Submitting to Inktomi?

[Jeff]

Hello,

I just did a searh at http://search.positiontech.com/ using my domain name and it came up with 13 different links that point to my site and have a few questions.

1. Since my site is already showing up in Inktomi, do I really need to submit it? Half of the content (Title and Description) is from pages I had up about a month ago and the other half is recent.

2. I have about 100 pages on my site with about 10 main sections then the rest are sub-pages. If I should submit my site to Inktomi, would it be the same if I just submitted mydomain.com or is it better to pay for each url and do mydomain.com/page1.html then mydomain.com/page2.html?


Jeff

[ihelpyou]

Sure, each Url you pay for is assured of getting crawled and indexed. If you don't pay, some pages will be indexed and some will not. The benefit of paying is that you are assured of being indexed and assured of getting re-spidered every 48 hours.

You are however, NOT assured of any kind of ranks. Your site has to be optimized/ready for that. If not, no amount of paying or indexing will do you much good.

[MsSearch]

If you already have pages in Inktomi, then i wouldn;t pay to submit unless you want the 48 hr re-spidering feature.

As Inktomi (a member here) stated in another thread, "The entire index is refreshed every 14-21 days" so if you already have pages in the index, I wouldn't pay for them. I would pay for other pages within your site that you want to appear in their database that aren't already showing up.

[Jeff]

Hi Doug,

Thanks for the info. I have one more question for you. I was just reading over positiontech.com's FAQ and under the technical guidelines it says:

The Web page must permit so called "spidering" technology, such as not using a "robots.txt" file.

I've always heard it was good to use a robots.txt file to prevent certain pages/directories from being spidered. Should I get rid of it? I have about 8 directories that I don't want to show up. I don't have any links to these sections on my site so will the SE's pick them up?


Jeff

[Jeff]

Hi MsSearch,

Thanks for your reply. I don't really have a lot of money to spend so I would like to concentrate on submitting it to spiders/indexes that my site does not show up in, but want to make sure it is showing up good in the most popular SEs.

Even though my site showed up 13 different times in the URL I posted above, I tried looking it up on hotbot.com using: originurl:http://www.mydomain.com and originurl:http://mydomain.com and nothing was found. Do you know why it would show up on positiontech's search but not hotbot's since they use the same database? I guess I should at least submit my main URL since it isn't showing up in hotbot.


Jeff

[potato]

I have about 20% of my more important pages Inktomi-listed with no difference in search.positiontech.com compared to hotbot.lycos.com.

I am using a robot.txt file which excludes a few directories which are in develpoment state. My logfiles tell me that slurp, the inktomi spider passed by without trying to spider them.

in contradiction to the cited Inktomi guideline which seems to disallow robot.txt files they publish the opposite at http//:www.inktomi.com/slurp.html (http://www.inktomi.com/slurp.html)
especially: * robots.txt: Slurp obeys the Robot Exclusion Standard. Specifically, Slurp adheres to the 1994 Robots Exclusion Standard (RES). Where the 1996 proposed standard disambiguates the 1994 standard, the proposed standard is followed.

Since this seems to make sense, is standard and applies to my logfile watches I simply believe that the cited Inktomi guidelines contain a typo.

[Alan Perkins]

Originally posted by Jeff
The Web page must permit so called "spidering" technology, such as not using a "robots.txt" file.
They just mean if you exclude the page using your robots.txt file, the page will not be spidered. i.e. don't exclude Slurp from reading a page using robots.txt, then pay for inclusion of that page.

BTW you might try using this Search Engine Site Lookup Script (http://www.searchmechanics.com/look/look.htm) to see the pages you have indexed in Hotbot and other engines.

[Advisor]

If your page(s) are already in the inktomi database as seen at PositionTech or Hotbot, then I wouldn't pay to be included. It won't boost your rankings.

Jill