Inktomi spiders

[ifish]

Is there a difference between the spider for the "pure search" and the one for the paid inclusion? I've got three different slurp spiders from inktomi in my logs hitting the site. I only have one page "paid include" and am trying to decide whether or not to include others. Obviously, if the "free spider" is hitting the site i will not. thanks for the input

[Hope]

I cannot answer your question since I don't pay for inclusion. I do have 3 or 4 different slurps hitting my logs on a weekly basis.

Ad far as I know the paid inclusion spider will only hit the pages that are being paid for. If this is infact true, then any slurp that is hitting other pages are not the paid spiders. You are getting other pages spidered for free and then you have no need to pay for them to be spidered.

Does that make sense?

[JuniorHarris]

Inktomi has a number of spiders with different UserAgent tags. If you post a couple of samples I'm sure someone (read MakeMeTop) can answer those questions.

I know we have had the PFI spider and the free spider read our site...I just don't remember off hand which was which, and I don't want to post any misleading information. However here is a list of UserAgents I have from Inktomi: Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)

[ifish]

thanks.
1) HOPE- it does, but we paid toinclude so many pages that it's almost impossible to determine a difference.

2) junior- thanks. inktomi agents that I see in our logs for last month are:
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)


ANyone have any idea if any of those are free vs. paid or if there is a difference?

[JuniorHarris]

I was hoping Barry would chime in...I suppose I can always guess!~;)

Slurp/cat; - PFI (paid for indexing) crawler
Slurp/si; - Free range crawler
Slurp.so/goo - Free range crawler


After checking the logs, I'm fairly certain that /cat is the PFI spider, and the other are the range free spiders. Of course would love to hear another's opinion....

[nuthin]

Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) - is goo.ne.jp inktomi spider which crawls for free when you submit sites into goo's index.

Don't know about the rest or the PFI spider. But im pretty sure one of the ones above is hotbots inktomi spider.
As we get 2 inktomi hotbot spiders crawling our sites alot one is goo's and the other one i assume is Hotbot's since we submit to hotbots index for free.

[MakeMeTop]

Been away, but I agree that after PFI you will see Slurp/cat which is the PFI spider. You will then get hit sometime afterwards by Slurp/si which checks for inbound/outbound links to the PFI page submitted to ensure it is not an orphan page - but is also a free crawler.

Slurp/so is the main free crawler.

[JuniorHarris]

Thanks for the update Barry...welcome back, hope you had a wonderful vacation!~ :)

[ifish]

Thanks for the info. Now here's a question. I see the PFI spider slurp/cat and the link spider (which comes as a result of the PFI) slurp/si. I have also been spidered by slurp/goo. I have never seen any non-pfi pages show up in inktomi results. Do you know (or what do you think) if because some of my pages are PFI, that "general" free spidered pages are not included. What is mean is, since the PFI spiders have my main domain, do you think that the "free" spiders which are gathering non-pfi sites ignore mine (and anyone who is using PFI)? I'm trying to determine the value of PFI. It certainly gets you in faster, but if it can be done for free a la Google, and if the PFI prevents free spidering, maybe we're all better off skipping the PFI.

Trying to figure this all out and it seems to just make my head hurt. Just when i get that rock to the top of the hill, zoom, it rolls back down right over me!

[JuniorHarris]

PFI is great for constantly changing content...as to keep the listings current.

It is possible to have non-PFI pages included, it just seems to take a long time. That said, I've noticed a great deal of non-PFI inktomi indexing in just the past few weeks. It appears the latest update has indeed included non-PFI pages and PFI pages for us.:eyes:

[ihelpyou]

Inktomi seems to be picking up new Non-PFI pages pretty darn quick these days. Sometimes within 2 weeks or so. That ain't too bad. :)