Inktomi spiders...

[ifish]

Does anyone know how the free Intomi spiders work? Here's the question- do the spider through your entire site, a la google, so that all of your pages can be included and can theoretically show up individually for certain search phrases, or do they only spider the home page to include that into the database? The reason i ask is that we have a large site, all of the pages of which are spidered by google (so we know they are spiderable). The free spider from Ink only hits our site a few times a month ( as opposed to the tens of thousands like google). We currently use PFI to make sure many of our internal pages are included in MSN results, and are considering dropping it. It just dawned on me though that although ink spiders for free also, internal pages may not be presented in the results. ANyone with any ideas, opinions, experience, etc.?

[bigDugan]

I think you may of already asked this question in another thread (http://www.ihelpyouservices.com/forums/showthread.php?s=&threadid=4640&highlight=Inktomi+spiders).from JuniorHarris:
It is possible to have non-PFI pages included, it just seems to take a long time. That said, I've noticed a great deal of non-PFI inktomi indexing in just the past few weeks.from IHelpYou:
Inktomi seems to be picking up new Non-PFI pages pretty darn quick these days. Sometimes within 2 weeks or so. That ain't too badSo, the answer seems to be YES.

[ifish]

Sorry BD if i seemed redundant but i i thought of it in a different way. I did see about Ink adding new pages, but i had never thoughtabout whether those pages were "deep" pages or simply the main/index page. i know that google is great at deep indexing and am simply trying to decide whether or not to continue to pay for our inner pages to be spidered, or whether ink will do a similar job as google, and show some of our inner pages to search results. again, i apologize if i was redundant- i actually thought of this as a different idea.

[bigDugan]

No problem. It just appeared to be the same question to me. About the only suggestion I have, is maybe cancel one of your PFIs and see if that page gets indexed by the free spider. To me, it would seem redundant for the free spider to index any pages that were already spidered by the PFI spider. SO, maybe you wouldn't see the free spider on those pages. Maybe someone else can shed some light on that?

[ihelpyou]

3,4,5 bots at a time have pounded these very forums for about 4 days straight. They are crawling deep and grabbing any and every page they can.

I'd say they are getting aggressive and wish to be very relevant in the months to come.

[ifish]

Thanks both BD and Doug. I'll be interested to know if these deep crawled pages get indexed. BD- yeah, we pretty much decided to let the pages drop, but you understand the worry- it's hard to get off the nipple, if you know what i mean.

[bigDugan]

>> it's hard to get off the nipple<<
Isn't that the truth! But I think when the kid turns 16, it's time to stop breast feeding! :eek:

Best of luck to you! I hope they do get crawled, it may just be a matter of time.