Inktomi takes a long time to deep spider

[usbnuts]

Does anyone here know the behaviour of Inktomi deep spidering?

I see deep spidering as the ONLY way to boost more traffic into my site as it has a wide variety of topics.

Inktomi takes A LONG TIME to deep crawl. I don't know why.

[Mel]

HiUSBnuts:

Did you use the inktomi paid submission? If so they tend to crawl only the pages you paid for.

[usbnuts]

inktomi paid submission will call inktomi spider to index our pages every 48hrs, but its spider also crawls other part of the site if you wait long enough. However, you just don't know when the crawler will come.

[ihelpyou]

The spider from INK that crawls your site is not the same one that recrawls your paid pages every 48hrs. The 48hr spider will only index the page you paid for.

The one you are seeing crawling comes around randomly. It usually simply follows a link from somewhere else. Either another web site or a directory. It will also index your pages.

This is another reason I have not paid for Ink as of yet. Until I feel the need, no point in doing so.

[Mel]

Hey Doug:

Do the different INK spiders have different names and if so do youknow them?

I would like to be able to track who is doing what and when.

[ihelpyou]

inktomisearch , etc with different numbers are the only ones I know as no sites pay so do not know the paid agent for Ink.

[JuniorHarris]

Paid UserAgent:
Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

[ihelpyou]

Thanks JH!

I bet it is taking you a long time to catch up to all the posts of the last week. :)