Does Looksmart have a spider?

[Mel]

Hi All:

today something came crawling around one of my sites according to my stats. It was under the guise of a Mozilla4 compatible browser. but it resolved back to http://wallaroo.looksmart.com. He gobbled up 31 pages and quietly departed. Anyone know who or what this might be?

[ihelpyou]

They may have a spider that simply checks periodically if a site is still online. That particular agent is a human from Looksmart. They may have followed a link from here to visit you. :)

[ihelpyou]

I was wrong Mel. Go figure. :) That is definitely a spider that Looksmart sends out. It hits these forums on a daily basis.

[JuniorHarris]

Mel did it hit pages which were not already listed in Looksmart? :eyes:

[Mel]

Hi JH
It spidered all the pages on my site recently listed in looksmart.

[JuniorHarris]

So it may just be verifying the pages as Doug suggested, but I also wonder if it is pulling any additional information.

[ihelpyou]

Yes, leads me to believe that not only does LS give us a category of keywords, BUT also has some kind of algo working and spiders a page occassionally to adjust ranks accordingly. Kind of like a real search engine. :eek:

We/I have wondered a little about this.