Robot vs crawler

[whl626]

May I know the difference between robot and crawler ?

[ihelpyou]

No difference. :)

That is if you are talking about the 'robot' that search engines use. If talking about "The Jetsons" robot kind of thing, well then, that's a different story. One day we all may have our own personal robot to cook dinner for us. :D

[whl626]

In fact, I am talking about search engine.

[qwerty]

Search engine robots and crawlers are the same thing. They're also called "spiders".

[icwcorp]

You're not talking about the robot.txt file you put on your website are you?

[whl626]

icw, when I check my site stastictics it lists out the number of links to my site from robot or crawler. In fact, I don't know what they are all about.

To my knowledge, they are kind of software used by various search engine to determine if a site has good content or not to be placed in front of the search list :D

By the way, what is robot.txt ? Sorry, if my question sounds silly. I am a technical bubble-brain anyway :p :p

[qwerty]

Robots, or bots, spiders, or crawlers, are used by search engines to run through web pages, indexing the content and following the links. It's how search engines find pages, and how they find out what's on those pages.

A robots.txt file is a text file you put at the root of your web site to tell robots if you don't want them to crawl through specific sections of your site. You can get more information on all this here: http://www.robotstxt.org/wc/robots.html

[whl626]

Very informative site with good examples :)