Mel
22-09-2001, 02:59/02:59AM
Hi all:
Most of us are aware of using the robots.txt file for controlling which spiders get access to which directories on our sites, but the problem of banning those unsavoury or malicous spiders requires a lot of research.
Brett Tabke at WebMasterWorld has compiled a very complete robots.txt file at http://www.searchengineworld.com/robots.txt and allows free use of it.
If you wonder why this is necessary read Bretts explaination at http://www.searchengineworld.com/newsletter/2001/
Most of us are aware of using the robots.txt file for controlling which spiders get access to which directories on our sites, but the problem of banning those unsavoury or malicous spiders requires a lot of research.
Brett Tabke at WebMasterWorld has compiled a very complete robots.txt file at http://www.searchengineworld.com/robots.txt and allows free use of it.
If you wonder why this is necessary read Bretts explaination at http://www.searchengineworld.com/newsletter/2001/