SEO Best Practices Search Engine Forums  

Go Back   SEO Best Practices Search Engine Forums > Web Site Promotion and Marketing > Web Site Promotion, Hosting, and Servers > Bad Robots

Bad Robots Post any rogue robots you see crawling around. Find out ways to squash them and kill them.

 
 
Thread Tools Rate Thread Display Modes
Prev Previous Post   Next Post Next
  #1  
Old 26-09-2007
WebSavvy's Avatar
WebSavvy WebSavvy is offline
retired from the web ...
 
Join Date: Dec 2001
Location: somewhere out there ...
Age: 49
Posts: 17,908
Rep Power: 20
WebSavvy is on a distinguished road
BotSeer?

Can someone please explain the "logic" behind this??? I know bots that are compliant are supposed to ask for robots.txt file -- BUT, have you ever heard of one that wanted to INDEX just your robots.txt file?

Honestly, this is useless and IMO it shouldn't be indexed in any friggin SE!!!
http://botseer.ist.psu.edu/

Found that "bot" hitting several of my sites today. It can't get anything from my sites though because I use a whitelist via .htaccess and it was served a nice big fat juicy ACCESS DENIED message.

Just thought I'd give the rest of you a heads-up on this one if you hadn't already noticed it hitting your site.
Reply With Quote
 

Bookmarks


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



All times are GMT -4. The time now is 06:44/06:44AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Copyright ©1998 - 2010, ihelpyou, Inc.