Can someone please explain the "logic" behind this??? I know bots that are compliant are supposed to ask for robots.txt file -- BUT, have you ever heard of one that wanted to INDEX just your
robots.txt file?
Honestly, this is useless and IMO it shouldn't be indexed in any friggin SE!!!
http://botseer.ist.psu.edu/
Found that "bot" hitting several of my sites today. It can't get anything from my sites though because I use a whitelist via .htaccess and it was served a nice big fat juicy ACCESS DENIED message.
Just thought I'd give the rest of you a heads-up on this one if you hadn't already noticed it hitting your site.