Bernard
14-09-2005, 13:48/01:48PM
I've set up a vbulletin forum on a new site and excluded a ton of pages and directories with robots.txt.
User-agent: *
Disallow: /cgi-bin/
...
Disallow: /online.php
Checking Who's online, I see MSN Bot requesting pages that have been disallowed such as the the who's online page. Has anyone else noticed something similar?
User-agent: *
Disallow: /cgi-bin/
...
Disallow: /online.php
Checking Who's online, I see MSN Bot requesting pages that have been disallowed such as the the who's online page. Has anyone else noticed something similar?