MSN BOT is crawling to much

[lundens]

Is any else experiencing too much crawling from MSN-BOT? My log files are getting out of hand because MSN-BOT is crawling dynamic application parts of my site like crazy. After cleaning up log files a few days ago, I now have another 60,000 hits by MSN bot. It is basically crawlling an interactive map application following links which will basically allow it to traverse the entire world over and over.

My site is cold fusion and fusebox so I don't have any subdirectories to turn MSNBot off at. Everything is going through index.cfm. Any ideas on how to control this bot without completely eliminating my listing from this engine?

Thanks,
Adam

[Trafficdeveloper]

Why do you want to get rid of it? Check to see how your rankings do in MSN. This happened to me a few years ago, but with a Googlebot, and PR and rankings jumped nicely after several months of extreme spider activity.

Or you could email msn and tell them which bot is going haywire on your site and tell them to reign it in.

[lundens]

The main issues is 175,000 hits and 3 Gig of data just to support msnbot in May thus far is adding up to more than I can afford to use. I think I have a 10 Gig Bandwidth hosting plan and typically never use half of it in a month. Also 25 Meg in server log files are more than I want to deal with a day as well.

I am trying to add the robots meta tag to have it not follow links on the interactive map application, and I will report back if it tops the infinite loop which the msnbot has got itself into.

I found the following thread of interest.
http://www.webmasterworld.com/forum97/62.htm

[WebSavvy]

MSNs 'bot swallowed 3 GIG bandwidth on your site for May alone? OMG! That's waaaaay too much!!!

[bwelford]

Currently I'm finding the MSN bot is spidering more frequently than the Googlebots on a regular basis. I wonder what they're up to? :confused:

[rustybrick]

What I have seen for may is:

SITE A:
Googlebot hits: 3.26%
Googlebot bytes: 11.97%

MSNbot hits: 1.50%
MSNbot bytes: 5.06%

SITE B:
Googlebot hits: 17.48%
Googlebot bytes: 21.26%

MSNbot hits: 4.06%
MSNbot bytes: 4.23%

Looks to me msn doesn't do such a bad job compared to Google in the hits to bytes ratio.

[lundens]

It looks like msnbot has slowed down today. Only a couple thousand hits today. I did not have my ISP turn them off, but did put in meta robots tags and send a email to the msnbot asking to please stop. I am going to turn back on logging for its IP and see if it is still crawling and obeying meta tags or if it has just stopped entirely.

MSNBot appears to be willing to dive much deeper into dynamic content. Having a site build in cold fusion and fusebox where everything goes through one entry page, this is seemingly a good thing. But the infinite loop that msnbot got itself into on m y interactive map application is not a good thing. Since longitude and latitude are part of my url variables, msnbot could theoretically have had to have quintillion hits to exhaust all the possibilities. I would like to see the useful results which come out of that database!

I'll keep you all posted. Thanks for the responses.

[GuyFromChicago]

Originally posted by bwelford
Currently I'm finding the MSN bot is spidering more frequently than the Googlebots on a regular basis. I wonder what they're up to? :confused:

IMO MSN is starting to deep crawl for the ucoming release of their new SE. I'm sure they're testing/tweaking a bunch before it goes live.

I've been seeing the same thing, the MSN bot is taking more from my site(s) than Google. Until about 2 weeks ago, that had never happened.