ASP.NET 500 errors for search bots

[flipperweb]

Hi,

I'm building a dynamic ASP.NET based website. I have had MSN, Yahoo and google crawling it for months. Over the past couple of days I'm seeing in the logs loads of 500 errors when the bots are crawling.

the site is:
http://www.snowmad.co.uk

The home page seems to be ok in the logs its when following the gear and resorts links.

When accesing from IE, Firefox the URLs requested are fine. I've installed the lynx browser too as apparently this is a lot like how the bots see the site.

Can anyone help. I can't think how to debug this issue as I'm not getting the errors.

Many thanks

Alex

[chrishirst]

and the logged errors are ?


does your code use sessions, or postback, or viewstate anywhere?

BTW I'd get rid of the single pixel link that is before the doctype It's of no use whatsoever.

[flipperweb]

Originally posted by chrishirst
and the logged errors are ?


does your code use sessions, or postback, or viewstate anywhere?

BTW I'd get rid of the single pixel link that is before the doctype It's of no use whatsoever.

Cheers Chris..

The errors in my logfiles were only 500. Anyhow no use of sessions, postback or viewstate.. I think I've cracked it.. I was looking around other forums on this site and learnt of webbug.. Great little tool allowed me to access the site not from a browser.

Well it looks like if you access the site using http 1.0 I get an error. My code was inspecting the host header which wasn't as expected when being accessed using http 1.0 rather than 1.1.
(using Request.Url.host)

That piece of code was redundant anyhow so I've stripped it so all ok now...

The single pixel link is actually pumped in by my redirector. I have a number of sites on the same IP address and I have an ASPX page which detirmines the URL being requested and does a server transfer. I wasn't sure whether the bots would like that so I added in a link to the site requested also..
thanks.

Alex

[g1smd]

The "500 Error" occurs when a server is misconfigured, the server is overloaded by requests or something unexpected happens in a script.

Those errors can be very difficult to track down. WebBug is a good help, but well done on finding and fixing the problem.


As for your HTML code, the DOCTYPE must be the very first line of code on the page. There must be no HTML above it at all. Ever.