Seeing 304-Error codes in the logs

[Advisor]

I just received an email from someone who's recently started seeing 20% of her traffic getting 304-errors (not 404).

Her server people told her this: 304 Error codes mean that the visitor's program checked to see if that specific URL had been modified, and your server responded "304", no the URL has not been modified from the date you provided. These requests are generally done by cache servers, like the ones from AOL. They can also come from many search engines that are just checking to see if the page has changed. So the source of that traffic that is getting 304 codes are cache servers, proxy servers and search engine bots.
But it seems awfully strange. I just checked my server logs and I don't have any 304's. Does anyone know much about these errors and the real reason why a site might be getting them. It is apparently something new happening for this site.

<added>Just got this from her also:

Here's what my stat's indicated for November 2002...to help you visualize
the issue.

Hits by Response Code

Code 200 - OK 91163
Code 206 - Partial Content 425
Code 301 - Moved Permanently 17
Code 304 - Not Modified 24044
Code 400 - Bad Request 92
Code 401 - Unauthorized 149
Code 403 - Forbidden 11
Code 404 - Not Found 5584
Code 405 - Method Not Allowed 2
Code 408 - Request Timeout 30
</added>

Thanks!

Jill

[Dan0]

Without seeing the logs, I can't tell you a lot, but you can track these things down pretty readily in most cases by doing reverse lookups etc.

It could be a spider, a cache/proxy server, or someone using offline browsing in MSIE etc. Most likely a spider, but could also be a local proxy server from the hosting provider or their upstream.

[dvduval]

Jill, I don't think Not Modified is considered an error by WebTrends. I'm pretty sure the way Webtrends reports 304 is under "cached hits" in the technical statistics. My Webtrends report doesn't actually list 304.

On the other hand, on another site my Webalizer report does list 304's. Here is one such report.

Code 200 - OK 12495
Code 206 - Partial Content 12
Code 301 - Moved Permanently 3
Code 302 - Found 79
Code 304 - Not Modified 6204
Code 400 - Bad Request 6
Code 404 - Not Found 679
Code 500 - Internal Server Error 14

[dvduval]

And a report from another log:

233978: 200 OK
222: 206 Partial content
46: 301 Document moved permanently
1326: 302 Document found elsewhere
61722: 304 Not modified since last retrieval
45: 400 Bad request
30: 401 Authentication required
21735: 404 Document not found
14: 500 Internal server error

[Dan0]

Jill, you won't see a lot of 304's if you:

1. Keep your site so up to date that every document has always been recently modified.
or
2. Have your web server sending out no-cache and expires headers, so that proxies aren't keeping the pages.

[dvduval]

2. Have your web server sending out no-cache and expires headers, so that proxies aren't keeping the pages.

Dan0,

In relation to Googlebot, are there any advantages to doing this?

Also, would this have speed disadvantages for some users?

[Advisor]

Thanks guys! It sounds like it's nothing to worry about then. (Like her server people told her.) I'll pass the info along.

Jill

[Alan Perkins]

304 is a good valid response, meaning the visitor saw the Web page they requested (without being redirected).

304 is a server response to a client If-Modified-Since request and it means "Not Modified". When the client receives this response, it knows it can safely use its cached version of the requested resource.

Googlebot has recently started sending If-Modified-Since fields in its HTTP requests, presumably to save bandwidth and indexing time.

[Dan0]

David:

As I understand things, Googlebot ignores "no-cache" headers. You have to use the META tag to keep from getting cached at Google.

Preventing proxies from caching your pages does have speed disadvantages, especially when the 'net is slow. But it's all relative. If a page may take an extra second to load, is that a disaster? Only if you've got a slow-loading page to begin with.

The main reason you'd use these headers is when you have a dynamic site where the same URL returns different information to the user based on the state of their session. For example, we have a custom-built PHP purchasing application, where because of the requirements, we have to POST form data.

This means that when you add something to your shopping cart, it takes you to the same URL again, instead of something like "cart.php?session=1234&command=add&item=123."
If we let anyone cache those pages, the shopping cart would break.

Same thing applies if your home page is dynamically generated, e.g. a news portal like Slashdot. Off the shelf content management applications like PHP-Nuke send out the right headers automatically, so it's not always a server configuration thing.

[dvduval]

Thanks, DanO

Your a gentleman and a scholar!