PDA

View Full Version : HTTP headers

crifer
23-02-2002, 21:52/09:52PM
Wow.. the new forum looks great, i had a hard time finding out when to post my question :) I still don't know if it's right to post it here :)

I have read alot about HTTP headers recently and web / proxy caches, how does SE handle does or do they have any effects at all on ranks or when sites get spidered?
ihelpyou
23-02-2002, 22:15/10:15PM
LOL Yep, that is the goal, to get members to crawl all over the forums to get familiar with them. I think it's working. ;) It won't take long to get to know the new setup. Just many more forums now with many forums off the front page.

Don't really know a good answer for you. Anyone? I really don't think they matter much for the spiders as they obey a robots.txt file and that's about it.
Alan Perkins
27-02-2002, 09:54/09:54AM
A massive topic Crifer!

Yes, HTTP headers impact how much a site is spidered. In short a spider will be looking for a 200 reply in order to index a page. 300 series replies can have interesting if unpredictable effects.

As for cache/procy issues, these are mostly features of HTTP/1.1 and most spiders are mainly HTTP/1.0 (although they send a Host field). But bear in mind that search engine indexes are, in themselves, caches so an indexing robot may choose to obey certain cache directives.
crifer
27-02-2002, 18:45/06:45PM
Yes, i come to think about that when i read about it. Is there any documented research or personal notice that you are really sure of? It have to affect the ranking or the re-spidering a little.

I wonder what a SE would do if you have a shell script that change the header every day. Would it noticed that the content is same?

I found headers quite intresting, if a SE use and go by the headers there is alot that you can control on the site and make it quicker to load once the basic design of the page has been downloaden.
Alan Perkins
27-02-2002, 19:00/07:00PM
I wonder what a SE would do if you have a shell script that change the header every day. Would it noticed that the content is same?Depends what you changed about the header. The short answer is yes, if it cared to try it would probably notice the content was the same.

if a SE use and go by the headers there is alot that you can control on the site and make it quicker to load once the basic design of the page has been downloadenYes. :)