Robots Meta Tags

[Hope]

Hey Folks,

I know I haven't been around in ages, but I have a quick question for you.

Do these robot meta tags work again? I know for a while they were a waste of my time. Is this still the case?

Thanks

[Connie]

Hi Hope. :hi: Glad the lost sheep found her way home.

The robots meta tag has always worked as for as I know.

If your asking about the meta description, and or the meta key word tag, you will get different opinions.

Whether the SEs put any weight on them now, who knows what they might do in the future? So they have always been important to me.

In regard to the meta key word tag, Yahoo has suggested putting misspellings of your key words there, so at Least Yahoo is using the key word tag for something.

The main thing is to use all tags and attributes properly.

[ihelpyou]

:hi: Hope!

[g1smd]

Major search engines obey the robots protocol. Note that there have been several extensions to the existing protocol in recent months.

There are many bad bots (e.g. scrapers) that do not obey the robots protocol. You need to take additional measures for those.

[Hope]

We host over 400 websites. Is it worth my time to put the robots index, follow tag on each and every page of these sites?

[g1smd]

You only need the noindex tag on pages that you do NOT want to be indexed.

The default action if you say now't is that they WILL be spidered and indexed.

[Comeran]

Like G1 said, you don't need anything on pages that you want indexed. You only need to include the noindex tag on the pages that you don't want indexed everything else should be fine.

I haven't seen you since the long and infamous "I am furious" post.

How have things been?

Com-

[Arnabme]

I think the Robots.txt file get preference ahead of the robots meta tag so why not pull in those pages in a single folder and then put it in the Robots file.

[chrishirst]

Originally posted by Arnabme
I think the Robots.txt file get preference ahead of the robots meta tag so why not pull in those pages in a single folder and then put it in the Robots file.

The robots.txt doesn't get preferential treatment as such, but it is read prior to any indexing, so items excluded there are not indexed.

With the robots meta element, the page has to be crawled and indexed before the bots "know" they shouldn't be indexing that page.
Also they will continue to request the page just to check that it still shouldn't be indexed.

Of course this only applies to compliant bots/UAs

[g1smd]

Items exluded using robots.txt can still appear in the SERPs as URL-only entries if anything else links to that URL.

The robots.txt exclusion stops the page being spidered but does not stop the search engine from simply recording that the page exists.



Using the meta tag, the page will be spidered (in order to read the tag), but nothing will be indexed for that page. It will not appear in the SERPs.



So, the robots.txt exclusion is more about saving bandwidth on the server, and the meta tag is more abot keeping pages out of the index.