To "Robots" or "Meta" - THAT is the question

[vinyl-junkie]

I posed this question on another message board, and all I got was a generic "how to use meta tags" type of response. Perhaps others here can offer more insight. Here goes:

I've found all kinds of websites that explain the basics of meta tags but not one that specifically covers my question. Here's some background.

I have several pages with dynamic content. Each page has a picture and some of the same stuff on each page (look here (http://www.napathon.net/BeeGees.asp) for one example). What I end up with is (shortening the URL here for readability) BeeGees.asp?page=2, BeeGees.asp?page=3, and so on.

I'd like to index the page itself once but not for each page. A "robots" meta tag of "NoIndex, Follow" will get me everything except the first page. I'm wondering if this robots.txt example will do the trick.
User-agent: *
Disallow: /BeeGees.asp?page=2
Disallow: /BeeGees.asp?page=3
etc. Will this work or is there a better way to accomplish what I want? :confused:

Thanks in advance.

[scottiecl]

Hi Pat-

I may not be understanding your question correctly, but it would be easier to put the files you do not want indexed in a directory by themselves, then disallow that directory.

Somehow, I don't think that is what you are asking though. Can you explain a little more?

[vinyl-junkie]

I was afraid I wasn't clear enough the first time. Let me try that again. I want to have the first page, http://www.napathon.net/BeeGees.asp and its links indexed, but from page 2 onward, I only want the links within the page indexed and not the page itself.

So in other words, on page 1, I want to have an "Index, Follow" but on all subsequent pages, I want to have "NoIndex, Follow" or an equivalent using my robots.txt file. Does that make a little more sense?

[g1smd]

The robots.txt file can be used for excluding individual files but is best used for excluding complete directories.

In your case individual meta tags inside each page would probably be best. However, make sure that you also put "index,follow" in all of the ones that you do want to be indexed, just so that the robot is totally sure what you want to do.

[qwerty]

Well, I don't consider myself an expert on robots.txt, but if I understand the question correctly, you'd like the spiders NOT to index pages, but to follow links from those pages, and you want to specify the pages for which this is done.

As far as I know, disallowing a file or directory in robots.txt will not just keep the spider from indexing a page; it will keep it from looking at it. Therefore, it won't be able to follow the links on it.

If that's what you want, you probably need to use the robots meta tag, and even then, I'm not sure if a setting of "no index, follow" is possible. But, if you put the pages into a site map, the spider will find them that way, and a "no index" on the robots meta tag on those pages should do what you want.

Can I ask why you don't want those pages indexed? Is it a matter of not wanting users to start on them?

[polarmate]

Generate your robots meta tag based on the value of your variable 'Page'
eg

pg = Request.QueryString("Page")
if pg >=2 then
'noindex, follow in your meta tag
else
'index, follow in your meta tag
end if

[vinyl-junkie]

Originally posted by polarmate
Generate your robots meta tag based on the value of your variable 'Page' Of course! This will get me exactly what I want! Gee, now that you mention it, it's so obvious, why didn't I think of it myself? :D

[vinyl-junkie]

Originally posted by qwerty
Can I ask why you don't want those pages indexed? Is it a matter of not wanting users to start on them?
I guess it probably doesn't make much sense unless you look at a couple of the pages. As for why I have some of the same things from one page to the next, when I was first building my website, I asked for some design advice on another message board, and I was told I had too much text and not enough graphics on my pages. So when I built my pages that display my music collection, I decided to have a featured artist and/or album for each section, as appropriate.

You could get into a whole philosophical discussion, I suppose, on what makes a good web design and whether mine is any good or not. Right or wrong though, this is what I've done.

[Alan Perkins]

Good advice from polarmate. :)

A small caveat to the strategy: the only part of the robots meta tag you can reasonably rely upon is "index"/"noindex".

"follow"/"nofollow" is very flakey. Links are not always followed in pages that contain a "noindex, follow" robots tag (or pages that don't, for that matter :D)

[vinyl-junkie]

Originally posted by Alan Perkins
"follow"/"nofollow" is very flakey. Links are not always followed in pages that contain a "noindex, follow" robots tag (or pages that don't, for that matter :D) Do you think I'd be better off using "index, follow" on those pages? I'd rather have redundant pages indexed than to have pages missing from the index. :confused:

[polarmate]

I believe index, follow is the default behavior, Pat.

Here's a thought: Your objective seems to be get the links indexed but the reason that you are going through the trouble of noindex/follow is that you don't want the same text indexed. Why not change that content based on the page variable, too?
Page 1 has a basic intro on Maurice Gibb
Page 2 has more on Maurice's music
Page 3 has some stuff on his involvement in social activities other than music
Page 4 has stuff on his sudden death
Page 5 has Barry Gibb and so on...

All you need to do is to create html pages for this content, map this to each page number and include the relevant page in the resulting ASP page.

Not only will all your links get indexed, you may also get the added advantage of becoming a kind of a web reference on the Bee Gees.

I know that you are not looking for advice of this nature but ...well, it's a thought!

[vinyl-junkie]

Originally posted by polarmate
I know that you are not looking for advice of this nature but ...well, it's a thought! Quite the contrary! I appreciate any sort of advice I can get. When you think about it, even though this forum is about search engines and that's certainly important, having a website that people want to spend any sort of time at is just as important, maybe even more so!

You've given me a great idea! I'm going to give it some thought and see if I can build some more interesting content for my site based on what you've suggested. Thanks! :D

[polarmate]

:cheers: Pat!!
Let us know how it goes!
:cheers: