Robots Tag Vs. Robots Text File

[agatlin]

If I use the robots meta tag, do I really need a robots. txt file. (I want everything indexed. My robots tag is set to all.)

Any suggestions?

[Hope]

Hi Anthony,

If you want everything indexed, then there is no real need for either. The robots are set to index all by default. The robots meta tag is pretty useless recently. If you have cgi on your site, I would recommend you adding that to robot.txt for exclusion.

[Advisor]

Yep, I agree with Hope, you don't need to do anything if you want everything indexed.

Jill

[agatlin]

Good. That is excellent info.

[lots0cash]

If you want the spiders to have complete access to index your site. You might want to consider a blank robots.txt file. I did this a while ago (based on the advice of Alan, Blue and some others) and so far it has worked great.

[superplay]

So is the robot.txt file only to disallow certain pages from being indexed?
Is it placed in the cgi-bin?
Does the robot.txt file keep a log or do you need a program for that?

[ihelpyou]

Welcome to the forums superplay! :hi:

No, the robots.txt is put into your first level of the site. Example:

yourdomainisgreat.com/robots.txt

You can do whatever you wish to do with it. Disallow certain directories or pages, or allow them or both. More info here:

www.robotstxt.org

[seowheel]

If I want all pdf's on my site to be excluded should I put up a robots.txt page and have it look like this:

User-Agent: *
Disallow: /pdf

If I want a specific pdf file not to be found would I do this instead:

User-Agent: *
Disallow: /nameofdoc.pdf

Do I need to put anything at all on the page that I want excluded- or just have it on my robots.txt file?

Just want to make sure I'm on the right page.

[Advisor]

You could put all pdfs in a directory called pdfs and then exclude them this way:

User-Agent: *
Disallow: /pdf

I'm not sure about the other things though as robots.txt is definitely not my specialty. I will defer to the others here to answer your questions more definitively!

Jill

[Alan Perkins]

Jill's last post was slightly inaccurate. :eek:Originally posted by seowheel
If I want all pdf's on my site to be excluded should I put up a robots.txt page and have it look like this:

User-Agent: *
Disallow: /pdfThat will disallow any URL which, starting from your root directory, begins with the letters "pdf", e.g. "/pdfinfo.htm", "/pdf/file.pdf", etc. Assuming you put all your PDFs in a folder called "/pdf", the command you want is

User-Agent: *
Disallow: /pdf/Note that slash after the pdf.Originally posted by seowheel
If I want a specific pdf file not to be found would I do this instead:

User-Agent: *
Disallow: /nameofdoc.pdfAssuming that the nameofdoc.pdf file is at the level of your domain, yes. If it was in your pdf folder, you would use the following code to just exclude that one file but allow the other PDFs in the folder to be indexed:User-Agent: *
Disallow: /pdf/nameofdoc.pdf
Originally posted by seowheel
Do I need to put anything at all on the page that I want excluded- or just have it on my robots.txt file? The beauty of the robots.txt file is that you don't have to put anything on the page - which is useful when it's not a page you are trying to exclude, but say an image or a PDF.

Conversely, the beauty of the robots meta tag is that you don't have to put it in the robots.txt file! There are certain instances when this really helps, e.g. in highly dynamic environments, or when you don't have access to documents at the root level of your domain.

If you're really interested in this stuff you may want to read my White Paper on The robots.txt file and the robots meta tag (http://www.ebrandmanagement.com/whitepapers/robots2.htm).

[Advisor]

Jill's last post was slightly inaccurate. Why am I not surprised! I should have just waited for someone who knew what they were talking about, to answer it!

Thanks, Alan!

Jill