Adsense and robots.txt

[Papadoc]

I have a client that wants to keep a couple of sections off limits to search engines because he doesn't want people coming in that way... long story. He is currently doing that very well with a robots.txt file.

But he would also like to add Adsense to this section as a default ad spot. Does anyone know if G can read these pages for content matching, but not index them if the robots.txt file excludes it?

[Bernard]

I like the new avatar!

If robots.txt blocks the Mediapartners-Google* bot, AdSense will probably just serve public service ads:

https://www.google.com/adsense/faq#basics2

[Papadoc]

Thanks Bernard. It's a one size fits all occasion avatar. Screaming, laughing, shock, horror, surprise, or taken just after my morning constitution... the morning after a night of fried jalapenos and margaritas... you decide. Emotional Rorschach testing I suppose.

Is there a way to disallow all bots but the Mediapartners-Google bot?

[Bernard]

I believe you can just add the following to the end of your robots.txt file:User-agent: Mediapartners-Google*
Disallow:

[Papadoc]

Wouldn't that do just the opposite? Allow all BUT Mediapartners-Google* when the idea is to allow ONLY Mediapartners-Google* bot? Or do I have something backwards here?
:confused:

[Bernard]

When you specify a specific user agent, the following instruction(s) pertain(s) to it only. The instruction published above is a blank disallow (which means that the entire site is crawlable for that agent). IIRC, you can precede that text with:User-agent: *
Disallow: /
Which would disallow spidering of the site to all spiders (except the ones which have specific instructions like Mediapartners-Google*).

[polarmate]

I believe you can just add the following to the end of your robots.txt file
It has to be at the beginning of the file.

See: http://www.robotstxt.org/wc/exclusion-admin.html - see the section "To allow a single robot"

The correct code would be:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /

Interesting avatar, PapaDoc!!

[Bernard]

Thanks Manisha!