PDA

View Full Version : how to exclude https page with robots


loki
30-03-2004, 09:01/09:01AM
can someone help me exclude a secure page i have from gg(currently using the hosting provider's digital cert)

the URL of the secure page is

https://cobra.xssl.net/%7Edomain.com/payment.html

actually anything under this root would be even better. i guess it would be something like:

User-agent: googlebot
Disallow: https://cobra.xssl.net/

TIA

g1smd
30-03-2004, 17:02/05:02PM
That last line should be:

Disallow: /

loki
31-03-2004, 04:23/04:23AM
as i understand it, that will disallow EVERYTHING on the domain from googlebot!!!

Hope
31-03-2004, 14:02/02:02PM
You are sending people off your site into a shopping cart?

If this is the case, you might have to rely on a robots meta tag of index, nofollow.

loki
31-03-2004, 15:39/03:39PM
IT'S A STANDARD PAGE REPURPOSED AS A SSL PAYMENT PAGE. UNFORTUNATELY GOOGLE GRABBED IT, FOLLOWED THE LINKS AND NOW I HAVE 'DUPLICATE' (oops) pages starting with https.

i've added noindex,follow and re-written all the links as absolute links with http prefixes.

i just thought that adding the exclisuon to the robots.txt file might speed things up.

g1smd
31-03-2004, 18:47/06:47PM
Yes, true, but the robots.txt file goes in the root of the domain, and specifies which folders and files are to be excluded. You cannot use the robots.txt file to state a domain for exclusion.

Disallow: https://cobra.xssl.net/ is not valid code.

Putting Disallow: / in the root folder of cobra.xssl.net is valid, but if that isn't what you want to do, then you are using completely the wrong thing here.

loki
01-04-2004, 03:37/03:37AM
sounds like you know what you're talking about!

the server i am on for this domain is called 'cobra' hence when i use their digital certicate the ssl page has the https://cobra.xssl.net/%7Edomain.com/payment.html URL but conbra is not my domain name.

i think i'll go with the noindex,follow tag for now.

thanks for your help.