Robots Text

[MHutch77]

When it comes to robot text files, could you try and use an "allow" command to have a search engine look for a file or link similiar to how the disallow "suggests" (as of course not all listen to the robots text) that they stop looking in that direction?

I have been curious and reading some posts on this thread brought the thought back to my mind?

[MazY]

The "disallow" is, as you suggest, there to help prevent pages that you do not want indexing from being indexed.

An "allow" is not required as if the spider detects no "disallow" against a page then it will assume that it is ok to index it.

Sadly, we have little control over how the spider indexes and how much it indexes (other than making their journey through the site as easy as possible).

How much it indexes varies between search engines. Google and FAST are notoriously good at scooping anything up from your site that does not have a disallow next to it in the robots.txt.

Many state that Google is the king of the deep crawl but I contest that this title actually belongs to FAST.

What we need is a "mustgofetch" tag. :D or even better, a "PleaseFindItThisTime" tag.

PleaseFindItThisTime: /my-best-page.htm

[ihelpyou]

Welcome to the forums MHutch77! :hi:

[Blue]

:hi:

[MHutch77]

Thanks for the welcome. I've been reading people's posts for a while now, before this last question re-surfaced in my mind and I thought I'd ask.

Thanks for the answer by the way.

One other question on the same subject though, if using frames and say..you dont have the info in the <noframes> tag, could an allow tag be used to point the spiders in the right direction?

[ihelpyou]

I have no idea if that may work or not. I really doubt it. You do need to fill the <noframes> tag with some good optimized content. The spiders always give the robots.txt file a calling when they visit, so I do not think they would read the "allow" as such anywhere else.

[JuniorHarris]

MazY I believe FAST is indeed the king of crawl, with Google a solid second!!