New structure

[WebSavvy]

Over the past several months I've spent a lot of time restructuring the directory, and making a big push toward WAI compliance.

Currently, our category URLs are mixed case (upper & lower)
Example: Category-Name

In our new setup, the category URLs are all lowercase;
Example: category-name

Sending old URLs to new URLs is easy enough (301).

However, we have categories that are empty either due to lack of submission for that category or lack of quality in what has been submitted to that category.

Google has indexed all of our empty cats (and full ones too).

In our new setup, empty cats are disabled.
So, now we have the problem:

Using 301 to send old to new
Disabled cats would be old to new (301)

But, 301 them ... where?

I've been speaking with a few others regarding this, and it's been suggested to 301 the disabled child category to it's enabled parent.

At first, I thought this would be a good idea, but ... it creates a new issue.

It's telling the SEs that the old category which has a new URL, but is now disabled that it has moved permanently to the parent category.

When that child category does become enabled, it might not get indexed because we've told the SEs it's moved permanently.

Now, here's what I'm thinking might be a better idea?

I'll create a page that's simply called :: disabled.php
Then 302 all old (now disabled) URLs to this page.

Then when the disabled child category has listings, we would simply remove the 302 to the disabled.php page and 301 it to the proper category URL.

Is this the best way? Does anyone else have any suggestions?

[SEFL]

I'm not sure if this can be done in PHP, but it can in ASP so I'd imagine it could.

But here's how I'd handle it:

Gather up, using PHP, a list of all the empty cats/subcats/etc. and their associated URLs.
Open up your robots.txt file and modify it on the fly to disallow those URLs.
Wait until Google deindexes them.
Repeat this process, removing and adding URLs as categories gain or lose content respectively.

That way, there's no 301 issue at all, and if the categories do need to be readded you can do so. I might combine this with a 302 just to be safe, though.

[WebSavvy]

Google doesn't like to "deindex" any of my files. Been that route before.

Also, I've had many files disallowed in robots.txt and Google ignored it.

The files disallowed by robots.txt would show up in Google as
URL-ONLY files (e.g., no titles, descriptions, content).

I don't even want the URL-ONLY version there, as it'll no longer be a valid URL.

I already have the list of cats & urls.

I'm thinking 302 to disabled.php would probably be the best way to move forward with this.

Understand that we will be disabling around 3600 categories. Adding that many lines to robots.txt would make it too HUGE.

[SEFL]

Okay, then I've got nothing else. You used up all the ideas I had. :)

[Dave Hawley]

Deb, if G is ignoring your robots.txt file I doubt you will get into any hot water with them.

About the only other way is to add
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
To the pages.

[g1smd]

Debs, you're over-thinking this one.

The redirect from old sub-folder to new parent-folder should be a 301 redirect, in order to get the old folder out of the index (that may take a long time, but it does not matter, as the redirect will dump visitors to the right place on the new site).

When the new sub-folder eventually becomes active, then the links from the new parent-folder will get it indexed. The lack of 301 redirect from the old sub-folder to the new sub-folder will be irrelevant at that time.

Don't use any 302 redirects. Those are very likely to cause a whole load of trouble that you won't be needing.

[WebSavvy]

OK, g1. Thanks. :)

Thanks for the comments too, Dave and Adam. :)