I’ve been playing around with the Google Webmaster Tools and have noticed one thing on the site verification process, that is not present on the Diagnostics/Crawl Errors tab…
Last attempt [date]: We’ve detected that your 404 (file not found) error page returns a status of 200 (Success) in the header.
Google will be unable to verify your site [for this service] if it is setup in a way where all URLs that do not exist, redirect back to the main page. But more importantly, from the look of things, this also has an effect of receiving a penalty and/or keeping your site out of the Google index.
DynamicSide.NET was like that, and Enotarize.com was even worse, not redirecting back, but rather just displaying the main page on any and all URLs [what can I say, its a very simple site, pre-beta].
And I was wondering why I have not been seeing much traffic on those two from Google search.
With DynamicSide.NET, there seemed to be some type of a penalty [nothing under the webmaster tools, but I have a feeling something is going on]. And with Enotarize.com, it looked like it was not being indexed at all, for some time. I’m sure that also had/has an effect on DeveloperSide.NET, as I’m linking back to both sites.
On another note, content that did exist on your site at one point, that has been removed, should return HTTP Status code ‘410′ “gone”, over the ‘404′ “not-found”.
And what about a situation where content has moved URLs, yet there is no simple way for you to redirect… A good example of this situation is under our Forums, with the switch from phpBB to SMF. They both have non-compatible ways of associating URLs to posts and topics. Right now I’m just redirecting everything hitting the old phpBB ‘viewtopic.php’ URLs to the main forum URL. Perhaps I should be using a ‘303′ “see-other” code…