RelevantText.com

Making the most of server errors

January 11th, 2008 Jason Posted in geek, seo |

Nobody thinks twice about planning for and dealing with 404 errors on their website. It’s going to happen, right? Not because you didn’t properly redirect when you moved a page or something, of course! But you expect that somebody will mistype a URL someday, and you plan for it and have your fancy or funny 404 page in place on launch day.

I was reminded today that people often don’t deal with 500 server errors at all, but on a large dynamic site these errors are just as bound to happen as 404s, and they’re far more troublesome. They are unpredictable, untrackable (unless you want to trawl through server logs, which I for one don’t), and harbingers of doom for your site because more often than not, they are indicators of something very bad going on behind the scenes…and you can bet your AdSense check that if a user sees them, a search spider does, too. When a spider hits a server error, it’s usually dead in the water, and that spells disaster for your rankings.

The good news is, it’s actually not too hard to deal with them properly.

One of my large corporate sites was having some massive issues with server response time last year, and as a result we were seeing a significant uptick in the number of 500 errors being reported in Google WMT’s crawl stats.

For the most part, it seemed like simply backing up and reloading the page usually got past the error, but GoogleBot isn’t going to do that. We really had no way of knowing just how pervasive the problem was, but we knew it was affecting the user experience, and clearly killing GoogleBot on a regular basis. While the technology group worked on the backend issues, we stemmed the problem from the front end by creating a custom error page to display any time a 500 error occurred.

The criteria were minimal:

1. Improve the user experience when an error occurs
2. Provide search spiders a way to continue through the site, and
3. Be able to solidly track the number of server errors being delivered as part of our overall statistics

Fortunately, both .NET and Apache make it very easy to define a custom page to display when a server error happens.

In Apache, it’s dead simple - add a line to your .htaccess file like this:

ErrorDocument 500 /friendly500.html

(the nice thing here is that you don’t need to tweak the server config file, which you probably can’t do if you don’t manage your own servers…)

Microsoft servers are a little more involved. For a friendly error page in .NET, IIS tells yout to edit the web config file to include this code:

<customErrors mode="On" defaultRedirect="errors/friendly500.html">
</customErrors>

As noted here on Techrepublic, you may define different pages for different errors:

<customErrors mode="RemoteOnly" defaultRedirect="errors/ErrorPage.aspx">
<error statusCode="400" redirect="errors/friendly400.html" />
<error statusCode="401" redirect="errors/friendly401.html" />
<error statusCode="403" redirect="errors/friendly403.html" />
<error statusCode="404" redirect="errors/friendly404.html" />
<error statusCode="408" redirect="errors/friendly408.html" />
<error statusCode="500" redirect="errors/friendly500.html" />
<error statusCode="503" redirect="errors/friendly503.html" />
</customErrors>

The page can have either an .aspx or .html extension, but keep in mind that if the server is having problems there’s no sense in trying to deliver another dynamic page. Keep it static.

One caveat : IE will try to display a friendly error message of its own, unless the error page is over 512k, so put some text on it.

As our existing 404 page is essentially a sitemap, we quickly realized that we could simply duplicate it as ‘error.html’ and with a few text changes, use that. Users now get a friendly “Oops!” message, and spiders and users alike have a variety of useful links enabling them to continue navigating the site instead of going elsewhere.

Results?

A snapshot report from Google WMT in July showed 218 server errors that happened during their crawls in the previous two weeks. Today, there are none listed at all. (To be fair, the tech guys have been doing loads of work to make things run better as well, and credit where credit is due.) But we can also now see in our statistics that regardless of what GoogleBot is seeing, the error page has actually loaded…um…let’s just say “rather a lot” this month so far, and we can now start assembling solid numbers of how much the server issues are affecting the user experience and to argue for even more improvement work on the backend.