If you kept trying to access your Facebook page Thursday and got an error message, you weren’t alone. Millions of Facebook users around the world got the same message during a 2.5-hour blackout that will go down in the social-networking giant’s four-year history as the worst-ever outage.
Facebook was quick to apologize and offer an explanation to its more than 500 million users. The story is one of technical glitches and lessons learned for the young company, which is the focus of a feature film to be released Oct. 1 called The Social Network.
As Facebook’s Robert Johnson explained it, the key flaw that caused the outage to be so severe boiled down to the “unfortunate handling of an error condition.” Essentially, an automated system for verifying configuration values ended up causing much more damage than it fixed.
“The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store,” Johnson said. “This works well for a transient problem with the cache, but it doesn’t work when the persistent store is invalid.”
A Spiraling Problem
Facebook got its social network back online by making a change to the persistent copy of a configuration value that was interpreted as invalid. With the invalid configuration value, every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.
“To make matters worse, every time a client got an error attempting to query one of the databases, it interpreted it as an invalid value and deleted the corresponding cache key. This meant that even after the original problem had been fixed, the stream of queries continued,” Johnson said….