Imagine the sinking feeling of logging in to your Gmail account and finding it empty. Ben Treynor, Google’s vice president of engineering and site reliability czar, invited us to picture that in our mind’s eye.
But many Gmail users didn’t have to use their imagination — it actually happened. Indeed, Treynor reported that 0.02 percent (about 150 million) of Gmail users woke up Monday to find their Gmail account was empty. No important e-mail to answer. E-mail archives up in smoke. It wasn’t a pretty picture. And Treynor is sorry.
“The good news is that e-mail was never lost and we’ve restored access for many of those affected,” Treynor said. “Though it may take longer than we originally expected, we’re making good progress and things should be back to normal for everyone soon.”
What About Google Disaster Recovery?
That was the update as of 9:30 p.m. EST Monday night. After asking Gmail users to imagine the worst, Treynor suggested some of them were thinking: “How could this happen if Google has multiple copies of your data in multiple data centers?”
Indeed, Google made a pretty big deal about its disaster-recovery services about this time last year. In March 2010, the company promised that Google Apps customers wouldn’t need to worry about disaster striking and used e-mail as a prime example. So what happened?
“Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue,” Treynor said.
Essentially, to protect Google Apps user information from these rare bugs, Google also backs up the data to tape. Since the tapes are offline, Treynor explained, they’re protected from such software bugs. It follows, then, that…