Recent Events: Data Center Explosion and Recovery

On May 31st, 3:45pm Pacific Time a transformer exploded outside of The Planet's H1 data center in Texas destroying an electrical room and causing a power outage throughout the entire data center. We had a total of 25 servers located in this data center.

Upon any servers going offline we receive instant notification via a toolbar we have on our computers as well as phone notification:



Immediately after determining that there was some sort of network issue we called up our contacts at The Planet (our server host) for information on what was going on. Details were very sketchy at first with no clear information getting out. After a while we learned that there was some kind of fire that knocked out the electrical room.

We spent a lot of time with our account manager on the phone over the next few hours. We attempted to get our servers physically moved to another data center, but this idea got shot down.

Faced with a continued outage with no information on when power would be restored, we decided our best course of action would be to setup 24 new machines in one of The Planet's data centers. We were told that all of our old servers had 100% of their data in tact, but we would need to wait for power to come back online before we could access that data. Our backup server was not located in the H1 data center, and as such we decided to begin to restore as much data as possible from it to our new servers.

At 1:10am Pacific Time our first brand new server came online. We worked feverishly to setup this server and others as they came online throughout the night. We began to restore as many forums as we could from the backups that we had available. By 4:00am we had a few servers back online, albeit in a restored state from April while we awaited full data recovery.

Continuing throughout the night we restored as much data as was possible until we had each server back online to a usable state. A little bit over 24 hours after this event started, Martyn and I got some much needed sleep.

Starting at approximately 4:30am on June 2nd we began to see some servers come back online from the H1 data center. Martyn began the process of data recovery from these servers at this time. By 8:43am we already had our first server fully restored to the point it was at from before the original explosion at the data center.

We worked throughout the day and night from June 2nd to 3rd getting all servers back online with as much data recovered as possible. I am happy to report that 100% of all data was fully restored.

In an effort to ensure faster recovery of data in the future, we are purchasing 2 brand new backup servers which will be placed in separate data centers at The Planet. These should be online within a couple of days.

I would like to thank you for your patience and understanding, your overwhelming words of support and encouragement have been very appreciated by our staff.

Patrick Clinger
Founder, ProBoards.com

P.S.
We've decided to name this event the:
First Incredible Rapid Explosion of Servers - 2008.

UBBC code to use this image:
[url=http://blog.proboards.com/2008/06/recent-events-data-center-explosion-and.html] [img]http://blog.proboards.com/survivor.jpg[/img] [/url]
HTML code to use this image:
<a href="http://blog.proboards.com/2008/06/recent-events-data-center-explosion-and.html"><img src="http://blog.proboards.com/survivor.jpg" alt="I Survived." /></a>