View Single Post
Old 9 Dec 2017, 09:18 PM   #3
Cornerstone of the Community
Join Date: Jan 2008
Posts: 697

Representative of:
That is correct. This is a news article from the power outage event

This was compounded by the fact that our DC's generators didn't kick in in time and there was power loss to the servers. This is pretty much a worst case scenario for a DC, and for anyone running production services.

The good news is that our investments over the past few years in infrastructure allowed us to weather this event with minimal losses. We have 4 separate, independent but synced storage systems. Two of them were corrupted by the unexpected shutdown and the data integrity checks are still taking place in fact. They were pretty much out of the game. We also have other storage systems that run ZFS which is pretty much immune to unexpected power loss so they got kicked up to primary storage role.

Since our main domain runs on 60 seconds refresh in DNS we were able to quickly divert the flow of mail until all services were back up. We are still syncing mail across storage servers now but I am 100% confident that no mail has been lost.

This event was pretty much out of our hands and given these facts I believe we did the best we could. We will be looking into DC-level replication in the near future to avoid such a situation. It's the first time in 12 years this has happened to us! Pretty unique
George_B is offline   Reply With Quote