polarismail down
 

Hi guys,
Is polarismail.com and emailarray.com down for all of you as well?

There was a huge power outage in Montreal which took out their data center. I just chatted with them and they are expecting it will be several more hours before they are fully up again.

That is correct. This is a news article from the power outage event

http://montrealgazette.com/storyline...ntown-montreal

This was compounded by the fact that our DC's generators didn't kick in in time and there was power loss to the servers. This is pretty much a worst case scenario for a DC, and for anyone running production services.

The good news is that our investments over the past few years in infrastructure allowed us to weather this event with minimal losses. We have 4 separate, independent but synced storage systems. Two of them were corrupted by the unexpected shutdown and the data integrity checks are still taking place in fact. They were pretty much out of the game. We also have other storage systems that run ZFS which is pretty much immune to unexpected power loss so they got kicked up to primary storage role.

Since our main domain runs on 60 seconds refresh in DNS we were able to quickly divert the flow of mail until all services were back up. We are still syncing mail across storage servers now but I am 100% confident that no mail has been lost.

This event was pretty much out of our hands and given these facts I believe we did the best we could. We will be looking into DC-level replication in the near future to avoid such a situation. It's the first time in 12 years this has happened to us! Pretty unique