Disaster Recovery Disasters
 

Given the big disaster that happened before when NYI tried to use its backup generators when utility power went down, I'd really like to see NYI's logs showing who tested their various UPS systems, when, and what the results were, that show that testing is done regularly (not just that they promise to do so).

http://en.wikipedia.org/wiki/Broadba...#Current_Event prompts me to request this info.

Fastmail/Opera, can you please get and post this, or point me to where it is on NYI's site?

While this would be interesting to see, I am even more interested in the progress on reported plans for a backup site outside NYI. The existence of such a (tested) site would make me much less concerned about possible extended downtime at NYI.

Hmm. No answer. I'll send a support request.


:(

Got a reply from NYI.

That's all nice to know actually, which is why I suggested they put it on their website, which as you can see, they thought was a good idea as well.

Note: We don't have any servers at the NYI NJ facility, so that part isn't relevant to us.

However we do have some external servers that are currently a realtime read-only backup of *all* data at NYI (all users, all databases, all email, all files, all address books, etc). We plan sometime later this year to make this a fully read-write replica and use geoip to load balance users between them in regular mode, or failover to one of them in a complete disaster mode. More information when that actually is done.

Rob

Thank-you; good to see you here.

Regards

Nigel

The devil is in the details....
 

Darn, lost my earlier draft post !:( v2:

Thanks for following up and the good news, Rob!:D

Because of my mixed experience with auditing standards during the TD Ameritrade saga, I'm cautious when a certification is provided and suggested that it's proof of something.

I looked into what it means to be SSAE16 Type II Compliant, and found this:

http://www.datacenterknowledge.com/a...e-5-yard-line/ :mad:

To summarize, SSAE16 is fatally flawed. Such compliance doesn't really provide much reassurance of anything that's important. Mike Klein does imply that it's helpful if the actual audit report can be read. They do say that the report is available to customers. Rob, have you obtained or read a copy? Can you share it?:confused:

Well, this big guy named Sandy did a stress test on NYI's disaster recovery procedures, and it performed very well. :D

I wasn't aware that they had an outage. If there was no outage there was no disaster recovery, only a disaster avoidance. Did I miss something?

The phone lines went out (Verizon's systems weren't set up to run without mains power.) The power was out for days. Physical access was restricted or cut off for days. The systems ran successfully on diesel power. Diesel deliveries were successful despite the turmoil. The term 'disaster recovery' is firmly established as covering the contingency measures that made continued operation possible. So yeah, you missed something.

I would agree with your synopsis PON. Backup generators are expected to kick in automatically (on power failure) and usually do. They do need to be tested though (at least monthly) on any building whose operation is deemed essential, to the needs of everyday life.

Rob? Any news on this? It would be reassuring to know that you'd obtained and read a copy of both the report, and the critique at the URL I listed.

Not hearing back is not reassuring.

I would love to understand why your email is so critical that this is an issue.

If it is so critical, can I suggest that $40 isn't enough to spend on your email?