Disaster Recovery Disasters

elvey · 10 May 2012, 06:34 AM

Given the big disaster that happened before when NYI tried to use its backup generators when utility power went down, I'd really like to see NYI's logs showing who tested their various UPS systems, when, and what the results were, that show that testing is done regularly (not just that they promise to do so).

http://en.wikipedia.org/wiki/Broadba...#Current_Event prompts me to request this info.

Fastmail/Opera, can you please get and post this, or point me to where it is on NYI's site?

BritTim · 10 May 2012, 08:06 PM

While this would be interesting to see, I am even more interested in the progress on reported plans for a backup site outside NYI. The existence of such a (tested) site would make me much less concerned about possible extended downtime at NYI.

elvey · 15 May 2012, 03:05 AM

Hmm. No answer. I'll send a support request.

robmueller · 21 May 2012, 01:01 PM

Got a reply from NYI.

Quote:

The issue we had in 2003 was that the Generator was managed by the building at the time, whereas now we have our own infrastructure and control all aspects of our infrastructure from the fuel tanks to the generator itself. In addition to Quarterly PMs by the Manufacturer, we perform Monthly Generator Testing on Load using Load Banks that simulate our load. We also perform regular PMs on the UPS units and ATS systems to insure that the infrastructure is performing as expected. We also have our Bridgewater, NJ Facility, 40 miles outside NYC which allows us to not only maintain copies of all backed up data, but allows our customers to have access to a full fledged DR facility with dark fiber connectivity for realtime syncing of data. Now that we have the new website (nyi.net), I think your suggestion is a great one and we'll setup a page that clearly and openly states the amount of effort that goes into providing maximum uptime. Let me know if you need any additional details.

...

Of particular interest is the fact that it's quite unique in our market to maintain *all* of the infrastructure, particularly in NYC. Usually at least some part of it is shared, be it the fuel supply, glycol supply, etc. In our case, in both facilities, we are responsible for all aspects of the infrastructure. We refer to that on the website as "First Party Facility" but we can certainly do a better job of making that distinction.

...

The other point that I'd make is that we are now SSAE16 Type II Compliant which means that we have independent auditors come in and verify the controls that we have in place for our infrastructure including environmental, power, networking, security, etc. This report is done biannually and available to our customers.

That's all nice to know actually, which is why I suggested they put it on their website, which as you can see, they thought was a good idea as well.

Note: We don't have any servers at the NYI NJ facility, so that part isn't relevant to us.

However we do have some external servers that are currently a realtime read-only backup of *all* data at NYI (all users, all databases, all email, all files, all address books, etc). We plan sometime later this year to make this a fully read-write replica and use geoip to load balance users between them in regular mode, or failover to one of them in a complete disaster mode. More information when that actually is done.

NJSS · 21 May 2012, 04:13 PM

Rob

Thank-you; good to see you here.

Regards

Nigel

elvey · 31 May 2012, 08:10 AM

Darn, lost my earlier draft post !

v2:

Thanks for following up and the good news, Rob!

Because of my mixed experience with auditing standards during the TD Ameritrade saga, I'm cautious when a certification is provided and suggested that it's proof of something.

I looked into what it means to be SSAE16 Type II Compliant, and found this:

http://www.datacenterknowledge.com/a...e-5-yard-line/

To summarize, SSAE16 is fatally flawed. Such compliance doesn't really provide much reassurance of anything that's important. Mike Klein does imply that it's helpful if the actual audit report can be read. They do say that the report is available to customers. Rob, have you obtained or read a copy? Can you share it?

elvey · 21 Nov 2012, 10:46 AM

Well, this big guy named Sandy did a stress test on NYI's disaster recovery procedures, and it performed very well.

PON · 21 Nov 2012, 06:59 PM

I wasn't aware that they had an outage. If there was no outage there was no disaster recovery, only a disaster avoidance. Did I miss something?

elvey · 23 Nov 2012, 01:23 AM

Quote:

Originally Posted by PON

I wasn't aware that they had an outage. If there was no outage there was no disaster recovery, only a disaster avoidance. Did I miss something?

The phone lines went out (Verizon's systems weren't set up to run without mains power.) The power was out for days. Physical access was restricted or cut off for days. The systems ran successfully on diesel power. Diesel deliveries were successful despite the turmoil. The term 'disaster recovery' is firmly established as covering the contingency measures that made continued operation possible. So yeah, you missed something.

David · 23 Nov 2012, 05:59 AM

Quote:

Originally Posted by PON

I wasn't aware that they had an outage. If there was no outage there was no disaster recovery, only a disaster avoidance. Did I miss something?

I would agree with your synopsis PON. Backup generators are expected to kick in automatically (on power failure) and usually do. They do need to be tested though (at least monthly) on any building whose operation is deemed essential, to the needs of everyday life.

elvey · 22 Feb 2013, 01:14 AM

Quote:

Originally Posted by elvey

Darn, lost my earlier draft post !

v2:

Thanks for following up and the good news, Rob!

Because of my mixed experience with auditing standards during the TD Ameritrade saga, I'm cautious when a certification is provided and suggested that it's proof of something.

I looked into what it means to be SSAE16 Type II Compliant, and found this:

http://www.datacenterknowledge.com/a...e-5-yard-line/

To summarize, SSAE16 is fatally flawed. Such compliance doesn't really provide much reassurance of anything that's important. Mike Klein does imply that it's helpful if the actual audit report can be read. They do say that the report is available to customers. Rob, have you obtained or read a copy? Can you share it?

Rob? Any news on this? It would be reassuring to know that you'd obtained and read a copy of both the report, and the critique at the URL I listed.

elvey · 25 Jun 2013, 08:24 AM

Quote:

Originally Posted by elvey

Rob? Any news on this? It would be reassuring to know that you'd obtained and read a copy of both the report, and the critique at the URL I listed.

Not hearing back is not reassuring.

hobbes · 27 Jun 2013, 10:16 PM

I would love to understand why your email is so critical that this is an issue.

If it is so critical, can I suggest that $40 isn't enough to spend on your email?

10 May 2012, 06:34 AM	#1
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,458	Disaster Recovery Disasters Given the big disaster that happened before when NYI tried to use its backup generators when utility power went down, I'd really like to see NYI's logs showing who tested their various UPS systems, when, and what the results were, that show that testing is done regularly (not just that they promise to do so). http://en.wikipedia.org/wiki/Broadba...#Current_Event prompts me to request this info. Fastmail/Opera, can you please get and post this, or point me to where it is on NYI's site?

15 May 2012, 03:05 AM	#3
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,458	Hmm. No answer. I'll send a support request. Last edited by elvey : 15 May 2012 at 03:08 AM. Reason: Request sent.

31 May 2012, 08:10 AM	#6
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,458	The devil is in the details.... Darn, lost my earlier draft post ! v2: Thanks for following up and the good news, Rob! Because of my mixed experience with auditing standards during the TD Ameritrade saga, I'm cautious when a certification is provided and suggested that it's proof of something. I looked into what it means to be SSAE16 Type II Compliant, and found this: http://www.datacenterknowledge.com/a...e-5-yard-line/ To summarize, SSAE16 is fatally flawed. Such compliance doesn't really provide much reassurance of anything that's important. Mike Klein does imply that it's helpful if the actual audit report can be read. They do say that the report is available to customers. Rob, have you obtained or read a copy? Can you share it?

10 May 2012, 08:06 PM	#2
BritTim The "e" in e-mail Join Date: May 2003 Location: mostly in Thailand Posts: 3,090	While this would be interesting to see, I am even more interested in the progress on reported plans for a backup site outside NYI. The existence of such a (tested) site would make me much less concerned about possible extended downtime at NYI.

21 May 2012, 04:13 PM	#5
NJSS Master of the @ Join Date: Jul 2002 Location: Hampshire, UK Posts: 1,238	Rob Thank-you; good to see you here. Regards Nigel

21 Nov 2012, 10:46 AM	#7
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,458	Well, this big guy named Sandy did a stress test on NYI's disaster recovery procedures, and it performed very well.

21 Nov 2012, 06:59 PM	#8
PON Essential Contributor Join Date: Mar 2002 Location: Wicklow, Ireland Posts: 449	I wasn't aware that they had an outage. If there was no outage there was no disaster recovery, only a disaster avoidance. Did I miss something?

27 Jun 2013, 10:16 PM	#13
hobbes Registered User Join Date: Aug 2003 Location: UK Posts: 463	I would love to understand why your email is so critical that this is an issue. If it is so critical, can I suggest that $40 isn't enough to spend on your email?