RFE: Guaranteed uptime

koffg · 12 Nov 2005, 10:31 PM

This is my proposal for an uptime guarantee FM could use, obviously not written in legalese, please comment!

--
All Fastmail/messagingengine.com services, with the exception of the blog, forum, and Wiki, will be unavailable for no more than one hour per month. If the server is slow to the point of unusability, as determined by an independent monitor, that will be considered an outage. Each outage less than ten minutes in duration will be counted as a full ten minutes.

Email that is delayed more than five minutes (from the time delivered to FM servers to the time it arrives in a user's account) will be counted as an outage of ten minutes.

Two half hours of scheduled maintenance are alloted per month. To be considered scheduled, and not an outage, a notice must be posted on the front page of FM 24 hours in advance of said maintenance.
--

Serious refunds/credit/penalties as appropriate. The actual numbers aren't important, the idea is. If FM wants to provide a so-called world class email service, this needs to be implemented. Several people have already offered to pay more money to get this level of service (what customers!). This base guarantee should apply to Full and Enhanced accounts, and perhaps there should be a new tier of service for people who are willing to pay for 99.9999999% uptime. There is an undisputed need.

Let me stress the importance of an independent (i.e. community run, publicly available, FM funded) monitor than can ping, with fine granularity, the uptime of each of FM's servers as well as email delivery speed.

Comments more than welcome.

fmfan · 12 Nov 2005, 11:12 PM

Quote:

Originally posted by koffg
...
There is an undisputed need. ...

Seems like it to me - class of customer that has need for higher uptime, more reliability and will pay for this needed requirement for their day-to-day operations.

messagingengine.com

nickbeee · 12 Nov 2005, 11:15 PM

I'm surprised nobody has suggested this earlier.

Whilst "enhanced" buys me more resources, in hindsight there is no kind of service level agreement as you have suggested above. Personally I would be happy to pay a little extra if it meant minimising any down-time for issues such as this.

There is, however, an option for paying customers of FastMail to cancel their accounts, I suspect there will be quite a few who do this. Personally I will look for reassurances in the longer term and hope that FM re-think their contingency plans .

koffg · 12 Nov 2005, 11:23 PM

Quote:

Personally I will look for reassurances in the longer term and hope that FM re-think their contingency plans .

I feel the same way, which is the reason I wrote up this proposal. FM has got to do more, this time, than just offer a free month and say "this won't happen again", again. Here's hoping they consider this.

krbumgarner · 12 Nov 2005, 11:44 PM

Five 9's

In today's world, keeping IT systems and networks up and running is more important than ever before. How much downtime is acceptable? 99%? 99.9%? How many 9's? The answer may surprise you.

If systems are up... Then yearly downtime is...
90%
876 hours

99%
87 hours, 36 minutes

99.9%
8 hours, 45.5 minutes

99.99%
52 minutes, 33 seconds

99.999%
5 minutes, 15 seconds

So while 99% uptime sounds good, the reality is that IT would be down 1.7 hours per week!

That amount of downtime is completely unacceptable to most organizations and a significant direct threat to profitability for high volume e-businesses.

Today's definition of "high availability" is a system at "five 9's" (99.999%).

Quoted from "Forsythe - Five 9's"

--------------------------------------------------------

This is an article that I found while putting together some documentation for work. The company I work for sells software that requires high-availablity hardware (9-1-1 dispatch). A large array of disks isn't necessarily a bad thing ... as long as you don't ever have to restore it. When you put all those eggs into the same basket, a means of keeping that data available is greatly desired. A method of keeping that data available outside the scope of your OS ... mirrored arrays are not cheap, but they get the job done. For example, an EMC replicated SAN using MirrorView would do the trick ... (1) array crashes, (2) failover to secondary array, (3) users still have access to their data, (4) repair offline array, (5) sync arrays, (6) back to fully a redundant operation.

EMC MirrorView

99.9999999 is NINE 9's ... that's less than a minute per year ... I think that'd be overly-excessive demands for email.

Granted, I'm one who lives by my email ... everyone knows that if they send me an email they're going to get a faster response than by trying to call me. When I decide to 'wait until I get home to answer an email' I get follow-ups asking if I'm alright ... why haven't I answered yet, etc.

I don't believe that freaking out and ranting and raving will get us anywhere in this situation. But I do believe that an outage that is anticipated to last for more than 4-6 hours that is unplanned is truly excessive and totally blows the word RELIABLE out the window.

Good luck to all who are suffering withdrawl symptoms from being without our email ...

-Keith

**Sherry** · 13 Nov 2005, 03:11 AM

Sorry koffg but I can't see any of the suggestions as reasonable. I know I wouldn't want to run a business with any of that in place knowing just a few DoS attacks could wipe me out. In fact, hiring another person (or even 2 people) to handle "all" the email they would get with users saying "my mail was 10 minutes late so I want my compensation" could cause the cost of our service to climb unreasonably high. Checking the headers and showing the user it was bogged down at the other end (which happens a lot) could be a big, time consuming job.

As for maintenance, I sure wouldn't want them to be worried about missing the 24 hour notice when seeing a teeny sign making them think they want to perform the maintenance a little earlier than normal. I run maintenance on my pc weekly but "many" times I see a little slow down and do it earlier than normal (depending on my pc usage that week).

However I think letting everyone know their future plans and what's done to prevent another disaster would be something the users could/should expect. Actually, they already have said one thing they are/may do and that's use smaller volumes instead of one huge one so if something similar (or the same) thing happened the restores would go much much faster.

Anyway, this is all IMHO because I know how very unpredictable hardware/software is and wouldn't want to grantee either if it meant loosing my whole business.

Sherry

Chipper · 13 Nov 2005, 03:47 AM

Quote:

Originally posted by krbumgarner
Five 9's

In today's world, keeping IT systems and networks up and running is more important than ever before. How much downtime is acceptable? 99%? 99.9%? How many 9's? The answer may surprise you.

If systems are up... Then yearly downtime is...
90%
876 hours

99%
87 hours, 36 minutes

99.9%
8 hours, 45.5 minutes

99.99%
52 minutes, 33 seconds

99.999%
5 minutes, 15 seconds

So while 99% uptime sounds good, the reality is that IT would be down 1.7 hours per week!

That amount of downtime is completely unacceptable to most organizations and a significant direct threat to profitability for high volume e-businesses.

Today's definition of "high availability" is a system at "five 9's" (99.999%).

Quoted from "Forsythe - Five 9's"

--------------------------------------------------------

This is an article that I found while putting together some documentation for work. The company I work for sells software that requires high-availablity hardware (9-1-1 dispatch). A large array of disks isn't necessarily a bad thing ... as long as you don't ever have to restore it. When you put all those eggs into the same basket, a means of keeping that data available is greatly desired. A method of keeping that data available outside the scope of your OS ... mirrored arrays are not cheap, but they get the job done. For example, an EMC replicated SAN using MirrorView would do the trick ... (1) array crashes, (2) failover to secondary array, (3) users still have access to their data, (4) repair offline array, (5) sync arrays, (6) back to fully a redundant operation.

EMC MirrorView

99.9999999 is NINE 9's ... that's less than a minute per year ... I think that'd be overly-excessive demands for email.

Granted, I'm one who lives by my email ... everyone knows that if they send me an email they're going to get a faster response than by trying to call me. When I decide to 'wait until I get home to answer an email' I get follow-ups asking if I'm alright ... why haven't I answered yet, etc.

I don't believe that freaking out and ranting and raving will get us anywhere in this situation. But I do believe that an outage that is anticipated to last for more than 4-6 hours that is unplanned is truly excessive and totally blows the word RELIABLE out the window.

Good luck to all who are suffering withdrawl symptoms from being without our email ...

-Keith

Very interesting stuff, thank you!

Personally, I could handle 99.9 (average of about 10 mins per week). However, the other variable would be the "longest acceptable outage." In other words, while I think many people could handle a 10 minute issue, the question would be that if those 8 hours 45 mins were to happen all at once...

Again, I don't have any "mission critical" reasons to need as much reliablility as others need. So... my response is not to "shoot down" what you said, but just to add something to the discussion.

koffg · 13 Nov 2005, 04:05 AM

Sherry, I appreciate your sentiment. I also don't want to see FM make promises they couldn't keep, or that would risk their business. To keep me as a customer, however, their implicit guarantee of providing a reliable service needs to be made explicit.

Again I'll say that the figures in my proposal aren't important. 24-hours notice or two-hours notice is fine by me. An hour or three hours of downtime a month. The point is to keep FM more conscientious with regards to their maintenance and server design, and to set user expectations. Such a guarantee would help right what I'd consider their growing reputation as unreliable.

I agree that it would be a bit of work to implement monitoring of timely delivery. Users should not be able to make claims of delayed delivery, but rather this would be determined through the independent monitoring server I mentioned in my original post. I have enough technical knowledge to know it's possible, with FM's cooperation.

schmoe · 13 Nov 2005, 12:16 PM

I too think this is unnecessary, complicated, and costly. I have my own personal contract with fastmail. Its an implied contract and not a legal contract. Fastmail doesn't even know about it. It goes like this:

Fastmail shall provide me a quality email service. On an annual basis, I will acess Fastmail's performance and determine whether to continue. If I like the quality of service, I will reup. If I dislike the quality of service, I will not reup.

So far, I have reupped twice, and so long as this two day outage does not become a regular occurence, I'll reup a third time.

David · 13 Nov 2005, 01:10 PM

This is one of the few threads, where I can honestly say that I agree with everyone. The reality is that different people have different needs (and different amounts of money to spend) The individual searches for a provider that will meet their needs; often they will not be able to find exactly what they want, and will buy something close, or build something themself, to meet their individual needs.

Fastmail is competing in a highly competitive market place, where the going rate (for an email account) is somewhere between about $16 and $70 a year, for a variety of different email services. Within the confines of these numbers, an individual will usually be able to locate a company, who will sell them what they want to buy.

I suspect though, that the service that koffg is wanting, could not be provided for much less than $100 a year. It's a different marketplace, that few will wish to purchase into.

Regardless, it is a valid market, that some will patronise - if the service is provided though.

hadaso · 14 Nov 2005, 06:55 AM

I tend not to belive such "service guaranteed" statements. When I hear/read such statements what I understand is "part of the money you pay us for service is actually not used to provide you with our service but instead is paid to our insurer who is providing the financial backing these statements are based on..."

I don't see how FastMail can promise that they won't have 3 disks in a RAID-6 array fail at the same time. And they already said how they can make recovery time much faster in the unlikely event that 3 disks would simultaneously fail again...

Edit: I've read a bit about RAID-6. The "6" figure has nothing to do with the number of disks in the RAID array. But it means it can withstand failure of 2 disks but not 3.

greenplum · 15 Nov 2005, 02:53 AM

I do believe also that if for example NYI have 500 servers and one goes off for 40 hrs in a 1000 hrs the downtime would not be 1000 - 40 = 0.4% After taking into consideration the other 499 with no problems the amount would not even register as a percentage .. However if you were on that server it would certainly register with you (as has been proved in the past few days)

hadaso · 15 Nov 2005, 05:10 AM

The servers are FastMail's servers, I think. They are loated on NYI facilities that also provide the hardware maintenance. Perhaps I misunderstand something. Correct me if I'm wrong!

greenplum · 15 Nov 2005, 05:52 AM

Quote:

Originally posted by hadaso
The servers are FastMail's servers, I think. They are loated on NYI facilities that also provide the hardware maintenance. Perhaps I misunderstand something. Correct me if I'm wrong!

Just using that as an example as to why anyone's claimed figures are a waste of space as they are never as straightforward as the original statement.
Even Fastmail can claim, if they so wish that 3 of their servers have had no problems; so while 4 was down the other 3 were clocking up 3 x 55hrs (or whatever)uptime

Sorry if it sounds complicated

12 Nov 2005, 10:31 PM	#1
koffg Junior Member Join Date: Nov 2005 Posts: 5	RFE: Guaranteed uptime This is my proposal for an uptime guarantee FM could use, obviously not written in legalese, please comment! -- All Fastmail/messagingengine.com services, with the exception of the blog, forum, and Wiki, will be unavailable for no more than one hour per month. If the server is slow to the point of unusability, as determined by an independent monitor, that will be considered an outage. Each outage less than ten minutes in duration will be counted as a full ten minutes. Email that is delayed more than five minutes (from the time delivered to FM servers to the time it arrives in a user's account) will be counted as an outage of ten minutes. Two half hours of scheduled maintenance are alloted per month. To be considered scheduled, and not an outage, a notice must be posted on the front page of FM 24 hours in advance of said maintenance. -- Serious refunds/credit/penalties as appropriate. The actual numbers aren't important, the idea is. If FM wants to provide a so-called world class email service, this needs to be implemented. Several people have already offered to pay more money to get this level of service (what customers!). This base guarantee should apply to Full and Enhanced accounts, and perhaps there should be a new tier of service for people who are willing to pay for 99.9999999% uptime. There is an undisputed need. Let me stress the importance of an independent (i.e. community run, publicly available, FM funded) monitor than can ping, with fine granularity, the uptime of each of FM's servers as well as email delivery speed. Comments more than welcome. Last edited by koffg : 12 Nov 2005 at 11:24 PM.

12 Nov 2005, 11:44 PM	#5
krbumgarner Junior Member Join Date: Nov 2005 Posts: 1	What is "five 9's of up-time" Five 9's In today's world, keeping IT systems and networks up and running is more important than ever before. How much downtime is acceptable? 99%? 99.9%? How many 9's? The answer may surprise you. If systems are up... Then yearly downtime is... 90% 876 hours 99% 87 hours, 36 minutes 99.9% 8 hours, 45.5 minutes 99.99% 52 minutes, 33 seconds 99.999% 5 minutes, 15 seconds So while 99% uptime sounds good, the reality is that IT would be down 1.7 hours per week! That amount of downtime is completely unacceptable to most organizations and a significant direct threat to profitability for high volume e-businesses. Today's definition of "high availability" is a system at "five 9's" (99.999%). Quoted from "Forsythe - Five 9's" -------------------------------------------------------- This is an article that I found while putting together some documentation for work. The company I work for sells software that requires high-availablity hardware (9-1-1 dispatch). A large array of disks isn't necessarily a bad thing ... as long as you don't ever have to restore it. When you put all those eggs into the same basket, a means of keeping that data available is greatly desired. A method of keeping that data available outside the scope of your OS ... mirrored arrays are not cheap, but they get the job done. For example, an EMC replicated SAN using MirrorView would do the trick ... (1) array crashes, (2) failover to secondary array, (3) users still have access to their data, (4) repair offline array, (5) sync arrays, (6) back to fully a redundant operation. EMC MirrorView 99.9999999 is NINE 9's ... that's less than a minute per year ... I think that'd be overly-excessive demands for email. Granted, I'm one who lives by my email ... everyone knows that if they send me an email they're going to get a faster response than by trying to call me. When I decide to 'wait until I get home to answer an email' I get follow-ups asking if I'm alright ... why haven't I answered yet, etc. I don't believe that freaking out and ranting and raving will get us anywhere in this situation. But I do believe that an outage that is anticipated to last for more than 4-6 hours that is unplanned is truly excessive and totally blows the word RELIABLE out the window. Good luck to all who are suffering withdrawl symptoms from being without our email ... -Keith Last edited by krbumgarner : 13 Nov 2005 at 12:15 AM.

13 Nov 2005, 04:05 AM	#8
koffg Junior Member Join Date: Nov 2005 Posts: 5	Sherry, I appreciate your sentiment. I also don't want to see FM make promises they couldn't keep, or that would risk their business. To keep me as a customer, however, their implicit guarantee of providing a reliable service needs to be made explicit. Again I'll say that the figures in my proposal aren't important. 24-hours notice or two-hours notice is fine by me. An hour or three hours of downtime a month. The point is to keep FM more conscientious with regards to their maintenance and server design, and to set user expectations. Such a guarantee would help right what I'd consider their growing reputation as unreliable. I agree that it would be a bit of work to implement monitoring of timely delivery. Users should not be able to make claims of delayed delivery, but rather this would be determined through the independent monitoring server I mentioned in my original post. I have enough technical knowledge to know it's possible, with FM's cooperation. Last edited by koffg : 13 Nov 2005 at 04:12 AM.

13 Nov 2005, 12:16 PM	#9
schmoe Essential Contributor Join Date: Oct 2003 Posts: 385	thats just silly I too think this is unnecessary, complicated, and costly. I have my own personal contract with fastmail. Its an implied contract and not a legal contract. Fastmail doesn't even know about it. It goes like this: Fastmail shall provide me a quality email service. On an annual basis, I will acess Fastmail's performance and determine whether to continue. If I like the quality of service, I will reup. If I dislike the quality of service, I will not reup. So far, I have reupped twice, and so long as this two day outage does not become a regular occurence, I'll reup a third time.

14 Nov 2005, 06:55 AM	#11
hadaso The "e" in e-mail Join Date: Oct 2002 Location: Holon, Israel. Posts: 4,863	I tend not to belive such "service guaranteed" statements. When I hear/read such statements what I understand is "part of the money you pay us for service is actually not used to provide you with our service but instead is paid to our insurer who is providing the financial backing these statements are based on..." I don't see how FastMail can promise that they won't have 3 disks in a RAID-6 array fail at the same time. And they already said how they can make recovery time much faster in the unlikely event that 3 disks would simultaneously fail again... Edit: I've read a bit about RAID-6. The "6" figure has nothing to do with the number of disks in the RAID array. But it means it can withstand failure of 2 disks but not 3. Last edited by hadaso : 14 Nov 2005 at 03:37 PM.

12 Nov 2005, 11:15 PM	#3
nickbeee Junior Member Join Date: Jul 2003 Location: Nottingham/UK Posts: 29	I'm surprised nobody has suggested this earlier. Whilst "enhanced" buys me more resources, in hindsight there is no kind of service level agreement as you have suggested above. Personally I would be happy to pay a little extra if it meant minimising any down-time for issues such as this. There is, however, an option for paying customers of FastMail to cancel their accounts, I suspect there will be quite a few who do this. Personally I will look for reassurances in the longer term and hope that FM re-think their contingency plans .

13 Nov 2005, 03:11 AM	#6
Sherry Moderator Join Date: Dec 2002 Location: USA Posts: 8,687	Sorry koffg but I can't see any of the suggestions as reasonable. I know I wouldn't want to run a business with any of that in place knowing just a few DoS attacks could wipe me out. In fact, hiring another person (or even 2 people) to handle "all" the email they would get with users saying "my mail was 10 minutes late so I want my compensation" could cause the cost of our service to climb unreasonably high. Checking the headers and showing the user it was bogged down at the other end (which happens a lot) could be a big, time consuming job. As for maintenance, I sure wouldn't want them to be worried about missing the 24 hour notice when seeing a teeny sign making them think they want to perform the maintenance a little earlier than normal. I run maintenance on my pc weekly but "many" times I see a little slow down and do it earlier than normal (depending on my pc usage that week). However I think letting everyone know their future plans and what's done to prevent another disaster would be something the users could/should expect. Actually, they already have said one thing they are/may do and that's use smaller volumes instead of one huge one so if something similar (or the same) thing happened the restores would go much much faster. Anyway, this is all IMHO because I know how very unpredictable hardware/software is and wouldn't want to grantee either if it meant loosing my whole business. Sherry

13 Nov 2005, 01:10 PM	#10
David Ultimate Contributor Join Date: Dec 2001 Location: Canada. Posts: 10,355	This is one of the few threads, where I can honestly say that I agree with everyone. The reality is that different people have different needs (and different amounts of money to spend) The individual searches for a provider that will meet their needs; often they will not be able to find exactly what they want, and will buy something close, or build something themself, to meet their individual needs. Fastmail is competing in a highly competitive market place, where the going rate (for an email account) is somewhere between about $16 and $70 a year, for a variety of different email services. Within the confines of these numbers, an individual will usually be able to locate a company, who will sell them what they want to buy. I suspect though, that the service that koffg is wanting, could not be provided for much less than $100 a year. It's a different marketplace, that few will wish to purchase into. Regardless, it is a valid market, that some will patronise - if the service is provided though.

15 Nov 2005, 02:53 AM	#12
greenplum Essential Contributor Join Date: Sep 2003 Location: N.E. England Posts: 423	I do believe also that if for example NYI have 500 servers and one goes off for 40 hrs in a 1000 hrs the downtime would not be 1000 - 40 = 0.4% After taking into consideration the other 499 with no problems the amount would not even register as a percentage .. However if you were on that server it would certainly register with you (as has been proved in the past few days) Last edited by greenplum : 15 Nov 2005 at 03:00 AM.

15 Nov 2005, 05:10 AM	#13
hadaso The "e" in e-mail Join Date: Oct 2002 Location: Holon, Israel. Posts: 4,863	The servers are FastMail's servers, I think. They are loated on NYI facilities that also provide the hardware maintenance. Perhaps I misunderstand something. Correct me if I'm wrong!