DDoS attack against FastMail

[17pm]

Originally Posted by DumbGuy

I'd say cut 'em some slack. According to their blog, they had enormous pressure on them from their ISP and hundreds of other businesses that were affected as collateral damage. It sounded like they resisted paying as much as they could.

Jesus Christ, they had pressure from their ISP. What are they going to do when three/four letter security agencies pressure them to give them a backdoor or hand them user data?

I think this definitely proves that protonmail is not to be trusted.

[ioneja]

Originally Posted by dodorkahedron

I'm very impressed with how FM and NYI handled this. I appreciate the advance notice of possible service interruptions this week as they mitigate any further attacks. I much prefer that to paying ransom to miscreants!

My account is due for renewal, and I'm pleased to say I will be on board for another five years.

Agreed. FM seems to have handled this well. Keep up the good work FM team!

[Just Bill]

Originally Posted by 17pm

Jesus Christ, they had pressure from their ISP. What are they going to do when three/four letter security agencies pressure them to give them a backdoor or hand them user data?

I think this definitely proves that protonmail is not to be trusted.

EXACTLY. I wouldn't trust them as far as I could spit.

[remotetech]

I'm a ProtonMail user, and this is the email we were sent by ProtonMail:

"Dear ProtonMail Community,

As many of you know, last week ProtonMail came under a massive distributed denial-of-service (DDoS) attack which knocked our service offline for several days. Unfortunately, we were initially unable to defend against such a massive attack and suffered downtime as a result. Despite the ferocity of the attack, our server security measures and end-to-end encryption meant we were able to keep user data secure.

This incident was one of the largest cyberattacks ever in Switzerland and caused enough damage to knock an entire datacenter offline. In an attempt to keep ProtonMail offline, upstream ISPs were also attacked, knocking hundreds of other businesses offline in countries as far away as Russia. The main attack began on Wednesday, November 4th, and it was not until the evening of Saturday, November 7th that we were able to bring the situation until control. Full details about the attack can be found on our blog here.

There is no doubt that the purpose of the attack was to keep ProtonMail offline for as long as possible. In doing so, the attackers wanted to deny email privacy to nearly a million people worldwide. The attackers hoped to destroy our community, but this attack has only served to bring us all together, united by a common cause and vision for the future. Our vision for an Internet that respects privacy and freedom can be assaulted, but it will never be destroyed.

Instead of weakening ProtonMail, these attacks have only made us stronger, and rallied more people to our cause. Collectively, the ProtonMail community raised $50,000 for the ProtonMail Defense Fund in just three days, giving us the resources to defeat the current attack and protect against future ones. In defending ProtonMail, we were joined by Radware, one of the world's premier DDoS protection companies. We also redesigned our network infrastructure to have a dedicated link to a Tier 1 carrier in Zurich. In addition to the privacy benefits of controlling all traffic in and out of our datacenter, this also makes our network far more difficult to attack.

Our cause is also joined by IP-Max, the best network experts in Switzerland. The IP-Max team worked extremely long hours for several days in a row to bring us back up. And they did it entirely on a volunteer basis, simply to support our community. Building an entire network from scratch and bringing it online in a few days requires an incredible effort, and it was only with their assistance that we were able to come back online as quickly as we did.

The result is that ProtonMail is now stronger than ever. Not only did we mitigate the largest DDoS attack in Switzerland in a couple days, we also gained the ability to resist such attacks in the future. We would like to thank the entire ProtonMail community for your many kind words of encouragement and support during this difficult time. We built ProtonMail for you, and it is truly an honor to have you standing behind us, in both good times and bad times. We look forward to continuing on this journey towards a more private and free Internet with all of you.

Best Regards,

The Entire ProtonMail Team"

As an IT guy, I can tell you that 100GB DDoS attacks are no joke. They are very difficult to manage, if at all. It's easy to blackhole a single IP on an edge router; it's orders of magnitude more difficult when there are thousands of IPs to contend with. Yes, some of you are leery about the fact they "paid" and still had their rear ends handed to them. I agree that paying was a mistake, but let's all see how this plays out.

[BritTim]

Originally Posted by remotetech

I'm a ProtonMail user
... snip ...
As an IT guy, I can tell you that 100GB DDoS attacks are no joke. They are very difficult to manage, if at all. It's easy to blackhole a single IP on an edge router; it's orders of magnitude more difficult when there are thousands of IPs to contend with. Yes, some of you are leery about the fact they "paid" and still had their rear ends handed to them. I agree that paying was a mistake, but let's all see how this plays out.

As another IT guy, I have no illusions about the difficulty of dealing with large scale DDoS attacks. However, I do not think ProtonMail was properly prepared to respond. It takes a lot of work to set up a resistant architecture, and regularly collect baseline network statistics so anomalies in traffic patterns can be quickly detected and analyzed. Fastmail was prepared but not complacent. That is just as well, because the worst may yet be to come.

[remotetech]

Originally Posted by BritTim

As another IT guy, I have no illusions about the difficulty of dealing with large scale DDoS attacks. However, I do not think ProtonMail was properly prepared to respond. It takes a lot of work to set up a resistant architecture, and regularly collect baseline network statistics so anomalies in traffic patterns can be quickly detected and analyzed. Fastmail was prepared but not complacent. That is just as well, because the worst may yet be to come.

Indeed. I think ProtonMail made the same mistake so many others make when they do one thing well. They were focused on the "security" of the end product and not on the security of the entire thing, front to back -- and that oversight costed them -- literally, in time, money, negative attention.

Per the email I posted above and their own blog, they have taken the necessary steps to help mitigate this kind of attack in future. As Bruce Schneier is fond of saying, "security is a process, not a product". So very true. No amount of router security, firewall security, you name it is going to do any good unless you have a process to deal with the problem should it arise. Apparently some very talented network people stepped up to assist them so they should be well taken care of going forward.

As always, I'm glad to see FM's usual QC shine through with this. I somehow doubt NYI doesn't have the chops to deal with these issues, as they are one of the best in the business, if not the best in the US.

[robn]

Originally Posted by remotetech

As always, I'm glad to see FM's usual QC shine through with this. I somehow doubt NYI doesn't have the chops to deal with these issues, as they are one of the best in the business, if not the best in the US.

As always, we can't sing their praises highly enough. They were onto the problem in a flash, even on the weekend and worked with their network peers to get various protections in place without us really having to get involved (we were mostly just updating each other on progress).

I've seen and heard of responses ranging from "we'll just blackhole your network" to "what's a DDos?" to "good luck!". I heard of one case where the network provider terminated a contract in response to a DDoS. Nothing like from NYI - no excuses, no blame, just got on with fixing the problem.

Incidentally, this is why it's so hard to choose secondary datacentres. NYI have spoiled us rotten so no one else will ever look as good, and it's really hard to know beforehand how they will respond in a crisis.

[remotetech]

Originally Posted by robn

As always, we can't sing their praises highly enough. They were onto the problem in a flash, even on the weekend and worked with their network peers to get various protections in place without us really having to get involved (we were mostly just updating each other on progress).

I've seen and heard of responses ranging from "we'll just blackhole your network" to "what's a DDos?" to "good luck!". I heard of one case where the network provider terminated a contract in response to a DDoS. Nothing like from NYI - no excuses, no blame, just got on with fixing the problem.

Incidentally, this is why it's so hard to choose secondary datacentres. NYI have spoiled us rotten so no one else will ever look as good, and it's really hard to know beforehand how they will respond in a crisis.

Interesting. It is amazing that in 2015, so many people involved in networking don't know how to mitigate threats. NYI has a stellar reputation in the business precisely because they don't rest on their laurels. They are always trying to make it better, faster, safer for their customers. If I could only convince my bosses to move us over to NYI...

Thanks again for your own stellar product with FM. Been a user for almost 10 years for my primary email accounts and see no reason to even look at anyone else.

[fhapgood]

Boston Mass USA 20:23 EST

[verbovet]

Already up. It seems, 10 minutes of network problems.

[fhapgood]

... as of 12:43 EST

[fhapgood]

working fine now

[n5bb]

Yes, as was warned in the Fastmail blog, DDOS attacks on various email providers are happening this week. You can see the resolution at the Fastmail status page, which provides much more transparency than most other email providers:
http://www.fastmailstatus.com/

Rob Mueller of Fastmail was interviewed about these latest DDOS attacks here:
Ransom attacks likely to fade as small email providers resist

Bill

[fhapgood]

Thanks. Good references.

[pjwalsh]

Detailed account of the ProtonMail DDoS attack.
TechRepublic, Nov 13

Inside the ProtonMail siege: How two small companies fought off one of Europe's largest DDoS attacks
What started as a simple digital ransom quickly escalated into a trans-continental networking battle