FastMail offline?

[dartsmaster]

Is FastMail offline at the moment? I can't access the website at any of the URL's, and the IMAP server is not responding in Outlook Express. Anyone else having problems?

[PhilaFrog]

Same here... Looks like the website is down

[andrew99]

Oh dear! I don't think there is anything important for me and I'm sure they will sort it soon...

[jmh]

yeah, same problem here. this is ironic because i'm trying to divest from hotmail for many reasons and my search led me to fastmial, which i liked.
and the first day i try to access it, i can't!
is this a common problem for fastmail users?
i don't want to switch to fastmail if i'm going to plagued w/ similar problems.
-jmh

[Jeremy Howard]

I'm really sorry folks--normally with have a script that runs every 15 mins and warns us if there's any problems (first it tries to restart the effect service, and then pages us if it stil doesn't work). But last night we turned the script off before we took down FastMail.FM to make some quick changes (didn't want the script running while we knew FastMail.FM was down!) and then forgot to turn it on again after we brought everything up.

Then for some reason (still looking into why) the IMAP server stopped responding a couple of hours ago, and the test script didn't run, so the IMAP server didn't get restarted and we didn't get paged .

Again, I'm really sorry. I know how annoying it is to have your mail offline. I'll discuss with Rob how we're going to upload changes to FastMail.FM in the future without stopping the test script, so that this problem can't happen again. I'll post back to this thread when we have a new process worked out.

[Jeremy Howard]

Well, I can now officially announce that the whole thing was entirely my fault. In one sense I'm kind of glad that our pager notification was down, since otherwise Rob would have been woken up at 2am due to a problem entirely of my making. I added some additional security checks into our firewall yesterday to block any suspicious packets (e.g. spoofed packets), but misconfigured things such that some purely internal packets (e.g. going from the IMAP server to the web server) could set off the alarms. When this occured it blocked some of it's own internal traffic!

I've turned off the additional security checks for now until I've had a chance to ensure this can't happen again. Basically I'm going to both turn off the checking of internal traffic, change the configuration to reduce false alarms, and change the script that is triggered when an alarm goes off so that it only blocks relevent packets.

Also, we're changing our procedure for updating the server so we won't have to turn the test script off any more. We currently run a script to commit changes to the server, and are adding to that script something that pauses the test script, updates the changes, and then restarts the test script. If the test script is paused for more than an hour, it automatically restarts (although this shouldn't ever happen, because the command we type to start up the web server also starts up the test script now).

[andrew99]

Thanks for letting us know. I appreciate knowing

Andrew

[Lvsheng]

Oh, I am a bit late here. I do remember that I have the same experience too on that period. When FM fails, I just thought that Jeremy tell us before that it is still a Preview. Thus FM will average offline 4 mins per day because developer wanna put in new feature. I am wrong though...

[DaveHanson]

down again, at least for the last 15 minutes.

On the one hand, I SALUTE you guys for a heroic effort! And the communication is MOST WELCOME.

On the other hand, I'm thinking I jumped in a little early for routing all pop mail through your excellent service, since I have been experiencing noticable outages every day now and my livelihood (college professor) is now largely dependent on e-mails.. Perhaps you should start charging pronto so that you can hire a night guy to monitor things?

Any visibility on when these problems will be under control? Feel free to point me to another thread. Thank you!

Best regards, Dave

[Jeremy Howard]

It was down for 20 minutes. Our hosting provider apparently experienced a network flood. See http://www.webhostingtalk.com/showth...804#post186804 for my request for more detail from our upstream provider (which hasn't been responded to yet).

Our monitoring is working fine, but there's not much we can do when our host's network gets flooded. If this isn't shortly resolved to our satisfaction then we'll have to move to another host. However we did a lot of research before we chose this host and these problems certainly aren't the norm there (we tested their web-site for a week and experienced a single outage, which only lasted 1 minute; also I spoke to a number of their customers who were very happy with their service over a number of years).

So hopefully this is just one of those weeks . I'll let you know if we get any clarification from our web host. As you can see from their setup, they should be able to handle most problems, so I will be pressing them to understand what's going on...

[Jeremy Howard]

OK, our host just posted this on webhostingtalk:

We are being flooded again...
Someone does not like us very much this weekend.
We are in the middle of putting up a flood box, so we can track this, and find out who is doinng this, and block them from our network.

Looks like a solution is being put in place. I'll post more information as it becomes available.

FastMail.FM had one outage today of 20 minutes, and one outage yesterday of 10 minutes. Hopefully the flood box will be installed before this happens again.

BTW, there's a good description here of how this floods work, and why they are so hard to deal with. The fact that we've got the assistance of our network provider in this case means that it should be dealt with effectively.

[OtaconTV]

That was a long page, I actually understood some of that. Why isn't the law altered to punish minors that are hackers to be punished as a adult? Imagine a hacker jail. That's scary when you imagine your own computer could be part of that.

[Jeremy Howard]

Careful what you wish for OtaconTV... the law is being changed following Sep 11 to make intrusion of computer systems equivalent to a 'terrorist attack' in certain (but broadly defined) situations.

Do you really want your 14 y.o. child who's just trying to be cool being convicted as a terrorist (which means substantially lessened rights when charged, massively inflated penalties, etc...)

(Err, OK, maybe you don't have a 14 y.o. child... but just pretend for the sake of argument, huh? )

[kirill]

I agree with J, children are children after all.

--
Kirill

[Jeremy Howard]

Now, back on topic after my little rant... Here's some great news from our upstream provider, taken unedited from the webhostingtalk forum, regarding the "flood box" that will identify and fix problems with network flooding in the future:

Box is already in place now, and it is fully functional as of about an hour ago. We had already written a software program to handle this earlier today, in response to our first flood....but had not had a chance to install it before the second flood hit.
What it does...is take a sampling of all packets passing thru our network, over our main switch, and shows what servers and ips are being utilized the most, as well as where the most traffic to and from the network is coming from. Now when/if we get flooded again, we just look at that monitor box, see the IP being targeted, and where the flood is coming from, and we blocks them at the router level

I gotta say, this is all pretty impressive. Within 48 hours of the first attack, they've built and rolled out a system for dealing with it. It sounds a lot like the programs that we put in place at FastMail.FM for avoiding mail abuse, except that our systems do automated lockout whereas this 'flood box' still requires manual input, so we should still probably expect a 5-10 minute outage if this happens again. Let's see how well it holds up...