Fastmail Down Again And Again? - Page 2

brong · 7 Sep 2011, 04:33 AM

No outage back then doesn't mean no outage now. We certainly did have an outage now - I estimate it was about 1/2 hour.

The bad news - same issue as last time, and the first blade again, but in our SECOND bladecentre. A blade died and flooded the network so hard that it stopped all traffic to other blades in the same centre. Pretty impressive.

The good news - we have a separate management network, so I could not only see immediately (well, within a few minutes of actually being aware of the issue and getting my arse in front of a computer with access) I could find the offending host and kill it as soon as I'd taken a screenshot of the kernel panic for posterity.

At which point everything started working again, just like magic. Except for one service, which stayed sad. I'll have to look into why it doesn't recover from DB outages later... for now I'm just restarting it everywhere.

541 · 7 Sep 2011, 04:33 AM

Quote:

Originally Posted by flob

It'd be nice if the "representatives of Opera" would check with their colleagues before making snide comments.

I have to agree with that. The remark was unprofessional.

akorvemaker · 7 Sep 2011, 04:33 AM

The comments from Neil and Johan look pretty reasonable to me. Also, looking at the timestamps on their posts, they were made about 12 hours before this recent outage that is mentioned on the status blog.

brong · 7 Sep 2011, 04:36 AM

Quote:

Originally Posted by 541

I have to agree with that. The remark was unprofessional.

Quoting something 12 hours ago and assuming it's related to a half hour outage now is pretty unprofessional. I saw that post then, and I confirmed that everything was fine. I have a pretty accurate timestamp for this outage starting - only it didn't page me immediately and I couldn't get to a computer immediately when it did.

Rob and Richard also came online to help, but I had already resolved the immediate problem by the time they contacted me, so I sent Rob back to bed and Richard back to his visiting family.

tobiasdr · 7 Sep 2011, 04:38 AM

Are you aware of any email being lost? Test messages i sent from my Gmail account about half an hour ago just when this recent error surfaced have not arrived yet.

541 · 7 Sep 2011, 04:39 AM

Quote:

Originally Posted by brong

Quoting something 12 hours ago and assuming it's related to a half hour outage now is pretty unprofessional. I saw that post then, and I confirmed that everything was fine. I have a pretty accurate timestamp for this outage starting - only it didn't page me immediately and I couldn't get to a computer immediately when it did.

Rob and Richard also came online to help, but I had already resolved the immediate problem by the time they contacted me, so I sent Rob back to bed and Richard back to his visiting family.

The outage was only a half hour ago for me, not 12 hours ago.

tobiasdr · 7 Sep 2011, 04:41 AM

Alright, my test messages arrived now.

brong · 7 Sep 2011, 04:48 AM

Email should not be lost - our servers were not accepting new email without being able to confirm with the database one way or the other, so it will be queueing remotely. 1/2 hour is not long enough for any sane site to start discarding queued emails.

bbrener · 7 Sep 2011, 04:52 AM

The time stamps are inaccurate. My time stamps show 04.00am and here in Phoenix it was 12.00pm and 3.00pm EST. They seem to change when you log in and not when you are just viewing. They were current when they were made. I was watching the posts as new entries came in.

brong · 7 Sep 2011, 04:58 AM

I'm not sure what you mean, bbrener. The 'Date:' header, or the time they were delieverd to your mailbox (internaldate)?

bbrener · 7 Sep 2011, 05:13 AM

No, I mean the date of the post shown in the forum. I believe that was the '12 hours ago' that may have been referred to. I can send you screenshots.
Your post shows a time of 04.58am (Australian time?) in the forum if I just view it. If I log in to the forum, the time of your post changes to 12.58pm which is my current time in PST and obtained from my profile obviously. I was pointing out that some might get confused by that.

rabarberski · 7 Sep 2011, 05:54 AM

@brong: Although you might be aware, I didn't see it mentioned anywhere but beta.fastmail.fm is still down
(site reports "MessagingEngine.com Server Outage ...")

Regular server is OK, and I am fine using it, so not really an issue

brong · 7 Sep 2011, 06:05 AM

rabarberski: well noticed. See if you can guess which machine was the culprit

We don't have multiple servers for the betaweb, so when it's down, it's down. I'm in the process of trying to bring it back up.

rabarberski · 7 Sep 2011, 06:06 PM

Quote:

Originally Posted by brong

We don't have multiple servers for the betaweb, so when it's down, it's down. I'm in the process of trying to bring it back up.

Beta all fine now.

COUME · 7 Sep 2011, 09:47 PM

Quote:

Originally Posted by brong

Email should not be lost - our servers were not accepting new email without being able to confirm with the database one way or the other, so it will be queueing remotely. 1/2 hour is not long enough for any sane site to start discarding queued emails.

I received notification from LinkedIn that my email address was bouncing emails. btw, it was a domain hosted with FM not a FM email address so it.

Thanks

7 Sep 2011, 04:33 AM	#16
brong The "e" in e-mail Join Date: Jul 2004 Location: Melbourne, Australia Posts: 2,696 Representative of: Fastmail.fm	No outage back then doesn't mean no outage now. We certainly did have an outage now - I estimate it was about 1/2 hour. The bad news - same issue as last time, and the first blade again, but in our SECOND bladecentre. A blade died and flooded the network so hard that it stopped all traffic to other blades in the same centre. Pretty impressive. The good news - we have a separate management network, so I could not only see immediately (well, within a few minutes of actually being aware of the issue and getting my arse in front of a computer with access) I could find the offending host and kill it as soon as I'd taken a screenshot of the kernel panic for posterity. At which point everything started working again, just like magic. Except for one service, which stayed sad. I'll have to look into why it doesn't recover from DB outages later... for now I'm just restarting it everywhere.

7 Sep 2011, 04:33 AM	#18
akorvemaker Master of the @ Join Date: Nov 2002 Location: Canada Posts: 1,015	The comments from Neil and Johan look pretty reasonable to me. Also, looking at the timestamps on their posts, they were made about 12 hours before this recent outage that is mentioned on the status blog.

7 Sep 2011, 04:38 AM	#20
tobiasdr Member Join Date: Sep 2011 Posts: 43	Are you aware of any email being lost? Test messages i sent from my Gmail account about half an hour ago just when this recent error surfaced have not arrived yet.

7 Sep 2011, 04:41 AM	#22
tobiasdr Member Join Date: Sep 2011 Posts: 43	Alright, my test messages arrived now.

7 Sep 2011, 04:48 AM	#23
brong The "e" in e-mail Join Date: Jul 2004 Location: Melbourne, Australia Posts: 2,696 Representative of: Fastmail.fm	Email should not be lost - our servers were not accepting new email without being able to confirm with the database one way or the other, so it will be queueing remotely. 1/2 hour is not long enough for any sane site to start discarding queued emails.

7 Sep 2011, 04:52 AM	#24
bbrener Essential Contributor Join Date: May 2002 Location: Phoenix, AZ Posts: 467	The time stamps are inaccurate. My time stamps show 04.00am and here in Phoenix it was 12.00pm and 3.00pm EST. They seem to change when you log in and not when you are just viewing. They were current when they were made. I was watching the posts as new entries came in.

7 Sep 2011, 04:58 AM	#25
brong The "e" in e-mail Join Date: Jul 2004 Location: Melbourne, Australia Posts: 2,696 Representative of: Fastmail.fm	I'm not sure what you mean, bbrener. The 'Date:' header, or the time they were delieverd to your mailbox (internaldate)?

7 Sep 2011, 05:13 AM	#26
bbrener Essential Contributor Join Date: May 2002 Location: Phoenix, AZ Posts: 467	No, I mean the date of the post shown in the forum. I believe that was the '12 hours ago' that may have been referred to. I can send you screenshots. Your post shows a time of 04.58am (Australian time?) in the forum if I just view it. If I log in to the forum, the time of your post changes to 12.58pm which is my current time in PST and obtained from my profile obviously. I was pointing out that some might get confused by that.

7 Sep 2011, 05:54 AM	#27
rabarberski Master of the @ Join Date: Nov 2006 Location: Ghent, Belgium Posts: 1,027	@brong: Although you might be aware, I didn't see it mentioned anywhere but beta.fastmail.fm is still down (site reports "MessagingEngine.com Server Outage ...") Regular server is OK, and I am fine using it, so not really an issue

7 Sep 2011, 06:05 AM	#28
brong The "e" in e-mail Join Date: Jul 2004 Location: Melbourne, Australia Posts: 2,696 Representative of: Fastmail.fm	rabarberski: well noticed. See if you can guess which machine was the culprit We don't have multiple servers for the betaweb, so when it's down, it's down. I'm in the process of trying to bring it back up.