Mis-delivered mail shows fastmail out of control

biffbulkie · 28 Nov 2002, 10:03 PM

In a chilling indication of how fastmail is spiraling out of control,
my mailbox this morning contained 6 emails that were addressed
to other people who have domains hosted at Fastmail. (I checked the MX records for all 3 domains.) The person addressed at each domain happens to be my fastmail username. I assume that fastmail cannot find these domains and the mail defaults to the fastmail user with that name (insane).

So if you use any of the following domains:

mackraz.com
trickbit.to
brooker.us

You need to check with Fastmail immediately to straighten this out.

I guess what most disturbs me is that if the system cannot find
joeblow@whateverdomain.com it throws the mail into joeblow@fastmail.fm 's mailbox.

Tango · 28 Nov 2002, 10:20 PM

If this is true, this is indeed worrying...I hope Jeremy can clarify this matter soon...

Jeremy Howard · 28 Nov 2002, 10:29 PM

Quote:

Originally posted by biffbulkie
I guess what most disturbs me is that if the system cannot find
joeblow@whateverdomain.com it throws the mail into joeblow@fastmail.fm 's mailbox.

No, that's not what happens. There was unfortunately a misconfiguration with these domains which led to the problem that you saw. In fact, I sent you an email about this problem about 10 minutes ago.

The misconfiguration occured because a user entered a virtual domain address, but no target address. We failed to check for this possibility in our web application. The lack of a target address caused a number of virtual domains to become corrupted for a few hours today. We are currently attempting to contact those people with domains impacted by this problem.

The failure to check for empty targets has actually been in the web app the whole time--it's just that today was the first time that a user entered an address without a target, thus triggering the problem.

biffbulkie · 28 Nov 2002, 10:32 PM

Apparently this has been corrected. The reply from Fastmail indicated that the virtual domain file had been corrupted causing some own domain addresses to not resolve correctly.

I am still angry, as I think this is a Cardinal Sin in the email business. I don't ask for much, but I do INSISIT that email
end up only in the inbox it was intended. I wonder how much of my own domain mail might be scattered out there in Fastmail land, in stranger's email boxes?

Tango · 28 Nov 2002, 10:33 PM

when it rains, it pours...personally, I find this problem more disturbing than any of the outages we've experienced this past few days...anyways, I hope all such bugs can be anticipated and eliminated totally!

Jeremy Howard · 28 Nov 2002, 10:35 PM

Quote:

Originally posted by biffbulkie
I am still angry, as I think this is a Cardinal Sin in the email business. I don't ask for much, but I do INSISIT that email end up only in the inbox it was intended.

You have every right to be angry. I'm really sorry that this happened. We do try really hard to check and double-check the validity of the delivery process, but in this case something slipped through the cracks.

***Edwin*** · 28 Nov 2002, 10:36 PM

Looking at it from the glass-half-full perspective, the total elapsed time between a significant bug being reported on this Forum and a patch being put in place was 29 minutes (and at midnight local time for the core Fastmail team!)

It would probably take that long just to find somebody (anybody) to contact at most faceless companies... who would then need to escalate the issue, organize a bug fixing meeting (and perhaps a budget meeting), etc. etc.

Jeremy Howard · 28 Nov 2002, 10:41 PM

Quote:

Originally posted by Tango
...I hope all such bugs can be anticipated and eliminated totally!

That would be nice... unfortunately there's never a guarantee that any system is 100% bug free... In hundreds of thousands of lines of code, there's bound to be places where there are problems, even although we do try to test every possibility before committing any new code.

For instance, in this particular section of the code, which checks for the validity of new alias records, we have around two dozen separate tests to try to check for every possible potential problem. However, even that wasn't enough in this case to avoid the bug--we had a section that checked for the validity of the target address, which unfortunately treated an empty address as "valid".

ppetru · 28 Nov 2002, 10:51 PM

Quote:

Originally posted by Edwin
Looking at it from the glass-half-full perspective, the total elapsed time between a significant bug being reported on this Forum and a patch being put in place was 29 minutes (and at midnight local time for the core Fastmail team!)

Sure, it's just that when you've lost half a day's worth of mail you tend to see the glass half empty

Other than that, thanks to the team for fixing it but they still owe us some explanations about what they're doing to prevent this in the future.

Petru

Jeremy Howard · 28 Nov 2002, 11:00 PM

Petru, you'll find that those explanations are already sitting in your email inbox.

DippyTwitty · 28 Nov 2002, 11:32 PM

Quote:

Originally posted by Tango
I hope all such bugs can be anticipated and eliminated totally!

That would be utopia, of course.... but in reality where most of us live, you have two options:

Hire a team of engineers with arms full of SQA manuals
Do the best job you can, and charge appropriately

Obviously, unless we all want to pay Jeremy/Rob NASA/Military style costs for the first option, you're going to have to accept the 2nd.

Being in software development myself, Jeremy/Rob could probably build a system, and test it thoroughly enough to ensure that it won't break. But by this time they would have maybe finished a simple mail backend, and wouldn't have started the interface yet!

All of us would love bug-free software, but we all want to get it for $39.95 a year. You're dreaming if you think that's even feasible.

Just my $0.02 rant for the day

Dip

Tango · 28 Nov 2002, 11:36 PM

Heh, I know the enormity of the task...I am just particularly concerned about bugs where the privacy of a user is compromised...

ppetru · 29 Nov 2002, 12:40 AM

Ok, everything seems to be working again now. The lost mail won't come back, but hey, it's just mail

Sorry everyone (especially Jeremy) for my earlier tone and thanks for working on this.

Petru

rwillis · 29 Nov 2002, 12:50 AM

Realizing that no system is bug free, I am very happy with the level of service we get from the FastMail team. I suppose this is easy for me to say since I have experienced any major problems (that I'm aware of) in recent days. Good work, guys!

aha · 30 Nov 2002, 02:29 AM

Quote:

Originally posted by Jeremy Howard
In hundreds of thousands of lines of code, there's bound to be places where there are problems, even although we do try to test every possibility before committing any new code.

Just curious ... but approximately how many lines of code are there in FastMail?

28 Nov 2002, 10:03 PM	#1
biffbulkie Essential Contributor Join Date: Jan 2002 Location: New England Posts: 358	Mis-delivered mail shows fastmail out of control In a chilling indication of how fastmail is spiraling out of control, my mailbox this morning contained 6 emails that were addressed to other people who have domains hosted at Fastmail. (I checked the MX records for all 3 domains.) The person addressed at each domain happens to be my fastmail username. I assume that fastmail cannot find these domains and the mail defaults to the fastmail user with that name (insane). So if you use any of the following domains: mackraz.com trickbit.to brooker.us You need to check with Fastmail immediately to straighten this out. I guess what most disturbs me is that if the system cannot find joeblow@whateverdomain.com it throws the mail into joeblow@fastmail.fm 's mailbox. Last edited by biffbulkie : 28 Nov 2002 at 10:11 PM.

29 Nov 2002, 12:40 AM	#13
ppetru Junior Member Join Date: Mar 2002 Location: Brasov, Romania Posts: 25	Everything working now Ok, everything seems to be working again now. The lost mail won't come back, but hey, it's just mail Sorry everyone (especially Jeremy) for my earlier tone and thanks for working on this. Petru

28 Nov 2002, 10:20 PM	#2
Tango Cornerstone of the Community Join Date: Mar 2002 Posts: 529	If this is true, this is indeed worrying...I hope Jeremy can clarify this matter soon...

28 Nov 2002, 10:32 PM	#4
biffbulkie Essential Contributor Join Date: Jan 2002 Location: New England Posts: 358	Apparently this has been corrected. The reply from Fastmail indicated that the virtual domain file had been corrupted causing some own domain addresses to not resolve correctly. I am still angry, as I think this is a Cardinal Sin in the email business. I don't ask for much, but I do INSISIT that email end up only in the inbox it was intended. I wonder how much of my own domain mail might be scattered out there in Fastmail land, in stranger's email boxes?

28 Nov 2002, 10:33 PM	#5
Tango Cornerstone of the Community Join Date: Mar 2002 Posts: 529	when it rains, it pours...personally, I find this problem more disturbing than any of the outages we've experienced this past few days...anyways, I hope all such bugs can be anticipated and eliminated totally!

28 Nov 2002, 10:36 PM	#7
*Edwin* Administrator Join Date: Aug 2001 Location: UK Posts: 3,118	Looking at it from the glass-half-full perspective, the total elapsed time between a significant bug being reported on this Forum and a patch being put in place was 29 minutes (and at midnight local time for the core Fastmail team!) It would probably take that long just to find somebody (anybody) to contact at most faceless companies... who would then need to escalate the issue, organize a bug fixing meeting (and perhaps a budget meeting), etc. etc.

28 Nov 2002, 11:00 PM	#10
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	Petru, you'll find that those explanations are already sitting in your email inbox.

28 Nov 2002, 11:36 PM	#12
Tango Cornerstone of the Community Join Date: Mar 2002 Posts: 529	Heh, I know the enormity of the task...I am just particularly concerned about bugs where the privacy of a user is compromised...

29 Nov 2002, 12:50 AM	#14
rwillis Essential Contributor Join Date: Apr 2002 Location: PA Posts: 211	Realizing that no system is bug free, I am very happy with the level of service we get from the FastMail team. I suppose this is easy for me to say since I have experienced any major problems (that I'm aware of) in recent days. Good work, guys!