"X-Delivered-to:" header missing from recent Fastmail lost mail notification

[Jacinto]

I think I cursed myself this morning with my prior post.

After writing the post, I received a message from Fastmail with the subject "[FastMail] The recent server failure and your account" advising that messages had been lost from one of my accounts last month.

However, it also advised that all lost messages were recovered (I confirmed it with my back-up account at another server).

The python script (getmail) that I use to fetch mail from Fastmail employs the "X-Delivered-to" header to deliver messages to the various local mailboxes.

Fastmail normally adds the header to all messages. However, this morning's lost mail message did not include it.

As a result, getmail went into a tail spin and was unable to fetch that message and others. I had to manually fetch the message.

Would someone at Fastmail kindly explain why the header was missing from the message.

Also, if this is going to be a recurring behavior, I would appreciate a hint for a sieve script to add the header when not added by Fastmail.

Thank you.

--
Jacinto

[elvey]

It should be on all messages. My sieve script will also break badly if it goes missing. Hope to hear that this has been investigated and …

[BritTim]

Originally Posted by Jacinto

I think I cursed myself this morning with my prior post.

After writing the post, I received a message from Fastmail with the subject "[FastMail] The recent server failure and your account" advising that messages had been lost from one of my accounts last month.

However, it also advised that all lost messages were recovered (I confirmed it with my back-up account at another server).

The python script (getmail) that I use to fetch mail from Fastmail employs the "X-Delivered-to" header to deliver messages to the various local mailboxes.

Fastmail normally adds the header to all messages. However, this morning's lost mail message did not include it.

As a result, getmail went into a tail spin and was unable to fetch that message and others. I had to manually fetch the message.

Would someone at Fastmail kindly explain why the header was missing from the message.

Also, if this is going to be a recurring behavior, I would appreciate a hint for a sieve script to add the header when not added by Fastmail.

Thank you.

--
Jacinto

There is a long blog post explaining the 26 Feb disaster and how Fastmail attempted to recover. Many messages were recovered by copying them from various sources. In the process, most likely, some headers normally added by Fastmail's servers did not make it into the message. One will charitably assume that occurrences like those that happened last month will not become a regular feature of our interaction with Fastmail.

[Jacinto]

Originally Posted by BritTim

There is a long blog post explaining the 26 Feb disaster and how Fastmail attempted to recover. Many messages were recovered by copying them from various sources. In the process, most likely, some headers normally added by Fastmail's servers did not make it into the message. One will charitably assume that occurrences like those that happened last month will not become a regular feature of our interaction with Fastmail.

Thank you, BritTim.

The header wasn't missing from the recovered messages, though.

As I said in my original post, it was missing from the message advising about lost and recovered messages.

As to being charitable, although I ended up not losing any mail from this account (haven't heard yet about my other two accounts), I save my charity for the poor, hungry and downtrodden -- not for a commercial enterprise that collects a fee (albeit small) for its services.

--
Jacinto

[robn]

I'm not sure exactly how Bron sent the messages (haven't seen the code), but I expect they were directly injected into the target mailbox, bypassing the normal delivery channels where (among other things) the X-Delivered-To header is added. Its unusual, but this wasn't exactly a normal situation.

[Jacinto]

Originally Posted by robn

I'm not sure exactly how Bron sent the messages (haven't seen the code), but I expect they were directly injected into the target mailbox, bypassing the normal delivery channels where (among other things) the X-Delivered-To header is added. Its unusual, but this wasn't exactly a normal situation.

If that was the case, a sieve stanza to add the missing header would be of no use.

Since I received the message without the header yesterday morning, more than two weeks after February 26th, not sure how yesterday "wasn't exactly a normal situation."

--
Jacinto

[danieldk]

Originally Posted by Jacinto

Since I received the message without the header yesterday morning, more than two weeks after February 26th, not sure how yesterday "wasn't exactly a normal situation."

I think what he is saying is that the messages that were sent out yesterday about the situation two weeks ago, were perhaps directly inserted in the maildirs of the affected accounts rather than being sent through normal SMTP delivery.

[DrStrabismus]

I think you've setup getmail in a rather fragile way. Consider using the MultiGuesser with a default destination. That way getmail can fall back to To/Cc if X-delivered-to is missing and fall back to the default if an email is missing any usable headers. The default should be the same destination as the main user@userdomain address, since that's where FM sends account related emails.

I see that if you receive an email to two addresses in the same account FM delivers one copy with "X-Delivered-To: Unknown" rather than two separate copies. The MultiGuesser should handle this (almost) correctly by falling through and matching all the local addresses in To/Cc. I say "almost" because if one or more of the addresses is not in the To/Cc list (Bcc'ed) Fastmail puts one of these addresses in the X-Delivered-To header in place of "Unknown".

[me0000]

Originally Posted by BritTim

One will charitably assume that occurrences like those that happened last month will not become a regular feature of our interaction with Fastmail.

Yeah right

[Jacinto]

Originally Posted by danieldk

I think what he is saying is that the messages that were sent out yesterday about the situation two weeks ago, were perhaps directly inserted in the maildirs of the affected accounts rather than being sent through normal SMTP delivery.

Thank you, danieldk.

Actually, I understood what he was saying, that's why I made my earlier comment about sieve not being of any use.

--
Jacinto

[Jacinto]

Originally Posted by DrStrabismus

I think you've setup getmail in a rather fragile way. Consider using the MultiGuesser with a default destination. That way getmail can fall back to To/Cc if X-delivered-to is missing and fall back to the default if an email is missing any usable headers. The default should be the same destination as the main user@userdomain address, since that's where FM sends account related emails.

My setup is with a "MultiSorter" destination and a default Maildir which delivers based on the true envelope recipient. It has served me well for years.

As you know the "Cc," "Bcc," etc. headers are not necessarily the envelope recipient. Hence, I consider "MultiSorter" far more accurate and robust than "MultiGuesser" (I believe that getmail's author would agree with me).

Instead of the user having to change local mail fetching parameters because of a server aberration, the server's administrators should go the extra mile to be consistent as to not inconvenience paying subscribers.

--
Jacinto

[DrStrabismus]

Originally Posted by Jacinto

As you know the "Cc," "Bcc," etc. headers are not necessarily the envelope recipient. Hence, I consider "MultiSorter" far more accurate and robust than "MultiGuesser" (I believe that getmail's author would agree with me).

But MultiGuesser only uses To/Cc if there is no address in X-Delivered-to

[Jacinto]

Originally Posted by DrStrabismus

But MultiGuesser only uses To/Cc if there is no address in X-Delivered-to

MultiGueser does not parse for a "X-Delivered-to" header, bur for a "Delivered-to" header instead. Please see code snippet below from "destinations.py."

    def _deliver_message(self, msg, delivered_to, received):
               self.log.trace()
               matched = []
               header_addrs = []
               fieldnames = (
                    ('delivered-to', ),
                    ('envelope-to', ),
                    ('x-envelope-to', ),
                    ('apparently-to', ),
                    ('resent-to', 'resent-cc', 'resent-bcc'),
                    ('to', 'cc', 'bcc'),
               )

Since both "envelope-to" and "x-envelope-to" are included, I suppose that it wouldn't be too difficult to hack the code and add "x-delivered-to" to the tuple.

Because my getmail sources are so heavily customized, I'll have to ponder that. If getmail refuses to fetch a message again because of a missing "X-Delivered-to" header, I probably will.

By the way, I know enough python to be dangerous.

--
Jacinto

[brong]

Yes, it was injected directly into the users' Inbox - deliberately to avoid filters. As others have said, it's a legal email, and your tools should be able to deal with it. X-Delivered-to: would also be missing if you had a Draft moved to Inbox by mistake, or if you detached an attached message using the classic interface...

[Jacinto]

Originally Posted by brong

Yes, it was injected directly into the users' Inbox - deliberately to avoid filters. As others have said, it's a legal email, and your tools should be able to deal with it. . . .

Hello Bron.

I am surprised by your change in attitude. Your obstinate and defensive reply is not consistent with your candid and transparent explanation of recent events that a lot of us praised.

You are blaming the user or the user's software when the aberration was with the server.

A paying subscriber is entitled to expect that messages are delivered to her or his Inbox in a coherent manner (as they usually are).

When they are not, whether intentionally or otherwise, and the user is affected by it and complains about it, the user doesn't want to hear from the mail host that it was the user's fault (when it wasn't).

You cannot become smug in any business and take the customers or clients for granted. It takes a log time to gain a client, but only a split second to lose one.

--
Jacinto