Duplicate-message "hiding" behavior in macOS Mail.app

[jdmc]

NOTE: The content of this post started out as a question, but I think I've found the answer. I'm posting all the details here anyway, in case they may prove useful to someone else in the future.

— — —

Original message title: Strange behavior of a possibly re-sent incoming message

I normally work with my FastMail account using macOS Mail.app as my IMAP client. On Sunday morning (yesterday) I was going through one of my account's mailboxes, unsubscribing to a lot of unsolicited recurring emails from various vendors for my business. In one particular case, there was an email that was shown in my message list as having been received at 9:42am on Fri Aug 31 [my time, U.S. CDT (-0500)]. I replied to the message with an unsubscribe request, as per the instructions in the message. I also marked the originally-received message with a gray flag in Mail.app, which is my way of recording the fact that I unsubscribed from a certain email. A couple of minutes later, I received a short email confirming that my unsubscribe request had been processed. Nothing unusual as of yet.

But half an hour later, something weird happened. As it appeared in my message list in Mail.app, the original message — which had been received on Friday morning and now had a gray flag applied to it — suddenly relocated itself to the top of the message list, with a new "Date Received" of that same moment, namely 9:02am on Sun Sep 2 [my time]. This new "version" of the message (if that's the right word) maintained the gray flag that I had applied to the original. However, upon viewing the full headers of the new version, it became evident that although the message had originated from the sender on Friday morning, at some point in the delivery chain, the timestamps suddenly changed to Sunday morning. Meanwhile, the original "version" message no longer appeared in the message list at its previous location, when sorted by date received.

In other words, the way the change appeared in the Mail.app message list made it look as if the original message had somehow been edited to show a different time of arrival, though it had the same message content. This impression was strengthened by the fact that the gray flag that I had applied to the original was now visible next to this new "version", while the old version had disappeared entirely.

Interestingly, however, when I logged into the FM Web interface and viewed the email subfolder in question, both instances of the message appeared, each with its true time of arrival. Comparing the raw source of the two, the delivery headers revealed that for some unknown reason, a host in the *.prod.outlook.com subdomain had re-issued the message to the next receiving host in the chain, in such a way that the "Message-ID:" header value was unchanged from the original.

One might expect that when the re-issued message arrived at FastMail, the system would've noticed that it bore the same Message-ID as the original version, and would therefore suppress its delivery — basically meaning, delete and disregard the new version. However, this did not happen. While researching this issue, I came across a post in this forum from 14 years ago (!) in which FastMail cofounder Jeremy Howard stated: "We expire dupe delivery records after 1 day." If this remains true today, it would explain why the second arrival of the message was not detected as a duplicate, as it came about 48 hours after the original.

So, both instances of the message were actually present in the same mail subfolder at Fastmail. It turns out, apparently, that it was Mail.app that had noticed that the two messages shared the same Message-ID value, and had responded by hiding the older version from view, while carrying over to the duplicate the display of the gray flag I had manually applied to the original.

I returned to the FM Web interface and moved the later instance of the message to the Trash folder. After giving macOS Mail some time to notice and adjust to the change, the original instance of the message reappeared in the message list displayed in Mail.app, just as it had before the arrival of the duplicate. Evidently, Mail.app "hides" all but the most recently received instance of an email message with the same Message-ID value, but only if the duplicates are within the same mail subfolder. A bit odd, perhaps, but there ya go!

Cheers,
~ Justin