Duplicate Emails - one copy in Spam folder

gardenweed · 28 Jan 2017, 08:01 AM

I find that sometimes, when an email has been sent to spam/junk, a duplicate will also exist elsewhere.

An example is today, I received an email from @facebookmail.com
I have a rule that says any incoming email from that domain is filed directly to a folder called Subscriptions.
An email arrived from that domain today.
One copy sits in Subscriptions, another copy sits in Junk.
The time/date stamp on each is identical.
On a quick look, the headers appear to be identical.
(Interestingly they show passes for DKIM, SPF and DMARC, yet one is sent to junk)

Any ideas as to why two copies are created?

n5bb · 28 Jan 2017, 11:54 AM

The most likely cause is that you are using a custom sieve script with :copy so that the filing rule doesn't cancel the default target folder. See: https://tools.ietf.org/html/rfc3894
If this rule is before the spam filter, the spam filter will only affect the copy of that message which targets default delivery folder.

The other way that a single incoming message can generate two messages is if you use more than one delivery targets - one the default (which is your account main address), a comma, then a folder at your account using a plus+address (or an address at an external domain which redirects back to your account). However, I can't think of an easy way for the two messages to be treated differently by the spam filter unless the message was redirected to your account in such a way that your filing rule acted differently. This can get very complex and I doubt that is what is going on.

Under the first method above you could even have another normal filing rule (without :copy) to the same folder at a later part of your sieve script (after the spam filter). Under normal conditions (multiple message suppression) only one of those two messages would remain in the folder. But if the message was classified as spam, the two copies would be filed in separate folders, so the duplicate suppression would not be applicable.

Bill

gardenweed · 28 Jan 2017, 06:50 PM

Thanks for having a look at this Bill.

I do have some custom sieve using 'copy'.
They are all "redirect: copy abcname @ xyzdomain", where this email address is external to FM, eg gmail, hotmail.

I don't have any copy script directing to a FM folder.

I'll keep an eye on it to see if I can spot a pattern.

jhollington · 28 Jan 2017, 10:32 PM

Do you by any chance have a "keep" directive in a block of Sieve code that would affect that message? This would have the same affect as using the ":copy" directive with "fileinto."

Obviously, if it's in the same block as the "fileinto" directive, you're going to end up with two copies, and the original "kept" copy would still be eligible for spam filtering (assuming that's done later in the script). However, that's the obvious approach — it's still possible to have a "keep" directive elsewhere in the script that could inadvertently affect that message. I tend to avoid using "keep" in my Sieve scripts entirely for that reason.

gardenweed · 28 Jan 2017, 10:40 PM

Had a search for any 'keep' in the sieve, but none found.

n5bb · 29 Jan 2017, 01:25 AM

Remember that there are various ways for messages to appear in your Fastmail account, and various ways for messages to be filed into folders (alias targets, plus addressing, subdomain addressing, rules, etc.). The issue might be caused by an IMAP email client or POP3 fetch. It's also possible for message to arrive at your spam folder via sieve commands you created rather than the default Fastmail spam filter. It might be a good time to remove as much custom sieve as possible and use the normal rules system (which has many more features than in the past).

Bill

gardenweed · 29 Jan 2017, 07:00 AM

Hmm. Haven't used an email client recently.
I guess the issue must be in the sieve code.

My sieve has become messy.
Not because I made it messy with my own edits, but because of the automated changes that FM has implemented on the sieve as they've changed the back end over the last couple of years.

I mostly use the Rules these days (ever since it was allowed to be used along with sieve).

I'll try commenting out the editable sieve that looks redundant.
If that doesn't break anything, I'll go back and delete the commented areas to clean up the sieve.

28 Jan 2017, 08:01 AM	#1
gardenweed Cornerstone of the Community Join Date: Jun 2008 Location: Perth Posts: 664	Duplicate Emails - one copy in Spam folder I find that sometimes, when an email has been sent to spam/junk, a duplicate will also exist elsewhere. An example is today, I received an email from @facebookmail.com I have a rule that says any incoming email from that domain is filed directly to a folder called Subscriptions. An email arrived from that domain today. One copy sits in Subscriptions, another copy sits in Junk. The time/date stamp on each is identical. On a quick look, the headers appear to be identical. (Interestingly they show passes for DKIM, SPF and DMARC, yet one is sent to junk) Any ideas as to why two copies are created?

28 Jan 2017, 11:54 AM	#2
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 8,929	Quite possible if custom sieve used The most likely cause is that you are using a custom sieve script with :copy so that the filing rule doesn't cancel the default target folder. See: https://tools.ietf.org/html/rfc3894 If this rule is before the spam filter, the spam filter will only affect the copy of that message which targets default delivery folder. The other way that a single incoming message can generate two messages is if you use more than one delivery targets - one the default (which is your account main address), a comma, then a folder at your account using a plus+address (or an address at an external domain which redirects back to your account). However, I can't think of an easy way for the two messages to be treated differently by the spam filter unless the message was redirected to your account in such a way that your filing rule acted differently. This can get very complex and I doubt that is what is going on. Under the first method above you could even have another normal filing rule (without :copy) to the same folder at a later part of your sieve script (after the spam filter). Under normal conditions (multiple message suppression) only one of those two messages would remain in the folder. But if the message was classified as spam, the two copies would be filed in separate folders, so the duplicate suppression would not be applicable. Bill

28 Jan 2017, 06:50 PM	#3
gardenweed Cornerstone of the Community Join Date: Jun 2008 Location: Perth Posts: 664	Thanks for having a look at this Bill. I do have some custom sieve using 'copy'. They are all "redirect: copy abcname @ xyzdomain", where this email address is external to FM, eg gmail, hotmail. I don't have any copy script directing to a FM folder. I'll keep an eye on it to see if I can spot a pattern.

28 Jan 2017, 10:32 PM	#4
jhollington Essential Contributor Join Date: Apr 2008 Posts: 371	Do you by any chance have a "keep" directive in a block of Sieve code that would affect that message? This would have the same affect as using the ":copy" directive with "fileinto." Obviously, if it's in the same block as the "fileinto" directive, you're going to end up with two copies, and the original "kept" copy would still be eligible for spam filtering (assuming that's done later in the script). However, that's the obvious approach — it's still possible to have a "keep" directive elsewhere in the script that could inadvertently affect that message. I tend to avoid using "keep" in my Sieve scripts entirely for that reason.

28 Jan 2017, 10:40 PM	#5
gardenweed Cornerstone of the Community Join Date: Jun 2008 Location: Perth Posts: 664	Had a search for any 'keep' in the sieve, but none found.

29 Jan 2017, 01:25 AM	#6
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 8,929	Remember that there are various ways for messages to appear in your Fastmail account, and various ways for messages to be filed into folders (alias targets, plus addressing, subdomain addressing, rules, etc.). The issue might be caused by an IMAP email client or POP3 fetch. It's also possible for message to arrive at your spam folder via sieve commands you created rather than the default Fastmail spam filter. It might be a good time to remove as much custom sieve as possible and use the normal rules system (which has many more features than in the past). Bill

29 Jan 2017, 07:00 AM	#7
gardenweed Cornerstone of the Community Join Date: Jun 2008 Location: Perth Posts: 664	Hmm. Haven't used an email client recently. I guess the issue must be in the sieve code. My sieve has become messy. Not because I made it messy with my own edits, but because of the automated changes that FM has implemented on the sieve as they've changed the back end over the last couple of years. I mostly use the Rules these days (ever since it was allowed to be used along with sieve). I'll try commenting out the editable sieve that looks redundant. If that doesn't break anything, I'll go back and delete the commented areas to clean up the sieve.