Stop from going to Spam

[JamesHenderson]

Originally Posted by xyzzy

When I refer to a sieve section number I am referring to the numbered comments in the portions of the sieve script you cannot edit. For example section 3 starts with the comment,

### 3. Sieve generated for spam protection

Or to use your examples section 5 is Vacation and section 6 calendar. It's thus easier and quicker to refer to the explicitly numbered comments rather than counting edit sections.

Below is the portion of my initialization code (just after the require of course). Note, I tend to comment a lot, just my coding style.

#
# Initialize known_sender to reflect what X-Spam-Known-Sender indicates (i.e., sender is in the Contacts
# list) or for contacts that use subdomains that cannot be exhaustively specified in the Contacts list.
# Note there is also a special case where X-Spam-Known-Sender will be set to "no" even if the contact is
# on the Contacts list. That can happen if there is something bogus in the headers.  When that happens
# "in-addressbook" is added to the end of X-Spam-Known-Sender so that even with "no" it is still known
# it's in the Contacts list.  The known_sender switch being true always guarantees known senders make it
# to through even those with suspicious headers.
#
if header :matches "X-Spam-Known-Sender" ["yes*", "no*in-addressbook*"] {
  set "known_sender" "true";
} elsif address :matches "From" ["*@*abc.tld",         # whitelisted patterns that contacts doesn't support
                                 "*@*xyz.tld",         # if contacts supported full globbing instead of
                                 "*@*.gov",            # just *@domain.tld these tests woudn't be needed.
                                 "*uvwh*"] {
  set "known_sender" "true"; 
} elsif address :matches "To" ["*@LISTSERV.*"] {       # allow subscribed listserv mail lists
  set "known_sender" "true";                           # "To" is the list, "From" is users posting to list
} else {
  set "known_sender" "false";
}

So I then use "known_sender" in places I want to override that might send the message into spam. For example to bypass my black list testing and more to the point to bypass FM own sieve spam check code. So just before section 1 I have:

#
# Sender is not blacklisted or foreign at this point...
#
# Always bypass the spam score checks for what is considered senders in addition to X-Spam-Known-Sender.
# Unfortunately this also bypasses save-on-SMTP identities and UI generated discard rules but that can't
# be helped since that code below cannot be edited.  Hopefully these will never be important enough for
# known contacts (or whitelisted ones).  If they are then they will have to be explicitly coded.
#
if string :is "${known_sender}" "false" {

And just before section 4 the matching brace to the above test:

} # not known sender

Hi,

So I have tried this, but everything is now ending up in my inbox. I have traced the error to this line:

if header :matches "X-Spam-Known-Sender" ["yes*", "no*in-addressbook*"]

and specifically this part:

"no*in-addressbook*"

For whatever reason, if "X-Spam-Known-Sender" is "no", "known_sender" is being set to "true"

Is there something wrong or out of date with "no*in-addressbook*"?

cheers,
James.

[xyzzy]

If you look in section 3 you will see FM's sieve's own code test' X-Spam-Known-Sender for stuff being in the the contacts list as:

if not header :matches "X-Spam-Known-Sender" "yes*" {

So the first part of my test checks for the same pattern.

My second test for "no*in-addressbook*" covers cases like the following:

X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged"); in-addressbook
X-Spam-known-sender: no ("Email failed DMARC policy for domain"); in-addressbook

because I want names in my contacts list to always get through to my inbox no matter what. Let's not get into a philosophical debate about this. It's just my preference.

The additional match ("no*in-addressbook*") is for "no" followed by anything followed by "in-addressbook" as illustrated in those examples. Probably don't need the trailing star but it shouldn't make a difference.

I wrote that code based only on my what I have seen. I only have my own email to use as source examples. If there's other situations where something in the contacts list can cause X-Spam-Known-Sender to say "no" and also not say "in-addressbook" for a contact that is in the contacts list I don't know about them.

So in all your matches that went into the inbox do you have any where X-Spam-Known-Sender was "no" and didn't contain "in-addressbook"? Are you sure there isn't some pattern matching mistake in the address :matches "From" section?

[BritTim]

I cannot see any particular error with your code, but I always try to simplify tests as far as possible. Tests with several wildcards make me nervous because they can cause tests to timeout. I would be inclined to write those tests as

if header :contains "X-Spam-Known-Sender" ["yes", "in-addressbook"]

Since the possible values of the X-Spam-Known-Sender header are pretty constrained, I would not worry about mismatches here.

[JamesHenderson]

Originally Posted by xyzzy

If you look in section 3 you will see FM's sieve's own code test' X-Spam-Known-Sender for stuff being in the the contacts list as:

if not header :matches "X-Spam-Known-Sender" "yes*" {

So the first part of my test checks for the same pattern.

My second test for "no*in-addressbook*" covers cases like the following:

X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged"); in-addressbook
X-Spam-known-sender: no ("Email failed DMARC policy for domain"); in-addressbook

because I want names in my contacts list to always get through to my inbox no matter what. Let's not get into a philosophical debate about this. It's just my preference.

The additional match ("no*in-addressbook*") is for "no" followed by anything followed by "in-addressbook" as illustrated in those examples. Probably don't need the trailing star but it shouldn't make a difference.

I wrote that code based only on my what I have seen. I only have my own email to use as source examples. If there's other situations where something in the contacts list can cause X-Spam-Known-Sender to say "no" and also not say "in-addressbook" for a contact that is in the contacts list I don't know about them.

So in all your matches that went into the inbox do you have any where X-Spam-Known-Sender was "no" and didn't contain "in-addressbook"? Are you sure there isn't some pattern matching mistake in the address :matches "From" section?

gotcha - looking closer at the spam that went into my inbox, they all were using my email address as the "from" address, and X-spam-known-header was:

X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged");
 in-addressbook

...so your script worked, but as I never send emails to myself, I would still want this kind of email to get spam checked in section 3 - easy enough to add a test for that situation.

thanks!

[xyzzy]

The setting of the known_sender to true and the way it's tested is mainly to get it past the FM generated spam test (but I also use it in other places as well). So once beyond there you're own your own to filter any way you want.

I also have an organize rule for emails sent to myself. I filter them not as spam but place them in a folder named "Unexpected". No one knows my actual FM email address. Only my forwarding service's address is known. So any email sent with an explicit To of my FM address I put in a folder of their own called "unexpected". The only legitimate time the To should be my actual FM address is when I am doing it for testing purposes or FM is sending me something directly. So part of the test is to let those emails reach my inbox (with one exception mentioned later). Below is the sieve code (albeit generated with an actual organize rule - sieve code "reads" so much better and is more concise than the organize rule UI page which defined them).

if 
  allof(
  address :is "To" "xxxxx@fastmail.com",
  not address :matches "From" "*@fastmail.com"
  )
{
  fileinto "INBOX.Unexpected@fastmail^com";
}

This happens to be my first organize rule in the list hence the if test and not elsif. If no other organize rule picks these off it will end up in the inbox by default.

But there is as case I do want to pick off and I can't do it with an organize rule (or if I can it's not worth the effort since it's easier in sieve). That's the case fastmail.com directly sends me email. Obviously they do know my actual FM email address too!

I have it grouped in a special block of code following section 8 where do some other similar operations (alias address filtering but that's off topic). Without going into the actual context of what I do there since that complicates the description I include a test that basically looks like:

if allof(address :is "From" "support@fastmail.com",
                header :matches "Subject" "*.* * Mail fetch retrieval error") {
    fileinto "${1}^${2}";
  }

If this test matches I then filter those messages into the folder named as the email address since I name top level folders by their email address (that's my convention - any subfolders would be special cases of that email address which is why I have it grouped with my alias address handling - actual code is more complicated to handle subfolder name extraction and the conversion of dots to ^'s).

The reason for this test is I have (had) a mail fetch account where about 30% of the time resulted in "Mail fetch retrieval error" being reported from fastmail.com. In the "old days" when mail fetch was using POP it was a timeout error and I could test for this and just ignore it since eventually a future fetch would succeed. Now with IMAP it's just a general "Mail fetch retrieval error" so I sort them into the email address 's folder. If I didn't they would end up in my inbox or if I didn't exclude fastmail.com from my "unexpected" test they would go into "unexpected". I want these error reports to go into the mail fetch identity's folder not elsewhere.

I have 4 other mail fetch identities set up. This problem only happens with one of the (the yahoo servers). While the above test is a general filtering rule to handle these errors, I've since changed one offending email account to forward from there and disabled my mail fetch. Screw 'em!

[Terry]

Would this work

elsif
allof(
not header :contains "X-Spam-Known-Sender" "in-addressbook",
address :is "To" "your email address"
)
{
fileinto "INBOX";
}

[xyzzy]

It's valid code but what are you trying to accomplish and where is this code being placed? As a organize rule it would act only on non-spam.

On face value is says "send any message from anyone not in your contacts" to the inbox. The To, in most cases, will be always be you.

Is this in reference to something I said or some of the code I included in one of the previous posts? I'm a bit confused about the context here.

[Terry]

I just thought that it may work for the OP.

I would have thought the sent email would be put in his inbox instead of the spam folder.

[xyzzy]

Oh, all the way back there. Not sure why that email is being classed as spam if the sender is in the contacts list. Maybe the same kind of X-Spam-Known-Sender examples I show in post 17 so the spam bypass test at the start of section 3 fails and the stuff really has a high enough spam scores.

The code I showed is a more general case for overriding the spam check when the sender is in the contacts list. Senders will always get past the spam check no matter what. Your code isn't going to work since as a organize rule it's too late since the OP's message would still be picked off by the spam test. And if you place your code before the spam check almost all email that's not in the contacts will end up in the inbox.

So to summarize the object of the "sport" was to only bypass the spam check if the sender is in the contacts list leaving everything else the same. That's all. Once past the spam check organize rules can do their thing to filter what they want and what's left will end up in the inbox by default.

[JamesHenderson]

Originally Posted by xyzzy

Not sure why that email is being classed as spam if the sender is in the contacts list. I supposed then X-Spam-Known-Sender is no for some reason.

they spoofed my own email address so the email was:
from: me
to: me

The spam rules caught it correctly, but your script was (of course) bypassing the spam rules because the sender (me) was in my contacts list.

This will be a problem for all spam that uses your own contacts as their from: address (I guess because you also are also in their contacts list when their email was compromised). I get those from time to time.

[JamesHenderson]

So, I have been thinking about this some more - anything to stay busy during the lockdown

I wrote these lines above section 1:

### 0.  Whitelist test 
if address :matches "from" "name@domain.tld"
{
	deleteheader "X-Spam-Known-Sender";
	addheader "X-Spam-Known-Sender" "yes from sieve Whitelist";
}

And I sent an email to myself from name@domain.tld

The code "worked" in that I got this header:

X-Spam-Known-Sender: yes from sieve Whitelist

There were no other versions (i.e. the deleteheader worked), but the email still went into my spam folder.

Agh! Where has my logic failed? The spam section starts with

if not header :matches "X-Spam-Known-Sender" "yes*" {

so surely it should have worked? The sieve test webpage also reported it should have worked (I know that test isn't perfect)..

Here are the headers in question:

X-Spam-Known-Sender: yes from sieve Whitelist
Return-Path: <name@domain.tld>
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42])
	 by sloti5d1t04 (Cyrus 3.1.7-1130-gd0f8b30-fmstable-20200414v1) with LMTPA;
	 Tue, 14 Apr 2020 06:29:16 -0400
X-Cyrus-Session-Id: sloti5d1t04-1586860156-318776-2-9389288406526221069
X-Sieve: CMU Sieve 3.0
X-Spam-sender-reputation: 0 (email)
X-Spam-score: 5.8
X-Spam-hits: BAYES_50 0.8, DCC_CHECK 1.1, FREEMAIL_FROM 0.001, HTML_MESSAGE 0.001,
  ME_SENDERREP_DENY 4, RCVD_IN_DNSWL_NONE -0.0001,
  RCVD_IN_MSPIKE_H2 -0.001, SPF_HELO_PASS -0.001, SPF_PASS -0.001,
  LANGUAGES unknown, BAYES_USED user, SA_VERSION 3.4.2
X-Spam-source: IP='40.92.19.13',
  Host='mail-dm6nam11olkn2013.outbound.protection.domain.tld',
  Country='US', FromHeader='com', MailFrom='com'
X-Spam-charsets: plain='iso-8859-1', html='iso-8859-1'
X-Resolved-to: me@fastmail.com
X-Delivered-to: me@mydomain.tld
X-Mail-from: name@domain.tld

Any takers?

[SideshowBob]

Reproducing the problem exactly was a bit of a hassle for me, but I tried the following at the top of my custom sieve

if header :contains "subject" "wibble" {
   deleteheader "X-Spam-Known-Sender";
   addheader "X-Spam-Known-Sender" "yes from sieve Whitelist";
   if header :matches "X-Spam-Known-Sender" "yes*"{
      fileinto "INBOX.test";
   }
   stop;
}

and I found it worked as expected.

Are you sure the email isn't being sent to spam later? In a rule or by an external spam filter in a client. This would go some way to explaining the next bit.

This looks like a problem:

X-Spam-sender-reputation: 0 (email)
...
X-Spam-hits: BAYES_50 0.8, ...  ME_SENDERREP_DENY 4,

The sender reputation is supposedly based on how you've treated email from the domain or address. Hopefully the address reputation overrides the domain. Presumably you've allowed email from this address to remain in your spam folder and so contribute to the bad reputation.

[JamesHenderson]

Hi SideshowBob,

Thanks for verifying my snippet - appreciated.

I used an address that I knew would "normally" end up in junk otherwise I couldn't test the snippet for being able to whitelist.

I knew it generated the header correctly, but couldn't understand why it then didn't stop Fastmail's sieve section 3 from sending to junk.

I take your point about something else sending it to junk instead, but I have no other rules that would do that and I only use the web interface or Fastmail's own app (iPhone/iPad).

My intention is to create a whitelist without addresses being in my contacts (for addresses that I will never send to).

J.

edit: I should add that if I add the email address to my contacts, it does not go to junk - it goes to my inbox.

[xyzzy]

Originally Posted by SideshowBob

and I found it worked as expected.

It did? Who were you sending that to? Yourself? If so then the test of X-Spam-Known-Sender for "yes*" was always going to test true.

Here's what I tried:

if allof(true, header :is "Subject" "test") {
  deleteheader "X-Spam-Known-Sender";
  addheader "X-Spam-Known-Sender" "yes!";
  if header :matches "X-Spam-Known-Sender" "*" {
    addheader "Debug" "X-Spam-Known-Sender=${1}";
  } else {
    addheader "Debug" "Huh?";
  }
  stop;
}

Note I have "Debug" as an added header to display. And here's what I see in the received email (sending to myself):

X-Spam-Known-Sender: yes!
Debug: X-Spam-Known-Sender=yes ("Self sent message"); in-addressbook,  self-send

Since I'm sending to myself X-Spam-Known-Sender is of course "yes" but I modified it to just "yes!" (i.e., with an exclamation point at the end) to tell the difference. But it appears to be testing the original header and not the modified one.

I've submitted a ticket on this.

Update: Sieve Tester has the same behavior. I wonder if there's something in an RFC about this.

[JamesHenderson]

Originally Posted by xyzzy

But it appears to be testing the original header and not the modified one.[/i]

Aha! ...well done!

I couldn't understand why my script *only* worked when the email address was still in my contacts (and didn't work when it wasn't), but that would explain it. ...albeit I don't understand how it is possible to test a header that has been deleted.

I appreciate you sending the bug report. I did ask my original question of Fastmail support but haven't received an answer yet, If you want to associate your bug report with my query, the ticket ID is: PTN636591

I wonder if it is possible to to explicitly test the for of the last occurrence of a header (in the same way that *deleteheader* can have an index number to specify which occorance is deleted (not specifying an index deletes all occurrences).