Google groups

[mayzie@fastmail]

Originally Posted by n5bb

I agree with all of the comments by JeremyNicoll. [list][*]Please look at the contents of that message. Does it appear to be a good message, or is it a spam or phishing message? Do you know the sender? If so, you might ask them (if possible in a different way, such as over the telephone or using a different non-Gmail email address) if they actually sent that message.

Yes, the message is fine - it's a regular email to the group mailing list which in this case happens to be a neighbourhood group. I get 10-15ish messages a day from this mailing list, and about half go to into the Spam folder for reasons I can't figure out. The occasional email might be offering items for sale or something, but many go into my Spam folder that are just talking about regular neighbourhood issues.

Originally Posted by n5bb

Also look at this Fastmail help topic:
Improving spam protection
Scroll down to Your personal spam database and tell us how many spam and non-spam messages have been reported.

Spam learned
25551

Non-spam learned
849

I'm starting to think the problem is likely related to my mail forwarding. I thought entering the SMTP addresses in the "Forwarding hosts" field would remove those issues, but perhaps it's not enough.

[n5bb]

Originally Posted by mayzie@fastmail

... I'm starting to think the problem is likely related to my mail forwarding. I thought entering the SMTP addresses in the "Forwarding hosts" field would remove those issues, but perhaps it's not enough.

We didn't understand that those were forwarded messages! Yes, forwarding in general breaks DMARC email authenticity checks (DKIM and SPF). Some other email services (such as Yahoo and Gmail) block some incoming messages which don't follow DMARC rules. See:
https://dmarcian.com/yahoo-and-google-dmarc-required/
Also see Fastmail help:
https://www.fastmail.help/hc/en-us/a...cation#inbound

The best thing to do is to use a Fastmail managed mail address to receive these messages directly (without forwarding) or to use mail fetch for Fastmail to grab mail from the other accounts. See:
https://www.fastmail.help/hc/en-us/a...ses-with-fetch

Bill

[JeremyNicoll]

Originally Posted by mayzie@fastmail

I'm starting to think the problem is likely related to my mail forwarding. I thought entering the SMTP addresses in the "Forwarding hosts" field would remove those issues, but perhaps it's not enough.

One problem here is that I'd expect Google to have many servers from which they could forward mails. Maybe you don't have the right ones listed? And even if you do (now) by tomorrow they could have changed. I think you'd need to ask FM Support if there's some way to specify "Google's servers" without having ever to know all those servers' names.


Regarding resetting a BAYES database, I think it's possible to have that done by asking FM support.


Regarding the impossibility of listing many (unknown to you) senders in Contacts (or wherever) I would HOPE that the contents of the "From:" header in a mail is irrelevant for these mail-list/discussion group emails. The sender that matters might be (eg) "<groupname>@groups.io" eg "H390-MVS@groups.io", and whatever the equivalent is in one of the google groups mails.


Even for genuine personal mails where at least it's more likely that they were sent from one person (or a system pretending to be one person) the value in the "From:" header is very often misleading. Then, I'd hope that FM would be looking at the sender info on the SMTP envelope which routed the mail to you at FM, not the FROM value.

But - again - you might need to ask FM about that.

[hadaso]

Incoming email from mail lists usually have some list-specific headers that can be used for filtering. These are from the headers of a message from Google Groups:

Precedence: list
Mailing-list:
List-ID:
List-Post:
List-Help:
List-Archive:
List-Unsubscribe:

[JeremyNicoll]

Originally Posted by hadaso

Incoming email from mail lists usually have some list-specific headers that can be used for filtering. These are from the headers of a message from Google Groups:

Precedence: list
Mailing-list:
List-ID:
List-Post:
List-Help:
List-Archive:
List-Unsubscribe:

Yes, indeed, but no ordinary mail rule that tests one of these headers will take effect until after the FM-provided spam filter actions have occurred, so they'd be too late to fix the OP's problem. [Most of my own mail-list mail rules do use the List-Id: header, but automatic spam filtering isn't affecting me.]

The only non-devious-Sieve-way to bypass the spam filtering for specific senders is by putting those senders in one's Contacts list, which is why I was thinking-aloud above about what FM regard as the sender of a mail-list/group mail; the List-Xxxxx headers don't solve that problem.


Of course one can also choose to create custom Sieve content before any of the auto-generated Sieve rules, but that shouldn't be necessary.

[mayzie@fastmail]

Just an update that I did change my subscription to one of the mailing lists to my Fastmail account - so no more forwarding. I thought that solved the problem completely but there are still a handful of messages going to Spam (fewer than before, I think, but still occasional).

The BAYES scores are still high, even for innocuous messages discussing local issues.

Interestingly, even though the From line in my email shows the individual sender's name only, I see that the raw data shows this:

Sender: dufferingrovefriends@googlegroups.com

And that address is definitely in my Contacts.

Here's the Spam headers of a recent email that went to Spam even though it wasn't forwarded:

X-Spam-known-sender: no
X-Spam-sender-reputation: 0 (email)
X-Spam-score: 7.7
X-Spam-hits: BAYES_80 2, HEADER_FROM_DIFFERENT_DOMAINS 0.001, MAILING_LIST_MULTI -1,
ME_HAS_VSSU 0.001, ME_SC_NH -0.001, ME_SENDERREP_DENY 4,
RCVD_IN_DNSWL_NONE -0.0001, RCVD_IN_MSPIKE_H3 0.001,
RCVD_IN_MSPIKE_WL 0.001, RCVD_IN_PSBL 2.7, SPF_HELO_NONE 0.001,
SPF_PASS -0.001, LANGUAGES en, BAYES_USED user, SA_VERSION 4.0.1
X-Spam-source: IP='209.85.161.58', Host='mail-oo1-f58.google.com', Country='US',
FromHeader='ca', MailFrom='com'
X-Spam-charsets: plain='UTF-8'

I guess the only thing is to contact Fastmail support.

[JeremyNicoll]

Originally Posted by mayzie@fastmail

Interestingly, even though the From line in my email shows the individual sender's name only, I see that the raw data shows this:

Sender: dufferingrovefriends@googlegroups.com

And that address is definitely in my Contacts.

The issue with entries in Contacts maybe getting round the FM spam filters is whether or not FM's logic looks at mail-list-specific headers for mail-list mails. Only FM will know for sure... And if not, why not?

Originally Posted by mayzie@fastmail

J
The BAYES scores are still high, even for innocuous messages discussing local issues.

Here's the Spam headers of a recent email that went to Spam even though it wasn't forwarded:

X-Spam-known-sender: no
X-Spam-sender-reputation: 0 (email)
X-Spam-score: 7.7
X-Spam-hits: BAYES_80 2, HEADER_FROM_DIFFERENT_DOMAINS 0.001, MAILING_LIST_MULTI -1,
ME_HAS_VSSU 0.001, ME_SC_NH -0.001, ME_SENDERREP_DENY 4,
RCVD_IN_DNSWL_NONE -0.0001, RCVD_IN_MSPIKE_H3 0.001,
RCVD_IN_MSPIKE_WL 0.001, RCVD_IN_PSBL 2.7, SPF_HELO_NONE 0.001,
SPF_PASS -0.001, LANGUAGES en, BAYES_USED user, SA_VERSION 4.0.1
X-Spam-source: IP='209.85.161.58', Host='mail-oo1-f58.google.com', Country='US',
FromHeader='ca', MailFrom='com'
X-Spam-charsets: plain='UTF-8'

I guess the only thing is to contact Fastmail support.

Yes...

BAYES isn't the biggest problem in this example though: it's only caused 2 of your 7.7 points. The rest come from ME_SENDERREP_DENY (4 points) & RCVD_IN_PSBL (2.7 points) ... which of course add up to 6.7 pts, but the MAILING_LIST_MULTI (-1 point) reduces the total score by 1.

Both ME_SENDERREP_DENY (4 points) & RCVD_IN_PSBL (2.7 points) mean that the server which sent this specific email to you has a terrible reputation.

Did emails which are now NOT going into spam get sent by a different server?

[mayzie@fastmail]

Originally Posted by JeremyNicoll

Both ME_SENDERREP_DENY (4 points) & RCVD_IN_PSBL (2.7 points) mean that the server which sent this specific email to you has a terrible reputation.

I checked headers for a message that correctly went to my Inbox and it also has the "Senderrep_deny4" but not the RCVD_IN_PSBL issues:

X-Spam-hits: BAYES_60 1.5, MAILING_LIST_MULTI -1, ME_HAS_VSSU 0.001, ME_SC_NH -0.001,
ME_SENDERREP_DENY 4, RCVD_IN_DNSWL_NONE -0.0001, RCVD_IN_MSPIKE_H3 0.001,
RCVD_IN_MSPIKE_WL 0.001, SPF_HELO_NONE 0.001, SPF_PASS -0.001,
LANGUAGES en, BAYES_USED user, SA_VERSION 4.0.1

Originally Posted by JeremyNicoll

Did emails which are now NOT going into spam get sent by a different server?

I'm not exactly sure how to find the server info in the raw heading data. Would it be this (from a message that incorrectly went to Spam):

Received: from mail-oo1-f62.google.com (mail-oo1-f62.google.com [209.85.161.62])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by phl-mx-05.messagingengine.com (Postfix) with ESMTPS id B2A055D200D8
for ...

and this (from a message that correctly went to Inbox):

Received: from mail-oa1-f59.google.com (mail-oa1-f59.google.com [209.85.160.59])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by phl-mx-02.messagingengine.com (Postfix) with ESMTPS id AC04A30A00E0
for ...

If *.google.com is the server name, every message seems to have a slightly different one.

[JeremyNicoll]

Originally Posted by mayzie@fastmail

I checked headers for a message that correctly went to my Inbox and it also has the "Senderrep_deny4" but not the RCVD_IN_PSBL issues:

X-Spam-hits: BAYES_60 1.5, MAILING_LIST_MULTI -1, ME_HAS_VSSU 0.001, ME_SC_NH -0.001,
ME_SENDERREP_DENY 4, RCVD_IN_DNSWL_NONE -0.0001, RCVD_IN_MSPIKE_H3 0.001,
RCVD_IN_MSPIKE_WL 0.001, SPF_HELO_NONE 0.001, SPF_PASS -0.001,
LANGUAGES en, BAYES_USED user, SA_VERSION 4.0.1

This mail was ok, but only just: BAYES gave it 1.5 pts, which added to the ME_SENDERREP_DENY 4 would have exceeded the FM default spam filter score threshold of 5 points ... if it wasn't for the MAILING_LIST_MULTI -1 which brought the score down to 4.5.

Originally Posted by mayzie@fastmail

I'm not exactly sure how to find the server info in the raw heading data. Would it be this (from a message that incorrectly went to Spam):

Received: from mail-oo1-f62.google.com (mail-oo1-f62.google.com [209.85.161.62])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by phl-mx-05.messagingengine.com (Postfix) with ESMTPS id B2A055D200D8
for ...

and this (from a message that correctly went to Inbox):

Received: from mail-oa1-f59.google.com (mail-oa1-f59.google.com [209.85.160.59])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by phl-mx-02.messagingengine.com (Postfix) with ESMTPS id AC04A30A00E0
for ...

If *.google.com is the server name, every message seems to have a slightly different one.

Yes, those headers look relevant, as they detail the movement of the mails concerned between google and FM. The google server names might matter less than their IP addresses (which are also different - 209.85.161.62 vv 209.85.160.59).


Not only were different google servers involved, two different FM servers were too: phl-mx-05.messagingengine.com and phl-mx-02.messagingengine.com. That raises the possibility that FM have SpamAssassin configured differently on their various incoming-mail servers, which might be why you're seeing differences in behaviour.

[voxson]

I've had similar problems many times. The only certain way to fix it is to put Sieve code in the very first box. These are applied before spam checking.

To avoid creating Sieve code yourself, make the rules using the web interface so it auto creates code and then copy-paste it to the first box.

[JeremyNicoll]

Originally Posted by voxson

I've had similar problems many times. The only certain way to fix it is to put Sieve code in the very first box. These are applied before spam checking.

To avoid creating Sieve code yourself, make the rules using the web interface so it auto creates code and then copy-paste it to the first box.

It would be better if FM could fix it (for everyone) though, not least because FM might one day disallow "custom Sieve code".


Also the auto-generated blocks of Sieve code, each block corresponding to one rule in the GUI rule editor, these days have unnecessary extra logic in them to fit into a more complex framework than simple-as-possible rules need to have.

If the OP's only solution is custom Sieve code, I think they'd be better to use the simplest possible code. Several people here can advise on that, not least how to test it safely.

[voxson]

The best option would be, in my opinion for each rule in the GUI editor to have an additional check box option to run before/after spam check.

[hadaso]

Originally Posted by JeremyNicoll

... FM might one day disallow "custom Sieve code".
....

I really hope this never happens because it would make my mail filtering a complete mess.


The logic of the mail filtering system is wrong: it applies the same rules regardless of incoming address. There is absolutely no reason to apply spam filtering for addresses that never receive spam, or to apply aggressive filtering for addresses that receive little spam.


My Sieve script checks incoming address first (X-Delivered-To header) and then there are separate blocks of code applied to separate groups of incoming mail. The messages sent to my work related addresses have a separate set of rules that apply only to them, and the messages sent to the address I use to correspond with friends and family have an entirely separate set of rules, an the many addresses I use for correspondence with various commercial sites have yet another separate set of rules).


On the topic of this thread: if messages sent from a Google groups server receive a high spam score just because of the IP of the sender (ME_SENDERREP_DENY 4) then something is wrong in the way Fastmail handles it. It probably affect other users too. They should check why a Google server generates such a huge contribution to the span score.