Filtering using 'headers doesn't contain'

[50ftQueenie]

Hi fellow Runbox users,

Filters are among the options with Runbox that I use the most.

Most filters I have created look like this:
Message where: [option] [contain] [...] will be [option] [if applicable]

Recently I added two filters using [doesn't contain]. These filters look like this:

Message where: [header] [doesn't contain] X-DSPAM-Result: Whitelisted
will be [saved to folder] Trash
Message where: [header] [doesn't contain] X-DSPAM-Result: Innocent
will be [saved to folder] Trash
Both filters are running at order value 1 (before I set the value to 0, but it did not make any difference)

The strange thing is that messages that do match these filters are sent to the Trash folder, but that is exactly what I don't want the filters to do . I'm positive that these filters do not conflict with the filters I have already created.

So my question is: could anyone please test this and see what happens?

All comments and suggestions are very much appreciated of course Thanx!

Kind regards,
50ftQ.

[marc_otten]

Hi Queenie,

Correct me if I'm wrong, but you say that you've created the following filters:[list=1][*]Message where: [header] [doesn't contain] X-DSPAM-Result: Whitelisted will be [saved to folder] Trash[*]Message where: [header] [doesn't contain] X-DSPAM-Result: Innocent will be [saved to folder] Trash[/list=1] I've translated this to normal English (for my own understanding) and this means something like: when the messageheader does not contain X-DSPAM-Result "Whitelist" or "Innocent", move the message to the Trash-folder.

So actually, the filters seem to behave like you wanted them to (eg: they are moving messages to trash). Or am I missing something?

-Marc

[50ftQueenie]

Hi Marc,

Thanks for your quick response: ) The translation is correct, but in my opinion the filters don't behave like I wanted them to.

It would be correct if the messages are moved to the Trash folder if I would choose to say: when the messageheader contains (instead of what I chose: does not contain) X-DSPAM-Result 'Whitelist' or 'Innocent', move the message to the Trash folder.

I would expect that all messages matching my rules are delivered into my Inbox and all other messages that do not match these rules are moved to the Trash folder.

Kind regards,
50ftQ.

[carverrn]

What are you wanting to do that you created these filters? I don't think you're filters do what you expect. Here's why ...

Runbox spam filtering works by first putting headers in the messages that flag the messages as spam or non-spam. This is done when the messages arrive.

Later, during the filter processing phase, there is a "built-in" Runbox filter that runs at an Order value of -1 (see Order Values for Filter Processing which I have made a "sticky" message since I keep digging it up so often) which moves all messages flagged as spam to the designated spam folder.

Since the "move spam" filter runs at -1, it runs before your filter (with either a 0 or 1 Order value). Since most messages that "don't contain" the phrases "X-DSPAM-Result: Whitelisted" or "X-DSPAM-Result: Innocent" probably have "X-DSPAM-Result: Spam", will already have been moved to the designated spam folder.

Now lets say the message says "X-DSPAM-Result: Whitelisted". It gets to your first filter and passes just fine. BUT (see it's a big but), when it gets to the second filter it checks for "X-DSPAM-Result: Innocent". The message DOES NOT contain "X-DSPAM-Result: Innocent" so it will move it to the Trash. I bet that's not what you wanted is it?

If the message contained "X-DSPAM-Result: Innocent", then it wouldn't contain "X-DSPAM-Result: Whitelisted" and your first filter would move it to the Trash. Not what you wanted either is it?

So, most likely, pretty much all of your mail is probably going to the Trash. Is that what you're seeing?

One final issue to keep in mind. When the Runbox servers get bogged down with too much incoming mail Runbox will shutdown DSPAM to speed up the incoming mail processing. When this happens, there will be no X-DSPAM headers at all in a message. So checking for "doesn't contain" on an X-DSPAM header will catch all the messages at those times.

Regards,
Rich

[carverrn]

Originally posted by 50ftQueenie
I would expect that all messages matching my rules are delivered into my Inbox and all other messages that do not match these rules are moved to the Trash folder.

That may be the mistake there.

The test goes:

IF (condition is true) THEN (perform action)

So in your case:

IF (doesn't contain "X-DSPAM-Result: Whitelisted") THEN (save to folder Trash)

and

IF (doesn't contain "X-DSPAM-Result: Innocent") THEN (save to folder Trash)

Regards,
Rich

[50ftQueenie]

Hi Rich,

Sorry for replying so late. I've got a new job and somehow I've managed to stay over 6 hours behind the computer every day according to the statistics of Workpace. And exceeding the maximum hours of computerwork as stated by Dutch law... So I'm trying to bring down that average, also when I'm at home.

Rich, thank you for answering my question. I do understand your post now in which you refer to your excellent post which was made sticky. I just did not realise at first that my filters would not work as I thought, so all of my messages did end up in the Trash folder indeed.

My first thought was to find a way in decreasing the number of spam messages by having them directly deleted instead of moving them to a folder (in my case the Trash folder). So I came up with the idea of the filters using 'Whitelisted' and 'Innocent', just to see if that would work well. But it turned out not and you're completely right that messages may lack X-DSPAM headers, because I've seen that a couple of times now (spam messages ending up in my Inbox, because I set a filter that moves messages containing 'X-DSPAM-Result: Spam' to the Trash folder).

I already have a filter that deletes messages (X-SPAM-Level: ***) instead of moving them to the Trash folder. For me that works well, because I've never received any legitimate message with a higher X-SPAM-Level than 2. And up til now I've never got any complaint about a message that was not received by me, fortunately. But of course all messages with a lower score than 2 end up in the Trash folder and I receive quite a few every day.

But to decrease the number of spam messages in my Trash folder I came up with another idea. I haven't tested it yet, but hopefully it can work well given your own personal circumstances. I've analysed all of my messages to see what kind of tests were run in the X-SPAM-Status header and compared it to spam messages. Some tests (see http://spamassassin.apache.org/tests_3_0_x.html) do come up more than once in spam messages, but do not appear in legitimate messages (in my case!).

For instance these are some tests:
- URIBL_SBL
- URIBL_OB_SURBL
- RCVD_IN_NJABL_DUL
- RCVD_IN_SORBS_DUL

So I'm thinking of setting up some filters that delete all messages that contain any of the above tests. Of course keeping in mind how likely it is that you will receive a legitimate message in the future that does contain one of the tests. And thus risking of losing that message without knowing it. So I would like to emphasize that this idea may work for me, but not for others.

Well, I hope I can add something (useful or not ) to the discussion of decreasing spam messages and I'm curious about the reactions.

Kind regards,
50ftQ.

[marc_otten]

Hi 50ftQueenie,

But to decrease the number of spam messages in my Trash folder I came up with another idea. I haven't tested it yet, but hopefully it can work well given your own personal circumstances. I've analysed all of my messages to see what kind of tests were run in the X-SPAM-Status header and compared it to spam messages. Some tests (see http://spamassassin.apache.org/tests_3_0_x.html) do come up more than once in spam messages, but do not appear in legitimate messages (in my case!).

This actually sounds like a good idea. I've set up some 'personal' filters for my own convenience and they're actually very usefull in my "war on spam".

<SNIP> Of course keeping in mind how likely it is that you will receive a legitimate message in the future that does contain one of the tests. And thus risking of losing that message without knowing it. So I would like to emphasize that this idea may work for me, but not for others</SNIP>

To make sure you do not loose legitimate messages, you can consider keeping your spam in a "Spam-folder". Then monitor this "Spam-folder" on a regular bases (for example, once a week) to verify whether legitimate mail is in it or not. After a while, the desciscion can be made to skip the "spam-folder" and delete the filtered messages immediately.

Well, I hope I can add something (useful or not ) to the discussion of decreasing spam messages and I'm curious about the reactions.

Keep posting! You seem to have a good understanding of filtering and stuff. The ideas you posted triggered me to add some "personal filters"! Thanks for the tips.

Regards,
Marc

[carverrn]

I do have a filter that says if the header contains "RCVD_IN" to move to a blacklisted folder. From time to time it does catch email that DSPAM missed and SpamAssassin scored low enough to get through.

Regards,
Rich

[marc_otten]

Phew. . .you sure make heavy use of filters.

I'm only taking my "first steps" in filtering, because I see an increase of spam in my Runbox-account. Just to be sure, I'm using a "monitored spam" strategy: I filter all possible spam to a specific spam-folder and once in a while I manually check all mail in it before deleting the entire contents. This prevents me from accidentaly deleting legitimate messages.

The information 50ftqueenie provided me with (this page) was of great use, because it makes it possible for me to identify some spam I receive with better accuracy...as did your tip about filtering on "RCVD_IN" did

Thanks,
Marc

[carverrn]

My incoming spam seems to fluctuate. It had been down but as of a couple of days ago it has picked up again and is between 100-200 a day.

Regards,
Rich

[marc_otten]

Hehehe...

My spam-load increased from nothing to 1-10 messages a day...so - when comparing to your ammount of spam - I don't have that much to complain Almost all of my spam is caught by the Spam-functionallities within RMM. I got it filtered into a spam-folder.

However, sometimes a message slips through the maze and arrives in my Inbox Therefor, I'm working on customized filters.

-Marc

[jbs]

You know what would be terrific? Some sort of an empty button on the spam folder! :-)

Originally posted by Geir on 15 January 2005
An "empty" link next to the Spam folder shouldn't be a problem at all. I'm sure we can fix that next week.

- Geir

[marc_otten]

I'd like to have that as well...when it's obvious (from reading the email-subjects) that all messages in my spam-folder in fact are spam-messages, it would be very convenient to have an "empty folder " button.

-Marc

[carverrn]

There should be an option for "empty". However, I wouldn't use it personally. It's a bit risky for me. When I review my Spam Folder I don't like to scan more than 50 messages at a time. Scanning more than that at once makes my eyes blurry and I'm not sure if I've really caught any non-spam (although it's rare that there is some non-spam there). I then select all the message on the page and use my "Delete Permanently" javascript (I actually use the one built into my Runbox Toolbar 2) to delete them for good.

Regards,
Rich

[50ftQueenie]

Originally posted by marc_otten
Hi 50ftQueenie, This actually sounds like a good idea. I've set up some 'personal' filters for my own convenience and they're actually very usefull in my "war on spam".

I've set up some filters now, but I can't really say yet if they're useful or not, because I added them today.

Keep posting! You seem to have a good understanding of filtering and stuff. The ideas you posted triggered me to add some "personal filters"! Thanks for the tips.

*grin* Well, not always (see the first post in this topic , but receiving spam makes you be more creative on how to decrease it. I myself have a very short username. I'm with Runbox for a couple of years now and in the beginning I did not receive any spam at all. At that time I didn't realise that the amount of spam was increasing that fast and that it would also affect me. Although I can decide to take another username, to me it's not worth all the hassle informing people about a new emailaddress.

Originally posted by carverrn
I do have a filter that says if the header contains "RCVD_IN" to move to a blacklisted folder.

The only exceptions I made are:
RCVD_IN_BSP_TRUSTED (Sender is in Bonded Sender Program (trusted relay)) and
RCVD_IN_BSP_OTHER (Sender is in Bonded Sender Program (other relay)).
The first test appeared in the headers of messages sent by eBay. Therefore I suspect it might also appear in other legitimate services.

Kind regards,
50ftQ.