How To Improve SPAM Filtering using Runbox Filters

[carverrn]

2005-07-15: This information hasn't been updated since Runbox implemented the trainable DSPAM filter. But much of the information is still relevant.

A blocklist doesn't do much good for most SPAM since SPAM doesn't usually
have a valid "From" address nor use the same "From" address everytime.


Here's what seems to work very well for me.

(1) For "Detect junk mail" select "Yes, reject if possible"
I no longer recommend selecting this option. See below for details.

This allows Runbox to reject a message at the time it's being received,
if it is flagged as SPAM and if everyone receiving the message has agreed
to reject SPAM. I originally didn't use this option but the Linpro guys (the
ones hosting the Runbox mail servers) convinced me it's a good choice.

2005-07-15: After using "reject if possible" for a while I decided it's not a feature worth using. It can cause more problems than it solves. First, if you receive messages from groups like Yahoo Groups or Google Groups it can bounce spam messages back to Yahoo or Google which will cause your address to be marked as "bouncing" and you'll stop getting mail from your groups until you reactivate your address. Second, since a lot of spam is bulk mailed to many users at the same domain there's a good chance that for a given spam mail at least one user will not have selected this option so for that message it's disabled for everyone. Finally, it's probably more likely to bounce a valid message sent just to you that was incorrectly flagged as spam and you'd never know it.

(2) Create a "Spam" folder and a filter defined as:

Order: -2
Messages where: "Header" "contains" "X-Spam-Flag: YES"
will be: "saved to folder" "Spam"

By selecting "reject if possible" for the "Detect junk mail" option,
we have also disabled the ability to move SPAM to a Spam folder. This
defines our own Spam Filter that will do the same thing the original
Spam Filter did.

If you want your Spam Filter to be more restrictive, like have a cutoff
of 4 instead of the default of 5, you can define your filter like this:

Order: -2
Messages where: "Header" "contains" "X-Spam-Level: ****"
will be: "saved to folder" "Spam"

For this filter, any message with a score greater than 4 (e.g. 4.1) will
be moved to the Spam folder.


(3) Create a "NotInWhiteList" folder and a filter defined as:

Order: 999
Messages where: "Header" "doesn't contain" "USER_IN_WHITELIST"
will be: "saved to folder" "NotInWhiteList"

This should be your very last filter. If a message makes it all the way to
this filter, it's next stop would be the Inbox. This filter will make sure
that the message is from someone in your Whitelist, if not, it moves it to
the "NotInWhileList" folder.

Initially you will have messages in your "NotInWhiteList" folder that you
either want to add to your Whitelist or create a filter to move it to a
folder (could even be the Inbox). You will also have SPAM that wasn't caught
by SpamAssassin. Eventually, as you update your Whitelist and filters, the
"NotInWhiteList" folder will contain mostly uncaught SPAM.


(4) Messages over 250K in size are not scanned by SpamAssassin. The reason
being that mail that large is unlikely to be SPAM. Unless these messages are
moved to a folder by a filter, they would end up in the "NotInWhiteList"
folder because they will never contain the "X-Spam-Status" header for the
"USER_IN_WHITELIST" check. To avoid this you can add the following filter:

Order: 998
Messages where: "Header" "doesn't contain" "X-Spam-Status"
will be: "saved to folder" "Inbox"

This moves ANY message that hasn't been scanned by SpamAssassin to the Inbox.

Or you could create a NotSpamChecked folder and move it there.


(5) Add addresses of people/places you expect "ham" (not spam) email from
to the Spam Filter whitelist.

When SpamAssassin checks a message it will check the whilelist for the
"From" address. If it finds the address, it will add "USER_IN_WHITELIST"
to the "X-Spam-Status" header in the message.


(6) Add filters for Mailing Lists and Groups. For mailing lists or groups
(e.g. Yahoo Groups) that use the senders real email in the "From" address
instead of the list/group email address, setup filters to move these messages
to the proper folder. If they use the list email address in the "From" address
you could White List it instead but you still might want to have a filter to
move them to a folder.


(7) Add other filters as needed/wanted.

Messages are checked by SpamAssassin as they arrive at the Runbox servers.
However, messages flagged as *SPAM* are not filtered until your filters are
processed. The Spam Filter runs at an order value of "-2".

If you want your filters to run BEFORE the Spam Filter, allowing you to handle
ALL messages, your filter order values must be less than -2 (-3 through -99).

If you want your filters to run AFTER the Spam Filter, allowing flagged SPAM
to be filtered, your filter order values must be greater than -2 (-1 through
999).

CAUTION: You can have multiple filters with the same order value. Within
the same order value, filters are processed in the order in which the entry
was first added to the database. The filters page will show the filters in
the order processed.


It may seem a bit complicated but it's working really well for me.


Regards,
Rich

[jbs]

Rich,

Thanks for an excellent set of instructions. I've dodged spam thus far but I think I will start building up my whitelist just in case.

Along those lines I'm wondering whether there is an easy way to add a lot of addresses to the whitelist. I'd love to be able totake my contact manager software, dump out a file of all the email addresses and add them to my whitelist. Doing so I'd have 99% of my wanted email already flagged for whitelist.

I tried using the RBTB2 and entering multiple addresses in the input field, separated by spaces, commas or semicolons but could only get it to work with one at a time. Is that the way it works?

Along similar lines, I know your toolbar has an excellent tool for mass-uploading to one's address book . . . does being in the address book make a contact automatically count as whitelisted?

If not, then can I respectfully suggest to the Runbox team (or the Toolbar "team" ;-) a mechanism for mass uploading of whitelist names?

--Jason

[carverrn]

Hi Jason,

Along those lines I'm wondering whether there is an easy way to add a lot of addresses to the whitelist.

There is a way, "kind of". You can use my Edit Whitelist bookmarklet. It will display all the Whitelist entries in an edit box so that you can add/paste/delete/cut/edit. Each address should be on a line by itself. If you can export your addresses that way then you can simply cut and paste the whole list. Otherwise you'll have to cut and paste one at a time.

I tried using the RBTB2 and entering multiple addresses in the input field, separated by spaces, commas or semicolons but could only get it to work with one at a time. Is that the way it works?

RBTB2 ... I like that

It currently only handles one address at a time. I could take a look at handling multiple addresses.

Along similar lines, I know your toolbar has an excellent tool for mass-uploading to one's address book . . . does being in the address book make a contact automatically count as whitelisted?

Unfortunately no. It would be nice if this was possible in the future.

Regards,
Rich

[jbs]

Originally posted by carverrn
There is a way, "kind of". You can use my Edit Whitelist bookmarklet. It will display all the Whitelist entries in an edit box so that you can add/paste/delete/cut/edit. Each address should be on a line by itself. If you can export your addresses that way then you can simply cut and paste the whole list. Otherwise you'll have to cut and paste one at a time.

Rich-

Well, I hate to correct you in public like this, but you were wrong when you said there is a way 'kind of'. In fact, your bookmarklet (surprise surprise) is PERFECT!! Not only did it allow me to wasily and quickly add 500 email addresses to my whitelist, but it also id'ed for me a dozen or so cases where I had improperly formatted email addresses in my Palm Pilot!

The process was relatively simple. I exported the email field of my Palm address book to a CSV file, opened in excel, then copied into notepad and back into excel (this split records where I had multiple email addresses separated by carriage returns into separate cells in Excel. Then I split data to columns using commas as delimiters, so that any fields where I had multiple emails separate by commas would go into separate cells. I pasted the cells that were now in columns B, C, etc into column A and sorted alphabetically. I then pasted them into your form (about 100 at a time) and voila!

Just tested it with an email from myself (fortunately I'm on my whitelist -- it's obviously not that exclusive) and saw the beloved USER_IN_WHITELIST flag. I've never used this before because of the difficulty in adding addresses one at a time, and a concern that if I only added them from emails I got, I would slways be getting emails in my "NotInWhitelist" folder from that person you'd not heard from in a long while.

Even though Spam's not been a problem for me yet at this new address, I'm going to modify my growing system of filters to mimic what you've described above, and start to build a good solid pre-emptive system for dealing with spam.

Thanks again, Rich! You rock!!

--Jason

[Liz]

Just FYI, I'm working on incorporating Rich's excellent "scheme" in the help pages, just have to find a way that doesn't send the insecure users into spasms of panic.

Also, the current placing of the Toolbar and bookmarklets links is way too hidden away, I think (in the Help pages, under email add-ons).

Liz

[petrs]

that would be great - but please, make it 'idiot-proof' so even I can undestand it

[Liz]

If I understand it, you will too.

Liz

[carverrn]

For the Spam Filtering, I know the process sounds a bit complicated although it's not too bad. I tried to add a little explanation of "why" it was being added but it might be easier for some if you just say "trust me, it will work" and just show the filters to add. Some better formatting via HTML and aa few screen shots might make it easier to follow too.

For the Whilelist editing, I said "kind of" because it would be a lot easier if it could import CSV files directly. I could probably do this with the import stuff form the Import Addressbook bookmarklet. I should probably write an Export Addressbook too then. I guess it's time to work on bookmarklets again.

Rich

[MikhailT]

i was wondering if anybody have time to make a visual repersentation of what carvern is talkin about. Not a lot of people would understand and i admit even tho im an IT major, i had trouble understand where is where and to input what.

[Liz]

Not a bad idea...if anyone would like to set up a dummy account with dummy contacts and make some screenshots with text a la the POP/IMAP setups guides (JPEG please), they will be rewarded. Do mail us first though, in case more than one person volunteers.

Liz

[Gnome]

Hi, problem with setting these filters around case sensitivity of the in the email address.

My work address gets sent as:
Joseph.Cairns@ibx.com

But on the white list I'm only able to put:
joseph.cairns@ibx.com

There's no way for me to enter anything with upper case, so unfortunately, the server isn't putting the "USER_IN_WHITELIST" tag on this header.

The process should be converting everything to lower case before it compares addresses but it doesn't seem to be doing this.

Thanks!

You might want to reformat your address so it doesn't get spammed (userATdomain instead of realuser@realdomain). Also, you don't need to put your username, enough to put your UserName for the example.

[Gnome]

How my email admins at work set up my address' format is outside of my control.

This is coming from my worlk server with the capitals but I'm unable to enter an address with capitals in the white list.

Edit Note: regardless email addresses are not case sensitive and User@xyz.com should match user@xyz.com.

according to the RFC's, the localpart is case sensitive.
I ment how you write your address in your post. (write supportATrunbox instead of support@runbox).

[Gnome]

Ahh gotcha have a million things going on here at once and my head is spinning.

My address at work is send only so no worries there, but I'll keep it in mind for future reference, thanks!

according to the RFC's, the localpart is case sensitive.

RFC?