Not too happy with new Spam filter

[Jericko]

Ok, I have been using the new SPAM feature for about 3 weeks now. I went from getting about 40-50 spam emails per day to zero. But the problem is, all my emails go into my SPAM folder. So instead of looking in my inbox for my emails, I have to look in my SPAM folder for my emails, which is more of a pain. Maybe I am missing these features, but if there not there, they need to be!

1. Add all people in address book to whitelist button, as opposed to adding them one by one.

2. When a good email gets classified as SPAM, add a button to put that address in your whitelist! What good does that "not spam" button do if it only moves the email to your inbox.

I was really excited about this new SPAM filter, but heck I could have wrote a script that said put EVERYTHING in my junk folder unlesss I say otherwise.

[marc_otten]

Hmm..

funny. SPAM-filtering works very, VERY smooth for me. What are your settings???

[Jericko]

Setting?

The only thing I see is "Use trainable filter" which i have checked. I would not call it trainable, since I am the one that has to do ALL the work. I dont understand why when I click on a email and the "Not Spam" button, it cant remember that address is not spam. I am just really disapointed in this new "trainable filter", and am probably going to turn it off soon.

[Geir]

Jericko,

I think you should give it some more time. Your filter is far from "mature", and although your accumulated accuracy is currently 89.9% it will get a lot better.

We will most likely implement those extra buttons you mentioned.

- Geir

[Jericko]

Originally posted by Geir
Jericko,

I think you should give it some more time. Your filter is far from "mature", and although your accumulated accuracy is currently 89.9% it will get a lot better.

We will most likely implement those extra buttons you mentioned.

- Geir

Its been worse then that, I just stop marking my legit emails as "Not Spam" just to move them to my inbox. I at 1st thought the "Not Spam" button would "train" the spam filter, but I soon found out it does nothing.

[carverrn]

Originally posted by Jericko
Its been worse then that, I just stop marking my legit emails as "Not Spam" just to move them to my inbox. I at 1st thought the "Not Spam" button would "train" the spam filter, but I soon found out it does nothing.

That's not true that it does nothing. Your misunderstanding comes from your expectation that DSPAM is maintaining something like a whitelist and that when you click "Not Spam" it should be adding it to the whitelist. DSPAM does not have a whitelist. So clicking "Not Spam" does not instantly make messages from a certain person "not spam" anymore. It will take a number of messages from that person that aren't considered spam to make the presence of that name in the message statistically significant enough to make the message not be considered spam. If you quit using "Not Spam" and just move the messages then you're no longer "training" your spam filter. Runbox has configured SpamAssassin to help train DSPAM and SpamAssassin DOES use the Whitelist in the Filter section. If you add names to the Whitelist it will help SpamAssassin to train your DSPAM filter.

My description may not be 100% accurate but I thinks it's close enough to explain what's going on.

Regards
Rich

[tore]

Originally posted by carverrn
DSPAM does not have a whitelist. So clicking "Not Spam" does not instantly make messages from a certain person "not spam" anymore. It will take a number of messages from that person that aren't considered spam to make the presence of that name in the message statistically significant enough to make the message not be considered spam.

Actually, it's more explicit about it (and indeed it's called a whitelist). It DSPAM-specific, though, not used by SpamAssassin or anything else. It tokenizes the entire From-header, and if it has seen this token >=20 times in a non-spam and 0 times in a spam, future emails matching this token is automatically classified as nonspam (note that SpamAssassin might disagree). Once you correct one of these messages as a spam, the token is removed from the whitelist.

If you quit using "Not Spam" and just move the messages then you're no longer "training" your spam filter.

Using DSPAM, but failing to correct all misclassifications (both spam and nonspam), is a very bad idea. Not only won't it be effective in spotting the spams, but it will also dramatically increase the risk of misclassifying nonspams. I assume this effect is what Jericko is seeing. DSPAM always assumes it was able to correctly classify a message, so if you don't correct it when it's made an error, the chance of it doing another error on a similar email is much higher. Once it's made two uncorrected errors, the third will follow shortly. And so on. So if one does not correct errors, it must be turned off, or in the end every email will be classified the same (spam or nonspam).

Tore

[carverrn]

Actually, it's more explicit about it (and indeed it's called a whitelist). It DSPAM-specific, though, not used by SpamAssassin or anything else. It tokenizes the entire From-header, and if it has seen this token >=20 times in a non-spam and 0 times in a spam, future emails matching this token is automatically classified as nonspam (note that SpamAssassin might disagree). Once you correct one of these messages as a spam, the token is removed from the whitelist.

Thanks for clarifying that Tore. When I was reading about DSPAM and whitelisting I guess I misunderstood what I read. I got the impression that the tokenized From addresses were lumped in with all the other tokens and that it wasn't a separate whitelist.

Regards,
Rich

[csrgavh]

I have to be counted as one who is extremely pleased with the new DSPAM filter. I do agree that one nice feature to be added for usefulness would be an "Add to Whitelist" button.

Regards,
Craig

[carverrn]

My Runbox Toolbar 2 has an "Add To Whitelist"

Rich

[tore]

Originally posted by carverrn
When I was reading about DSPAM and whitelisting I guess I misunderstood what I read. I got the impression that the tokenized From addresses were lumped in with all the other tokens and that it wasn't a separate whitelist.

No, you understood perfectly. :-) It is inserted into the database the same way all other tokens are, the difference is that when classifying (deciding if an email is spam or not, based on its tokens) that token is given special regard, iff it has been seen >19 times in nonspams and <1 times in spams. If that's true, a match on the From-token alone will make the entire classification result in "nonspam".

You're right it's not a separate whitelist, it's not even a "list" at all - only a token that is looked at specially when classifying. The effect, however, is that of a conventional auto-whitelist.

Tore

[Geir]

In addition to Dspam's autowhitelist, SpamAssassin has its own whitelist that you can manage from the Manager:Filter section.

When Dspam is activated, a filter rule is created in your Exim filter file. It distinguishes "spam" from "ham" by looking at the SA and Dspam headers, and saves messages considered to be "spam" to your Junk folder -- unless the sender's address is found in your SA whitelist.

Although the inner workings of our SA/Dspam setup might sound complicated, it is (supposed to be) easy to use: Keep training your filter, SA-whitelist specific addresses if you need to, and don't create any spam-related filters preceding SA/Dspam -- unless you know what you're doing.

- Geir

[Gnome]

Has the new spam filter been released from beta yet? I don't see the options for it on my account and would love to get going with the new filter.

[carverrn]

As I understand it they are not officially rolling it out yet until they resolve some hardware issues (i.e. get some new servers installed). The servers running the spam filters are apparently the same ones running some of the other services like POP3. Since the spam filtering can get quite intensive when lots of emails are pouring in it can effect performance of the other services. At least this is my understanding of the situation.

However, they will gladly turn on DSPAM for anyone requesting it. Just PM Geir or Liz with your account name and they'll switch it on for your account.

Regards,
Rich

[Geir]

We have resolved most of the scalability issues with Dspam.

Regular spam scanning by SpamAssassin is currently distributed among the various MX servers. Dspam runs on a dedicated server (currently Laika, which used to be the main database server) that should be powerful enough, as the Dspam setup is much more efficient now.

Until recently we have had to store all Dspam-scanned messages temporarily in the Dspam database (in addition to the extracted tokens), because the data passed to Dspam via the "Report Spam/Not Spam" functions wasn't exactly identical to the data Dspam initially scanned upon receiving the message (both Dspam, SA, etc add headers during message delivery).

By utilizing more sophisticated configuration available in the new Dspam version combined with some changes in internal message delivery, we no longer need to store those messages in the Dspam DB (in addition to the user's Maildir) -- which greatly reduces database size and server load.

There are still some minor tasks to complete before Dspam is ready for official launch, but as there are no foreseen problems we expect it to be ready quite soon.

Until then, as Rich said, drop us a message if you'd like to test it.

- Geir