Spam to random tagged addresses.

[elvey]

Well, it happened, I just got spam to
<randomaphanumericstring>@<myuserid>.fastmail.fm
Actually, I got spam that was from that address, sent through an open relay, to aol, which refused it, so the open relay bounced it to me.

I prepared for this by using
<companyname><checksum>@<myuserid>.fastmail.fm as my tagged addresses. Now I have to figure out if/how to make Sieve calculate the checksum and check it. The checksum is a count of the # of letters [aeirst] in <companyname>
e.g. dell0@... yahoo1@... ecost3@....

Any ideas?

[Jeremy Howard]

Gosh Elvey, that's a clever idea! I guess you want something like:

header "subject" :regex "^.{3}@.{3}.fastmail.fm
header "subject" :regex "^.{4}@.{4}.fastmail.fm
header "subject" :regex "^.{5}@.{5}.fastmail.fm
...

I haven't checked this - not sure if Sieve supports this particular regex code (and it's 5am, so I'm not checking now! )

[kurianja]

Elvey, sorry but I can't figure out why you are doing this checksum thing. Does that mean you always sign up for subscriptions using these checksum addresses? What advantage does that give over giving say ibm@yourfmid, microsoft@yourfmid addresses?

[kurianja]

Hm.. I can't see what Jeremy's sieve does either. Probably thats because I am not understanding Elveys question correctly.

[flashbang]

kurian:

I think the point is so that elvey knows which emails are valid (only the ones with the checksum)..That way, he can filter based on that property, and greatly reduce the spam to his account.

Now, <any random alphanumeric string>@username.fastmail.fm wont pass through, because it doesn tpass the checksum test..

This vastly reduces the amount of spam he can/will get..

At least, this is my understanding of it

As for Jeremy's code.. My experience with sieve, or regular expressions in general is nothing, so i dont understand it either

Will this work?

anyof (
address :regex :localpart "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){1}1$",
address :regex :localpart "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){2}2$"
)

(edit)
For bounces, the "Received" headers should be tested, not the "X-Delivered-To" header.

[DrStrabismus]

I suspect that tacking .6 on the end of the localpart and using

address :localpart :matches "X-Delivered-To" "*?.6"

would be just as effective. Spammers don't tayler their spam to individuals, and a determined crank's first guess could well be sending ten copies with different digits.

One of the nice features of Spamgourmet is that it's extra levels of security are not used to filter email, they are used to determine whether a new address can be autocreated. That means that an individual can crank-up the level of security as they need, without having to worry about whitelisting the addresses they created in the past. Even if FM doesn't want to adapt their Perl script, there is a lot to be learned from Spamgourmet.

[LrdVader]

Originally posted by DrStrabismus
One of the nice features of Spamgourmet is that it's extra levels of security are not used to filter email, they are used to determine whether a new address can be autocreated. That means that an individual can crank-up the level of security as they need, without having to worry about whitelisting the addresses they created in the past. Even if FM doesn't want to adapt their Perl script, there is a lot to be learned from Spamgourmet.

Yep, Spamgourmet rocks. I find myself using it instead of FM's subdomain addressing for questionable signups, since it gives me more control. If I get spammed, I don't have to worry about Sieve scripts, or how much bandwidth the spam is costing me before it gets discarded, or dictionary attacks against the subdomain, or any of that stuff. If I get spam, I just log in and disable that address.

I would love to see FM get this kind of control.

[elvey]

Originally posted by Jeremy Howard
Gosh Elvey, that's a clever idea! I guess you want something like:

header "subject" :regex "^.{3}@.{3}.fastmail.fm
header "subject" :regex "^.{4}@.{4}.fastmail.fm
header "subject" :regex "^.{5}@.{5}.fastmail.fm
...

I haven't checked this - not sure if Sieve supports this particular regex code (and it's 5am, so I'm not checking now! )

I'm so exhausted after a 14 hr day I can't grok this either.


I guess it's a check that counts the number of chars in the localpart?

[FMradio]

Originally posted by Daniel T
Will this work?

anyof (
address :regex :localpart "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){1}1$",
address :regex :localpart "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){2}2$"
)

(edit)
For bounces, the "Received" headers should be tested, not the
"X-Delivered-To" header.

I tested a slight variation of the code above and happy to report that it
does work - just need to also add a "0" count case - for example, to
validate "foo0" which contains none of the [aeirst].

I also tested another trick which might yield more entropy from
simpler regex. Basic example takes the first letter of the address to
derive checksum just as the next letter of the alphabet - ie,
"antelope" + checksum = "antelopeb". (And, "zebra" would be
"zebraa"). This can be done with one (long) line of regex code, yielding
a 1/26 chance of random letters passing the check.

Adding just one more regex statement to map an additional letter of
the address yields a 2-letter checksum - with a 1/676 chance of a
random string matching. For example, "antelope" + checksum
mapping the first 2 letters = "antelopebo".

One might imagine any number of variations on this idea, all easy
enough for a human to generate and a regex to test.

The Sieve code I actually used for testing is only slightly more complex,
to accomodate a switch between the checksum methods. Method "1"
maps the first letter, method "2" maps the first 2 letters, and method
"3" counts [aeirst].

Examples of addresses that pass through this Sieve:

"foo.1g" (method 1)
"foo.2gp" (method 2)
"foo.3.0", "eleven.3.3" (method 3)

Note that for method 3, number of characters before ".3" is limited to
10 for the sake of this example (using 10 regex statements). It would
be nice to have an elegant way to handle longer addresses without
adding additional lines of regex code. I guess it could also all go into one
long line, but not sure that would be much of an improvement.
Anyway - here's the Sieve:

require ["envelope", "fileinto", "regex"];

if header :contains "X-Delivered-to" "@filtered.example.com"
{
  # "generic" checksum address format is one or more letters + '.'
  # + one or more digits + one or more non-numeric character, 
  # optionally followed by any character(s) ("^[a-z]+\\.1[^0-9]+.*@").  
  # The number after the '.' must correspond to a specified checksum 
  # method.  Address format requirements may be more strict for 
  # specific checksum methods, hence initial tests may differ from
  # general format.

  if address :regex "X-Delivered-to" "^[a-z]+\\.1[a-z]@" {
  # method 1 requires one or more letters + ".1" + only one letter
    if not address :regex "X-Delivered-to" "^a.*b@|^b.*c@|^c.*d@|^d.*e@|^e.*f@|^f.*g@|^g.*h@|^h.*i@|^i.*j@|^j.*k@|^k.*l@|^l.*m@|^m.*n@|^n.*o@|^o.*p@|^p.*q@|^q.*r@|^r.*s@|^s.*t@|^t.*u@|^u.*v@|^v.*w@|^w.*x@|^x.*y@|^y.*z@|^z.*a@" 
    {
      fileinto "INBOX.Junk Mail";
      stop;
    }
  } elsif address :regex "X-Delivered-to" "^[a-z]{2,}\\.2[a-z]{2}@" {
    # method 2 requires 2 or more letters + ".2" + 2 letters
    if anyof
    ( 
      not address :regex "X-Delivered-to" "^a.*b.@|^b.*c.@|^c.*d.@|^d.*e.@|^e.*f.@|^f.*g.@|^g.*h.@|^h.*i.@|^i.*j.@|^j.*k.@|^k.*l.@|^l.*m.@|^m.*n.@|^n.*o.@|^o.*p.@|^p.*q.@|^q.*r.@|^r.*s.@|^s.*t.@|^t.*u.@|^u.*v.@|^v.*w.@|^w.*x.@|^x.*y.@|^y.*z.@|^z.*a.@",
      not address :regex "X-Delivered-to" "^.a.*b@|^.b.*c@|^.c.*d@|^.d.*e@|^.e.*f@|^.f.*g@|^.g.*h@|^.h.*i@|^.i.*j@|^.j.*k@|^.k.*l@|^.l.*m@|^.m.*n@|^.n.*o@|^.o.*p@|^.p.*q@|^.q.*r@|^.r.*s@|^.s.*t@|^.t.*u@|^.u.*v@|^.v.*w@|^.w.*x@|^.x.*y@|^.y.*z@|^.z.*a@"
    ) {
      fileinto "INBOX.Junk Mail";
      stop;
    }
  } elsif address :regex "X-Delivered-to" "^[a-z]{1,10}\\.3\\.[0-9]{1,2}@" {
    # method 3 requires 1-10 letters + ".3." + 1-2 digits
    if not anyof
    ( 
      address :regex "X-Delivered-To" "^[^aeirst]*\\.3\\.0@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){1}\\.3\\.1@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){2}\\.3\\.2@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){3}\\.3\\.3@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){4}\\.3\\.4@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){5}\\.3\\.5@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){6}\\.3\\.6@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){7}\\.3\\.7@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){8}\\.3\\.8@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){9}\\.3\\.9@",
      address :regex "X-Delivered-To" "^[^aeirst]*([aeirst][^aeirst]*){10}\\.3\\.10@"
    ) {
      fileinto "INBOX.Junk Mail";
      stop;
    }
  } else {
    fileinto "INBOX.Junk Mail";
    stop;
  }
  fileinto "INBOX.Valid Checksum";
}

attempting to edit for typography ... ouch.

Originally posted by FMradio
I tested a slight variation of the code above and happy to report that it
does work - just need to also add a "0" count case - for example, to
validate "foo0" which contains none of the [aeirst].

Thanks for the (very useful) reply!

Originally posted by FMradio
Note that for method 3, number of characters before ".3" is limited to 10 for the sake of this example (using 10 regex statements). It would be nice to have an elegant way to handle longer addresses without adding additional lines of regex code. [...]

Count mod 10

The change is to add

(([aeirst][^aeirst]*){10})*

after the negated class, in the cases for 3.1@ through 3.10@.

edit: your comment ('method 3 requires 1-10 letters + ".3." + 1-2 digits') is not accurate; only the characters in the set ([aeirst]) are limited, to 10 repeats or less. For example, 'emaildiscussions.3.9@' is accepted.

[FMradio]

Originally posted by Daniel S
your comment ('method 3 requires 1-10 letters + ".3." + 1-2 digits') is not accurate; only the characters in the set ([aeirst]) are limited, to 10 repeats or less. For example, 'emaildiscussions.3.9@' is accepted.

Right, thanks for clarifying this. To avoid confusion (ie, when dealing with the rare case where an address consists entirely of countable letters - say, "raresttest.3.10@") I had set the 10-character limit as an arbitrary requirement in the initial test for method 3:

 "^[a-z]{1,10}\\.3\\.[0-9]{1,2}@"

(This is where 'emaildiscussions.3.9@' would be blocked)

But - using your idea to count [aeirst] modulo 10, there would be no need to artificially limit a valid address to 10 characters, and only a single-digit checksum would be needed. In that case, the initial check for valid method 3 address might look something like:

 "^[a-z]+\\.3\\.[0-9]@"

Edit: I've added the example Sieve code to a page in the FastMail wiki - hopefully more good ideas and code for checksums in Sieve can be developed collaboratively there, in parallel with the discussion here.

[FMradio]

Okay, well - some significant improvements have already been made
to the previous example via the wiki page, thanks to Daniel S.
(Does the 'S.' stand for Sieve? )

To allow slightly simpler regex statements, the different checksum
methods are now presented as separate Sieve scripts. The "method
number" is no longer part of the localpart of the address (the examples
are now coded to catch mail at different subdomains, ie
"foogp@method2.example.com", but that's not really necessary
either.) Note - omitting the extra code to handle method version
number for the sake of simplified example Sieves - however, probably
a good idea to iadd a version number switch to allow forward and
backward compatibility when implementing for actual use.

Method 3 has been modified to count [aeirst] modulo 10 so that the
length of a valid address is no longer artificially restricted. Tested the
new code, and it works - I think it's a very neat solution.

There must be at least a few more interesting ways to implement
checksums via Sieve. This is fun stuff, and a potentially powerful
method to reduce the amount of unsolicited email getting through
to "catchall" inboxes.

# "method 1" - address must begin and end with different letters.
# Non-alphabetic characters in the middle are allowed. Minimum length: 2.
# checksum: last letter maps "first + 1" (alphabetical order)
# (b is checksum for a, c is checksum for b, ..., a is checksum for z)
# Example: pass: "foog@", "zebr.123.a@", "summit@"; fail: "foo@", "a@".

require ["fileinto", "regex"];

if address :contains "X-Delivered-to" "@method1.example.com" {
  if not address :regex "X-Delivered-to" "^[a-z].*[a-z]@"
  {
    fileinto "INBOX.Junk Mail";
    stop;
  }
  # only validate the checksum if generic method1 address requirement tested out above
  if not address :regex "X-Delivered-to" "^(a.*b|b.*c|c.*d|d.*e|e.*f|f.*g|g.*h|h.*i|i.*j|j.*k|k.*l|l.*m|m.*n|n.*o|o.*p|p.*q|q.*r|r.*s|s.*t|t.*u|u.*v|v.*w|w.*x|x.*y|y.*z|z.*a)@"
  {
    fileinto "INBOX.Junk Mail";
    stop;
  }
  # valid @method1 address passes through to reach this point
  # can now process further to fileinfo, forward, notify, vacation, etc.
}

# "method 2" - address must begin with a pair of letters, and end with a different pair.
# Non-alphabetic characters in the middle are allowed. Minimum length: 4.
# checksum: the second to last letter maps "first + 1", and the last letter
# maps "second + 1" (alphabetical order as in method 1)
# Example: pass "foogp@", fail "foog@"

require ["fileinto", "regex"];

if address :contains "X-Delivered-to" "@method2.example.com"
{
  if not address :regex "X-Delivered-to" "^[a-z]{2}.*[a-z]{2}@"
  {
    fileinto "INBOX.Junk Mail";
    stop;
  }
  # only validate the checksum if generic method2 address requirement tested out above
  if not allof
  (
    address :regex "X-Delivered-to" "^(a.*b.|b.*c.|c.*d.|d.*e.|e.*f.|f.*g.|g.*h.|h.*i.|i.*j.|j.*k.|k.*l.|l.*m.|m.*n.|n.*o.|o.*p.|p.*q.|q.*r.|r.*s.|s.*t.|t.*u.|u.*v.|v.*w.|w.*x.|x.*y.|y.*z.|z.*a.)@",
    address :regex "X-Delivered-to" "^(.a.*b|.b.*c|.c.*d|.d.*e|.e.*f|.f.*g|.g.*h|.h.*i|.i.*j|.j.*k|.k.*l|.l.*m|.m.*n|.n.*o|.o.*p|.p.*q|.q.*r|.r.*s|.s.*t|.t.*u|.u.*v|.v.*w|.w.*x|.x.*y|.y.*z|.z.*a)@"
  ) {
    fileinto "INBOX.Junk Mail";
    stop;
  }
  # valid @method2 address passes through to reach this point
  # can now process further to fileinfo, forward, notify, vacation, etc.
}

# method 3 requires address ending with a number from 0 to 10
# minimum length: 1 (only "0@"; there are six 2-character options).
# Assuming N is the number of the localpart letters which are in [aeirst],
# the number to append is: 0 if N = 0, 10 if N is a multiple of 10,
# or N mod 10 (i.e., rightmost digit of N) otherwise.
# Examples: pass "null0@", pass "emaildiscussions9@", pass "raresttest10@",
# pass "startagainafterten4@", pass "0@", pass "r1@";
# fail "foo2@", fail "counted0@", fail "10@"

require ["fileinto", "regex"];

if address :contains "X-Delivered-to" "@method3.example.com"
{
  # First, simply check that the localpart ends with a number.
  # (Want 0 to 10 for checksum, larger numbers accepted to handle case where
  #  original address might end with a number too - ie "lucky7".
  if not address :regex "X-Delivered-to" "[0-9]@"
  {
    # Messages that don't end in a number.
    fileinto "INBOX.Junk Mail";
    stop;
  }
  # Only validate the checksum if generic method3 address requirement tested out above:
  # Check the that number equals the (Num [aeirst] in localpart) mod 10.
  if not anyof
  (
    address :regex "X-Delivered-To" "^([^aeirst]*[^aeirst1])?0@", # matches "0@" and "xyz0@" but not "xyz10@"
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){1}1@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){2}2@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){3}3@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){4}4@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){5}5@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){6}6@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){7}7@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){8}8@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})*([aeirst][^aeirst]*){9}9@",
    address :regex "X-Delivered-To" "^[^aeirst]*(([aeirst][^aeirst]*){10})+10@"
  ) {
    # Messages that end in a number that doesn't equal to the number of [aeirst]'s.
    fileinto "INBOX.Junk Mail";
    stop;
  } else { # Optional: action to take on legitimate messages. Can be left empty.
  }
  # valid @method3 address passes through to reach this point
  # can now process further to fileinfo, forward, notify, vacation, etc.
}
# Legitimate messages skip the rest of this elsif-chain; if more spam
# filtering follows, it should with an 'if' rather than an 'elsif'.

Edit: Fixed vbulletin formatting glitch (by replacing 'a' with '&amp;#97;' as suggested by Daniel S. - thanks for the tip!).
Edit: Updated example Sieves to match current contents of http://wiki.fastmail.fm/index.php/SieveChecksums
Edit: Note recommending use of version number switch.

Originally posted by FMradio
the wiki page

Edited it again...

Originally posted by FMradio
Daniel S.
(Does the 'S.' stand for Sieve? )

No comment
And if I were a C# programmer, would I then be "Daniel #"?

Originally posted by FMradio
EDIT: Unfortunately, formatting seems to be getting mangled by something I don't understand in vbulletin - beware of spurious spaces and line breaks in regex statements below. (Compare with correctly formatted example code at http://wiki.fastmail.fm/index.php/SieveChecksums if any questions.)

Known bug (documented on the Forum Tips sticky in the About This Site forum). I've found that escaping one character in (on average) every line in the Compose Form with its HTML entity (I usually use &amp;#97; - the letter 'a' ('a' is ASCII 97)) fixes this problem. Don't know why. It also works in links, BTW.

But it would be easier to just keep the most recent version on the Wiki - no formatting bugs there (except in the diffs? added a comment on the Wiki page).

[elvey]

FmRadio - thanks for taking my idea and running with it - cool! (or should I say k3w1?) Some great enhancements to make the regex less CPU-intensive. BTW, I guess the bug in cyryus that made ^ and $ stop working has been fixed - since your sample uses "^"!