Greylisting of incoming email - Page 4

robmueller · 1 Aug 2006, 07:47 PM

Re: Verizon

This problem was not greylisting related, but was related to the "too many unknown recipients" policy.

It seems verizon use an address verifaction system (e.g. something like http://www.postfix.org/ADDRESS_VERIFICATION_README.html). The problem with this is that when a spammer tries and sends lots of emails to verizon with forged MAIL FROM addresses, it generates lots of false RCPT TO attempts to us, which end up causing the verizon servers to be blocked because they're trying to deliver email to lots of unknown fastmail users.

I see address verification as useful, so I've now got a workaround that should stop other hosts that implement this sensibly getting blocked either.

hadaso: That does seem a bit annoying, though I would have thought that after a couple of emails that IP would end up whitelisted and not subject to greylisting. I'll look into the helo IP = client IP thing to see if that could help.

jjd: Yes some sending servers may be delayed for a while if they only retry every 24 hours, but remember:
1. It's only machines on dialup/adsl connections that are greylisted
2. If a machine passes greylisting twice in 24 hours, it gets whitelisted for a while (the more times it passes, the longer it gets whitelisted)

rikki Tikki: I'd be surprised if it's greylisting. As mentioned, it's only dialup/adsl hosts that are greylisted. I doubt that the e-booking system you use is using a dialup server to send it's email. If you have some existing emails from them, you can check the Received: headers. Look for something like:

Code:

Received: from xxx (yyy [a.b.c.d])
by mx1.messagingengine.com (Postfix) with ESMTP id zzz
for <me@mydomain.tld>; Thu, 27 Jul 2006 12:51:51 -0400 (EDT)

And paste that received header here.

Rob

paleolith · 1 Aug 2006, 09:34 PM

Quote:

Originally posted by robmueller
This problem was not greylisting related, but was related to the "too many unknown recipients" policy.

Rob,

Do you have a name for the latter? I think there's some confusion here because this thread is titled "greylisting", plus you haven't given a specific name for the "too many unknown recipients" policy. As a result we are tending to think of them as one thing and confusing the issues. It doesn't help that the methods and results have a lot of similarities.

A short sweet name -- "greylisting and TMUR" or something better -- would help dispel this confusion.

Thanks for the explanations.

Edward

Mal · 2 Aug 2006, 12:43 AM

newhampshire,

You can set your reaction to Backscatter messages from the web interface here:

Options, Spam/Virus Protection, {scroll down} Backscatter action.

jjd · 2 Aug 2006, 02:24 AM

Ok, I think I missed one KEY point in the rules... that being the greylisting only applies when there ARE UNKNOWN recipents. Messages to a known recipent should pass thorugh. If that's the case it could be fine. (although, personalized spam will still get through)

jjd · 2 Aug 2006, 02:27 AM

Rob, I'd still like to suggest implimenting a greylisted folder for each user. Along with bouncing the message back, give the user a copy. Just do this for a little while until the users get confident in the system. For nothing else than it will help users confidence in this system and it would help us help you when we discover items that slip through the cracks.

(This could be a support nightmare with items arriving only in the greylisted folder before the sender has a chance to retry, but maybe you can pull some server side hold on those messages until they pass fail the resend test. )

paleolith · 2 Aug 2006, 02:47 AM

jjd,

When FM rejects a message from a greylisted (dial-up) server, it doesn't receive the message and send it back. It only reaches an early step in the SMTP protocol, where the sender gives the recipient ID. At that point FM sends back the 453 message.

So you could not have a "greylisted folder", because there are no messages. You'd have to create an entirely new notification mechanism.

For TMUR (Too Many Unknown Recipient) hosts, the process is similar.

And frankly, I'd have a hundred notifications per day based on what I've seen so far. If one message got stopped that I wanted immediately, I wouldn't be able to find the needle in the haystack anyway.

Edward

elvey · 2 Aug 2006, 06:09 AM

Thanks, Rob.

Quote:

Originally posted by paleolith
jjd,

And frankly, I'd have a hundred notifications per day based on what I've seen so far. If one message got stopped that I wanted immediately, I wouldn't be able to find the needle in the haystack anyway.

Edward

Me too. Good point.
I occasionally tweak filters and extrapolate to determine what's coming in. I get sent thousands of spams per day, which advanced filtering keep at bay.

Oh, and the management at verizon deserve to be called things unprintable.
http://www.spamhaus.org/sbl/listings...sp=verizon.net

bkuhn01 · 3 Aug 2006, 12:45 AM

I've actually noticed a significant increase in spam since this change. It might be a coincidence but...

Thoughts?

paleolith · 3 Aug 2006, 01:22 AM

Quote:

Originally posted by bkuhn01
I've actually noticed a significant increase in spam since this change. It might be a coincidence but...

Thoughts?

Coincidence.

There's no way this measure could increase spam. Whether it decreases spam depends on the type and amount of spam you get. If it didn't help you much, then it wouldn't have a chance to balance out anything else that increased your spam.

Did you do anything new recently? This is your first post here -- did you recently post somewhere else that reveals your email address? Posting to Usenet with your real email address will bump your spam immediately, though it seems to fall back to baseline in a few days.

Anything else that might have revealed your address to the slimers?

It isn't necessarily anything you did though. Your addy may have gotten on a new CD that a couple of big spammers recently added to their lists, or something like that.

Edward

n5bb · 3 Aug 2006, 03:16 AM

Spam arrives sporatically and the arrival time has a high statistical deviation. Since SpamAssassin and other spam-limiting techniques are being periodically updated, and service providers are closing spam-generating accounts on a daily basis, and PC security software is blocking or eliminating rouge spam distribution daily, and spammers change their techniques periodically, it's hard to tell if something you change makes a real difference on your spam unless you look over several days.

When I turn off my spam protection blocking in my Sieve script, the amount of incoming spam does seem to be lower than a few months ago (before Fastmail changed several basic spam blocking features). But I have no way of knowing if the incoming raw spam from the Internet has changed -- it might have tripled in that interval, for all that I know.

Fastmail staff must have some idea of how much spam is reduced using the various standard techniques. My guess is that 80% of the raw spam is blocked up-front, and that SpamAssassin and backscatter blocking eliminates essentially all of the remainder. I have eliminated some manual spam blocking I was using in my personal Sieve script recently due to the backscatter and greylisting improvements.

One improvement I think we will see is that since we aren't so dependent on SpamAssassin and other content-based techniques, we can relax our spam blocking score levels. This (and whitelisting) should essentially eliminate false positives.

Bill

lazarus · 3 Aug 2006, 11:42 PM

unfortunately my setup is being seriously clobbered by the greylisting. My setup is as follows:

I own my own domain which is setup to primary MX to my home server which is on a static IP (Verizon business FIOS) - I pay extra to run servers and have a static ip.

This then filters the addresses to a number of places - but most then get forwarded to fastmail. At the moment 80% of deliveries are hitting the greylisting filter and being delayed.

my ip is in the 71.248.x.x all email being sent from me are to my valid fastmail address (plus optional folder).

Can this be fixed?

JasonWard · 3 Aug 2006, 11:59 PM

Quote:

Originally posted by lazarus
unfortunately my setup is being seriously clobbered by the greylisting.

From what I've read about the Greylisting written by Rob I would have thought your server would not now be being greylisted at all.

Unless I have misunderstood something your server should by now be whitelisted.

Perhaps you should email Fastmail support about the issue.

Jason

paleolith · 4 Aug 2006, 03:09 AM

lazarus,

First, since you are running your own mail server, just set the sending retry time to ten seconds for the first retry. AFAIK, FM has no lower limit on the time to retry -- perhaps they will have to eventually, but not now. In fact ISTR that hadaso and maybe another user posted results of tests showing that they could retry immediately and succeed (though they were testing AED rather than greylisting).

Second, definitely write support. Since you have a static IP address, they can probably manually whitelist that address.

Finally, which of the 453 responses are you getting? (Rob posted then in the first post in the thread.) It matters: if you are in the greylisting due to appearing to be a dial-up, then you should have been whitelisted quickly. This implies that the problem is AED on the /24 (too many unknown recipients from 71.248.x.* addresses). Does this fit the text of the 453 responses?

Edward

hadaso · 4 Aug 2006, 06:31 AM

Quote:

Originally posted by paleolith
... hadaso and maybe another user posted results of tests showing that they could retry immediately and succeed (though they were testing AED rather than greylisting).
...

I was testing greylisting by connecting from home using "telnet in1.smtp.messagingengine.com 25". What I saw then was that it was possible to just issue the same "rcpt to" command twice to pass the greylisting. I tried it again now and it didn't work. The second time I get the message

Code:

453 <ofer@hadaso.net>: Recipient address rejected: Still temporary deferral,
try again soon

It seems that now one needs to start a new SMTP session to pass greylisting. I thought that having an A record of the domain in the helo parameter would bypass greylisting, so I tried the domain I got from reverse dns for the IP I have right now, and it didn't work (of course I might have put in an extra character or some typo, though actually I just copied the domain from DNSstuff reverseDNS results (and verified there's an A record for that domain that resolves to the IP. And I verified by sending mail and checking the received header that FastMail actually receives the session directly from me and not through some kind of port 25 fforwarding that some ISPs do, and checked in the headers that the reverseDNS that FastMail got is the same that I used).

paleolith · 4 Aug 2006, 07:07 AM

Quote:

Originally posted by hadaso
I was testing greylisting

Ack, so you were. Sorry. That'll teach me to post without checking. (Fat chance.)

Quote:

It seems that now one needs to start a new SMTP session to pass greylisting.

Looking back now, I see that Rod's original explanation said

The delay time for successful resending varies slightly randomly depending on certain factors about the IP.

so that might explain your varying results. Also, a single successful retry is supposed to whitelist you for 24 hours -- not sure from your comments whether that is happening.

Edward

1 Aug 2006, 07:47 PM	#46
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	Re: Verizon This problem was not greylisting related, but was related to the "too many unknown recipients" policy. It seems verizon use an address verifaction system (e.g. something like http://www.postfix.org/ADDRESS_VERIFICATION_README.html). The problem with this is that when a spammer tries and sends lots of emails to verizon with forged MAIL FROM addresses, it generates lots of false RCPT TO attempts to us, which end up causing the verizon servers to be blocked because they're trying to deliver email to lots of unknown fastmail users. I see address verification as useful, so I've now got a workaround that should stop other hosts that implement this sensibly getting blocked either. hadaso: That does seem a bit annoying, though I would have thought that after a couple of emails that IP would end up whitelisted and not subject to greylisting. I'll look into the helo IP = client IP thing to see if that could help. jjd: Yes some sending servers may be delayed for a while if they only retry every 24 hours, but remember: 1. It's only machines on dialup/adsl connections that are greylisted 2. If a machine passes greylisting twice in 24 hours, it gets whitelisted for a while (the more times it passes, the longer it gets whitelisted) rikki Tikki: I'd be surprised if it's greylisting. As mentioned, it's only dialup/adsl hosts that are greylisted. I doubt that the e-booking system you use is using a dialup server to send it's email. If you have some existing emails from them, you can check the Received: headers. Look for something like: Code: Received: from xxx (yyy [a.b.c.d]) by mx1.messagingengine.com (Postfix) with ESMTP id zzz for <me@mydomain.tld>; Thu, 27 Jul 2006 12:51:51 -0400 (EDT) And paste that received header here. Rob

3 Aug 2006, 12:45 AM	#53
bkuhn01 Junior Member Join Date: Aug 2006 Posts: 3	increase in spam I've actually noticed a significant increase in spam since this change. It might be a coincidence but... Thoughts?

3 Aug 2006, 11:42 PM	#56
lazarus Junior Member Join Date: Mar 2003 Posts: 18	it's biting me hard unfortunately my setup is being seriously clobbered by the greylisting. My setup is as follows: I own my own domain which is setup to primary MX to my home server which is on a static IP (Verizon business FIOS) - I pay extra to run servers and have a static ip. This then filters the addresses to a number of places - but most then get forwarded to fastmail. At the moment 80% of deliveries are hitting the greylisting filter and being delayed. my ip is in the 71.248.x.x all email being sent from me are to my valid fastmail address (plus optional folder). Can this be fixed?

2 Aug 2006, 12:43 AM	#48
Mal Senior Member Join Date: May 2002 Location: Wanstead, England Posts: 170	newhampshire, You can set your reaction to Backscatter messages from the web interface here: Options, Spam/Virus Protection, {scroll down} Backscatter action.

2 Aug 2006, 02:24 AM	#49
jjd Junior Member Join Date: Aug 2006 Posts: 3	Ok, I think I missed one KEY point in the rules... that being the greylisting only applies when there ARE UNKNOWN recipents. Messages to a known recipent should pass thorugh. If that's the case it could be fine. (although, personalized spam will still get through)

2 Aug 2006, 02:27 AM	#50
jjd Junior Member Join Date: Aug 2006 Posts: 3	Rob, I'd still like to suggest implimenting a greylisted folder for each user. Along with bouncing the message back, give the user a copy. Just do this for a little while until the users get confident in the system. For nothing else than it will help users confidence in this system and it would help us help you when we discover items that slip through the cracks. (This could be a support nightmare with items arriving only in the greylisted folder before the sender has a chance to retry, but maybe you can pull some server side hold on those messages until they pass fail the resend test. )

2 Aug 2006, 02:47 AM	#51
paleolith Cornerstone of the Community Join Date: Mar 2002 Location: Florida Posts: 545	jjd, When FM rejects a message from a greylisted (dial-up) server, it doesn't receive the message and send it back. It only reaches an early step in the SMTP protocol, where the sender gives the recipient ID. At that point FM sends back the 453 message. So you could not have a "greylisted folder", because there are no messages. You'd have to create an entirely new notification mechanism. For TMUR (Too Many Unknown Recipient) hosts, the process is similar. And frankly, I'd have a hundred notifications per day based on what I've seen so far. If one message got stopped that I wanted immediately, I wouldn't be able to find the needle in the haystack anyway. Edward

3 Aug 2006, 03:16 AM	#55
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 8,930	Spam arrives sporatically and the arrival time has a high statistical deviation. Since SpamAssassin and other spam-limiting techniques are being periodically updated, and service providers are closing spam-generating accounts on a daily basis, and PC security software is blocking or eliminating rouge spam distribution daily, and spammers change their techniques periodically, it's hard to tell if something you change makes a real difference on your spam unless you look over several days. When I turn off my spam protection blocking in my Sieve script, the amount of incoming spam does seem to be lower than a few months ago (before Fastmail changed several basic spam blocking features). But I have no way of knowing if the incoming raw spam from the Internet has changed -- it might have tripled in that interval, for all that I know. Fastmail staff must have some idea of how much spam is reduced using the various standard techniques. My guess is that 80% of the raw spam is blocked up-front, and that SpamAssassin and backscatter blocking eliminates essentially all of the remainder. I have eliminated some manual spam blocking I was using in my personal Sieve script recently due to the backscatter and greylisting improvements. One improvement I think we will see is that since we aren't so dependent on SpamAssassin and other content-based techniques, we can relax our spam blocking score levels. This (and whitelisting) should essentially eliminate false positives. Bill

4 Aug 2006, 03:09 AM	#58
paleolith Cornerstone of the Community Join Date: Mar 2002 Location: Florida Posts: 545	lazarus, First, since you are running your own mail server, just set the sending retry time to ten seconds for the first retry. AFAIK, FM has no lower limit on the time to retry -- perhaps they will have to eventually, but not now. In fact ISTR that hadaso and maybe another user posted results of tests showing that they could retry immediately and succeed (though they were testing AED rather than greylisting). Second, definitely write support. Since you have a static IP address, they can probably manually whitelist that address. Finally, which of the 453 responses are you getting? (Rob posted then in the first post in the thread.) It matters: if you are in the greylisting due to appearing to be a dial-up, then you should have been whitelisted quickly. This implies that the problem is AED on the /24 (too many unknown recipients from 71.248.x.* addresses). Does this fit the text of the 453 responses? Edward