Does FastMail have a better way then IMAP Idle to know when you get new emails?

jd2066 · 28 Jul 2019, 06:56 AM

Hey,
I use the FastMail service in an unusual way.
I have IMAP/STMP services on my personal home server that I can access remotely and such that I prefer over using remotely hosted services for various reasons.
However my home server is not reliable enough to have messages directly delivered to it (partly because I may do something that temporarily makes my server unavailable or my cable internet connection goes down and whatnot).

I use getmail to fetch any new messages from my FastMail account with the following configuration file:

Code:

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.fastmail.com
username = mymailbox@example.com
password = mypassword
mailboxes = ("INBOX",)
move_on_delete = "Retrieved using getmail"

[options]
received = false
delivered_to = false
read_all = false
delete = true
verbose = 1
message_log = /var/log/getmailrc-justin/mymailboxe@mydomain.log
message_log_verbose = false

[destination]
type = MDA_external
path = /usr/libexec/dovecot/dovecot-lda
arguments = ("-e", "-f", "%(sender)", "-d", "justin")

I run getmail using the command

Code:

getmail -g /home/justin/.config/getmail -r rc-mymailbox@example.com -i Inbox

.

It creates an IMAP Idle connection to FastMail so getmail knows instantly if there is new mail available and retrieves the email and then puts the email on FastMail in my "Retrieved using getmail" folder which I have set on FastMail to auto-delete the emails in every 30 days or so.

If getmail cannot connect to FastMail for whatever reason, I can login to FastMail.com and every email in the Inbox folder will be one that wasn't retrieved so I can see emails that I did not get due to some error or something and when getmail is running again, those messages will be retrieved.

I prefer IMAP idle over POP3 due to the instant knowing of a new email being available, with POP3 I would need to check every 5 minutes or so.

The MDA_external commend hands the email over to the Dovecot LDA service to run sieve filters on the email and then store the email in the file system so they can be be accessed via the Dovecot IMAP service.

However over the past few months I frequently get this error in the getmail log file "getmailOperationError error (IMAP error (command: UID => socket error: EOF))" which apparently means the connection to FastMail's IMAP service using IMAP Idle has been disconnected.
I'm not sure why but getmail's behavior when the IMAP Idle connection is disconnected is to just to close so until I manually restart getmail again I don't get email retreival.

I'm not sure if FastMail changed something recently, something happened with my internet provider or whatnot but I keep needing to restart getmail at least once a day and usually more now.

In the past, only my internet connection going away could cause the IMAP Idle connection to stop working so I rarely had to restart getmail again.

If there was some FastMail API I could use that was a more reliable way to know about new emails then I could just use that and tell it to run getmail whenever that API says new email is available.

Before FastMail I was using Google's email for domains service but their IMAP server worked in non-standard ways along with other things that I did not like.

BritTim · 28 Jul 2019, 08:17 AM

As far as I know, there is no publicly announced API from FastMail that does what you want. I believe FastMail now uses JMAP on the backend, and that ought to provide a better interface for notification of changes than IMAP IDLE. FastMail also has push email when dealing with Apple Mail on both iOS and MacOS. No documentation I have seen on either.

n5bb · 28 Jul 2019, 10:21 AM

Here is how Fastmail does this with JMAP:

Bill

EricG · 28 Jul 2019, 01:03 PM

Does the server have a static IP? If not, NAT will forget about idle connections.

All email clients deal with connection reset, so getmail has to also. The IMAP IDLE spec says the server can hang up after 29 minutes.

TenFour · 28 Jul 2019, 07:46 PM

Why not just forward the email from Fastmail or wherever to your local server-based email? I have found forwarding to be faster than IMAP in most cases with other services.

SideshowBob · 29 Jul 2019, 12:15 AM

Quote:

Originally Posted by jd2066

However over the past few months I frequently get this error in the getmail log file "getmailOperationError error (IMAP error (command: UID => socket error: EOF))" which apparently means the connection to FastMail's IMAP service using IMAP Idle has been disconnected.
I'm not sure why but getmail's behavior when the IMAP Idle connection is disconnected is to just to close so until I manually restart getmail again I don't get email retreival.

Just wrap getmail in a loop.

jd2066 · 29 Jul 2019, 10:42 AM

Quote:

Originally Posted by TenFour

Why not just forward the email from Fastmail or wherever to your local server-based email? I have found forwarding to be faster than IMAP in most cases with other services.

I tried that once but as my server is on a network with a dynamic IP address and sometimes it can be down for a couple days, forwarding is just not that reliable.
The email server doing the forwarding will usually give up after a certain amount of tries and then I don't get the email at all.
Using IMAP fetch is much better as it's my server asking the email server for email instead of the email server assumed that my server is always available.

Quote:

Originally Posted by EricG

Does the server have a static IP? If not, NAT will forget about idle connections.

The server has a static internal IP address, the external IP address is shared between all devices including the server via an ASUS router that uses NAT routing.

Quote:

Originally Posted by EricG

All email clients deal with connection reset, so getmail has to also. The IMAP IDLE spec says the server can hang up after 29 minutes.

True, I'll have to check but I think the main getmail developers didn't want to bother handling that because they don't care to have proper IMAP Idle support due to the limitations of IMAP Idle.
As getmail is an open source project, someone can and did add IMAP Idle support but I don't think the main developers of getmail really care for the feature at all so they don't intend on improving the feature.

SideshowBob · 29 Jul 2019, 11:40 PM

Quote:

Originally Posted by jd2066

I'll have to check but I think the main getmail developers didn't want to bother handling that because they don't care to have proper IMAP Idle support due to the limitations of IMAP Idle.

My understanding is that getmail breaks out of IDLE after 5 minutes if no new mail arrives.

I don't see that getmail is doing anything wrong here. In some cases action may be needed after a networking problem, for example restarting a tunnel. getmail's exiting so a calling script can handle the problem before restarting getmail is sensible IMO.

Your problem is trivial to fix by calling getmail from a loop.

28 Jul 2019, 06:56 AM	#1
jd2066 Junior Member Join Date: Jul 2016 Posts: 3	Does FastMail have a better way then IMAP Idle to know when you get new emails? Hey, I use the FastMail service in an unusual way. I have IMAP/STMP services on my personal home server that I can access remotely and such that I prefer over using remotely hosted services for various reasons. However my home server is not reliable enough to have messages directly delivered to it (partly because I may do something that temporarily makes my server unavailable or my cable internet connection goes down and whatnot). I use getmail to fetch any new messages from my FastMail account with the following configuration file: Code: [retriever] type = SimpleIMAPSSLRetriever server = imap.fastmail.com username = mymailbox@example.com password = mypassword mailboxes = ("INBOX",) move_on_delete = "Retrieved using getmail" [options] received = false delivered_to = false read_all = false delete = true verbose = 1 message_log = /var/log/getmailrc-justin/mymailboxe@mydomain.log message_log_verbose = false [destination] type = MDA_external path = /usr/libexec/dovecot/dovecot-lda arguments = ("-e", "-f", "%(sender)", "-d", "justin") I run getmail using the command Code: getmail -g /home/justin/.config/getmail -r rc-mymailbox@example.com -i Inbox . It creates an IMAP Idle connection to FastMail so getmail knows instantly if there is new mail available and retrieves the email and then puts the email on FastMail in my "Retrieved using getmail" folder which I have set on FastMail to auto-delete the emails in every 30 days or so. If getmail cannot connect to FastMail for whatever reason, I can login to FastMail.com and every email in the Inbox folder will be one that wasn't retrieved so I can see emails that I did not get due to some error or something and when getmail is running again, those messages will be retrieved. I prefer IMAP idle over POP3 due to the instant knowing of a new email being available, with POP3 I would need to check every 5 minutes or so. The MDA_external commend hands the email over to the Dovecot LDA service to run sieve filters on the email and then store the email in the file system so they can be be accessed via the Dovecot IMAP service. However over the past few months I frequently get this error in the getmail log file "getmailOperationError error (IMAP error (command: UID => socket error: EOF))" which apparently means the connection to FastMail's IMAP service using IMAP Idle has been disconnected. I'm not sure why but getmail's behavior when the IMAP Idle connection is disconnected is to just to close so until I manually restart getmail again I don't get email retreival. I'm not sure if FastMail changed something recently, something happened with my internet provider or whatnot but I keep needing to restart getmail at least once a day and usually more now. In the past, only my internet connection going away could cause the IMAP Idle connection to stop working so I rarely had to restart getmail again. If there was some FastMail API I could use that was a more reliable way to know about new emails then I could just use that and tell it to run getmail whenever that API says new email is available. Before FastMail I was using Google's email for domains service but their IMAP server worked in non-standard ways along with other things that I did not like.

28 Jul 2019, 10:21 AM	#3
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 8,923	JMAP new message push Here is how Fastmail does this with JMAP: JMAP Push Mechanism JMAP staying in sync example code (may not meet final JMAP standard) Bill

28 Jul 2019, 08:17 AM	#2
BritTim The "e" in e-mail Join Date: May 2003 Location: mostly in Thailand Posts: 3,090	As far as I know, there is no publicly announced API from FastMail that does what you want. I believe FastMail now uses JMAP on the backend, and that ought to provide a better interface for notification of changes than IMAP IDLE. FastMail also has push email when dealing with Apple Mail on both iOS and MacOS. No documentation I have seen on either.

28 Jul 2019, 01:03 PM	#4
EricG Essential Contributor Join Date: Aug 2009 Location: Canada Posts: 296	Does the server have a static IP? If not, NAT will forget about idle connections. All email clients deal with connection reset, so getmail has to also. The IMAP IDLE spec says the server can hang up after 29 minutes.

28 Jul 2019, 07:46 PM	#5
TenFour Master of the @ Join Date: Feb 2017 Location: USA Posts: 1,719	Why not just forward the email from Fastmail or wherever to your local server-based email? I have found forwarding to be faster than IMAP in most cases with other services.