EmailDiscussions.com

EmailDiscussions.com (http://www.emaildiscussions.com/index.php)
-   Email Help Needed! (http://www.emaildiscussions.com/forumdisplay.php?f=7)
-   -   IMAP Connection Very Slow - Suggestions ? (http://www.emaildiscussions.com/showthread.php?t=74403)

InquiringMind 23 May 2019 06:05 AM

IMAP Connection Very Slow - Suggestions ?
 
One of my email accounts is on Inbox.com

I have a 75 Mb/s. ( ~ 9.375 MB/s ) internet connection and get approximately that on both download and upload.

When I try to download an email with 5 MB of attachment by IMAP on Outlook 2010 or the latest version of Thunderbird, it takes about 25 seconds. So my IMAP download speed is about 0.2 MB/s.

If I use the same programs to access my Fastmail.com account, download is way faster.

I have tried turning off Norton Antivirus and Disabled Antispam plugin for Outlook.

I have updated Outlook with the latest MS Updates.

Is there any way to speed up this download problem?
Inbox.com says they do not throttle.

JeremyNicoll 23 May 2019 07:06 PM

First, you wouldn't expect to see quite as much as 75 Mbps (or as you say 9+ MBps) as a traffic rate from a 75 Mbps connection. Included in that is all the 'protocol' overhead of managing the flow of traffic. You might more reasonably expect a sustained flow of about 8 MBps.

That only happens if a server is sending traffic at that speed or potentially even faster, AND every computer that your traffic passes through en-route to you barely delays it.

Inbox.com may not "throttle" ie deliberately slow down the rate at which their servers send data, but that doesn't mean that some problem in their system might not make data transmission unintentionally slow.

Depending on where in the world you are, and where their servers are, local problems in internet infastructure might affect you. Another possibility (if they have servers in several countries) is that you might temporarily be getting data from a server that's a long way from you rather than close to you.

It's really hard to tell. Worse, because you're seeing this as a problem fetching a mail attachment, other people can't experiment to see if they can get that same attachment at a reasonable speed.

InquiringMind 24 May 2019 04:51 AM

When I use Inbox.com's web interface, and download the email and attachments, everything is speedy. No long delays.

I would just use the webmail interface, but I want to back up my emails locally onto my PC and I also want to sort through them using the tools that a program like Outlook provides, which is more sophisticated than the web interface. This will allow me to delete unneeded emails with less time spent on the process.

This is very frustrating.

If I can access the same thing fast through the web interface, and if Outlook accesses Fastmail without the same delays, can I deduce that the problem lies with Inbox.com's IMAP server ?

JeremyNicoll 24 May 2019 06:19 AM

The trouble is, I don't know (and probably you don't know) how complicated Inbox.com's server architecture is. You don't know if the IMAP server is in the same place as the web server, and you don't know if when you're using the web server whether it talks to the same IMAP server as the one you'd be accessing if using an email client directly.

And, I can talk about "the" web server or "the "IMAP server, there may be lots of each of them, even if they all appear externally to have the same IP address.

Does this slow download happen with every attachment or just those of a certain size?

InquiringMind 24 May 2019 11:57 AM

Every attachment and the delay is proportional to size.

5 sec for 1 MB, 10 sec for 2 MB, 20 sec for 4 MB, 25 sec for 5 MB.

JeremyNicoll 24 May 2019 07:58 PM

If TB and/or Outlook have configurable levels of logging, turn it on, at a high level of detail and try just to grab the relevant attachment (ie keep other stuff out of the detailed logs if possible). Are the logs showing any kinds of error? - this may be useless because for all I know, if eg TB makes multiple requests for something it might not log that.

You said that Inbox say they don't throttle. But did you ask them that (which is perhaps too vague) or did you ask them specifically why your attachment downloads are slow via TB/Outlook and fast in their webmail system?

Ask them if the servers are in the same place.

Ask them if their webmail system is in some way optimised compared with their public-facing IMAP server. It may be that the webmail system runs on the speediest machines with the best internet access, and the IMAP server - which maybe they expect fewer people to use - is older & slower, or mis-configured, or swamped with users. Unless they can shed light on this I don't see what progress you'll make.

InquiringMind 25 May 2019 01:00 AM

I will try to get the type of information you wrote about and will post after.

InquiringMind 25 May 2019 08:37 AM

Here is a very detailed answer I received from Tech Support:

There is a reason why IMAP download is slower than web download as many IMAP clients (including Outlook and Thunderbird) typically open multiple connections to the server and perform multiple operations at the same time. So when you are looking at a single email Outlook could still be performing another download in the background, slowing download of the current email. And it's not only that it eats bandwidth but those multiple connections are "fighting" for exclusive access to mailbox, which each operation requires. So when one command is being processed, other waits (on server) for its completion before it can execute and return data to client.

Also, back to the difference between web and IMAP. Attachments are in general binary data (although also text file can be an attachment, typically it is image, PDF, Word document, etc. and those are binary data). Because SMTP, POP3 and IMAP are plain text protocols, binary data could not be directly included in email, they must be encoded. For binary files the most used format is Base64 encoding, which enlarges the encoded data by 1/3 of original un-encoded data (each 3 bytes are encoded into 4 bytes). To lower storage requirements on our servers, we perform automatic base64 decoding and we store attachments in binary form. When downloading attachment via web, we can send the binary data directly, so that's why it is quite fast. However when email (MIME format) is requested via POP3 or IMAP, server has to encode the attachment again to create valid message in MIME format. The encoding takes some time. In addition IMAP protocol allows that IMAP client requests only portion of the MIME message to allow recovery if download fails, i.e. chunked download (give me first X bytes of message, give me next X bytes, give me next X bytes, ....). If the client automatically performs chunked download for large enough emails, it means, that each time client receives request for one chunk, it has to construct full valid MIME message (encode all attachments and whole messages) in order to be able to seek to requested starting index and provide given number of bytes from that index. Now imagine the email has 40MB and that the the client uses chunk size of 100 KB (for simplicity just suppose than 1MB = 10*100KB). To download the whole message, client has to perform 400 requests. Each request means, that our IMAP server has to retrieve raw email data (40MB), perform encoding (resulting in about 52 MB MIME message) and locate requested 100 KB chunk and return it to client. In addition IMAP server typically runs on different server than where mailbox is located, so those 40MB are transferred through local area network (LAN) in our server room. I.e. the download of 40MB (in fact 52MB) message requires in this scenario transfer of 400*40MB between IMAP server and storage server over LAN, which is about 15 GB!

Outlook etc might use different algorithms to improve the speed of IMAP but we have no plans at this time to change our system.

InquiringMind 25 May 2019 08:43 AM

Unless someone has a better suggestion, it seems like I need to work with the headers only in IMAP.

Delete what I do not need to save.

Then download folders of those emails I want to save locally on long weekends where I can leave the connection otherwise undisturbed.

Any suggestions?

JeremyNicoll 25 May 2019 09:51 PM

I'm impressed that Inbox's technical support people provided a thorough answer.

It seems they've optimised their system for users of their webmail systems, which is fair enough.

The problem with IMAP seems to be that /if/ your client requests downloads in chunks, the Inbox backend system is very inefficient. I can't quite see why anyone would design a system that says: the user wants an attachment (in chunks) so over & over again we'll do a lot of work... I would have expected their system to do the work once, cache the re-encoded attachment file, hand it out in chunks, and destroy it afterwards.

I think you should ask Thunderbird support whether there's a way to make sure that an IMAP account in TB does not request data in chunks, or perhaps only does so if the attachment size is above some threshhold (maybe a lot bigger than most of your attachments). You have a fairly fast connection and presumably it's pretty reliable - not dialup on a flakey connection - so needless chunking (so that chunks can be re-requested shouldn't be needed). See: https://lists.mozilla.org/listinfo/support-thunderbird

If that's not possible in TB then I'd suggest you try a POP3 session, just for retrieval of attachments. POP3 is a simple protocol and it just asks for the entire contents of an email in one operation. I don't know enough about TB to know if you'd change the existing account from IMAP to POP3 and then back again (I suspect not though, as that might screw up IMAP flags). More likely you'd disable the IMAP account (ie leave it defined), define a new POP3 account for the same server, then use it to grab the whole of the mails concerned. You might have a problem pointing the POP3 server at the mails concerned as POP3 might just see those in the Inbox account's inbox, not other folders as well. You might also need to tell POP3 to leave the mails on the server (at least while experimenting).

JeremyNicoll 25 May 2019 09:58 PM

Another option: you said you have a Fastmail account. Define a new alias there and a filter so that any mail sent to it ends up in a separate folder.

Forward the mails whose attachments you want to save, to the fastmail address. Use fastmail's web interface to download the mails (and their attachments). I've done this when I wanted to download things from another provider's Roundcube system, as in Roundcube (or at least in an ancient version of RC) you can only download one email at a time.

The advantage of forwarding to fastmail is that the bulk of the internet traffic will be between Inbox and fastmail, not on your internet connection, and you don't care how long that takes.

InquiringMind 26 May 2019 02:47 AM

If the content were the only thing important in the emails I wish to save, then that would be an excellent suggestion.

Unfortunately, the time stamp is just as important, and when you forward an email, you get an additional time stamp, and it can be confusing if you need to present the email as evidence.

But thank you for the suggestion.

n5bb 26 May 2019 04:34 AM

Why not just use Fastmail instead of Inbox.com? You can change the generating system target address so that the messages are directly sent to a Fastmail alias for new messages. Then import the existing messages to your Fastmail account via a big IMAP transfer:
https://www.fastmail.com/help/receive/migratemail.html

Since you are importing the messages using IMAP (similar to an IMAP client), they should be in correct format rather than using forwarding headers.

Bill

InquiringMind 28 May 2019 10:19 AM

Dear Bill,

Quote:

Why not just use Fastmail instead of Inbox.com? You can change the generating system target address so that the messages are directly sent to a Fastmail alias for new messages
For reasons to complex to go into here, I cannot change email right now. In the future it may be possible. Fastmail does not seem to have the same slow rate of transfer, though I have way more messages with large attachments in the Inbox.com so it may not be a fair comparison.

Quote:

Then import the existing messages to your Fastmail account via a big IMAP transfer:
https://www.fastmail.com/help/receive/migratemail.html
Since you are importing the messages using IMAP (similar to an IMAP client), they should be in correct format rather than using forwarding headers.
This is fascinating and was unknown to me before. I will try an experiment with some unimportant emails that have large attachments to see if the header and time stamp is affected. Thank you for telling me about this feature that I would never have suspected existed.

emoore 13 Jul 2019 09:57 AM

Quote:

Originally Posted by JeremyNicoll (Post 610231)
I think you should ask Thunderbird support whether there's a way to make sure that an IMAP account in TB does not request data in chunks, or perhaps only does so if the attachment size is above some threshhold (maybe a lot bigger than most of your attachments).

http://kb.mozillazine.org/Entire_mes...n_IMAP_message mentions that mail.imap.fetch_by_chunks controls whether Thunderbird tries to fetch a message body (or any other MIME body part) in chunks. You can toggle that setting in tools -> options -> advanced -> general -> config editor (the Thunderbird equivalent of about:config) . However, its normally not defined by default in prefs.js (the file that stores the settings, which the config editor edits). In that case you will probably have to right click on the body of the config editor, select Boolean from the context menu, enter mail.imap.fetch_by_chunks as the name of the setting and then select false as its value. That will add the setting.

FYI Default values of settings are stored in omni.ja (a renamed .jar file), but its not safe to edit that file as it may get replaced when Thunderbird updates. Use the config editor.


All times are GMT +9. The time now is 06:01 PM.


Copyright EmailDiscussions.com 1998-2022. All Rights Reserved. Privacy Policy