Technical question re IMAP performance and folders

jsamuelson · 26 Jul 2007, 06:04 PM

As the title says, a technical question on improving performance with IMAP.

I have archived old mail by year, in the format Archive, then subfolders by year, so they appear as Archive.2003 etc in the webmail interface. They appear properly nested in my mail client (Mail.app on Tiger).

Each year folder has an average of say 2000 messages archived.

In my ever-continuing attempts to improve performance, I have split each year into quarters - ie Archive.2003.1Q etc (again appears nested in the client).

My question is this - will this IMPROVE or DEGRADE performance? How does the database handle messages/folders and what is the most efficient, speedy way to file messages?

I have done this for some folders already but thought an IMAP guru may be able to give an instant answer.

Thanks, John

jsamuelson · 26 Jul 2007, 06:38 PM

A quick update - this does seem to improve performance. Is there a quantitive way to work out the optimum number of messages per folder? Has Runbox ever tested in this way?

Thanks, John

jsamuelson · 26 Jul 2007, 10:31 PM

Well I take it back, and should have known better. An enormous chunk of email has now gone astray, permanently gone from both client and server, during one of the copy/move operations.

Now I am going to have to trawl through an old image of my hard drive to try and extract mbox files and then the messages that are missing, which is a non-trivial exercise.

I am going to file a support request and a complaint. No email system that I rely on for my business has any right being this flaky. Ridiculous.

Geir · 27 Jul 2007, 01:29 AM

John,

Our apologies for the problems you've been having. As you might know we're working on improving both our IMAP server and overall performance.

A larger number of messages in a folder will typically entail a longer processing time on both the server and client side. The type of client in use will also make a difference -- Mail.app is known to be excessively resource intensive and we recommend using for instance Thunderbird instead.

The messages that disappeared are most likely found in our backup, but we need your Runbox username to restore them. Please let us know either here or by email to Runbox Support, and we'll have it done right away.

- Geir

Dreambiz · 27 Jul 2007, 03:59 AM

Quote:

Originally Posted by Geir

A larger number of messages in a folder will typically entail a longer processing time on both the server and client side. The type of client in use will also make a difference -- Mail.app is known to be excessively resource intensive and we recommend using for instance Thunderbird instead.

To Geir: People have their personal preferences and with all due respect, I think it would be more in your customer's (and potential customer's) interest if Runbox *supported* their choices rather than dismissing them or encouraging them to change to something because it's better for *you*. If Runbox has an issue with the way Mail.app does something I'd like to suggest you discuss it with Apple directly.

To John: I use Mail.App (and love it) and I keep a large number of sent messages (and refer to them frequently). I did just what you did and broke them out into multiple segments and it definitely sped things up. I didn't lose any messages that I'm aware of, but I did get a whole bunch of bad dates on chunks that were moved (that I was told was due to a problem on the server side that has since been fixed). I hope Runbox is able to recover your data.

- Randy

jsamuelson · 27 Jul 2007, 04:03 AM

Hi Geir,

I have been corresponding with Liz to try and find the missing messages.

If I wasn't quite so busy myself I would have more patience, but this is testing...as I get older and grouchier I am much less inclined to spend time troubleshooting.

Runbox really needs to sort out IMAP. The guys at Fastmail, for example, are out on the bleeding edge working out how the iPhone ticks and how to support P-IMAP for it, if possible. They have their problems too though...

I won't beat you up (figuratively!) about it anymore, but something needs to be done. I prefer to remain a Runbox client but these issues might force/drive me away.

Thanks for your reply.

Regards,

John

jsamuelson · 27 Jul 2007, 04:16 AM

Quote:

Originally Posted by Dreambiz

To Geir: People have their personal preferences and with all due respect, I think it would be more in your customer's (and potential customer's) interest if Runbox *supported* their choices rather than dismissing them or encouraging them to change to something because it's better for *you*. If Runbox has an issue with the way Mail.app does something I'd like to suggest you discuss it with Apple directly.

To John: I use Mail.App (and love it) and I keep a large number of sent messages (and refer to them frequently). I did just what you did and broke them out into multiple segments and it definitely sped things up. I didn't lose any messages that I'm aware of, but I did get a whole bunch of bad dates on chunks that were moved (that I was told was due to a problem on the server side that has since been fixed). I hope Runbox is able to recover your data.

- Randy

I must agree. If you have ever used a Mac and tried to use Thunderbird, you will know that a) it is also not the best IMAP client with many of its own quirks, and b) it is a very un-Mac-like experience.

Mail.app offers Spotlight (can be clunky but nothing else comes close), smart folders, system-wide hooks, services, and I use Mail Tags which streamlines my life. It is the best combination of features for me and many others and to my knowledge I do not know of any other email service that specifically warns people not to use it.

I have used Mail.app, Thunderbird, Mulberry, Entourage and Outlook with Runbox and they all have ups and downs.

Randy - what IMAP server are you using? Standard on port 143, SSL on 993, or beta on port 1143?

What settings do you use in Mail? Have you input the IMAP folder path of INBOX or have you left it blank? Do you cache emails/attachments? What folder structure have you used? Do you use the built-in junk filter or a different one (eg SpamSieve)?

Sorry for the barrage of questions, I rarely get a chance to speak to another Mail.app + Runbox user.

Regards,

John

jsamuelson · 27 Jul 2007, 07:55 PM

And the saga continues. I decided to test the 1143 beta IMAP server with my other, personal account, changed the settings, and it destroyed Mail.app, corrupting the preferences for the whole app and all mbox files for that account.

I reloaded the files from a backup and replaced the corrupt plist file and it was solidly repeatable 3 times.

Change server info = borked.

So I have managed to delete the account by setting it offline first and all seems well again.

The conclusion is that if you change the server, you must delete the account first and then set it up afresh.

Very weird, very annoying, never encountered before.

Geir · 27 Jul 2007, 09:15 PM

Quote:

Originally Posted by Dreambiz

To Geir: People have their personal preferences and with all due respect, I think it would be more in your customer's (and potential customer's) interest if Runbox *supported* their choices rather than dismissing them or encouraging them to change to something because it's better for *you*.

Our impression has been that Mail.app was resource intensive both on the client and the server side, and that some of its innovative features are rather asocial in the way they consume resources. For instance, the "smart folder" functionality, which lets the client perform continuous searches on the server, is a neat idea in theory -- but it hogs a disproportionate amount of server resources, potentially affecting all users of the service.

Runbox will naturally try to conform with, and encourage, new usage patterns that increase the users' efficiency, but as far as I can tell Mail.app has issues affecting the local system as well: http://www.google.com/search?q=mail%2eapp+performance

Quote:

Originally Posted by Dreambiz

If Runbox has an issue with the way Mail.app does something I'd like to suggest you discuss it with Apple directly.

The issues mentioned appear to be general knowledge, and realistically it's unlikely that Runbox can have much impact on Apple's plans for their own email application. It's probably better that we focus our time and energy on improving our own services.

For these reasons we thought it might be helpful to alert Mail.app users that there are alternatives that might work better for them in general. Furthermore, Runbox supports open source alternatives whenever possible, such as those offered by the Mozilla foundation (Thunderbird and Firefox).

- Geir

jsamuelson · 27 Jul 2007, 09:49 PM

Spotlight and smart folders rely on a local index, which I would assume might hit the server when initially being built/mail indexed but shouldn't hit it for each search or smart folder update.

In fact I have just run a search on two terms on my mail folders and observed NO network activity. Therefore my search had no impact on Runbox's servers. And trust me, your IMAP server is not that fast...my search results were almost instantaneous.

Just to prove my point I have unplugged my ethernet cable and run more searches. Now I know FOR SURE it was not beating up your server because it was impossible.

Maill.app does have an annoying habit of checking all folders for updates sometimes, and I admit it needs more robust IMAP folder subscription and caching logic but I do not believe that this should be an issue for Runbox. Many clients do this. It is unfair to malign Mail.app for common bad habits.

If I am incorrect in any way, let me know, I am genuinely curious to learn if you have formed your impression in an empirical way. I am email/IT savvy but no true expert.

Regards,

John

Geir · 27 Jul 2007, 10:11 PM

John: As you probably already know we restored your email from backup yesterday -- hopefully you'll find all the missing messages there.

I've never experienced that the local IMAP files are corrupted when changing the server info in the settings -- but then I've never had the pleasure of trying Mail.app.

Our support manager here at Runbox (Liz) uses a Mac but I believe she prefers Thunderbird.

Hopefully your Runbox experience will improve from now on, though.

- Geir

Geir · 27 Jul 2007, 10:22 PM

Quote:

Originally Posted by jsamuelson

Just to prove my point I have unplugged my ethernet cable and run more searches. Now I know FOR SURE it was not beating up your server because it was impossible.

That's very interesting. Our empirical evidence of Mail.app at times causing extreme server load is confined to a distinct correlation between our IMAP server logs and information from the user about the client and settings in use.

I'd be interested in the details of your Mail.app settings. Perhaps it's set to only perform searches on local data, or to only synchronize on certain occasions?

- Geir

Dreambiz · 28 Jul 2007, 03:15 AM

Since both Geir and John seem to be curious about Mail settings. Here are mine (that work fine):

IMAP port: 1143
IMAP Path Prefix: INBOX
Incoming Mail Server: IMAP.RUNBOX.COM
Automatically Synchronize Changed Mailboxes: NOT selected (but it does seem to resync/update when you select different folders within the local client)
Keep copies of messages for offline viewing: ALL MESSAGES AND THEIR ATTACHMENTS
Include when automatically checking for new mail: SELECTED
Store draft messages on the server: NOT selected
Store sent messages on the server: SELECTED
Delete sent messages when: NEVER (I do this manually when I want it done)
Store junk messages on the server: NOT selected
Delete junk messages when: NEVER (I do this manually after quickly reviewing them for legitimate email)
Move deleted messages to the trash mailbox: SELECTED
Store deleted messages on the server: SELECTED
Permanently erase deleted messages when: NEVER (I do this manually)
Check for new mail: MANUALLY (again, it appears to do this whenever I select a folder/sub-folder)
When searching all mailboxes, include results from (trash, junk, encrypted messages): NONE SELECTED
Remove unedited downloads: AFTER MESSAGE IS DELETED

I think those are the main ones. Any questions? There will be a test at the end of class...

- Randy

Dreambiz · 28 Jul 2007, 03:16 AM

Quote:

Originally Posted by jsamuelson

Spotlight and smart folders rely on a local index, which I would assume might hit the server when initially being built/mail indexed but shouldn't hit it for each search or smart folder update.

In fact I have just run a search on two terms on my mail folders and observed NO network activity. Therefore my search had no impact on Runbox's servers. And trust me, your IMAP server is not that fast...my search results were almost instantaneous.

Just to prove my point I have unplugged my ethernet cable and run more searches. Now I know FOR SURE it was not beating up your server because it was impossible.

My experience is the same as John's in this regard.

jsamuelson · 28 Jul 2007, 03:34 AM

Hi Geir - for the sake of keeping the original thread updated, I have seen some performance benefit to breaking up large message folders into chunk of say 500 messages apiece.

Liz was very responsive, but from quick notes I made while transferring messages before everything went pear-shaped I am still out a few hundred messages, some from July this year and some from December 2006 still.

I have a previous backup of my mail folders, so I know if I really find that these messages were vital I can dig them out, but I am running out of energy and enthusiasm so now I just want to get everything stable again!

The plist file which stores all Mail's settings, plugin settings, rules etc was corrupted repeatedly when I changed the server, and Mail would also lock up when redownloading mail for the other account. It is all very bizarre. It simply did not like the change in server settings. OSX is normally much more resilient than this.

Spotlight - which Panther did not have - relies on a local index maintained by the system. Any application can provide data to the index through plugins etc. Some of the performance issues people have complained about has been Spotlight deciding to index/reindex items and sucking down system resources. Once it has finished it is very snappy though, really fantastic - and available system-wide. Some people have experienced worse issues with Spotlight database corruption, but for a first gen technology it is very solid.

Even if Liz doesn't like Mail, I would be surprised if she wasn't a fan of Spotlight.

The major change for Mail.app was that now, instead of a single database of mail, each message was split out into an .emlx file so that each individual message can be indexed fully. I am fairly sure this creates some extra load as well.

I would be interested to see how different versions of Mail impact your server. I suspect Panther Mail was much heavier.

Listen, if you want to give me a test account and a few thousand messages to mess about with I am very happy to work with you so you can look at how certain settings or scenarios work out.

My Runbox experience up till yesterday has been generally excellent, hence so few posts even though I think I've used the service for 3/4 years. Let's get that straight!

If I didn't think you were going to respond well I wouldn't waste my time posting here.

Regards,

John

26 Jul 2007, 06:04 PM	#1
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	Technical question re IMAP performance and folders As the title says, a technical question on improving performance with IMAP. I have archived old mail by year, in the format Archive, then subfolders by year, so they appear as Archive.2003 etc in the webmail interface. They appear properly nested in my mail client (Mail.app on Tiger). Each year folder has an average of say 2000 messages archived. In my ever-continuing attempts to improve performance, I have split each year into quarters - ie Archive.2003.1Q etc (again appears nested in the client). My question is this - will this IMPROVE or DEGRADE performance? How does the database handle messages/folders and what is the most efficient, speedy way to file messages? I have done this for some folders already but thought an IMAP guru may be able to give an instant answer. Thanks, John

26 Jul 2007, 06:38 PM	#2
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	A quick update - this does seem to improve performance. Is there a quantitive way to work out the optimum number of messages per folder? Has Runbox ever tested in this way? Thanks, John

26 Jul 2007, 10:31 PM	#3
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	Well I take it back, and should have known better. An enormous chunk of email has now gone astray, permanently gone from both client and server, during one of the copy/move operations. Now I am going to have to trawl through an old image of my hard drive to try and extract mbox files and then the messages that are missing, which is a non-trivial exercise. I am going to file a support request and a complaint. No email system that I rely on for my business has any right being this flaky. Ridiculous.

27 Jul 2007, 01:29 AM	#4
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	John, Our apologies for the problems you've been having. As you might know we're working on improving both our IMAP server and overall performance. A larger number of messages in a folder will typically entail a longer processing time on both the server and client side. The type of client in use will also make a difference -- Mail.app is known to be excessively resource intensive and we recommend using for instance Thunderbird instead. The messages that disappeared are most likely found in our backup, but we need your Runbox username to restore them. Please let us know either here or by email to Runbox Support, and we'll have it done right away. - Geir

27 Jul 2007, 04:03 AM	#6
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	Hi Geir, I have been corresponding with Liz to try and find the missing messages. If I wasn't quite so busy myself I would have more patience, but this is testing...as I get older and grouchier I am much less inclined to spend time troubleshooting. Runbox really needs to sort out IMAP. The guys at Fastmail, for example, are out on the bleeding edge working out how the iPhone ticks and how to support P-IMAP for it, if possible. They have their problems too though... I won't beat you up (figuratively!) about it anymore, but something needs to be done. I prefer to remain a Runbox client but these issues might force/drive me away. Thanks for your reply. Regards, John

27 Jul 2007, 07:55 PM	#8
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	And the saga continues. I decided to test the 1143 beta IMAP server with my other, personal account, changed the settings, and it destroyed Mail.app, corrupting the preferences for the whole app and all mbox files for that account. I reloaded the files from a backup and replaced the corrupt plist file and it was solidly repeatable 3 times. Change server info = borked. So I have managed to delete the account by setting it offline first and all seems well again. The conclusion is that if you change the server, you must delete the account first and then set it up afresh. Very weird, very annoying, never encountered before.

27 Jul 2007, 09:49 PM	#10
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	Spotlight and smart folders rely on a local index, which I would assume might hit the server when initially being built/mail indexed but shouldn't hit it for each search or smart folder update. In fact I have just run a search on two terms on my mail folders and observed NO network activity. Therefore my search had no impact on Runbox's servers. And trust me, your IMAP server is not that fast...my search results were almost instantaneous. Just to prove my point I have unplugged my ethernet cable and run more searches. Now I know FOR SURE it was not beating up your server because it was impossible. Maill.app does have an annoying habit of checking all folders for updates sometimes, and I admit it needs more robust IMAP folder subscription and caching logic but I do not believe that this should be an issue for Runbox. Many clients do this. It is unfair to malign Mail.app for common bad habits. If I am incorrect in any way, let me know, I am genuinely curious to learn if you have formed your impression in an empirical way. I am email/IT savvy but no true expert. Regards, John

27 Jul 2007, 10:11 PM	#11
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	John: As you probably already know we restored your email from backup yesterday -- hopefully you'll find all the missing messages there. I've never experienced that the local IMAP files are corrupted when changing the server info in the settings -- but then I've never had the pleasure of trying Mail.app. Our support manager here at Runbox (Liz) uses a Mac but I believe she prefers Thunderbird. Hopefully your Runbox experience will improve from now on, though. - Geir

28 Jul 2007, 03:15 AM	#13
Dreambiz Essential Contributor Join Date: Sep 2004 Posts: 329	Since both Geir and John seem to be curious about Mail settings. Here are mine (that work fine): IMAP port: 1143 IMAP Path Prefix: INBOX Incoming Mail Server: IMAP.RUNBOX.COM Automatically Synchronize Changed Mailboxes: NOT selected (but it does seem to resync/update when you select different folders within the local client) Keep copies of messages for offline viewing: ALL MESSAGES AND THEIR ATTACHMENTS Include when automatically checking for new mail: SELECTED Store draft messages on the server: NOT selected Store sent messages on the server: SELECTED Delete sent messages when: NEVER (I do this manually when I want it done) Store junk messages on the server: NOT selected Delete junk messages when: NEVER (I do this manually after quickly reviewing them for legitimate email) Move deleted messages to the trash mailbox: SELECTED Store deleted messages on the server: SELECTED Permanently erase deleted messages when: NEVER (I do this manually) Check for new mail: MANUALLY (again, it appears to do this whenever I select a folder/sub-folder) When searching all mailboxes, include results from (trash, junk, encrypted messages): NONE SELECTED Remove unedited downloads: AFTER MESSAGE IS DELETED I think those are the main ones. Any questions? There will be a test at the end of class... - Randy

28 Jul 2007, 03:34 AM	#15
jsamuelson Senior Member Join Date: Oct 2006 Location: Switzerland Posts: 138	Hi Geir - for the sake of keeping the original thread updated, I have seen some performance benefit to breaking up large message folders into chunk of say 500 messages apiece. Liz was very responsive, but from quick notes I made while transferring messages before everything went pear-shaped I am still out a few hundred messages, some from July this year and some from December 2006 still. I have a previous backup of my mail folders, so I know if I really find that these messages were vital I can dig them out, but I am running out of energy and enthusiasm so now I just want to get everything stable again! The plist file which stores all Mail's settings, plugin settings, rules etc was corrupted repeatedly when I changed the server, and Mail would also lock up when redownloading mail for the other account. It is all very bizarre. It simply did not like the change in server settings. OSX is normally much more resilient than this. Spotlight - which Panther did not have - relies on a local index maintained by the system. Any application can provide data to the index through plugins etc. Some of the performance issues people have complained about has been Spotlight deciding to index/reindex items and sucking down system resources. Once it has finished it is very snappy though, really fantastic - and available system-wide. Some people have experienced worse issues with Spotlight database corruption, but for a first gen technology it is very solid. Even if Liz doesn't like Mail, I would be surprised if she wasn't a fan of Spotlight. The major change for Mail.app was that now, instead of a single database of mail, each message was split out into an .emlx file so that each individual message can be indexed fully. I am fairly sure this creates some extra load as well. I would be interested to see how different versions of Mail impact your server. I suspect Panther Mail was much heavier. Listen, if you want to give me a test account and a few thousand messages to mess about with I am very happy to work with you so you can look at how certain settings or scenarios work out. My Runbox experience up till yesterday has been generally excellent, hence so few posts even though I think I've used the service for 3/4 years. Let's get that straight! If I didn't think you were going to respond well I wouldn't waste my time posting here. Regards, John