Help with IMAP archiving strategy

[CaptainSlog]

Hi All,

I'm using FastMail to pull together all of my POP3 email accounts into one IMAP account so I can use any device anywhere with a consistent interface.

At the moment I am using Thunderbird just on my laptop. This accesses all of the POP accounts directly, I use filters to sort them into local folders and I have accumulated well over 2Gbytes of emails as a result.

Now I've got a desktop PC, and in the future a smartphone. I'd like the laptop and desktop to have access to all the past emails for searching, but the smartphone only to the latest emails because it's resource limited.

What's the best stategy? I could, for example, have IMAP folders with the latest emails each of which has a subfolder with the archived emails. The laptop and desktop could subscribe to both, but the smartphone only to the normal IMAP folders and not the archives. I would have to manually move some emails regularly from the normal IMAP folders to the archives (unless this can be done automatically somehow).

Any ideas?
Thanks!
Mark.

[BritTim]

This is a very good question! I do not think there is a simple answer.

It would be nice if FM supported smart folders (folders created on the fly as a result of a search on the underlying folders). This would allow tricks like having folder Recent_In and Recent_Out constructed as a global search across all real folders of emails created within, say, the last 7 days.

As things stand, I think it may be possible to create a tool that runs on the desktop that maintains real folders constructed from suitable search criteria. I am very busy for the next couple of weeks, but will look into combining this with IMAPSize backup scripts when I get a chance.

[CaptainSlog]

Thanks! There's a possible solution, it seems some IMAP clients are able to download just the latest headers and content and maintain a latest set (I think Thunderbird can do this), but not sure if smartphone clients can (?).

It would be nice though to have server side filters to do this automatically, like 'move any emails older than 30 days in a folder to a specified archive folder'. I'm using server side filters to move incoming messages into folders so they are pre-sorted from the client's viewpoint. It would be great if the filtering in FastMail was a little more flexible, like being able to apply two conditions. One of the things I wanted was to junk all messages with the same To: and From: headers like a lot of SPAM does.

Next thing now is how to synchronisze contacts..

[n5bb]

There are several options available for archiving at the client side or the server side:

• In Thunderbird, look at the Tools>Account Settings>Synchronization & Storage settings for a particular account.
  • You can choose to synchronize all messages, or only the most recent messages.
  • You can choose to not download messages which are over a given size.
  • And you can choose to permanently delete old messages automatically.
• In Thunderbird, you can Archive a message by hitting the A key while reading a message in the message list screen. This will move that message to an Archive folder in the client, and mark that message as deleted (with line through it) in the Fastmail web interface. You can purge messages marked for deleted in a folder to clean up these marked for deletion messages. And you can move archived messages in Thunderbird back to an IMAP folder and they will reappear on the Fastmail server.
• In the Fastmail web interface Options>Folders screen you have some options for dealing with old messages:
  • In the settings for a particular folder, you can choose to auto-purge messages (permanently delete them) after a given number of days.
  • Click the Purge/Archive button at the bottom of the screen. This displays a screen which lets you archive messages (send you a zip file containing those messages) or delete old messages.
• All mobile IMAP email clients I have ever used have the ability to only look at the most recent messages (either by quantity or number of days).

Bill

[paleolith]

I keep an archive by using a Sieve script to copy messages to an archive folder. Though my script is much more complex, for this purpose it boils down to

addflag "\\Seen";
fileinto :copy "INBOX.archive";
removeflag "\\Seen";
fileinto "INBOX.edward";

INBOX.edward is the working inbox. If I had a smart phone, I would then subscribe to INBOX.edward but not to INBOX.archive. This way, I simply delete messages when I'm through with them -- I don't have to remember to archive them.

This has the obvious disadvantage that the desktop clients will download the messages twice, once for each mailbox. However, if you only need the archive to be on the server, you could set the desktop clients (one or both) to download only headers from INBOX.archive. In practice, I never notice the double download. I'm only using one computer on a regular basis, and I seldom use 10% of my monthly data transfer allowance from FM. YMMV of course.

Note that manipulating the "seen" flag is only so that messages in the archive will appear read rather than unread. Showing them unread is useless, since a different copy gets read. But the script would work with only the two "fileinto" lines.

It would be possible to make the copy on the desktop client and then upload it to the archive. But since upload speeds are usually much less than download speeds, making the copy on the server will be preferable in most cases.

If you want your old archives to be accessible on both laptop and desktop, you could create an IMAP archive for them and move the messages there. But if these are static archives, it might be easier just to make a copy in local folders on both systems.

The above does not archive outgoing messages. Thunderbird has some options for that -- I'm using Postbox, which is based on TB but not identical, so I'm not sure which options apply to TB. You might be able to simply save outgoing messages to INBOX.archive. In my case, I have PB set up to BCC all outgoing messages to sent-mail@mydomain.org, I enable "sent-mail" as a valid recipient for my domain, and have a script rule (before the other one) that says

if header :contains "X-Delivered-to" "sent-mail@mydomain.org" {
  addflag "\\Seen";
  fileinto "INBOX.archive";
  stop;
}

Again, there's issues of multiple copies being transferred. In this case, only one copy is uploaded, but then both desktop and laptop will download the sent-mail message unless configured not to as above. The same issue will apply to any method of archiving outgoing messages on the server.

I don't have a solution for synchronizing contacts yet ... that was my next project. Neither FM nor TB allows LDAP write access, so that's not a possibility. That's as far as I've gotten.

BTW, I recommend against automatically junking when to=from. Though spammers often do this, so do many of my friends who want to sent to a BCC list but either their mail client won't let them leave the To blank or they don't realize they can. Spam Assassin is far more accurate -- I've removed most of my specific spam rules since converting to SA.

Edward

[JMac]

I use Mailstore Home (free program) to archive ALL my messages. Keeps them in a separate database on my PC which I then back up locally and online.

Not accessible from a smartphone though.

Jim

[CaptainSlog]

Thanks for that detailed reply Edward.

I've been looking at the contacts issue too. I currently use SyncKolab add-on for TB which uses an IMAP folder to store contact info, this works OK for standard contacts (although a small issue with mailing lists, it works but sometimes complains about duplicates. I just skip that dialog box). So my TB contacts are now fully synchronized, but the web contacts aren't yet.

SyncKolab also has a standard vCard format in which each contact is stored in vCard format as a separate email in the IMAP folder. FM already has the ability to read this vCard format via a file upload and add them to the contacts, with a little help server side I think this could be used to sync the web contacts to the IMAP folder entries. The header information of each vCard email (and also a few blank lines at the end) need to be stripped off, and instead of uploading a file the web interface could made to point to an IMAP folder instead.

To sync the other way round (if a contact is added to the contacts via the web interface) requires a new vCard format email adding to the IMAP folder, possibly after checking for duplicates.

So it looks like it might be possible to sync FM contacts fo the TB contacts relatively easily using SyncKolab and a little server side scripting (?)

Mark.

[CaptainSlog]

Originally Posted by JMac

I use Mailstore Home (free program) to archive ALL my messages. Keeps them in a separate database on my PC which I then back up locally and online.

Not accessible from a smartphone though.

Jim

Thanks. I actually use IMAPSize for real long-term backup, it's nice though if all emails are accessible via IMAP.

[CaptainSlog]

@Bill: Thanks for that info on Thunderbird. If most IMAP clients can do this then archiving isn't too much of a problem, the next big issue is contacts syncing

[ChinaLamb]

Originally Posted by CaptainSlog

Thanks. I actually use IMAPSize for real long-term backup, it's nice though if all emails are accessible via IMAP.

I choose mailstore, since imapsize loses a lot of message information. Mailstore is able to keep all information.

[JMac]

Hey China, how do you like the new version of Mailstore? It appearsd that they have changed the way searches are done though the developer says that didn't change at all. I notice that when I use a partial email address as a search term I get no results at all.

Jim

[FredOnline]

Jim, I use Mail Store 5 too - I just did a few searches on partial e-mail addresses and it worked fine for me . . .

[ChinaLamb]

Originally Posted by FredOnline

Jim, I use Mail Store 5 too - I just did a few searches on partial e-mail addresses and it worked fine for me . . .

I havn't noticed any changes myself.