Integration of email into OS file system

pjroutledge · 27 Jun 2023, 01:06 PM

Has anybody come across an email client that integrates email into an OS file system?

At the moment I feel like it's necessary to maintain separate systems, one for files and one for emails. Or, I end up with files remaining stored wihin emails in my email service/client, rather than searchable, indexed and version controlled in my file system.

I realise that I can export individual emails as files into file system folders, but what I'd like is an integrated OS and email client that just do it.

So I would have a folder structure that served to organise files and emails together. If I saved an email to, say, a folder called 'Holidays', it would be stored as a file along with all the other 'Holidays' files (spreadsheets, text documents, notes, etc). Ideally, the email client would have the option of showing all of the contents of file system folders - both emails and non-email files.

Essentially I suppose the idea is that the folder structure shown in the email client is actually the underlying OS folder structure and individual emails are stored as individual documents in that folder structure.

The ideal email client would also be able to keep track of the email documents and present them as threaded conversations if desired.

hadaso · 28 Jun 2023, 08:14 PM

In the first years I was using email it worked for me quite like that (that wasduring the 1990's)/
All incoming email was in a file in the home directory of my user account on the university department's Unix system. I think the file was called ".mbox", or perhaps ".mail"/ I don't remember those details. Perhaps the file was not in my home directory but on some public directory that had an inbox file for easch user.
Everything was command line (and the monitor was a black screen with green or white text).
The mail command opened the file of incoming mail for reading/ After reading I could save the read message to a default file (perhaps that one was .mbox"?) but I usually saved each message to a mail file in a folder that held other files related to the same subject, so I had evething classified by subject, including a file that holds all correspondence related to the topic of each directory.
Eventually email clients evolved differently, archiving email messages in one place that is separate from all other related files (and also the annoying practice of separating outgoing messages in a separate "sent" folder).
I still have all my files from back then, but I never found the time to collect all those mail files from all the folders and integrating them with my more recent mail (that I received since around the year 2000, and is all in my FastMail account).

JeremyNicoll · 28 Jun 2023, 09:40 PM

There are/were some email clients that keep everything as individual files.

The advantages are: it's a simple idea, if something corrupts a single file you lose only its contents, external tools can search (or otherwise manipulate the emails). Even if the client goes spectacularly wrong you still have all your data in an accessible form.

The disadvantages are: it can be a LOT slower than systems which keep multiple mails in one single file. Each time the email client creates a new file or even just opens, reads and closes one, the file system will have to update its index of where all the files are (even if only updating meta data like "time of last access" for a file). The mail client may have to update its indexes every time too ... and may have to write those back to disk to "commit" the data each time, too (so that if your machine crashes at that point all the files are in sync next time the client starts).

If the file-system has a limit on the number of files that it can store overall, hundreds of thousands of separate emails might be a problem - and if your backups of all those files are also separate files (rather than eg zips) then the limit will be reached sooner.

Also some file systems have (or had) limits to the number of files one could store in an individual folder. A system I used about 20 years ago only allowed 77 or so files per folder. So to store thousands of files under a single folder that folder had to have multiple subfolders (and sometimes those also had subfolders) and the individual data files were scattered through the sub-[sub-]folders ... and the email client had to know where IT had put them (separate from the file-system itself knowing where things were).

Also, that advantage (that an external tool can search/modify the individual files) brings with it the disadvantage that a tool like that (or malware) can damage the relationship between all the separate files. For example, when you, using the mail client, open a mail client folder, you will want a near-instant display of summary info about all those mails. It can't be done instantly - certainly not by opening each separate mail file and parsing out salient data. So systems like that will cache a summary/index. But because that's also a separate file, there's a risk if something changes its contents out of sync with the individual mail files, or eg a tool changes the subject lines of some of the stored emails but doesn't also update the index. All such systems will have utilities that rebuild the indexes, and - hopefully - also some way to detect that an index might not be in sync with the folder's contents.

The speed arguments are much less important nowadays when even quite basic PCs are so much faster than those 20-30 years ago, and when many people use SSDs rather than spinning rust .. or even floppy disks.

There's lots of halfway points between "everything is an individual fle" and "everything is in one huge file/database".

Eg a client might decide to store all its control data in a single file (eg an SQLite database). The file is either open (exclusively by the mail client) or it isn't ... so no other program should be able to alter any of that control data while the client is running. All the separate tables of control info are in that one file, so - provided there's no logic errors in the mail client - they should always be in sync. It'd be fast, and the database query language will allow subsets of control info to be extracted - so some searches etc will be very fast. BUT it'll be harder for external programs to read the control info. Not impossible, but not as easy as opening a plain text index in a text editor....

I'm not well-informed about mainstream mail clients - those I've used were either for minority platforms, years ago, or did do the kind of thing you mention but are no longer maintained.

The Claws email client (mainly on linux though there is a Windows port) has some choices about which mail file format one uses on the PC (if indeed you're storing your mail on your own machine rather than on servers) and I think at least one of those - the "MH" format - uses separate files per mail. See: https://www.claws-mail.org/

Claws is VERY versatile, but that also means it's hard for users to get to grips with. Documentation is - to my mind - sparse. There is however a claws-users mail list (which I lurk on - as I do many mail lists because they are interesting. I don't currently use Claws.)

27 Jun 2023, 01:06 PM	#1
pjroutledge Senior Member Join Date: Jan 2010 Location: Melbourne, Oz Posts: 133	Integration of email into OS file system Has anybody come across an email client that integrates email into an OS file system? At the moment I feel like it's necessary to maintain separate systems, one for files and one for emails. Or, I end up with files remaining stored wihin emails in my email service/client, rather than searchable, indexed and version controlled in my file system. I realise that I can export individual emails as files into file system folders, but what I'd like is an integrated OS and email client that just do it. So I would have a folder structure that served to organise files and emails together. If I saved an email to, say, a folder called 'Holidays', it would be stored as a file along with all the other 'Holidays' files (spreadsheets, text documents, notes, etc). Ideally, the email client would have the option of showing all of the contents of file system folders - both emails and non-email files. Essentially I suppose the idea is that the folder structure shown in the email client is actually the underlying OS folder structure and individual emails are stored as individual documents in that folder structure. The ideal email client would also be able to keep track of the email documents and present them as threaded conversations if desired.

28 Jun 2023, 08:14 PM	#2
hadaso The "e" in e-mail Join Date: Oct 2002 Location: Holon, Israel. Posts: 4,855	In the first years I was using email it worked for me quite like that (that wasduring the 1990's)/ All incoming email was in a file in the home directory of my user account on the university department's Unix system. I think the file was called ".mbox", or perhaps ".mail"/ I don't remember those details. Perhaps the file was not in my home directory but on some public directory that had an inbox file for easch user. Everything was command line (and the monitor was a black screen with green or white text). The mail command opened the file of incoming mail for reading/ After reading I could save the read message to a default file (perhaps that one was .mbox"?) but I usually saved each message to a mail file in a folder that held other files related to the same subject, so I had evething classified by subject, including a file that holds all correspondence related to the topic of each directory. Eventually email clients evolved differently, archiving email messages in one place that is separate from all other related files (and also the annoying practice of separating outgoing messages in a separate "sent" folder). I still have all my files from back then, but I never found the time to collect all those mail files from all the folders and integrating them with my more recent mail (that I received since around the year 2000, and is all in my FastMail account).

28 Jun 2023, 09:40 PM	#3
JeremyNicoll Essential Contributor Join Date: Dec 2017 Location: Scotland Posts: 490	There are/were some email clients that keep everything as individual files. The advantages are: it's a simple idea, if something corrupts a single file you lose only its contents, external tools can search (or otherwise manipulate the emails). Even if the client goes spectacularly wrong you still have all your data in an accessible form. The disadvantages are: it can be a LOT slower than systems which keep multiple mails in one single file. Each time the email client creates a new file or even just opens, reads and closes one, the file system will have to update its index of where all the files are (even if only updating meta data like "time of last access" for a file). The mail client may have to update its indexes every time too ... and may have to write those back to disk to "commit" the data each time, too (so that if your machine crashes at that point all the files are in sync next time the client starts). If the file-system has a limit on the number of files that it can store overall, hundreds of thousands of separate emails might be a problem - and if your backups of all those files are also separate files (rather than eg zips) then the limit will be reached sooner. Also some file systems have (or had) limits to the number of files one could store in an individual folder. A system I used about 20 years ago only allowed 77 or so files per folder. So to store thousands of files under a single folder that folder had to have multiple subfolders (and sometimes those also had subfolders) and the individual data files were scattered through the sub-[sub-]folders ... and the email client had to know where IT had put them (separate from the file-system itself knowing where things were). Also, that advantage (that an external tool can search/modify the individual files) brings with it the disadvantage that a tool like that (or malware) can damage the relationship between all the separate files. For example, when you, using the mail client, open a mail client folder, you will want a near-instant display of summary info about all those mails. It can't be done instantly - certainly not by opening each separate mail file and parsing out salient data. So systems like that will cache a summary/index. But because that's also a separate file, there's a risk if something changes its contents out of sync with the individual mail files, or eg a tool changes the subject lines of some of the stored emails but doesn't also update the index. All such systems will have utilities that rebuild the indexes, and - hopefully - also some way to detect that an index might not be in sync with the folder's contents. The speed arguments are much less important nowadays when even quite basic PCs are so much faster than those 20-30 years ago, and when many people use SSDs rather than spinning rust .. or even floppy disks. There's lots of halfway points between "everything is an individual fle" and "everything is in one huge file/database". Eg a client might decide to store all its control data in a single file (eg an SQLite database). The file is either open (exclusively by the mail client) or it isn't ... so no other program should be able to alter any of that control data while the client is running. All the separate tables of control info are in that one file, so - provided there's no logic errors in the mail client - they should always be in sync. It'd be fast, and the database query language will allow subsets of control info to be extracted - so some searches etc will be very fast. BUT it'll be harder for external programs to read the control info. Not impossible, but not as easy as opening a plain text index in a text editor.... I'm not well-informed about mainstream mail clients - those I've used were either for minority platforms, years ago, or did do the kind of thing you mention but are no longer maintained. The Claws email client (mainly on linux though there is a Windows port) has some choices about which mail file format one uses on the PC (if indeed you're storing your mail on your own machine rather than on servers) and I think at least one of those - the "MH" format - uses separate files per mail. See: https://www.claws-mail.org/ Claws is VERY versatile, but that also means it's hard for users to get to grips with. Documentation is - to my mind - sparse. There is however a claws-users mail list (which I lurk on - as I do many mail lists because they are interesting. I don't currently use Claws.)