Best alternative to maildir?

[Nicolas Sebrecht]

On Fri, Sep 19, 2014 at 11:28:13PM +0200, Bart Schouten wrote:

> There is a good reason why not any commercial (let's say, really popular) 
> email client would never store email as individual files. 99.5% of users 
> would complain about it. People who go into Thunderbird see a neat lists 
> of files that coincide with their IMAP folders. They are not burdened with 
> the details of what is inside those files, because they have a tool for 
> that: It's called their email program (Thunderbird itself).

OfflineIMAP was designed to have a local maildir as a user of IMAP mail
servers.

> My backups get TUCKED AWAY inside even encrypted rsyncable tarballs. Why 
> on EARTH do I want other tools to be able to operate on that? In general a 
> backup wants to be restored in its entirety and when that is not the case, 
> the backup tool itself (in this case OfflineIMAP) can arrange for its 
> individual extraction.

OfflineIMAP is NOT a backup system on its own. It acts as a mirorring
system and let the user do the backups on TOP of his local synced
maildir.

Though, we have the read only mode to prevent from local mistakes to
propagate on the IMAP server.

That's what I do for all my mailboxes. One is 1.7+ Go of raw mails.

> With email at least, I rarely make manual mistakes because we have Trash 
> folders that provide a two-level delete. So my backup is meant for 
> safekeeping, not for operating on. In the rare event that I do need to 
> operate on it, it really won't be hard, I can promise you that.

I backup my emails not only because of my mistakes but also to prevent
from mistakes of my provider. All providers have and will lose emails of
customers at some point in time.

>            Even for a mail spool, I would not depend on individual files. 
> You are putting too much emphasis and too much reliance and too much 
> dependence on the filesystem handling your operational tasks when it is 
> not even designed for that. It is abusing your fs.

No offense, but I guess we will all disagree here and the reasons stand
in two points.

First, we do trust more filesystems than any email software handling its
own "home-made hidden filesystem which doesn't assume its name" to store
emails. As you say, mainly because system/kernel filesystems are much
more widely tested.

Second, we don't rely on the local filesystems for our backups. Having a
synced mirrored local maildir alone is not a backup.
Backuping is about having available archives we create FROM the local maildir. 

>                 Sure I lack the tools presently to automate it. I have a 
> sync script setup to a USB stick and that works fine, I don't have it 
> loaded now. If I need a third location for this text of this email, I 
> would mount it and start syncing that. Currently I need a sync from IMAP 
> to local files more than anything else. But in principle, individual 
> drafts of individual emails are not a meaningful distinction or 
> separation. If I could sync the entire folder at once, that would be 
> exactly what I need. Why then store as individual files? What purpose does 
> it have?

Explanations above.

> The only reason you can say mbox files are more prone to corruption is 
> because you are now depending on the functioning of the filesystem proper 
> which has seen much greater development than anything any individual tool 
> could probably allow or mandate. So you are abusing something that is just 
> well-tested and fully-developed even though it is not the proper place for 
> that functionality.

You assume we are "abusing" the filesystem. We think filesystems are the
best already-well-tested lines of code we can use for the purpose of
storing such data (emails).

> Any reasonable "folder" implementation should have indices as PART of the 
> container itself. And then your tool should have a sublayer that handles 
> ALL access into individual "sections" such as would be, for example, 
> emails. If that sublayer is developed well enough, there should be no 
> issues more than in a filesystem that is equally well-developed.

"If that sublayer is developed well enough"... and as a user you don't
hit a bug at some point.

>                                                                  SQLite 
> should be such a "sublayer". Perhaps it is not perfectly ideal but it 
> should do the job for now...

SQLite or any other database engine is the additional layer we like to
avoid. The more layers and lines of code are involved, the less it
becomes reliable.

We use SQLite only for the cache of OfflineIMAP. We don't care about the
cache if it's dead.

SQLite is the choice for the cache because it's easily embedded in
OfflineIMAP and doesn't require us to set up a database engine like
mysql. Also, SQLite is the default alternative of the historical plain
text system we used to maintain the cache.

> Right? Maybe I'm wrong about that... I was abit hesitant about SQLite for 
> this from the beginning.... Any regular database stores indices separate 
> from the data itself, I believe. That may seem common sense, but it is 
> not, because you have now created a dependency that is, from the level of 
> the filesystem, external. Thunderbird [...]

For what you'd like, you'll have to rewrite big pieces of OfflineIMAP
because it was designed only for the maildir format from the beginning.

If technically possible, I would not recommend it. Sorry.

Hope you'll find what you're looking for!

-- 
Nicolas Sebrecht