A comment on rewrites

[John Goerzen]

This is (collectively) your project now, so by all means feel free to 
ignore me entirely.  I wanted to share a bit of experience about 
rewrites of large free software projects.

Most of the time, when there is an established Open Source project with 
a fairly significant codebase, and some people go off to rewrite it from 
scratch, the rewrite never gets done.  If enough people from the 
original project get invested in the rewrite, it may even kill the 
original project and wind up with no replacement.  There are exceptions, 
of course, but I've seen this pattern several times.

Part of the reason for this was already mentioned by someone else: if 
the project is large enough to merit thoughts of a rewrite, chances are 
it's also pretty large and complex.  OfflineIMAP has somewhere around 50 
configuration options, three mail backend drivers, 4 UIs (plus a fifth 
Gnome one in development), and 8 years of testing and use.  It also has 
various hooks for extending it with custom Python code.

This won't be duplicated overnight in any language with any library.  In 
fact, it will take quite a bit of time to duplicate.  I should know: 
I've attempted a rewrite twice, running out of steam myself.  The first 
was a few years back when I rewrote it to use Python Twisted.  I got it 
to actually work to a certain extent before giving up in frustration at 
the difficulty of debugging in Twisted.  More recently I tried to 
rewrite it in Haskell, but ran out of time due to other commitments.

There are some problems with OfflineIMAP, but a whole rewrite isn't 
needed to fix them.  The CPU-bound problem can be dealt with quite 
easily by using Sqlite or something like it for the LocalStatus storage. 
  There have been patches sent to the list in the past that got about 
90% of the way to being accepted, in fact.  (If I recall correctly, they 
ultimately weren't because they didn't migrate existing LocalStatus 
data.)  There may be other ways to address it also.  The problem is 
unlikely to be language, but rather algorithm in a particular area.

If a rewrite is attempted, it should be recognized that:

1) This will take a significant amount of time before it can begin to 
displace OfflineIMAP 6.2.x

2) In the meantime, the 6.2.x branch will need to be kept active or else 
the community will be lost before the rewrite is finished.

3) Ease of migration is important.

I think it unlikely that choice of language will impact OfflineIMAP 
performance or reliability.  A well-written algorithm will always be 
much faster than the networked IMAP server.  OfflineIMAP's current 
algorithms were never designed with tens of thousands of messages per 
folder in mind, and it is natural that they aren't very fast in such 
situations.  (Note: I say that as someone with strong opinions in 
programming languages; choice of language *does* impact some things, but 
for this project, is unlikely to impact performance.)

It may be that incremental changes result in, say, 3 years down the road 
none of my OfflineIMAP code survives.  That is fine, and a sign of an 
active project.  Such a process for change is more likely to succeed 
than an entire rewrite that is completed in 1 year, and also more likely 
to have high quality after 3.

One other question is whether a rewritten project ought to be called 
"OfflineIMAP".  The answer probably depends upon how well it can grok 
~/.offlineimaprc, the ~/.offlineimap metadata storage, and implements 
the full OfflineIMAP feature set.  If it doesn't do all of this pretty 
well, it probably ought to be named something else.

Having said that: once again, this is your project, so to those of you 
that are willing to contribute to the community -- whether it be a 
rewrite or not, whether it is called OfflineIMAP or not -- more power to 
you, and thanks for the work you do.

-- John