[PATCH 5/7] Re: Don't keep sqlite connections open

[Sebastian Spaeth]

On Sat, 7 May 2011 07:08:35 -0700 (PDT), chris coleman wrote:
> I bring this up because I see the project has been spending valuable energy reinventing some basic database technology that already exists and is tested by millions of users already-

So far I don't see us reinventing database technology. I just implemented
_using_ some database tech, but that is no more or less work than using a
database abstraction layer.

I implemented the sqlite database backend, so that we take advantage of
existing database technology and prevent all the things you list
below. I don't see how enabling us to plug in 5 different rather than 1
suitable database makes us reinvent the wheel any less or makes us any safer.

> a) flat files 
> that are written to disk so often so they can play the role of journaling and transaction logs - to prevent crashes from losing/corrupting data.  

The flat files support has been written in 2002, neither sqlite nor persistence
frameworks were around then. At least not easily usable and with nice
python bindings. Not much energy has gone into the writing part
since. :) 

> b) cached local copy of a remote database index (this is what the LocalStatus file is). 

> With the trend being toward larger and larger IMAP inboxes (unlimited email storage available nearly everywhere), there are larger and larger LocalStatus cache files.
> The chances of crashing 
> and losing/corrupting data in the middle of a multi-megabyte write that takes place several times per second  - is going to get bigger not smaller.  

Yes, mailstores are going to be bigger, that's why we are optionally
offering a database cache now.

> So, seriously, why 
> not just let a proven reliable database handle the data integrity 
> concerns?  Even a 64MB laptop can run free mysql with room to spare so it can't be because of system requirements...

Such as sqlite? We are! Why I am opposed to MySQL?

1) Installation is harder. 2) I don't want mysql constantly running just
because I need to store 32,000 integer numbers on my disk. Should I
start up MySQL every 10 minutes to sync or do you want me to keep mysql
running all the time on my netbook? Just not gonna happen :)

> I would say that if you're willing to try, the next step should be to point out which source files contain database calls.
> Do you have, or could you write up, a document that lists the source files that call the db.

Yes: offlineimap/folder/LocalStatus.py (or alternatively
offlineimap/folder/LocalStatusSQLite.py). That's it. There are about 5
calls to a database function.

> And the unwritten rules that must be followed when making calls to the db. 
> That is the hard part.

Don't corrupt the db? ;-P

> The next step after that is easy.  Just have to alter the calls to talk through one of the high rated persistence frameworks... and test.

As I said, it is easy to add new backends. And as I said, *I* am
not interested in doing it.

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110508/eb791307/attachment-0001.sig>