Test suite ?

[Nicolas Sebrecht]

On Tue, Aug 09, 2011 at 01:00:50PM +0200, Vladimir Marek wrote:
> > [ Could you please NOT set the Mail-Followup-To header. This breaks
> >   threads. ]
> 
> I'm sorry about that. I turned off Mail-Followup-To for this alias
> (hopefully).

Thanks.

> > > I can see three parts - IMAP, Maildir and state directory
> > > 
> > > IMAP
> > >  - simple socket server
> > >  - scripted scenarios
> > 
> > I think you're a bit confused, here. Scenarios is on the client side
> > whereas everything else you put in this section is about the server.
> 
> Maybe we are thinking about different tests here? What I had in mind was
> something like:
> 
> Let's build fake IMAP server which does not support APPENDUID and
> SEARCH. Now run OfflineIMAP with exactly this config and see if it
> synchronizes my mails correctly. Let's check that it still works against
> (scripted) Gmail like server.
> 
> So I'm thinking about script driven IMAP-like server.

Haha... Interesting! Here is a new approach we never had before.

Of course, it looks more powerfull than a true IMAP server as it also
helps to check for known IMAP breakages.

The idea sounds fine to me.

> > >  - single threaded only
> > 
> > Why? It looks like a strong limitation, to me.
> 
> Because that would be more difficult to implement:) Also it will be
> harder to make OfflineIMAP behave consistently between two test runs.

Thinking to this again, I don't think it would be a problem. What we
need is to be able to test multithreaded OfflineIMAP.

> > >  - should we check order of commands sent? Or should it be rather state
> > >    machine (find corresponding output to a given input)? Strictly
> > >    ordered commands means tighter checks, but does OfflineIMAP always
> > >    send commands in the same order?
> > 
> > Should we care about low-level issues _in a test suites_? My answer is
> > NO! Please, don't!
> > 
> > First, because we aren't working at the IMAP level. Imaplib2 does this.
> > If we decide to switch of library, everything done in this area is lost.
> 
> That's probably poor explanation on my side. My question is if
> OfflineIMAP is always guaranteed to send the IMAP commands in the same
> order. And yes, if we switch Imaplib2 for something else (not likely to
> happen I guess), I would expect some tests to break. But is that bad?
> Maybe we had certain sequence of IMAP commands because of some broken
> IMAP server?

We should do what a IMAP server would do for the IMAP protocol.  Notice
that introducing known server side oddities is not my point, here.

I mean the fake server should not do assumptions on what offlineimap is
handling IMAP protocol. Facing the fake IMAP server to another IMAP
client should just work.

> > Second, because we care about emails. I mean we have to know what was
> > uploaded, downloaded, modified, etc.
> 
> Agreed. My first intention was to try to make sure that happens with all
> possible strange IMAP servers. But maybe that's too much work ...

I think it is too much for now but it could be done in the longer term.
This would be a very interesting feature not too hard to add later.

> > Notice I was wrong and did not know that a user-level installation of
> > Dovecot is possible. So, a dovecot (whatever) installation into a
> > dedicated git repository is easy.
> 
> I don't know Dovecot, but I hoped to be able to simulate strange IMAP
> servers and cases in the tests. Like Gmail, UIDVALIDITY changed, not
> supported SEARCH command. If there would be a way to make Dovecot behave
> exactly as we like, that would be excellent!

I don't think we can do smart things like that with Dovecot. It would
ask to change the source code, re-compile, etc.

Another approach could be to write an IMAP proxy instead of IMAP server
and change sent data on the fly.

I'm not convinced it would be easier to implement and maintain in the
long term, though. ,-)

-- 
Nicolas Sebrecht