IDLE again

[exarkun at twistedmatrix.com]

On 05:45 am, dom.lobue at gmail.com wrote:
>
>Jean-Paul,
>
>I read over the IMAP IDLE RFC and went through the IMAP4 twisted
>library and I've sketched out a rough outline of how to implement
>IDLE. I've run into some things in Twisted however that I don't really
>understand well, and I'm hoping you can point me in the correct
>direction.

Cool.  That was quick. :)  Before I get into things, since this thread 
is likely to go into a lot of Twisted-specific details which may not be 
generally interesting, if there's anyone who'd like off the cc list, 
please speak up. :)
>First, just to verify my understanding: the IMAP4Client class is a
>Protocol class.

Yep.  And to expand on that, instances of Protocol classes typically 
have a one-to-one relationship with a connection.
>All IMAP commands are represented by at least two
>methods in the IMAP4Client class - one for what to do when the command
>is received from the server, and one for when the command is sent to
>the server.

Generally, though there are some exceptions.  For example, AUTHENTICATE 
is implemented with one method that starts by possibly sending a 
CAPABILITY (IMAP4Client.authenticate) command, then another method which 
will actually send AUTHENTICATE (IMAP4Client.__cbAuthenticate), then two 
more methods for dealing with the response to the AUTHENTICATE 
(IMAP4Client.__cbContinueAuth and IMAP4Client.__cbAuthTLS).

Another way to look at it is like this.  For each supported protocol 
action, there is at least one public method on IMAP4Client to initiate 
this action by sending some bytes to the server.  All bytes received by 
IMAP4Client from the server are parsed according to the state the client 
is in, what commands are outstanding, etc.  Depending on the state and 
the bytes, callbacks might be invoked as a result of this, possibly 
delivering the results of a protocol action initiated earlier to the 
calling application code.

This doesn't disagree with what you said too much, it just re-states it 
in slightly more general terms.
>Assuming this assertion to be true, the broad strokes of
>the IDLE implementation is as follows:
>
>-IDLE is engaged: command is sent to server turning on IDLE, schedule
>an IDLE reset in 29 minutes, and an attribute ( _IDLE_Enabled for
>example ) is set to True.

Basically, yes.  One subtle point, though - since the server might 
reject the IDLE command, the client shouldn't assume it has entered the 
IDLE state until it receives a positive acknowledgement of the command 
from the server (eg the "+ idling" line from the RFC).
>-In the command dispatcher code: checks if IDLE is enabled or not. If
>enabled, it appends IDLE to all incoming commands and sends a DONE to
>the server before any new commands are sent. (On the incoming commands
>part: what I mean is if the method originally to be called was
>"incoming_exists", instead it would go to "incoming_existsIDLE".)

This part could probably bear some elaboration.  I think the question 
here is how the unsolicited information should best be made available to 
the application code which caused the IDLE to be issued (if I've 
misunderstood what you were getting at here, let me know).  One 
possibility is the existing IMailboxListener interface - if you look at 
the very end of the implementation of IMAP4Client, you'll find three no- 
op methods, modeChanged, flagsChanged, and newMessages.  These are 
intended for subclasses to override and are already called by 
IMAP4Client when unsolicited information is given by the server in 
response to a command.  It may make sense to direct data provided during 
an IDLE to these callbacks, or others similar to them.
>-When IDLE is confirmed off by server: delete scheduled IDLE reset.

Yep.
>And that's basically it I think. Fancy stuff like downloading the
>messages that IDLE notifies you about are handled in the
>ClientFactory, right?

Perhaps by a factory, or perhaps by something else.  When I've said 
"application code" above, this is what I'm talking about - the code that 
someone else has written which uses IMAP4Client somehow in order to do 
something IMAP4 related.  In our case, offlineimap would be the 
application code. :)  It doesn't make much difference to the IMAP4Client 
implementation who or what is using it, so it could be a ClientFactory 
or another protocol or a GUI or any number of other things.
>Some things that I'm not all that clear on and could use your help to
>understand:
>How do you cancel a previously queued/scheduled callback?

It would probably make sense to be more specific here.  "Callback" might 
mean a lot of things.

Deferreds, the central callback-management API used in Twisted, don't 
directly support cancellation (though we consider adding such support 
from time to time).  Generally APIs which want to offer cancellation do 
it by some other means separate from the Deferred they return.  For 
example, a number of APIs accept a "timeout" parameter which is a form 
of cancellation.  These APIs internally use the timed call features of 
the Twisted reactor to make the operation fail if it does not complete 
within the given time frame, resulting in an "errback" on the Deferred 
(just a callback for errors).

Actually canceling an operation depends on what the operation is and how 
it's implemented.  For example, the IMAP4 protocol itself offers no 
mechanism for canceling a command which the server has already received 
and begun processing, aside from prematurely closing the connection.  So 
if you issue a FETCH, you don't have much of a way to avoid receiving 
the results.

I'm not sure with what aim you bring up cancellation, so I don't think I 
can be any more specific than this now.  Let me know if I didn't 
actually answer your question.
>How are multiple connections to the same server handled? And more
>importantly: how do you have a command in one session use a callback
>on another already-open connection?

This is simpler than it seems.  The answer is mostly what you'd expect 
if you asked it about a non-Twisted-based app.  If you want protocol 
instance A to do something to protocol instance B, you make sure A has a 
reference to B and then you have A call a method on B.  There are lots 
of approaches to making sure that reference is available, but you don't 
really need anything fancier than an attribute somewhere - using the 
factory is a common approach, since  ClientFactory sets itself as the 
"factory" attribute on each protocol instance it creates.

Hope that was helpful,
Jean-Paul