exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Fri Dec 4 06:32:27 GMT 2009
On 05:45 am, dom.lobue at gmail.com wrote:
>I read over the IMAP IDLE RFC and went through the IMAP4 twisted
>library and I've sketched out a rough outline of how to implement
>IDLE. I've run into some things in Twisted however that I don't really
>understand well, and I'm hoping you can point me in the correct
Cool. That was quick. :) Before I get into things, since this thread
is likely to go into a lot of Twisted-specific details which may not be
generally interesting, if there's anyone who'd like off the cc list,
please speak up. :)
>First, just to verify my understanding: the IMAP4Client class is a
Yep. And to expand on that, instances of Protocol classes typically
have a one-to-one relationship with a connection.
>All IMAP commands are represented by at least two
>methods in the IMAP4Client class - one for what to do when the command
>is received from the server, and one for when the command is sent to
Generally, though there are some exceptions. For example, AUTHENTICATE
is implemented with one method that starts by possibly sending a
CAPABILITY (IMAP4Client.authenticate) command, then another method which
will actually send AUTHENTICATE (IMAP4Client.__cbAuthenticate), then two
more methods for dealing with the response to the AUTHENTICATE
(IMAP4Client.__cbContinueAuth and IMAP4Client.__cbAuthTLS).
Another way to look at it is like this. For each supported protocol
action, there is at least one public method on IMAP4Client to initiate
this action by sending some bytes to the server. All bytes received by
IMAP4Client from the server are parsed according to the state the client
is in, what commands are outstanding, etc. Depending on the state and
the bytes, callbacks might be invoked as a result of this, possibly
delivering the results of a protocol action initiated earlier to the
calling application code.
This doesn't disagree with what you said too much, it just re-states it
in slightly more general terms.
>Assuming this assertion to be true, the broad strokes of
>the IDLE implementation is as follows:
>-IDLE is engaged: command is sent to server turning on IDLE, schedule
>an IDLE reset in 29 minutes, and an attribute ( _IDLE_Enabled for
>example ) is set to True.
Basically, yes. One subtle point, though - since the server might
reject the IDLE command, the client shouldn't assume it has entered the
IDLE state until it receives a positive acknowledgement of the command
from the server (eg the "+ idling" line from the RFC).
>-In the command dispatcher code: checks if IDLE is enabled or not. If
>enabled, it appends IDLE to all incoming commands and sends a DONE to
>the server before any new commands are sent. (On the incoming commands
>part: what I mean is if the method originally to be called was
>"incoming_exists", instead it would go to "incoming_existsIDLE".)
This part could probably bear some elaboration. I think the question
here is how the unsolicited information should best be made available to
the application code which caused the IDLE to be issued (if I've
misunderstood what you were getting at here, let me know). One
possibility is the existing IMailboxListener interface - if you look at
the very end of the implementation of IMAP4Client, you'll find three no-
op methods, modeChanged, flagsChanged, and newMessages. These are
intended for subclasses to override and are already called by
IMAP4Client when unsolicited information is given by the server in
response to a command. It may make sense to direct data provided during
an IDLE to these callbacks, or others similar to them.
>-When IDLE is confirmed off by server: delete scheduled IDLE reset.
>And that's basically it I think. Fancy stuff like downloading the
>messages that IDLE notifies you about are handled in the
Perhaps by a factory, or perhaps by something else. When I've said
"application code" above, this is what I'm talking about - the code that
someone else has written which uses IMAP4Client somehow in order to do
something IMAP4 related. In our case, offlineimap would be the
application code. :) It doesn't make much difference to the IMAP4Client
implementation who or what is using it, so it could be a ClientFactory
or another protocol or a GUI or any number of other things.
>Some things that I'm not all that clear on and could use your help to
>How do you cancel a previously queued/scheduled callback?
It would probably make sense to be more specific here. "Callback" might
mean a lot of things.
Deferreds, the central callback-management API used in Twisted, don't
directly support cancellation (though we consider adding such support
from time to time). Generally APIs which want to offer cancellation do
it by some other means separate from the Deferred they return. For
example, a number of APIs accept a "timeout" parameter which is a form
of cancellation. These APIs internally use the timed call features of
the Twisted reactor to make the operation fail if it does not complete
within the given time frame, resulting in an "errback" on the Deferred
(just a callback for errors).
Actually canceling an operation depends on what the operation is and how
it's implemented. For example, the IMAP4 protocol itself offers no
mechanism for canceling a command which the server has already received
and begun processing, aside from prematurely closing the connection. So
if you issue a FETCH, you don't have much of a way to avoid receiving
I'm not sure with what aim you bring up cancellation, so I don't think I
can be any more specific than this now. Let me know if I didn't
actually answer your question.
>How are multiple connections to the same server handled? And more
>importantly: how do you have a command in one session use a callback
>on another already-open connection?
This is simpler than it seems. The answer is mostly what you'd expect
if you asked it about a non-Twisted-based app. If you want protocol
instance A to do something to protocol instance B, you make sure A has a
reference to B and then you have A call a method on B. There are lots
of approaches to making sure that reference is available, but you don't
really need anything fancier than an attribute somewhere - using the
factory is a common approach, since ClientFactory sets itself as the
"factory" attribute on each protocol instance it creates.
Hope that was helpful,
More information about the OfflineIMAP-project