python, offlineimap and wakeups (outch)

[Sebastian Spaeth]

On Wed, 09 Mar 2011 13:14:45 -0500, Ethan Glasser-Camp <glasse at cs.rpi.edu> wrote:
> On 03/09/2011 06:39 AM, Sebastian Spaeth wrote:
> > Some research revealed the a thread.join() will wake up often too see if
> > our threads are still alive. And true, invoking offlineimap with the -1
> > option makes the wakeups go down from 20 to 0.1 per second!

> Can you elaborate? Attached please see a simple program that doesn't 
> seem to trigger wakeups at all (no output from "sudo powertop -d | grep 
> python").

Yep, the issue is not join per se, but calling it with a timeout. (or in
our case, I introduced the timeout to Queue.get() in our
threadexitmonitorloop (or whatever it is called). Without a timeout, we
won't wake up, but we will also not react to ctrl-c (not sure about
SIGTTERM). With a timeout, we react to ctrl-c, but wake up every 50ms.
(modify your sample code to have a timeout in the .join() and recheck
with powertop).

This is

> I do see code in the threading.Condition.wait() method, which 
> sleeps/busy waits until it gets notified, with a comment that "if we 
> sleep the whole timeout time, we'll be unresponsive". But that only 
> happens if we join() with a timeout, and as far as I can tell, we don't 
> provide a timeout on any join() calls in offlineimap.

No, it's not in the join() call, but it's:
            thrd = exitthreads.get(True, 60)
in threadutil.py (basically, they all boil down to a lock with
timeout).

> Anyhow, that comment raises the question: Responsive to what? 
> KeyboardInterrupt? You can try with the attached program, thread.join() 
> doesn't respond to KeyboardInterrupt either. If it really is 
> Thread.join, maybe we can subclass threading.Thread and define join in a 
> different manner.

With a timeout it will react to KeyboardInterrrupt, without it won't.

This is a nice dilemma, either no ctrl-c or frequent wakeups. THe only
way I can see out of that is to use a trick like this one:
http://code.activestate.com/recipes/496735-workaround-for-missed-sigint-in-multithreaded-prog/

which uses 2 processes, one waiting for a signal like ctrl-c, and the
other processes crunching along without timeouts. When a ctrl-c comes
along the watcher process sends an os.kill(sig.SIGKILL) to the child
process. This adds some complexity, and having 2 processes might be a
bit ugly, but I don't see a way around it (unless we decide that we
don't need either ctrl-c or don't care about wakeups).

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20110310/06c0d9f3/attachment-0001.sig>