python, offlineimap and wakeups (outch)
Sebastian at SSpaeth.de
Thu Mar 10 07:20:33 GMT 2011
On Wed, 09 Mar 2011 13:14:45 -0500, Ethan Glasser-Camp <glasse at cs.rpi.edu> wrote:
> On 03/09/2011 06:39 AM, Sebastian Spaeth wrote:
> > Some research revealed the a thread.join() will wake up often too see if
> > our threads are still alive. And true, invoking offlineimap with the -1
> > option makes the wakeups go down from 20 to 0.1 per second!
> Can you elaborate? Attached please see a simple program that doesn't
> seem to trigger wakeups at all (no output from "sudo powertop -d | grep
Yep, the issue is not join per se, but calling it with a timeout. (or in
our case, I introduced the timeout to Queue.get() in our
threadexitmonitorloop (or whatever it is called). Without a timeout, we
won't wake up, but we will also not react to ctrl-c (not sure about
SIGTTERM). With a timeout, we react to ctrl-c, but wake up every 50ms.
(modify your sample code to have a timeout in the .join() and recheck
> I do see code in the threading.Condition.wait() method, which
> sleeps/busy waits until it gets notified, with a comment that "if we
> sleep the whole timeout time, we'll be unresponsive". But that only
> happens if we join() with a timeout, and as far as I can tell, we don't
> provide a timeout on any join() calls in offlineimap.
No, it's not in the join() call, but it's:
thrd = exitthreads.get(True, 60)
in threadutil.py (basically, they all boil down to a lock with
> Anyhow, that comment raises the question: Responsive to what?
> KeyboardInterrupt? You can try with the attached program, thread.join()
> doesn't respond to KeyboardInterrupt either. If it really is
> Thread.join, maybe we can subclass threading.Thread and define join in a
> different manner.
With a timeout it will react to KeyboardInterrrupt, without it won't.
This is a nice dilemma, either no ctrl-c or frequent wakeups. THe only
way I can see out of that is to use a trick like this one:
which uses 2 processes, one waiting for a signal like ctrl-c, and the
other processes crunching along without timeouts. When a ctrl-c comes
along the watcher process sends an os.kill(sig.SIGKILL) to the child
process. This adds some complexity, and having 2 processes might be a
bit ugly, but I don't see a way around it (unless we decide that we
don't need either ctrl-c or don't care about wakeups).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 197 bytes
Desc: not available
More information about the OfflineIMAP-project