[PATCH] Remove unneeded workaround for Darwin
vbeffara at ens-lyon.fr
Tue Mar 1 15:10:00 GMT 2011
SS> There is a clumsy workaround for Darwin that chunks reads into
SS> 8kb blocks to avoid huge memory allocations.
I am the author of the workaround - sorry for the clumsiness !
SS> First, this fix should not only be required but on FreeBSD2.6
SS> too (see http://bugs.python.org/issue3531). Second, decent
SS> python versions (I checked 2.6) already chunk in the SSL case
SS> anyway, so there is no need to do that again. Remove that level
SS> of indirection.
That's probably fine then. Initially I only patched the non-SSL version,
and that was not enough, but possibly things have changed since.
SS> http://evanjones.ca/python-memory.html claims that this problem
SS> has been fixed since python 2.5, so we might consider removing
SS> the workaround completely even for the non-SSL case.
It is most definitely _not_ fixed on the version of python 2.5 shipped
by Apple in mac os 10.6 (which is 2.5.4 i.e. not the very last of the
branch). It is fixed in python 2.5.5 (the one in Fink) and in python 2.6
(the one coming with the system i.e. 2.6.1) as well.
By the way, 'python' now defaults to version 2.6 in snow leopard, so I
guess the problem will solve itself anyway eventually.
vincent at elrond ~/Desktop > /usr/bin/python2.5 bug.py
python2.5(40154) malloc: *** mmap(size=10489856) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "bug.py", line 9, in <module>
s = f.read(BLOCKSIZE)
vincent at elrond ~/Desktop > /usr/bin/python2.6 bug.py
vincent at elrond ~/Desktop > cat bug.py
BLOCKSIZE = 10*1024*1024
data = 
for i in range(10000):
s = f.read(BLOCKSIZE)
assert len(s) == 0
Not sure about previous versions of the OS, but removing the patch would
effectively mean dropping python 2.5 compatibility on the mac, at least
until apple uprades it. Which if you ask me is 100% fine given that 2.6
ships with the system, but maybe someone will disagree.
SS> Increase the chunk size on Mac from 8kb to 64kb. Even Macs should be
SS> able to take that amount of memory usage nowadays.
Now _that_ is totaly fine, the problem was never the size of the window
but the number of calls to read() in the first place. Apparently with no
hint, read() ended up reading character by character, many times, but
with a largish buffer anyway, which was realloc()ed after the read - and
that is a noop on darwin, hence the crash.
| | UMPA - ENS Lyon | Mél: vbeffara at ens-lyon.fr |
| Vincent Beffara | 46 allée d'Italie | Tél: (+33) 4 72 72 85 25 |
| | 69364 Lyon Cedex 07 | Fax: (+33) 4 72 72 84 80 |
More information about the OfflineIMAP-project