[pymvpa] Optimization Results
Michael Hanke
michael.hanke at gmail.com
Mon May 11 12:34:38 UTC 2009
Hi Valentin,
I have to run in a few minutes, but I want to thank you for all your
efforts. Doing this validation stuff is not very entertaining, but
nevertheless very important. it looks like on the most important aspect
(accuracy) we are doing quite well:
> I ran a searchlight for all 8 FIR timebins for a single subject. And the
> resulting accuracy maps are almost the same (99.99%)
That is good to know!
I am not suprised by the performance of PyMVPA in terms of speed. Only
few pieces are optimized at all. Let me quote "First do it right, then ..."
However, optimization (especially for the intense stuff, such as
searchlights) is important and I am glad that you are working on it.
Even if it is not yet merged into the mainline, it will get the attention
it deserves, but these days are crazy and it will take a few weeks till
things calm down a bit.
Over the summer we will go over the Dataset baseclass and at that point
also the profiling you did will have its impact.
So far for the moment -- I am sure Yarik will comment too -- when he
woke up ;-)
Thanks again,
Michael
PS: I updated the movie -- watch for your name ;-)
http://www.pymvpa.org/history.html
On Mon, May 11, 2009 at 01:27:24PM +0200, Valentin Haenel wrote:
> Hi,
>
> when i started my Lab Rotation here in berlin 2 Months ago i was given the task
> of evaluating PyMVPA as a possible alternative to the Matlab+SPM combo. To this
> end i replicated part of the following study:
>
> http://www.ncbi.nlm.nih.gov/pubmed/19111624
>
> In particular i took the FIR betas and ran a searchlight on these. I soon
> realized that PyMVPA was quite a bit slower than Matlab, and so i sat down to
> improve things. (hence all the optimizations commits in the val/* branches on
> alioth)
>
> All the comparisons were done on the same machine:
>
> 2 x DualCore AMD Opteron 2220 (4 cores)
> 16246 MB RAM
>
> I ran a searchlight for all 8 FIR timebins for a single subject. And the
> resulting accuracy maps are almost the same (99.99%)
>
> Matlab+SPM = 22 min
>
> Here is a list of what i managed to squeeze out
>
> PyMVPA:
>
> 1. No improvements 2h 42 min
> 2. _svm.py - remove for loops in python code 2h 31 min
> 3. Searchlight index cache (multiple datasets) 2h 8 min
> 4. 2 and 3 combinded 1h 55 min
>
> 5. 4 + LIBSVM Wrapper optimization 1h 2 min
> 6. 5 + comment out deepcopy in base.py 57 min
> 7. 6 + python -O switch 53 min
>
>
> My current profiling round came up with the following:
>
> --------------------------------------------------------------------------------
>
> Sat May 9 13:11:32 2009 runprof6
>
> 2342453314 function calls (2308610267 primitive calls) in 10262.765 CPU seconds
>
> Ordered by: internal time
> List reduced from 3531 to 40 due to restriction <40>
>
> ncalls tottime percall cumtime percall filename:lineno(function)
> 553055208/534337859 1295.902 0.000 1413.098 0.000 state.py:306(__getattribute__)
> 120986219 621.773 0.000 1048.780 0.000 state.py:257(_checkIndex)
> 65194885 594.444 0.000 1539.628 0.000 state.py:682(isEnabled)
> 30235296/27355744 403.370 0.000 1132.826 0.000 state.py:377(_action)
> 107353443/96915067 313.182 0.000 502.194 0.000 state.py:1084(__getattribute__)
> 204855562 289.925 0.000 289.925 0.000 {method 'has_key' of 'dict' objects}
> 39594431 282.294 0.000 471.186 0.000 verbosity.py:505(__call__)
> 5759104 273.423 0.000 1693.267 0.000 state.py:408(reset)
> 14397760 220.419 0.000 485.896 0.000 _svm.py:179(convert2SVMNode)
> 11608194 161.626 0.000 341.870 0.000 mask.py:216(isValidInId)
> 25915986 160.144 0.000 1338.768 0.000 state.py:773(<lambda>)
> 1439776 159.463 0.000 1286.389 0.001 svm.py:125(_train)
> 2879552 157.217 0.000 418.094 0.000 base.py:1131(selectSamples)
> 1439776 149.204 0.000 532.763 0.000 _svm.py:206(__init__)
> 28795520 146.755 0.000 146.755 0.000 {mvpa.clfs.libsvmc._svmc.svm_node_array_set}
> 90912297/89472064 133.970 0.000 138.536 0.000 {len}
> 76353402 121.160 0.000 121.160 0.000 attributes.py:248(isEnabled)
> 1439780 120.549 0.000 2015.755 0.001 _svmbase.py:220(__repr__)
> 3599448 120.319 0.000 282.827 0.000 base.py:103(__init__)
> 1439776 108.342 0.000 211.624 0.000 _svm.py:91(__init__)
> 9974753 107.149 0.000 107.149 0.000 mask.py:221(getOutId)
> 11608194 104.920 0.000 121.404 0.000 support.py:306(isInVolume)
> 359944 99.851 0.000 9511.643 0.026 cvtranserror.py:117(_call)
> 28795582 95.811 0.000 193.214 0.000 attributes.py:94(reset)
> 1439776 92.656 0.000 880.621 0.001 svm.py:191(_predict)
> 54352246 90.825 0.000 90.825 0.000 verbosity.py:343(<lambda>)
> 14397771 83.953 0.000 240.313 0.000 state.py:329(__getitem__)
> 31675997 83.106 0.000 83.106 0.000 {range}
> 8278713 78.118 0.000 400.513 0.000 state.py:371(setvalue)
> 2879554 76.175 0.000 1504.840 0.001 state.py:756(_getEnabled)
> 14757770 74.728 0.000 498.424 0.000 state.py:1105(__setattr__)
> 4319352 73.530 0.000 158.611 0.000 function_base.py:900(unique)
> 1439776 72.381 0.000 72.381 0.000 {mvpa.clfs.libsvmc._svmc.svm_train}
> 1439776 72.335 0.000 592.177 0.000 splitters.py:283(splitDataset)
> 1439776 68.823 0.000 6321.527 0.004 transerror.py:1220(_precall)
> 11158265 67.908 0.000 257.093 0.000 attributes.py:231(_set)
> 39594431 66.210 0.000 66.210 0.000 verbosity.py:344(<lambda>)
> 42522592 66.051 0.000 66.051 0.000 {isinstance}
> 12957993 65.684 0.000 405.324 0.000 state.py:766(<lambda>)
> 12957993 65.623 0.000 406.477 0.000 state.py:768(<lambda>)
>
> --------------------------------------------------------------------------------
>
> So it looks like the 'Collection' class in state.py is using up alot of time, not
> because of the implementation, but because of the number of function calls. One
> of the suggestions Tiziano Zito had, was to refactor the collections class and
> maybe inherit from one of the builtin types such as Set. However we are unsure
> as to what 'Collection' class actually is supposed to do, so any hints regarding
> this would be greately appreciated.
>
>
> V-
>
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
--
GPG key: 1024D/3144BE0F Michael Hanke
http://apsy.gse.uni-magdeburg.de/hanke
ICQ: 48230050
More information about the Pkg-ExpPsy-PyMVPA
mailing list