[Pkg-exppsy-pymvpa] model selection and other questions

[Emanuele Olivetti]

Yaroslav Halchenko wrote:
> ...
> indeed... nice discussion should help - as I suggested - may be we can
> make voice conference some time next week? I am still waiting if Anton
> follows up on that thread in scipy mailing list
>
>   

Just emailed my pointers privately to you and Michael.

>
>> It was cross validation, as provided by libsvm.
>> Once you have
>> a function to evaluate the quality of your hyperparameters then
>> it is just matter of optimize it.
>>     
> indeed it is a function call which returns point-estimate of some value,
> but it is not a function per se - ie you cant (or it is computationally
> way too demanding) get reliable estimate of
> its gradient/Hessian for efficient optimization. What would be cool is
> to have analytically derived criterion -- there are some upper-bound
> generalization performance estimates for SVMs and SMLR. That would be
> really cool to see if optimization based on them would be fruitful.
>
>   

:) indeed It worked (with some exceptions) and it was clearly slooow.
GPR and GPC have analytical criteria so they are clearly faster.
But I guess any learning method have fast criteria if you digg
in the literature.


>> At that time I used scipy.optimize but it seems somewhat unstable on
>> my datasets. Dmitrey (OpenOpt's author) suggested the Shor's
>> r-algorithm ("ralg") for my setting.  I tried it on GPR (nice result!)
>> and hope to test on SVR when possible.
>>     
> Thanks for sharing the experience -- we should give them a spin too but
> lots of changes should happen first to make that viable  -- for
> instance, now considerable amount of 'training' of SVMs is taken by
> transforming data into 'library' space.

I don't get the point, but I'll catch it eventually, since I'm playing
with pyMVPA now.

Bye,

Emanuele