[Debian-med-packaging] Function multiversioning for Phylogeny tools [Fwd: Help to build library in generic form, avx and sse3]

[Andreas Tille]

Hi Alexandros,

recently I asked you about suggestions what else besides RaXML and ExaML
might be interesting packaging targets for the Debian Med team.  I
somehow lost the message (I'm archiving now mine here on the public
Debian Med developers list ;-)) but you suggested

  ExaBayes: https://github.com/aberer/exabayes
  RogueNaRok: https://github.com/aberer/RogueNaRok
  mPTP: https://github.com/Pas-Kapli/mptp

by their influence.  I had a look into the first two.  The second was
easy and thus a roguenarok package is now in the Debian New queue for
inspection by ftpmaster.  ExaBayes is a bit more complex and I'm in
discussion with Andre Aberer (in CC) how to proceed[1].

When trying to package libpll (independently whether it will be used to
link ExaBayes or not in the final package) I stumbled about the problem
that in contrast to the documentation no generic library (without sse3
and avx is created) and asked for help on the usual Debian channels.

I'm quoting below the most interesting answer which is probably
interesting for several tools of your lab since I remember some past
discussion about RaXML where also different binaries are created which
might not be only confusing for users but also could lead to situation
where it is out of control of the user what version is choosen.

I observed it in wrappers like biopython, bioperl and other suites that
in all cases only the simple version is called and not the hardware
enhanced versions are choosen.  I tried to work around this situation by
providing a wrapper in /usr/bin/DefaultNameOfTheTool which checks the
hardware and redirects the command to the proper executable but finally
that's a hack I'd like to get rid of rather sooner than later.

So I'd like you to read the mail below about function multi-versioning
and consider to take this technique over into your source code to
support all users with sufficient hardware profiting from the maximum
speed they could gain from their hardware.  If you want me to open
issues in Github for the single projects I can do so but at first I
wanted to make you aware about this in general since it is not only a
single software affected and you might like to coordinate things in your
lab.

Kind regards

        Andreas.

[1] https://github.com/aberer/exabayes/issues/4

----- Forwarded message from Christian Seiler <christian at iwakd.de> -----

Date: Tue, 14 Mar 2017 16:03:01 +0100
From: Christian Seiler <christian at iwakd.de>
To: Andreas Tille <andreas at an3as.eu>, Debian Mentors List <debian-mentors at lists.debian.org>, Debian Med Packaging Team <debian-med-packaging at lists.alioth.debian.org>
Subject: Re: Help to build library in generic form, avx and sse3

On 03/14/2017 03:46 PM, Andreas Tille wrote:
> I've started packaging Phylogenetic Likelihood Library[1].  Since it
> makes heavy use of amd64 features it comes with specific support of AVX
> and SSE3.  My plan is to provide binary packages amd64 only named
> libpll-avx1 and libpll-sse3-1 with the according features plus a generic
> library libpll-generic1 for all architectures.  Upstream supports the
> creation of separate avx and sse3 libs out of the box but I failed to
> create the generic version.  So I have two questions:
> 
>   1. Could anybody please have a look at the automake stuff to
>      enable the build of the generic lib in addition to the other
>      two.  I tried several switches but failed. :-(
> 
>   2. What do you think about the plan to support specific hardware
>      features in separate binary packages?

GCC from version 6 (which is in Debian Stretch) supports function
multi-versioning (and GCC from 4.8 onwards, which is even in Jessie,
supports a subset of that), which allows you to do the following:

 - have a function with generic C/C++ code be compiled multiple
   times in different variants, and have the most optimal variant
   be selected at runtime (requires GCC 6),

   e.g.

   __attribute__((target_clones("avx2","sse3","default")))
   double foo(double a, double b) {
     return a + b;
   }

 - manually write different versions of the function and mark
   them accordingly (requires GCC 4.8)

   __attribute__((target("default")))
   double foo(double a, double b) {
     return a + b;
   }

   __attribute__((target("sse3")))
   double foo(double a, double b) {
     SOME_FANCY_SSE3_CODE;
   }

   __attribute__((target("avx2")))
   double foo(double a, double b) {
     SOME_FANCY_AVX2_CODE;
   }

So from a purely technical perspective I think the best solution
would probably be to work with upstream to allow them to support
FMV properly - and then you only need to compile a single library
version that will work everywhere, but will select the optimal
algorithm depending on the machine it's run - win/win.

Further reading:
https://lwn.net/Articles/691932/

Regards,
Christian


----- End forwarded message -----

-- 
http://fam-tille.de