[Pkg-opencl-devel] opencl-icd selection

[giuseppe.bilotta at gmail.com]

I'm reviving an old-ish thread to express my thoughts on the ICD
installation/selection thing. Sorry for being so late to the game, I
only recently joined the mailing list.

Rebecca Palmer wrote:

> ==Defaulting to installing all ICDs?==
> 
> Currently, OpenCL-using applications Depend: on opencl-icd, which means 
> "whichever ICD apt happens to pick, often not one that actually works on 
> that hardware".
> I suggested earlier [0] that we solve this problem (at least for the 
> ~40% of users whose hardware has a free ICD [1]) by making it instead 
> mean "all ICDs unless the user explicitly chooses one", similar to the 
> video drivers:
> 
> #application/library that needs OpenCL
> #do *not* depend on any specific ICD (i.e. #739176 is not a bug), as 
> doing so prevents automatic installation of the others
> python-pyopencl Depends: libopencl1, opencl-icd
> 
> #new metapackage, and also still the virtual package
> #list the CPU implementation first, so --no-install-recommends picks it 
> opencl-icd Depends: pocl-opencl-icd | beignet | mesa-opencl-icd
> Recommends: pocl-opencl-icd, beignet, mesa-opencl-icd
> 
> #ICDs keep the Provides, so if the user does explicitly install one, the 
> system won't waste space on installing the others
> pocl-opencl-icd Provides: opencl-icd
> beignet Provides: opencl-icd
> mesa-opencl-icd Provides: opencl-icd
> nvidia-opencl-icd Provides: opencl-icd

I like this idea. My only perplexity is for the following scenario:
let's say that a user has a machine with only a discrete GPU for which
they are using the properitary driver (NVIDIA or AMD). They also did
_not_ install all the video drivers, so (among other things) they don't
have mesa installed.

Now, they install a package that Depends on opencl-icd: in this case,
installing mesa-opencl-icd or beignet will bring in all of mesa as a
dependency. Would it be possible to specify the dependencies in such a
way that GPU-related ICDs are installed only if the corresponding mesa
packages are already installed (or would be otherwise installed anyway
in this run)?

> For this to work, both ICDs and applications need to properly handle the 
> case of an ICD being installed without the corresponding hardware:
> -ICDs must report CL_DEVICE_NOT_FOUND, not crash/hang/exit.
> -Applications must then try the next platform, not assume that OpenCL 
> isn't available at all.  They should preferably look for GPUs before 
> CPUs, but as the default platform order appears to be alphabetical 
> (placing pocl last), with the current free ICDs the GPU happens to be first.

There is also one other condition: ICDs should not conflict with each
other (as is the case currently with mesa and pocl/beignet).

> Currently, one ICD (beignet #745363) and roughly half the applications 
> (pyopencl #745456, vxl #745535, erlang-cl, bfgminer, clinfo, hwloc, oce, 
> gegl) do not do this, but they all appear to be easy to fix; I have 
> filed patches for the first 3, and can do so for the others if we agree 
> that this is a good idea.
> 
> As it is likely that OpenCL is also being used with applications 
> obtained outside the Debian archive (which we hence can't fix), should 
> we patch whatever decides the platform order (ocl-icd-libopencl1?) to 
> put the working-on-this-hardware platform first, or is it reasonable to 
> say "if you're doing that, choose the right ICD manually"?

The only situation that would seriously be unfixable (at the
distribution level) is this:
* the user is using an unpatched/unpatchable application (e.g.
  proprietary or obtained outside of Debian);
* the user is using an unpatched/unpatchable ICD loader (e.g.
  amd-libopencl1);
* ICDs for unsupported hardware are enumerated before the ones for
  supported hardware by the ICD loader.

In this case, but I would say only in this case (modulo see below) the
only thing that can be done is to suggest the user to either switch to
ocl-icd-libopencl1 or to uninstall ICDs for unsupported hardware.

This, of course, assuming all OpenCL applications in Debian are fixed to
try all platforms until a working one is found, and ocl-icd is fixed to
enumerate ICD with supported hardware first.

To this end, I've extended my clinfo to also test the ICD loader
behavior in case of NULL platform/properties when doing things such as
querying for device IDs or creating contexts of specific types.

Now, the standard says that the behavior in these cases (NULL
patform/context properties) is implementation-defined. The Khronos ICD
loader (and its derivatives such as amd-libopencl1) just throw a
CL_INVALID_PLATFORM error, whereas ocl-icd does _almost_ the right
thing, in that it acts as if the default platform was specified.

The only limitations of the current ocl-icd strategy are that when no
default platform is set by the user, it picks the first one, which might
have no supporting hardware (e.g. mesa on my machine) at all, or no
supporting hardware of the specific type (e.g. beignet when a CPU
context was asked for, or pocl when a GPU context was asked for).

I think that just extending ocl-icd to have a smarter default context
selection (and possibly ICD ordering) would fix it. My idea is the
following:

* when probing for ICDs, ocl-icd should also query the number and types
  of devices available;
* ocl-icd should internally sort the ICDs by type (GPU > CPU) and number
  of devices;
* when a (default) context of a specific type is asked, it should pick
  the topmost ICD _with devices of that type_.

We should probably discuss this with the ocl-icd authors too.

> [1] Popcon installs: nvidia-opencl-icd (non-free) 2502, amd-opencl-icd 
> (non-free but replaceable by mesa-opencl-icd+pocl-opencl-icd) 1570, 
> amd-opencl-icd-legacy (non-free) 197, beignet 137, mesa-opencl-icd 38

Just a nitpick: aside from hardware support, amd-opencl-icd{,-legacy}
are _not_ 100% replaceable by {mesa-opencl-icd,beignet} +
pocl-opencl-icd, because the former allow you to create a mixed context
(a context with both a GPU and the CPU as devices), whereas the latter
forces you to use two contexts from two different platforms to use both.
Ideally, it would be nice if mesa-opencl-icd included support for CPU
devices as well (or does it, and I didn't know about it?)

Best regards,

-- 
Giuseppe "Oblomov" Bilotta