LLVM PPA: Script source availability

[Whisperity]

On Mon, Feb 21, 2022 at 5:00 PM Matthias Klose <doko at debian.org> wrote:
>
> On 2/21/22 16:24, Whisperity wrote:
> > [Current workflow without update-alternatives and having to deal with suffixes]
> >
> > S(ysadmin): *uploads newer LLVM release, suddenly clang-15 is the
> > version available*
> > U(ser): *tries to run commands from history or from scripts that have
> > "clang-13", or purely "clang" in them*
> > U: Why are my scripts not working, it keeps saying command not found?
> > S: We updated versions, you need to replace it.
> > U: *replaces in certain places only, because honest mistake, runs
> > script, it fails after some hours of work*
> > U: What?! *<curated list of expletives aimed at sysadmin>*
> > S: *<curated list of expletives aimed at both apt/debian and the user>*
> > U: *does some more replacing, hopefully not missing anything this time*
> > *It worked.*
> >
> > [With update-alternatives]
> >
> > S: *installs new release, sets system-wide alternative to new version*
> > S: *may or may not uninstall old version*
> > U: *all scripts use "clang", "clang++", "ld.lld", whatever*
> > U: *runs their scripts or commands to produce a new build*
> > *It works. Everyone is happy.*
> >
> > The default meaning of "clang" would as expected, I would of course
> > want to stay the same, whichever version the distribution is at right
> > now.
>
> I think you are over-simplifying your preferred solution.
>
>  - a user might not be aware of where your clang, clang++ symlinks
>    are pointing to.

But I do not want them to be aware of. They have a PATH and they are
using whatever the system administrator (in this case, me) gives them
(in /usr/local right now, but preferably in /usr). If they do not like
it in particular, they are free to download whatever binary zipball,
extract it where they like, and add it to their PATH. (Or compile it
themselves.) And then "clang" will be their specific version.

>  - compilation breaks, because newer clang++ defaults to a
>    newer c++ standard version. But where does it come from?
>  - much harder to debug in bug reports which version actually
>    is used.
>  - clang/clang++ don't use shared libraries a lot, but in case
>    of a version change, these are most likely not ABI compatible.
>    You don't care for your use case, but others do.

Assuming a good enough build system (which people have already done
when they claimed that every system pulling its own weight *SHOULD*
meaningfully support CC, CXX, LDFLAGS, and all these environment
variables), the standard version is just another flag which gets
embedded into the calls generated by the compiler.

But putting that aside, there will most likely be no bug reports
related to this. Installing or choosing a different alternative (other
than what the distribution supplies, installs, and selects by default)
has to be a conscious decision by the user (or their system
administrator, if it is a different person).

If I send you a bug report right now which only contains "clang++
--blah-blah", you also cannot be sure that whatever I call "clang++"
is what you think you are supplying me with. It could be a renamed
binary, it could be something I compiled myself, it could be that I
did the update-alternatives stuff, etc. Same deal as if I say "gcc
--whatever" is broken. How will you know that it is gcc 7.5, 8.3,
10.2? And as long as people do not go around relentlessly using the
PPA (production systems most likely don't) *AND* changing their
selected alternatives (this should also be uncommon?), then the vast
majority of the bug reports of people using strictly the true and
official package sources would keep using whichever version is the
default on that system.

And the moment you are unsure about this, I believe you would ask for
"Please tell us what `clang --version`" outputs, at which point it
should be immediately apparent which version the person is using.

What I am trying to argue here is that *messing it up* is not an easy
task for the average user because they have to do conscious,
sysadmin-privileged things. And even if they do or do not mess it up,
if someone wants to malicious sends, essentially "crappy bug reports",
they can still do that.
But the tradeoff is that the "database" (see below) that someone needs
to consciously decide for changing versions is coming from the
rightful source (including an automation!) itself, as opposed to being
manually created by one human here, and another human there, who might
miss out or overlook something. (That is a personal experience, I
managed to mess up some of the symlinks at first because I symlinked
something related to clang-format or clang-query (I do not recall
exactly which), the binary itself, but not some other tool they would
also like to use from the PATH, and stuff was not working. That is
when I thought about "Yeah maybe I could actually look into what
package installs what..." )


In addition, currently even /usr/bin/clang is a symlink to
/usr/lib/llvm-XXX/bin/clang. So in case someone is building off of
using LLVM's libraries as their binary dependency, the shared object
names and all the RPATH includes a versioned path.
The dynamic loader dependencies of clang themselves also contain a
version tag in their name.

All that would change, is that the client-side executables (in
/usr/bin, which is automatically in PATH on a proper system at least)
could point to this, or that, version.

For more aspects of compatibility (but I would like to thank both of
you for these inputs because they gave me a lot to think about!),
there are two levels:

 1. Your current "command-line" doing "clang --whatever" suddenly
breaks because the underlying Clang version is backwards incompatible.
(Be it command-line, ABI, whatever.) While this is certainly a
possibility and an issue, this is the same issue as if your "gcc"
version was upgraded. (Or any other tool, we could also say "java"
here. ^(1) ) If there is no trust between system administrators and
users to notify them about an upgrade, that is not an issue on the
package or package maintainer level. Your "gcc --whatever" will also
break if I decide to suddenly do a full upgrade on the system to a
newer release of Debian.

 2. My question primarily concerns the nightly/custom PPA that houses
the latest and greatest builds. I'm trying to make this distinction in
the political domain -- although I understand the scripts might be
coming from the exact same source, which makes making this distinction
technical harder -- because once you are using a custom source
repository or PPA or whatever (read: not official), then you generally
should be expecting less (or: no) compatibility or bugfree-ness,
right?


-----

^(1) Heck, the alternatives for `editor` is even worse in terms of
compatibility. Currently, on my computer, ed, nano, mcedit, vim.gtk3
(these come from the official packages) and neovim (self-compiled and
added to the alternative set) are the options. Now from all this, I
believe only vim and neovim are *somewhat* compatible (in terms of
invocation) with each other, but definitely not with the rest. Maybe
the only compatibility that is true for all is that they open a list
of files specified in their argv. :)

This makes me feel as if somewhere along the line some of us (not
meaning you or me directly, but members of the community at large)
misunderstood the reasoning behind the existence of
update-alternatives. Which might be true, but it is divergent to argue
for "compatibility" between the use cases of update-alternatives
system, where the longer-living and more widely used examples
themselves disregard it.


-----

>
> If you are such convinced about the alternatives, and if you have
> control over all development systems, why don't you install these
> alternatives yourself? update-alternatives is not bound to packages.

That is what I was doing (see the actual executed commands in previous
mail). The problem with that is that it is not trivial what binaries
(that should be available, with the right (needed) version, in PATH)
this or that package supplies.

My train of thought was:
 1. The package making script surely knows what they build and what
outputs are produced.
 2. We can collect this info in some meaningful fashion during the build.
 3. This can generate the appropriate update-alternatives call(s) and
bake them into the package.
 4. People like me no longer have to deal with manually inspecting the
.deb files, prepping, copy-pasting, sending, etc., and running the
update-alternatives commands, on every machine, all the time,
themselves. Just the "select" mechanism which does the symlinks right.

>
> >>>> For example, someone already mentioned Java there, and indeed (taken
> >>>> from an Ubuntu 18.04.6 LTS system):
> >>>>
> >>>>> ╰─ update-alternatives --query java
> >>>>> Name: java
> >>>>> Link: /usr/bin/java
> >>>>> Slaves:
> >>>>>   java.1.gz /usr/share/man/man1/java.1.gz
> >>>>> Status: auto
> >>>>> Best: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
> >>>>> Value: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
> >>>>>
> >>>>> Alternative: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
> >>>>> Priority: 1111
> >>>>> Slaves:
> >>>>>   java.1.gz /usr/lib/jvm/java-11-openjdk-amd64/man/man1/java.1.gz
> >>>>>
> >>>>> Alternative: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
> >>>>> Priority: 1081
> >>>>> Slaves:
> >>>>>   java.1.gz /usr/lib/jvm/java-8-openjdk-amd64/jre/man/man1/java.1.gz
> >>>>
> >>>> There were a lot of changes from Java 8 to 11 on the command-line
> >>>> interface which makes the two incompatible. (Although most of those
> >>>> are not everyday-used flags.)
> >> Yeah, Matthias (who particpated to the discussions on launchpad) and I (at some point)
> >> have been maintaining openjdk packages too.
> >>
> >> It also uses a similar approach to llvm-defaults:
> >> https://packages.debian.org/buster/default-jre (and others)
>
> The alternatives system for Java comes from these days when you wanted to
> install the i386 client build on an amd64 system, mostly for compatibility with
> browser plugins (all the openjdk binary packages are Multi-Arch: same).  These
> days are gone. So maybe we should stop managing the java symlinks with
> alternatives as well.
>
> Matthias

This is side-tracky, but as far as I can see, there is no "java-8" or
"java-11" binary, but rather the thing itself is in the non-trivial
"/usr/lib/jvm/..." path. So all the people who do not know the
intricacies of how Java is installed (on Debian at least) and will
just try to run "javac" or "java" for their job or schoolwork, will
end up seeing error messages or deal with non-trivial configuration,
which makes the entire thing lose user-friendliness.

Another side-note: I do believe that the default version of GCC
changed between Ubuntu 18.04 LTS revisions at some point in the past,
either from GCC-5 to GCC-7, or from GCC-7.y to 7.5. And yet, we kept
using the "gcc" or "g++" call and things generally kept working.