Bug#1036920: another problem class from /usr-merge [Re: Bug#1036920: systemd: please ship a placeholder in /usr/lib/modules-load.d/]

[Helmut Grohne]

Hi,

I fear we are not done with empty directory loss yet. This is a
technical update for future reference.

On Wed, May 31, 2023 at 11:59:58AM +0200, Helmut Grohne wrote:
> On Tue, May 30, 2023 at 11:53:00AM +0200, Helmut Grohne wrote:
> > In effect, this bug report is an instance of a bug class. I am in the
> > process of quantifying its effects, but I do not have useful numbers at
> > this time. As an initial gauge, I think it is about 2000 binary packages
> > that ship empty directories (which does not imply them to be affected,
> > rather this is to be seen as a grossly imprecise upper bound).
> 
> I did some more analysis work here and have to admit that I know my data
> model has a weakness that may result in false negatives. I'd have to do
> a complete reimport of packages and eventually will, so for now I'm
> dealing with incomplete data here. I note that content indices do not
> cover empty directories, so you really have to download loads of .debs
> to find these.

We're much further in understanding the problem now.
> Anyway, to gauge the problem, we're effectively looking for a
> combination of packages A and B such that:
> 
>  * A ships an empty directory.
>  * That empty directory is a path affected by aliasing (either in /usr
>    or /).
>  * B also ships that directory (e.g. non-empty) in the "other"
>    representation of that path.

This earlier representation is incomplete. Moving a file from / to /usr
(which currently is prohibited by the moratorium) as part of a simple
package upgrade also triggers the loss behaviour.

In any case, https://subdivi.de/~helmut/dumat.yaml now reports the
affected instances (both when multiple packages are involved and upgrade
scenarios) and it's not that many.

> So yeah, this bug class is clearly not one to panic about. As we move
> files from / to /usr, I expect this bug class to gain more occurences. I
> am not aware of a generic solution and it seems diversions won't cut it.
> If you can propose any generic workaround or recipe for this situation,
> I'm all ears. The placeholder file sounds ugly, but might work.

We've progressed somewhat since. At this time, my favourite mitigation
is deleting affected directories and I've sent a number of patches for
applicable situations. Unfortunately, some empty directories exist with
reason and we need another mitigation for them.

The options seem to be as follows:

# M9

While dpkg does not allow diverting a directory, you can trick it into
doing that by temporarily moving the directory out of the way. So
diversions are an option here though they're really ugly.

# M17

A way to not loose these directories is to keep them in both aliased and
canonical location. While this works fairly reliably (until we attempt
to remove the aliased location), it is incompatible with other
mitigations such as shipping the aliasing symlinks in a package (M11).

# M20

We can also accept the temporary loss of these directories and restore
them using maintainer scripts. In upgrade scenarios, this is trivial as
the postinst script runs after the loss event. In the package removal
scenario, the package being removed can activate an interest-noawait
trigger to restore the directory.

I've implemented and attached a prototype for this approach.

# M21

The obvious way to avoid loss of empty directories is to make them
non-empty. Adding a placeholder file is sufficient here.


All of these mitigations can be selected per-occasion (though selecting
M17 once precludes a number of other ones). I don't think there is a
clear winner here. We can leave the choice of mitigation to the affected
maintainers.

Rough outline on a per-package basis:
 * cockpit-tests: deleted #1043322
 * firmware-b43-installer: /lib/firmware/b43 triggerless M20
 * firmware-b43legacy-installer: /lib/firmware/b43legacy triggerless M20
 * fwupd: delete #1041752
 * gretl: deleted #1041835
 * lib32lsan0 : deleted #1042482
 * libjte-dev: deleted #1041753
 * libmpeg3-dev: deleted #1041756
 * libswe-dev: delete #1041757
 * libx32lsan0: deleted #1042482
 * netplan-generator: multiple, TBD
 * openrc: TBD, maybe triggerless M20?
 * pcp: delete #1041754
 * pkg-config/pkgconf/pkgconf-bin: TBD, maybe M21?
 * printer-driver-foo2zjs: /lib/firmware/hp triggerless M20
 * python3-expeyes: deleted #1041755
 * systemd,udev: multiple, some deleted, others TBD

The takeaway here is that the majority of cases are handled via
deletion and we'll be finding solutions with the maintainers of those
four packages.

If you want to help mitigate these instances, you may add placeholder
files or delete empty directories at any time. Adding diversions,
maintainer scripts or triggers is something we need to consider at a
later stage.

Helmut
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.sh
Type: application/x-sh
Size: 868 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20230815/b0796663/attachment.sh>