[Pkg-raspi-maintainers] Proposal: pythonize recipe generation

Sat Nov 20 14:54:36 GMT 2021

Hi,

Cyril Brulebois <cyril at debamax.com> (2021-08-28):
> The goal was *not* to propose something smart, elegant, fancy, that
> would require some advance knowledge of one particular language, or
> designing some DSL, or something else. It was basically trying to make
> the existing approach more reliable.

I'd like to nudge this proposal a bit, especially after the commit
adding bookworm builds, which re-introduces issues that were addressed
already… hard to spot in 68ebe5c3858d02b5b7b0de60ad56b29481484ca7 though!

I'm attaching a big-yaml.diff comparing all yaml recipes built from
master against those built from the pythonize branch.

Excerpts follow:

* for all recipes:

    -    - raspi-firmware
         - linux-image-rpi
    +    - raspi-firmware

I'm going to start with a little “I'm sorry”, but I thought it would
make sense to group all firmware packages together after the linux-image
package, so that's a little noise appearing in all recipes. That happens
for the raspi-firmware or the raspi3-firmware package depending on the
build.

* for all recipes:

       - create-file: /etc/apt/sources.list
    -    trailing-newline: '1'
    -    contents: |
    +    contents: |+
    […]
    +

This fixes a long-standing misconception around the trailing newlines…

* for all bookworm recipes:

       deb http://deb.debian.org/debian bookworm main contrib non-free
+      deb http://security.debian.org/debian-security bookworm-security main contrib non-free

This fixes a missing security suite for all bookworm builds (whether
that's immediately useful is another story, I don't see why an existing
suite should not be configured right away, so that users don't have to
remember they need to add it on their own).

* for all bookworm recipes:

       - chroot: /
         shell: |
           rm -f /etc/machine-id /var/lib/dbus/machine-id
    -      touch /etc/machine-id

This fixes an extraneous 'touch' case that was documented as being
needed only for buster… see 26a7de63b0bb3de1b5d0c4d0529240721c322dbb in
master vs. 592c0df22e3edf4ff416946473b1862a5c1627b9 in pythonize.

* for raspi_4_buster recipe:

    -      # Backports are _not_ enabled by default.
    -      # Enable them by uncommenting the following line:
    -      # deb http://deb.debian.org/debian buster-backports main contrib non-free
    -      deb http://deb.debian.org/debian/ buster-backports main contrib non-free # raspi 4 needs a kernel and raspi-firmware newer than buster's
    +      # raspi 4 needs kernel and firmware newer than buster's
    +      deb http://deb.debian.org/debian/ buster-backports main contrib non-free
    +

This fixes some long-standing schizophrenia as to whether we're enabling
or not enabling backports… Granted, buster might go away at some point,
but the same can happen for whatever build requires backports.

* for raspi_4_buster recipe:

    -      sed -i s#raspi-firmware#raspi-firmware/buster-backports# ${ROOT?}/etc/systemd/system/rpi-reconfigure-raspi-firmware.service

There were several attempts at getting the right handling for firmware
packages:
 - 89ef8242851a201426c4df36efa89aa43646a06a
 - 42f88e3d308c940487c8c6d2934e898b0f10f949 (fix for syntax error)
 - feb61f3361ea2db92b5f12a68f1365cbee2799b4 (trying to sed in the .service config)
 - 3058cf8e8289c9dba69f635384375d50972382db (trying to patch the right file)
 - 7aed761507b2ca7d04aee1fb75b662522922a587 (trying to dodge a syntax error)

whereas the sed maze can be replaced with the following logic:
 - any buster build pulls raspi3-firmware and requires a firmware fixup
   accordingly;
 - except the buster 4 build which pulls raspi-firmare from backports
   and doesn't require that;
 - and any other build pulls raspi-firmware and doesn't require that.

That's the fix_firmware variable in the pythonize branch…

Another advantage we gain by not duplicating recipes all over the place
is that we can fix stuff once and for all, see:
 - 3f9e671fedd54b4a332e5a93e156489600203fad in boot-consistency, which
   required a follow-up 2b2bb9d6d78404c53bc1ccd31e8094aa1acb2921 once
   that branch was finally merged into master, since bookworm builds
   were added inbetween;
 - versus cab748ab0c63913dbef44e33dece9df48c52b7e2 in pythonize…

We could also finally get rid of the rpiN (buster) vs. rpi_N (all
others) disparity, which is nowhere problematic, but underlines a
little more how many disparities we've been carrying over…

This approach also makes it possible to add something for various
families in a single shot, see c7f08c54cbeed27b876b45d1460f53a27a771376
in the python+compute-modules branch…

I'm happy to address any issues you'd have with this approach, I'm
really eager to getting more reliability and consistency across all
builds…

For the avoidance of doubt: I don't *need* to push this approach onto
everyone else. I'm already forking recipes to address my customer's
needs anyway. But I'd love to contribute anything interesting (e.g.
CM3/CM4 support) into Debian directly. The current sed&grep-in-Makefile
approach doesn't make it easy. And I hate to see brain cycles wasted,
fixing the same bugs over and over again.

Cheers,
-- 
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: big-yaml.diff
Type: text/x-diff
Size: 21389 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-raspi-maintainers/attachments/20211120/7f007aaf/attachment-0001.diff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-raspi-maintainers/attachments/20211120/7f007aaf/attachment-0001.sig>