[Pkg-zfsonlinux-devel] Bug#1089488: Bug#1089488: zfs-dkms: module build fails for kernel: 6.12.3-amd64 (x86_64)

[Craig Sanders]

On Sat, Dec 07, 2024 at 03:44:43PM -0600, Steve M. Robbins wrote:
> Package: zfs-dkms
> Version: 2.2.6-2
> Severity: important
> Tags: upstream
> 
> [...] I suspect an upstream fix is required.

Yes, this is quite common with zfs (and other out-of-tree kernel
modules, e.g. the proprietary nvidia driver).

Every so often, the linux kernel changes something and becomes incompatible
with the ZFS module.  It takes some time for a) the zfs devs to update zfs
to be compatible with the new kernel and b) for the updated ZFS module to
be packaged for debian or for any patches to be back-ported to the current
zfs-dkms module.

In this particular case, ZFS 2.2.6 is incompatible with kernel 6.12.  You'll
need to wait for the upcoming ZFS 2.2.7 to be released and packaged for
debian.  See:

   https://github.com/openzfs/zfs/pull/16720

(Alternatively, the zfs-dkms devs may backport the 6.12 compatibility patch
from https://github.com/openzfs/zfs/pull/16793 like they did for 6.11
compatibility some time ago. probably depends on when 2.2.7 is going to be
released and whether it's worth the effort for a temporary soon-to-be-obsolete
fix)

If you use ZFS (or ANY other non-standard kernel module, including other dkms
modules) then it is strongly recommended that you **DO NOT UPGRADE YOUR KERNEL
UNTIL YOU HAVE VERIFIED THAT THE NEW KERNEL WORKS WITH THE MODULE(S) YOU
NEED**.

This can not be emphasised enough - if you need the module to boot (e.g.
because you have a zfs root fs) then compiling a kernel with an incompatible
version of the module can render your system incapable of booting.


If you use the packaged debian kernel images & headers, then use apt-mark to
put them on hold so they don't auto-upgrade.

    apt-mark hold linux-image-amd64 linux-headers-amd64

When there's a new kernel you want to install and you're absolutely certain
that it is compatible with zfs-dkms and any other non-standard modules you
need, then upgrade the kernel and remember to hold it again afterwards. e.g.
with:

   apt-get install linux-image-amd64 linux-headers-amd64 ; apt-mark hold linux-image-amd64 linux-headers-amd64

If you compile your kernels by downloading them from kernel.org or wherever
then just don't upgrade until you've verified that the new kernel works with
the version of zfs you're using.

You can keep up to date with ZFS versions and bug reports and other info at

   https://zfsonlinux.org/

and

   https://openzfs.org/

and

   https://github.com/openzfs/zfs/issues



FYI, all this has been the case since basically forever and is unlikely to
ever change (linux kernel devs do not care about out-of-tree drivers and
expect driver developers to keep up with any changes in the kernel - and
they're not wrong about that).

I've been doing this (holding off on kernel upgrades until I know they're
going to work) for well over 10 years with zfs-dkms and, on some of my
systems, even longer with nvidia-kernel-dkms.  If you don't want to break your
system with incompatible kernel upgrades, I very strongly recommend you do the
same.

Bigger, newer version numbers aren't always better. Especially if they don't
work for your particular circumstances.

A working and bootable kernel is far better than a shinier, newer kernel that
doesn't boot and doesn't work with the modules you need.


PS: If you're not sure that a new kernel is going to be compatible, I
recommend running a VM with the dkms modules you need and testing all kernel
upgrades and all zfs-dkms package upgrades on the VM before you upgrade your
real systems.  A broken VM is less of a problem than a broken system and much
easier to fix, especially if the VM is on a ZVOL (or QCOW2 file, or a raw file
on a zfs dataset) and you snapshot it before testing the upgrade.  I have a
debian VM called "ztest" with an ext4 rootfs and a couple of small file-based
zpools for exactly this purpose. Test that it compiles, test that it boots,
and test that any zfs pools on the VM actually work - e.g. by rsyncing lots of
data to them and by scrubbing them.

In fact, it's a good idea to test not just zfs & kernel upgrades but all major
upgrades (e.g. "apt-get dist-upgrade") on a testing VM before applying them to
any real systems. Back in the dim dark past before KVM, I used to use spare or
"sacrificial" systems for this kind of thing but virtual machines are better
and far less hassle.