[Pkg-zfsonlinux-devel] Bug#1057018: zfs-dkms: VERIFY3 failed; PANIC at range_tree.c:435; cause all operations to hang until hard reboot

[zoyd]

Package: zfs-dkms
Version: 2.1.11-1
Severity: important
Tags: lfs upstream
X-Debbugs-Cc: j3cipgk393oz at explodemail.com

Dear Maintainer,

When doing incremental replication of encrypted datasets, `zfs send`
crashes and `zfs recv` hangs without responding to any kill signals. Any
subsequent zfs operations hang infinitely until a hard power cycle is
forced (due to reboot similarly hanging).

The following appears in kernel log:

```
VERIFY3(size != 0) failed (0 != 0)
PANIC at range_tree.c:435:range_tree_remove_impl()
Showing stack for process 299994
CPU: 1 PID: 299994 Comm: z_wr_iss Tainted: P           OE     5.9.0-5-amd64 #1 Debian 5.9.15-1
Hardware name: *REDACTED*
Call Trace:
 dump_stack+0x6b/0x83
 spl_panic+0xd4/0xfc [spl]
 ? kfpu_end+0x3f/0x70 [zcommon]
 ? zfs_btree_insert_into_leaf+0x22e/0x330 [zfs]
 ? abd_iterate_func.part.0+0x13b/0x1b0 [zfs]
 ? zfs_btree_add_idx+0xa5/0x230 [zfs]
 ? zfs_btree_find+0x5d/0x320 [zfs]
 ? zfs_btree_find+0x188/0x320 [zfs]
 range_tree_remove_impl+0xa92/0xf00 [zfs]
 ? metaslab_df_alloc+0x1c3/0x5c0 [zfs]
 ? multilist_sublist_insert_tail+0x2b/0x50 [zfs]
 metaslab_alloc_dva+0xbd3/0x14a0 [zfs]
 ? _cond_resched+0x16/0x40
 metaslab_alloc+0xcf/0x280 [zfs]
 zio_dva_allocate+0xd4/0x8d0 [zfs]
 ? irq_exit_rcu+0x3e/0xc0
 ? common_interrupt+0x77/0x130
 ? asm_common_interrupt+0x1e/0x40
 zio_execute+0x81/0x120 [zfs]
 taskq_thread+0x2df/0x520 [spl]
 ? wake_up_q+0xa0/0xa0
 ? zio_gang_tree_free+0x60/0x60 [zfs]
 ? taskq_thread_spawn+0x50/0x50 [spl]
 kthread+0x11b/0x140
 ? __kthread_bind_mask+0x60/0x60
 ret_from_fork+0x22/0x30
INFO: task z_wr_iss:765697 blocked for more than 120 seconds.
      Tainted: P           OE     5.9.0-5-amd64 #1 Debian 5.9.15-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:z_wr_iss        state:D stack:    0 pid:765697 ppid:     2 flags:0x00004000
Call Trace:
 __schedule+0x281/0x8a0
 schedule+0x4a/0xb0
 schedule_preempt_disabled+0xa/0x10
 __mutex_lock.constprop.0+0x13a/0x480
 metaslab_alloc_dva+0x413/0x14a0 [zfs]
 ? _cond_resched+0x16/0x40
 metaslab_alloc+0xcf/0x280 [zfs]
 zio_dva_allocate+0xd4/0x8d0 [zfs]
 ? zio_vdev_io_done+0x49/0x220 [zfs]
 zio_execute+0x81/0x120 [zfs]
 taskq_thread+0x2df/0x520 [spl]
 ? wake_up_q+0xa0/0xa0
 ? zio_gang_tree_free+0x60/0x60 [zfs]
 ? taskq_thread_spawn+0x50/0x50 [spl]
 kthread+0x11b/0x140
 ? __kthread_bind_mask+0x60/0x60
 ret_from_fork+0x22/0x30
INFO: task txg_sync:765752 blocked for more than 120 seconds.
      Tainted: P           OE     5.9.0-5-amd64 #1 Debian 5.9.15-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
```


There is an upstream patch available: 

The fix should be present already in bookworm-backports but is sever
enough that I believe it should be brought into stable.

Upstream issue: https://github.com/openzfs/zfs/issues/11893
Upstream patch targeting 2.1.12: https://github.com/openzfs/zfs/pull/14843

-- System Information:
Debian Release: 12.2
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.9.0-5-amd64 (SMP w/12 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages zfs-dkms depends on:
ii  debconf [debconf-2.0]  1.5.82
ii  dkms                   3.0.10-8+deb12u1
ii  file                   1:5.44-3
ii  libc6-dev [libc-dev]   2.36-9+deb12u3
ii  libpython3-stdlib      3.11.2-1+b1
ii  lsb-release            12.0-1
ii  perl                   5.36.0-7
ii  python3-distutils      3.11.2-3

Versions of packages zfs-dkms recommends:
ii  linux-libc-dev  6.1.55-1
ii  zfs-zed         2.1.11-1
ii  zfsutils-linux  2.1.11-1

Versions of packages zfs-dkms suggests:
pn  debhelper  <none>

-- debconf information:
* zfs-dkms/note-incompatible-licenses:
  zfs-dkms/stop-build-for-unknown-kernel: true
  zfs-dkms/stop-build-for-32bit-kernel: true