After a bit of further debugging it seems that the problem occurs when using the nvidia-drm.modeset=1 kernel parameter. Without it the kernel warnings don't appear. Unfortunately then things like nvidia-vaapi-driver don't work (which worked normally with it set).