[pkg-go] Bug#1115262: Error ... creating read-write layer... permission denied

[Ian Jackson]

Package: podman
Version: 4.3.1+ds1-8+deb12u1+b1

The tag2upload service, which uses podman via autopkgtest-virt-podman,
failed earlier today.  I don't understand the cause, but I think it
must be some kind of race.

The symptoms are as follows.

The current failure looks like this.  This happens every time we try
to "open the testbed" with autopkgtest-virt-podman:

Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: autopkgtest-virt-podman [17:47:48]: disabling init based on image label
Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: <VirtSubproc>: failure: ['podman', 'run', '--detach=true', '--volume', '/tmp/autopkgtest-virt-docker.shared
.gmd824eq:/tmp/autopkgtest-virt-docker.shared.gmd824eq', '--network=host', 'localhost/autopkgtest/debian:bookworm', 'sleep', 'infinity'] failed (exit status 125, stde
rr 'Error: creating container storage: creating read-write layer with ID "dbec61978e3ec997def1e4ae0e8a7a94729ccff325aba6dbefce69bb396614e8": open /srv/builder.tag2upl
oad.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/builder: permission denied\n')
Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280275]: [t2u-oracled tag2upload-builder-01.debian.org,1280275][2025-09-14T17:47:48] virt-server: failed with error 
exit status 12

We're using rootless podman containers.

I investigated by logging in as the service user:

tag2upload-builder at tag2upload-builder-01:~$ ls -al /srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/
total 12
drwxr-xr-x  3 tag2upload-builder tag2upload-builder 4096 Sep 11 03:55 .
dr-xr-sr-x 17 tag2upload-builder tag2upload-builder 4096 Sep 11 03:57 ..
drwx------  4             100999             100999 4096 Sep 11 03:55 builder
tag2upload-builder at tag2upload-builder-01:~$ 

Immediately predecing the first failure, the logs contain this:

Sep 14 15:30:27 tag2upload-oracle-01 using-these[1261274]: Connection to tag2upload-builder-01.debian.org closed by remote host.
Sep 14 17:47:25 tag2upload-oracle-01 using-these[841]: [t2u-oracled tag2upload-builder-01.debian.org,841][2025-09-14T17:47:25] group_leader worker=716680: died due to
 fatal signal PIPE

This was probably due to a transient network problem between two of
the tag2upload service's hosts.  The process reported there as
receiving SIGPIPE was the parent of autopkgtest-virt-podman.  I don't
think autopkgtest-virt-podman would have got any signal as a result of
this event.  It would have seen an EOF on its stdin and if it
attempted to write to its stdout it would probably have seen
EPIPE/SIGPIPE.

I conjecture that podman (or autopkgtest-virt-podman) leaves a broken,
stuck, state if things fail at the wrong moemnt.

I tried to run our image rebuild script in the hope of fixing things,
but without success:

+ podman image prune --force
Error: openfdat
/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/d3c5473984fc03ec91dcf6a31e183f1ba34c5aebacbd73d6b973c94b4a14d869/home/builder:
permission denied

I tried moving the directory out with mv (transcript below) but that
didn't work.

I then did this:

  tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$ mv dir dir.broken
  tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$ 

I then ran rm -rf dir.broken to delete what I could, to try to recover
disk space.  This seems to have allowed the system to work again.

I will file a DSA ticket to ask them to remvoe the `dir.broken`.

Ian.

2a565a034c8cc4c53501 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken
tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ rm -rf 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken
rm: cannot remove '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/sys': Permission denied
...
rm: cannot remove '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/var/cache/man/ko/cat8': Permission denied
...
tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ mkdir /srv/builder.tag2upload.debian.org/broken
tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ mv 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken /srv/builder.tag2upload.debian.org/broken/.
mv: cannot move '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken' to '/srv/builder.tag2upload.debian.org/broken/./9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken': Permission denied
tag2upload-builder at tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ ls


-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.