[pkg-go] Bug#982288: podman: Can't run caontainers - failed to connect to container's attach socket

Sam Morris sam at robots.org.uk
Mon Feb 8 11:05:54 GMT 2021 

Package: podman
Version: 3.0.0~rc2+dfsg1-2+b1
Severity: grave
Justification: renders package unusable
X-Debbugs-Cc: sam at robots.org.uk

After upgrading to podman 3, I can't run any containers any more.

    $ podman run --rm -it docker.io/library/debian:10 
    Error: failed to connect to container's attach socket: /run/user/876099160/libpod/tmp/socket/3178d20b8a3a42642dc6a7f32884df47019bc4a2a82af94fe4928b00ed3293c9/attach: no such file or directory

The directory /run/user/876099160/libpod/tmp/socket is empty.

According to unix(7), socket paths are limited to 108 bytes, but the
path in the error message is slightly longer than that:

    $ echo -n /run/user/876099160/libpod/tmp/socket/18654637587d169f834095dce40d4812378e0056936974c9b7073ba1ae767bfa/attach | wc -c
    109

Podman had a similar sounding bug a couple of years ago,
<https://bugzilla.redhat.com/show_bug.cgi?id=1641800>, but that was
fixed in podman 0.11.1.

Full debug output:

    INFO[0000] podman filtering at log level debug          
    DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm -it docker.io/library/debian:10) 
    DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
    DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.1 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/876099160/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/ipa.example.com/sam.morris/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/876099160/libpod/tmp VolumePath:/home/ipa.example.com/sam.morris/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/ipa.example.com/sam.morris/.config/cni/net.d}} 
    DEBU[0000] Reading configuration file "/etc/containers/containers.conf" 
    DEBU[0000] Merged system config "/etc/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.1 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/876099160/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/ipa.example.com/sam.morris/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/876099160/libpod/tmp VolumePath:/home/ipa.example.com/sam.morris/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/ipa.example.com/sam.morris/.config/cni/net.d}} 
    DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
    DEBU[0000] Initializing boltdb state at /home/ipa.example.com/sam.morris/.local/share/containers/storage/libpod/bolt_state.db 
    DEBU[0000] Using graph driver overlay                   
    DEBU[0000] Using graph root /home/ipa.example.com/sam.morris/.local/share/containers/storage 
    DEBU[0000] Using run root /run/user/876099160/containers 
    DEBU[0000] Using static dir /home/ipa.example.com/sam.morris/.local/share/containers/storage/libpod 
    DEBU[0000] Using tmp dir /run/user/876099160/libpod/tmp 
    DEBU[0000] Using volume path /home/ipa.example.com/sam.morris/.local/share/containers/storage/volumes 
    DEBU[0000] Set libpod namespace to ""                   
    DEBU[0000] [graphdriver] trying provided driver "overlay" 
    DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
    DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
    DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
    DEBU[0000] Initializing event backend journald          
    DEBU[0000] using runtime "/usr/bin/crun"                
    DEBU[0000] using runtime "/usr/bin/runc"                
    INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
    INFO[0000] Setting parallel job count to 13             
    DEBU[0000] parsed reference into "[overlay@/home/ipa.example.com/sam.morris/.local/share/containers/storage+/run/user/876099160/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] exporting opaque data as blob "sha256:e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] parsed reference into "[overlay@/home/ipa.example.com/sam.morris/.local/share/containers/storage+/run/user/876099160/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] exporting opaque data as blob "sha256:e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] using systemd mode: false                    
    DEBU[0000] No hostname set; container's hostname will default to runtime default 
    DEBU[0000] Loading default seccomp profile              
    DEBU[0000] Allocated lock 0 for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] parsed reference into "[overlay@/home/ipa.example.com/sam.morris/.local/share/containers/storage+/run/user/876099160/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] exporting opaque data as blob "sha256:e7d08cddf791fe3245267654331eb21b805458b3412d368018009355855044a3" 
    DEBU[0000] created container "799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6" 
    DEBU[0000] container "799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6" has work directory "/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata" 
    DEBU[0000] container "799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6" has run directory "/run/user/876099160/containers/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata" 
    DEBU[0000] Handling terminal attach                     
    DEBU[0000] overlay: mount_data=lowerdir=/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/l/ZOACAX2GHPZLNKP57FFUL6MJVO,upperdir=/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/31a1df5f3d6c2ef588196d2fbfd8880257cd626a5eca01d9731eb6036cda40e2/diff,workdir=/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/31a1df5f3d6c2ef588196d2fbfd8880257cd626a5eca01d9731eb6036cda40e2/work,context="system_u:object_r:svirt_lxc_file_t:s0:c220,c538" 
    DEBU[0000] Made network namespace at /run/user/876099160/netns/cni-7a9bd1cb-9235-02c2-4e8a-88e7e29cf688 for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] mounted container "799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6" at "/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/31a1df5f3d6c2ef588196d2fbfd8880257cd626a5eca01d9731eb6036cda40e2/merged" 
    DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/876099160/netns/cni-7a9bd1cb-9235-02c2-4e8a-88e7e29cf688 tap0 
    DEBU[0000] Created root filesystem for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 at /home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/31a1df5f3d6c2ef588196d2fbfd8880257cd626a5eca01d9731eb6036cda40e2/merged 
    DEBU[0000] Workdir "/" resolved to host path "/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay/31a1df5f3d6c2ef588196d2fbfd8880257cd626a5eca01d9731eb6036cda40e2/merged" 
    DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
    DEBU[0000] Setting CGroups for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 to user.slice:libpod:799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
    DEBU[0000] Created OCI spec for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 at /home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata/config.json 
    DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
    DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -c 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 -u 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 -r /usr/bin/crun -b /home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata -p /run/user/876099160/containers/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata/pidfile -n pedantic_lalande --exit-dir /run/user/876099160/libpod/tmp/exits --socket-dir-path /run/user/876099160/libpod/tmp/socket -s -l k8s-file:/home/ipa.example.com/sam.morris/.local/share/containers/storage/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/user/876099160/containers/overlay-containers/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/ipa.example.com/sam.morris/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/876099160/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/876099160/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6]"
    INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6.scope 
    DEBU[0000] Received: 606074                             
    INFO[0000] Got Conmon PID as 606071                     
    DEBU[0000] Created container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 in OCI runtime 
    DEBU[0000] Attaching to container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] Removing container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] Removing all exec sessions for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] Cleaning up container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] Tearing down network namespace at /run/user/876099160/netns/cni-7a9bd1cb-9235-02c2-4e8a-88e7e29cf688 for container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] Successfully cleaned up container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 
    DEBU[0000] unmounted container "799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6" 
    DEBU[0000] Container 799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6 storage is already unmounted, skipping... 
    DEBU[0000] ExitCode msg: "failed to connect to container's attach socket: /run/user/876099160/libpod/tmp/socket/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/attach: no such file or directory" 
    Error: failed to connect to container's attach socket: /run/user/876099160/libpod/tmp/socket/799c00afbed3b994eb0eafae01dcbc55d1fe659a7dac52289c2634fb20dae8c6/attach: no such file or directory

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing-debug
  APT policy: (550, 'testing-debug'), (550, 'testing'), (530, 'unstable-debug'), (530, 'unstable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.9.0-4-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_USER
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: SELinux: enabled - Mode: Permissive - Policy name: default

Versions of packages podman depends on:
ii  conmon                           2.0.20-1
ii  containernetworking-plugins      0.9.0-1
ii  crun                             0.16+dfsg-1
ii  golang-github-containers-common  0.33.1+ds1-3
ii  init-system-helpers              1.60
ii  libc6                            2.31-9
ii  libdevmapper1.02.1               2:1.02.175-2
ii  libgpgme11                       1.14.0-1+b2
ii  libseccomp2                      2.5.1-1
ii  runc                             1.0.0~rc92+dfsg1-5+b1

Versions of packages podman recommends:
ii  buildah                                           1.19.3+dfsg1-1
ii  fuse-overlayfs                                    1.3.0-1
ii  golang-github-containernetworking-plugin-dnsname  1.1.1+ds1-4+b1
ii  slirp4netns                                       1.0.1-1
ii  tini                                              0.19.0-1
ii  uidmap                                            1:4.8.1-1

Versions of packages podman suggests:
ii  containers-storage  1.23.9+dfsg1-1+b1
pn  docker-compose      <none>

-- no debconf information

More information about the Pkg-go-maintainers mailing list