[Piuparts-devel] [Git][debian/piuparts][master] 17 commits: Start 1.4.5 development. Use `gbp dch --since=1.4.4 --multimaint- merge` to...
Nicolas Dandrimont (@olasd)
gitlab at salsa.debian.org
Thu Nov 14 11:52:32 GMT 2024
Nicolas Dandrimont pushed to branch master at Debian / piuparts
Commits:
00d751e8 by Nicolas Dandrimont at 2024-09-24T19:12:59+02:00
Start 1.4.5 development. Use `gbp dch --since=1.4.4 --multimaint- merge` to generate a changelog for this release.
- - - - -
b89ead63 by Nicolas Dandrimont at 2024-09-27T22:31:30+02:00
dracut requires removing essential packages
A dependency on systemd-sysv > init was added to fix #1056382.
- - - - -
c3415566 by Nicolas Dandrimont at 2024-09-27T22:36:31+02:00
The package is called dracut-core, not dracut
- - - - -
f0f1b5e8 by Nicolas Dandrimont at 2024-09-27T22:41:54+02:00
All dracut* packages need the override, not only dracut-core
- - - - -
9907f43b by Nicolas Dandrimont at 2024-10-10T23:14:34+01:00
packagesdb: resolve virtual packages when detecting dependency cycles
The rust ecosystem currently has a dependency cycle going through 2
virtual packages. piuparts fails to schedule tests for these packages,
as well as any package that ends up depending on them, as their status
gets reported as `unknown`.
- - - - -
9f58e16c by Nicolas Dandrimont at 2024-10-11T17:13:30+01:00
bind-mount the minimal /dev devices if they can't be opened
When the piuparts temporary directory is mounted nodev, e.g. when it's
under the default /tmp tmpfs in current Debian sid, the minimal /dev
entries can be created with mknod, but they can't be opened. In that
case, bind-mount them.
Closes: #1084234
- - - - -
bfd35acb by Jochen Sprickerhof at 2024-10-25T18:38:41+02:00
Add --bootstrapcmd to use alternative bootstrap commands
- - - - -
45617e70 by Nicolas Dandrimont at 2024-11-14T09:16:15+00:00
Merge branch 'support_mmdebstrap' into 'develop'
Add --bootstrapcmd to use alternative bootstrap commands
See merge request debian/piuparts!70
- - - - -
37eb3270 by Helmut Grohne at 2024-11-14T11:09:01+01:00
piuparts: reexecute via unshare when run as non-root
When piuparts is run as root, it currently errors out and exits. Rather
than doing that, we now try reexecuting it inside an unshared namespace
since we recently made piuparts work when run in an unprivileged user
namespace. This enables running piuparts as non-root out of the box.
- - - - -
c6cbcebc by Helmut Grohne at 2024-11-14T11:09:01+01:00
install piuparts to /usr/bin
Since we can now run piuparts as a regular user, install it in a place
where a user can run it.
- - - - -
6fff741b by Helmut Grohne at 2024-11-14T11:09:01+01:00
suggest installing uidmap
Unprivileged piuparts only works when uidmap is installed.
- - - - -
504c8654 by Helmut Grohne at 2024-11-14T11:09:01+01:00
piuparts.1: document what it takes run piuparts as non-root
Reported-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4dad8cbe by Nicolas Dandrimont at 2024-11-14T11:09:01+01:00
Add autopkgtest for unshared piuparts
- - - - -
1e373449 by Helmut Grohne at 2024-11-14T11:09:01+01:00
attempt to fix unshared piuparts on gitlab runners
We get "Permission denied" while debootstrap tries to mount its proc
filesystem. I guess that we somehow fail the mnt_already_visible test in
fs/namespace.c. If that holds true, mounting an initial proc should
help.
- - - - -
bb303dd7 by Nicolas Dandrimont at 2024-11-14T11:09:01+01:00
skip unshare autopkgtest if --mount-proc is unavailable
The salsaci runners are running on docker, which shadows enough parts of
/proc that unshare is unable to remount it in the new namespace. Skip
the test if that's the case.
- - - - -
aa9a95df by Nicolas Dandrimont at 2024-11-14T10:28:41+00:00
Merge branch 'helmutg/feature-nonroot' into 'develop'
make it easy to run piuparts as a regular user
See merge request debian/piuparts!60
- - - - -
f58a0290 by Nicolas Dandrimont at 2024-11-14T12:21:31+01:00
Prepare changelog for piuparts 1.5.0 release
- - - - -
19 changed files:
- Makefile
- conf/piuparts-slave.sudoers
- custom-scripts/scripts/pre_remove_exceptions
- debian/changelog
- debian/control
- debian/piuparts.install
- debian/piuparts.lintian-overrides
- debian/tests/all-python-versions
- debian/tests/common.sh
- debian/tests/control
- debian/tests/smoke-test
- + debian/tests/unshare
- docs/README_pejacevic.txt
- docs/piuparts/piuparts.1.txt
- instances/piuparts.conf-template.pejacevic
- instances/piuparts.conf.anbe
- piuparts.py
- piupartslib/packagesdb.py
- slave-bin/slave_stop.in
Changes:
=====================================
Makefile
=====================================
@@ -1,5 +1,5 @@
prefix = /usr/local
-sbindir = $(prefix)/sbin
+bindir = $(prefix)/bin
sharedir = $(prefix)/share
mandir = $(sharedir)/man
man1dir = $(mandir)/man1
@@ -30,7 +30,7 @@ define placeholder_substitution
-e 's/__PIUPARTS_VERSION__/$(version)/g' \
-e 's%@libdir@%$(libdir)%g' \
-e 's%@sharedir@%$(sharedir)%g' \
- -e 's%@sbindir@%$(sbindir)%g' \
+ -e 's%@bindir@%$(bindir)%g' \
$< > $@
endef
@@ -147,8 +147,8 @@ install-master: build-master-stamp install-common
#install -m 0644 known_problems/*.conf $(DESTDIR)$(etcdir)/piuparts/known_problems/
install-slave: install-common
- install -d $(DESTDIR)$(sbindir)
- install -m 0755 piuparts $(DESTDIR)$(sbindir)/
+ install -d $(DESTDIR)$(bindir)
+ install -m 0755 piuparts $(DESTDIR)$(bindir)/
install -d $(DESTDIR)$(sharedir)/piuparts
install -m 0755 piuparts-slave $(DESTDIR)$(sharedir)/piuparts/
=====================================
conf/piuparts-slave.sudoers
=====================================
@@ -3,7 +3,7 @@
#
# The piuparts slave needs to handle chroots.
-piupartss ALL = NOPASSWD: /usr/sbin/piuparts *, \
+piupartss ALL = NOPASSWD: /usr/bin/piuparts *, \
/bin/umount /srv/piuparts.debian.org/tmp/tmp*, \
/usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \
/usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp*
=====================================
custom-scripts/scripts/pre_remove_exceptions
=====================================
@@ -66,13 +66,14 @@ case "$PIUPARTS_DISTRIBUTION" in
;;
*)
case ${PIUPARTS_OBJECTS%%=*} in
- init|\
+ dracut*|\
education-thin-client|\
grub-efi-amd64-signed|\
grub-efi-ia32-signed|\
- ltsp-client|\
+ init|\
+ ltsp-client-core-dbgsym|\
ltsp-client-core|\
- ltsp-client-core-dbgsym)
+ ltsp-client)
log_debug
# requires removal of essential packages
if [ ! -f /etc/apt/apt.conf.d/piuparts-allow-remove-essential ]
=====================================
debian/changelog
=====================================
@@ -1,3 +1,21 @@
+piuparts (1.5.0) unstable; urgency=medium
+
+ [ Helmut Grohne ]
+ * piuparts: reexecute via unshare when run as non-root
+ * install piuparts to /usr/bin
+ * suggest installing uidmap
+
+ [ Jochen Sprickerhof ]
+ * Add --bootstrapcmd to use alternative bootstrap commands
+
+ [ Nicolas Dandrimont ]
+ * bind-mount the minimal /dev devices if they can't be opened (Closes: #1084234)
+ * packagesdb: resolve virtual packages when detecting dependency cycles
+ * Add autopkgtest for unshared piuparts
+ * Script updates for dracut-related packages
+
+ -- Nicolas Dandrimont <olasd at debian.org> Thu, 14 Nov 2024 12:19:16 +0100
+
piuparts (1.4.4) unstable; urgency=medium
[ Alexandre Detiste ]
=====================================
debian/control
=====================================
@@ -41,10 +41,11 @@ Depends:
${sphinxdoc:Depends}
Recommends:
adequate,
- debootstrap,
+ debootstrap | mmdebstrap,
Suggests:
schroot,
docker.io,
+ uidmap,
Description: .deb package installation, upgrading, and removal testing tool
piuparts tests that .deb packages (as used by Debian) handle
installation, upgrading, and removal correctly. It does this by
@@ -167,7 +168,7 @@ Depends:
${misc:Depends},
${python3:Depends},
Recommends:
- debootstrap,
+ debootstrap | mmdebstrap,
Description: dependencies for running piuparts slave from git
piuparts is meant as a quality assurance tool for people who create .deb
packages to test them before they upload them to the Debian package archive.
=====================================
debian/piuparts.install
=====================================
@@ -1,2 +1,2 @@
-/usr/sbin/piuparts
+/usr/bin/piuparts
/etc/piuparts/scripts*
=====================================
debian/piuparts.lintian-overrides
=====================================
@@ -3,5 +3,5 @@ uses-dpkg-database-directly [etc/piuparts/scripts-debug-purge/post_remove_postrm
uses-dpkg-database-directly [etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x]
uses-dpkg-database-directly [etc/piuparts/scripts/post_remove_exceptions]
uses-dpkg-database-directly [etc/piuparts/scripts/pre_remove_40_find_missing_md5sums]
-uses-dpkg-database-directly [usr/sbin/piuparts]
+uses-dpkg-database-directly [usr/bin/piuparts]
debian-news-entry-has-unknown-version 0.45 [usr/share/doc/piuparts/NEWS.Debian.gz:1]
=====================================
debian/tests/all-python-versions
=====================================
@@ -13,7 +13,7 @@ cd "$AUTOPKGTEST_TMP"
create_packages
for pyvers in $(py3versions -vi); do
- test_this "python$pyvers" -X dev /usr/sbin/piuparts t.deb
+ test_this "python$pyvers" -X dev /usr/bin/piuparts t.deb
done
exit 0
=====================================
debian/tests/common.sh
=====================================
@@ -5,7 +5,7 @@ set -e
test_this() {
echo
echo "running $@"
- $@
+ "$@"
}
create_packages () {
=====================================
debian/tests/control
=====================================
@@ -1,7 +1,11 @@
Tests: smoke-test
-Depends: piuparts, debootstrap
+Depends: piuparts, debootstrap, mmdebstrap
Restrictions: needs-root
Tests: all-python-versions
Depends: piuparts, python3-all, debootstrap
Restrictions: needs-root
+
+Tests: unshare
+Depends: piuparts, uidmap, debootstrap
+Restrictions: needs-root, skippable
=====================================
debian/tests/smoke-test
=====================================
@@ -15,6 +15,9 @@ create_packages
# this should always succeed
test_this piuparts t.deb
+# this should always succeed
+test_this piuparts --bootstrapcmd="mmdebstrap --skip=check/empty --variant=minbase" t.deb
+
# it is an error if this succeeds
test_this piuparts f.deb && false
=====================================
debian/tests/unshare
=====================================
@@ -0,0 +1,40 @@
+#!/bin/sh
+
+set -eu
+
+if [ -z "$AUTOPKGTEST_NORMAL_USER" ]; then
+ echo "No normal user available, test cannot be run"
+ exit 77
+fi
+
+AUTOPKGTEST_NORMAL_UID="$(id -u "$AUTOPKGTEST_NORMAL_USER")"
+AUTOPKGTEST_NORMAL_GID="$(id -g "$AUTOPKGTEST_NORMAL_USER")"
+
+if ! grep -qx "$AUTOPKGTEST_NORMAL_UID:.*" /etc/subuid; then
+ echo "Adding a subuid allocation for $AUTOPKGTEST_NORMAL_USER"
+ echo "$AUTOPKGTEST_NORMAL_UID:1000000:65536" >> /etc/subuid
+fi
+
+if ! grep -qx "$AUTOPKGTEST_NORMAL_GID:.*" /etc/subgid; then
+ echo "Adding a subgid allocation for $AUTOPKGTEST_NORMAL_USER's group"
+ echo "$AUTOPKGTEST_NORMAL_GID:1000000:65536" >> /etc/subgid
+fi
+
+if ! unshare --user --map-auto --setuid 0 --setgid 0 --mount --pid --fork --mount-proc true; then
+ echo "Unshare failed, bailing"
+ exit 77
+fi
+
+. "$(dirname "$0")/common.sh"
+
+echo running "$0"
+
+test_this piuparts --version
+
+cd "$AUTOPKGTEST_TMP"
+
+create_packages
+
+chown "$AUTOPKGTEST_NORMAL_USER:" t.deb f.deb
+
+test_this runuser -u "$AUTOPKGTEST_NORMAL_USER" -- piuparts t.deb
=====================================
docs/README_pejacevic.txt
=====================================
@@ -103,7 +103,7 @@ This is actually done by DSA:
.. code-block:: text
# The piuparts slave needs to handle chroots.
- piupartss ALL = NOPASSWD: /usr/sbin/piuparts *, \
+ piupartss ALL = NOPASSWD: /usr/bin/piuparts *, \
/bin/umount /srv/piuparts.debian.org/tmp/tmp*, \
/usr/bin/test -f /srv/piuparts.debian.org/tmp/tmp*, \
/usr/bin/rm -rf --one-file-system /srv/piuparts.debian.org/tmp/tmp*
=====================================
docs/piuparts/piuparts.1.txt
=====================================
@@ -33,7 +33,7 @@ When processing changes files, by default, all packages in a changes file will b
:program:`piuparts` outputs to the standard output some log messages to show what is going on. If a log file is used, the messages go there as well.
-:program:`piuparts` needs to be run as root.
+:program:`piuparts` requires root rights to test packages. It does not have to be run in the initial namespace though. When running it as non-root, it'll create a new Linux namespace and rerun itself as root inside said namespace. For this to work, your user needs have an subuid range (which happens by default since a few years) and you need to install the `uidmap` package to provide setuid helpers :program:`newuidmap` and :program:`newgidmap`.
OPTIONS
-------
@@ -67,6 +67,10 @@ Options must come before the other command line arguments.
Bind-mount a directory inside the chroot.
+.. option:: --bootstrapcmd cmd
+
+ Set alternative chroot bootstrap command (including command line options). The default is ``debootstrap --variant=minbase``. Needs to be compatible with typical ``debootstrap`` command line options such as ``--arch`` and ``--include``. For ``mmdebstrap`` use: ``--bootstrapcmd=mmdebstrap --skip=check/empty --variant=minbase``.
+
.. option:: -d name, --distribution name
Which Debian distribution to use: a code name (for example ``bullseye``, ``bookworm`` or ``sid``) or ``experimental``. The default is ``sid`` (= ``unstable``).
=====================================
instances/piuparts.conf-template.pejacevic
=====================================
@@ -220,7 +220,7 @@ piuparts-command =
sudo
env PYTHONPATH=%(PYTHONPATH)s
timeout -s INT -k 5m 80m
- /srv/piuparts.debian.org/sbin/piuparts
+ /srv/piuparts.debian.org/bin/piuparts
PYTHONPATH = /srv/piuparts.debian.org/lib/python3/dist-packages
master-directory = /srv/piuparts.debian.org/master
slave-directory = /srv/piuparts.debian.org/slave
=====================================
instances/piuparts.conf.anbe
=====================================
@@ -450,7 +450,7 @@ piuparts-command =
nice
env PYTHONPATH=%(PYTHONPATH)s
timeout -s INT -k 5m 110m
- /srv/piuparts/sbin/piuparts
+ /srv/piuparts/bin/piuparts
PYTHONPATH = /srv/piuparts/lib/python3/dist-packages
master-directory = /srv/piuparts/master
slave-directory = /srv/piuparts/slave
=====================================
piuparts.py
=====================================
@@ -158,6 +158,7 @@ class Settings:
self.extra_repos = []
self.testdebs_repo = None
self.debian_distros = []
+ self.bootstrapcmd = []
self.keep_sources_list = False
self.keyring = None
self.do_not_verify_signatures = False
@@ -1186,7 +1187,7 @@ class Chroot:
options.append("--arch=%s" % settings.arch)
run(
prefix
- + ["debootstrap", "--variant=minbase"]
+ + settings.bootstrapcmd
+ options
+ [settings.debian_distros[0], self.name, settings.distro_config.get_mirror(settings.debian_distros[0])]
)
@@ -1227,22 +1228,29 @@ class Chroot:
}
for devname, devnum in chardevices.items():
devname = "/dev/" + devname
+ inner_path = self.name + devname
isdevice = False
try:
- isdevice = stat.S_ISCHR(os.stat(self.name + devname).st_mode)
+ isdevice = stat.S_ISCHR(os.stat(inner_path).st_mode)
except FileNotFoundError:
# Try creating missing devices. If that fails with -EPERM, we
# likely are in an unprivileged namespace and resort to bind
# mounting them individually.
try:
- os.mknod(self.name + devname, stat.S_IFCHR | 0o666, devnum)
- os.chmod(self.name + devname, 0o666) # Override umask
+ os.mknod(inner_path, stat.S_IFCHR | 0o666, devnum)
+ os.chmod(inner_path, 0o666) # Override umask
isdevice = True
except OSError as err:
if err.errno != errno.EPERM:
raise
# Create a regular file to serve as a mount point.
- os.mknod(self.name + devname, stat.S_IFREG)
+ os.mknod(inner_path, stat.S_IFREG)
+ if isdevice:
+ try:
+ open(inner_path, "rb").close()
+ except PermissionError:
+ # Could not open the device, the filesystem is probably nodev.
+ isdevice = False
if not isdevice:
self.mount(devname, devname, opts=["bind"])
@@ -2888,6 +2896,12 @@ def parse_command_line():
help="Directory to be bind-mounted inside the chroot.",
)
+ parser.add_option(
+ "--bootstrapcmd",
+ default="debootstrap --variant=minbase",
+ help="Set alternative chroot bootstrap command (including command-line options).",
+ )
+
parser.add_option(
"-d",
"--distribution",
@@ -3412,6 +3426,7 @@ def parse_command_line():
settings.extra_repos = opts.extra_repo
settings.testdebs_repo = opts.testdebs_repo
settings.debian_distros = opts.distribution
+ settings.bootstrapcmd = shlex.split(opts.bootstrapcmd)
settings.keep_sources_list = opts.keep_sources_list
if opts.keyring:
settings.keyring = opts.keyring
@@ -3615,7 +3630,31 @@ def main():
# check if user has root privileges
if os.getuid():
- print("You need to be root to use piuparts.")
+ path_env = os.environ["PATH"].split(":")
+ for d in ("/usr/sbin", "/sbin"):
+ if d not in path_env:
+ path_env.append(d)
+ os.environ["PATH"] = ":".join(path_env)
+ # Make debootstrap (<< 1.0.134) happy when it cannot do mknod.
+ os.environ.setdefault("container", "lxc")
+ logging.info("Running non-root. Reexecuting in a usernamespace.")
+ os.execvp(
+ "unshare",
+ [
+ "unshare",
+ "--user",
+ "--map-auto",
+ "--setuid",
+ "0",
+ "--setgid",
+ "0",
+ "--mount",
+ "--pid",
+ "--fork",
+ "--mount-proc",
+ *sys.argv,
+ ],
+ )
sys.exit(1)
logging.info("-" * 78)
=====================================
piupartslib/packagesdb.py
=====================================
@@ -488,7 +488,7 @@ class PackagesDB:
deps.append(dep)
dep_pkg = self.get_package(dep, recurse=True, resolve_virtual=True)
if dep_pkg is not None:
- more += dep_pkg.dependencies()
+ more += [self._resolve_virtual(pkg) for pkg in dep_pkg.dependencies()]
return deps
def _get_dependency_cycle(self, package_name):
@@ -503,7 +503,7 @@ class PackagesDB:
dep_pkg = self.get_package(dep, recurse=True, resolve_virtual=True)
if dep_pkg is not None and package_name in self._get_recursive_dependencies(dep_pkg):
circular.append(dep)
- more += dep_pkg.dependencies()
+ more += [self._resolve_virtual(pkg) for pkg in dep_pkg.dependencies()]
return circular
def _is_successfully_tested(self, package):
@@ -620,7 +620,8 @@ class PackagesDB:
if package["Package"] in circular_deps:
testable = True
for dep, dep_state in dep_states:
- if dep in circular_deps:
+ resolved = self._resolve_virtual(dep)
+ if resolved in circular_deps:
# allow any non-error dep_state on the cycle for testing
# (error states are handled by the error propagation above)
pass
@@ -779,6 +780,12 @@ class PackagesDB:
else:
return self._packages[package_name][header]
+ def _resolve_virtual(self, package_name):
+ if package_name in self._virtual_packages:
+ return self._virtual_packages[package_name][0]
+ else:
+ return package_name
+
def get_package_state(self, package_name, resolve_virtual=True, recurse=True):
self._compute_package_states()
if package_name in self._package_state:
@@ -790,7 +797,7 @@ class PackagesDB:
return self._package_state[package_name]
if package_name in self._virtual_packages:
if resolve_virtual:
- provider = self._virtual_packages[package_name][0]
+ provider = self._resolve_virtual(package_name)
return self._package_state[provider]
else:
return "virtual"
=====================================
slave-bin/slave_stop.in
=====================================
@@ -45,7 +45,7 @@ while pgrep --full '/usr/bin/python3 @sharedir@/piuparts/piuparts-slave' > /dev/
echo -n "$(date -u +%T) - "
pgrep --full '/usr/bin/python3 @sharedir@/piuparts/piuparts-slave' | xargs -r echo -n "slaves running: "
SLEEP=$(( $i * $i ))
- BUSY=$(ps fax | grep -v grep | grep '/usr/bin/python3 @sbindir@/piuparts' | awk '{print $NF}')
+ BUSY=$(ps fax | grep -v grep | grep '/usr/bin/python3 @bindir@/piuparts' | awk '{print $NF}')
if [ -n "$BUSY" ] ; then
# really/meaningful busy
echo
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/82404cf70d4361a02ba8dfbaaa1b8671952d502e...f58a0290b668b002ac607a98685297b90bde7c95
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/82404cf70d4361a02ba8dfbaaa1b8671952d502e...f58a0290b668b002ac607a98685297b90bde7c95
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20241114/43ed5c49/attachment-0001.htm>
More information about the Piuparts-devel
mailing list