[Piuparts-devel] [Git][debian/piuparts][feature/gitlab-ci] 41 commits: custom-scripts/scripts/pre_test_root_password: fix grammar in comment
Nicolas Dandrimont (@olasd)
gitlab at salsa.debian.org
Wed May 24 14:37:35 BST 2023
Nicolas Dandrimont pushed to branch feature/gitlab-ci at Debian / piuparts
Commits:
24e89b75 by Holger Levsen at 2022-02-02T10:12:37+01:00
custom-scripts/scripts/pre_test_root_password: fix grammar in comment
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
190399be by Holger Levsen at 2022-02-02T10:13:31+01:00
custom-scripts/scripts/pre_test_root_password: further improve grammar in comment
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e7538053 by Luca Boccassi at 2022-07-13T10:36:42+01:00
piuparts: detect files moving between / and /usr on upgrade
The TC issued a recommendation to avoid moving files between
bin|sbin|lib* and /usr/bin|sbin|lib* for the Bookworm cycle.
Detect such moves and raise an error.
For more details see:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
- - - - -
4489752a by Luca Boccassi at 2022-07-13T10:36:42+01:00
piuparts: detect files moving between / and /usr on dist-upgrade
Same as the check on package install/upgrade, but across distribution
dist-upgrades.
- - - - -
59c00b8b by Luca Boccassi at 2022-07-13T10:36:42+01:00
p.conf: enable --warn-on-usr-move for bullseye/bookworm
- - - - -
92c1faff by Nicolas Dandrimont at 2022-07-13T20:06:49+02:00
Merge branch 'usrmerge' into develop
piuparts: detect files moving between / and /usr on upgrade
See merge request https://salsa.debian.org/debian/piuparts/-/merge_requests/37
- - - - -
98b4fdcd by Luca Boccassi at 2022-07-17T20:17:08+01:00
custom_scripts: ensure usr-is-merged/usrmerge does not fail the migration test
The new usr-is-merged metapackage intentionally fails to install
on systems that are not merged-usr, unless a specific override
flag is passed.
usrmerge does a live-upgrade of the chroot, which piuparts does not
expect.
Use the skip flag, so that migration is not blocked.
Debootstrap will start using this flag once the package is
available in testing. For details see:
https://sources.debian.org/src/usrmerge/27/debian/README.Debian/#L66
and:
https://lists.debian.org/debian-ctte/2022/07/msg00019.html
- - - - -
c3352499 by Nicolas Dandrimont at 2022-07-17T22:06:03+02:00
Merge branch 'usr-is-merged' into 'develop'
custom_scripts: ensure usr-is-merged does not fail the migration test
See merge request https://salsa.debian.org/debian/piuparts/-/merge_requests/38
- - - - -
67a85a4f by David Steele at 2022-11-02T13:14:39-04:00
Add Stable Release tasks to README_pejacevic.txt
... to update summary.json for the new distribution environment.
- - - - -
3331b140 by Nicolas Dandrimont at 2023-01-11T11:35:00+01:00
piuparts-slave: run isort
- - - - -
ad7e30b3 by Nicolas Dandrimont at 2023-01-11T11:59:39+01:00
piuparts-slave: replace pipes.quote with shlex.quote
- - - - -
b9ca98d1 by Nicolas Dandrimont at 2023-01-11T11:59:52+01:00
piuparts.py: run isort
- - - - -
fc0047c9 by Nicolas Dandrimont at 2023-01-11T11:59:52+01:00
piuparts.py: replace pipes.quote with shlex.quote
- - - - -
2cfae2aa by Nicolas Dandrimont at 2023-01-11T12:00:00+01:00
test_piuparts: replace mox3 with unittest.mock
Closes: #1028453
- - - - -
2fd10804 by Nicolas Dandrimont at 2023-01-11T12:05:53+01:00
Migrate lintian overrides to pointed hints
- - - - -
cd45cc8f by Nicolas Dandrimont at 2023-01-11T13:19:53+01:00
Simplify docs install in Makefile
- - - - -
224e9beb by Nicolas Dandrimont at 2023-01-11T13:25:39+01:00
Use sphinxdoc dh plugin
- - - - -
0ac90edf by Nicolas Dandrimont at 2023-01-11T13:25:39+01:00
Update changelog
- - - - -
fbca117e by Nicolas Dandrimont at 2023-01-11T13:31:48+01:00
Release to unstable
- - - - -
5b5a213d by Holger Levsen at 2023-01-11T15:57:37+01:00
Start 1.1.7 development.
d/changelog entries will be written on release
using the git commit messages.
Use 'gbp dch --since debian/1.1.6'
to write d/changelog entries since that last release.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1190e637 by Holger Levsen at 2023-01-11T15:58:23+01:00
Bump standards version to 4.6.2, no changes needed.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
320b7cbb by Nicolas Dandrimont at 2023-01-11T16:15:00+01:00
Makefile: use dpkg-parsechangelog -S instead of sed
- - - - -
f5dc1171 by Holger Levsen at 2023-01-18T11:25:49+01:00
Add debian/source/lintian-overrides to override a bunch of very-long-line-length-in-source-file warnings
I fail to understand why the 4th one isn't working.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
c08f7a71 by Holger Levsen at 2023-01-18T12:32:10+01:00
d/control: fix cute-field warning, thanks lintian.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
838f800f by Holger Levsen at 2023-01-18T13:23:50+01:00
Bump standards version to 4.6.2, no changes needed.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
d4c2447a by Holger Levsen at 2023-01-18T13:37:13+01:00
d/piuparts.lintian-overrides: add debian-news-entry-has-unknown-version 0.45 due to trimmed d/chanelog in binary packages
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
bb6dd5b1 by Holger Levsen at 2023-01-18T13:43:42+01:00
Drop unused d./piuparts-(master|slave).lintian-overrides
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
25138b9b by Holger Levsen at 2023-01-18T15:50:17+01:00
d/control: remove version constraints on the depens on python3-debianbts, debootstrap amd debsums as they are met since (at least) buster
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
402a97af by Holger Levsen at 2023-01-18T20:58:09+01:00
piuparts.py, tests and docs: handle new "non-free-firmware" section in Debian. Closes: #1028661.
(Strictly speaking this is only used for bookworm, unstable, experimental and
later suites like trixie, but specifying which suite has which sections is
out of scope for piuparts itself.)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
8c272296 by Holger Levsen at 2023-01-18T22:45:24+01:00
release as 1.1.7
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4ba9949a by Holger Levsen at 2023-01-18T22:53:30+01:00
Start 1.1.8 development.
d/changelog entries will be written on release
using the git commit messages.
Use 'gbp dch --since debian/1.1.7'
to write d/changelog entries since that last release.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
82960e1e by Holger Levsen at 2023-01-19T00:18:07+01:00
fix speling error in old chanelog entry..
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
956829be by David Steele at 2023-02-08T12:15:37-05:00
Doc - don't include oldoldstable in global summary
- - - - -
0463b1e4 by Cyril Brulebois at 2023-03-15T20:13:35+01:00
piuparts(1): fix syntax for long options.
There are two kind of changes, one simply dropping the extraneous dash,
e.g.:
-*-*-allow-database*::
+*--allow-database*::
And the other that drops the extraneous dash as well, but also inserts a
separator before the =value part, e.g.:
-*-*-arch*='arch'::
+*--arch*\ ='arch'::
This is documented as a known gotcha:
Separation of inline markup: As said above, inline markup spans must
be separated from the surrounding text by non-word characters, you
have to use a backslash-escaped space to get around that.
Source: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#gotchas
Additionally, a missing '*' is fixed for the -p option.
- - - - -
10d9a216 by Cyril Brulebois at 2023-03-15T20:59:48+01:00
piuparts(1): fix syntax for bullet lists.
Turn leading '.' into '*', and delete the '+' before the note. That
doesn't seem to be supported at the moment:
[…]/docs/piuparts/piuparts.1.txt:28: WARNING: Bullet list ends without a blank line; unexpected unindent.
Let's keep it simple instead of trying to add some admonition inside a
bulleted list.
- - - - -
891436bf by Nicolas Dandrimont at 2023-03-15T22:50:26+01:00
piuparts(1): fix a lot of ReStructuredText markup issues
- - - - -
19d05e64 by Evgeni Golov at 2023-04-02T19:09:32+00:00
explicitly state that piuparts needs direct access to docker storage
- - - - -
6182fd58 by Nicolas Dandrimont at 2023-04-02T19:51:41+00:00
Merge branch 'evgeni-develop-patch-18078' into 'develop'
explicitly state that piuparts needs direct access to docker storage
See merge request debian/piuparts!41
- - - - -
41e1da76 by Nicolas Dandrimont at 2023-05-24T15:00:46+02:00
Introduce basic salsa-ci config
- - - - -
adb84017 by Nicolas Dandrimont at 2023-05-24T15:00:46+02:00
Enable diffoscope on salsa-ci reprotest
- - - - -
f1252503 by Nicolas Dandrimont at 2023-05-24T15:36:54+02:00
Split salsaci overrides for different stages
- - - - -
25 changed files:
- + .gitlab-ci.yml
- + .gitlab-ci/check-salsaci-overrides
- + .gitlab-ci/salsaci-overrides.yml
- Makefile
- custom-scripts/scripts/post_purge_exceptions
- custom-scripts/scripts/pre_install_exceptions
- custom-scripts/scripts/pre_test_root_password
- debian/changelog
- debian/control
- − debian/piuparts-master.lintian-overrides
- − debian/piuparts-slave.lintian-overrides
- debian/piuparts.lintian-overrides
- debian/rules
- + debian/source/lintian-overrides
- docs/README_pejacevic.txt
- docs/README_server.txt
- docs/conf.py
- docs/piuparts/piuparts.1.txt
- instances/piuparts.conf-template.pejacevic
- + known_problems/file_moved_usr_error.conf
- + known_problems/file_moved_usr_issue.conf
- piuparts-report.py
- piuparts-slave.py
- piuparts.py
- tests/test_piuparts.py
Changes:
=====================================
.gitlab-ci.yml
=====================================
@@ -0,0 +1,12 @@
+---
+include:
+ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
+ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
+ - .gitlab-ci/salsaci-overrides.yml
+
+stages:
+ - lint
+ - salsaci/provisioning
+ - salsaci/build
+ - salsaci/publish
+ - salsaci/test
=====================================
.gitlab-ci/check-salsaci-overrides
=====================================
@@ -0,0 +1,127 @@
+#!/usr/bin/python3
+#
+# Copyright © 2022 Nicolas Dandrimont (nicolas at dandrimont.eu)
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see <https://www.gnu.org/licenses/>
+
+from __future__ import annotations
+
+import os
+import sys
+
+import requests
+import yaml
+
+"""Check that salsaci-overrides.yml overrides all of the common salsaci pipelines"""
+
+KNOWN_TOPLEVEL_KEYS = {"include", "variables", "defaults"}
+
+
+def merge(source, destination):
+ """
+ run me with nosetests --with-doctest file.py
+
+ >>> a = { 'first' : { 'all_rows' : { 'pass' : 'dog', 'number' : '1' } } }
+ >>> b = { 'first' : { 'all_rows' : { 'fail' : 'cat', 'number' : '5' } }, 'list': [1, 2] }
+ >>> merge(b, a) == { 'first' : { 'all_rows' : { 'pass' : 'dog', 'fail' : 'cat', 'number' : '5' } }, 'list': [1, 2] }
+ True
+ """
+ for key, value in source.items():
+ if isinstance(value, dict):
+ # get node or create one
+ node = destination.setdefault(key, {})
+ merge(value, node)
+ elif isinstance(value, list):
+ node = destination.setdefault(key, [])
+ destination[key] = destination[key] + value
+ else:
+ destination[key] = value
+
+ return destination
+
+
+def get_gitlabci_config(git_root: str) -> Dict[str, Any]:
+ """Retrieve the full gitlab-ci config out of the git root"""
+
+ gitlabci_path = os.path.join(git_root, ".gitlab-ci.yml")
+ return yaml.safe_load(open(gitlabci_path, "r"))
+
+
+def fetch_config(url: str) -> Dict[str, Any]:
+ """Fetch a gitlab-ci config, resolving any includes"""
+ req = requests.get(url)
+ req.raise_for_status()
+
+ config = yaml.safe_load(req.content)
+
+ merged_config = {}
+ for include in config.pop("include", []):
+ included_config = fetch_config(include)
+ merge(included_config, merged_config)
+
+ merge(config, merged_config)
+ return merged_config
+
+
+def get_salsaci_config(gitlabci_config: Dict[str, Any]) -> Dict[str, Any]:
+ """Retrieve the config of the salsaci jobs, out of .gitlab-ci.yml includes"""
+ includes = gitlabci_config.get("include", [])
+ if not includes:
+ raise ValueError("No includes found in .gitlab-ci.yml")
+
+ for include in includes:
+ if include.endswith("pipeline-jobs.yml"):
+ break
+ else:
+ raise ValueError("No include for pipeline-jobs.yml found in .gitlab-ci.yml")
+
+ return fetch_config(include)
+
+
+def get_salsaci_overrides(
+ git_root: str, gitlabci_config: Dict[str, Any]
+) -> Dict[str, Any]:
+ """Retrieve the salsaci overrides, out of .gitlab-ci.yml includes"""
+ includes = gitlabci_config.get("include", [])
+ if not includes:
+ raise ValueError("No includes found in .gitlab-ci.yml")
+
+ for include in includes:
+ if include.endswith("salsaci-overrides.yml"):
+ break
+ else:
+ raise ValueError("No include for salsaci-overrides.yml found in .gitlab-ci.yml")
+
+ return yaml.safe_load(open(os.path.join(git_root, include), "r"))
+
+
+if __name__ == "__main__":
+ git_root = os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), ".."))
+ gitlabci_config = get_gitlabci_config(git_root)
+ salsaci_config = get_salsaci_config(gitlabci_config)
+ salsaci_overrides = get_salsaci_overrides(git_root, gitlabci_config)
+
+ keys_missing = {
+ key
+ for key in salsaci_config.keys()
+ - salsaci_overrides.keys()
+ - KNOWN_TOPLEVEL_KEYS
+ if not key.startswith(".")
+ }
+
+ for key in keys_missing:
+ print(key, salsaci_config[key])
+
+ if keys_missing:
+ sys.exit(1)
=====================================
.gitlab-ci/salsaci-overrides.yml
=====================================
@@ -0,0 +1,80 @@
+---
+check-salsaci-overrides:
+ stage: lint
+ image: debian:bullseye
+ before_script:
+ - apt-get update
+ - apt-get -y dist-upgrade
+ - apt-get -y install python3 python3-requests python3-yaml
+ script:
+ - ./.gitlab-ci/check-salsaci-overrides
+
+extract-source:
+ stage: salsaci/provisioning
+build:
+ stage: salsaci/build
+build i386:
+ stage: salsaci/build
+build arm64:
+ stage: salsaci/build
+build armel:
+ stage: salsaci/build
+build armhf:
+ stage: salsaci/build
+build source:
+ stage: salsaci/build
+aptly:
+ stage: salsaci/publish
+reprotest-environment:
+ stage: salsaci/test
+test-build-any:
+ stage: salsaci/test
+test-build-all:
+ stage: salsaci/test
+reprotest:
+ stage: salsaci/test
+atomic-reprotest:
+ stage: salsaci/test
+lintian:
+ stage: salsaci/test
+piuparts:
+ stage: salsaci/test
+autopkgtest:
+ stage: salsaci/test
+reprotest-domain_host:
+ stage: salsaci/test
+reprotest-user_group:
+ stage: salsaci/test
+reprotest-exec_path:
+ stage: salsaci/test
+blhc:
+ stage: salsaci/test
+reprotest-aslr:
+ stage: salsaci/test
+rc-bugs:
+ stage: salsaci/test
+reprotest-build_path:
+ stage: salsaci/test
+reprotest-fileordering:
+ stage: salsaci/test
+reprotest-home:
+ stage: salsaci/test
+reprotest-timezone:
+ stage: salsaci/test
+missing-breaks:
+ stage: salsaci/test
+reprotest-num_cpus:
+ stage: salsaci/test
+test-crossbuild-arm64:
+ stage: salsaci/test
+reprotest-locales:
+ stage: salsaci/test
+reprotest-kernel:
+ stage: salsaci/test
+reprotest-umask:
+ stage: salsaci/test
+reprotest-time:
+ stage: salsaci/test
+
+variables:
+ SALSA_CI_REPROTEST_ENABLE_DIFFOSCOPE: 1
=====================================
Makefile
=====================================
@@ -10,11 +10,11 @@ site3 = $(libdir)/python3/dist-packages
htdocsdir = $(sharedir)/piuparts/htdocs
etcdir = $(prefix)/etc
-distribution=${shell dpkg-parsechangelog | sed -n 's/^Distribution: *//p'}
+distribution=${shell dpkg-parsechangelog -SDistribution}
ifeq ($(distribution),UNRELEASED)
-version := ${shell echo "`dpkg-parsechangelog | sed -n 's/^Version: *//p'`~`date +%Y%m%d%H%M`~`git describe --dirty`"}
+version := ${shell echo "`dpkg-parsechangelog -SVersion`~`date +%Y%m%d%H%M`~`git describe --dirty`"}
else
-version := ${shell dpkg-parsechangelog | sed -n 's/^Version: *//p'}
+version := ${shell dpkg-parsechangelog -SVersion}
endif
@@ -83,42 +83,7 @@ install-doc: build-stamp
install -d $(DESTDIR)$(docdir)/
install -m 0644 docs/README.txt docs/README_server.txt $(DESTDIR)$(docdir)/
# html
- install -d $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/*.html $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/searchindex.js $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/objects.inv $(DESTDIR)$(docdir)/html/
- install -d $(DESTDIR)$(docdir)/html/_static/
- install -m 0644 docs/build/_static/* $(DESTDIR)$(docdir)/html/_static
- install -d $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/index.html $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/piuparts.1.html $(DESTDIR)$(docdir)/html/piuparts/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/piuparts_slave_run.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/piuparts_slave_join.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/piuparts_slave_stop.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -d $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/*.html $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/searchindex.js $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/objects.inv $(DESTDIR)$(docdir)/html/
- install -d $(DESTDIR)$(docdir)/html/_static/
- install -m 0644 docs/build/_static/* $(DESTDIR)$(docdir)/html/_static
- install -d $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/index.html $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/piuparts.1.html $(DESTDIR)$(docdir)/html/piuparts/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/piuparts_slave_run.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/piuparts_slave_join.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/piuparts_slave_stop.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
+ cp -r docs/build/ $(DESTDIR)$(docdir)/html/
# manpages
install -d $(DESTDIR)$(man1dir)
install -m 0644 piuparts.1 $(DESTDIR)$(man1dir)/
=====================================
custom-scripts/scripts/post_purge_exceptions
=====================================
@@ -53,6 +53,13 @@ case ${PIUPARTS_OBJECTS%%=*} in
log_debug
sleep 10
;;
+ *usr-is-merged*|*usrmerge*)
+ # Remove the exception flag that we added in pre_install_exceptions, if we added it
+ if [ -f /etc/unsupported-skip-usrmerge-conversion.piuparts ]; then
+ log_debug
+ rm -f /etc/unsupported-skip-usrmerge-conversion*
+ fi
+ ;;
esac
if [ -f /var/run/piuparts-stamp-dkms-installed ]; then
=====================================
custom-scripts/scripts/pre_install_exceptions
=====================================
@@ -106,6 +106,20 @@ case ${PIUPARTS_OBJECTS%%=*} in
mkdir -p /var/state/samhain/
touch /var/state/samhain/samhain_file
;;
+ *usr-is-merged*|*usrmerge*)
+ # usr-is-merged intentionally fails to install unless either the system is merged-usr, or
+ # the exception flag is set.
+ # usrmerge does a live-upgrade of the chroot, which piuparts does not expect, so skip it.
+ # debootstrap will take care of this in a future update, so remember if we were the ones
+ # that added the flag file, to remove it in the post_purge_exceptions script.
+ # For details on the flag file, see: https://sources.debian.org/src/usrmerge/27/debian/README.Debian/#L66
+ # and: https://lists.debian.org/debian-ctte/2022/07/msg00019.html
+ if [ ! -f /etc/unsupported-skip-usrmerge-conversion ]; then
+ log_debug
+ touch /etc/unsupported-skip-usrmerge-conversion.piuparts
+ echo "this system will not be supported in the future" > /etc/unsupported-skip-usrmerge-conversion
+ fi
+ ;;
esac
if [ "$PIUPARTS_DISTRIBUTION" = "jessie" ]; then
=====================================
custom-scripts/scripts/pre_test_root_password
=====================================
@@ -2,6 +2,6 @@
set -e
# sudo refuses removal if no root password is set, so set one
-# do this unconditionally, as there are quite some packages depending on sudo
+# unconditionally, as there are quite some packages depending on sudo
# (and since its harmless and fast)
yes "yes" 2>/dev/null | passwd
=====================================
debian/changelog
=====================================
@@ -1,3 +1,61 @@
+piuparts (1.1.8) UNRELEASED; urgency=medium
+
+ * Start 1.1.8 development. d/changelog entries will be written on
+ release using the git commit messages.
+
+ -- Holger Levsen <holger at debian.org> Wed, 18 Jan 2023 22:53:30 +0100
+
+piuparts (1.1.7) unstable; urgency=medium
+
+ [ Holger Levsen ]
+ * piuparts.py, tests and docs: handle new "non-free-firmware" section in
+ Debian. Closes: #1028661.
+ * d/control:
+ - remove version constraints on the (build-)depends on python3-debianbts,
+ debootstrap amd debsums as they are met since (at least) buster.
+ - fix cute-field warning, thanks lintian.
+ - bump standards version to 4.6.2, no changes needed.
+ * Add debian/source/lintian-overrides to override a bunch of very-long-line-
+ length-in-source-file warnings.
+ * d/piuparts.lintian-overrides: add debian-news-entry-has-unknown-version
+ 0.45 due to trimmed d/changelog in binary packages.
+ * Drop unused d./piuparts-(master|slave).lintian-overrides.
+
+ [ Nicolas Dandrimont ]
+ * Makefile: use dpkg-parsechangelog -S instead of sed.
+
+ -- Holger Levsen <holger at debian.org> Wed, 18 Jan 2023 22:45:04 +0100
+
+piuparts (1.1.6) unstable; urgency=medium
+
+ [ David Steele ]
+ * Remove oldold*stable dists from json summary.
+
+ [ Holger Levsen ]
+ * custom-scripts/scripts/pre_test_root_password: fix grammar in comment.
+
+ [ Luca Boccassi ]
+ * piuparts: detect files moving between / and /usr on upgrade.
+ * piuparts: detect files moving between / and /usr on dist-upgrade.
+ * p.conf: enable --warn-on-usr-move for bullseye/bookworm.
+ * custom_scripts: ensure usr-is-merged/usrmerge does not fail the
+ migration test.
+
+ [ David Steele ]
+ * Add Stable Release tasks to README_pejacevic.txt.
+
+ [ Nicolas Dandrimont ]
+ * piuparts-slave: run isort.
+ * piuparts-slave: replace pipes.quote with shlex.quote.
+ * piuparts.py: run isort.
+ * piuparts.py: replace pipes.quote with shlex.quote.
+ * test_piuparts: replace mox3 with unittest.mock (Closes: #1028453)
+ * Migrate lintian overrides to pointed hints.
+ * Simplify docs install in Makefile.
+ * Use sphinxdoc dh plugin.
+
+ -- Nicolas Dandrimont <olasd at debian.org> Wed, 11 Jan 2023 13:31:41 +0100
+
piuparts (1.1.5) unstable; urgency=medium
[ Nicolas Dandrimont ]
@@ -10,8 +68,8 @@ piuparts (1.1.5) unstable; urgency=medium
* Remove oldstable results from summary.json.
[ Athos Ribeiro ]
- * d/rules: set GO111MODULE to auto to maintain pre Go 1.16 behavior. Closes: #991970.
- (MR: debian/piuparts!35)
+ * d/rules: set GO111MODULE to auto to maintain pre Go 1.16 behavior.
+ Closes: #991970. (MR: debian/piuparts!35)
-- Holger Levsen <holger at debian.org> Thu, 14 Oct 2021 15:23:26 +0200
=====================================
debian/control
=====================================
@@ -5,7 +5,7 @@ Uploaders: Nicolas Dandrimont <olasd at debian.org>,
Holger Levsen <holger at debian.org>
Section: devel
Priority: optional
-Standards-Version: 4.5.1
+Standards-Version: 4.6.2
Rules-Requires-Root: no
Build-Depends:
debhelper-compat (= 12),
@@ -13,9 +13,8 @@ Build-Depends:
python3-all,
python3-apt,
python3-debian,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-distro-info,
- python3-mox3,
python3-nose,
python3-six,
python3-sphinx,
@@ -26,22 +25,22 @@ Build-Depends-Indep:
Homepage: https://piuparts.debian.org
Vcs-Git: https://salsa.debian.org/debian/piuparts.git
Vcs-Browser: https://salsa.debian.org/debian/piuparts
-Xs-Go-Import-Path: salsa.debian.org/debian/piuparts
+XS-Go-Import-Path: salsa.debian.org/debian/piuparts
Package: piuparts
Architecture: all
Depends:
piuparts-common (= ${binary:Version}),
# keep this list in sync with piuparts-slave-from-git-deps
- debootstrap (>= 1.0.89~),
- debsums (>= 2.2.2~),
- libjs-sphinxdoc,
+ debootstrap,
+ debsums,
lsb-release,
lsof,
mount,
python3-debian,
${python3:Depends},
${misc:Depends},
+ ${sphinxdoc:Depends}
Recommends:
adequate
Suggests:
@@ -67,7 +66,7 @@ Depends:
# keep this list in sync with piuparts-master-from-git-deps
adduser,
openssh-server,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-setproctitle,
tango-icon-theme,
xz-utils,
@@ -100,7 +99,7 @@ Depends:
# this list is synced from piuparts-master
adduser,
openssh-server,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-setproctitle,
tango-icon-theme,
xz-utils,
@@ -157,8 +156,8 @@ Depends:
screen,
sudo,
# this list is synced from piuparts
- debootstrap (>= 1.0.89~),
- debsums (>= 2.2.2~),
+ debootstrap,
+ debsums,
libjs-sphinxdoc,
lsb-release,
lsof,
=====================================
debian/piuparts-master.lintian-overrides deleted
=====================================
@@ -1,5 +0,0 @@
-# It's ok, because it depends on piuparts-common that has the dependency.
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/master/detect_well_known_errors #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-analyze #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-master-backend #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-report #!/usr/bin/python3
=====================================
debian/piuparts-slave.lintian-overrides deleted
=====================================
@@ -1,2 +0,0 @@
-# It's ok, because it depends on piuparts-common that has the dependency.
-piuparts-slave: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-slave #!/usr/bin/python3
=====================================
debian/piuparts.lintian-overrides
=====================================
@@ -1,6 +1,7 @@
# it is impossible not to do this without losing compatibility with ancient releases
-uses-dpkg-database-directly etc/piuparts/scripts-debug-purge/post_remove_postrm_set-x
-uses-dpkg-database-directly etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x
-uses-dpkg-database-directly etc/piuparts/scripts/post_remove_exceptions
-uses-dpkg-database-directly etc/piuparts/scripts/pre_remove_40_find_missing_md5sums
-uses-dpkg-database-directly usr/sbin/piuparts
+uses-dpkg-database-directly [etc/piuparts/scripts-debug-purge/post_remove_postrm_set-x]
+uses-dpkg-database-directly [etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x]
+uses-dpkg-database-directly [etc/piuparts/scripts/post_remove_exceptions]
+uses-dpkg-database-directly [etc/piuparts/scripts/pre_remove_40_find_missing_md5sums]
+uses-dpkg-database-directly [usr/sbin/piuparts]
+debian-news-entry-has-unknown-version 0.45 [usr/share/doc/piuparts/NEWS.Debian.gz:1]
=====================================
debian/rules
=====================================
@@ -7,7 +7,7 @@ export GOCACHE := $(CURDIR)/.gocache
export GO111MODULE := auto
%:
- dh $@ --with python3
+ dh $@ --with python3,sphinxdoc
override_dh_auto_build:
$(MAKE) prefix=/usr build build-doc
=====================================
debian/source/lintian-overrides
=====================================
@@ -0,0 +1,4 @@
+very-long-line-length-in-source-file 650 > 512 [htdocs/news.tpl:214]
+very-long-line-length-in-source-file 671 > 512 [htdocs/index.tpl:46]
+very-long-line-length-in-source-file 777 > 512 [docs/piuparts/piuparts.1.txt:27]
+very-long-line-length-in-source-file 515 > 512 [known_problems/db_setup_issue.conf:4]
=====================================
docs/README_pejacevic.txt
=====================================
@@ -226,11 +226,29 @@ daily by '~piupartsm/crontab'.
More checks should be added as we become aware of them.
+:ref:`top <top3>`
+
+:blue:`New Stable Release`
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The server exports data, used by DDPO et al. to display test results by
+package. With a new release of Debian Stable, a number of tasks are required to
+update the summary data appropriately:
+
+- Edit 'piuparts.conf.pejacevic' to set 'json-sections = none' for the sections
+ that now represent oldstable ('foo', 'bar2foo', 'foo-proposed', etc.).
+- In particular, make sure that oldoldstable releases, and older, are not
+ included in the global report ('distro-info' only goes back to oldstable).
+- Update the 'distro-info-data' package, so that 'piuparts-report' can correctly
+ assign sections to distribution names.
+
+
:ref:`top <top3>`
:blue:`Authors`
^^^^^^^^^^^^^^^
-Last updated: February 2017
+Last updated: November 2022
Holger Levsen <holger at layer-acht.org>
+David Steele <steele at debian.org>
=====================================
docs/README_server.txt
=====================================
@@ -480,7 +480,7 @@ section, too, and will serve as defaults for all other sections
* "area" is the archive area used to get the list of packages to
be tested. The Packages file for this area will be loaded. The
default is "main" and the possible values depend on the vendor,
- for Debian these are main, contrib, non-free.
+ for Debian these are main, contrib, non-free and non-free-firmware.
* "components" sets the archive areas that will be available when
testing the packages selected via the "area" setting. These will
=====================================
docs/conf.py
=====================================
@@ -171,3 +171,5 @@ epub_title = project
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
+
+option_emphasise_placeholders = True
=====================================
docs/piuparts/piuparts.1.txt
=====================================
@@ -3,377 +3,442 @@
piuparts(1)
===========
:doctype: manpage
- :revdate: 2017-03-06
-
+ :revdate: 2023-01-18
+.. program:: piuparts
SYNOPSIS
--------
-*piuparts* ['-apvV'] ['-d' _distro_] ['-i' _filename_] ['-I' _regexp_] ['-l' _logfile_] ['-m' _url_] ['--bindmount' _dir_] [_package_]... [_changes_file_]...
+**piuparts** [:option:`-a` :option:`-p` :option:`-v` :option:`-V`] [:option:`-d` *distro*] [:option:`-i` *filename*] [:option:`-I` *regexp*] [:option:`-l` *logfile*] [:option:`-m` *URL*] [:option:`--bindmount` *dir*] [*package* ...] [*changes\_file* ...]
DESCRIPTION
-----------
-*piuparts* tests that Debian packages handle installation, upgrading, and removal correctly. It does this by creating a minimal Debian installation in a chroot, and installing, upgrading, and removing packages in that environment, and comparing the state of the directory tree before and after. *piuparts* reports any files that have been added, removed, or modified during this process.
+:program:`piuparts` tests that Debian packages handle installation, upgrading, and removal correctly. It does this by creating a minimal Debian installation in a chroot, and installing, upgrading, and removing packages in that environment, and comparing the state of the directory tree before and after. :program:`piuparts` reports any files that have been added, removed, or modified during this process.
-*piuparts* is meant as a quality assurance tool for people who create Debian packages to test them before they upload them to the Debian package
+:program:`piuparts` is meant as a quality assurance tool for people who create Debian packages to test them before they upload them to the Debian package
archive.
By default, piuparts can do three different tests:
-. A simple install-purge test within one Debian distribution (chosen with the '-d' option, unstable by default). It sets up the chroot with the desired distribution, then installs and purges the packages, and reports problems.
+* A simple install-purge test within one Debian distribution (chosen with the :option:`-d` option, unstable by default). It sets up the chroot with the desired distribution, then installs and purges the packages, and reports problems.
-. A simple install-upgrade-purge test within one Debian distribution. This test is like the install-purge test, but it installs the packages first via *apt-get* and then from the package files given on the command line. If the command line has package names (option '--apt' used), or no tested package is known to *apt-get* (new packages), this test is skipped, otherwise it is performed automatically.
+* A simple install-upgrade-purge test within one Debian distribution. This test is like the install-purge test, but it installs the packages first via :program:`apt-get` and then upgrades them with the package files given on the command line. If the command line has package names (option :option:`--apt` used), or no tested package is known to :program:`apt-get` (new packages), this test is skipped, otherwise it is performed automatically.
-. An upgrade test between Debian releases. This test is enabled by using the '-d' option multiple times and disables the other two tests. It sets up the chroot with the first distribution named, then upgrades it to each successive one, and then remembers the directory tree state at the end. After this, it starts over with the chroot of the first distribution, installs the desired packages (via *apt-get*), and does the successive upgrading (via *apt-get dist-upgrade*). Then, if package files (and not just package names) were given on the command line, it installs them. Finally, it reports problems against the state of the directory tree at the last distribution compared with the state without the packages having been installed. This test can be quite slow to execute.
-+
-Note that this does not work with experimental, because *apt-get* does not automatically upgrade to packages in experimental. To test a particular package or group of packages in experimental, use the second test.
+* An upgrade test between Debian releases. This test is enabled by using the :option:`-d` option multiple times and disables the other two tests. It sets up the chroot with the first distribution named, then upgrades it to each successive one, and then remembers the directory tree state at the end. After this, it starts over with the chroot of the first distribution, installs the desired packages (via :command:`apt-get install`), and does the successive upgrading (via :command:`apt-get dist-upgrade`). Then, if package files (and not just package names) were given on the command line, it installs them. Finally, it reports problems against the state of the directory tree at the last distribution compared with the state without the packages having been installed. This test can be quite slow to execute. Note that this does not work with experimental, because :program:`apt-get` does not automatically upgrade to packages in experimental. To test a particular package or group of packages in experimental, use the second test.
-Command line arguments are the paths to package files (e.g., *piuparts_1.0-1_all.deb*), paths to changes files (e.g., *piuparts_1.0-1_i386.changes*), or names of packages, if the '--apt' option is given.
+Command line arguments are the paths to package files (e.g., ``piuparts_1.0-1_all.deb``), paths to changes files (e.g., ``piuparts_1.0-1_i386.changes``), or names of packages, if the :option:`--apt` option is given.
-When processing changes files, by default, all packages in a changes file will be processed together with all individual packages given on the command line. Then each package given on the command line is processed in a single group. If the '--single-changes-list' is used, the packages in all changes files are processed together along with any individual packages that were given on the command line. To avoid this behaviour, it is possible to specify '--single-packages'.
+When processing changes files, by default, all packages in a changes file will be processed together with all individual packages given on the command line. Then each package given on the command line is processed in a single group. If :option:`--single-changes-list` is used, the packages in all changes files are processed together along with any individual packages that were given on the command line. To avoid this behaviour, it is possible to specify :option:`--single-packages`.
-*piuparts* outputs to the standard output some log messages to show what is going on. If a log file is used, the messages go there as well.
+:program:`piuparts` outputs to the standard output some log messages to show what is going on. If a log file is used, the messages go there as well.
-*piuparts* needs to be run as root.
+:program:`piuparts` needs to be run as root.
OPTIONS
-------
Options must come before the other command line arguments.
-*-a*, *--apt*::
- The package arguments on the command line are to be treated as package names and installed via *apt-get* instead of being names of package files, to be installed via *dpkg -i*.
+.. option:: -a, --apt
+
+ The package arguments on the command line are to be treated as package names and installed via :command:`apt-get install` instead of being names of package files, to be installed via :command:`dpkg -i`.
+
+.. option:: --allow-database
-*-*-allow-database*::
Allow starting MySQL and PostgreSQL database servers in the chroot for
packages requiring database access in their maintainer scripts.
Do not use this option if there is already a database server running on the
- system running piuparts (or piuparts-slave)!
+ system running :program:`piuparts` (or :program:`piuparts-slave`)!
In master-slave setups with multiple slaves running on one host collisions
- may occur, these will be detected by 'detect_piuparts_issues' and the
+ may occur, these will be detected by :program:`detect_piuparts_issues` and the
affected packages will be tested again.
-*-*-arch*='arch'::
- Create chroot and run tests for (non-default) architecture 'arch'. The default is the output from 'dpkg --print-architecture'.
+.. option:: --arch arch
+
+ Create chroot and run tests for (non-default) architecture *arch*. The default is the output from :command:`dpkg --print-architecture`.
+
+.. option:: -b tarball, --basetgz tarball
-*-b* 'tarball', *-*-basetgz*='tarball'::
- Use tarball as the contents of the initial chroot, instead of building a new one with debootstrap.
+ Use :file:`tarball` as the contents of the initial chroot, instead of building a new one with :program:`debootstrap`.
- The tarball can be created with the '-s' option, or you can use one that *pbuilder* has created (see '-p'). If you create one manually, make sure the root of the chroot is the root of the tarball.
+ The tarball can be created with the :option:`-s` option, or you can use one that :program:`pbuilder` has created (see :option:`-p`). If you create one manually, make sure the root of the chroot is the root of the tarball.
+
+.. option:: --bindmount dir
-*-*-bindmount*='dir'::
Bind-mount a directory inside the chroot.
-*-d* 'name', *-*-distribution*='name'::
- Which Debian distribution to use: a code name (for example bullseye, bookworm or sid) or experimental. The default is sid (=unstable).
+.. option:: -d name, --distribution name
+
+ Which Debian distribution to use: a code name (for example ``bullseye``, ``bookworm`` or ``sid``) or ``experimental``. The default is ``sid`` (= ``unstable``).
+
+.. option:: -D flavor, --defaults flavor
+
+ Use default settings suitable for a particular flavor of Debian: either ``debian`` or ``ubuntu``. The default is ``debian``.
-*-D* 'flavor', *-*-defaults*='flavor'::
- Use default settings suitable for a particular flavor of Debian: either debian or ubuntu. The default is debian.
+.. option:: --do-not-verify-signatures
-*-*-do-not-verify-signatures*::
- Do not verify signatures from the Release files when running debootstrap. Also set APT::Get::AllowUnauthenticated accordingly in /etc/apt/apt.conf in the chroots.
+ Do not verify signatures from the Release files when running debootstrap. Also set ``APT::Get::AllowUnauthenticated`` accordingly in :file:`/etc/apt/apt.conf` in the chroots.
-*-*-dpkg-force-confdef*::
- Make dpkg use --force-confdef, which lets dpkg always choose the default action when a modified conffile is found. This option will make piuparts ignore errors it was designed to report and therefore should only be used to hide problems in depending packages. 'This option shall normally not be used.' (See #466118.)
+.. option:: --dpkg-force-confdef
-*-*-dpkg-noforce-unsafe-io*::
- Prevent running dpkg with --force-unsafe-io. --force-unsafe-io causes dpkg to skip certain file system syncs known to cause substantial performance degradation on some filesystems. Thus, including this option reverts to safe but slower behavior. The --dpkg-noforce-unsafe-io is required for running tests on distributions older than squeeze.
+ Make :program:`dpkg` use ``--force-confdef``, which lets :program:`dpkg` always choose the default action when a modified conffile is found. This option will make :program:`piuparts` ignore errors it was designed to report and therefore should only be used to hide problems in depending packages. *This option shall normally not be used.* (See `#466118 <https://bugs.debian.org/466118>`_.)
+
+.. option:: --dpkg-noforce-unsafe-io
+
+ Prevent running :program:`dpkg` with ``--force-unsafe-io``. ``--force-unsafe-io`` causes :program:`dpkg` to skip certain file system syncs known to cause substantial performance degradation on some filesystems. Thus, including this option reverts to safe but slower behavior. The :option:`--dpkg-noforce-unsafe-io` is required for running tests on distributions older than squeeze.
+
+.. option:: --no-eatmydata
-*-*-no-eatmydata*::
Prevent use of eatmydata. The --no-eatmydata option is required for running tests on squeeze or older distributions.
-*-*-extra-old-packages*='pkg1[,pkg2]...'::
+.. option:: --extra-old-packages pkg1[,pkg2]...
+
Install additional old packages before upgrading. Allows testing package renames/merges where the old package is no longer available in the new distribution and the new one utilizes Conflicts/Replaces. The argument is a comma separated list of package names and the option can be given multiple times.
For install/purge tests these packages will be installed before the package
that is to be tested.
-*-e* 'dirname', *-*-existing-chroot*='dirname'::
+.. option:: -e dirname, --existing-chroot dirname
+
Use the specified directory as source for the new chroot, instead of building
- a new one with debootstrap. This is similar to '--basetgz', but the contents
- are not archived. See also the --hard-link option.
+ a new one with debootstrap. This is similar to :option:`--basetgz`, but the contents
+ are not archived. See also the :option:`--hard-link` option.
+
+.. option:: --distupgrade-to-testdebs
-*-*-distupgrade-to-testdebs*::
Use the "testdebs" repository to override the packages in the distupgrade
target distribution. This allows one to test complex upgrade paths before the
packages enter the archive.
-*-*-extra-repo*='deb-line'::
- Provide an additional (unparsed) line to be appended to sources.list, e.g.::
+.. option:: --extra-repo deb-line
+ Provide an additional (unparsed) line to be appended to :file:`sources.list`, for example::
'deb <URL> <distrib> <components>'
or
'deb file:// </bind/mount> ./'
Useful for e.g. backports, security or local repositories that cannot be
- handled by '--mirror'. May be repeated to add more than one line.
+ handled by :option:`--mirror`. May be repeated to add more than one line.
+
+.. option:: --fake-essential-packages pkg1[,pkg2]...
-*-*-fake-essential-packages*='pkg1[,pkg2]...'::
Install additional packages in the base chroot that are not removed after
the test. These are available during purge and for checking against
mistreatment.
Takes a comma separated list of package names and can be given multiple
times.
-*-*-hard-link*::
- When the --existing-chroot option is used, and the source directory is on the
+.. option:: --hard-link
+
+ When the :option:`--existing-chroot` option is used, and the source directory is on the
same filesystem, hard-link files instead of copying them. This is faster, but
any modifications to files will be reflected in the originals.
-*-i* 'filename', *-*-ignore*='filename'::
+.. option:: -i filename, --ignore filename
+
Add a filename to the list of filenames to be ignored when comparing changes
- before and after installation. By default, piuparts ignores files that always
- change during a package installation and uninstallation, such as *dpkg*
+ before and after installation. By default, :program:`piuparts` ignores files that always
+ change during a package installation and uninstallation, such as :program:`dpkg`
status files. The filename should be relative to the root of the chroot
- (e.g., *var/lib/dpkg/status*).
- Filenames prefixed with a ':' will be logged verbosely if found.
+ (e.g., :file:`var/lib/dpkg/status`).
+ Filenames prefixed with a ``:`` will be logged verbosely if found.
This option can be used as many times as necessary.
-*-I* 'regexp', *-*-ignore-regexp*='regexp'::
+.. option:: -I regexp, --ignore-regexp regexp
+
Add a regular expression pattern to the list of patterns for filenames to be
ignored when comparing changes before and after installation.
- Patterns prefixed with a ':' will log verbosely all matching files.
+ Patterns prefixed with a ``:`` will log verbosely all matching files.
This option can be used as many times as necessary.
-*-*-install-purge-install*::
+.. option:: --install-purge-install
+
Purge package after installation and reinstall. All dependencies are available during purge.
-*-*-install-recommends*::
+.. option:: --install-recommends
+
Enable installation of Recommends.
-*-*-install-suggests*::
+.. option:: --install-suggests
+
Enable installation of Suggests.
-*-*-install-remove-install*::
+.. option:: --install-remove-install
+
Remove package after installation and reinstall. For testing installation in config-files-remaining state.
-*-k*, *-*-keep-env*::
- Depending on which option is passed, keep the environment used for testing after the program ends::
+.. option:: -k, --keep-env
+
+ Depending on which option is passed, keep the environment used for testing after the program ends
+
* By default it doesn't remove the temporary directory for the chroot,
- * or if --schroot is used, the schroot session is not terminated,
- * or if --docker-image is used, the container created is not destroyed.
+ * or if :option:`--schroot` is used, the schroot session is not terminated,
+ * or if :option:`--docker-image` is used, the container created is not destroyed.
+
+.. option:: -K filename, --keyring filename
+
+ Use :file:`filename` as the keyring to use with debootstrap when creating chroots.
-*-K*, *-*-keyring*='filename'::
- Use FILE as the keyring to use with debootstrap when creating chroots.
+.. option:: --keep-sources-list
-*-*-keep-sources-list*::
- Don't modify the chroot's etc/apt/sources.list.
+ Don't modify the chroot's :file:`/etc/apt/sources.list`.
+
+.. option:: --list-installed-files
-*-*-list-installed-files*::
List the files added to the chroot after the installation of the package and after the installation of the package dependencies.
-*-*-lvm-volume*='lvm-volume'::
- Use the specified lvm-volume as source for the chroot, instead of building a
+.. option:: --lvm-volume lvm-volume
+
+ Use the specified *lvm-volume* as source for the chroot, instead of building a
new one with debootstrap. This creates a snapshot of the given LVM volume and
mounts it to the chroot path.
-*-*-lvm-snapshot-size*='snapshot-size'::
- Use the specified snapshot-size as snapshot size when creating a new LVM
+.. option:: --lvm-snapshot-size snapshot-size
+
+ Use the specified *snapshot-size* as snapshot size when creating a new LVM
snapshot (default: 1G)
-*-l* 'filename', *-*-log-file*='filename'::
- Append log file to _filename_ in addition to the standard output.
+.. option:: -l filename, --log-file filename
+
+ Append log file to :file:`filename` in addition to the standard output.
+
+.. option:: --log-level level
-*-*-log-level*='level'::
- Display messages from loglevel LEVEL, possible values are: error, info, dump, debug. The default is dump.
+ Display messages from loglevel *level*, possible values are: *error*, *info*, *dump*, *debug*. The default is *dump*.
-*-*-max-command-output-size*='size'::
- Set the maximum permitted command output to _size_ (in MB) for debugging
+.. option:: --max-command-output-size size
+
+ Set the maximum permitted command output to *size* (in MB) for debugging
runs exceeding the default of 8 MB.
-*-*-merged-usr*::
- When using debootstrap to create the chroot, use the '--merged-usr' option
- to create a chroot with /bin, /lib, /sbin being symlinks to their /usr
+.. option:: --merged-usr
+
+ When using debootstrap to create the chroot, use the ``--merged-usr`` option
+ to create a chroot with :file:`/bin`, :file:`/lib`, :file:`/sbin` being symlinks to their :file:`/usr`
counterparts.
-*-m* 'url', *-*-mirror*='url'::
- Which Debian mirror to use. The default is the first mirror named in */etc/apt/sources.list* or *http://deb.debian.org/debian* if none is found. This option may be used multiple times to use multiple mirrors. Only the first mirror is used with *debootstrap*.
+.. option:: -m URL, --mirror URL
+
+ Which Debian mirror to use. The default is the first mirror named in :file:`/etc/apt/sources.list` or ``http://deb.debian.org/debian`` if none is found. This option may be used multiple times to use multiple mirrors. Only the first mirror is used with :program:`debootstrap`.
+
+ The 'components' that are used for a mirror can also be set with this option: a space separated list within the same argument (so you need to quote the entire argument in the shell). If no components are given explicitly, the usual Debian components are used (*main*, *contrib*, *non-free* and *non-free-firmware*). For the mirrors read from :file:`/etc/apt/sources.list`, the components are read from the same place.
- The 'components' that are used for a mirror can also be set with this option: a space separated list within the same argument (so you need to quote the entire argument in the shell). If no components are given explicitly, the usual Debian components are used (main, contrib, and non-free). For the mirrors read from */etc/apt/sources.list*, the components are read from the same place.
+ Note that ``file:`` addresses work if the directories are made accessible from within the chroot with :option:`--bindmount`.
- Note that file: addresses works if the directories are made accessible from within the chroot with '--bindmount'.
+.. option:: --no-adequate
-*-*-no-adequate*::
- Don't run adequate after installation. The default is to run adequate, provided it is installed.
+ Don't run :program:`adequate` after installation. The default is to run :program:`adequate`, provided it is installed.
-*-*-no-check-valid-until*::
- Set apt option Acquire::Check-Valid-Until=false in the chroot to ignore the
+.. option:: --no-check-valid-until
+
+ Set :program:`apt` option ``Acquire::Check-Valid-Until=false`` in the chroot to ignore the
expiration of Release files. This is needed for testing archived releases.
-*-*-no-diversions*::
+.. option:: --no-diversions
+
Don't check for broken diversions.
-*-n*, *-*-no-ignores*::
- Forget all built-in and other ignores that have been set so far. Any '-i' or '-I' arguments that come after this one will be obeyed, but none of the ones that come before.
+.. option:: -n, --no-ignores
+
+ Forget all built-in and other ignores that have been set so far. Any :option:`-i` or :option:`-I` arguments that come after this one will be obeyed, but none of the ones that come before.
+
+.. option:: -N, --no-symlinks
-*-N*, *-*-no-symlinks*::
Don't check for broken symlinks.
-*-*-fail-if-inadequate*::
- Fail on inadequate results from running adequate. The default is to just issue those errors as warnings.
+.. option:: --fail-if-inadequate
+
+ Fail on inadequate results from running :program:`adequate`. The default is to just issue those errors as warnings.
+
+.. option:: --fail-on-broken-symlinks
-*-*-fail-on-broken-symlinks*::
Fail on broken symlinks. The default is to just issue those errors as warnings.
-*-*-no-upgrade-test*::
+.. option:: --no-upgrade-test
+
Skip testing upgrade from an existing version in the archive.
-*-*-no-install-purge-test*::
+.. option:: --no-install-purge-test
+
Skip the install and purge test.
-*-p, *-*-pbuilder*::
- Use */var/cache/pbuilder/base.tgz* as the base tarball. This is a shorthand so that you don't need to use '-b' for it.
+.. option:: -p, --pbuilder
+
+ Use :file:`/var/cache/pbuilder/base.tgz` as the base tarball. This is a shorthand so that you don't need to use :option:`-b` for it.
-*-*-pedantic-purge-test*::
- Be pedantic when checking if a purged package leaves files behind. If this option is not set, files left in */tmp* are ignored.")
+.. option:: --pedantic-purge-test
-*-*-proxy*='URL'::
- Use the proxy at URL to access the Debian mirror(s). Takes precedence over
- the 'http_proxy' environment variable. Using a local proxy is recommended
- because piuparts may use large amounts of bandwidth to repeatedly download
+ Be pedantic when checking if a purged package leaves files behind. If this option is not set, files left in :file:`/tmp` are ignored.
+
+.. option:: --proxy URL
+
+ Use the proxy at *URL* to access the Debian mirror(s). Takes precedence over
+ the :envvar:`http_proxy` environment variable. Using a local proxy is recommended
+ because :program:`piuparts` may use large amounts of bandwidth to repeatedly download
the same files.
-*-s* 'filename', *-*-save*='filename'::
- Save the chroot, after it has been set up, as a tarball into *filename*. It can then be used with '-b'.
+.. option:: -s filename, --save filename
+
+ Save the chroot, after it has been set up, as a tarball into :file:`filename`. It can then be used with :option:`-b`.
+
+.. option:: -B filename, --end-meta filename
+
+ Load chroot package selection and file meta data from :file:`filename`. See the function :func:`install_and_upgrade_between_distros` in :file:`piuparts.py` for defaults. Mostly useful for large scale distro upgrade tests.
+
+.. option:: -S filename, --save-end-meta filename
+
+ Save chroot package selection and file meta data in :file:`filename` for later use. See the function :func:`install_and_upgrade_between_distros` in :file:`piuparts.py` for defaults. Mostly useful for large scale distro upgrade tests.
+
+.. option:: --scriptsdir directory
+
+ Directory where are custom scripts are placed. By default, this is not set. For more information about this, read :file:`README_server.txt`
-*-B* 'FILE', *-*-end-meta*='FILE'::
- Load chroot package selection and file meta data from FILE. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests.
+.. option:: --schroot SCHROOT-NAME
-*-S* 'FILE', *-*-save-end-meta*='FILE'::
- Save chroot package selection and file meta data in FILE for later use. See the function install_and_upgrade_between_distros() in piuparts.py for defaults. Mostly useful for large scale distro upgrade tests.
+ Use schroot session named *SCHROOT-NAME* for the testing environment, instead of building a new one with :program:`debootstrap`.
-*-*-scriptsdir*='DIR'::
- Directory where are custom scripts are placed. By default, this is not set. For more information about this, read README_server.txt
+.. option:: --docker-image DOCKER-IMAGE
-*-*-schroot*='SCHROOT-NAME'::
- Use schroot session named SCHROOT-NAME for the testing environment, instead of building a new one with debootstrap.
+ Use a container created from the docker image *DOCKER-IMAGE* for the testing environment, instead of building a new one with :program:`debootstrap`.
+ This only supports ``overlay2`` for now and it uses the ``MergedDir`` layer where :program:`piuparts` can access, add, edit and remove files easily by directly accessing the directory.
-*-*-docker-image*='DOCKER-IMAGE'::
- Use a container created from the docker image DOCKER-IMAGE for the testing environment, instead of building a new one with debootstrap.
- This only supports overlay2 for now and it uses the `MergedDir` layer where piuparts can access, add, edit and remove files easily.
+.. option:: --single-changes-list
-*-*-single-changes-list*::
- When processing changes files, piuparts will process the packages in each individual changes file separately. This option will set piuparts to scan the packages of all changes files together along with any individual package files that may have been given on the command line.
+ When processing changes files, :program:`piuparts` will process the packages in each individual changes file separately. This option will set :program:`piuparts` to scan the packages of all changes files together along with any individual package files that may have been given on the command line.
-*-*-single-packages*::
- Process every package file or package name individually, thus piuparts process runs multiple times. This option can be useful with conflicting packages.
+.. option:: --single-packages
+
+ Process every package file or package name individually, thus :program:`piuparts` process runs multiple times. This option can be useful with conflicting packages.
+
+.. option:: --shell-on-error
-*-*-shell-on-error*::
Start an interactive shell in the chroot after an error occurred.
- This should help debugging failures directly inside the piuparts test environment.
+ This should help debugging failures directly inside the :program:`piuparts` test environment.
The chroot cleanup will continue after the shell terminates.
- Note: This does not work if the piuparts command is prefixed with 'timeout',
+ Note: This does not work if the :program:`piuparts` command is prefixed with 'timeout',
which is usually the case in command lines directly copied from logfiles
from a master-slave setup. Removing the 'timeout' part is sufficient.
-*-*-skip-minimize*::
- Allow skip minimize chroot step. This is useful when you want to test several packages with piuparts. You can prepare a tarball already minimized and skip this step in all the tests. This is the default now.
+.. option:: --skip-minimize
+
+ Allow skip minimize chroot step. This is useful when you want to test several packages with :program:`piuparts`. You can prepare a tarball already minimized and skip this step in all the tests. This is the default now.
+
+.. option:: --minimize
+
+ Minimize the chroot with debfoster. This used to be the default until `#539142 <https://bugs.debian.org/539142>`_ was fixed.
-*-*-minimize*::
- Minimize the chroot with debfoster. This used to be the default until #539142 was fixed.
+.. option:: --skip-cronfiles-test
-*-*-skip-cronfiles-test*::
Skip testing the output from the cron files left in the system after remove a package.
-*-*-skip-logrotatefiles-test*::
+.. option:: --skip-logrotatefiles-test
+
Skip testing the output from the logrotate files left in the system after remove a package.
-*-*-testdebs-repo*='deb-line'::
- Provide an additional line to be appended to sources.list, e.g.::
+.. option:: --testdebs-repo deb-line
+
+ Provide an additional line to be appended to sources.list, for example::
'deb [ trusted=yes ] <URL> <distrib> <components>'
or
'deb [ trusted=yes ] file:// </bind/mount> ./'
- If only an URL or local path is given as argument, "deb [ trusted=yes ]",
- "file://", and "./" will be prepended/appended as needed.
+ If only an URL or local path is given as argument, ``deb [ trusted=yes ]``,
+ ``file://``, and ``./`` will be prepended/appended as needed.
The "testdebs" repository provides the packages to be tested (and some
additional dependencies, if needed, e.g. all packages built from the same
source package as the (binary) package being tested) and can be used for
testing complex installation and upgrade scenarios involving dependencies
that are not yet in the archive. This repository will be available only
for installing the target packages. Dependency resolution will be done by
- apt-get. The packages to be tested can be passed as .debs or as package
- names (with '--apt').
+ :program:`apt-get`. The packages to be tested can be passed as ``.deb``\ s or as package
+ names (with :option:`--apt`).
- The "trusted=yes" option causes this (and only this) repository to be
+ The ``trusted=yes`` option causes this (and only this) repository to be
trustworthy even if the Packages file is not signed, such that a
- (globally acting) '--do-not-verify-signatures' will not be needed.
+ (globally acting) :option:`--do-not-verify-signatures` will not be needed.
+
+.. option:: -t directory, --tmpdir directory
+
+ Use *directory* as the place where temporary files and directories are created. The default is the environment variable :envvar:`TMPDIR`, or :file:`/tmp` if not set.
+ Note: the temporary directory must *not* be mounted with the *nodev* or *nosuid* mount option.
-*-t directory*, *-*-tmpdir*='directory'::
- Use directory as the place where temporary files and directories are created. The default is the environment variable *TMPDIR*, or */tmp* if not set.
- Note: the temporary directory must *not* be mounted with the _nodev_ or _nosuid_ mount option.
+.. option:: --update-retries num-retries
-*-*-update-retries*='num-retries'::
- Rerun 'apt-get update` up to "num-retries" times.
+ Rerun :command:`apt-get update` up to *num-retries* times.
Useful to work around temporary network failures and hashsum mismatch errors.
-*-*-upgrade-before-dist-upgrade*::
- Perform two-stage upgrades: 'apt-get upgrade && apt-get dist-upgrade'.
+.. option:: --upgrade-before-dist-upgrade
+
+ Perform two-stage upgrades: :command:`apt-get upgrade && apt-get dist-upgrade`.
+
+.. option:: -v, --verbose
-*-v*, *-*-verbose*::
This option no longer has any meaning, but it is still accepted for backwards compatibility.
-*-V*, *-*-version*::
+.. option:: -V, --version
+
Write out the version number of the program.
-*-*-warn-on-debsums-errors*::
+.. option:: --warn-on-debsums-errors
+
Print a warning rather than failing if debsums reports modified files.
-*-*-warn-on-leftovers-after-purge*::
+.. option:: --warn-on-leftovers-after-purge
+
Print a warning rather than failing if files are left behind after purge.
-*-*-warn-on-others*::
+.. option:: --warn-on-others
+
Print a warning rather than failing if files are left behind, modified, or removed by a package that was not given on the command-line.
This way, you can basically isolate the purge test to your own packages. If a package that is brought in as a dependency doesn't purge cleanly, the test will not fail because of it (but a warning message will be printed).
Behavior with multiple packages given on the command-line could be problematic, particularly if the dependency tree of one package in the list includes another in the list. Therefore, it is recommended to use this option with one package at a time.
+.. option:: --warn-on-usr-move disabled|warn|fail
+
+ Whether to enable the test (with a warning or a failure) that checks if files are moved between :file:`/\{bin|sbin|lib*\}` and :file:`/usr/\{bin|sbin|lib*\}`.
+ Accepted values: *disabled* (default), *warn*, *fail*.
+
EXAMPLES
--------
-Assume that you have just built a new version of your Debian package, to be uploaded to Debian unstable. It is in *../foo_1.0-2_i386.deb* and you would like to know whether it installs and uninstalls properly. Here's what you would do::
+Assume that you have just built a new version of your Debian package, to be uploaded to Debian unstable. It is in ``../foo_1.0-2_i386.deb`` and you would like to know whether it installs and uninstalls properly. Here's what you would do::
- ----
- piuparts ../foo_1.0-2_i386.deb
- ----
+ piuparts ../foo_1.0-2_i386.deb
If the package exists in the Debian archive already, the above command also tests that it upgrades properly.
To do the same test, but using a particular mirror, and only the main component, you would do this::
- ----
- piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.deb
- ----
+ piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.deb
-If you want to do the same as above but for your changes files, pass in your changes files when running piuparts, and piuparts will process each package in the changes files as though you had passed all those packages on the command line to piuparts yourself. For example::
+If you want to do the same as above but for your changes files, pass in your changes files when running :program:`piuparts`, and :program:`piuparts` will process each package in the changes files as though you had passed all those packages on the command line to :program:`piuparts` yourself. For example::
- ----
- piuparts ../foo_1.0-2_i386.changes
+ piuparts ../foo_1.0-2_i386.changes
- piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.changes
- ----
+ piuparts -m 'http://gytha/debian main' ../foo_1.0-2_i386.changes
If you want to test that a package installs properly in the stable (currently bullseye) Debian release, then can be upgraded to the testing (currently bookworm) and unstable (sid) versions, and then uninstalled without problems, you would give the following command::
- ----
- piuparts -a -d bullseye -d bookworm -d sid foo
- ----
+ piuparts -a -d bullseye -d bookworm -d sid foo
ENVIRONMENT
-----------
-*TMPDIR* Location for temporary files and directories. If not set, use */tmp*. See also the '-t' ('--tmpdir') option.
+.. envvar:: TMPDIR
+
+ Location for temporary files and directories. If not set, use :file:`/tmp`. See also the :option:`--tmpdir` option.
NOTES
-----
-Output of commands run by piuparts is limited to three megabytes. To change this limit, the source code needs to be edited.
+Output of commands run by :program:`piuparts` is limited to three megabytes. To change this limit, the source code needs to be edited.
Commands exceeding this limit will be aborted.
SEE ALSO
--------
-*pbuilder* (1), *debootstrap* (8)
+:manpage:`pbuilder(1)`, :manpage:`debootstrap(8)`
AUTHOR
------
=====================================
instances/piuparts.conf-template.pejacevic
=====================================
@@ -40,15 +40,18 @@ flags-end-oldstable = %(flags-end-buster)s
# common flags for tests starting in bookworm
flags-start-bookworm =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests ending in bookworm
flags-end-bookworm =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests starting in bullseye
flags-start-bullseye =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests ending in bullseye
flags-end-bullseye =
=====================================
known_problems/file_moved_usr_error.conf
=====================================
@@ -0,0 +1,13 @@
+# detect a file moving between bin/sbin/lib* and usr/bin|sbin|lib*
+#
+PATTERN='(WARN|FAIL): File\(s\) moved between /.* and /usr/.*:'
+WHERE='fail bugged affected'
+ISSUE=0
+HEADER='File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}'
+HELPTEXT='
+<p>
+The Technical Committee recommended against moving files between between /{bin|sbin|lib*}
+and /usr/{bin|sbin|lib*} during the Bookworm development cycle. See
+<a href="https://bugs.debian.org/994388#80">Debian bug #994388.</a>
+</p>
+'
=====================================
known_problems/file_moved_usr_issue.conf
=====================================
@@ -0,0 +1,13 @@
+# detect a file moving between bin/sbin/lib* and usr/bin|sbin|lib*
+#
+PATTERN='(WARN|FAIL): File\(s\) moved between /.* and /usr/.*:'
+WHERE='pass'
+ISSUE=1
+HEADER='File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}'
+HELPTEXT='
+<p>
+The Technical Committee recommended against moving files between between /{bin|sbin|lib*}
+and /usr/{bin|sbin|lib*} during the Bookworm development cycle. See
+<a href="https://bugs.debian.org/994388#80">Debian bug #994388.</a>
+</p>
+'
=====================================
piuparts-report.py
=====================================
@@ -501,6 +501,8 @@ linktarget_by_template = [
("missing_md5sums_error.tpl", "...and logfile reports missing md5sums"),
("unowned_lib_symlink_error.tpl", "...and logfile reports unowned lib symlinks"),
("piuparts-depends-dummy_error.tpl", "...and logfile reports piuparts-depends-dummy.deb could not be installed"),
+ ("file_moved_usr_error,tpl", "...and logfile reports a file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"),
+ ("file_moved_usr_issue,tpl", "but logfile reports a file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"),
("unclassified_failures.tpl", "due to unclassified failures"),
]
=====================================
piuparts-slave.py
=====================================
@@ -24,17 +24,18 @@ Lars Wirzenius <liw at iki.fi>
"""
from __future__ import print_function
+import fcntl
+import logging
import os
-import sys
+import random
+import shlex
import stat
-import time
-import logging
-from signal import alarm, signal, SIGALRM, SIGINT, SIGKILL, SIGHUP, SIGUSR1
import subprocess
-import fcntl
-import random
+import sys
+import time
+from signal import SIGALRM, SIGHUP, SIGINT, SIGKILL, SIGUSR1, alarm, signal
+
import apt_pkg
-import pipes
import piupartslib.conf
import piupartslib.packagesdb
@@ -851,7 +852,7 @@ def log_name(package, version):
def command2string(command):
"""Quote s.t. copy+paste from the logfile gives a runnable command in the shell."""
- return " ".join([pipes.quote(arg) for arg in command])
+ return " ".join([shlex.quote(arg) for arg in command])
def run_test_with_timeout(cmd, maxwait, kill_all=True):
=====================================
piuparts.py
=====================================
@@ -2,7 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright 2005 Lars Wirzenius (liw at iki.fi)
-# Copyright © 2007-2018 Holger Levsen (holger at layer-acht.org)
+# Copyright © 2007-2023 Holger Levsen (holger at layer-acht.org)
# Copyright © 2010-2019 Andreas Beckmann (anbe at debian.org)
#
# This program is free software; you can redistribute it and/or modify it
@@ -35,38 +35,38 @@ from __future__ import print_function
VERSION = "__PIUPARTS_VERSION__"
-import distro_info
-import time
+import json
import logging
import optparse
-import sys
-import tempfile
-import shutil
import os
-import tarfile
-import stat
-import re
-import json
+import pathlib
import pickle
+import re
+import shlex
+import shutil
+import stat
import subprocess
+import sys
+import tarfile
+import tempfile
+import time
import traceback
import uuid
-import apt_pkg
-import pipes
from collections import namedtuple
-from signal import alarm, signal, SIGALRM, SIGTERM, SIGKILL
+from signal import SIGALRM, SIGKILL, SIGTERM, alarm, signal
+
+import apt_pkg
+import distro_info
try:
from debian import deb822
except ImportError:
from debian_bundle import deb822
-import piupartslib.conf
-
-from six.moves import urllib
-
import six
+from six.moves import urllib
+import piupartslib.conf
apt_pkg.init_system()
@@ -101,7 +101,7 @@ class Defaults:
class DebianDefaults(Defaults):
def get_components(self):
- return ["main", "contrib", "non-free"]
+ return ["main", "contrib", "non-free", "non-free-firmware"]
def get_mirror(self):
return [("http://deb.debian.org/debian", self.get_components())]
@@ -220,6 +220,7 @@ class Settings:
self.warn_on_debsums_errors = False
self.warn_on_install_over_symlink = False
self.warn_if_inadequate = True
+ self.warn_on_usr_move = "disabled"
self.pedantic_purge_test = False
self.ignored_files = [
# /root/.rnd should *not* be listed here, see #750099
@@ -504,7 +505,7 @@ def indent_string(str):
def command2string(command):
"""Quote s.t. copy+paste from the logfile gives a runnable command in the shell."""
- return " ".join([pipes.quote(arg) for arg in command])
+ return " ".join([shlex.quote(arg) for arg in command])
def unqualify(packages):
@@ -1833,6 +1834,45 @@ class Chroot:
return True
return False
+ def check_files_moved_usr(self, packages=[], files_before={}, files_after={}, warn_only=None):
+ """Check that no files were moved from /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"""
+
+ if settings.warn_on_usr_move == "disabled":
+ return
+
+ # For each path that is a file in each package, check that it did not move between
+ # /bin, /sbin or /lib* to the corresponding location under /usr, and viceversa.
+ # See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ # If a file moved in a package that we are inspecting, print an error by default.
+ # Otherwise, print a warning.
+ broken = []
+ for old_path in files_before:
+ # '/' is a separate element in the parts list
+ old_path_parts = pathlib.Path(old_path).parts
+ if len(old_path_parts) < 3:
+ continue
+
+ if old_path_parts[1] == 'usr' and (old_path_parts[2] in ['bin', 'sbin'] or old_path_parts[2].startswith('lib')):
+ new_path = os.path.join("/", *old_path_parts[2:])
+ elif old_path_parts[1] in ['bin', 'sbin'] or old_path_parts[1].startswith('lib'):
+ new_path = "/usr" + old_path
+ else:
+ continue
+
+ # Skip over directories, multiple packages can ship files in the same directories
+ if new_path in files_after and os.path.isfile(self.relative(new_path)):
+ broken.append("%s %s => %s %s" % (old_path, files_before[old_path], new_path, files_after[new_path]))
+
+ if broken:
+ if settings.warn_on_usr_move == "warn" or warn_only:
+ logging.warning("WARN: File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}: %s" % indent_string("\n".join(broken)))
+ else:
+ logging.error("FAIL: File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}: %s" % indent_string("\n".join(broken)))
+ panic()
+ else:
+ logging.debug("No file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}.")
+
+
def check_for_broken_symlinks(self, warn_only=None, file_owners={}):
"""Check that all symlinks in chroot are non-broken."""
if not settings.check_broken_symlinks:
@@ -2456,6 +2496,9 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.check_for_no_processes()
chroot.check_for_broken_symlinks()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_before = chroot.get_files_owned_by_packages()
+
if settings.install_remove_install:
chroot.remove_packages(packages, ignore_errors=True)
@@ -2468,10 +2511,12 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.disable_testdebs_repo()
- file_owners = chroot.get_files_owned_by_packages()
+ file_owners_after = chroot.get_files_owned_by_packages()
chroot.check_for_no_processes()
- chroot.check_for_broken_symlinks(file_owners=file_owners)
+ chroot.check_for_broken_symlinks(file_owners=file_owners_after)
+ if settings.warn_on_usr_move != "disabled":
+ chroot.check_files_moved_usr(packages, file_owners_before, file_owners_after)
# Remove all packages from the chroot that weren't there initially.
chroot.restore_selections(chroot_state, packages)
@@ -2479,9 +2524,9 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.run_scripts("post_test")
chroot.check_for_no_processes(fail=True)
- chroot.check_for_broken_symlinks(file_owners=file_owners)
+ chroot.check_for_broken_symlinks(file_owners=file_owners_after)
- return check_results(chroot, chroot_state, file_owners)
+ return check_results(chroot, chroot_state, file_owners_after)
def save_meta_data(filename, chroot_state):
@@ -2610,6 +2655,9 @@ def install_and_upgrade_between_distros(package_files, packages_qualified):
chroot.check_for_no_processes()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_before = chroot.get_files_owned_by_packages()
+
os.environ["PIUPARTS_PHASE"] = "distupgrade"
chroot.upgrade_to_distros(settings.debian_distros[1:-1], distupgrade_packages, settings.upgrade_before_dist_upgrade)
@@ -2634,6 +2682,10 @@ def install_and_upgrade_between_distros(package_files, packages_qualified):
chroot.check_for_no_processes()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_after = chroot.get_files_owned_by_packages()
+ chroot.check_files_moved_usr(packages, files_before=file_owners_before, files_after=file_owners_after)
+
# Remove all packages from the chroot that weren't in the reference chroot.
chroot.restore_selections(chroot_state, packages_qualified)
@@ -3008,6 +3060,10 @@ def parse_command_line():
default=False,
help="Fail if broken symlinks are detected.")
+ parser.add_option("--warn-on-usr-move", action="store", default="disabled",
+ help="Whether to enable the test (with a warning or a failure) that checks if files are moved "
+ "between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}. Accepted values: 'disabled' (default), 'warn', 'fail'.")
+
parser.add_option("--log-level", action="store", metavar='LEVEL',
default="dump",
help="Displays messages from LEVEL level, possible values are: error, info, dump, debug. The default is dump.")
@@ -3103,6 +3159,7 @@ def parse_command_line():
settings.warn_on_debsums_errors = opts.warn_on_debsums_errors
settings.warn_on_install_over_symlink = opts.warn_on_install_over_symlink
settings.warn_if_inadequate = not opts.fail_if_inadequate
+ settings.warn_on_usr_move = opts.warn_on_usr_move
settings.pedantic_purge_test = opts.pedantic_purge_test
settings.ignored_files += opts.ignore
settings.ignored_patterns += opts.ignore_regex
@@ -3138,6 +3195,10 @@ def parse_command_line():
logging.error("Scripts directory is not a directory: %s" % sdir)
panic()
+ if settings.warn_on_usr_move not in ["disabled", "warn", "fail"]:
+ logging.error("--warn-on-usr-move must be one of 'disabled', 'warn', 'fail'")
+ panic()
+
if not settings.debian_distros:
settings.debian_distros = defaults.get_distribution()
=====================================
tests/test_piuparts.py
=====================================
@@ -1,63 +1,69 @@
-import unittest
-from mox3 import mox
import os
import shutil
+import unittest
+from unittest.mock import patch
+
import piuparts
from piuparts import is_broken_symlink
class DefaultsFactoryTests(unittest.TestCase):
-
def setUp(self):
- self.mox = mox.Mox()
self.df = piuparts.DefaultsFactory()
piuparts.settings = piuparts.Settings()
- def tearDown(self):
- self.mox.UnsetStubs()
-
def test_new_defaults_return_debian_defaults(self):
# mock the guess_flavor function as it runs lsb_release in a subprocess
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('debian')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/debian-archive-keyring.gpg')
- self.assertEqual(defaults.get_components(), ["main", "contrib", "non-free"])
- self.assertEqual(defaults.get_mirror(), [("http://deb.debian.org/debian", ["main", "contrib", "non-free"])])
- self.assertEqual(defaults.get_distribution(), ['sid'])
+ with patch.object(
+ self.df, "guess_flavor", return_value="debian"
+ ) as guess_flavor_mock:
+ defaults = self.df.new_defaults()
+ guess_flavor_mock.assert_called_once()
+
+ self.assertEqual(
+ defaults.get_keyring(), "/usr/share/keyrings/debian-archive-keyring.gpg"
+ )
+ self.assertEqual(defaults.get_components(), ["main", "contrib", "non-free", "non-free-firmware"])
+ self.assertEqual(
+ defaults.get_mirror(),
+ [("http://deb.debian.org/debian", ["main", "contrib", "non-free", "non-free-firmware"])],
+ )
+ self.assertEqual(defaults.get_distribution(), ["sid"])
def test_new_defaults_return_ubuntu_defaults(self):
- # mock the guess_flavor function as it runs lsb_release in a subprocess
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('ubuntu')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/ubuntu-archive-keyring.gpg')
- self.assertEqual(defaults.get_components(), ["main", "universe", "restricted", "multiverse"])
- self.assertEqual(defaults.get_mirror(), [("http://archive.ubuntu.com/ubuntu", ["main", "universe", "restricted", "multiverse"])])
+ with patch.object(
+ self.df, "guess_flavor", return_value="ubuntu"
+ ) as guess_flavor_mock:
+ defaults = self.df.new_defaults()
+ guess_flavor_mock.assert_called_once()
+
+ self.assertEqual(
+ defaults.get_keyring(), "/usr/share/keyrings/ubuntu-archive-keyring.gpg"
+ )
+ self.assertEqual(
+ defaults.get_components(), ["main", "universe", "restricted", "multiverse"]
+ )
+ self.assertEqual(
+ defaults.get_mirror(),
+ [
+ (
+ "http://archive.ubuntu.com/ubuntu",
+ ["main", "universe", "restricted", "multiverse"],
+ )
+ ],
+ )
def test_new_defaults_panics_with_unknown_flavor(self):
- # mock the guess_flavor function as it runs lsb_release in a subprocess
- # and the panic function as it would use sys.exit()
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('centos')
- self.mox.StubOutWithMock(piuparts, 'panic')
- piuparts.panic().AndReturn('Oh dear! Its CentOS!')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- # panic() would cause sys.exit() so no Defaults object would
- # ever be returned
- self.assertEqual(defaults, None)
+ with patch.object(
+ self.df, "guess_flavor", return_value="centos"
+ ) as guess_flavor_mock, patch.object(
+ piuparts, "panic", side_effect=SystemExit
+ ) as panic_mock:
+ with self.assertRaises(SystemExit):
+ self.df.new_defaults()
+
+ guess_flavor_mock.assert_called_once()
+ panic_mock.assert_called_once()
class IsBrokenSymlinkTests(unittest.TestCase):
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/e18b442fe3f202bc6a7edd53382c76b4493fde3b...f1252503d21b6942164d2e8abd36ed772dfe2ddd
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/e18b442fe3f202bc6a7edd53382c76b4493fde3b...f1252503d21b6942164d2e8abd36ed772dfe2ddd
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20230524/5322df46/attachment-0001.htm>
More information about the Piuparts-devel
mailing list