[Piuparts-devel] [Git][debian/piuparts][phahn/docker-fixes] 33 commits: Remove oldold*stable dists from json summary
Philipp Hahn (@pmhahn)
gitlab at salsa.debian.org
Tue Sep 19 08:25:42 BST 2023
Philipp Hahn pushed to branch phahn/docker-fixes at Debian / piuparts
Commits:
adfef741 by David Steele at 2021-10-23T12:50:11-04:00
Remove oldold*stable dists from json summary
Among other things, distro_info reports them as "unknown".
- - - - -
24e89b75 by Holger Levsen at 2022-02-02T10:12:37+01:00
custom-scripts/scripts/pre_test_root_password: fix grammar in comment
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
190399be by Holger Levsen at 2022-02-02T10:13:31+01:00
custom-scripts/scripts/pre_test_root_password: further improve grammar in comment
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e7538053 by Luca Boccassi at 2022-07-13T10:36:42+01:00
piuparts: detect files moving between / and /usr on upgrade
The TC issued a recommendation to avoid moving files between
bin|sbin|lib* and /usr/bin|sbin|lib* for the Bookworm cycle.
Detect such moves and raise an error.
For more details see:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
- - - - -
4489752a by Luca Boccassi at 2022-07-13T10:36:42+01:00
piuparts: detect files moving between / and /usr on dist-upgrade
Same as the check on package install/upgrade, but across distribution
dist-upgrades.
- - - - -
59c00b8b by Luca Boccassi at 2022-07-13T10:36:42+01:00
p.conf: enable --warn-on-usr-move for bullseye/bookworm
- - - - -
92c1faff by Nicolas Dandrimont at 2022-07-13T20:06:49+02:00
Merge branch 'usrmerge' into develop
piuparts: detect files moving between / and /usr on upgrade
See merge request https://salsa.debian.org/debian/piuparts/-/merge_requests/37
- - - - -
98b4fdcd by Luca Boccassi at 2022-07-17T20:17:08+01:00
custom_scripts: ensure usr-is-merged/usrmerge does not fail the migration test
The new usr-is-merged metapackage intentionally fails to install
on systems that are not merged-usr, unless a specific override
flag is passed.
usrmerge does a live-upgrade of the chroot, which piuparts does not
expect.
Use the skip flag, so that migration is not blocked.
Debootstrap will start using this flag once the package is
available in testing. For details see:
https://sources.debian.org/src/usrmerge/27/debian/README.Debian/#L66
and:
https://lists.debian.org/debian-ctte/2022/07/msg00019.html
- - - - -
c3352499 by Nicolas Dandrimont at 2022-07-17T22:06:03+02:00
Merge branch 'usr-is-merged' into 'develop'
custom_scripts: ensure usr-is-merged does not fail the migration test
See merge request https://salsa.debian.org/debian/piuparts/-/merge_requests/38
- - - - -
67a85a4f by David Steele at 2022-11-02T13:14:39-04:00
Add Stable Release tasks to README_pejacevic.txt
... to update summary.json for the new distribution environment.
- - - - -
3331b140 by Nicolas Dandrimont at 2023-01-11T11:35:00+01:00
piuparts-slave: run isort
- - - - -
ad7e30b3 by Nicolas Dandrimont at 2023-01-11T11:59:39+01:00
piuparts-slave: replace pipes.quote with shlex.quote
- - - - -
b9ca98d1 by Nicolas Dandrimont at 2023-01-11T11:59:52+01:00
piuparts.py: run isort
- - - - -
fc0047c9 by Nicolas Dandrimont at 2023-01-11T11:59:52+01:00
piuparts.py: replace pipes.quote with shlex.quote
- - - - -
2cfae2aa by Nicolas Dandrimont at 2023-01-11T12:00:00+01:00
test_piuparts: replace mox3 with unittest.mock
Closes: #1028453
- - - - -
2fd10804 by Nicolas Dandrimont at 2023-01-11T12:05:53+01:00
Migrate lintian overrides to pointed hints
- - - - -
cd45cc8f by Nicolas Dandrimont at 2023-01-11T13:19:53+01:00
Simplify docs install in Makefile
- - - - -
224e9beb by Nicolas Dandrimont at 2023-01-11T13:25:39+01:00
Use sphinxdoc dh plugin
- - - - -
0ac90edf by Nicolas Dandrimont at 2023-01-11T13:25:39+01:00
Update changelog
- - - - -
fbca117e by Nicolas Dandrimont at 2023-01-11T13:31:48+01:00
Release to unstable
- - - - -
5b5a213d by Holger Levsen at 2023-01-11T15:57:37+01:00
Start 1.1.7 development.
d/changelog entries will be written on release
using the git commit messages.
Use 'gbp dch --since debian/1.1.6'
to write d/changelog entries since that last release.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1190e637 by Holger Levsen at 2023-01-11T15:58:23+01:00
Bump standards version to 4.6.2, no changes needed.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
320b7cbb by Nicolas Dandrimont at 2023-01-11T16:15:00+01:00
Makefile: use dpkg-parsechangelog -S instead of sed
- - - - -
f5dc1171 by Holger Levsen at 2023-01-18T11:25:49+01:00
Add debian/source/lintian-overrides to override a bunch of very-long-line-length-in-source-file warnings
I fail to understand why the 4th one isn't working.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
c08f7a71 by Holger Levsen at 2023-01-18T12:32:10+01:00
d/control: fix cute-field warning, thanks lintian.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
838f800f by Holger Levsen at 2023-01-18T13:23:50+01:00
Bump standards version to 4.6.2, no changes needed.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
d4c2447a by Holger Levsen at 2023-01-18T13:37:13+01:00
d/piuparts.lintian-overrides: add debian-news-entry-has-unknown-version 0.45 due to trimmed d/chanelog in binary packages
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
bb6dd5b1 by Holger Levsen at 2023-01-18T13:43:42+01:00
Drop unused d./piuparts-(master|slave).lintian-overrides
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
25138b9b by Holger Levsen at 2023-01-18T15:50:17+01:00
d/control: remove version constraints on the depens on python3-debianbts, debootstrap amd debsums as they are met since (at least) buster
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
402a97af by Holger Levsen at 2023-01-18T20:58:09+01:00
piuparts.py, tests and docs: handle new "non-free-firmware" section in Debian. Closes: #1028661.
(Strictly speaking this is only used for bookworm, unstable, experimental and
later suites like trixie, but specifying which suite has which sections is
out of scope for piuparts itself.)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
8c272296 by Holger Levsen at 2023-01-18T22:45:24+01:00
release as 1.1.7
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1f5e0b54 by Philipp Hahn at 2023-09-19T09:23:07+02:00
fix(docker) getting Container ID
in case the image does not exists locally docker will pull it from any
remote registry, but will include the progress information in the
output:
> Unable to find image 'docker-registry.XXX/YYY:ZZZ' locally
> ZZZ: Pulling from YYY
...
> Digest: sha256:4b9bfe7b0a6c970e3613c04f267ba6319cfceb8cc120b0435d9ee7b8037a1f06
> Status: Downloaded newer image for docker-registry.XXX/YYY:ZZZ
> 8be38c89d12937b98c8be5ab7466dd45b0e4a306862f282b58077ac7193251eb
The old code expected the output to be the container ID, which fails in that case.
- - - - -
482c6e48 by Philipp Hahn at 2023-09-19T09:23:12+02:00
refactor(docker) Let `docker inspect` return path
Directly use golang templating to only get the MergedDir path instead of
using JSON in Python.
Include path in debug output.
Let logging.debug() handle the variable substitution.
- - - - -
21 changed files:
- Makefile
- custom-scripts/scripts/post_purge_exceptions
- custom-scripts/scripts/pre_install_exceptions
- custom-scripts/scripts/pre_test_root_password
- debian/changelog
- debian/control
- − debian/piuparts-master.lintian-overrides
- − debian/piuparts-slave.lintian-overrides
- debian/piuparts.lintian-overrides
- debian/rules
- + debian/source/lintian-overrides
- docs/README_pejacevic.txt
- docs/README_server.txt
- docs/piuparts/piuparts.1.txt
- instances/piuparts.conf-template.pejacevic
- + known_problems/file_moved_usr_error.conf
- + known_problems/file_moved_usr_issue.conf
- piuparts-report.py
- piuparts-slave.py
- piuparts.py
- tests/test_piuparts.py
Changes:
=====================================
Makefile
=====================================
@@ -10,11 +10,11 @@ site3 = $(libdir)/python3/dist-packages
htdocsdir = $(sharedir)/piuparts/htdocs
etcdir = $(prefix)/etc
-distribution=${shell dpkg-parsechangelog | sed -n 's/^Distribution: *//p'}
+distribution=${shell dpkg-parsechangelog -SDistribution}
ifeq ($(distribution),UNRELEASED)
-version := ${shell echo "`dpkg-parsechangelog | sed -n 's/^Version: *//p'`~`date +%Y%m%d%H%M`~`git describe --dirty`"}
+version := ${shell echo "`dpkg-parsechangelog -SVersion`~`date +%Y%m%d%H%M`~`git describe --dirty`"}
else
-version := ${shell dpkg-parsechangelog | sed -n 's/^Version: *//p'}
+version := ${shell dpkg-parsechangelog -SVersion}
endif
@@ -83,42 +83,7 @@ install-doc: build-stamp
install -d $(DESTDIR)$(docdir)/
install -m 0644 docs/README.txt docs/README_server.txt $(DESTDIR)$(docdir)/
# html
- install -d $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/*.html $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/searchindex.js $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/objects.inv $(DESTDIR)$(docdir)/html/
- install -d $(DESTDIR)$(docdir)/html/_static/
- install -m 0644 docs/build/_static/* $(DESTDIR)$(docdir)/html/_static
- install -d $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/index.html $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/piuparts.1.html $(DESTDIR)$(docdir)/html/piuparts/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/piuparts_slave_run.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/piuparts_slave_join.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/piuparts_slave_stop.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -d $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/*.html $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/searchindex.js $(DESTDIR)$(docdir)/html/
- install -m 0644 docs/build/objects.inv $(DESTDIR)$(docdir)/html/
- install -d $(DESTDIR)$(docdir)/html/_static/
- install -m 0644 docs/build/_static/* $(DESTDIR)$(docdir)/html/_static
- install -d $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/index.html $(DESTDIR)$(docdir)/html/piuparts/
- install -m 0644 docs/build/piuparts/piuparts.1.html $(DESTDIR)$(docdir)/html/piuparts/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -m 0644 docs/build/piuparts_slave_run/piuparts_slave_run.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_run/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -m 0644 docs/build/piuparts_slave_join/piuparts_slave_join.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_join/
- install -d $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/index.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
- install -m 0644 docs/build/piuparts_slave_stop/piuparts_slave_stop.8.html $(DESTDIR)$(docdir)/html/piuparts_slave_stop/
+ cp -r docs/build/ $(DESTDIR)$(docdir)/html/
# manpages
install -d $(DESTDIR)$(man1dir)
install -m 0644 piuparts.1 $(DESTDIR)$(man1dir)/
=====================================
custom-scripts/scripts/post_purge_exceptions
=====================================
@@ -53,6 +53,13 @@ case ${PIUPARTS_OBJECTS%%=*} in
log_debug
sleep 10
;;
+ *usr-is-merged*|*usrmerge*)
+ # Remove the exception flag that we added in pre_install_exceptions, if we added it
+ if [ -f /etc/unsupported-skip-usrmerge-conversion.piuparts ]; then
+ log_debug
+ rm -f /etc/unsupported-skip-usrmerge-conversion*
+ fi
+ ;;
esac
if [ -f /var/run/piuparts-stamp-dkms-installed ]; then
=====================================
custom-scripts/scripts/pre_install_exceptions
=====================================
@@ -106,6 +106,20 @@ case ${PIUPARTS_OBJECTS%%=*} in
mkdir -p /var/state/samhain/
touch /var/state/samhain/samhain_file
;;
+ *usr-is-merged*|*usrmerge*)
+ # usr-is-merged intentionally fails to install unless either the system is merged-usr, or
+ # the exception flag is set.
+ # usrmerge does a live-upgrade of the chroot, which piuparts does not expect, so skip it.
+ # debootstrap will take care of this in a future update, so remember if we were the ones
+ # that added the flag file, to remove it in the post_purge_exceptions script.
+ # For details on the flag file, see: https://sources.debian.org/src/usrmerge/27/debian/README.Debian/#L66
+ # and: https://lists.debian.org/debian-ctte/2022/07/msg00019.html
+ if [ ! -f /etc/unsupported-skip-usrmerge-conversion ]; then
+ log_debug
+ touch /etc/unsupported-skip-usrmerge-conversion.piuparts
+ echo "this system will not be supported in the future" > /etc/unsupported-skip-usrmerge-conversion
+ fi
+ ;;
esac
if [ "$PIUPARTS_DISTRIBUTION" = "jessie" ]; then
=====================================
custom-scripts/scripts/pre_test_root_password
=====================================
@@ -2,6 +2,6 @@
set -e
# sudo refuses removal if no root password is set, so set one
-# do this unconditionally, as there are quite some packages depending on sudo
+# unconditionally, as there are quite some packages depending on sudo
# (and since its harmless and fast)
yes "yes" 2>/dev/null | passwd
=====================================
debian/changelog
=====================================
@@ -1,3 +1,54 @@
+piuparts (1.1.7) unstable; urgency=medium
+
+ [ Holger Levsen ]
+ * piuparts.py, tests and docs: handle new "non-free-firmware" section in
+ Debian. Closes: #1028661.
+ * d/control:
+ - remove version constraints on the (build-)depends on python3-debianbts,
+ debootstrap amd debsums as they are met since (at least) buster.
+ - fix cute-field warning, thanks lintian.
+ - bump standards version to 4.6.2, no changes needed.
+ * Add debian/source/lintian-overrides to override a bunch of very-long-line-
+ length-in-source-file warnings.
+ * d/piuparts.lintian-overrides: add debian-news-entry-has-unknown-version
+ 0.45 due to trimmed d/chanelog in binary packages.
+ * Drop unused d./piuparts-(master|slave).lintian-overrides.
+
+ [ Nicolas Dandrimont ]
+ * Makefile: use dpkg-parsechangelog -S instead of sed.
+
+ -- Holger Levsen <holger at debian.org> Wed, 18 Jan 2023 22:45:04 +0100
+
+piuparts (1.1.6) unstable; urgency=medium
+
+ [ David Steele ]
+ * Remove oldold*stable dists from json summary.
+
+ [ Holger Levsen ]
+ * custom-scripts/scripts/pre_test_root_password: fix grammar in comment.
+
+ [ Luca Boccassi ]
+ * piuparts: detect files moving between / and /usr on upgrade.
+ * piuparts: detect files moving between / and /usr on dist-upgrade.
+ * p.conf: enable --warn-on-usr-move for bullseye/bookworm.
+ * custom_scripts: ensure usr-is-merged/usrmerge does not fail the
+ migration test.
+
+ [ David Steele ]
+ * Add Stable Release tasks to README_pejacevic.txt.
+
+ [ Nicolas Dandrimont ]
+ * piuparts-slave: run isort.
+ * piuparts-slave: replace pipes.quote with shlex.quote.
+ * piuparts.py: run isort.
+ * piuparts.py: replace pipes.quote with shlex.quote.
+ * test_piuparts: replace mox3 with unittest.mock (Closes: #1028453)
+ * Migrate lintian overrides to pointed hints.
+ * Simplify docs install in Makefile.
+ * Use sphinxdoc dh plugin.
+
+ -- Nicolas Dandrimont <olasd at debian.org> Wed, 11 Jan 2023 13:31:41 +0100
+
piuparts (1.1.5) unstable; urgency=medium
[ Nicolas Dandrimont ]
@@ -10,8 +61,8 @@ piuparts (1.1.5) unstable; urgency=medium
* Remove oldstable results from summary.json.
[ Athos Ribeiro ]
- * d/rules: set GO111MODULE to auto to maintain pre Go 1.16 behavior. Closes: #991970.
- (MR: debian/piuparts!35)
+ * d/rules: set GO111MODULE to auto to maintain pre Go 1.16 behavior.
+ Closes: #991970. (MR: debian/piuparts!35)
-- Holger Levsen <holger at debian.org> Thu, 14 Oct 2021 15:23:26 +0200
=====================================
debian/control
=====================================
@@ -5,7 +5,7 @@ Uploaders: Nicolas Dandrimont <olasd at debian.org>,
Holger Levsen <holger at debian.org>
Section: devel
Priority: optional
-Standards-Version: 4.5.1
+Standards-Version: 4.6.2
Rules-Requires-Root: no
Build-Depends:
debhelper-compat (= 12),
@@ -13,9 +13,8 @@ Build-Depends:
python3-all,
python3-apt,
python3-debian,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-distro-info,
- python3-mox3,
python3-nose,
python3-six,
python3-sphinx,
@@ -26,22 +25,22 @@ Build-Depends-Indep:
Homepage: https://piuparts.debian.org
Vcs-Git: https://salsa.debian.org/debian/piuparts.git
Vcs-Browser: https://salsa.debian.org/debian/piuparts
-Xs-Go-Import-Path: salsa.debian.org/debian/piuparts
+XS-Go-Import-Path: salsa.debian.org/debian/piuparts
Package: piuparts
Architecture: all
Depends:
piuparts-common (= ${binary:Version}),
# keep this list in sync with piuparts-slave-from-git-deps
- debootstrap (>= 1.0.89~),
- debsums (>= 2.2.2~),
- libjs-sphinxdoc,
+ debootstrap,
+ debsums,
lsb-release,
lsof,
mount,
python3-debian,
${python3:Depends},
${misc:Depends},
+ ${sphinxdoc:Depends}
Recommends:
adequate
Suggests:
@@ -67,7 +66,7 @@ Depends:
# keep this list in sync with piuparts-master-from-git-deps
adduser,
openssh-server,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-setproctitle,
tango-icon-theme,
xz-utils,
@@ -100,7 +99,7 @@ Depends:
# this list is synced from piuparts-master
adduser,
openssh-server,
- python3-debianbts (>= 2.10.0),
+ python3-debianbts,
python3-setproctitle,
tango-icon-theme,
xz-utils,
@@ -157,8 +156,8 @@ Depends:
screen,
sudo,
# this list is synced from piuparts
- debootstrap (>= 1.0.89~),
- debsums (>= 2.2.2~),
+ debootstrap,
+ debsums,
libjs-sphinxdoc,
lsb-release,
lsof,
=====================================
debian/piuparts-master.lintian-overrides deleted
=====================================
@@ -1,5 +0,0 @@
-# It's ok, because it depends on piuparts-common that has the dependency.
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/master/detect_well_known_errors #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-analyze #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-master-backend #!/usr/bin/python3
-piuparts-master: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-report #!/usr/bin/python3
=====================================
debian/piuparts-slave.lintian-overrides deleted
=====================================
@@ -1,2 +0,0 @@
-# It's ok, because it depends on piuparts-common that has the dependency.
-piuparts-slave: python3-script-but-no-python3-dep usr/share/piuparts/piuparts-slave #!/usr/bin/python3
=====================================
debian/piuparts.lintian-overrides
=====================================
@@ -1,6 +1,7 @@
# it is impossible not to do this without losing compatibility with ancient releases
-uses-dpkg-database-directly etc/piuparts/scripts-debug-purge/post_remove_postrm_set-x
-uses-dpkg-database-directly etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x
-uses-dpkg-database-directly etc/piuparts/scripts/post_remove_exceptions
-uses-dpkg-database-directly etc/piuparts/scripts/pre_remove_40_find_missing_md5sums
-uses-dpkg-database-directly usr/sbin/piuparts
+uses-dpkg-database-directly [etc/piuparts/scripts-debug-purge/post_remove_postrm_set-x]
+uses-dpkg-database-directly [etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x]
+uses-dpkg-database-directly [etc/piuparts/scripts/post_remove_exceptions]
+uses-dpkg-database-directly [etc/piuparts/scripts/pre_remove_40_find_missing_md5sums]
+uses-dpkg-database-directly [usr/sbin/piuparts]
+debian-news-entry-has-unknown-version 0.45 [usr/share/doc/piuparts/NEWS.Debian.gz:1]
=====================================
debian/rules
=====================================
@@ -7,7 +7,7 @@ export GOCACHE := $(CURDIR)/.gocache
export GO111MODULE := auto
%:
- dh $@ --with python3
+ dh $@ --with python3,sphinxdoc
override_dh_auto_build:
$(MAKE) prefix=/usr build build-doc
=====================================
debian/source/lintian-overrides
=====================================
@@ -0,0 +1,4 @@
+very-long-line-length-in-source-file 650 > 512 [htdocs/news.tpl:214]
+very-long-line-length-in-source-file 671 > 512 [htdocs/index.tpl:46]
+very-long-line-length-in-source-file 777 > 512 [docs/piuparts/piuparts.1.txt:27]
+very-long-line-length-in-source-file 515 > 512 [known_problems/db_setup_issue.conf:4]
=====================================
docs/README_pejacevic.txt
=====================================
@@ -226,11 +226,27 @@ daily by '~piupartsm/crontab'.
More checks should be added as we become aware of them.
+:ref:`top <top3>`
+
+:blue:`New Stable Release`
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The server exports data, used by DDPO et al. to display test results by
+package. With a new release of Debian Stable, a number of tasks are required to
+update the summary data appropriately:
+
+- Edit 'piuparts.conf.pejacevic' to set 'json-sections = none' for the sections
+ that now represent oldstable ('foo', 'bar2foo', 'foo-proposed', etc.).
+- Update the 'distro-info-data' package, so that 'piuparts-report' can correctly
+ assign sections to distribution names.
+
+
:ref:`top <top3>`
:blue:`Authors`
^^^^^^^^^^^^^^^
-Last updated: February 2017
+Last updated: November 2022
Holger Levsen <holger at layer-acht.org>
+David Steele <steele at debian.org>
=====================================
docs/README_server.txt
=====================================
@@ -480,7 +480,7 @@ section, too, and will serve as defaults for all other sections
* "area" is the archive area used to get the list of packages to
be tested. The Packages file for this area will be loaded. The
default is "main" and the possible values depend on the vendor,
- for Debian these are main, contrib, non-free.
+ for Debian these are main, contrib, non-free and non-free-firmware.
* "components" sets the archive areas that will be available when
testing the packages selected via the "area" setting. These will
=====================================
docs/piuparts/piuparts.1.txt
=====================================
@@ -3,7 +3,7 @@
piuparts(1)
===========
:doctype: manpage
- :revdate: 2017-03-06
+ :revdate: 2023-01-18
@@ -189,7 +189,7 @@ Options must come before the other command line arguments.
*-m* 'url', *-*-mirror*='url'::
Which Debian mirror to use. The default is the first mirror named in */etc/apt/sources.list* or *http://deb.debian.org/debian* if none is found. This option may be used multiple times to use multiple mirrors. Only the first mirror is used with *debootstrap*.
- The 'components' that are used for a mirror can also be set with this option: a space separated list within the same argument (so you need to quote the entire argument in the shell). If no components are given explicitly, the usual Debian components are used (main, contrib, and non-free). For the mirrors read from */etc/apt/sources.list*, the components are read from the same place.
+ The 'components' that are used for a mirror can also be set with this option: a space separated list within the same argument (so you need to quote the entire argument in the shell). If no components are given explicitly, the usual Debian components are used (main, contrib, non-free and non-free-firmware). For the mirrors read from */etc/apt/sources.list*, the components are read from the same place.
Note that file: addresses works if the directories are made accessible from within the chroot with '--bindmount'.
@@ -330,6 +330,10 @@ Options must come before the other command line arguments.
Behavior with multiple packages given on the command-line could be problematic, particularly if the dependency tree of one package in the list includes another in the list. Therefore, it is recommended to use this option with one package at a time.
+*-*-warn-on-usr-move*='disabled|warn|fail'::
+ Whether to enable the test (with a warning or a failure) that checks if files are moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}.
+ Accepted values: 'disabled' (default), 'warn', 'fail'.
+
EXAMPLES
=====================================
instances/piuparts.conf-template.pejacevic
=====================================
@@ -40,15 +40,18 @@ flags-end-oldstable = %(flags-end-buster)s
# common flags for tests starting in bookworm
flags-start-bookworm =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests ending in bookworm
flags-end-bookworm =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests starting in bullseye
flags-start-bullseye =
-# no flags needed
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ --warn-on-usr-move fail
# common flags for tests ending in bullseye
flags-end-bullseye =
@@ -891,6 +894,7 @@ upgrade-test-distros = jessie jessie-backports
[jessie]
precedence = 7
+json-sections = none
piuparts-flags =
%(flags-default)s
%(flags-start-jessie)s
=====================================
known_problems/file_moved_usr_error.conf
=====================================
@@ -0,0 +1,13 @@
+# detect a file moving between bin/sbin/lib* and usr/bin|sbin|lib*
+#
+PATTERN='(WARN|FAIL): File\(s\) moved between /.* and /usr/.*:'
+WHERE='fail bugged affected'
+ISSUE=0
+HEADER='File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}'
+HELPTEXT='
+<p>
+The Technical Committee recommended against moving files between between /{bin|sbin|lib*}
+and /usr/{bin|sbin|lib*} during the Bookworm development cycle. See
+<a href="https://bugs.debian.org/994388#80">Debian bug #994388.</a>
+</p>
+'
=====================================
known_problems/file_moved_usr_issue.conf
=====================================
@@ -0,0 +1,13 @@
+# detect a file moving between bin/sbin/lib* and usr/bin|sbin|lib*
+#
+PATTERN='(WARN|FAIL): File\(s\) moved between /.* and /usr/.*:'
+WHERE='pass'
+ISSUE=1
+HEADER='File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}'
+HELPTEXT='
+<p>
+The Technical Committee recommended against moving files between between /{bin|sbin|lib*}
+and /usr/{bin|sbin|lib*} during the Bookworm development cycle. See
+<a href="https://bugs.debian.org/994388#80">Debian bug #994388.</a>
+</p>
+'
=====================================
piuparts-report.py
=====================================
@@ -501,6 +501,8 @@ linktarget_by_template = [
("missing_md5sums_error.tpl", "...and logfile reports missing md5sums"),
("unowned_lib_symlink_error.tpl", "...and logfile reports unowned lib symlinks"),
("piuparts-depends-dummy_error.tpl", "...and logfile reports piuparts-depends-dummy.deb could not be installed"),
+ ("file_moved_usr_error,tpl", "...and logfile reports a file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"),
+ ("file_moved_usr_issue,tpl", "but logfile reports a file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"),
("unclassified_failures.tpl", "due to unclassified failures"),
]
=====================================
piuparts-slave.py
=====================================
@@ -24,17 +24,18 @@ Lars Wirzenius <liw at iki.fi>
"""
from __future__ import print_function
+import fcntl
+import logging
import os
-import sys
+import random
+import shlex
import stat
-import time
-import logging
-from signal import alarm, signal, SIGALRM, SIGINT, SIGKILL, SIGHUP, SIGUSR1
import subprocess
-import fcntl
-import random
+import sys
+import time
+from signal import SIGALRM, SIGHUP, SIGINT, SIGKILL, SIGUSR1, alarm, signal
+
import apt_pkg
-import pipes
import piupartslib.conf
import piupartslib.packagesdb
@@ -851,7 +852,7 @@ def log_name(package, version):
def command2string(command):
"""Quote s.t. copy+paste from the logfile gives a runnable command in the shell."""
- return " ".join([pipes.quote(arg) for arg in command])
+ return " ".join([shlex.quote(arg) for arg in command])
def run_test_with_timeout(cmd, maxwait, kill_all=True):
=====================================
piuparts.py
=====================================
@@ -2,7 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright 2005 Lars Wirzenius (liw at iki.fi)
-# Copyright © 2007-2018 Holger Levsen (holger at layer-acht.org)
+# Copyright © 2007-2023 Holger Levsen (holger at layer-acht.org)
# Copyright © 2010-2019 Andreas Beckmann (anbe at debian.org)
#
# This program is free software; you can redistribute it and/or modify it
@@ -35,38 +35,37 @@ from __future__ import print_function
VERSION = "__PIUPARTS_VERSION__"
-import distro_info
-import time
import logging
import optparse
-import sys
-import tempfile
-import shutil
import os
-import tarfile
-import stat
-import re
-import json
+import pathlib
import pickle
+import re
+import shlex
+import shutil
+import stat
import subprocess
+import sys
+import tarfile
+import tempfile
+import time
import traceback
import uuid
-import apt_pkg
-import pipes
from collections import namedtuple
-from signal import alarm, signal, SIGALRM, SIGTERM, SIGKILL
+from signal import SIGALRM, SIGKILL, SIGTERM, alarm, signal
+
+import apt_pkg
+import distro_info
try:
from debian import deb822
except ImportError:
from debian_bundle import deb822
-import piupartslib.conf
-
-from six.moves import urllib
-
import six
+from six.moves import urllib
+import piupartslib.conf
apt_pkg.init_system()
@@ -101,7 +100,7 @@ class Defaults:
class DebianDefaults(Defaults):
def get_components(self):
- return ["main", "contrib", "non-free"]
+ return ["main", "contrib", "non-free", "non-free-firmware"]
def get_mirror(self):
return [("http://deb.debian.org/debian", self.get_components())]
@@ -220,6 +219,7 @@ class Settings:
self.warn_on_debsums_errors = False
self.warn_on_install_over_symlink = False
self.warn_if_inadequate = True
+ self.warn_on_usr_move = "disabled"
self.pedantic_purge_test = False
self.ignored_files = [
# /root/.rnd should *not* be listed here, see #750099
@@ -504,7 +504,7 @@ def indent_string(str):
def command2string(command):
"""Quote s.t. copy+paste from the logfile gives a runnable command in the shell."""
- return " ".join([pipes.quote(arg) for arg in command])
+ return " ".join([shlex.quote(arg) for arg in command])
def unqualify(packages):
@@ -922,15 +922,19 @@ class Chroot:
def setup_from_docker(self, docker_image):
self.check_if_docker_storage_driver_is_supported()
- ret_code, output = run(['docker', 'run', '-d', '-it', docker_image, 'bash'])
- if ret_code != 0:
- logging.error("Couldn't start the container from '%s'" % docker_image)
- panic()
- self.docker_container = output.strip()
- ret_code, output = run(['docker', 'inspect', self.docker_container])
- container_data = json.loads(output)[0]
- self.name = container_data['GraphDriver']['Data']['MergedDir']
- logging.info("New container created '%s'" % self.docker_container)
+ with tempfile.TemporaryDirectory() as tmpdir:
+ tmp = os.path.join(tmpdir, "cid")
+ ret_code, output = run(['docker', 'run', '-d', '-it', '--cidfile', tmp, docker_image, 'bash'])
+ if ret_code != 0:
+ logging.error("Couldn't start the container from '%s'" % docker_image)
+ panic()
+
+ with open(tmp, "r") as fd:
+ self.docker_container = fd.read().strip()
+
+ ret_code, output = run(['docker', 'inspect', '-f', '{{ .GraphDriver.Data.MergedDir }}', self.docker_container])
+ self.name = output.strip()
+ logging.info("New container created %r at %r", self.docker_container, self.name)
def setup_from_lvm(self, lvm_volume):
"""Create a chroot by creating an LVM snapshot."""
@@ -1833,6 +1837,45 @@ class Chroot:
return True
return False
+ def check_files_moved_usr(self, packages=[], files_before={}, files_after={}, warn_only=None):
+ """Check that no files were moved from /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}"""
+
+ if settings.warn_on_usr_move == "disabled":
+ return
+
+ # For each path that is a file in each package, check that it did not move between
+ # /bin, /sbin or /lib* to the corresponding location under /usr, and viceversa.
+ # See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#80
+ # If a file moved in a package that we are inspecting, print an error by default.
+ # Otherwise, print a warning.
+ broken = []
+ for old_path in files_before:
+ # '/' is a separate element in the parts list
+ old_path_parts = pathlib.Path(old_path).parts
+ if len(old_path_parts) < 3:
+ continue
+
+ if old_path_parts[1] == 'usr' and (old_path_parts[2] in ['bin', 'sbin'] or old_path_parts[2].startswith('lib')):
+ new_path = os.path.join("/", *old_path_parts[2:])
+ elif old_path_parts[1] in ['bin', 'sbin'] or old_path_parts[1].startswith('lib'):
+ new_path = "/usr" + old_path
+ else:
+ continue
+
+ # Skip over directories, multiple packages can ship files in the same directories
+ if new_path in files_after and os.path.isfile(self.relative(new_path)):
+ broken.append("%s %s => %s %s" % (old_path, files_before[old_path], new_path, files_after[new_path]))
+
+ if broken:
+ if settings.warn_on_usr_move == "warn" or warn_only:
+ logging.warning("WARN: File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}: %s" % indent_string("\n".join(broken)))
+ else:
+ logging.error("FAIL: File(s) moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}: %s" % indent_string("\n".join(broken)))
+ panic()
+ else:
+ logging.debug("No file moved between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}.")
+
+
def check_for_broken_symlinks(self, warn_only=None, file_owners={}):
"""Check that all symlinks in chroot are non-broken."""
if not settings.check_broken_symlinks:
@@ -2456,6 +2499,9 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.check_for_no_processes()
chroot.check_for_broken_symlinks()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_before = chroot.get_files_owned_by_packages()
+
if settings.install_remove_install:
chroot.remove_packages(packages, ignore_errors=True)
@@ -2468,10 +2514,12 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.disable_testdebs_repo()
- file_owners = chroot.get_files_owned_by_packages()
+ file_owners_after = chroot.get_files_owned_by_packages()
chroot.check_for_no_processes()
- chroot.check_for_broken_symlinks(file_owners=file_owners)
+ chroot.check_for_broken_symlinks(file_owners=file_owners_after)
+ if settings.warn_on_usr_move != "disabled":
+ chroot.check_files_moved_usr(packages, file_owners_before, file_owners_after)
# Remove all packages from the chroot that weren't there initially.
chroot.restore_selections(chroot_state, packages)
@@ -2479,9 +2527,9 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
chroot.run_scripts("post_test")
chroot.check_for_no_processes(fail=True)
- chroot.check_for_broken_symlinks(file_owners=file_owners)
+ chroot.check_for_broken_symlinks(file_owners=file_owners_after)
- return check_results(chroot, chroot_state, file_owners)
+ return check_results(chroot, chroot_state, file_owners_after)
def save_meta_data(filename, chroot_state):
@@ -2610,6 +2658,9 @@ def install_and_upgrade_between_distros(package_files, packages_qualified):
chroot.check_for_no_processes()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_before = chroot.get_files_owned_by_packages()
+
os.environ["PIUPARTS_PHASE"] = "distupgrade"
chroot.upgrade_to_distros(settings.debian_distros[1:-1], distupgrade_packages, settings.upgrade_before_dist_upgrade)
@@ -2634,6 +2685,10 @@ def install_and_upgrade_between_distros(package_files, packages_qualified):
chroot.check_for_no_processes()
+ if settings.warn_on_usr_move != "disabled":
+ file_owners_after = chroot.get_files_owned_by_packages()
+ chroot.check_files_moved_usr(packages, files_before=file_owners_before, files_after=file_owners_after)
+
# Remove all packages from the chroot that weren't in the reference chroot.
chroot.restore_selections(chroot_state, packages_qualified)
@@ -3008,6 +3063,10 @@ def parse_command_line():
default=False,
help="Fail if broken symlinks are detected.")
+ parser.add_option("--warn-on-usr-move", action="store", default="disabled",
+ help="Whether to enable the test (with a warning or a failure) that checks if files are moved "
+ "between /{bin|sbin|lib*} and /usr/{bin|sbin|lib*}. Accepted values: 'disabled' (default), 'warn', 'fail'.")
+
parser.add_option("--log-level", action="store", metavar='LEVEL',
default="dump",
help="Displays messages from LEVEL level, possible values are: error, info, dump, debug. The default is dump.")
@@ -3103,6 +3162,7 @@ def parse_command_line():
settings.warn_on_debsums_errors = opts.warn_on_debsums_errors
settings.warn_on_install_over_symlink = opts.warn_on_install_over_symlink
settings.warn_if_inadequate = not opts.fail_if_inadequate
+ settings.warn_on_usr_move = opts.warn_on_usr_move
settings.pedantic_purge_test = opts.pedantic_purge_test
settings.ignored_files += opts.ignore
settings.ignored_patterns += opts.ignore_regex
@@ -3138,6 +3198,10 @@ def parse_command_line():
logging.error("Scripts directory is not a directory: %s" % sdir)
panic()
+ if settings.warn_on_usr_move not in ["disabled", "warn", "fail"]:
+ logging.error("--warn-on-usr-move must be one of 'disabled', 'warn', 'fail'")
+ panic()
+
if not settings.debian_distros:
settings.debian_distros = defaults.get_distribution()
=====================================
tests/test_piuparts.py
=====================================
@@ -1,63 +1,69 @@
-import unittest
-from mox3 import mox
import os
import shutil
+import unittest
+from unittest.mock import patch
+
import piuparts
from piuparts import is_broken_symlink
class DefaultsFactoryTests(unittest.TestCase):
-
def setUp(self):
- self.mox = mox.Mox()
self.df = piuparts.DefaultsFactory()
piuparts.settings = piuparts.Settings()
- def tearDown(self):
- self.mox.UnsetStubs()
-
def test_new_defaults_return_debian_defaults(self):
# mock the guess_flavor function as it runs lsb_release in a subprocess
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('debian')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/debian-archive-keyring.gpg')
- self.assertEqual(defaults.get_components(), ["main", "contrib", "non-free"])
- self.assertEqual(defaults.get_mirror(), [("http://deb.debian.org/debian", ["main", "contrib", "non-free"])])
- self.assertEqual(defaults.get_distribution(), ['sid'])
+ with patch.object(
+ self.df, "guess_flavor", return_value="debian"
+ ) as guess_flavor_mock:
+ defaults = self.df.new_defaults()
+ guess_flavor_mock.assert_called_once()
+
+ self.assertEqual(
+ defaults.get_keyring(), "/usr/share/keyrings/debian-archive-keyring.gpg"
+ )
+ self.assertEqual(defaults.get_components(), ["main", "contrib", "non-free", "non-free-firmware"])
+ self.assertEqual(
+ defaults.get_mirror(),
+ [("http://deb.debian.org/debian", ["main", "contrib", "non-free", "non-free-firmware"])],
+ )
+ self.assertEqual(defaults.get_distribution(), ["sid"])
def test_new_defaults_return_ubuntu_defaults(self):
- # mock the guess_flavor function as it runs lsb_release in a subprocess
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('ubuntu')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- self.assertEqual(defaults.get_keyring(), '/usr/share/keyrings/ubuntu-archive-keyring.gpg')
- self.assertEqual(defaults.get_components(), ["main", "universe", "restricted", "multiverse"])
- self.assertEqual(defaults.get_mirror(), [("http://archive.ubuntu.com/ubuntu", ["main", "universe", "restricted", "multiverse"])])
+ with patch.object(
+ self.df, "guess_flavor", return_value="ubuntu"
+ ) as guess_flavor_mock:
+ defaults = self.df.new_defaults()
+ guess_flavor_mock.assert_called_once()
+
+ self.assertEqual(
+ defaults.get_keyring(), "/usr/share/keyrings/ubuntu-archive-keyring.gpg"
+ )
+ self.assertEqual(
+ defaults.get_components(), ["main", "universe", "restricted", "multiverse"]
+ )
+ self.assertEqual(
+ defaults.get_mirror(),
+ [
+ (
+ "http://archive.ubuntu.com/ubuntu",
+ ["main", "universe", "restricted", "multiverse"],
+ )
+ ],
+ )
def test_new_defaults_panics_with_unknown_flavor(self):
- # mock the guess_flavor function as it runs lsb_release in a subprocess
- # and the panic function as it would use sys.exit()
- self.mox.StubOutWithMock(self.df, 'guess_flavor')
- self.df.guess_flavor().AndReturn('centos')
- self.mox.StubOutWithMock(piuparts, 'panic')
- piuparts.panic().AndReturn('Oh dear! Its CentOS!')
- self.mox.ReplayAll()
-
- defaults = self.df.new_defaults()
- self.mox.VerifyAll()
-
- # panic() would cause sys.exit() so no Defaults object would
- # ever be returned
- self.assertEqual(defaults, None)
+ with patch.object(
+ self.df, "guess_flavor", return_value="centos"
+ ) as guess_flavor_mock, patch.object(
+ piuparts, "panic", side_effect=SystemExit
+ ) as panic_mock:
+ with self.assertRaises(SystemExit):
+ self.df.new_defaults()
+
+ guess_flavor_mock.assert_called_once()
+ panic_mock.assert_called_once()
class IsBrokenSymlinkTests(unittest.TestCase):
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/72c50f0753638d7ca6dd2fb5d1083b747c64be6d...482c6e486529522f49f4822a40c23ead589842d6
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/72c50f0753638d7ca6dd2fb5d1083b747c64be6d...482c6e486529522f49f4822a40c23ead589842d6
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20230919/9037ff7c/attachment-0001.htm>
More information about the Piuparts-devel
mailing list