[Piuparts-devel] [Git][debian/piuparts][develop] 14 commits: p-s: re-exec on SIGUSR1, picking up updated code and new config sections
Holger Levsen
gitlab at salsa.debian.org
Fri Jun 14 14:41:46 BST 2019
Holger Levsen pushed to branch develop at Debian / piuparts
Commits:
e7f15575 by Andreas Beckmann at 2019-06-14T13:41:30Z
p-s: re-exec on SIGUSR1, picking up updated code and new config sections
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1ff98edf by Andreas Beckmann at 2019-06-14T13:41:30Z
slave-bin/slave_reexec: new, send SIGUSR1 to all running slaves
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
d731b3e5 by Andreas Beckmann at 2019-06-14T13:41:30Z
network_issues: relax the 'Package .* .* not found in .*' pattern
there might be no alternative version available
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b2700e50 by Andreas Beckmann at 2019-06-14T13:41:30Z
rotate_master_logs: delete master logs older than 90 days
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b633123f by Andreas Beckmann at 2019-06-14T13:41:30Z
detect_piuparts_issues: clean up stale temporary and empty files
usually caused by piuparts-master running on a full disk
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
53cb626d by Andreas Beckmann at 2019-06-14T13:41:30Z
add logratate exception for chef-server-api
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3f607470 by Andreas Beckmann at 2019-06-14T13:41:30Z
redo the kFreeBSD-only exception
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
85994b31 by Andreas Beckmann at 2019-06-14T13:41:30Z
add skip_distupgrade_from checking the install candidate version
use where a later version might fix the bug and enable dist-upgrade testing
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
eaf4cc43 by Andreas Beckmann at 2019-06-14T13:41:30Z
enigmail/jessie is not installable with thunderbird 60
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3d4d3408 by Andreas Beckmann at 2019-06-14T13:41:30Z
xul-ext-sogo-connector/jessie is not installable with thunderbird 60
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7ffe64c7 by Andreas Beckmann at 2019-06-14T13:41:30Z
xul-ext-compactheader/wheezy,jessie is not installable with thunderbird 60
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
fcf0a7db by Andreas Beckmann at 2019-06-14T13:41:30Z
several firefox,tunderbird/wheezy,jessie cruft packages are uninstallable
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7a8cde3f by Andreas Beckmann at 2019-06-14T13:41:30Z
lib/db: factor out common log processing code
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
23e1f4a7 by Andreas Beckmann at 2019-06-14T13:41:30Z
lib/db: remove stale .kpr files after receiving new logs
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
9 changed files:
- custom-scripts/scripts-multi-distro-upgrade/pre_remove_exceptions_multi-distro-upgrade
- custom-scripts/scripts/is_testable_uninstallable
- debian/changelog
- master-bin/detect_network_issues.in
- master-bin/detect_piuparts_issues.in
- master-bin/rotate_master_logs.in
- piuparts-slave.py
- piupartslib/packagesdb.py
- + slave-bin/slave_reexec.in
Changes:
=====================================
custom-scripts/scripts-multi-distro-upgrade/pre_remove_exceptions_multi-distro-upgrade
=====================================
@@ -18,6 +18,13 @@ case ${PIUPARTS_OBJECTS} in
log_debug
touch /var/log/chef/solr.log
;;
+ chef-server-api=None)
+ # removed after wheezy
+ # logrotate config references /var/log/chef/server.log /var/log/chef/merb*log
+ log_debug
+ touch /var/log/chef/server.log
+ touch /var/log/chef/merb-dummy.log
+ ;;
dancer-ircd=None)
# removed after wheezy
# logrotate config references /var/log/dancer-ircd/ircd.log
=====================================
custom-scripts/scripts/is_testable_uninstallable
=====================================
@@ -34,6 +34,19 @@ skip_distupgrade()
fi
}
+# use where a later version might fix the bug and enable dist-upgrade testing
+skip_distupgrade_from()
+{
+ from_ver=$1
+ shift
+ pkg=${PIUPARTS_OBJECTS%%=*}
+ cand_ver=$(apt-cache policy "$pkg" | awk '{if ($1 == "Candidate:") {print $2}}')
+
+ if [ "$from_ver" = "$cand_ver" ]; then
+ skip_distupgrade "$@"
+ fi
+}
+
# * For uninstallable *broken* packages we usually run the normal
# installation+purge tests (which will fail and get the package marked
@@ -133,6 +146,27 @@ fi
if [ "${PIUPARTS_DISTRIBUTION}" = "wheezy" ]; then
case ${PIUPARTS_OBJECTS%%=*} in
+ firefox-esr-dev|\
+ iceowl-l10n-bn-bd|\
+ iceowl-l10n-pa-in|\
+ iceowl-l10n-ta-lk|\
+ iceweasel-l10n-ak|\
+ iceweasel-l10n-be|\
+ iceweasel-l10n-csb|\
+ iceweasel-l10n-ku|\
+ iceweasel-l10n-lg|\
+ iceweasel-l10n-nso|\
+ iceweasel-l10n-ta-lk|\
+ iceweasel-l10n-zu|\
+ lightning-l10n-bn-bd|\
+ lightning-l10n-pa-in|\
+ lightning-l10n-ta-lk|\
+ thunderbird-dev|\
+ thunderbird-l10n-bn-bd|\
+ thunderbird-l10n-pa-in|\
+ thunderbird-l10n-ta-lk)
+ skip_distupgrade "#928226: uninstallable cruft package"
+ ;;
firmware-b43-lpphy-installer)
skip_distupgrade "#623703: firmware-b43-lpphy-installer: download location no longer available"
;;
@@ -146,6 +180,9 @@ if [ "${PIUPARTS_DISTRIBUTION}" = "wheezy" ]; then
nautilus-dropbox)
skip_distupgrade "#770402: nautilus-dropbox: downloads awfully slow, hangs during installation"
;;
+ xul-ext-compactheader)
+ skip_distupgrade "#837387: xul-ext-compactheader: uninstallable in wheezy with icedove (>= 1:45)"
+ ;;
esac
fi
@@ -153,6 +190,27 @@ fi
if [ "${PIUPARTS_DISTRIBUTION}" = "jessie" ]; then
case ${PIUPARTS_OBJECTS%%=*} in
+ enigmail)
+ skip_distupgrade_from "2:1.9.9-1~deb8u1" "#909000: enigmail 2.0 needed after thunderbird 60 upload"
+ ;;
+ firefox-esr-dev|\
+ icedove-dev|\
+ icedove-l10n-bn-bd|\
+ icedove-l10n-pa-in|\
+ icedove-l10n-ta-lk|\
+ iceowl-l10n-bn-bd|\
+ iceowl-l10n-pa-in|\
+ iceowl-l10n-ta-lk|\
+ iceweasel-dev|\
+ lightning-l10n-bn-bd|\
+ lightning-l10n-pa-in|\
+ lightning-l10n-ta-lk|\
+ thunderbird-dev|\
+ thunderbird-l10n-bn-bd|\
+ thunderbird-l10n-pa-in|\
+ thunderbird-l10n-ta-lk)
+ skip_distupgrade "#928226: uninstallable cruft package"
+ ;;
fusionforge-plugin-mediawiki)
skip_distupgrade "#929139: fusionforge-plugin-mediawiki: uninstallable cruft package"
;;
@@ -160,10 +218,16 @@ if [ "${PIUPARTS_DISTRIBUTION}" = "jessie" ]; then
skip_distupgrade "#885441: geoip-database-contrib: download location no longer available"
;;
steam)
- skip_distupgrade "#772598: cannot preseed licence acceptance"
+ skip_distupgrade_from "1.0.0.49-1" "#772598: steam: cannot preseed licence acceptance"
;;
susv4)
- skip_distupgrade "#900882: download location no longer available"
+ skip_distupgrade_from "7.20161013~deb8u1" "#900882: susv4: download location no longer available"
+ ;;
+ xul-ext-compactheader)
+ skip_distupgrade_from "2.0.8-1" "#837387: xul-ext-compactheader: uninstallable in jessie with icedove (>= 1:45)"
+ ;;
+ xul-ext-sogo-connector)
+ skip_distupgrade_from "31.0.0-1" "#909313: xul-ext-sogo-connector: not compatible with thunderbird 60"
;;
esac
@@ -236,6 +300,18 @@ if [ "${PIUPARTS_DISTRIBUTION}" = "sid-debug" ]; then
fi
+
+if [ "$(uname)" = "Linux" ]; then
+
+ case ${PIUPARTS_OBJECTS%%=*} in
+ console-setup-freebsd)
+ skip_always "non-Linux package"
+ ;;
+ esac
+
+fi
+
+
case ${PIUPARTS_OBJECTS%%=*}@${PIUPARTS_DISTRIBUTION} in
acgvision-agent at squeeze)
# removed after squeeze
@@ -255,11 +331,6 @@ case ${PIUPARTS_OBJECTS%%=*}@${PIUPARTS_DISTRIBUTION} in
chef-server-api at wheezy)
skip_distupgrade "#681573: chef-solr: fails to install if policy-rc.d forbids to start services"
;;
- console-setup-freebsd@*)
- if [ "$(uname)" = "Linux" ]; then
- skip_always "non-Linux package"
- fi
- ;;
cpushare at wheezy)
# removed after wheezy
skip_distupgrade "#689885: Ships a folder in /var/run or /var/lock"
@@ -349,14 +420,4 @@ case ${PIUPARTS_OBJECTS%%=*}@${PIUPARTS_DISTRIBUTION} in
usrmerge@*)
skip_always "performs piuparts-unfriendy filesystem layout changes"
;;
- xul-ext-compactheader at wheezy)
- if [ "$(apt-cache show xul-ext-compactheader | sed -n '/^Version:/s/^Version: //p')" = "2.0.5-1" ]; then
- skip_distupgrade "#837387: uninstallable in wheezy with icedove (>= 1:45)"
- fi
- ;;
- xul-ext-compactheader at jessie)
- if [ "$(apt-cache show xul-ext-compactheader | sed -n '/^Version:/s/^Version: //p')" = "2.0.8-1" ]; then
- skip_distupgrade "#837387: uninstallable in jessie with icedove (>= 1:45)"
- fi
- ;;
esac
=====================================
debian/changelog
=====================================
@@ -7,8 +7,15 @@ piuparts (1.0.1) UNRELEASED; urgency=medium
- Run pre_remove scripts before computing packages to be removed.
- Fix spurious failure to remove packages with names ending with '+'.
* piuparts.conf.anbe: Add some more example sections.
+ * piupartslib/packagesdb.py:
+ - Remove stale .kpr files after receiving new logs.
* piuparts-slave.py:
- Generate separate tarball names for --merged-usr chroots.
+ - Re-exec on SIGUSR1, picking up updated code and new config sections.
+ * master-bin/detect_piuparts_issues:
+ - Clean up stale temporary and empty files.
+ * master-bin/rotate_master_logs: Delete master logs older than 90 days.
+ * slave-bin/slave_reexec: New, send SIGUSR1 to all running slaves.
[ Holger Levsen ]
* generate_daily_report: improve statistics.
@@ -160,7 +167,7 @@ piuparts (0.96) unstable; urgency=medium
[ Andreas Beckmann ]
* bug-templates: Raise severity of "fails to purge" errors to serious.
- (see #918379)
+ (see #918379)
* distros.conf: Add jessie-lts as a full distro.
* piuparts.conf:
- Use jessie-lts as upgrade target and rename section
=====================================
master-bin/detect_network_issues.in
=====================================
@@ -62,7 +62,7 @@ add_pattern "E: Version '.*' for '.*' was not found"
add_pattern 'E: Method file has died unexpectedly!'
add_pattern "E: Sub-process rred received a segmentation fault."
add_pattern "ERROR: Command failed \(status=-7\):.*'apt-get', 'update'"
-add_pattern 'Package .* .* not found in .*, .* is available'
+add_pattern 'Package .* .* not found in .*(, .* is available)?'
add_pattern "AppStream system cache was updated, but problems were found: Metadata files have errors"
FILE=$(mktemp)
=====================================
master-bin/detect_piuparts_issues.in
=====================================
@@ -1,7 +1,7 @@
#!/bin/sh
# Copyright © 2009-2018 Holger Levsen (holger at layer-acht.org)
-# Copyright © 2011-2017 Andreas Beckmann (anbe at debian.org)
+# Copyright © 2011-2019 Andreas Beckmann (anbe at debian.org)
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
@@ -96,6 +96,10 @@ for SECTION in $SECTIONS ; do
for subdir in fail bugged affected ; do
test -d $MASTER/$SECTION/$subdir || continue
grep -r -l -E --include '*.log' "$PATTERN" $MASTER/$SECTION/$subdir >> $FILE 2>/dev/null
+ done
+ for subdir in fail bugged affected untestable pass ; do
+ test -d $MASTER/$SECTION/$subdir || continue
+ find $MASTER/$SECTION/$subdir -name 'tmp??????' -mtime +1 -delete 2>/dev/null
find $MASTER/$SECTION/$subdir -name '*.log' -size 0 >> $FILE
done
done
=====================================
master-bin/rotate_master_logs.in
=====================================
@@ -1,6 +1,6 @@
#!/bin/sh
#
-# Copyright © 2017 Andreas Beckmann (anbe at debian.org)
+# Copyright © 2017-2019 Andreas Beckmann (anbe at debian.org)
# Copyright © 2018 Holger Levsen (holger at layer-acht.org)
#
# This program is free software; you can redistribute it and/or modify it
@@ -48,6 +48,12 @@ for SECTION in $SECTIONS ; do
xz -v9 $log >>$OUTPUT 2>&1
fi
done
+
+ # delete master logs older than 90 days, but keep at least five old logs
+ oldlogs=$(ls -t1 $SECTION/master.*.log.* 2>/dev/null | tail -n +5)
+ if [ -n "$oldlogs" ]; then
+ find $oldlogs -mtime +90 | xargs -r rm -fv >>$OUTPUT 2>&1
+ fi
done
LOGDATEDIR="$HTDOCS/logs/$(date -u '+%Y/%m/%d' -d '7 days ago')"
=====================================
piuparts-slave.py
=====================================
@@ -29,7 +29,7 @@ import sys
import stat
import time
import logging
-from signal import alarm, signal, SIGALRM, SIGINT, SIGKILL, SIGHUP
+from signal import alarm, signal, SIGALRM, SIGINT, SIGKILL, SIGHUP, SIGUSR1
import subprocess
import fcntl
import random
@@ -50,6 +50,7 @@ MAX_WAIT_TEST_RUN = 90 * 60
interrupted = False
old_sigint_handler = None
got_sighup = False
+got_sigusr1 = False
def setup_logging(log_level, log_file_name):
@@ -129,6 +130,14 @@ def sighup_handler(signum, frame):
print('SIGHUP: Will flush finished logs.')
+def sigusr1_handler(signum, frame):
+ global got_sigusr1
+ global got_sighup
+ got_sigusr1 = True
+ got_sighup = True
+ print('SIGUSR1: Will restart.')
+
+
class MasterIsBusy(Exception):
def __init__(self):
@@ -975,6 +984,7 @@ def create_file(filename, contents):
def main():
setup_logging(logging.INFO, None)
signal(SIGHUP, sighup_handler)
+ signal(SIGUSR1, sigusr1_handler)
# For supporting multiple architectures and suites, we take command-line
# argument(s) referring to section(s) in the configuration file.
@@ -1015,6 +1025,10 @@ def main():
for section in sorted(sections, key=lambda section: (section.precedence(), section.sleep_until())):
test_count += section.run(do_processing=(test_count == 0))
+ if got_sigusr1:
+ logging.info("Restarting...")
+ os.execv(__file__, sys.argv)
+
if test_count == 0 and got_sighup:
# clear SIGHUP state after flushing all sections
got_sighup = False
=====================================
piupartslib/packagesdb.py
=====================================
@@ -236,6 +236,9 @@ class LogDB:
def _log_name(self, package, version):
return "%s_%s.log" % (package, version)
+ def _kpr_name(self, package, version):
+ return "%s_%s.kpr" % (package, version)
+
def log_exists2(self, package, version, subdirs):
log_name = self._log_name(package, version)
for subdir in subdirs:
@@ -274,6 +277,11 @@ class LogDB:
self.remove_file(full_name)
self._evict(full_name)
+ def remove_kpr(self, subdir, package, version):
+ full_name = os.path.join(subdir, self._kpr_name(package, version))
+ if os.path.exists(full_name):
+ self.remove_file(full_name)
+
def stat(self, subdir, package, version):
full_name = os.path.join(subdir, self._log_name(package, version))
return os.stat(full_name)
@@ -896,32 +904,24 @@ class PackagesDB:
self._logdb.create(self._recycle, package, version, "")
self._logdb.remove(self._reserved, package, version)
- def pass_package(self, package, version, log):
+ def _process_log(self, package, version, log, subdir, result):
self._check_for_acceptability_as_filename(package)
self._check_for_acceptability_as_filename(version)
self._remove_logs_if_reserved(package, version)
- if self._logdb.create(self._ok, package, version, log):
- self._record_submission("pass", package, version)
+ if self._logdb.create(subdir, package, version, log):
+ self._record_submission(result, package, version)
+ self._logdb.remove_kpr(subdir, package, version)
else:
- raise LogfileExists(self._ok, package, version)
+ raise LogfileExists(subdir, package, version)
+
+ def pass_package(self, package, version, log):
+ self._process_log(package, version, log, self._ok, "pass")
def fail_package(self, package, version, log):
- self._check_for_acceptability_as_filename(package)
- self._check_for_acceptability_as_filename(version)
- self._remove_logs_if_reserved(package, version)
- if self._logdb.create(self._fail, package, version, log):
- self._record_submission("fail", package, version)
- else:
- raise LogfileExists(self._fail, package, version)
+ self._process_log(package, version, log, self._fail, "fail")
def make_package_untestable(self, package, version, log):
- self._check_for_acceptability_as_filename(package)
- self._check_for_acceptability_as_filename(version)
- self._remove_logs_if_reserved(package, version)
- if self._logdb.create(self._evil, package, version, log):
- self._record_submission("untestable", package, version)
- else:
- raise LogfileExists(self._evil, package, version)
+ self._process_log(package, version, log, self._evil, "untestable")
def _get_rdep_dict(self):
"""Return dict of one-level reverse dependencies by package"""
=====================================
slave-bin/slave_reexec.in
=====================================
@@ -0,0 +1,30 @@
+#!/bin/sh
+set -e
+
+# Copyright © 2019 Andreas Beckmann (anbe at debian.org)
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>
+
+#
+# Send all running piuparts-slaves SIGUSR1 to reexec themselves,
+# picking up new code and sections.
+#
+
+for pid in $(pgrep --full '/usr/bin/python @sharedir@/piuparts/piuparts-slave')
+do
+ echo "Sending SIGUSR1 to ${pid}..."
+ kill -USR1 $pid
+ sleep 5
+done
+echo "Done."
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/f4ace9e549d485179543d3aa29b7913a69de6346...23e1f4a79f8694954822d20401d3171a2e0246dd
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/f4ace9e549d485179543d3aa29b7913a69de6346...23e1f4a79f8694954822d20401d3171a2e0246dd
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20190614/52a1d5ea/attachment-0001.html>
More information about the Piuparts-devel
mailing list