[Piuparts-devel] [Git][debian/piuparts][ptmx] 11 commits: master-bin/detect_well_known_errors.py: suppress output if nothing new is found in a section.
Mathieu Parent
gitlab at salsa.debian.org
Sun Oct 28 20:31:41 GMT 2018
Mathieu Parent pushed to branch ptmx at Debian / piuparts
Commits:
011b9dec by Holger Levsen at 2018-10-24T15:00:01Z
master-bin/detect_well_known_errors.py: suppress output if nothing new is found in a section.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
15caf640 by Holger Levsen at 2018-10-24T19:55:43Z
master-bin/detect_well_known_errors.py: avoid more output with low signal
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
5e263f80 by Holger Levsen at 2018-10-24T23:35:31Z
master-bin/generate_daily_report: include number of failures in daily mail.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e587a83b by Holger Levsen at 2018-10-25T16:20:54Z
fix typo in comment
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3897b5fa by Holger Levsen at 2018-10-25T16:26:31Z
slave-bin/detect_slave_problems: increase amount of time (from 30 to 60m) a slave has to be inactive before complaining.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
93c4bbb1 by Andreas Beckmann at 2018-10-26T15:18:37Z
permit expired stretch-{lts,backports{,-sloppy}} signing keys
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7fcac454 by Andreas Beckmann at 2018-10-26T15:19:05Z
file-rc is gone from sid
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4db8eb02 by Andreas Beckmann at 2018-10-26T15:19:10Z
add exception for nsswitch.conf on stretch->buster upgrades
switch passwd/group/shadow from compat to files
libc-bin only upgrades pristine /etc/nsswitch.conf
(i.e. w/o modifications by libnss-myhostname and friends)
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
8b9d62ef by Andreas Beckmann at 2018-10-26T15:19:15Z
p: logrotate files can only be placed directly in /etc/logrotate.d/
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
22a001c9 by Andreas Beckmann at 2018-10-26T15:19:20Z
p: simplify cronfiles and logrotatefiles handling
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
a99ecade by Mathieu Parent at 2018-10-28T20:28:36Z
p: use mknod instead of touch to create missing /dev/ptmx mountpoint (Closes: #911334)
Working around findutils bug #912180:
mkdir foo
mknod foo/null c 1 3
touch foo/bindmntoverfile
mount --bind foo/null foo/bindmntoverfile
find foo -type f -ls
3857271640 0 crw-r--r-- 1 root root 1, 3 Oct 24 07:50 foo/bindmntoverfile
When using piuparts on a chroot without /dev/ptmx [noptmx],
scripts/pre_remove_50_find_bad_permissions fails with:
ERROR: BAD PERMISSIONS
crw-rw-rw-. 1 root root 5, 2 Oct 16 03:49 /dev/ptmx
In this case, piuparts does something like this:
touch /dev/ptmx # if not exists
mount -o bind /dev/pts/ptmx /dev/ptmx # if dev/ptmx was not a symlink
The kernel doc [devpts.txt] recommends instead:
mknod /dev/ptmx c 5 2
And this is what debootstrap does [debootstrap].
After this change, piuparts will do:
mknod /dev/ptmx c 5 2 # if not exists
mount -o bind /dev/pts/ptmx /dev/ptmx # if dev/ptmx was not a symlink
The behavior of piuparts called after debootstrap will not change.
The only behavior changing is when dev/ptmx doesn't exist at all.
[noptmx]: This is the case when chroot comes from the debian:unstable Docker image
[devpts.txt]: https://www.kernel.org/doc/Documentation/filesystems/devpts.txt
[debootstrap] https://salsa.debian.org/installer-team/debootstrap/blob/6f3f6f8b76e2d1a24ddbf05f065439412c3b81a1/functions#L1263-1268, introduced by https://salsa.debian.org/installer-team/debootstrap/commit/c997b80c064c6c1d36ec69da1850722f795f43e4
- - - - -
9 changed files:
- custom-scripts/scripts/post_chroot_unpack_allow_unauthenticated
- custom-scripts/scripts/post_distupgrade_exceptions
- custom-scripts/scripts/pre_distupgrade_allow_unauthenticated
- custom-scripts/scripts/pre_install_exceptions
- debian/changelog
- master-bin/detect_well_known_errors.py
- master-bin/generate_daily_report.in
- piuparts.py
- slave-bin/detect_slave_problems.in
Changes:
=====================================
custom-scripts/scripts/post_chroot_unpack_allow_unauthenticated
=====================================
@@ -9,7 +9,10 @@ Acquire::Check-Valid-Until "false";
EOF
fi
-if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ]; then
+if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ] || \
+ [ "$PIUPARTS_DISTRIBUTION" = "squeeze-lts" ] || \
+ [ "$PIUPARTS_DISTRIBUTION" = "squeeze-backports" ] || \
+ [ "$PIUPARTS_DISTRIBUTION" = "squeeze-backports-sloppy" ]; then
echo "Creating /etc/apt/apt.conf.d/unauthenticated-squeeze ..."
tee /etc/apt/apt.conf.d/unauthenticated-squeeze <<EOF
# The squeeze signing key has expired.
=====================================
custom-scripts/scripts/post_distupgrade_exceptions
=====================================
@@ -84,6 +84,12 @@ fi
if [ "$PIUPARTS_DISTRIBUTION" = "buster" ]; then
+ # libc-bin only upgrades pristine /etc/nsswitch.conf
+ if grep -q '^passwd:.*compat' /etc/nsswitch.conf ; then
+ echo "Switching from compat to files in /etc/nsswitch.conf"
+ sed -r -i '/^(passwd|group|shadow):/ s/compat/files/' /etc/nsswitch.conf
+ fi
+
# dpkg does not properly clean up directories getting empty and no longer shipped
for dir in /etc/dbus-1/system.d /etc/dbus-1
do
=====================================
custom-scripts/scripts/pre_distupgrade_allow_unauthenticated
=====================================
@@ -23,6 +23,8 @@ Acquire::Check-Valid-Until "false";
EOF
elif [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze-backports" ]; then
:
+elif [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze-lts" ]; then
+ :
elif [ -e /etc/apt/apt.conf.d/unauthenticated-squeeze ]; then
echo "FAIL: /etc/apt/apt.conf.d/unauthenticated-squeeze exists"
exit 1
=====================================
custom-scripts/scripts/pre_install_exceptions
=====================================
@@ -13,7 +13,7 @@ case ${PIUPARTS_OBJECTS%%=*} in
log_debug
echo 'Yes, do as I say!' | apt-get -y --force-yes install file-rc
;;
- squeeze*|stretch*|sid)
+ squeeze*|stretch*)
# force installation and removal of essential package sysv-rc
log_debug
yes 'Yes, do as I say!' | apt-get -y --force-yes install file-rc
=====================================
debian/changelog
=====================================
@@ -1,6 +1,22 @@
piuparts (0.94) UNRELEASED; urgency=medium
- * ...
+ [ Holger Levsen ]
+ * master-bin/detect_well_known_errors.py: suppress output if nothing new is
+ found in a section, or section is busy or has been recently processed.
+ * master-bin/generate_daily_report: include number of failures in daily mail.
+ * slave-bin/detect_slave_problems: increase amount of time (from 30 to 60m)
+ a slave has to be inactive before complaining.
+
+ [ Andreas Beckmann ]
+ * piuparts.py:
+ - Logrotate files can only be placed directly in /etc/logrotate.d/.
+ * scripts/post_distupgrade_exceptions:
+ - Handle /etc/nsswitch.conf stretch -> buster upgrade if modifications
+ from libnss-myhostname and friends are present.
+
+ [ Mathieu Parent ]
+ * piuparts.py: Use mknod instead of touch to create missing /dev/ptmx
+ mountpoint, working around findutils bug #912180 (Closes: #911334)
-- Holger Levsen <holger at debian.org> Sun, 21 Oct 2018 14:03:22 +0200
=====================================
master-bin/detect_well_known_errors.py
=====================================
@@ -113,7 +113,7 @@ def detect_well_known_errors(sections, config, problem_list, recheck, recheck_fa
(section, next_try) = todo.popleft()
now = time.time()
if (now < next_try):
- logging.info("Sleeping while sections are busy")
+ # sleeping, section has been tried recently
time.sleep(max(30, next_try - now) + 30)
try:
(del_cnt, add_cnt) = \
@@ -123,11 +123,11 @@ def detect_well_known_errors(sections, config, problem_list, recheck, recheck_fa
total_add += add_cnt
current_time=time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime())
if del_cnt == 0 and add_cnt == 0:
- logging.info("%s - %s: nothing new" % (current_time, section))
+ pass # nothing new
else:
logging.info("%s - %s: parsed logfiles: %d removed, %d added" % (current_time, section, del_cnt, add_cnt))
except Busy:
- logging.info("Section is busy")
+ # section is busy
todo.append((section, time.time() + 300))
except MissingSection:
pass
=====================================
master-bin/generate_daily_report.in
=====================================
@@ -67,7 +67,7 @@ FAILURESTAMP=$HTDOCS/last-failure-report.stamp
test -f $FAILURESTAMP || touch -d @0 $FAILURESTAMP # start at the epoch
touch $FAILURESTAMP.new # for the next report
-echo "New failures:" >> $DAILYREPORT
+DAILYTMP=$(mktemp)
for SECTION in $SECTIONS ; do
test -d $MASTER/$SECTION || continue
for DIRECTORY in fail bugged affected untestable ; do
@@ -75,7 +75,11 @@ for SECTION in $SECTIONS ; do
done
find $MASTER/$SECTION/fail $MASTER/$SECTION/bugged $MASTER/$SECTION/affected $MASTER/$SECTION/untestable \
-type f -name '*.log' -newer $FAILURESTAMP -exec ls -1 {} + 2>/dev/null
-done | sed s#^$MASTER#$URLBASE# >> $DAILYREPORT
+done | sed s#^$MASTER#$URLBASE# >> $DAILYTMP
+
+echo "$(cat $DAILYTMP | wc -l) new failures:" >> $DAILYREPORT
+cat $DAILYTMP >> $DAILYREPORT
+rm $DAILYTMP
echo "" >> $DAILYREPORT
date >> $DAILYREPORT
=====================================
piuparts.py
=====================================
@@ -1565,18 +1565,16 @@ class Chroot:
self.run_scripts("post_remove")
if not settings.skip_cronfiles_test:
- cronfiles, cronfiles_list = self.check_if_cronfiles(packages)
-
- if not settings.skip_cronfiles_test and cronfiles:
- self.check_output_cronfiles(cronfiles_list)
+ cronfiles = self.check_if_cronfiles(packages)
+ if cronfiles:
+ self.check_output_cronfiles(cronfiles)
if not settings.skip_logrotatefiles_test:
- logrotatefiles, logrotatefiles_list = self.check_if_logrotatefiles(packages)
-
- if not settings.skip_logrotatefiles_test and logrotatefiles:
- installed = self.install_logrotate()
- self.check_output_logrotatefiles(logrotatefiles_list)
- self.purge_packages(installed)
+ logrotatefiles = self.check_if_logrotatefiles(packages)
+ if logrotatefiles:
+ installed = self.install_logrotate()
+ self.check_output_logrotatefiles(logrotatefiles)
+ self.purge_packages(installed)
# Then purge all packages being depended on.
self.purge_packages(deps_to_purge)
@@ -1757,8 +1755,7 @@ class Chroot:
dev_ptmx_rel_path = self.relative("dev/ptmx")
if not os.path.islink(dev_ptmx_rel_path):
if not os.path.exists(dev_ptmx_rel_path):
- with open(dev_ptmx_rel_path, 'w'):
- pass
+ os.mknod(dev_ptmx_rel_path, 0666 | stat.S_IFCHR, os.makedev(5, 2))
self.mount(self.relative("dev/pts/ptmx"), "/dev/ptmx", opts="bind", no_mkdir=True)
p = subprocess.Popen(["tty"], stdout=subprocess.PIPE)
stdout, _ = p.communicate()
@@ -1829,9 +1826,9 @@ class Chroot:
"""Check if the packages have cron files under /etc/cron.d and in case positive,
it returns the list of files. """
+ # FIXME! Does not work for M-A: same packages
vdir = self.relative("var/lib/dpkg/info")
vlist = []
- has_cronfiles = False
for p in packages:
basename = p + ".list"
@@ -1846,12 +1843,10 @@ class Chroot:
mode = st[stat.ST_MODE]
# XXX /etc/cron.d/ files are NOT executables
if (mode & stat.S_IEXEC):
- if not has_cronfiles:
- has_cronfiles = True
vlist.append(pathname)
logging.info("Package " + p + " contains cron file: " + pathname)
- return has_cronfiles, vlist
+ return vlist
def check_output_cronfiles(self, list):
"""Check if a given list of cronfiles has any output. Executes
@@ -1874,9 +1869,9 @@ class Chroot:
"""Check if the packages have logrotate files under /etc/logrotate.d and in case positive,
it returns the list of files. """
+ # FIXME! Does not work for M-A: same packages
vdir = self.relative("var/lib/dpkg/info")
vlist = []
- has_logrotatefiles = False
for p in packages:
basename = p + ".list"
@@ -1885,14 +1880,12 @@ class Chroot:
for line in readlines_file(os.path.join(vdir, basename)):
pathname = line.strip()
- if pathname.startswith("/etc/logrotate.d/"):
+ if os.path.dirname(pathname) == "/etc/logrotate.d":
if os.path.isfile(self.relative(pathname.strip("/"))):
- if not has_logrotatefiles:
- has_logrotatefiles = True
vlist.append(pathname)
logging.info("Package " + p + " contains logrotate file: " + pathname)
- return has_logrotatefiles, vlist
+ return vlist
def install_logrotate(self):
"""Install logrotate for check_output_logrotatefiles, and return the
=====================================
slave-bin/detect_slave_problems.in
=====================================
@@ -49,18 +49,15 @@ STATEFILE=$SLAVEROOT/slave-problems
# clear the statefile daily and whine again
test $(file_age $STATEFILE) -lt 86000 || rm -f $STATEFILE
-# Only complain if screenlog is older than $IDLE_SPEEP + 1 minute (the slave
-# likes to sleep that long) and the problem is new or was not reported within
-# the previous 24 hours.
-if [ $(file_age $SCREENLOG) -le $(($IDLE_SLEEP + 60)) ]; then
+# Only complain if screenlog is older than $IDLE_SLEEP + 30 minutes
+# and the problem is new or was not reported within the previous 24 hours.
+if [ $(file_age $SCREENLOG) -le $(($IDLE_SLEEP + 1800)) ]; then
rm -f $STATEFILE
elif [ ! -f $STATEFILE ]; then
{
- echo "Either a test is running for a very long time (but no test"
- echo "should run longer than an hour), piuparts-slave hangs or is"
- echo "not running at all or wasn't started with"
- echo "~piupartss/bin/slave_run - please investigate and take"
- echo "appropriate measures!"
+ echo "Either a test is running for a very long time (more than $(( ($IDLE_SLEEP + 1800)/60 )) minutes),"
+ echo "or piuparts-slave hangs or is not running at all or wasn't started with"
+ echo "~piupartss/bin/slave_run - please investigate and take appropriate measures!"
echo
tail $SCREENLOG
} | mail -s "problem with piuparts-slave detected" piupartss
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/d3bc89f41531e1ce8d37fc82f70e5b2fa7290844...a99ecade05aeb855c5e17528403956300ee280c7
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/d3bc89f41531e1ce8d37fc82f70e5b2fa7290844...a99ecade05aeb855c5e17528403956300ee280c7
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20181028/fd2d13f1/attachment-0001.html>
More information about the Piuparts-devel
mailing list