[Piuparts-devel] [Git][debian/piuparts][master] 15 commits: continue development

Holger Levsen gitlab at salsa.debian.org
Thu Nov 8 11:11:39 GMT 2018


Holger Levsen pushed to branch master at Debian / piuparts


Commits:
0afdf66f by Holger Levsen at 2018-10-21T12:03:42Z
continue development

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
011b9dec by Holger Levsen at 2018-10-24T15:00:01Z
master-bin/detect_well_known_errors.py: suppress output if nothing new is found in a section.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
15caf640 by Holger Levsen at 2018-10-24T19:55:43Z
master-bin/detect_well_known_errors.py: avoid more output with low signal

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
5e263f80 by Holger Levsen at 2018-10-24T23:35:31Z
master-bin/generate_daily_report: include number of failures in daily mail.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
e587a83b by Holger Levsen at 2018-10-25T16:20:54Z
fix typo in comment

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
3897b5fa by Holger Levsen at 2018-10-25T16:26:31Z
slave-bin/detect_slave_problems: increase amount of time (from 30 to 60m) a slave has to be inactive before complaining.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
93c4bbb1 by Andreas Beckmann at 2018-10-26T15:18:37Z
permit expired stretch-{lts,backports{,-sloppy}} signing keys

Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
7fcac454 by Andreas Beckmann at 2018-10-26T15:19:05Z
file-rc is gone from sid

Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
4db8eb02 by Andreas Beckmann at 2018-10-26T15:19:10Z
add exception for nsswitch.conf on stretch->buster upgrades

switch passwd/group/shadow from compat to files
libc-bin only upgrades pristine /etc/nsswitch.conf
(i.e. w/o modifications by libnss-myhostname and friends)

Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
8b9d62ef by Andreas Beckmann at 2018-10-26T15:19:15Z
p: logrotate files can only be placed directly in /etc/logrotate.d/

Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
22a001c9 by Andreas Beckmann at 2018-10-26T15:19:20Z
p: simplify cronfiles and logrotatefiles handling

Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
a99ecade by Mathieu Parent at 2018-10-28T20:28:36Z
p: use mknod instead of touch to create missing /dev/ptmx mountpoint (Closes: #911334)

Working around findutils bug #912180:

    mkdir foo
    mknod foo/null c 1 3
    touch foo/bindmntoverfile
    mount --bind foo/null foo/bindmntoverfile
    find foo -type f -ls
    3857271640      0 crw-r--r--   1 root     root       1,   3 Oct 24 07:50 foo/bindmntoverfile

When using piuparts on a chroot without /dev/ptmx [noptmx],
scripts/pre_remove_50_find_bad_permissions fails with:

    ERROR: BAD PERMISSIONS
    crw-rw-rw-. 1 root root 5, 2 Oct 16 03:49 /dev/ptmx

In this case, piuparts does something like this:

    touch /dev/ptmx # if not exists
    mount -o bind /dev/pts/ptmx /dev/ptmx # if dev/ptmx was not a symlink

The kernel doc [devpts.txt] recommends instead:

    mknod /dev/ptmx c 5 2

And this is what debootstrap does [debootstrap].

After this change, piuparts will do:

    mknod /dev/ptmx c 5 2 # if not exists
    mount -o bind /dev/pts/ptmx /dev/ptmx # if dev/ptmx was not a symlink

The behavior of piuparts called after debootstrap will not change.
The only behavior changing is when dev/ptmx doesn't exist at all.

[noptmx]: This is the case when chroot comes from the debian:unstable Docker image
[devpts.txt]: https://www.kernel.org/doc/Documentation/filesystems/devpts.txt
[debootstrap] https://salsa.debian.org/installer-team/debootstrap/blob/6f3f6f8b76e2d1a24ddbf05f065439412c3b81a1/functions#L1263-1268, introduced by https://salsa.debian.org/installer-team/debootstrap/commit/c997b80c064c6c1d36ec69da1850722f795f43e4

- - - - -
62a6b3c4 by Ivo De Decker at 2018-10-31T16:27:48Z
Don't add results from outdated binaries to summary.json

Signed-off-by: Ivo De Decker <ivodd at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
ae52ab48 by Holger Levsen at 2018-10-31T16:28:50Z
add changelog entry for Ivo

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
21913449 by Holger Levsen at 2018-11-08T11:10:10Z
release as 0.94

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


10 changed files:

- custom-scripts/scripts/post_chroot_unpack_allow_unauthenticated
- custom-scripts/scripts/post_distupgrade_exceptions
- custom-scripts/scripts/pre_distupgrade_allow_unauthenticated
- custom-scripts/scripts/pre_install_exceptions
- debian/changelog
- master-bin/detect_well_known_errors.py
- master-bin/generate_daily_report.in
- piuparts-report.py
- piuparts.py
- slave-bin/detect_slave_problems.in


Changes:

=====================================
custom-scripts/scripts/post_chroot_unpack_allow_unauthenticated
=====================================
@@ -9,7 +9,10 @@ Acquire::Check-Valid-Until "false";
 EOF
 fi
 
-if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ]; then
+if [ "$PIUPARTS_DISTRIBUTION" = "squeeze" ] || \
+   [ "$PIUPARTS_DISTRIBUTION" = "squeeze-lts" ] || \
+   [ "$PIUPARTS_DISTRIBUTION" = "squeeze-backports" ] || \
+   [ "$PIUPARTS_DISTRIBUTION" = "squeeze-backports-sloppy" ]; then
 	echo "Creating /etc/apt/apt.conf.d/unauthenticated-squeeze ..."
 	tee /etc/apt/apt.conf.d/unauthenticated-squeeze <<EOF
 # The squeeze signing key has expired.


=====================================
custom-scripts/scripts/post_distupgrade_exceptions
=====================================
@@ -84,6 +84,12 @@ fi
 
 if [ "$PIUPARTS_DISTRIBUTION" = "buster" ]; then
 
+	# libc-bin only upgrades pristine /etc/nsswitch.conf
+	if grep -q '^passwd:.*compat' /etc/nsswitch.conf ; then
+		echo "Switching from compat to files in /etc/nsswitch.conf"
+		sed -r -i '/^(passwd|group|shadow):/ s/compat/files/' /etc/nsswitch.conf
+	fi
+
 	# dpkg does not properly clean up directories getting empty and no longer shipped
 	for dir in /etc/dbus-1/system.d /etc/dbus-1
 	do


=====================================
custom-scripts/scripts/pre_distupgrade_allow_unauthenticated
=====================================
@@ -23,6 +23,8 @@ Acquire::Check-Valid-Until "false";
 EOF
 elif [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze-backports" ]; then
 	:
+elif [ "$PIUPARTS_DISTRIBUTION_NEXT" = "squeeze-lts" ]; then
+	:
 elif [ -e /etc/apt/apt.conf.d/unauthenticated-squeeze ]; then
 	echo "FAIL: /etc/apt/apt.conf.d/unauthenticated-squeeze exists"
 	exit 1


=====================================
custom-scripts/scripts/pre_install_exceptions
=====================================
@@ -13,7 +13,7 @@ case ${PIUPARTS_OBJECTS%%=*} in
 				log_debug
 				echo 'Yes, do as I say!' | apt-get -y --force-yes install file-rc
 				;;
-			squeeze*|stretch*|sid)
+			squeeze*|stretch*)
 				# force installation and removal of essential package sysv-rc
 				log_debug
 				yes 'Yes, do as I say!' | apt-get -y --force-yes install file-rc


=====================================
debian/changelog
=====================================
@@ -1,3 +1,29 @@
+piuparts (0.94) unstable; urgency=medium
+
+  [ Holger Levsen ]
+  * master-bin/detect_well_known_errors.py: suppress output if nothing new is
+    found in a section, or section is busy or has been recently processed.
+  * master-bin/generate_daily_report: include number of failures in daily mail.
+  * slave-bin/detect_slave_problems: increase amount of time (from 30 to 60m)
+    a slave has to be inactive before complaining.
+
+  [ Andreas Beckmann ]
+  * piuparts.py:
+    - Logrotate files can only be placed directly in /etc/logrotate.d/.
+  * scripts/post_distupgrade_exceptions:
+    - Handle /etc/nsswitch.conf stretch -> buster upgrade if modifications
+      from libnss-myhostname and friends are present.
+
+  [ Mathieu Parent ]
+  * piuparts.py: Use mknod instead of touch to create missing /dev/ptmx
+    mountpoint, working around findutils bug #912180 (Closes: #911334)
+
+  [ Ivo De Decker ]
+  * piuparts-reports.py: Don't add results from outdated binaries to
+    summary.json.
+
+ -- Holger Levsen <holger at debian.org>  Thu, 08 Nov 2018 12:09:57 +0100
+
 piuparts (0.93) unstable; urgency=medium
 
   [ Andreas Beckmann ]


=====================================
master-bin/detect_well_known_errors.py
=====================================
@@ -113,7 +113,7 @@ def detect_well_known_errors(sections, config, problem_list, recheck, recheck_fa
         (section, next_try) = todo.popleft()
         now = time.time()
         if (now < next_try):
-            logging.info("Sleeping while sections are busy")
+            # sleeping, section has been tried recently
             time.sleep(max(30, next_try - now) + 30)
         try:
             (del_cnt, add_cnt) = \
@@ -123,11 +123,11 @@ def detect_well_known_errors(sections, config, problem_list, recheck, recheck_fa
             total_add += add_cnt
             current_time=time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime())
             if del_cnt == 0 and add_cnt == 0:
-                logging.info("%s - %s: nothing new" % (current_time, section))
+                pass # nothing new
             else:
                 logging.info("%s - %s: parsed logfiles: %d removed, %d added" % (current_time, section, del_cnt, add_cnt))
         except Busy:
-            logging.info("Section is busy")
+            # section is busy
             todo.append((section, time.time() + 300))
         except MissingSection:
             pass


=====================================
master-bin/generate_daily_report.in
=====================================
@@ -67,7 +67,7 @@ FAILURESTAMP=$HTDOCS/last-failure-report.stamp
 test -f $FAILURESTAMP || touch -d @0 $FAILURESTAMP  # start at the epoch
 touch $FAILURESTAMP.new  # for the next report
 
-echo "New failures:" >> $DAILYREPORT
+DAILYTMP=$(mktemp)
 for SECTION in $SECTIONS ; do
 	test -d $MASTER/$SECTION || continue
 	for DIRECTORY in fail bugged affected untestable ; do
@@ -75,7 +75,11 @@ for SECTION in $SECTIONS ; do
 	done
 	find $MASTER/$SECTION/fail $MASTER/$SECTION/bugged $MASTER/$SECTION/affected $MASTER/$SECTION/untestable \
 		-type f -name '*.log' -newer $FAILURESTAMP -exec ls -1 {} + 2>/dev/null
-done | sed s#^$MASTER#$URLBASE# >> $DAILYREPORT
+done | sed s#^$MASTER#$URLBASE# >> $DAILYTMP
+
+echo "$(cat $DAILYTMP | wc -l) new failures:" >> $DAILYREPORT
+cat $DAILYTMP >> $DAILYREPORT
+rm $DAILYTMP
 
 echo "" >> $DAILYREPORT
 date >> $DAILYREPORT


=====================================
piuparts-report.py
=====================================
@@ -1611,6 +1611,8 @@ class Section:
             summary = pkgsummary.new_summary()
 
             for reporting_section in reporting_sections:
+                source_done = {}
+                source_todo = {}
                 for binpkg in self._binary_db.get_all_packages():
                     pkgname = binpkg["Package"]
                     state = self._binary_db.get_package_state(pkgname)
@@ -1619,13 +1621,30 @@ class Section:
                     if flag == 'F':
                         block_cnt = self._binary_db.block_count(pkgname)
                     srcpkg = self._binary_db.get_package(pkgname).source()
+                    srcver = self._binary_db.get_package(pkgname).source_version()
+                    latest_srcver = self._source_db.get_test_versions(srcpkg)
                     url = source_summary_url(
                               web_host, self._doc_root,
                               self._config.section, srcpkg)
 
-                    pkgsummary.add_summary(
-                                   summary, reporting_section, srcpkg,
-                                   flag, block_cnt, url)
+                    if (srcver != latest_srcver):
+                        # if the binary package is cruft, we don't add it to
+                        # the report, but we keep track to make sure the
+                        # source package is added
+                        source_todo[srcpkg] = url
+                    else:
+                        pkgsummary.add_summary(
+                                       summary, reporting_section, srcpkg,
+                                       flag, block_cnt, url)
+                        source_done[srcpkg] = 1
+                for srcpkg in source_todo:
+                    # source packages that only have outdated binaries are
+                    # waiting for the build, so we add them as 'W'
+                    # (waiting-to-be-tested)
+                    if srcpkg not in source_done:
+                        pkgsummary.add_summary(
+                                       summary, reporting_section, srcpkg,
+                                       'W', 0, source_todo[srcpkg])
 
             pkgsummary.write_summary(summary, summary_path)
 


=====================================
piuparts.py
=====================================
@@ -1565,18 +1565,16 @@ class Chroot:
         self.run_scripts("post_remove")
 
         if not settings.skip_cronfiles_test:
-            cronfiles, cronfiles_list = self.check_if_cronfiles(packages)
-
-        if not settings.skip_cronfiles_test and cronfiles:
-            self.check_output_cronfiles(cronfiles_list)
+            cronfiles = self.check_if_cronfiles(packages)
+            if cronfiles:
+                self.check_output_cronfiles(cronfiles)
 
         if not settings.skip_logrotatefiles_test:
-            logrotatefiles, logrotatefiles_list = self.check_if_logrotatefiles(packages)
-
-        if not settings.skip_logrotatefiles_test and logrotatefiles:
-            installed = self.install_logrotate()
-            self.check_output_logrotatefiles(logrotatefiles_list)
-            self.purge_packages(installed)
+            logrotatefiles = self.check_if_logrotatefiles(packages)
+            if logrotatefiles:
+                installed = self.install_logrotate()
+                self.check_output_logrotatefiles(logrotatefiles)
+                self.purge_packages(installed)
 
         # Then purge all packages being depended on.
         self.purge_packages(deps_to_purge)
@@ -1757,8 +1755,7 @@ class Chroot:
         dev_ptmx_rel_path = self.relative("dev/ptmx")
         if not os.path.islink(dev_ptmx_rel_path):
             if not os.path.exists(dev_ptmx_rel_path):
-                with open(dev_ptmx_rel_path, 'w'):
-                    pass
+                os.mknod(dev_ptmx_rel_path, 0666 | stat.S_IFCHR, os.makedev(5, 2))
             self.mount(self.relative("dev/pts/ptmx"), "/dev/ptmx", opts="bind", no_mkdir=True)
         p = subprocess.Popen(["tty"], stdout=subprocess.PIPE)
         stdout, _ = p.communicate()
@@ -1829,9 +1826,9 @@ class Chroot:
         """Check if the packages have cron files under /etc/cron.d and in case positive,
         it returns the list of files. """
 
+        # FIXME! Does not work for M-A: same packages
         vdir = self.relative("var/lib/dpkg/info")
         vlist = []
-        has_cronfiles = False
         for p in packages:
             basename = p + ".list"
 
@@ -1846,12 +1843,10 @@ class Chroot:
                         mode = st[stat.ST_MODE]
                         # XXX /etc/cron.d/ files are NOT executables
                         if (mode & stat.S_IEXEC):
-                            if not has_cronfiles:
-                                has_cronfiles = True
                             vlist.append(pathname)
                             logging.info("Package " + p + " contains cron file: " + pathname)
 
-        return has_cronfiles, vlist
+        return vlist
 
     def check_output_cronfiles(self, list):
         """Check if a given list of cronfiles has any output. Executes
@@ -1874,9 +1869,9 @@ class Chroot:
         """Check if the packages have logrotate files under /etc/logrotate.d and in case positive,
         it returns the list of files. """
 
+        # FIXME! Does not work for M-A: same packages
         vdir = self.relative("var/lib/dpkg/info")
         vlist = []
-        has_logrotatefiles = False
         for p in packages:
             basename = p + ".list"
 
@@ -1885,14 +1880,12 @@ class Chroot:
 
             for line in readlines_file(os.path.join(vdir, basename)):
                 pathname = line.strip()
-                if pathname.startswith("/etc/logrotate.d/"):
+                if os.path.dirname(pathname) == "/etc/logrotate.d":
                     if os.path.isfile(self.relative(pathname.strip("/"))):
-                        if not has_logrotatefiles:
-                            has_logrotatefiles = True
                         vlist.append(pathname)
                         logging.info("Package " + p + " contains logrotate file: " + pathname)
 
-        return has_logrotatefiles, vlist
+        return vlist
 
     def install_logrotate(self):
         """Install logrotate for check_output_logrotatefiles, and return the


=====================================
slave-bin/detect_slave_problems.in
=====================================
@@ -49,18 +49,15 @@ STATEFILE=$SLAVEROOT/slave-problems
 # clear the statefile daily and whine again
 test $(file_age $STATEFILE) -lt 86000 || rm -f $STATEFILE
 
-# Only complain if screenlog is older than $IDLE_SPEEP + 1 minute (the slave
-# likes to sleep that long) and the problem is new or was not reported within
-# the previous 24 hours.
-if [ $(file_age $SCREENLOG) -le $(($IDLE_SLEEP + 60)) ]; then
+# Only complain if screenlog is older than $IDLE_SLEEP + 30 minutes
+# and the problem is new or was not reported within the previous 24 hours.
+if [ $(file_age $SCREENLOG) -le $(($IDLE_SLEEP + 1800)) ]; then
 	rm -f $STATEFILE
 elif [ ! -f $STATEFILE ]; then
 	{
-		echo "Either a test is running for a very long time (but no test"
-		echo "should run longer than an hour), piuparts-slave hangs or is"
-		echo "not running at all or wasn't started with"
-		echo "~piupartss/bin/slave_run - please investigate and take"
-		echo "appropriate measures!"
+		echo "Either a test is running for a very long time (more than $(( ($IDLE_SLEEP + 1800)/60 )) minutes),"
+		echo "or piuparts-slave hangs or is not running at all or wasn't started with"
+		echo "~piupartss/bin/slave_run - please investigate and take appropriate measures!"
 		echo
 		tail $SCREENLOG
 	} | mail -s "problem with piuparts-slave detected" piupartss



View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/6390f0a564a8c08fdb94c18b40d34c306f78e1ec...21913449777280d0a59f7dbb3c2e15de55154c2f

-- 
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/6390f0a564a8c08fdb94c18b40d34c306f78e1ec...21913449777280d0a59f7dbb3c2e15de55154c2f
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20181108/78df7aec/attachment-0001.html>


More information about the Piuparts-devel mailing list