[Piuparts-devel] [Git][debian/piuparts][master] 38 commits: continue development
Holger Levsen
gitlab at salsa.debian.org
Sun Dec 9 21:28:59 GMT 2018
Holger Levsen pushed to branch master at Debian / piuparts
Commits:
7513e22d by Holger Levsen at 2018-11-08T11:12:39Z
continue development
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b9644231 by Holger Levsen at 2018-11-25T13:53:19Z
piuparts.py: add /var/lib/debian-security-support/ to self.ignored_files. (Closes: #749317)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
8a4671ed by Holger Levsen at 2018-11-27T18:22:58Z
WIP: reduce number of mails per day, from currently 34 down to 32 - see #912591
- master-bin/detect_archive_issues: redirect logs to htdocs/logs to spare
us two mails per day.
- lib/write_log.sh: new helper to publish logs in htdocs/logs.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
71cd6fc8 by Holger Levsen at 2018-11-27T19:25:07Z
WIP: reduce number of mails per day, from 32 down to 24, see #912591.
master-bin/detect_piuparts_issues: redirect logs to htdocs/logs to spare
us eight mails per day.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4a28e24e by Holger Levsen at 2018-11-27T19:37:35Z
fixup 71cd6fc8: master-bin/detect_piuparts_issues
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f1978566 by Holger Levsen at 2018-11-27T19:40:59Z
WIP: reduce number of mails per day, from 24 down to 18, see #912591.
master-bin/detect_network_issues: redirect logs to htdocs/logs to spare us
six mails per day.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
44cfa21d by Holger Levsen at 2018-11-27T20:41:22Z
fixup, use month not minutes
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
a2c5e000 by Holger Levsen at 2018-11-27T20:45:12Z
piuparts-report.py: add navigation link to https://piuparts.d.o/logs/
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
ec3d6b90 by Holger Levsen at 2018-11-27T20:58:14Z
ignore vanishing files
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
fd98278a by Holger Levsen at 2018-11-27T22:49:13Z
fix grammar, thanks Haegar_
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
acc2a678 by Andreas Beckmann at 2018-11-28T09:49:31Z
p: clarify errors from installing objects over existing symlinks
objects can be files, directories, ...
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e63d9372 by Holger Levsen at 2018-11-28T10:37:39Z
WIP: reduce number of mails per day, from 18 down to 16, see #912591.
master-bin/reschedule_oldest_logs: redirect logs to htdocs/logs to spare
us two mails per day.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
025ed722 by Holger Levsen at 2018-11-28T11:29:24Z
update copyright
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
655b78b6 by Holger Levsen at 2018-11-28T11:36:48Z
WIP: reduce number of mails per day, from 16 down to 15, see #912591.
master-bin/rotate_master_logs: redirect logs to htdocs/logs to spare us
one mail per day.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f7faed86 by Holger Levsen at 2018-11-28T13:00:28Z
WIP: reduce number of mails per day, from 15 down to 2, see #912591.
- master-bin/master_shell_runner: new script to run python scripts to
redirect logs to htdocs/logs.
- conf/crontab-master.in: run detect_well_known_errors and piuparts-report
through the new master_shell_runner script.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7ed32f56 by Holger Levsen at 2018-11-28T13:42:06Z
shorten changelog
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b705a3c6 by Holger Levsen at 2018-11-28T13:45:20Z
conf/crontab-master.in: also run piuparts-analyze through the new master_shell_runner script.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
2e455394 by Holger Levsen at 2018-11-28T14:15:16Z
htdocs/news.tpl: explain the result of these changes.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
9f404e33 by Holger Levsen at 2018-11-28T14:24:53Z
also redirect stderr
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
bb7d3add by Holger Levsen at 2018-11-28T16:19:54Z
rename variable
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e7999f03 by Holger Levsen at 2018-11-28T16:35:37Z
add support for daily prefix text (to not repeat meta-information in the logs)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
53db400a by Holger Levsen at 2018-11-28T20:04:10Z
remove WIP from d/changelog, #912591 is basically done
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1808bc1e by Holger Levsen at 2018-11-29T09:50:43Z
WIP: reduce number of mails per day, from 4 down to 3, see #912591.
master-bin/report_untestable_packages: redirect logs to htdocs/logs to spare
one mail per day.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
107b4297 by Holger Levsen at 2018-11-29T10:47:05Z
WIP: reduce number of mails per day, from 3 down to 2, see #912591.
master-bin/generate_daily_report:
- redirect logs of called scripts to htdocs/logs.
- include listing of htdocs/logs/$TODAY into daily report.
- publish dailyreport.txt in htdocs/logs/$TODAY instead of htdocs/.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
c7428dd5 by Holger Levsen at 2018-11-29T10:56:04Z
d/control: use canonical URL for Vcs-Git, thanks lintian.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f03d3d27 by Holger Levsen at 2018-11-29T11:28:06Z
Add d/piuparts.lintian-overrides to override the incorrect uses-dpkg-database-directly warnings, after confirming we are not doing that.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b09e994f by Holger Levsen at 2018-11-29T12:32:37Z
improve output
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
64d19db9 by Holger Levsen at 2018-11-29T13:04:07Z
master-bin/rotate_master_logs: compress htdocs/logs older than a week.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
fb503cec by Holger Levsen at 2018-11-30T11:07:34Z
improve output
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7a15f345 by Holger Levsen at 2018-11-30T11:16:05Z
drop 'set -e
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3fd152b8 by Holger Levsen at 2018-12-02T13:46:50Z
redirect output to htdocs/logs
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
307ad431 by Holger Levsen at 2018-12-04T12:22:32Z
bug-templates/logrotate_exits_with_error_after_package_removal: clarify the cause of this problem.
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
d13b9f56 by Holger Levsen at 2018-12-05T13:00:15Z
xz --verbose leads to output on stderr...
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
8bf4ab90 by Andreas Beckmann at 2018-12-05T14:48:11Z
symlink_to_dir/dir_to_symlink are supported by dpkg/oldstable
Signed-off-by: Andreas Beckmann <anbe at debian.org>
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
40638b57 by Holger Levsen at 2018-12-05T14:49:49Z
add changelog entry for Andreas
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
7aa4aedd by Holger Levsen at 2018-12-06T12:01:48Z
fixup d13b9f56: xz --verbose leads to output on stderr...
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
50807bd5 by Holger Levsen at 2018-12-06T13:03:59Z
bug-templates/logrotate_exits_with_error_after_package_removal: explain 2nd possible cause and solution
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
0b74c92d by Holger Levsen at 2018-12-09T21:27:48Z
release as 0.95
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
23 changed files:
- bug-templates/copyright_file_missing_after_upgrade.mail
- bug-templates/logrotate_exits_with_error_after_package_removal.mail
- bug-templates/unhandled_symlink_to_directory_conversion.mail
- conf/crontab-master.in
- debian/changelog
- debian/control
- + debian/piuparts.lintian-overrides
- htdocs/news.tpl
- known_problems/installs_over_symlink_error.conf
- known_problems/installs_over_symlink_issue.conf
- + lib/write_log.sh
- master-bin/detect_archive_issues.in
- master-bin/detect_network_issues.in
- master-bin/detect_piuparts_issues.in
- master-bin/detect_well_known_errors.py
- master-bin/generate_daily_report.in
- + master-bin/master_shell_runner.in
- master-bin/report_stale_reserved_packages.in
- master-bin/report_untestable_packages.in
- master-bin/reschedule_oldest_logs.in
- master-bin/rotate_master_logs.in
- piuparts-report.py
- piuparts.py
Changes:
=====================================
bug-templates/copyright_file_missing_after_upgrade.mail
=====================================
@@ -33,7 +33,6 @@ https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#details-of-un
It is recommended to use the dpkg-maintscript-helper commands
'dir_to_symlink' and 'symlink_to_dir' (available since dpkg 1.17.14)
to perform the conversion, ideally using d/$PACKAGE.maintscript.
-Do not forget to add 'Pre-Depends: ${misc:Pre-Depends}' in d/control.
See dpkg-maintscript-helper(1) and dh_installdeb(1) for details.
=====================================
bug-templates/logrotate_exits_with_error_after_package_removal.mail
=====================================
@@ -9,9 +9,14 @@ Usertags: piuparts
Hi,
-during a test with piuparts I noticed your package's logrotate
-configuration causes logrotate to exit with an error after
-the package has been removed.
+during a test with piuparts I noticed your package's logrotate configuration
+causes logrotate to exit with an error after the package has been removed (*) or
+when logrote is run but no logfile exists.
+
+Usually the solution is to specify 'missingok' in the logrotate configuration.
+
+*) logrotate configuration files remain installed and executed after a package
+has been removed, they only get removed when the package is purged.
From the attached log (scroll to the bottom...):
=====================================
bug-templates/unhandled_symlink_to_directory_conversion.mail
=====================================
@@ -36,7 +36,6 @@ https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#details-of-un
It is recommended to use the dpkg-maintscript-helper commands
'dir_to_symlink' and 'symlink_to_dir' (available since dpkg 1.17.14)
to perform the conversion, ideally using d/$PACKAGE.maintscript.
-Do not forget to add 'Pre-Depends: ${misc:Pre-Depends}' in d/control.
See dpkg-maintscript-helper(1) and dh_installdeb(1) for details.
=====================================
conf/crontab-master.in
=====================================
@@ -13,9 +13,9 @@ PYTHONPATH=@libdir@/python2.7/dist-packages
# generate reports twice or three times a day
# (dinstall runs 1|7|13|19:52, so this is long after mirror pushes...)
#
-0 4-23/12 * * * piuparts-analyze
-0 2-23/3 * * * detect_well_known_errors
-0 0-23/3 * * * piuparts-report
+0 4-23/12 * * * master_shell_runner piuparts-analyze
+0 2-23/3 * * * master_shell_runner detect_well_known_errors
+0 0-23/3 * * * master_shell_runner piuparts-report
15 23 * * * generate_daily_report
#
=====================================
debian/changelog
=====================================
@@ -1,3 +1,40 @@
+piuparts (0.95) unstable; urgency=medium
+
+ [ Holger Levsen ]
+ * piuparts.py: add /var/lib/debian-security-support/ to self.ignored_files.
+ (Closes: #749317)
+ * Reduce number of mails per day, from 34 down to 1. (Closes: #912591)
+ - lib/write_log.sh: new helper to publish logs in htdocs/logs.
+ - master-bin/master_shell_runner: new script to run python scripts to
+ redirect logs to htdocs/logs.
+ - master-bin/generate_daily_report:
+ - redirect logs of called scripts to htdocs/logs.
+ - include listing of htdocs/logs/$TODAY into daily report.
+ - publish dailyreport.txt in htdocs/logs/$TODAY instead of htdocs/.
+ - master-bin/detect_archive_issues, detect_piuparts_issues,
+ detect_network_issues, reschedule_oldest_logs, rotate_master_logs,
+ report_untestable_packages and report_stale_reserved_packages: redirect
+ logs to htdocs/logs.
+ - conf/crontab-master.in: run detect_well_known_errors, piuparts-report
+ and piuparts-analyze through the new master_shell_runner script.
+ - master-bin/rotate_master_logs: compress htdocs/logs older than a week.
+ - piuparts-report.py: add navigation link to https://piuparts.d.o/logs/
+ - htdocs/news.tpl: explain the result of these changes.
+ * d/control: use canonical URL for Vcs-Git, thanks lintian.
+ * Add d/piuparts.lintian-overrides to override the incorrect
+ uses-dpkg-database-directly warnings, after confirming we are not doing
+ that.
+ * bug-templates/logrotate_exits_with_error_after_package_removal: clarify
+ possibles causes of this problem as well as the solution.
+
+ [ Andreas Beckmann ]
+ * Clarify errors from installing objects over existing symlinks.
+ * bug-templates/copyright_file_missing_after_upgrade and
+ bug-templates/unhandled_symlink_to_directory_conversion: adjust advice as
+ symlink_to_dir/dir_to_symlink are supported by dpkg in oldstable.
+
+ -- Holger Levsen <holger at debian.org> Sun, 09 Dec 2018 22:27:41 +0100
+
piuparts (0.94) unstable; urgency=medium
[ Holger Levsen ]
=====================================
debian/control
=====================================
@@ -24,7 +24,7 @@ Build-Depends:
Build-Depends-Indep:
git,
Homepage: https://piuparts.debian.org
-Vcs-Git: https://salsa.debian.org/debian/piuparts
+Vcs-Git: https://salsa.debian.org/debian/piuparts.git
Vcs-Browser: https://salsa.debian.org/debian/piuparts
Xs-Go-Import-Path: salsa.debian.org/debian/piuparts
=====================================
debian/piuparts.lintian-overrides
=====================================
@@ -0,0 +1,5 @@
+piuparts: uses-dpkg-database-directly etc/piuparts/scripts-debug-purge/post_remove_postrm_set-x
+piuparts: uses-dpkg-database-directly etc/piuparts/scripts-debug-remove/pre_remove_prerm_postrm_set-x
+piuparts: uses-dpkg-database-directly etc/piuparts/scripts/post_remove_exceptions
+piuparts: uses-dpkg-database-directly etc/piuparts/scripts/pre_remove_40_find_missing_md5sums
+piuparts: uses-dpkg-database-directly usr/sbin/piuparts
=====================================
htdocs/news.tpl
=====================================
@@ -6,9 +6,13 @@
</tr>
<tr class="normalrow">
<td class="contentcell2">
- <b>2018-10-19</b>: Three and a half years after stopping linking to packages.qa.debian.org(which was called <em>PTS</em>) we finally drop all references to the PTS and use <em>tracker.d.o</em> instead.
+ <b>2018-11-28</b>: The output of various maintenance scripts is now available at <a href="https://piuparts.debian.org/logs/" target="_blank">https://piuparts.debian.org/logs/</a> instead of sending it via mails to the piuparts.d.o maintainers only.
</td>
</tr>
+ <tr class="normalrow">
+ <td class="contentcell2">
+ <b>2018-10-19</b>: Three and a half years after stopping linking to packages.qa.debian.org (which was called <em>PTS</em>) we finally drop all references to the PTS and use <em>tracker.d.o</em> instead.
+ </td>
</tr>
<tr class="normalrow">
<td class="contentcell2">
=====================================
known_problems/installs_over_symlink_error.conf
=====================================
@@ -1,7 +1,7 @@
#
# detect packages which have the string "dirname part contains a symlink" in their logs
#
-PATTERN='silently overwrites files via directory symlinks|dirname part contains a symlink'
+PATTERN='silently overwrites files via directory symlinks|installs objects over existing directory symlinks|dirname part contains a symlink'
WHERE='fail bugged affected'
ISSUE=0
HEADER='Packages that install something over existing symlinks'
=====================================
known_problems/installs_over_symlink_issue.conf
=====================================
@@ -1,7 +1,7 @@
#
# detect packages which have the string "dirname part contains a symlink" in their logs
#
-PATTERN='silently overwrites files via directory symlinks|dirname part contains a symlink'
+PATTERN='silently overwrites files via directory symlinks|installs objects over existing directory symlinks|dirname part contains a symlink'
WHERE='pass'
ISSUE=1
HEADER='Packages that install something over existing symlinks'
=====================================
lib/write_log.sh
=====================================
@@ -0,0 +1,36 @@
+# Copyright © 2018 Holger Levsen (holger at debian.org)
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>
+
+#
+# Helper function for publishing logfiles
+#
+
+publish_logs() {
+ local LOG_OUTPUT=$1
+ local LOG_PREFIX=$2
+ local HTDOCS="$3"
+ local LOG=$4
+ local YEAR=$(date -u +%Y)
+ local MONTH=$(date -u +%m)
+ local DAY=$(date -u +%d)
+ local DIR="$HTDOCS/logs/$YEAR/$MONTH/$DAY"
+ mkdir -p "$DIR"
+ if [ -n "$LOG_PREFIX" ] && [ ! -s "$DIR/$LOG.txt" ] ; then
+ cat $LOG_PREFIX >> "$DIR/$LOG.txt"
+ rm -f $LOG_PREFIX >/dev/null
+ fi
+ cat $LOG_OUTPUT >> "$DIR/$LOG.txt"
+ rm -f $LOG_OUTPUT >/dev/null
+}
=====================================
master-bin/detect_archive_issues.in
=====================================
@@ -18,13 +18,17 @@
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
+get_config_value HTDOCS global output-directory
get_config_value SECTIONS global sections
get_config_value URLBASE global urlbase https://piuparts.debian.org
get_config_value DAYS global reschedule-untestable-days 7
ISSUE_LOG="$MASTER/archive_issues.txt"
+LOG_OUTPUT=`mktemp`
+LOG_PREFIX=$(mktemp)
STARTDATE=$(date -u +%s)
@@ -59,21 +63,28 @@ if [ -s $URLS ]; then
cat $URLS >> $ISSUE_LOG
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
- echo "Runtime: $RUNTIME"
- echo
- echo "Broken packages detected! The following packages have been moved to"
- echo "$section/untestable and will be tested again in $DAYS days."
- echo
- echo "Broken packages are usually a temporary problem in the archive and are"
- echo "caught by other tools like britney or https://qa.debian.org/dose/debcheck.html"
- echo "If it is always the same package failing, it's likely to be an issue in the"
- echo "package."
- echo
- grep -f $URLS $ISSUE_LOG | sort | uniq -c | sort -rn
- echo
- echo "Those problems were found in failed logs by grep'ing for these patterns:"
- echo " 'E: Broken packages'"
- echo " 'E: Unable to correct problems, you have held broken packages'"
- echo " 'E: Error, pkgProblemResolver::Resolve generated breaks'"
+ (
+ echo "Broken packages detected! The following packages have been moved to"
+ echo "$section/untestable and will be tested again in $DAYS days."
+ echo
+ echo "Broken packages are usually a temporary problem in the archive and are"
+ echo "caught by other tools like britney or https://qa.debian.org/dose/debcheck.html"
+ echo "If it is always the same package failing, it's likely to be an issue in the"
+ echo "package."
+ echo
+ echo "Those problems were found in failed logs by grep'ing for these patterns:"
+ echo " 'E: Broken packages'"
+ echo " 'E: Unable to correct problems, you have held broken packages'"
+ echo " 'E: Error, pkgProblemResolver::Resolve generated breaks'"
+ echo
+ ) > $LOG_PREFIX
+ (
+ echo "$(date -u)"
+ echo "Runtime: $RUNTIME"
+ echo
+ grep -f $URLS $ISSUE_LOG | sort | uniq -c | sort -rn
+ echo
+ ) > $LOG_OUTPUT
+ publish_logs $LOG_OUTPUT $LOG_PREFIX $HTDOCS archive_issues
fi
rm $LOGS $URLS
=====================================
master-bin/detect_network_issues.in
=====================================
@@ -18,11 +18,15 @@
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
+get_config_value HTDOCS global output-directory
STARTDATE=$(date -u +%s)
+LOG_OUTPUT=$(mktemp)
+LOG_PREFIX=$(mktemp)
#
# detect network/mirror problems
@@ -72,18 +76,25 @@ done
if [ -s $FILE ] ; then
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
- echo "Runtime: $RUNTIME"
- echo
- echo "Network problems on detected! The following logfiles have been deleted:"
- echo
- for log in $(sort -u $FILE)
- do
- echo "$log" | cut -d "/" -f5-
- grep -E "$PATTERN" "$log" | perl -e 'print grep !$seen{$_}++, <>;' | head -n 10 | sed "s/^/${tab}/"
- rm -f "$log"
- done
- echo
- echo "Those problems were found in failed logs by grep'ing for these patterns:"
- echo "$DISPLAY_PATTERN"
+ (
+ echo "Network problems detected! The following logfiles have been deleted:"
+ echo
+ echo "Those problems were found in failed logs by grep'ing for these patterns:"
+ echo "$DISPLAY_PATTERN"
+ echo
+ ) > $LOG_PREFIX
+ (
+ echo "$(date -u)"
+ echo "Runtime: $RUNTIME"
+ echo
+ for log in $(sort -u $FILE)
+ do
+ echo "$log" | cut -d "/" -f5-
+ grep -E "$PATTERN" "$log" 2>/dev/null | perl -e 'print grep !$seen{$_}++, <>;' | head -n 10 | sed "s/^/${tab}/"
+ rm -f "$log"
+ done
+ echo
+ ) > $LOG_OUTPUT
+ publish_logs $LOG_OUTPUT $LOG_PREFIX $HTDOCS network_issues
fi
rm $FILE
=====================================
master-bin/detect_piuparts_issues.in
=====================================
@@ -18,11 +18,15 @@
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
+get_config_value HTDOCS global output-directory
STARTDATE=$(date -u +%s)
+LOG_OUTPUT=$(mktemp)
+LOG_PREFIX=$(mktemp)
#
# detect piuparts problems
@@ -95,24 +99,30 @@ done
if [ -s $FILE ] ; then
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
- echo "Runtime: $RUNTIME"
- echo
- echo "piuparts problems detected! Please review the following logfiles/packages."
- echo "If it is always the same package failing, it's likely to be an issue in the"
- echo "package."
- echo
- echo "The following logfiles have been deleted:"
- echo
- for log in $(sort -u $FILE)
- do
- echo "$log" | cut -d "/" -f5-
- test -s "$log" || echo "${tab}Empty logfile."
- grep -E "$PATTERN" "$log" | perl -e 'print grep !$seen{$_}++, <>;' | head -n 10 | sed "s/^/${tab}/"
- rm -f "$log"
- done
- echo
- echo "Those problems were found in failed logs by grep'ing for these patterns:"
- echo "$DISPLAY_PATTERN"
- echo
+ (
+ echo "piuparts problems detected! Please review the following logfiles/packages."
+ echo "If it is always the same package failing, it's likely to be an issue in the"
+ echo "package."
+ echo
+ echo "Problems were found in failed logs by grep'ing for these patterns:"
+ echo "$DISPLAY_PATTERN"
+ echo
+ ) > $LOG_PREFIX
+ (
+ echo "$(date -u)"
+ echo "Runtime: $RUNTIME"
+ echo
+ echo "The following logfiles have been deleted:"
+ echo
+ for log in $(sort -u $FILE)
+ do
+ echo "$log" | cut -d "/" -f5-
+ test -s "$log" || echo "${tab}Empty logfile."
+ grep -E "$PATTERN" "$log" 2>/dev/null | perl -e 'print grep !$seen{$_}++, <>;' | head -n 10 | sed "s/^/${tab}/"
+ rm -f "$log"
+ done
+ echo
+ ) > $LOG_OUTPUT
+ publish_logs $LOG_OUTPUT $LOG_PREFIX $HTDOCS piuparts_issues
fi
rm $FILE
=====================================
master-bin/detect_well_known_errors.py
=====================================
@@ -3,6 +3,7 @@
# Copyright 2013 David Steele (dsteele at gmail.com)
# Copyright © 2014 Andreas Beckmann (anbe at debian.org)
+# Copyright © 2017-2018 Holger Levsen (holger at layer-acht.org)
#
# This file is part of Piuparts
#
@@ -132,9 +133,9 @@ def detect_well_known_errors(sections, config, problem_list, recheck, recheck_fa
except MissingSection:
pass
+ current_time=time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime())
+ logging.info("%s - total parsed logfiles: %d removed, %d added" % (current_time, total_del, total_add))
logging.info("")
- logging.info(time.strftime("%a %b %2d %H:%M:%S %Z %Y", time.localtime()))
- logging.info("total parsed logfiles: %d removed, %d added" % (total_del, total_add))
if __name__ == '__main__':
=====================================
master-bin/generate_daily_report.in
=====================================
@@ -1,6 +1,7 @@
#!/bin/sh
# Copyright © 2011-2013 Andreas Beckmann (anbe at debian.org)
+# Copyright © 2018 Holger Levsen (holger at layer-acht.org)
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
@@ -15,8 +16,8 @@
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <https://www.gnu.org/licenses/>
-
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
@@ -24,7 +25,6 @@ get_config_value HTDOCS global output-directory
get_config_value URLBASE global urlbase https://piuparts.debian.org
get_config_value PYTHONPATH global PYTHONPATH ''
-
export PYTHONPATH
LOCKFILE=$HTDOCS/daily.lock
@@ -36,22 +36,26 @@ fi
date > $LOCKFILE
R_STARTDATE=$(date -u +%s)
+R_LOGDATEDIR="$HTDOCS/logs/$(date -u '+%Y/%m/%d')"
#
-# various scripts to detect issues and cleanup
+# various scripts to detect issues and cleanup to be run right before we generate the daily report
+# (this code block contains cruft and could be refactored)
#
OUTPUT=$(mktemp)
+LOG_OUTPUT=$(mktemp)
for SCRIPT in archive_old_logs ; do
STARTDATE=$(date -u +%s)
@sharedir@/piuparts/master/$SCRIPT 2>&1 > $OUTPUT
FINALDATE=$(date -u +%s)
if [ -s $OUTPUT ] ; then
if grep -q "Runtime: " $OUTPUT ; then
- cat $OUTPUT | mail -s $SCRIPT $LOGNAME
+ cat $OUTPUT > $LOG_OUTPUT
else
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
- ( echo "Runtime: $RUNTIME" ; echo ; cat $OUTPUT ) | mail -s $SCRIPT $LOGNAME
+ ( echo "Runtime: $RUNTIME" ; echo ; cat $OUTPUT ) > $LOG_OUTPUT
fi
+ publish_logs $LOG_OUTPUT "" $HTDOCS $SCRIPT
fi
done
rm $OUTPUT
@@ -96,7 +100,15 @@ rm $expire
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $R_STARTDATE seconds" +%T)
-( echo "Runtime: $RUNTIME" ; echo ; cat $DAILYREPORT ) | mail -s generate_daily_report $LOGNAME
+(
+ echo "Runtime: $RUNTIME"
+ echo
+ echo $R_LOGDATEDIR
+ ls -lt $R_LOGDATEDIR
+ echo
+ cat $DAILYREPORT
+) | mail -s generate_daily_report $LOGNAME
+mv $DAILYREPORT $R_LOGDATEDIR
mv $FAILURESTAMP.new $FAILURESTAMP
rm -f $LOCKFILE
=====================================
master-bin/master_shell_runner.in
=====================================
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+# Copyright © 2018 Holger Levsen (holger at layer-acht.org)
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>
+
+. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
+
+get_config_value HTDOCS global output-directory
+
+LOG_OUTPUT=$(mktemp)
+
+$1 >$LOG_OUTPUT 2>&1
+publish_logs $LOG_OUTPUT "" $HTDOCS $1
=====================================
master-bin/report_stale_reserved_packages.in
=====================================
@@ -1,6 +1,6 @@
#!/bin/sh
-# Copyright 2009,2010 Holger Levsen (holger at layer-acht.org)
+# Copyright 2009-2018 Holger Levsen (holger at layer-acht.org)
# Copyright © 2011-2012 Andreas Beckmann (anbe at debian.org)
#
# This program is free software; you can redistribute it and/or modify it
@@ -16,12 +16,12 @@
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <https://www.gnu.org/licenses/>
-
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
-
+get_config_value HTDOCS global output-directory
#
# find packages which have been reserved for more than one day and unschedule them
@@ -34,11 +34,13 @@ for SECTION in $SECTIONS ; do
find $MASTER/$SECTION/reserved/ -mtime +$DAYS -name "*.log" 2>/dev/null >> $LOGS
done
if [ -s $LOGS ] ; then
- echo "Stale reserved packages detected, which have been reserved more then $DAYS days ago!"
- echo "These packages have been unscheduled."
- echo
+ LOG_OUTPUT=$(mktemp)
+ echo "Stale reserved packages detected, which have been reserved more then $DAYS days ago!" > $LOG_OUTPUT
+ echo "These packages have been unscheduled." >> $LOG_OUTPUT
+ echo >> $LOG_OUTPUT
for package_log in $(cat $LOGS) ; do
- rm -fv $package_log
+ rm -fv $package_log >> $LOG_OUTPUT
done
+ publish_logs $LOG_OUTPUT "" $HTDOCS report_stale_reserved_packages
fi
rm $LOGS
=====================================
master-bin/report_untestable_packages.in
=====================================
@@ -18,12 +18,16 @@
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
+get_config_value HTDOCS global output-directory
get_config_value DAYS global reschedule-untestable-days 7
STARTDATE=$(date -u +%s)
+LOG_OUTPUT=$(mktemp)
+LOG_PREFIX=$(mktemp)
#
# find packages which have been in untestable for more than $DAYS days and reschedule them for testing
@@ -41,13 +45,20 @@ done
if [ -s $LOGS ] ; then
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
- echo "Runtime: $RUNTIME"
- echo
- echo "Untestable packages detected, which have been tested more than $DAYS days ago!"
- echo "These packages have been rescheduled for piuparts testing."
- echo
- for package_log in $(cat $LOGS) ; do
- rm -fv $package_log | sed "s#$MASTER/##g" | tr -d "'"
- done
+ (
+ echo "Untestable packages detected, which have been tested more than $DAYS days ago!"
+ echo "These packages have been rescheduled for piuparts testing."
+ echo
+ ) > $LOG_PREFIX
+ (
+ echo "$(date -u)"
+ echo "Runtime: $RUNTIME"
+ echo
+ for package_log in $(cat $LOGS) ; do
+ rm -fv $package_log | sed "s#$MASTER/##g" | tr -d "'"
+ done
+ echo
+ ) > $LOG_OUTPUT
+ publish_logs $LOG_OUTPUT $LOG_PREFIX $HTDOCS report_untestable_packages
fi
rm $LOGS
=====================================
master-bin/reschedule_oldest_logs.in
=====================================
@@ -1,6 +1,6 @@
#!/bin/sh
-# Copyright © 2009-2017 Holger Levsen (holger at layer-acht.org)
+# Copyright © 2009-2018 Holger Levsen (holger at layer-acht.org)
# Copyright © 2011-2012 Andreas Beckmann (anbe at debian.org)
#
# This program is free software; you can redistribute it and/or modify it
@@ -18,12 +18,15 @@
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value MASTER global master-directory
get_config_value SECTIONS global sections
+get_config_value HTDOCS global output-directory
STARTDATE=$(date -u +%s)
OUTPUT=$(mktemp)
+LOG_OUTPUT=$(mktemp)
#
# reschedule 200 oldest log files, if they are older than 180 days
@@ -38,7 +41,6 @@ get_config_value COUNT global reschedule-old-count 200
get_config_value EXPIRE_FAIL_AGE global expire-fail-days 0
get_config_value FAIL_AGE global reschedule-fail-days 30
get_config_value FAIL_COUNT global reschedule-fail-count 25
-
get_config_value AUTO_RESCHEDULE global auto-reschedule yes
@@ -187,22 +189,27 @@ rm $QUEUED
FINALDATE=$(date -u +%s)
RUNTIME=$(date -u -d "0 $FINALDATE seconds - $STARTDATE seconds" +%T)
-echo "Runtime: $RUNTIME"
-echo
-
-if [ "$TOTAL" -gt "0" ]; then
- echo "Rescheduled $TOTAL logs."
-fi
-if [ "$TOTAL_EXPIRED" -gt "0" ]; then
- echo "Deleted $TOTAL_EXPIRED expired logs."
-fi
-if [ "$UNSCHEDULE" -gt "0" ]; then
- echo "Cancelled $UNSCHEDULE outdated rescheduling requests."
-fi
-if [ "$TOTAL_QUEUED" -gt "0" ]; then
- echo "Queued logs: $TOTAL_QUEUED"
-fi
-echo
-
-cat $OUTPUT
+(
+ echo "$(date -u)"
+ echo "Runtime: $RUNTIME"
+ echo
+ if [ "$TOTAL" -gt "0" ]; then
+ echo "Rescheduled $TOTAL logs."
+ fi
+ if [ "$TOTAL_EXPIRED" -gt "0" ]; then
+ echo "Deleted $TOTAL_EXPIRED expired logs."
+ fi
+ if [ "$UNSCHEDULE" -gt "0" ]; then
+ echo "Cancelled $UNSCHEDULE outdated rescheduling requests."
+ fi
+ if [ "$TOTAL_QUEUED" -gt "0" ]; then
+ echo "Queued logs: $TOTAL_QUEUED"
+ fi
+ echo
+ cat $OUTPUT
+ echo "##################################################################################################################" >> $OUTPUT
+ echo "##################################################################################################################" >> $OUTPUT
+ echo
+) > $LOG_OUTPUT
+publish_logs $LOG_OUTPUT "" $HTDOCS reschedule_oldest_logs
rm -f $OUTPUT
=====================================
master-bin/rotate_master_logs.in
=====================================
@@ -1,7 +1,7 @@
#!/bin/sh
-set -e
-
+#
# Copyright © 2017 Andreas Beckmann (anbe at debian.org)
+# Copyright © 2018 Holger Levsen (holger at layer-acht.org)
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
@@ -21,14 +21,15 @@ set -e
# * (delayed) compress rotated master logs
#
-
. @sharedir@/piuparts/lib/read_config.sh
+. @sharedir@/piuparts/lib/write_log.sh
get_config_value SECTIONS global sections
get_config_value MASTER global master-directory
-
+get_config_value HTDOCS global output-directory
today=$(date +%Y%m%d)
+OUTPUT=$(mktemp)
cd $MASTER
for SECTION in $SECTIONS ; do
@@ -37,14 +38,32 @@ for SECTION in $SECTIONS ; do
# rotate master.log larger than 1 MB
if [ -f $mlog ] && [ ! -L $mlog ] && [ $(stat -c %s $mlog) -gt 1048576 ] && [ ! -e $rlog ]; then
- mv -v $mlog $rlog
+ mv -v $mlog $rlog >> $OUTPUT
fi
# delayed compression of old master.log
for log in $SECTION/master.*.log
do
if [ -f "$log" ] && [ "$log" != "$rlog" ]; then
- xz -v9 $log
+ xz -v9 $log >>$OUTPUT 2>&1
fi
done
done
+
+LOGDATEDIR="$HTDOCS/logs/$(date -u '+%Y/%m/%d' -d '7 days ago')"
+if [ -d $LOGDATEDIR ] && [ -n "$(find $LOGDATEDIR -type f ! -name '*.xz')" ] ; then
+ echo >> $OUTPUT
+ echo "Compressing old piuparts.d.o logfiles." >> $OUTPUT
+ echo >> $OUTPUT
+ find $LOGDATEDIR -type f ! -name '*.xz' -exec xz -v9 {} \; >>$OUTPUT 2>&1
+fi
+
+if [ -s $OUTPUT ] ; then
+ LOG_OUTPUT=$(mktemp)
+ echo "$(date -u)" > $LOG_OUTPUT
+ echo >> $LOG_OUTPUT
+ cat $OUTPUT >> $LOG_OUTPUT
+ echo >> $LOG_OUTPUT
+ publish_logs $LOG_OUTPUT "" $HTDOCS rotate_master_logs
+fi
+rm $OUTPUT
=====================================
piuparts-report.py
=====================================
@@ -135,8 +135,9 @@ HTML_HEADER = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<td class="contentcell">
piuparts.d.o configuration:<br>
<a href="https://salsa.debian.org/debian/piuparts/tree/develop/instances" target="_blank">piuparts.conf</a>,<br>
- <a href="https://salsa.debian.org/debian/piuparts/blob/develop/conf/distros.conf" target="_blank">distros.conf</a><br> and
- <a href="https://salsa.debian.org/debian/piuparts/tree/develop/custom-scripts" target="_blank">scripts</a>
+ <a href="https://salsa.debian.org/debian/piuparts/blob/develop/conf/distros.conf" target="_blank">distros.conf</a>,<br>
+ <a href="https://salsa.debian.org/debian/piuparts/tree/develop/custom-scripts" target="_blank">scripts</a> and
+ <a href="https://piuparts.debian.org/logs/" target="_blank">logs</a>
</td>
</tr>
<tr class="normalrow">
=====================================
piuparts.py
=====================================
@@ -333,6 +333,7 @@ class Settings:
# actually, only modification should be permitted here, but not creation/removal
"/var/log/fontconfig.log",
# home directories of system accounts
+ "/var/lib/debian-security-support/", # #749317
"/var/lib/gozerbot/",
"/var/lib/nagios/", # nagios* (#668756)
"/var/lib/onioncat/", # onioncat
@@ -1488,13 +1489,15 @@ class Chroot:
bad.append(" %s -> %s" % (link, target))
if bad:
if overwrites:
- logging.error("FAIL: silently overwrites files via directory symlinks:\n" +
- indent_string("\n".join(bad)))
+ msg = "FAIL: silently overwrites files via directory symlinks:\n"
else:
- logging.info("dirname part contains a symlink:\n" +
- indent_string("\n".join(bad)))
+ msg = "installs objects over existing directory symlinks:\n"
+ msg += indent_string("\n".join(bad))
if not settings.warn_on_install_over_symlink:
+ logging.error(msg)
panic()
+ else:
+ logging.info(msg)
def remove_packages(self, packages, ignore_errors=False):
"""Remove packages in a chroot."""
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/21913449777280d0a59f7dbb3c2e15de55154c2f...0b74c92d6f9fa676e17802eab52d11a1774b2623
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/21913449777280d0a59f7dbb3c2e15de55154c2f...0b74c92d6f9fa676e17802eab52d11a1774b2623
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20181209/f8fa9714/attachment-0001.html>
More information about the Piuparts-devel
mailing list