[Piuparts-devel] [Git][debian/piuparts][develop] 6 commits: piuparts-master-backend: handle BrokenPipeError
Holger Levsen
gitlab at salsa.debian.org
Sat Dec 28 12:30:20 GMT 2019
Holger Levsen pushed to branch develop at Debian / piuparts
Commits:
9171821b by Nis Martensen at 2019-12-28T12:28:11Z
piuparts-master-backend: handle BrokenPipeError
No need to crash with a stacktrace when the slave is gone and the pipe
to it is broken. Just log this and exit 1.
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 433, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 423, in main
m = Master(sys.stdin, sys.stdout)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 154, in __init__
self._writeline("hello")
File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 111, in _writeline
self._output.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
dd4cd1a8 by Nis Martensen at 2019-12-28T12:29:01Z
fix piupartslib imports
During the port to python3 piupartslib was changed to no longer
auto-import some of its submodules. Import the required submodules
directly where now needed.
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1910, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1849, in main
section = Section(section_name, master_directory, doc_root, packagedb_cache=packagedb_cache)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 827, in __init__
self._load_package_database(section, master_directory)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 859, in _load_package_database
db = piupartslib.packagesdb.PackagesDB(prefix=sectiondir)
AttributeError: module 'piupartslib' has no attribute 'packagesdb'
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
cc30509e by Nis Martensen at 2019-12-28T12:30:06Z
piuparts-report: fix 'dict_keys has no attribute remove'
In python3 dict().keys() returns a dict_keys() view and not a list as in
python2. Replace the now unavailable remove() by a set subtraction.
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
section.generate_output(output_directory, section_names, problem_list, web_host)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
self.generate_html()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1574, in generate_html
self.cleanup_removed_packages(logs_by_dir)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1536, in cleanup_removed_packages
vdirs.remove("reserved")
AttributeError: 'dict_keys' object has no attribute 'remove'
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
484bc88e by Nis Martensen at 2019-12-28T12:30:06Z
piuparts-report: give hashlib the bytes it needs
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
section.generate_output(output_directory, section_names, problem_list, web_host)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
self.generate_html()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1580, in generate_html
self.print_by_dir(self._output_directory, logs_by_dir)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 943, in print_by_dir
desc_by_dir[vdir], vlist)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 930, in write_log_list_page
"logrows": "".join(lines),
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 890, in _write_template_html
write_template_html(filename, body, mapping, defer_if_unmodified=defer_if_unmodified, md5cache=self._md5cache)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 714, in write_template_html
content_md5 = hashlib.md5(htmlpage.safe_substitute(mapping)).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
08f9a354 by Nis Martensen at 2019-12-28T12:30:06Z
piuparts-report, piuparts: pickle needs binary file streams
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
section.generate_output(output_directory, section_names, problem_list, web_host)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
self.generate_html()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1600, in generate_html
pickle.dump(self._md5cache['new'], f)
TypeError: write() argument must be str, not bytes
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f70767fe by Nis Martensen at 2019-12-28T12:30:06Z
piuparts-report: handle EOFError with pickle.load()
Do not error out if the cache file is empty.
Fixes:
Traceback (most recent call last):
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
main()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
section.generate_output(output_directory, section_names, problem_list, web_host)
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
self.generate_html()
File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1563, in generate_html
self._md5cache['old'] = pickle.load(f)
EOFError: Ran out of input
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4 changed files:
- master-bin/detect_well_known_errors.py
- piuparts-master-backend.py
- piuparts-report.py
- piuparts.py
Changes:
=====================================
master-bin/detect_well_known_errors.py
=====================================
@@ -29,7 +29,7 @@ import argparse
import fcntl
from collections import deque
-import piupartslib
+import piupartslib.conf
from piupartslib.conf import MissingSection
from piupartslib.dwke import *
=====================================
piuparts-master-backend.py
=====================================
@@ -31,7 +31,8 @@ import fcntl
import time
import random
-import piupartslib
+import piupartslib.conf
+import piupartslib.packagesdb
from piupartslib.packagesdb import LogfileExists
from piupartslib.conf import MissingSection
@@ -426,6 +427,13 @@ def main():
pass
except URLError as e:
logging.error("ABORT: URLError: " + str(e.reason))
+ except BrokenPipeError:
+ logging.error("ABORT: BrokenPipeError")
+ logging.debug(timestamp() + " disconnected")
+ # https://docs.python.org/3/library/signal.html#note-on-sigpipe
+ devnull = os.open(os.devnull, os.O_WRONLY)
+ os.dup2(devnull, sys.stdout.fileno())
+ sys.exit(1)
logging.debug(timestamp() + " disconnected")
=====================================
piuparts-report.py
=====================================
@@ -47,7 +47,8 @@ try:
except:
pass
-import piupartslib
+import piupartslib.conf
+import piupartslib.packagesdb
from piupartslib.conf import MissingSection
from piupartslib.dwke import *
import piupartslib.pkgsummary as pkgsummary
@@ -710,7 +711,7 @@ def write_template_html(filename, body, mapping={}, defer_if_unmodified=False, m
"piuparts_version": "",
"time": "",
})
- content_md5 = hashlib.md5(htmlpage.safe_substitute(mapping)).hexdigest()
+ content_md5 = hashlib.md5(htmlpage.safe_substitute(mapping).encode()).hexdigest()
if md5cache is not None:
md5cache['new'][filename] = content_md5
@@ -1532,7 +1533,7 @@ class Section:
def cleanup_removed_packages(self, logs_by_dir):
vdirs = logs_by_dir.keys()
- vdirs.remove("reserved")
+ vdirs = vdirs - {"reserved"}
for vdir in vdirs:
for log in sorted(logs_by_dir[vdir]):
if log.endswith(".log"):
@@ -1558,9 +1559,9 @@ class Section:
def generate_html(self):
md5cachefile = os.path.join(self._output_directory, '.md5cache')
try:
- with open(md5cachefile, "r") as f:
+ with open(md5cachefile, "rb") as f:
self._md5cache['old'] = pickle.load(f)
- except IOError:
+ except (IOError, EOFError):
pass
logging.debug("Finding log files")
@@ -1595,7 +1596,7 @@ class Section:
logging.debug("Wrote %d out of %d html files, refreshed %d out of %d unmodified files" % ( \
self._md5cache['written'], len(self._md5cache['new']),
self._md5cache['refreshed'], self._md5cache['unmodified']))
- with open(md5cachefile, "w") as f:
+ with open(md5cachefile, "wb") as f:
pickle.dump(self._md5cache['new'], f)
logging.debug("Removing old log files")
=====================================
piuparts.py
=====================================
@@ -2460,14 +2460,14 @@ def install_upgrade_test(chroot, chroot_state, package_files, packages, old_pack
def save_meta_data(filename, chroot_state):
"""Save directory tree meta data into a file for fast access later."""
logging.debug("Saving chroot meta data to %s" % filename)
- with open(filename, "w") as f:
+ with open(filename, "wb") as f:
pickle.dump(chroot_state, f)
def load_meta_data(filename):
"""Load meta data saved by 'save_meta_data'."""
logging.debug("Loading chroot meta data from %s" % filename)
- with open(filename, "r") as f:
+ with open(filename, "rb") as f:
return pickle.load(f)
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/45242102a1c41f15e758b89a2ae87b7ee5ca4a5b...f70767fe582e7fedd394c5eea3af1c112e844e33
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/compare/45242102a1c41f15e758b89a2ae87b7ee5ca4a5b...f70767fe582e7fedd394c5eea3af1c112e844e33
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20191228/8f770e7d/attachment-0001.html>
More information about the Piuparts-devel
mailing list