[Piuparts-devel] [Git][debian/piuparts][develop] 2 commits: start preparing 1.1.3
Holger Levsen
gitlab at salsa.debian.org
Sun Feb 21 09:27:24 GMT 2021
Holger Levsen pushed to branch develop at Debian / piuparts
Commits:
df19a512 by Holger Levsen at 2021-02-21T10:27:00+01:00
start preparing 1.1.3
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
dbb8af43 by Michael Prokop at 2021-02-21T10:27:00+01:00
p: new option --update-retries <num> to re-run apt-get update
If the Packages file gets rewritten during `apt-get update`
execution, then piupart might fail with a hashsum mismatch
error, like:
| DEBUG: Starting command: ['chroot', '/var/cache/pbuilder/tmp.3l19K3MgeX_piuparts_tmp/tmp1m34xz', 'apt-get', 'update']
| DUMP:
| [...]
| Err:5 https://deb.example.com/autobuild gerrit_mika_foobar/main amd64 Packages
| File has unexpected size (1338 != 1346). Mirror sync in progress? [IP: 10.10.42.42 443]
| Hashes of expected file:
| - Filesize:1346 [weak]
| - SHA256:5af5db54c0d80e595a247a2070a1f8381468cbdacea1b62ead353d5bea074697
| - SHA1:61962e3dcd01055a4baf3ab8e318c17832b6b0fd [weak]
| - MD5Sum:92f602acb8051cfd90b3e33900a29932 [weak]
| Release file created at: Tue, 16 Feb 2021 01:13:26 +0000
| [...]
| Reading package lists...
| E: Failed to fetch https://deb.example.com/autobuild/dists/gerrit_mika_foobar/main/binary-amd64/Packages.gz File has unexpected size (1338 != 1346). Mirror sync in progress? [IP: 10.10.42.42 443]
| Hashes of expected file:
| - Filesize:1346 [weak]
| - SHA256:5af5db54c0d80e595a247a2070a1f8381468cbdacea1b62ead353d5bea074697
| - SHA1:61962e3dcd01055a4baf3ab8e318c17832b6b0fd [weak]
| - MD5Sum:92f602acb8051cfd90b3e33900a29932 [weak]
| Release file created at: Tue, 16 Feb 2021 01:13:26 +0000
| E: Some index files failed to download. They have been ignored, or old ones used instead.
| ERROR: Command failed (status=100): ['chroot', '/var/cache/pbuilder/tmp.3l19K3MgeX_piuparts_tmp/tmp1m34xz', 'apt-get', 'update']
By giving the user an option to retry the update procedure (up to
<num> times, as specified on the command line), we increase
chances for a successful build. This is useful especially in busy
continuous integration/delivery environments, lacking by-hash
support (see
https://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html),
for example all repositories using reprepro, which doesn't support
by-hash yet (see https://bugs.debian.org/820660).
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3 changed files:
- debian/changelog
- docs/piuparts/piuparts.1.txt
- piuparts.py
Changes:
=====================================
debian/changelog
=====================================
@@ -1,3 +1,10 @@
+piuparts (1.1.3) UNRELEASED; urgency=medium
+
+ * Team upload.
+ * [...]
+
+ -- Holger Levsen <holger at debian.org> Sun, 21 Feb 2021 10:26:15 +0100
+
piuparts (1.1.2) unstable; urgency=medium
* Team upload.
=====================================
docs/piuparts/piuparts.1.txt
=====================================
@@ -304,6 +304,10 @@ Options must come before the other command line arguments.
Use directory as the place where temporary files and directories are created. The default is the environment variable *TMPDIR*, or */tmp* if not set.
Note: the temporary directory must *not* be mounted with the _nodev_ or _nosuid_ mount option.
+*-*-update-retries*='num-retries'::
+ Rerun 'apt-get update` up to "num-retries" times.
+ Useful to work around temporary network failures and hashsum mismatch errors.
+
*-*-upgrade-before-dist-upgrade*::
Perform two-stage upgrades: 'apt-get upgrade && apt-get dist-upgrade'.
=====================================
piuparts.py
=====================================
@@ -182,6 +182,7 @@ class Settings:
self.scriptsdirs = []
self.bindmounts = []
self.allow_database = False
+ self.update_retries = None
# chroot setup
self.arch = None
self.basetgz = None
@@ -820,7 +821,8 @@ class Chroot:
# Useful for adjusting apt configuration e.g. for internal mirror usage
self.run_scripts("post_chroot_unpack")
- self.run(["apt-get", "update"])
+ self.aptupdate_run()
+
if settings.basetgz or settings.docker_image or settings.schroot or settings.existing_chroot:
self.run(["apt-get", "-yf", "dist-upgrade"])
self.minimize()
@@ -1005,6 +1007,28 @@ class Chroot:
"\n".join(lines) + "\n")
logging.debug("sources.list:\n" + indent_string("\n".join(lines)))
+ def aptupdate_run(self):
+ """Resynchronize the package index files.
+ If executed under --update-retries <num>, retry
+ its execution up to <num> times, useful e.g.
+ on temporary network failures or hashsum mismatch
+ errors."""
+ if not settings.update_retries:
+ self.run(["apt-get", "update"])
+ return
+
+ count = 0
+ for count, run in enumerate(range(settings.update_retries), 1):
+ (status, output) = self.run(["apt-get", "update"],
+ ignore_errors=True)
+ if status == 0:
+ break
+ else:
+ logging.info("apt-get update failed to execute " +
+ "(run #%d, re-trying up to %d times)" % (count, settings.update_retries))
+
+ return status
+
def enable_testdebs_repo(self, update=True):
if settings.testdebs_repo:
if settings.testdebs_repo.startswith("deb"):
@@ -1016,7 +1040,7 @@ class Chroot:
logging.debug("enabling testdebs repository '%s'" % debline)
create_file(self.relative("etc/apt/sources.list.d/piuparts-testdebs-repo.list"), debline + "\n")
if update:
- self.run(["apt-get", "update"])
+ self.aptupdate_run()
def disable_testdebs_repo(self):
if settings.testdebs_repo:
@@ -1171,7 +1195,7 @@ class Chroot:
self.create_apt_sources(distro)
# Run custom scripts before upgrade
self.run_scripts("pre_distupgrade")
- self.run(["apt-get", "update"])
+ self.aptupdate_run()
if apt_get_upgrade:
self.run(["apt-get", "-y", "upgrade"])
self.run(["apt-get", "-yf", "dist-upgrade"])
@@ -2939,6 +2963,11 @@ def parse_command_line():
help="Use DIR for temporary storage. Default is " +
"$TMPDIR or /tmp.")
+ parser.add_option("--update-retries", metavar="UPDATE_RETRIES", type="int",
+ help="Rerun `apt-get update` up to UPDATE_RETRIES "
+ "times. Useful to work around temporary network failures "
+ "and hashsum mismatch errors.")
+
parser.add_option("-v", "--verbose",
action="store_true", default=False,
help="No meaning anymore.")
@@ -3033,6 +3062,7 @@ def parse_command_line():
settings.scriptsdirs = opts.scriptsdir
settings.bindmounts += opts.bindmount
settings.allow_database = opts.allow_database
+ settings.update_retries = opts.update_retries
# chroot setup
settings.arch = opts.arch
settings.basetgz = opts.basetgz
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/31862192384ea68ad5d490c6fbf99f6f78f11314...dbb8af433b738db7ddbf36f5e80f877fc6f1aadc
--
View it on GitLab: https://salsa.debian.org/debian/piuparts/-/compare/31862192384ea68ad5d490c6fbf99f6f78f11314...dbb8af433b738db7ddbf36f5e80f877fc6f1aadc
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/piuparts-devel/attachments/20210221/cd298090/attachment-0001.htm>
More information about the Piuparts-devel
mailing list