[Piuparts-devel] Bug#767485: [PATCH] Support new scripts directive 'post_chroot_unpack'

[Michael Prokop]

The chroot gets updated via 'apt-get update;  apt-get -yf
dist-upgrade' after the tarball was extracted. This is necessary
to make sure the chroot is up2date before it gets snapshotted
(see #356678).  If the underlying chroot of the tarball was
(de)bootstrapped from a repository with a custom key (therefore
requiring either --keyring=... or (worse) --no-check-gpg to
debootstrap) it usually still doesn't include this key though.
Therefore 'apt-get update' will complain about it and the following
dist-upgrade actually does nothing:

| After purging files have been modified: /usr/share/doc/perl/changelog.Debian.gz owned by: perl-base

| 0m0.0s DEBUG: Unpacking /var/cache/pbuilder/base-wheezy-amd64.tgz into /tmp/tmpx5wqQb
| 0m0.0s DEBUG: Starting command: ['tar', '-C', '/tmp/tmpx5wqQb', '-zxf', '/var/cache/pbuilder/base-wheezy-amd64.tgz']
| 0m2.0s DEBUG: Command ok: ['tar', '-C', '/tmp/tmpx5wqQb', '-zxf', '/var/cache/pbuilder/base-wheezy-amd64.tgz']
| 0m2.0s DEBUG: Starting command: ['chroot', '/tmp/tmpx5wqQb', 'mount', '-t', 'proc', 'proc', '/proc']
| 0m2.0s DEBUG: Command ok: ['chroot', '/tmp/tmpx5wqQb', 'mount', '-t', 'proc', 'proc', '/proc']
| 0m2.0s DEBUG: sources.list:
|   deb http://debian.example.com/debian wheezy main
|   deb http://debian.example.com/debian wheezy contrib
|   deb http://debian.example.com/debian wheezy non-free
| 0m2.0s DEBUG: Created policy-rc.d and chmodded it.
| 0m2.0s DEBUG: Starting command: ['chroot', '/tmp/tmpx5wqQb', 'apt-get', 'update']
| 0m7.3s DUMP:
|   Get:1 http://debian.example.com wheezy Release.gpg [198 B]
|   Get:2 http://debian.example.com wheezy Release [5918 B]
|   Err http://debian.example.com wheezy Release
|
|   Fetched 6116 B in 0s (245 kB/s)
|   Reading package lists...
|   W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://debian.example.com wheezy Release: The fol
| lowing signatures couldn't be verified because the public key is not available: NO_PUBKEY 123456789424242F
|
|   W: Failed to fetch http://debian.example.com/debian/dists/wheezy/Release
|
|   W: Some index files failed to download. They have been ignored, or old ones used instead.
| 0m7.3s DEBUG: Command ok: ['chroot', '/tmp/tmpx5wqQb', 'apt-get', 'update']
| 0m7.3s DEBUG: Starting command: ['chroot', '/tmp/tmpx5wqQb', 'apt-get', '-yf', 'dist-upgrade']
| 0m7.7s DUMP:
|   Reading package lists...
|   Building dependency tree...
|   0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
| 0m7.7s DEBUG: Command ok: ['chroot', '/tmp/tmpx5wqQb', 'apt-get', '-yf', 'dist-upgrade']
| 0m7.7s DEBUG: Copying scriptsdir /tmp/piuparts-tests/scripts/ to /tmp/tmpx5wqQb/tmp/scripts/

As a result you might have different checksums reported in the
piupart run. For example if your chroot tarball uses perl-base of
Debian 7.6 but Debian 7.7 ships an updated perl-base package
nowadays you'll get:

| After purging files have been modified: /usr/share/doc/perl/changelog.Debian.gz owned by: perl-base

To make sure the dist-upgrade step doesn't fail we need a way to
install a custom key for apt usage *after* the unpacking of the
chroot but *before* the actual dist-upgrade. The is what the
post_chroot_unpack stage provides.

Thanks: Sipwise GmbH for sponsoring my development time
---
 README.txt                                                    |  3 +++
 .../scripts-unused-examples/post_chroot_unpack_key_setup.sh   |  9 +++++++++
 piuparts.py                                                   | 11 ++++++++---
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup.sh

diff --git a/README.txt b/README.txt
index 37faf2e..9c7e4d8 100644
--- a/README.txt
+++ b/README.txt
@@ -213,6 +213,9 @@ PIUPARTS_DISTRIBUTION.
 
 The following prefixes for scripts are recognized:
 
+'post_chroot_unpack' - after the chroot has been unpacked/debootrapped.
+Before the chroot gets updated/dist-upgraded initially.
+
 'post_setup_' - after the *setup* of the chroot is finished.
 Before metadata of the chroot is recorded for later comparison.
 
diff --git a/custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup.sh b/custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup.sh
new file mode 100644
index 0000000..06f1f96
--- /dev/null
+++ b/custom-scripts/scripts-unused-examples/post_chroot_unpack_key_setup.sh
@@ -0,0 +1,9 @@
+#!/bin/sh
+
+# we rely on wget being available, make sure to use "--include=wget" in your deboostrap cmdline
+echo "Setting up https://example.com/internal_key.asc for apt-get usage."
+wget -O - 'https://example.com/internal_key.asc' | apt-key add -
+
+echo "Running apt-get update to have a verified and working Debian repository available."
+apt-get update
+
diff --git a/piuparts.py b/piuparts.py
index de349ff..5053044 100644
--- a/piuparts.py
+++ b/piuparts.py
@@ -706,9 +706,6 @@ class Chroot:
             self.mount_proc()
             self.mount_selinux()
         self.configure_chroot()
-        if settings.basetgz or settings.schroot:
-            self.run(["apt-get", "-yf", "dist-upgrade"])
-        self.minimize()
 
         # Copy scripts dirs into the chroot, merging all dirs together,
         # later files overwriting earlier ones.
@@ -724,6 +721,14 @@ class Chroot:
                             and os.path.isfile(os.path.join(sdir, sfile)):
                         shutil.copy(os.path.join(sdir, sfile), dest)
 
+        # Run custom scripts after chroot has been unpacked/debootstrapped
+        # Useful for adjusting apt configuration e.g. for internal mirror usage
+        self.run_scripts("post_chroot_unpack")
+
+        if settings.basetgz or settings.schroot:
+            self.run(["apt-get", "-yf", "dist-upgrade"])
+        self.minimize()
+
         # Run custom scripts after creating the chroot.
         self.run_scripts("post_setup")
 
-- 
2.1.1