[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] 2 commits: reproducible: Support arbitrary package filters when generating deb822 output.

[Holger Levsen]

Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
607abf0b by Chris Lamb at 2018-12-06T13:56:34Z
reproducible: Support arbitrary package filters when generating deb822 output.

We were previously encountering errors when filtering excessively-long package
lists. In particular:

  /tmp/jenkins-script-DMnm8bZM: line 25: /usr/bin/grep-dctrl: Argument list too long

This was because we were constructing an excessively-long command-line along
the lines of "(foo|bar|baz...)". This commit changes the implementation to
iterate over Packages file instead to support arbitrary filter sizes.

This is related (but orthogonal...) to MR !15.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
6a6abf09 by Chris Lamb at 2018-12-06T13:56:51Z
Add support for calculating a PureOS package set.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


2 changed files:

- bin/reproducible_create_meta_pkg_sets.sh
- bin/reproducible_pkgsets.csv


Changes:

=====================================
bin/reproducible_create_meta_pkg_sets.sh
=====================================
@@ -21,8 +21,21 @@ ABORT=false
 
 # helper functions
 packages_list_to_deb822() {
-	ALL_PKGS=$(cat $TMPFILE | cut -d ":" -f1 | sed "s#([^()]*)##g ; s#\[[^][]*\]##g ; s#,##g ; s# #\n#g"  |sort -u | tr '\n' '|')
-	grep-dctrl -F Package -e '^('"$ALL_PKGS"')$' $PACKAGES > $TMPFILE
+	script=$(cat <<'EOF'
+		use Dpkg::Control;
+		my $fd;
+		open($fd, '<', $ARGV[0]) or die $!;
+		my %packages = map { chomp; $_ => 1 } <STDIN>;
+		while (1) {
+			my $deb822 = Dpkg::Control->new();
+			last unless $deb822->parse($fd);
+			print "$deb822\n" if $packages{$deb822->{'Package'}};
+		}
+		close($fd);
+EOF
+	)
+	perl -e "$script" $PACKAGES < $TMPFILE > $TMPFILE2
+	mv $TMPFILE2 $TMPFILE
 }
 
 tails_build_manifest_to_deb822() {
@@ -390,6 +403,20 @@ update_pkg_set_specific() {
 				ABORT=true
 			fi
 			;;
+		pureos)	# pureos
+			URL="https://repo.puri.sm/pureos/dists/green/main/source/Sources.gz"
+			echo "Downloading $URL now."
+			curl $URL | gunzip > $TMPFILE
+			if [ -s $TMPFILE ] ; then
+				echo "parsing $TMPFILE now..."
+				convert_from_deb822_into_source_packages_only
+			else
+				rm $TMPFILE
+				MESSAGE="Warning: could not download PureOS's Sources file, skipping pkg set..."
+				irc_message debian-reproducible $MESSAGE
+				ABORT=true
+			fi
+			;;
 		subgraph_OS)
 			# installed by Subgraph OS
 			# one day we will get a proper data provider from Subgraph OS...


=====================================
bin/reproducible_pkgsets.csv
=====================================
@@ -27,6 +27,8 @@ Debian distribution,grml
 Debian distribution,grml_build-depends
 Debian distribution,tails
 Debian distribution,tails_build-depends
+Debian distribution,pureos
+Debian distribution,pureos_build-depends
 Debian distribution,subgraph_OS
 Debian distribution,subgraph_OS_build-depends
 maintenance team,maint_debian-accessibility



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/compare/709fd65506d56c20bc433896d63e3ff5cba06ea6...6a6abf0920d8dd9d5b339e02ca99d581b8b97fcc

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/compare/709fd65506d56c20bc433896d63e3ff5cba06ea6...6a6abf0920d8dd9d5b339e02ca99d581b8b97fcc
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20181206/6fcac2e9/attachment-0001.html>