[Qa-jenkins-dev] Bug#864686: reproducible: reduce number of armhf builds per node
Vagrant Cascadian
vagrant at debian.org
Mon Jun 12 20:33:00 UTC 2017
Package: jenkins.debian.org
Severity: normal
Please consider merging the following patch, available in the
"armhf-reduced-builds" branch:
https://anonscm.debian.org/cgit/users/vagrant/jenkins.debian.net.git/
This may help reduce the impacts of Out-of-Memory errors on nodes when
building many packages simultaneously, though will likely result in a
slower build network...
live well,
vagrant
commit 314becd4473586e1fe47cb05477276aef9630a87
Author: Vagrant Cascadian <vagrant at debian.org>
Date: Mon Jun 12 13:08:44 2017 -0700
reproducible: Reduce number of armhf build jobs to 2 for 1gb of ram, 4
for 2gb of ram, and 6 for 4gb of ram, for a total of 51 jobs. Remove
disabled jobs for odc2a.
diff --git a/README b/README
index 02854897..ded67155 100644
--- a/README
+++ b/README
@@ -135,7 +135,7 @@ Installation tests inside chroot environments.
* The (current) purpose of https://tests.reproducible-builds.org is to show the potential of reproducible builds for
Debian - and six other projects currently. This is research, showing what could (and should) be done... check
https://wiki.debian.org/ReproducibleBuilds for the real status of the project for Debian!
-* For Debian, three suites, 'testing', 'unstable' and 'experimental', are tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 76 for 'armhf', which are each constantly testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers] which in turn
launch the actual link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins service.)
+* For Debian, three suites, 'testing', 'unstable' and 'experimental', are tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 51 for 'armhf', which are each constantly testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers] which in turn
launch the actual link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins service.)
* These builds on remote nodes run on very different hardware:
** for 'amd64' we are using four virtual machines, profitbricks-build(1+5+11+15)-amd64, which have 15 or 16 cores
and 48gb ram each. These nodes are sponsored by
link:https://jenkins.debian.net/userContent/thanks.html[Profitbricks].
diff --git a/bin/reproducible_build_service.sh b/bin/reproducible_build_service.sh
index 79bf24ee..500f15a6 100755
--- a/bin/reproducible_build_service.sh
+++ b/bin/reproducible_build_service.sh
@@ -116,93 +116,68 @@ choose_nodes() {
arm64_31) NODE1=codethink-sled14-arm64 NODE2=codethink-sled11-arm64 ;;
arm64_32) NODE1=codethink-sled13-arm64 NODE2=codethink-sled14-arm64 ;;
# to choose new armhf jobs:
- # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15 cbxi4pro0 ff2a ff2b ff64a jtk1a odc2a odxu4
odxu4b odxu4c odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a jtx1a ; do echo "$i: " ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
- # 8 jobs for quad-cores with 4 gb ram
- # 6 jobs for octo-cores with 2 gb ram
- # 6 jobs for hexa-cores with 2 gb ram
- # 6 jobs for quad-cores with 2 gb ram
- # 3 jobs for dual-cores with 1 gb ram
- # 3 jobs for quad-cores with 1 gb ram
+ # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15 cbxi4pro0 ff2a ff2b ff64a jtk1a odxu4 odxu4b odxu4c
odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a jtx1a ; do echo "$i: " ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
+ # 6 jobs for quad-cores with 4 gb ram
+ # 4 jobs for octo-cores with 2 gb ram
+ # 4 jobs for hexa-cores with 2 gb ram
+ # 4 jobs for quad-cores with 2 gb ram
+ # 4 jobs for dual-cores with 2 gb ram
+ # 2 jobs for dual-cores with 1 gb ram
+ # 2 jobs for quad-cores with 1 gb ram
#
# Don't forget to update README with the number of builders…!
#
- armhf_1) NODE1=bbx15-armhf-rb NODE2=odxu4-armhf-rb ;;
- armhf_2) NODE1=wbq0-armhf-rb NODE2=p64c-armhf-rb ;;
- armhf_3) NODE1=hb0-armhf-rb NODE2=p64b-armhf-rb ;;
- armhf_4) NODE1=ff4a-armhf-rb NODE2=wbq0-armhf-rb ;;
- armhf_5) NODE1=cbxi4pro0-armhf-rb NODE2=bpi0-armhf-rb ;;
- armhf_6) NODE1=ff4a-armhf-rb NODE2=cbxi4pro0-armhf-rb ;;
- armhf_7) NODE1=wbq0-armhf-rb NODE2=odxu4-armhf-rb ;;
- armhf_8) NODE1=hb0-armhf-rb NODE2=wbq0-armhf-rb ;;
- armhf_9) NODE1=ff4a-armhf-rb NODE2=bpi0-armhf-rb ;;
- armhf_10) NODE1=odxu4-armhf-rb NODE2=rpi2b-armhf-rb ;;
- armhf_11) NODE1=odxu4-armhf-rb NODE2=wbd0-armhf-rb ;;
- armhf_12) NODE1=wbd0-armhf-rb NODE2=cbxi4pro0-armhf-rb ;;
- armhf_13) NODE1=cbxi4pro0-armhf-rb NODE2=rpi2b-armhf-rb ;;
- armhf_14) NODE1=cbxi4a-armhf-rb NODE2=odxu4b-armhf-rb ;;
- armhf_15) NODE1=rpi2b-armhf-rb NODE2=odxu4c-armhf-rb ;;
- armhf_16) NODE1=odxu4b-armhf-rb NODE2=wbd0-armhf-rb ;;
- armhf_17) NODE1=odxu4c-armhf-rb NODE2=hb0-armhf-rb ;;
- armhf_18) NODE1=odxu4b-armhf-rb NODE2=odu3a-armhf-rb ;;
- armhf_19) NODE1=odxu4c-armhf-rb NODE2=opi2c-armhf-rb ;;
- armhf_20) NODE1=opi2b-armhf-rb NODE2=odxu4b-armhf-rb ;;
- armhf_21) NODE1=ff2a-armhf-rb NODE2=odxu4c-armhf-rb ;;
- armhf_22) NODE1=ff2a-armhf-rb NODE2=rpi2c-armhf-rb ;;
- armhf_23) NODE1=rpi2c-armhf-rb NODE2=odxu4b-armhf-rb ;;
- armhf_24) NODE1=rpi2c-armhf-rb NODE2=odxu4c-armhf-rb ;;
- armhf_25) NODE1=odxu4b-armhf-rb NODE2=ff2b-armhf-rb ;;
- armhf_26) NODE1=jtk1a-armhf-rb NODE2=ff2a-armhf-rb ;;
- armhf_27) NODE1=odxu4c-armhf-rb NODE2=cbxi4a-armhf-rb ;;
- armhf_28) NODE1=jtk1a-armhf-rb NODE2=ff2b-armhf-rb ;;
- armhf_29) NODE1=ff2b-armhf-rb NODE2=jtk1a-armhf-rb ;;
- armhf_30) NODE1=ff2b-armhf-rb NODE2=cbxi4b-armhf-rb ;;
- armhf_31) NODE1=ff2b-armhf-rb NODE2=opi2b-armhf-rb ;;
- armhf_32) NODE1=jtk1a-armhf-rb NODE2=cbxi4b-armhf-rb ;;
- armhf_33) NODE1=ff2a-armhf-rb NODE2=opi2b-armhf-rb ;;
- armhf_34) NODE1=cbxi4a-armhf-rb NODE2=opi2b-armhf-rb ;;
- armhf_35) NODE1=opi2a-armhf-rb NODE2=ff2b-armhf-rb ;;
- armhf_36) NODE1=opi2a-armhf-rb NODE2=cbxi4a-armhf-rb ;;
- armhf_37) NODE1=opi2a-armhf-rb NODE2=wbq0-armhf-rb ;;
- armhf_38) NODE1=cbxi4b-armhf-rb NODE2=jtk1a-armhf-rb ;;
- armhf_39) NODE1=cbxi4b-armhf-rb NODE2=cbxi4a-armhf-rb ;;
- armhf_40) NODE1=opi2b-armhf-rb NODE2=cbxi4b-armhf-rb ;;
- armhf_41) NODE1=opi2b-armhf-rb NODE2=cbxi4b-armhf-rb ;;
- armhf_42) NODE1=cbxi4b-armhf-rb NODE2=cbxi4a-armhf-rb ;;
- armhf_43) NODE1=cbxi4a-armhf-rb NODE2=opi2c-armhf-rb ;;
- armhf_44) NODE1=bbx15-armhf-rb NODE2=ff4a-armhf-rb ;;
- armhf_45) NODE1=ff4a-armhf-rb NODE2=p64b-armhf-rb ;;
- armhf_46) NODE1=wbq0-armhf-rb NODE2=bbx15-armhf-rb ;;
- armhf_47) NODE1=cbxi4pro0-armhf-rb NODE2=bbx15-armhf-rb ;;
- armhf_48) NODE1=bbx15-armhf-rb NODE2=p64c-armhf-rb ;;
- armhf_49) NODE1=bpi0-armhf-rb NODE2=ff4a-armhf-rb ;;
- armhf_50) NODE1=odxu4-armhf-rb NODE2=odu3a-armhf-rb ;;
- armhf_51) NODE1=odu3a-armhf-rb NODE2=cb3a-armhf-rb ;;
- armhf_52) NODE1=opi2c-armhf-rb NODE2=cb3a-armhf-rb ;;
- armhf_53) NODE1=cb3a-armhf-rb NODE2=ff4a-armhf-rb ;;
- armhf_54) NODE1=odu3a-armhf-rb NODE2=opi2c-armhf-rb ;;
- armhf_55) NODE1=opi2c-armhf-rb NODE2=odu3a-armhf-rb ;;
- armhf_56) NODE1=odu3a-armhf-rb NODE2=ff2a-armhf-rb ;;
- armhf_57) NODE1=opi2c-armhf-rb NODE2=ff64a-armhf-rb ;;
- armhf_58) NODE1=cbxi4a-armhf-rb NODE2=ff64a-armhf-rb ;;
- armhf_59) NODE1=jtk1a-armhf-rb NODE2=ff64a-armhf-rb ;;
- armhf_60) NODE1=cbxi4b-armhf-rb NODE2=jtx1a-armhf-rb ;;
- armhf_61) NODE1=p64c-armhf-rb NODE2=jtx1a-armhf-rb ;;
- armhf_62) NODE1=p64b-armhf-rb NODE2=jtx1a-armhf-rb ;;
- armhf_63) NODE1=p64b-armhf-rb NODE2=jtx1a-armhf-rb ;;
-# armhf_64) NODE1=p64c-armhf-rb NODE2=odc2a-armhf-rb ;;
-# armhf_65) NODE1=p64b-armhf-rb NODE2=odc2a-armhf-rb ;;
-# armhf_66) NODE1=p64c-armhf-rb NODE2=odc2a-armhf-rb ;;
- armhf_67) NODE1=ff64a-armhf-rb NODE2=ff2a-armhf-rb ;;
- armhf_68) NODE1=ff64a-armhf-rb NODE2=p64b-armhf-rb ;;
- armhf_69) NODE1=ff64a-armhf-rb NODE2=p64c-armhf-rb ;;
- armhf_70) NODE1=jtx1a-armhf-rb NODE2=opi2a-armhf-rb ;;
- armhf_71) NODE1=jtx1a-armhf-rb NODE2=opi2a-armhf-rb ;;
- armhf_72) NODE1=jtx1a-armhf-rb NODE2=opi2a-armhf-rb ;;
- armhf_73) NODE1=jtx1a-armhf-rb NODE2=ff4a-armhf-rb ;;
-# armhf_74) NODE1=odc2a-armhf-rb NODE2=bbx15-armhf-rb ;;
-# armhf_75) NODE1=odc2a-armhf-rb NODE2=cbxi4pro0-armhf-rb ;;
-# armhf_76) NODE1=odc2a-armhf-rb NODE2=odxu4-armhf-rb ;;
-
+ armhf_1) NODE1=bbx15-armhf-rb NODE2=jtx1a-armhf-rb ;;
+ armhf_2) NODE1=bbx15-armhf-rb NODE2=rpi2b-armhf-rb ;;
+ armhf_3) NODE1=cb3a-armhf-rb NODE2=bbx15-armhf-rb ;;
+ armhf_4) NODE1=cb3a-armhf-rb NODE2=bbx15-armhf-rb ;;
+ armhf_5) NODE1=hb0-armhf-rb NODE2=cb3a-armhf-rb ;;
+ armhf_6) NODE1=rpi2b-armhf-rb NODE2=cb3a-armhf-rb ;;
+ armhf_7) NODE1=rpi2c-armhf-rb NODE2=wbd0-armhf-rb ;;
+ armhf_8) NODE1=wbd0-armhf-rb NODE2=rpi2c-armhf-rb ;;
+ armhf_9) NODE1=bpi0-armhf-rb NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_10) NODE1=cbxi4pro0-armhf-rb NODE2=bpi0-armhf-rb ;;
+ armhf_11) NODE1=cbxi4pro0-armhf-rb NODE2=ff2a-armhf-rb ;;
+ armhf_12) NODE1=ff2a-armhf-rb NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_13) NODE1=ff2a-armhf-rb NODE2=ff2b-armhf-rb ;;
+ armhf_14) NODE1=ff2b-armhf-rb NODE2=ff2a-armhf-rb ;;
+ armhf_15) NODE1=ff2b-armhf-rb NODE2=ff64a-armhf-rb ;;
+ armhf_16) NODE1=ff64a-armhf-rb NODE2=ff2b-armhf-rb ;;
+ armhf_17) NODE1=ff64a-armhf-rb NODE2=jtk1a-armhf-rb ;;
+ armhf_18) NODE1=jtk1a-armhf-rb NODE2=ff64a-armhf-rb ;;
+ armhf_19) NODE1=jtk1a-armhf-rb NODE2=odxu4-armhf-rb ;;
+ armhf_20) NODE1=odxu4-armhf-rb NODE2=jtk1a-armhf-rb ;;
+ armhf_21) NODE1=odxu4-armhf-rb NODE2=odxu4b-armhf-rb ;;
+ armhf_22) NODE1=odxu4b-armhf-rb NODE2=odxu4-armhf-rb ;;
+ armhf_23) NODE1=odxu4b-armhf-rb NODE2=odxu4c-armhf-rb ;;
+ armhf_24) NODE1=odxu4c-armhf-rb NODE2=odxu4b-armhf-rb ;;
+ armhf_25) NODE1=odxu4c-armhf-rb NODE2=odu3a-armhf-rb ;;
+ armhf_26) NODE1=odu3a-armhf-rb NODE2=odxu4c-armhf-rb ;;
+ armhf_27) NODE1=odu3a-armhf-rb NODE2=opi2a-armhf-rb ;;
+ armhf_28) NODE1=opi2a-armhf-rb NODE2=odu3a-armhf-rb ;;
+ armhf_29) NODE1=opi2a-armhf-rb NODE2=opi2b-armhf-rb ;;
+ armhf_30) NODE1=opi2b-armhf-rb NODE2=opi2a-armhf-rb ;;
+ armhf_31) NODE1=opi2b-armhf-rb NODE2=opi2c-armhf-rb ;;
+ armhf_32) NODE1=opi2c-armhf-rb NODE2=opi2b-armhf-rb ;;
+ armhf_33) NODE1=opi2c-armhf-rb NODE2=p64b-armhf-rb ;;
+ armhf_34) NODE1=p64b-armhf-rb NODE2=opi2c-armhf-rb ;;
+ armhf_35) NODE1=p64b-armhf-rb NODE2=p64c-armhf-rb ;;
+ armhf_36) NODE1=p64c-armhf-rb NODE2=p64b-armhf-rb ;;
+ armhf_37) NODE1=p64c-armhf-rb NODE2=wbq0-armhf-rb ;;
+ armhf_38) NODE1=wbq0-armhf-rb NODE2=p64c-armhf-rb ;;
+ armhf_39) NODE1=wbq0-armhf-rb NODE2=cbxi4a-armhf-rb ;;
+ armhf_40) NODE1=cbxi4a-armhf-rb NODE2=wbq0-armhf-rb ;;
+ armhf_41) NODE1=cbxi4a-armhf-rb NODE2=cbxi4b-armhf-rb ;;
+ armhf_42) NODE1=cbxi4a-armhf-rb NODE2=cbxi4b-armhf-rb ;;
+ armhf_43) NODE1=cbxi4b-armhf-rb NODE2=cbxi4a-armhf-rb ;;
+ armhf_44) NODE1=cbxi4b-armhf-rb NODE2=cbxi4a-armhf-rb ;;
+ armhf_45) NODE1=cbxi4b-armhf-rb NODE2=ff4a-armhf-rb ;;
+ armhf_46) NODE1=ff4a-armhf-rb NODE2=cbxi4b-armhf-rb ;;
+ armhf_47) NODE1=ff4a-armhf-rb NODE2=jtx1a-armhf-rb ;;
+ armhf_48) NODE1=ff4a-armhf-rb NODE2=jtx1a-armhf-rb ;;
+ armhf_49) NODE1=jtx1a-armhf-rb NODE2=ff4a-armhf-rb ;;
+ armhf_50) NODE1=jtx1a-armhf-rb NODE2=ff4a-armhf-rb ;;
+ armhf_51) NODE1=jtx1a-armhf-rb NODE2=hb0-armhf-rb ;;
*) NODE1=undefined
;;
esac
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/qa-jenkins-dev/attachments/20170612/505ba15a/attachment.sig>
More information about the Qa-jenkins-dev
mailing list