Making schleuder build reproducibly
Georg Faerber
georg at riseup.net
Mon Oct 30 17:21:39 UTC 2017
Hi all,
On 17-06-15 21:19:12, Georg Faerber wrote:
> I really would like to make the build of schleuder, a gpg enabled
> mailing list, reproducible. However, I'm a bit lost on my own, that's
> why I'm searching for input with this mail:
tl;dr: Due to recent changes and fixes upstream, schleuder now does
build reproducible. For details, please read on.
--
Well, it took me quite a while and some time, but it was worth it, it
seems [1]. Here's a short summary of the changes:
- Sometimes, gpg-agent gets stuck, killing it before and after running
each test helps [2].
- Some tests expected specific dates (like 2016-12-06) to be returned by
gpg. However, these dates are non deterministic, they depend on locale
settings.
Quoting a comment in the corresponding upstream merge request:
"Oh my...
I wondered before why gpg doesn't show timezone-information anywhere
but assumed that they just don't store it or hide it because it
doesn't add actual information. TIL gpg knows about timezones and also
converts dates but doesn't tell about it...
So actually in order to compare a key-date I have to remember (or
guess) which timezone it was created in, and calulate the
time-difference to my local timezone, in order to guess if a
off-by-one-day-mismatch is maybe not an actual mismatch? I'm shaking
my head."
Therefore, we've relaxed the expected output, to not match specific
dates, but a specific format, like YYYY-MM-DD. [3]
- Some tests check if getting and / or refreshing keys via the keyserver
mechanism works. For this to work, a local keyserver is started, which
mocks the behaviour of the keyservers on the Internets. In the past,
we just started the keyserver before running the test and assumed it
to work. Sometimes this failed, because the keyserver took longer to
start, leading to failed tests, because it was in fact unreachable.
This was solved via introducing a check for the keyserver to be up,
before running the test. [4]
- I've never encountered the following during the "official" tests run
by jenkins, but I've run into this three times while using reprotest.
I'm not sure if this is a bug, race or a feature, but during the
second build, the build dir was read only.
In the past, the database in which schleuder stores list information
was just below the build dir, read only in these cases, leading to a
hanging test suite.
This was solved via introducing erb code parsing in the schleuder
config [5], and using this feature via env vars in Debian to move the
database dir below /tmp/ [6].
@dkg: It seems, there is still a bug / race in dirmngr, which leads to
errors like "can't connect to '127.0.0.1': no IP address for host" and
in turn "marking host '127.0.0.1' as dead". See the attached debug log for
details, the log was taken on October 1st with dirmrngr out of unstable.
I'm happy to debug this further, if needed.
That's all for now!
Thanks for this initiative and the work all of you're putting into it,
highly appreciated!
Cheers,
Georg
[1] https://tests.reproducible-builds.org/debian/history/schleuder.html
[2] https://0xacab.org/schleuder/schleuder/commit/8ecd8a4cdc76bc8fcdee54397fd0928d338146c1
[3] https://0xacab.org/schleuder/schleuder/commit/e52b6851b30d2578d2eb6c451425549a630d2ba6
[4] https://0xacab.org/schleuder/schleuder/commit/5acd2df470e420ef54ac1bf1193638ec8a23bbba
[5] https://0xacab.org/schleuder/schleuder/commit/d188224ce60f8850001825dc94a7d1fa470342ae
[6] https://anonscm.debian.org/cgit/pkg-ruby-extras/schleuder.git/commit/?id=f1cafc243766ecd0ad3ec84d8b1adf26ef2e0c66
-------------- next part --------------
2017-10-01 06:16:42 dirmngr[32131] listening on socket '/tmp/schleuder-test/example.org/list62/S.dirmngr'
2017-10-01 06:16:42 dirmngr[32132.0] permanently loaded certificates: 149
2017-10-01 06:16:42 dirmngr[32132.0] runtime cached certificates: 0
2017-10-01 06:16:42 dirmngr[32132.0] trusted certificates: 149 (148,0,0,1)
2017-10-01 06:16:42 dirmngr[32132.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt': No such file or directory
2017-10-01 06:16:42 dirmngr[32132.0] creating directory '/tmp/schleuder-test/example.org/list62/crls.d'
2017-10-01 06:16:42 dirmngr[32132.0] new cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt' created
2017-10-01 06:16:42 dirmngr[32132.6] handler for fd 6 started
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list62
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list62/dirmngr.conf
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service
2017-10-01 06:16:42 dirmngr[32132.6] connection from process 32128 (0:0)
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- GETINFO version
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> D 2.2.1
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> S KEYSERVER hkp://127.0.0.1:9999
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK
2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- KS_GET -- =admin at example.org
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: libdns initialized (tor mode)
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: libdns initialized (tor mode)
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_addr(): Success
2017-10-01 06:16:43 dirmngr[32132.6] resolve_dns_addr for '127.0.0.1': 'localhost' [already known]
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_addr(): Success
2017-10-01 06:16:43 dirmngr[32132.6] number of system provided CAs: 148
2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999
2017-10-01 06:16:43 dirmngr[32132.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:1819:socket_new: object 0x00007f9f403a6b20 for fd 7 created
2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:request:
2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> GET /pks/lookup?op=get&options=mr&search=admin at example.org&exact=on HTTP/1.0\r\n
2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> Host: localhost:9999\r\n
2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:request-header:
2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> \r\n
2017-10-01 06:16:43 dirmngr[32132.6] DBG: http.c:response:
2017-10-01 06:16:43 dirmngr[32132.6] DBG: >> HTTP/1.1 200 OK\r\n
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Content-Type: text/html;charset=utf-8'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Content-Length: 1762'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-XSS-Protection: 1; mode=block'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-Content-Type-Options: nosniff'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'X-Frame-Options: SAMEORIGIN'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Connection: close'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: 'Server: thin'
2017-10-01 06:16:43 dirmngr[32132.6] http.c:RESP: ''
2017-10-01 06:16:43 dirmngr[32132.6] DBG: chan_6 -> S SOURCE http://127.0.0.1:9999
2017-10-01 06:16:43 dirmngr[32132.6] DBG: (1762 bytes sent via D lines not shown)
2017-10-01 06:16:43 dirmngr[32132.6] DBG: chan_6 -> OK
2017-10-01 06:16:44 dirmngr[32132.6] DBG: chan_6 <- BYE
2017-10-01 06:16:44 dirmngr[32132.6] DBG: chan_6 -> OK closing connection
2017-10-01 06:16:44 dirmngr[32132.6] handler for fd 6 terminated
2017-10-01 06:16:45 dirmngr[32132.0] socket file has been removed - shutting down
2017-10-01 06:16:45 dirmngr[32132.0] dirmngr (GnuPG) 2.2.1 stopped
2017-10-01 06:16:46 dirmngr[32169] listening on socket '/tmp/schleuder-test/example.org/list63/S.dirmngr'
2017-10-01 06:16:46 dirmngr[32170.0] permanently loaded certificates: 149
2017-10-01 06:16:46 dirmngr[32170.0] runtime cached certificates: 0
2017-10-01 06:16:46 dirmngr[32170.0] trusted certificates: 149 (148,0,0,1)
2017-10-01 06:16:46 dirmngr[32170.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list63/crls.d/DIR.txt': No such file or directory
2017-10-01 06:16:46 dirmngr[32170.0] creating directory '/tmp/schleuder-test/example.org/list63/crls.d'
2017-10-01 06:16:46 dirmngr[32170.0] new cache dir file '/tmp/schleuder-test/example.org/list63/crls.d/DIR.txt' created
2017-10-01 06:16:47 dirmngr[32170.6] handler for fd 6 started
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list63
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list63/dirmngr.conf
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service
2017-10-01 06:16:47 dirmngr[32170.6] connection from process 32166 (0:0)
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 <- GETINFO version
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> D 2.2.1
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 <- KS_FETCH -- http://127.0.0.1:9999/keys/example.asc
2017-10-01 06:16:47 dirmngr[32170.6] number of system provided CAs: 148
2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999
2017-10-01 06:16:47 dirmngr[32170.6] DBG: dns: libdns initialized (tor mode)
2017-10-01 06:16:47 dirmngr[32170.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:1819:socket_new: object 0x00007f5c38007c60 for fd 7 created
2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:request:
2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> GET /keys/example.asc HTTP/1.0\r\n
2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> Host: 127.0.0.1:9999\r\n
2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:request-header:
2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> \r\n
2017-10-01 06:16:47 dirmngr[32170.6] DBG: http.c:response:
2017-10-01 06:16:47 dirmngr[32170.6] DBG: >> HTTP/1.1 200 OK\r\n
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Content-Type: text/html;charset=utf-8'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Content-Length: 1762'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-XSS-Protection: 1; mode=block'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-Content-Type-Options: nosniff'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'X-Frame-Options: SAMEORIGIN'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Connection: close'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: 'Server: thin'
2017-10-01 06:16:47 dirmngr[32170.6] http.c:RESP: ''
2017-10-01 06:16:47 dirmngr[32170.6] DBG: (1762 bytes sent via D lines not shown)
2017-10-01 06:16:47 dirmngr[32170.6] DBG: chan_6 -> OK
2017-10-01 06:16:48 dirmngr[32170.6] DBG: chan_6 <- BYE
2017-10-01 06:16:48 dirmngr[32170.6] DBG: chan_6 -> OK closing connection
2017-10-01 06:16:48 dirmngr[32170.6] handler for fd 6 terminated
2017-10-01 06:16:48 dirmngr[32170.0] socket file has been removed - shutting down
2017-10-01 06:16:48 dirmngr[32170.0] dirmngr (GnuPG) 2.2.1 stopped
2017-10-01 06:16:49 dirmngr[32207] listening on socket '/tmp/schleuder-test/example.org/list64/S.dirmngr'
2017-10-01 06:16:49 dirmngr[32208.0] permanently loaded certificates: 149
2017-10-01 06:16:49 dirmngr[32208.0] runtime cached certificates: 0
2017-10-01 06:16:49 dirmngr[32208.0] trusted certificates: 149 (148,0,0,1)
2017-10-01 06:16:49 dirmngr[32208.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list64/crls.d/DIR.txt': No such file or directory
2017-10-01 06:16:49 dirmngr[32208.0] creating directory '/tmp/schleuder-test/example.org/list64/crls.d'
2017-10-01 06:16:49 dirmngr[32208.0] new cache dir file '/tmp/schleuder-test/example.org/list64/crls.d/DIR.txt' created
2017-10-01 06:16:50 dirmngr[32208.6] handler for fd 6 started
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list64
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list64/dirmngr.conf
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service
2017-10-01 06:16:50 dirmngr[32208.6] connection from process 32204 (0:0)
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- GETINFO version
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> D 2.2.1
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- KEYSERVER --clear hkp://127.0.0.1:9999
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 -> OK
2017-10-01 06:16:50 dirmngr[32208.6] DBG: chan_6 <- KS_GET -- 0x98769E8A1091F36BD88403ECF71A3F8412D83889
2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: libdns initialized (tor mode)
2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:16:50 dirmngr[32208.6] DBG: dns: libdns initialized (tor mode)
2017-10-01 06:16:58 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Connection closed in DNS
2017-10-01 06:16:58 dirmngr[32208.6] resolve_dns_addr failed while checking '127.0.0.1': Connection closed in DNS
2017-10-01 06:16:58 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Success
2017-10-01 06:17:00 dirmngr[32208.6] number of system provided CAs: 148
2017-10-01 06:17:00 dirmngr[32208.6] DBG: http.c:connect_server: trying name='127.0.0.1' port=9999
2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:17:00 dirmngr[32208.6] can't connect to '127.0.0.1': no IP address for host
2017-10-01 06:17:00 dirmngr[32208.6] error connecting to 'http://127.0.0.1:9999': Unknown host
2017-10-01 06:17:00 dirmngr[32208.6] marking host '127.0.0.1' as dead
2017-10-01 06:17:00 dirmngr[32208.6] DBG: dns: resolve_dns_name(127.0.0.1): Success
2017-10-01 06:17:01 dirmngr[32208.6] DBG: dns: resolve_dns_addr(): Success
2017-10-01 06:17:01 dirmngr[32208.6] host '127.0.0.1' marked as dead
2017-10-01 06:17:01 dirmngr[32208.6] command 'KS_GET' failed: No keyserver available
2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 -> ERR 167772346 No keyserver available <Dirmngr>
2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 <- BYE
2017-10-01 06:17:01 dirmngr[32208.6] DBG: chan_6 -> OK closing connection
2017-10-01 06:17:01 dirmngr[32208.6] handler for fd 6 terminated
2017-10-01 06:17:02 dirmngr[32208.6] handler for fd 6 started
2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list64
2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list64/dirmngr.conf
2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service
2017-10-01 06:17:02 dirmngr[32208.6] connection from process 32243 (0:0)
2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 <- KILLDIRMNGR
2017-10-01 06:17:02 dirmngr[32208.6] DBG: chan_6 -> ERR 167788543 End of file <Dirmngr> - [closing connection]
2017-10-01 06:17:02 dirmngr[32208.0] socket file has been removed - shutting down
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20171030/35d6927b/attachment.sig>
More information about the Reproducible-builds
mailing list