[Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

Package: piuparts
Severity: wishlist

This is in relation to https://bugs.debian.org/850917, where we added
code to log update-alternatives calls.

Now that the calls are logged, we need to somehow make Debiman aware
of the calls. To that end, I wrote a program which will read all
available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
file with the links created by the various binary packages.

Find the source of that program at
https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go

I then ran the program like so on piuparts.debian.org:

% debiman-piuparts-distill \
    -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
    -output=debiman/testing.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
  -output=debiman/unstable.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
  -output=debiman/experimental.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
  -output=debiman/stable.json.gz

I then copied debiman/*.json.gz over to manziarly.debian.org,
re-generated the entire site, and slave alternative manpages now work
as expected :).

Obviously, the above was a one-off manual effort, so now we should
discuss how we’d like to set things up for a more permanent export.

Here’s a strawman proposal. We could:

1. Add a crontab entry, which triggers a shell script, which will
   update debiman-piuparts-distill, then run it on the logs
   directories.

2. Store the resulting *.json.gz files in a directory called
   /srv/piuparts.debian.org/htdocs/for-manpages.d.o

3. Download the files via HTTPS on manziarly.d.o.

Step ① requires installing the Go compiler
(https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
could avoid this if we’re all okay with just copying the
debiman-piuparts-distill binary onto piuparts.d.o, but referring to
e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
the piuparts git repository feels somewhat unclean.

Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
a patch.