[Piuparts-devel] Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hi Michael,

are you subscribed to this bug or do you need cc:s?

On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> This is in relation to https://bugs.debian.org/850917, where we added
> code to log update-alternatives calls.
> 
> Now that the calls are logged, we need to somehow make Debiman aware
> of the calls. To that end, I wrote a program which will read all
> available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> file with the links created by the various binary packages.
 
thanks for the nice summary and proposal!

> Find the source of that program at
> https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go

looks simple and clean enough!
 
> I then ran the program like so on piuparts.debian.org:

/me really likes that pejacevic is developer accessable! ;)

> % debiman-piuparts-distill \
>     -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
>     -output=debiman/testing.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>   -output=debiman/unstable.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>   -output=debiman/experimental.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>   -output=debiman/stable.json.gz

ok, so only parsing 4 suites… how long did these commands run?

> I then copied debiman/*.json.gz over to manziarly.debian.org,
> re-generated the entire site, and slave alternative manpages now work
> as expected :).

hehe, nice! got an example?

> Obviously, the above was a one-off manual effort, so now we should
> discuss how we’d like to set things up for a more permanent export.

sure!

> Here’s a strawman proposal. We could:
> 
> 1. Add a crontab entry, which triggers a shell script, which will
>    update debiman-piuparts-distill, then run it on the logs
>    directories.

sounds good, how often would you like to run it?

> 2. Store the resulting *.json.gz files in a directory called
>    /srv/piuparts.debian.org/htdocs/for-manpages.d.o

I don't really like the "for-manpages.d.o" directory name, but cannot think
of a better one either, so…

> 3. Download the files via HTTPS on manziarly.d.o.

great, so only step 1+2 are on the piuparts side :)

> Step ① requires installing the Go compiler
> (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> could avoid this if we’re all okay with just copying the
> debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> the piuparts git repository feels somewhat unclean.

nah, installing golang-go from jessie or jessie-backports is easy, it
just needs a patch for git.debian.org/git/mirror/debian.org.git for the files
debian/control and debian/changelog and a mail to admin at rt.d.o with the 
subject "Debian RT: .*" to generate a ticket… which golang-go version do
you want?

> Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> a patch.

Yup, seems very fine to me! Please do!

cmd/debiman-piuparts-distill/piuparts.go also could use a comments in main()
and explaining the purpose of the various functions ;-) I was just giving it 
a 2nd look after wanting to ask you to explain it/go-basics to me the next
time we met but then thought I should first try to understand it myself ;-) 
And then I noticed some more comments explaining/introducing the purpose
of the following code lines would be nice…

I suppose cmd/debiman-piuparts-distill/piuparts.go should live in master-bin
in piuparts.git or what would you propose? I don't think we wanna compile it
when building the piuparts-master package (for the Debian archive) but rather
only build it when installing on pejacevic…


-- 
cheers,
	Holger
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/piuparts-devel/attachments/20170523/e5643d3c/attachment.sig>