[Piuparts-devel] Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

Tue May 23 07:55:11 UTC 2017

On Tue, May 23, 2017 at 3:34 AM, Holger Levsen <holger at layer-acht.org>
wrote:

> Hi Michael,
>
> are you subscribed to this bug or do you need cc:s?
>

I’m not subscribed, so please keep me cc'ed :).

>
> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> > This is in relation to https://bugs.debian.org/850917, where we added
> > code to log update-alternatives calls.
> >
> > Now that the calls are logged, we need to somehow make Debiman aware
> > of the calls. To that end, I wrote a program which will read all
> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> > file with the links created by the various binary packages.
>
> thanks for the nice summary and proposal!
>
> > Find the source of that program at
> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541
> ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>
> looks simple and clean enough!
>
> > I then ran the program like so on piuparts.debian.org:
>
> /me really likes that pejacevic is developer accessable! ;)
>
> > % debiman-piuparts-distill \
> >     -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
> >     -output=debiman/testing.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
> >   -output=debiman/unstable.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
> >   -output=debiman/experimental.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
> >   -output=debiman/stable.json.gz
>
> ok, so only parsing 4 suites… how long did these commands run?
>

294,21s user 212,64s system 30% cpu 27:36,09 total

>
> > I then copied debiman/*.json.gz over to manziarly.debian.org,
> > re-generated the entire site, and slave alternative manpages now work
> > as expected :).
>
> hehe, nice! got an example?
>

Sure:
https://manpages.debian.org/unstable/postgresql-client-9.6/pg_dump.1.en.html
https://manpages.debian.org/unstable/vis/vi.1.en.html

>
> > Obviously, the above was a one-off manual effort, so now we should
> > discuss how we’d like to set things up for a more permanent export.
>
> sure!
>
> > Here’s a strawman proposal. We could:
> >
> > 1. Add a crontab entry, which triggers a shell script, which will
> >    update debiman-piuparts-distill, then run it on the logs
> >    directories.
>
> sounds good, how often would you like to run it?
>

Once a day would be sufficient, I think.

>
> > 2. Store the resulting *.json.gz files in a directory called
> >    /srv/piuparts.debian.org/htdocs/for-manpages.d.o
>
> I don't really like the "for-manpages.d.o" directory name, but cannot think
> of a better one either, so…
>
> > 3. Download the files via HTTPS on manziarly.d.o.
>
> great, so only step 1+2 are on the piuparts side :)
>
> > Step ① requires installing the Go compiler
> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> > could avoid this if we’re all okay with just copying the
> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> > the piuparts git repository feels somewhat unclean.
>
> nah, installing golang-go from jessie or jessie-backports is easy, it
> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
> files
> debian/control and debian/changelog and a mail to admin at rt.d.o with the
> subject "Debian RT: .*" to generate a ticket… which golang-go version do
> you want?
>

Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
from jessie-backports, please :)

>
> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> > a patch.
>
> Yup, seems very fine to me! Please do!
>

Great. Find a patch attached. Note that it’s untested, as golang-go is not
yet installed and I lack permission to run as piupartsm. I’m not sure what
your development process is for changes like this, but feel free to fix up
the patch as necessary.

>
> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
> main()
> and explaining the purpose of the various functions ;-) I was just giving
> it
> a 2nd look after wanting to ask you to explain it/go-basics to me the next
> time we met but then thought I should first try to understand it myself ;-)
> And then I noticed some more comments explaining/introducing the purpose
> of the following code lines would be nice…
>

Sure. I’ll add some commentary, and I’m also happy to explain any details —
just ask.

>
> I suppose cmd/debiman-piuparts-distill/piuparts.go should live in
> master-bin
> in piuparts.git or what would you propose? I don't think we wanna compile
> it
> when building the piuparts-master package (for the Debian archive) but
> rather
> only build it when installing on pejacevic…
>

It could continue living in github.com/Debian/debiman, or we could move it
to piuparts. The advantage of keeping it in github.com/Debian/debiman is
that we don’t need to duplicate code between different git repositories.
Installing/updating the command is as simple as running “go get -u
github.com/Debian/debiman/cmd/debiman-piuparts-distill”.

-- 
Best regards,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/piuparts-devel/attachments/20170523/8e5bd45f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-run-debiman-piuparts-distill-via-cron.patch
Type: text/x-patch
Size: 1916 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/piuparts-devel/attachments/20170523/8e5bd45f/attachment-0001.bin>