[Piuparts-devel] Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

Thu May 25 08:22:12 UTC 2017

On Tue, May 23, 2017 at 9:55 AM, Michael Stapelberg <stapelberg at debian.org>
wrote:

>
>
> On Tue, May 23, 2017 at 3:34 AM, Holger Levsen <holger at layer-acht.org>
> wrote:
>
>> Hi Michael,
>>
>> are you subscribed to this bug or do you need cc:s?
>>
>
> I’m not subscribed, so please keep me cc'ed :).
>
>
>>
>> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
>> > This is in relation to https://bugs.debian.org/850917, where we added
>> > code to log update-alternatives calls.
>> >
>> > Now that the calls are logged, we need to somehow make Debiman aware
>> > of the calls. To that end, I wrote a program which will read all
>> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
>> > file with the links created by the various binary packages.
>>
>> thanks for the nice summary and proposal!
>>
>> > Find the source of that program at
>> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effc
>> d85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>>
>> looks simple and clean enough!
>>
>> > I then ran the program like so on piuparts.debian.org:
>>
>> /me really likes that pejacevic is developer accessable! ;)
>>
>> > % debiman-piuparts-distill \
>> >     -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
>> >     -output=debiman/testing.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>> >   -output=debiman/unstable.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>> >   -output=debiman/experimental.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>> >   -output=debiman/stable.json.gz
>>
>> ok, so only parsing 4 suites… how long did these commands run?
>>
>
> 294,21s user 212,64s system 30% cpu 27:36,09 total
>
>
>>
>> > I then copied debiman/*.json.gz over to manziarly.debian.org,
>> > re-generated the entire site, and slave alternative manpages now work
>> > as expected :).
>>
>> hehe, nice! got an example?
>>
>
> Sure:
> https://manpages.debian.org/unstable/postgresql-client-9.
> 6/pg_dump.1.en.html
> https://manpages.debian.org/unstable/vis/vi.1.en.html
>
>
>>
>> > Obviously, the above was a one-off manual effort, so now we should
>> > discuss how we’d like to set things up for a more permanent export.
>>
>> sure!
>>
>> > Here’s a strawman proposal. We could:
>> >
>> > 1. Add a crontab entry, which triggers a shell script, which will
>> >    update debiman-piuparts-distill, then run it on the logs
>> >    directories.
>>
>> sounds good, how often would you like to run it?
>>
>
> Once a day would be sufficient, I think.
>
>
>>
>> > 2. Store the resulting *.json.gz files in a directory called
>> >    /srv/piuparts.debian.org/htdocs/for-manpages.d.o
>>
>> I don't really like the "for-manpages.d.o" directory name, but cannot
>> think
>> of a better one either, so…
>>
>> > 3. Download the files via HTTPS on manziarly.d.o.
>>
>> great, so only step 1+2 are on the piuparts side :)
>>
>> > Step ① requires installing the Go compiler
>> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
>> > could avoid this if we’re all okay with just copying the
>> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
>> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
>> > the piuparts git repository feels somewhat unclean.
>>
>> nah, installing golang-go from jessie or jessie-backports is easy, it
>> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
>> files
>> debian/control and debian/changelog and a mail to admin at rt.d.o with the
>> subject "Debian RT: .*" to generate a ticket… which golang-go version do
>> you want?
>>
>
> Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> from jessie-backports, please :)
>
>
>>
>> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
>> > a patch.
>>
>> Yup, seems very fine to me! Please do!
>>
>
> Great. Find a patch attached. Note that it’s untested, as golang-go is not
> yet installed and I lack permission to run as piupartsm. I’m not sure what
> your development process is for changes like this, but feel free to fix up
> the patch as necessary.
>
>
>>
>> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
>> main()
>> and explaining the purpose of the various functions ;-) I was just giving
>> it
>> a 2nd look after wanting to ask you to explain it/go-basics to me the next
>> time we met but then thought I should first try to understand it myself
>> ;-)
>> And then I noticed some more comments explaining/introducing the purpose
>> of the following code lines would be nice…
>>
>
> Sure. I’ll add some commentary, and I’m also happy to explain any details
> — just ask.
>

Commentary added:
https://github.com/Debian/debiman/commit/8306cf79924d18db55a1bc0123d63ac98ad2f2c4

Let me know if anything else deserves an explanation.

>
>
>>
>> I suppose cmd/debiman-piuparts-distill/piuparts.go should live in
>> master-bin
>> in piuparts.git or what would you propose? I don't think we wanna compile
>> it
>> when building the piuparts-master package (for the Debian archive) but
>> rather
>> only build it when installing on pejacevic…
>>
>
> It could continue living in github.com/Debian/debiman, or we could move
> it to piuparts. The advantage of keeping it in github.com/Debian/debiman
> is that we don’t need to duplicate code between different git repositories.
> Installing/updating the command is as simple as running “go get -u
> github.com/Debian/debiman/cmd/debiman-piuparts-distill”.
>
> --
> Best regards,
> Michael
>

-- 
Best regards,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/piuparts-devel/attachments/20170525/aa81e59a/attachment.html>