distributing .buildinfo files (Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master)

[Philipp Kern]

On 2017-09-02 23:48, Holger Levsen wrote:
> On Mon, Jul 03, 2017 at 07:23:29PM +0200, Philipp Kern wrote:
>> > Not yet.  We people from the reproducible team couldn't find a way to
>> > usefully talk to ftp-masters people, whom never replied to any of the
>> > questions in the thread at #763822 (they only did some quick comments on
>> > IRC, and we have been left on guessing what they would like…).
>> >
>> > Anyhow, .buildinfo files are stored in ftp-master, just not exported to
>> > the mirrors, you can find them in
>> > coccia.debian.org:/srv/ftp-master.debian.org/<something>.
>> 
>> So I suppose we talk about 13 GB[1] of static content in about 1.7M
>> files. Is that something that could be distributed through
>> static.debian.org if there are concerns around inodes for the main
>> mirrors? Given that they would be accessed mostly rarely[2]?
>> 
>> [1] 7.7kB (75%ile as mentioned in the referenced bug) * 55000 binary
>> packages * 10 architectures * 3 versions - so quite conservatively
>> [2] So supposedly a CDN wouldn't bring a lot of benefit as individual
>> files aren't likely to be hit frequently.
> 
> using static.debian.org seems to be a good idea to me, what would be
> needed to make
> this happen?
> 
> or, we could put them in a git repo instead, and use git.debian.org…

Git is an interesting thought for incremental mirroring. But then it 
also seems to be a poor choice for something that is an only growing 
repository of data.

What I think should be a requirement is that the data is pushed out 
before the mirror pulse. Otherwise you end up with a race where you try 
to mirror the data including the buildinfo but can't access it. (It's a 
little unfortunate that we don't simply put them onto the mirrors.

Kind regards
Philipp Kern