partial snapshot mirror amd64/bullseye/bookworm

Frédéric Pierret frederic.pierret at qubes-os.org
Mon Mar 15 09:29:21 GMT 2021


Hi,

Le 2/27/21 à 3:09 PM, Holger Levsen a écrit :
> Hi,
> 
> snapshot.debian.org is an awesome service for the wider free software community
> and especially for those working on reproducible builds. Sadly accessing *many*
> packages from it is limited and troublesome (see below for bug numbers), thus
> we (mostly Frédéric Pierret and myself) came up with the idea of setting up a
> partial mirror, covering only the years 2017 until now and arch:amd64 and arch:all
> only as well. (for a start, maybe we need 2015+2016 too and maybe we can afford
> to also host arm64 or some other architecture...)
> 
> Background: several projects (at NYU, from Qubes, from Debian, some independent
> researchers) want to setup rebuilders of Debian (bullseye) rebuilding all 30000
> source packages which need thousands of different snapshots and thus are regularily
> hit/hurt by these bugs:
> 
> #977653 Please document rate limits on snapshots.debian.org
> #960304 snapshot.debian.org: Snapshot repo repeatedly cutting off connection, returning partial content
> #969906 snapshot.debian.org: error 500 internal server error after some requests via Python
> 
> It's possible to work around them partly/sometimes, but it seems clear by
> now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence
> the idea of a partial snapshot mirror which is only suitable to rebuild bullseye
> am64 (but not previous release or other archs.) - and which can also be mirrored
> more easily than the whole of snapshot.d.o if the need arises.
> 
> Does that sound feasable/sensible from the snapshot.d.o team perspective?
> 
> The idea would be to ask Debian for money to buy four 16tb discs (one currently
> costs around 300€) and then have two shipped to Frédéric (for local development)
> and two shipped to OSUOSL.org, which probably could host them for us. (We've got
> an informal offer we need to formalize once/if this idea is deemed good.)
> 
> misc notes:
> 
> this is ment as prototype / temporary solution for the next 2-3 years to that we can
> continue to develop (several) tools for the verification of Reproducible Builds of
> amd64/bullseye and probably amd64/bookworm, but it's clear that in the long
> term we want solutions for other supported archs and releases after bookworm.
> 
> I'd very much hope that snapshot.d.o can be scaled, just this will take time, probably
> a lot, and so I think it makes sense to spend <1.5k€ to have an interim solution *now*.
> 
> In theory the machine offered to us informally at osuosl can take 6 sata drives...
> 
> feedback very much welcome!
> 
> 

A quick update: with respect to the original message and the RT ticket [rt.debian.org #8547] for having a one shot exported database to find a mirroring solution, I've started working on mirroring in another way without the help of the DB. Currently, a POC is running for 'amd64' and 'unstable only. For example a little tree output: https://gist.githubusercontent.com/fepitre/27e18815712fd1df96cb72993a78c4b7/raw/dacf81d2334305a8db5e9ea338d55ef31695b950/fragment.

Once I've got several timestamps synced for 'unstable', 'bullseye' ('amd64 and 'all' arches) I will properly check that I can use those local mirrors for package rebuild.

Best regards,
Frédéric

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20210315/7a7ce3a4/attachment.sig>


More information about the Reproducible-builds mailing list