partial snapshot mirror amd64/bullseye/bookworm

[Holger Levsen]

Hi,

snapshot.debian.org is an awesome service for the wider free software community
and especially for those working on reproducible builds. Sadly accessing *many*
packages from it is limited and troublesome (see below for bug numbers), thus
we (mostly Frédéric Pierret and myself) came up with the idea of setting up a
partial mirror, covering only the years 2017 until now and arch:amd64 and arch:all
only as well. (for a start, maybe we need 2015+2016 too and maybe we can afford
to also host arm64 or some other architecture...)

Background: several projects (at NYU, from Qubes, from Debian, some independent
researchers) want to setup rebuilders of Debian (bullseye) rebuilding all 30000
source packages which need thousands of different snapshots and thus are regularily
hit/hurt by these bugs:

#977653 Please document rate limits on snapshots.debian.org
#960304 snapshot.debian.org: Snapshot repo repeatedly cutting off connection, returning partial content
#969906 snapshot.debian.org: error 500 internal server error after some requests via Python

It's possible to work around them partly/sometimes, but it seems clear by
now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence 
the idea of a partial snapshot mirror which is only suitable to rebuild bullseye
am64 (but not previous release or other archs.) - and which can also be mirrored
more easily than the whole of snapshot.d.o if the need arises.

Does that sound feasable/sensible from the snapshot.d.o team perspective?

The idea would be to ask Debian for money to buy four 16tb discs (one currently
costs around 300€) and then have two shipped to Frédéric (for local development)
and two shipped to OSUOSL.org, which probably could host them for us. (We've got
an informal offer we need to formalize once/if this idea is deemed good.)

misc notes:

this is ment as prototype / temporary solution for the next 2-3 years to that we can
continue to develop (several) tools for the verification of Reproducible Builds of
amd64/bullseye and probably amd64/bookworm, but it's clear that in the long
term we want solutions for other supported archs and releases after bookworm.

I'd very much hope that snapshot.d.o can be scaled, just this will take time, probably
a lot, and so I think it makes sense to spend <1.5k€ to have an interim solution *now*.

In theory the machine offered to us informally at osuosl can take 6 sata drives...

feedback very much welcome!


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁       holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

I'm looking forward to Corona being a beer again and Donald a duck.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20210227/29a40e30/attachment.sig>