Bug#988191: ionos-1 proxy causes lots of temporary build failures by being slow to respond

[Helmut Grohne]

Package: jenkins.debian.org
Severity: important

The proxy running on ionos-1 is frequently overloaded and slow to
respond. This frequently results in apt or mmdebstrap failing to
download packages and makes a lot of jobs fail. For rebootstrap it seems
to roughly kill 1/3 of all jobs. As such the situation is a significant
waste of resources.

Looking into munin (thanks for providing that!), there seems to be a
good correlation between significant iowait
https://jenkins.debian.net/munin/debian.net/ionos1-amd64.debian.net/cpu.html
and the temporary failures. The system has 40G swap in use at the moment
and significant swap traffic. It seems unsurprising that a squid running
on such a system would not perform well and time out requests
occasionally.

A long term solution would be separating the proxy component from the
component that causes heavy swapping (whatever that is). The simplest
solution would be dedicating a separate small vm at the same datacenter
to the proxying. I suppose that 1 vcpu, 1G ram and like 30G of disk
space would be the minimum to support a proxy-only vm.

Do you agree with the proposed solution?

Helmut