[Piuparts-devel] Migration to testing blocked by broken piuparts?

[Andreas Beckmann]

On 08/01/2020 19.09, Boyuan Yang wrote:
> 在 2020-01-08三的 16:36 +0530，Pirate Praveen写道：
>>
>> On വ്യാ, Jan 2, 2020 at 21:43, debacle <debacle at debian.org> 
>> wrote:
>>> On 2020-01-02 11:53, Lisandro Damián Nicanor Pérez Meyer wrote:
>>>>  El jue., 2 ene. 2020 08:28, Julien Cristau <jcristau at debian.org> 
>>>> escribió:
>>>>  > No, it'll eventually get retried and (assuming it passes) the 
>>>> migration
>>>>  > block will lift.
>>>>
>>>>  And "eventually" is normally how many days?
>>>
>>> If I've learned one thing in Debian, it's patience!
>>
>> Its already 10 days [1] and newly uploaded packages are now getting 
>> migrated.

>> So I don't think the packages failed due to bug are getting 
>> any priority over the newly uploaded ones.

Correct.

>> I'd have expected them to be 
>> retried before new packages. Can this still be done?

Not easily. But I have some ideas for giving priority to
failures that are marked for retesting in the future.

> I checked the piuparts documentation just then [2] and found out that unlike
> ci.d.o or reproducible-build checks, the piuparts test will *not*
> automatically be retried for the same package of the same version (same
> upload).

Where is that written?

> If I understand correctly, it seems that those packages uploaded in
> the period when the piuparts were broken will never migrate to Testing unless
> another upload is made.

That is wrong.

piuparts is chewing on a significant backlog of packages:

jessie2bpo2stretch/ waiting-to-be-tested=1 waiting-for-dependency-to-be-tested=1
bullseye-rcmd/ waiting-to-be-tested=9 waiting-for-dependency-to-be-tested=0
stretch2buster-rcmd/ waiting-to-be-tested=10 waiting-for-dependency-to-be-tested=1
stable2sid/ waiting-to-be-tested=44 waiting-for-dependency-to-be-tested=31
testing2sid/ waiting-to-be-tested=45 waiting-for-dependency-to-be-tested=134
stable22sid/ waiting-to-be-tested=55 waiting-for-dependency-to-be-tested=815
sid-merged-usr/ waiting-to-be-tested=75 waiting-for-dependency-to-be-tested=83
wheezy2jessie-lts/ waiting-to-be-tested=232 waiting-for-dependency-to-be-tested=611
oldstable222sid/ waiting-to-be-tested=5345 waiting-for-dependency-to-be-tested=9246

The processing preferences in piuparts are
1) untested packages
2) packages marked for retesting, starting with failing ones

failed logs get marked for retesting after 2 days
passed logs get marked for retesting after three months
(this process gets throttled if the slaves cannot process them in time)
failed logs expire (i.e. they get deleted) after about a week
(throttled to about 50-100 logs per day)

This expiration process ensures progress in situations like these.
The ~200 failures in sid are all queued for rechecking, but wait
for idle slaves.
Using the expiration process, they should all get retested within a few days.

This does work. E.g. pocl migrated yesterday after being blocked by piuparts for a few days.

> This is definitely far from ideal. Is it possible to manually trigger a retry
> for each packages hit by this regression? At least asking maintainers to
> search throught britney update excuses [3] and making another upload is really
> not a valid option. BTW, searching "Rejected due to piuparts regression" in
> [3] gives 135 results.
> 
> [2] 
> https://wiki.debian.org/piuparts/FAQ#Q._Can_I_somehow_tell_piuparts_to_retest_my_package.3F
> 
> [3] https://release.debian.org/britney/update_excuses.html

Andreas

PS:

anbe at pejacevic:/srv/piuparts.debian.org/master$ for x in  */ ; do grep waiting-to-be-tested $x/master.log | tail -n 1 | awk "{ print \"$x \"  \$6 \" \" \$7 }" ; done | sort -n -t= -k2