Bug#1067232: limit diffoscope recursions on packages where diffoscope runs into a timeout

[Holger Levsen]

package: jenkins.debian.org
severity: wishlist

hi,

in https://salsa.debian.org/qa/jenkins.debian.net/-/merge_requests/163
James Addison suggested to use --max-container-depth=3 (or 5) for
when diffscope runs into a timeout on a package. (or rather not then,
but always, which why this MR wasnt accepted.)

However this lead to the following discussion on irc:

<h01ger> jayaddison: thanks for closing MR163!
<jayaddison> no probs!  thought it might confuse/distract someone later otherwise :)
<h01ger> jayaddison: thinking about it, we could probably run diffoscope with the reduced depth option on those packages we know diffoscope timeouts, eg all packages listed more than 4 times on https://tests.reproducible-builds.org/debian/index_breakages.html
<h01ger> the breakage job could generated that list, and then reproducible_build.sh could consume it and act accordingly
<jayaddison> interesting idea.  that was going to be my next question (how to maintain the list of relevant packages)
<jayaddison> I'm slightly averse to adaptive testing because it becomes a maintenance/debug challenge over time, but am thinking about it
<jayaddison> what's the effect of a diffoscope timeout at the moment?  no output at all?
< h01ger> thinking about it, this could cause an interesting jojo effect: git-annex is tested, diffoscope runs into a timeout. this happens more than 4 times -> git-annex is put on the list of packages which diffoscope is run on with reduced depth. thus, (hopefullly! now) diffoscope doesnt run into a timeout 
          anymore. this happens so often that git-annex appears less than 4 times on the breakages page -> 
<h01ger> git-annex is tested without a timeout again, rinse repeat, until diffoscope or git-annex is improved
<h01ger> yes, its basically empty, see eg https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/git-annex.html
<h01ger> basically true for any link on https://tests.reproducible-builds.org/debian/index_breakages.html
<jayaddison> ok.  I guess 'in theory' it'd be nice for diffoscope to diff to depth 1, then write output, then proceed to depth 2, rewrite output, etc.
<h01ger> nah, depth 3/4/5 should be fine, i hope
<jayaddison> first few levels are generally .deb, data/xz or something, and... I forget.  but fast/inexpensive compute-wise
<h01ger> yeah
<jayaddison> if we could write output after each layer, there should always be 'something' to show on the results page even if it later fails at depth 8 or whatever
<jayaddison> s/fails/times-out
<h01ger> i'd just go with layer X for a start, i doubt its useful to make this more sophisticated, esp from the start. also its sophisticated enough to only invoke this for a few timeouting packages

or maybe even simpler: first run diffoscope normally, then if that runs into a timeout,
run with --max-container-depth=3 (or 5). I think this would be acceptable with
only 286 suite/arch/package combinations currently which run into a timeout.

I'd still like to store/record somewhere (for later reading, eg to then semiautomatically
add this information to notes.git) that diffoscope ran into a timeout on this package,
but for the moment I don't have a plan how to that exactly.


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Deutschland ist so rechts, es wird sogar diskutiert ob die Nazis links waren.
(@elhotzo, 20220206)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-dev/attachments/20240320/330c22d1/attachment.sig>