Bug#1102062: zookeeper: FTBFS: expected: <1> but was: <0>

[tony mancill]

Hi Pierre,

On Sun, Dec 14, 2025 at 10:13:39PM +0100, Pierre Gruet wrote:
> > As a justification for disabling the tests by default, here are failure
> > counts by test for a sequence of 50 or so builds. Approximately a 50%
> > build failure rate and 10 distinct failing tests (so far):
> >
> > 9 [ERROR] RequestThrottlerTest.testRequestThrottler:235 expected: <5> but
> was: <4>
> > 8 [ERROR] SaslAuthRequiredMultiClientTest.testClientOpWithInvalidSASLPasswordAuthAfterSuccessLogin:76->ClientBase.createClient:185->ClientBase.createClient:190->ClientBase.createClient:205->ClientBase.createClient:224
> expected [0x1004bb8efa30001] ==> expected: <1> but was: <0>
> > 5 [ERROR] SnapshotAndRestoreCommandTest.testSnapshotAndRestoreCommand_streaming:168->validateSnapshotMetrics:398
> expected: <true> but was: <false>
> > 3 [ERROR] QuorumPeerMainTest.testLeaderOutOfView:884 expected: <LOOKING>
> but was: <FOLLOWING>
> > 3 [ERROR] SessionTest.testSessionStateNoDupStateReporting:294 expected:
> <[SyncConnected, Disconnected, Expired]> but was: <[SyncConnected,
> Disconnected]>
> > 1 [ERROR] WatcherCleanerTest.testDeadWatcherMetrics:168->ZKTestCase.waitForMetric:144->ZKTestCase.waitForMetric:150->ZKTestCase.waitFor:140
> metric "max_dead_watchers_cleaner_latency" failed to match after 30 seconds
> > 1 [ERROR] RestoreQuorumTest.testRestoreAfterQuorumLost:93 expected: <20>
> but was: <19>
> > 1 [ERROR] ReconfigRollingRestartCompatibilityTest.testRollingRestartWithHostAddedAndRemoved:317
> waiting for server 3 being up ==> expected: <true> but was: <false>
> > 1 [ERROR]
> DIFFSyncTest.testLeaderShutdown_AckProposalBeforeAckNewLeader:191 expected:
> <200000001> but was: <100000003>
> > 1 [ERROR] CnxManagerTest.testWorkerThreads:511 Mon Jul 07 06:25:51 UTC
> 2025 Incorrect number of Worker threads for sid=0 expected 4 found 2 ==>
> expected: <null> but was: <Mon Jul 07 06:25:51 UTC 2025 Incorrect number of
> Worker threads for sid=0 expected 4 found 2>
> >
> >
> > So, I have prepared an upload for unstable that skips running the test
> > suite by default but permits the builder to enable it by setting an
> > environment variable. Kind of the opposite of DEB_BUILD_OPTIONS=nocheck,
> > although I used a different, non-standard variable to avoid confusion.
> >
> > Any concerns with an upload now? I see that Paul has already tagged the
> > bug with trixie-ignore. I am not expecting this to migrate before the
> > release (unless the Release Managers feel otherwise).
> >
> > I will continue to work on addressing the root cause.
> 
> I will consider giving this "root cause" some attention in the upcoming
> days. Just in case, have you been able to gather more information about that
> up to now?

I have not made progress towards isolating the root cause of these build
failures.  They appear to be more likely when the build is performed on
a virtualized machine (via a hypervisor), but I have observed build
failures (very rarely) on physical machines too.

I do not believe the issues are Debian-specific, as I was able to
reproduce them in other contexts, but I don't have percentages to share.

One idea I have it to try to use Fray [1] to see if there might be a
common issue with the upstream, but I have not had time to pursue it.

Best regards,
tony

[1] https://github.com/cmu-pasta/fray