Review 582 (REQUEST_CHANGES at 95178349) found a residual fail-open of the
same class the PR set out to close. Acknowledgement coverage was decided by
comparing a count against a count:
covers_live_sessions = live_count_known and acked_count >= sessions_live_other
Nothing bound an acknowledgement to the identity of a session that actually
owed one, so acknowledgements supplied for the requesting session and for a
session that does not exist satisfied the obligations of two live sessions
that never answered - minting a clean, correctly signed proof and an allow
verdict from the restart gate.
Coverage is now derived from authoritative impact-report evidence:
- New `_required_ack_sessions()` derives the required session ids from the
report itself, via `ack_state` keys and/or `affected_sessions` filtered on
`live and not is_requester`. The requester is excluded only on explicit
`is_requester` evidence, never inferred.
- When both views are present they must name the same set, and the result is
reconciled against `counts.sessions_live_other`. Missing, malformed,
duplicated, contradictory, or unreconcilable identity evidence fails closed
and outranks every permitting path, including the timeout policy.
- Coverage requires every required id to carry an explicit acknowledgement
token keyed by that id. Acknowledgements for the requester, for unknown
ids, or for fabricated ids never increase coverage.
- Caller-supplied acknowledgement cardinality is no longer proof of anything.
Failure propagates unchanged through `acks_or_timeout` -> `proof.clean` ->
`failed_checks` -> `gate_apply_restart` verdict `deny` / `allow=False`.
The earlier missing-acknowledgement remediation is preserved in full: absent,
None, non-mapping, empty, partial, stale, and unparseable acks still fail
closed, `ack_timeout_policy_applied` stays strict `value is True`, and the
legitimate zero-live-sessions and explicit-timeout paths still pass.
Reviewer's reproduction, before and after this commit:
sessions_live_other = 2
report ack_state = {'other-0': 'pending', 'other-1': 'pending'}
supplied acks = {'req': 'ack', 'totally-bogus-session': 'ack'}
before: acks_or_timeout = True | proof.clean = True | gate allow
after: acks_or_timeout = False | proof.clean = False | gate deny
Tests: 22 new cases in `AcknowledgementIdentityBindingTests` covering the
reviewer's exact exploit, wrong-ids-with-sufficient-count, partial identity
match, requester-only acks, fabricated ids, unproven per-session states,
missing/malformed/contradictory identity evidence, count mismatch, and the
preserved success paths.
Verification:
- `pytest tests/test_drain_proof.py` -> 61 passed, 56 subtests
(baseline at 95178349: 39 passed, 26 subtests)
- Restart surface (6 modules) -> 154 passed, 68 subtests, exit 0
(baseline at 95178349: 132 passed, 38 subtests)
- Full `pytest tests/` -> 5177 passed vs baseline 5155 passed; the 23
failures are identical in both runs and pre-exist at 95178349.
Scope: drain_proof.py, tests/test_drain_proof.py.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRUZAf3Fr5n3kqhhiayN6C
(cherry picked from commit 4193b63f415b066ee292386c2c89bc3d2651a0cc)
The acks_or_timeout check treated an absent `acks` key as proof that no
session needed to acknowledge: `drain_state.get("acks") or {}` collapsed
absent, None, and empty into the same value, and the resulting empty mapping
satisfied `no_sessions_to_ack`. The impact report's counts.sessions_live_other
was never consulted, so absence of evidence was read as evidence of absence.
Reproduced at head 1cbbde0089: with
sessions_live_other = 3 and the acknowledgement key absent, acks_or_timeout
passed with detail "no other live sessions required to acknowledge", the proof
minted clean, and gate_apply_restart returned verdict allow — a restart
authorized against three live sessions with zero acknowledgement evidence, and
the resulting artifact carried a valid signature.
Whether acknowledgement is required is now derived from the impact report,
never from the shape of the drain state:
- _live_session_count() reads counts.sessions_live_other and returns None for a
missing, malformed, negative, or bool value, so an unreadable report fails
closed instead of reading as "nobody was live".
- Absent, None, non-mapping, empty, partially-covering, and unparseable or
stale acknowledgement data all fail closed while live sessions require
acknowledgement.
- _is_acknowledged() no longer coerces with str(); only an explicit
"ack"/"acked"/"acknowledged" string counts, so None, timestamps, and
"pending"/"stale" markers are never read as an acknowledgement.
- Present-but-unacknowledged entries fail closed even when the report claims
zero live sessions: that contradiction is not safe to resolve in favour of
the restart.
- ack_timeout_policy_applied stays strict (`value is True`), so an absent, null,
or non-boolean value cannot open the gate on its own.
The genuine no-other-live-sessions case still passes, now justified by the
report proving sessions_live_other == 0 rather than by the absence of data.
Adds AcknowledgementFailClosedTests: 14 cases / 26 subtests covering missing,
null, empty, malformed, stale, partial-coverage, and unproven-count inputs,
the valid-acknowledgement and zero-live-session paths, timeout-policy
strictness, and that a failed check blocks proof.clean, verification, and the
restart gate.
Restart-surface suite: 132 passed, 38 subtests (branch baseline 118 passed,
12 subtests; +14 new tests, no regressions).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VEaP3TohHLFWkp3Z2mmuZw
Add `drain_proof.py`: a machine-verifiable DrainProof artifact plus a
fail-closed verifier and the hard gate the sanctioned restart-apply path
must consult, so a restart can never proceed on a stale or false "ready"
claim (#655 umbrella, child of #658 coordinator / #659 drain / #660
checkpoints).
- DrainProof: HMAC-SHA256 keyed proof-id over a canonical body using a
per-process secret -> non-forgeable within the process; a proof minted in
a prior daemon process will not verify after restart. Short TTL (120s).
- build_drain_proof(): mints the proof from the #658 impact report + the
drain-mode outcomes. Checklist: no in-flight mutations, assignments
stopped, checkpoints complete, handoffs ok, leases handled, acks-or-
timeout. Every check fails closed on missing/ambiguous evidence; the
no-in-flight-mutations and leases-handled checks are derived from the
authoritative impact report, not self-reported.
- verify_drain_proof(): fail-closed — rejects missing, malformed, expired,
signature-mismatched (forged/tampered/prior-process), unclean, or
stale-fingerprint proofs; recomputes cleanliness from the checks rather
than trusting the flag.
- gate_apply_restart(): allow only on a valid clean proof; deny -> durable
incident descriptor; break-glass is the only bypass and is never silent.
- Checkpoint completeness is a supplied input, not a hard dependency on the
(still-unmerged #660) checkpoint schema.
Wire the gate into gitea_request_mcp_restart: dry_run=False now enforces the
hard gate (drain_proof_json required; break-glass via request_break_glass +
GITEA_BREAKGLASS_RESTART_AUTHORIZATION env). The tool still performs no
actual restart — execution remains a further child.
Tests: tests/test_drain_proof.py — 25 cases covering AC#1-4 (apply without
proof denied, successful drain verifiable, open unsafe mutation fails,
pass/fail/expired), forgery/tamper/wrong-secret/stale-fingerprint rejection,
break-glass bypass, and secret hygiene. 25/25 pass (coordinator suite
unaffected: 40/40 together).
Links #652#653#655#658#659#660.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01E7Fv9Bp2XWgvaWa4M1kdR7
(cherry picked from commit e7bcc952bb3e820fda95acbecefeaebfa5f8fcff)
sysadmin
jcwalker3
