feat(restart): pre-restart drain proof and hard gate (#661)
Add `drain_proof.py`: a machine-verifiable DrainProof artifact plus a fail-closed verifier and the hard gate the sanctioned restart-apply path must consult, so a restart can never proceed on a stale or false "ready" claim (#655 umbrella, child of #658 coordinator / #659 drain / #660 checkpoints). - DrainProof: HMAC-SHA256 keyed proof-id over a canonical body using a per-process secret -> non-forgeable within the process; a proof minted in a prior daemon process will not verify after restart. Short TTL (120s). - build_drain_proof(): mints the proof from the #658 impact report + the drain-mode outcomes. Checklist: no in-flight mutations, assignments stopped, checkpoints complete, handoffs ok, leases handled, acks-or- timeout. Every check fails closed on missing/ambiguous evidence; the no-in-flight-mutations and leases-handled checks are derived from the authoritative impact report, not self-reported. - verify_drain_proof(): fail-closed — rejects missing, malformed, expired, signature-mismatched (forged/tampered/prior-process), unclean, or stale-fingerprint proofs; recomputes cleanliness from the checks rather than trusting the flag. - gate_apply_restart(): allow only on a valid clean proof; deny -> durable incident descriptor; break-glass is the only bypass and is never silent. - Checkpoint completeness is a supplied input, not a hard dependency on the (still-unmerged #660) checkpoint schema. Wire the gate into gitea_request_mcp_restart: dry_run=False now enforces the hard gate (drain_proof_json required; break-glass via request_break_glass + GITEA_BREAKGLASS_RESTART_AUTHORIZATION env). The tool still performs no actual restart — execution remains a further child. Tests: tests/test_drain_proof.py — 25 cases covering AC#1-4 (apply without proof denied, successful drain verifiable, open unsafe mutation fails, pass/fail/expired), forgery/tamper/wrong-secret/stale-fingerprint rejection, break-glass bypass, and secret hygiene. 25/25 pass (coordinator suite unaffected: 40/40 together). Links #652 #653 #655 #658 #659 #660. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01E7Fv9Bp2XWgvaWa4M1kdR7 (cherry picked from commit e7bcc952bb3e820fda95acbecefeaebfa5f8fcff)
This commit is contained in:
+58
-8
@@ -2068,6 +2068,7 @@ import lease_lifecycle # noqa: E402
|
||||
import lease_policy # noqa: E402
|
||||
import workflow_dashboard # noqa: E402 # #605 live queue/lease dashboard
|
||||
import restart_coordinator # noqa: E402 # #658 MCP restart coordinator/impact
|
||||
import drain_proof # noqa: E402 # #661 pre-restart drain proof and hard gate
|
||||
import incident_bridge # noqa: E402
|
||||
import sentry_observability # noqa: E402 (#606 optional Sentry observability)
|
||||
import sentry_incident_bridge # noqa: E402 (#607 Sentry→Gitea incident bridge)
|
||||
@@ -22342,6 +22343,8 @@ def gitea_request_mcp_restart(
|
||||
request_override: bool = False,
|
||||
session_id: str | None = None,
|
||||
limit: int = 200,
|
||||
drain_proof_json: str | None = None,
|
||||
request_break_glass: bool = False,
|
||||
) -> dict:
|
||||
"""Evaluate a proposed MCP restart and return an impact preview (#658).
|
||||
|
||||
@@ -22351,10 +22354,15 @@ def gitea_request_mcp_restart(
|
||||
verdict, so the console (#642/#652) and operators can see what a restart
|
||||
would disrupt *before* any concurrent LLM work is destroyed.
|
||||
|
||||
This tool is **dry-run and never restarts anything.** The mutative apply
|
||||
path is a separate child gated by a drain proof (non-goal here); calling
|
||||
with ``dry_run=False`` still performs no restart and reports that apply is
|
||||
not yet available.
|
||||
This tool **never restarts a process.** In dry-run (the default) it returns
|
||||
only the impact preview. With ``dry_run=False`` it enforces the #661 hard
|
||||
gate: the apply request must present a valid, unexpired, clean drain proof
|
||||
(``drain_proof_json``) or it is denied and a durable incident descriptor is
|
||||
returned under ``incident``. Break-glass is the only bypass and is honoured
|
||||
only when ``request_break_glass`` is set *and* the environment carries
|
||||
``GITEA_BREAKGLASS_RESTART_AUTHORIZATION``. Even an authorized gate performs
|
||||
no restart here; actual execution is a further child. The gate outcome is
|
||||
reported under ``apply_gate`` / ``apply_authorized``.
|
||||
|
||||
Operator override authority is read from the process environment
|
||||
(``GITEA_OPERATOR_RESTART_OVERRIDE_AUTHORIZATION``), never self-asserted by
|
||||
@@ -22473,12 +22481,54 @@ def gitea_request_mcp_restart(
|
||||
payload["requesting_session_id"] = sid
|
||||
payload["operator_override_requested"] = bool(request_override)
|
||||
payload["operator_override_authorized"] = operator_authorized
|
||||
# Actual restart execution remains a further child; this tool never restarts
|
||||
# a process. What #661 adds is the *hard gate*: an apply request (dry_run
|
||||
# False) must present a valid, unexpired, clean drain proof, or it is denied
|
||||
# and a durable incident is raised. Break-glass is the only bypass and its
|
||||
# authorization is read from the environment, never self-asserted.
|
||||
payload["apply_supported"] = False
|
||||
if not dry_run:
|
||||
payload["reasons"] = list(payload.get("reasons") or []) + [
|
||||
"apply requested but not supported: sanctioned restart apply is "
|
||||
"gated by a drain proof (separate child); no restart performed (#658)"
|
||||
]
|
||||
proof_obj: dict | None = None
|
||||
proof_parse_error: str | None = None
|
||||
if drain_proof_json:
|
||||
try:
|
||||
parsed = json.loads(drain_proof_json)
|
||||
proof_obj = parsed if isinstance(parsed, dict) else None
|
||||
if proof_obj is None:
|
||||
proof_parse_error = "drain_proof_json is not a JSON object"
|
||||
except (ValueError, TypeError) as exc:
|
||||
proof_parse_error = f"invalid drain_proof_json: {_redact(str(exc))}"
|
||||
|
||||
break_glass_authorized = bool(
|
||||
(
|
||||
os.environ.get("GITEA_BREAKGLASS_RESTART_AUTHORIZATION") or ""
|
||||
).strip()
|
||||
)
|
||||
break_glass = bool(request_break_glass and break_glass_authorized)
|
||||
|
||||
expected_fp = drain_proof.impact_fingerprint(report.as_dict())
|
||||
gate = drain_proof.gate_apply_restart(
|
||||
proof=proof_obj,
|
||||
break_glass=break_glass,
|
||||
expected_impact_fingerprint=expected_fp,
|
||||
requesting_session_id=sid,
|
||||
)
|
||||
gate_payload = gate.as_dict()
|
||||
if proof_parse_error and not break_glass:
|
||||
gate_payload["reasons"] = [proof_parse_error] + list(
|
||||
gate_payload.get("reasons") or []
|
||||
)
|
||||
payload["apply_gate"] = gate_payload
|
||||
payload["apply_authorized"] = gate.allow
|
||||
payload["break_glass_requested"] = bool(request_break_glass)
|
||||
payload["break_glass_authorized"] = break_glass_authorized
|
||||
# Even an authorized gate performs no restart here: execution is a later
|
||||
# child. The gate proves the apply path *would* be permitted.
|
||||
payload["reasons"] = list(payload.get("reasons") or []) + list(
|
||||
gate_payload.get("reasons") or []
|
||||
)
|
||||
if not gate.allow and gate.incident is not None:
|
||||
payload["incident"] = gate.incident
|
||||
return payload
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user