feat(restart): audit lifecycle events and durable incidents (#665)

Add restart_audit with mcp.restart.* event schema, redacted emission via
gitea_audit, correlation ids, and incident materialization for failed drain
and break-glass. Wire gitea_request_mcp_restart to always audit impact
previews and fail closed on privileged apply when the audit sink is enabled
but write fails.

Closes #665
This commit is contained in:
2026-07-25 17:10:16 -04:00
parent 76f293eb28
commit 22e0a41bd5
5 changed files with 975 additions and 0 deletions
+93
View File
@@ -0,0 +1,93 @@
# MCP restart audit events and incidents (#665)
Restarts and recovery attempts leave a forensic trail. Failed drains and
break-glass paths also raise durable Gitea incident issues so unsafe restarts
cannot be silently repeated.
Parent umbrella: **#655**. Related: impact coordinator **#658**, drain proof
**#661**, restart classes **#663**, break-glass **#664**, post-restart reconcile
**#662**, vision **#652**, roadmap **#653**, console recovery **#642**.
## Components
| Piece | Where | Responsibility |
|-------|-------|----------------|
| Event schema + emission | `restart_audit.py` | `mcp.restart.*` vocabulary, redacted payload builder, append-only sink via `gitea_audit` |
| Fail-closed privileged gate | `restart_audit.require_audit_or_deny` | When `GITEA_AUDIT_LOG` is set and the write fails, privileged apply is denied |
| Incident descriptors | `restart_audit.build_incident_descriptor` | Durable follow-up issues (failed drain, break-glass, reconcile unresolved, unguarded) |
| Materializer | `restart_audit.materialize_incident` | Injected `create_issue_fn` (network kept out of pure tests) |
| Wiring | `gitea_request_mcp_restart` | Correlation id, impact-preview audit, apply-gate / break-glass audit + incident creation |
## Event vocabulary
| Event type | When |
|------------|------|
| `mcp.restart.impact_preview` | Every `gitea_request_mcp_restart` evaluation |
| `mcp.restart.drain_enter` | Drain window starts (schema reserved; emit from drain path) |
| `mcp.restart.drain_exit` | Drain window ends |
| `mcp.restart.drain_proof` | Drain-proof verification result |
| `mcp.restart.apply_gate` | Apply hard gate (`dry_run=False`) |
| `mcp.restart.break_glass` | Authorized break-glass bypass |
| `mcp.restart.post_restart_reconcile` | Post-restart reconcile outcome |
| `mcp.restart.narrower_recovery` | Narrower recovery attempt recorded |
| `mcp.restart.unguarded_detected` | Unguarded restart path detected |
All free text is redacted before sink write or issue body assembly. Emission
never raises; callers decide fail-closed policy.
## Correlation
Each restart lifecycle mints a short `correlation_id` (`rst-` + 16 hex) shared
across impact preview → apply gate → incident descriptors so operators can join
the trail.
## Privileged deny-on-audit-fail
Rollout policy (issue #665):
1. Configure `GITEA_AUDIT_LOG` so writes land.
2. Only then enforce deny when a privileged restart path cannot audit.
When audit is **not** configured, privileged apply still proceeds (no false
denials during rollout). When audit **is** configured and the write fails,
`apply_authorized` is cleared.
## Incidents
| Kind | Trigger |
|------|---------|
| `restart_failed_drain` | Apply denied by drain hard gate / failed proof |
| `restart_break_glass` | Any authorized break-glass apply |
| `restart_reconcile_unresolved` | Post-restart reconcile left work unresolved |
| `restart_unguarded_detected` | Unguarded restart attempt detected |
Break-glass **always** creates an incident descriptor (and a Gitea issue when
the create path is available). Failed drain does the same. Incident bodies
include correlation id, session, class, scope, proof id, and redacted reasons.
Default labels: `mcp-health`, `safety`, `observability`, `status:ready`,
`type:bug`, `workflow-hardening`.
## Tool payload surface
`gitea_request_mcp_restart` returns:
* `correlation_id` — lifecycle join key
* `restart_audit.impact_preview_written` — sink success for the preview event
* `restart_audit.apply_gate_written` — sink success for apply/break-glass (apply only)
* `restart_audit.incident_result` — materialization outcome when an incident was required
* `incident` — durable descriptor (when gate requires follow-up)
## Security
* No secrets in audit payloads or issue bodies.
* This module never restarts a process.
* Drain proof verification remains #661; audit only records the decision.
* Incident creation failures are recorded in `incident_result.reasons` and never
crash the restart evaluation path (audit write failure still fails closed for
privileged apply when the sink is enabled).
## Tests
See `tests/test_restart_audit.py`: schema, redaction, emission, deny policy,
incident materialization mocks, break-glass / failed-drain selection.