feat(restart): audit lifecycle events and durable incidents (#665)
Add restart_audit with mcp.restart.* event schema, redacted emission via gitea_audit, correlation ids, and incident materialization for failed drain and break-glass. Wire gitea_request_mcp_restart to always audit impact previews and fail closed on privileged apply when the audit sink is enabled but write fails. Closes #665
This commit is contained in:
@@ -0,0 +1,93 @@
|
||||
# MCP restart audit events and incidents (#665)
|
||||
|
||||
Restarts and recovery attempts leave a forensic trail. Failed drains and
|
||||
break-glass paths also raise durable Gitea incident issues so unsafe restarts
|
||||
cannot be silently repeated.
|
||||
|
||||
Parent umbrella: **#655**. Related: impact coordinator **#658**, drain proof
|
||||
**#661**, restart classes **#663**, break-glass **#664**, post-restart reconcile
|
||||
**#662**, vision **#652**, roadmap **#653**, console recovery **#642**.
|
||||
|
||||
## Components
|
||||
|
||||
| Piece | Where | Responsibility |
|
||||
|-------|-------|----------------|
|
||||
| Event schema + emission | `restart_audit.py` | `mcp.restart.*` vocabulary, redacted payload builder, append-only sink via `gitea_audit` |
|
||||
| Fail-closed privileged gate | `restart_audit.require_audit_or_deny` | When `GITEA_AUDIT_LOG` is set and the write fails, privileged apply is denied |
|
||||
| Incident descriptors | `restart_audit.build_incident_descriptor` | Durable follow-up issues (failed drain, break-glass, reconcile unresolved, unguarded) |
|
||||
| Materializer | `restart_audit.materialize_incident` | Injected `create_issue_fn` (network kept out of pure tests) |
|
||||
| Wiring | `gitea_request_mcp_restart` | Correlation id, impact-preview audit, apply-gate / break-glass audit + incident creation |
|
||||
|
||||
## Event vocabulary
|
||||
|
||||
| Event type | When |
|
||||
|------------|------|
|
||||
| `mcp.restart.impact_preview` | Every `gitea_request_mcp_restart` evaluation |
|
||||
| `mcp.restart.drain_enter` | Drain window starts (schema reserved; emit from drain path) |
|
||||
| `mcp.restart.drain_exit` | Drain window ends |
|
||||
| `mcp.restart.drain_proof` | Drain-proof verification result |
|
||||
| `mcp.restart.apply_gate` | Apply hard gate (`dry_run=False`) |
|
||||
| `mcp.restart.break_glass` | Authorized break-glass bypass |
|
||||
| `mcp.restart.post_restart_reconcile` | Post-restart reconcile outcome |
|
||||
| `mcp.restart.narrower_recovery` | Narrower recovery attempt recorded |
|
||||
| `mcp.restart.unguarded_detected` | Unguarded restart path detected |
|
||||
|
||||
All free text is redacted before sink write or issue body assembly. Emission
|
||||
never raises; callers decide fail-closed policy.
|
||||
|
||||
## Correlation
|
||||
|
||||
Each restart lifecycle mints a short `correlation_id` (`rst-` + 16 hex) shared
|
||||
across impact preview → apply gate → incident descriptors so operators can join
|
||||
the trail.
|
||||
|
||||
## Privileged deny-on-audit-fail
|
||||
|
||||
Rollout policy (issue #665):
|
||||
|
||||
1. Configure `GITEA_AUDIT_LOG` so writes land.
|
||||
2. Only then enforce deny when a privileged restart path cannot audit.
|
||||
|
||||
When audit is **not** configured, privileged apply still proceeds (no false
|
||||
denials during rollout). When audit **is** configured and the write fails,
|
||||
`apply_authorized` is cleared.
|
||||
|
||||
## Incidents
|
||||
|
||||
| Kind | Trigger |
|
||||
|------|---------|
|
||||
| `restart_failed_drain` | Apply denied by drain hard gate / failed proof |
|
||||
| `restart_break_glass` | Any authorized break-glass apply |
|
||||
| `restart_reconcile_unresolved` | Post-restart reconcile left work unresolved |
|
||||
| `restart_unguarded_detected` | Unguarded restart attempt detected |
|
||||
|
||||
Break-glass **always** creates an incident descriptor (and a Gitea issue when
|
||||
the create path is available). Failed drain does the same. Incident bodies
|
||||
include correlation id, session, class, scope, proof id, and redacted reasons.
|
||||
|
||||
Default labels: `mcp-health`, `safety`, `observability`, `status:ready`,
|
||||
`type:bug`, `workflow-hardening`.
|
||||
|
||||
## Tool payload surface
|
||||
|
||||
`gitea_request_mcp_restart` returns:
|
||||
|
||||
* `correlation_id` — lifecycle join key
|
||||
* `restart_audit.impact_preview_written` — sink success for the preview event
|
||||
* `restart_audit.apply_gate_written` — sink success for apply/break-glass (apply only)
|
||||
* `restart_audit.incident_result` — materialization outcome when an incident was required
|
||||
* `incident` — durable descriptor (when gate requires follow-up)
|
||||
|
||||
## Security
|
||||
|
||||
* No secrets in audit payloads or issue bodies.
|
||||
* This module never restarts a process.
|
||||
* Drain proof verification remains #661; audit only records the decision.
|
||||
* Incident creation failures are recorded in `incident_result.reasons` and never
|
||||
crash the restart evaluation path (audit write failure still fails closed for
|
||||
privileged apply when the sink is enabled).
|
||||
|
||||
## Tests
|
||||
|
||||
See `tests/test_restart_audit.py`: schema, redaction, emission, deny policy,
|
||||
incident materialization mocks, break-glass / failed-drain selection.
|
||||
@@ -33,6 +33,7 @@ recovery behavior for all nine classes.
|
||||
| `ControlPlaneDB.list_sessions` | `control_plane_db.py` | Read-only session inventory (the process-level unit a restart kills). |
|
||||
| `gitea_request_mcp_restart` | `gitea_mcp_server.py` | MCP tool: gathers inventory from the #613 DB, calls the coordinator, returns the report, and on `dry_run=False` runs the #661 drain-proof hard gate. Never restarts a process. |
|
||||
| `drain_proof.gate_apply_restart` | `drain_proof.py` | The #661 hard gate: verifies a drain proof against the current impact fingerprint, or records an authorized break-glass bypass. |
|
||||
| `restart_audit` | `restart_audit.py` | #665 forensic trail: `mcp.restart.*` events via `gitea_audit`, correlation ids, durable incidents for failed drain / break-glass. See [`mcp-restart-audit.md`](./mcp-restart-audit.md). |
|
||||
|
||||
## Dimensions evaluated
|
||||
|
||||
|
||||
Reference in New Issue
Block a user