sysadmin
22e0a41bd5
Add restart_audit with mcp.restart.* event schema, redacted emission via gitea_audit, correlation ids, and incident materialization for failed drain and break-glass. Wire gitea_request_mcp_restart to always audit impact previews and fail closed on privileged apply when the audit sink is enabled but write fails. Closes #665
94 lines
4.2 KiB
Markdown
94 lines
4.2 KiB
Markdown
# MCP restart audit events and incidents (#665)
|
|
|
|
Restarts and recovery attempts leave a forensic trail. Failed drains and
|
|
break-glass paths also raise durable Gitea incident issues so unsafe restarts
|
|
cannot be silently repeated.
|
|
|
|
Parent umbrella: **#655**. Related: impact coordinator **#658**, drain proof
|
|
**#661**, restart classes **#663**, break-glass **#664**, post-restart reconcile
|
|
**#662**, vision **#652**, roadmap **#653**, console recovery **#642**.
|
|
|
|
## Components
|
|
|
|
| Piece | Where | Responsibility |
|
|
|-------|-------|----------------|
|
|
| Event schema + emission | `restart_audit.py` | `mcp.restart.*` vocabulary, redacted payload builder, append-only sink via `gitea_audit` |
|
|
| Fail-closed privileged gate | `restart_audit.require_audit_or_deny` | When `GITEA_AUDIT_LOG` is set and the write fails, privileged apply is denied |
|
|
| Incident descriptors | `restart_audit.build_incident_descriptor` | Durable follow-up issues (failed drain, break-glass, reconcile unresolved, unguarded) |
|
|
| Materializer | `restart_audit.materialize_incident` | Injected `create_issue_fn` (network kept out of pure tests) |
|
|
| Wiring | `gitea_request_mcp_restart` | Correlation id, impact-preview audit, apply-gate / break-glass audit + incident creation |
|
|
|
|
## Event vocabulary
|
|
|
|
| Event type | When |
|
|
|------------|------|
|
|
| `mcp.restart.impact_preview` | Every `gitea_request_mcp_restart` evaluation |
|
|
| `mcp.restart.drain_enter` | Drain window starts (schema reserved; emit from drain path) |
|
|
| `mcp.restart.drain_exit` | Drain window ends |
|
|
| `mcp.restart.drain_proof` | Drain-proof verification result |
|
|
| `mcp.restart.apply_gate` | Apply hard gate (`dry_run=False`) |
|
|
| `mcp.restart.break_glass` | Authorized break-glass bypass |
|
|
| `mcp.restart.post_restart_reconcile` | Post-restart reconcile outcome |
|
|
| `mcp.restart.narrower_recovery` | Narrower recovery attempt recorded |
|
|
| `mcp.restart.unguarded_detected` | Unguarded restart path detected |
|
|
|
|
All free text is redacted before sink write or issue body assembly. Emission
|
|
never raises; callers decide fail-closed policy.
|
|
|
|
## Correlation
|
|
|
|
Each restart lifecycle mints a short `correlation_id` (`rst-` + 16 hex) shared
|
|
across impact preview → apply gate → incident descriptors so operators can join
|
|
the trail.
|
|
|
|
## Privileged deny-on-audit-fail
|
|
|
|
Rollout policy (issue #665):
|
|
|
|
1. Configure `GITEA_AUDIT_LOG` so writes land.
|
|
2. Only then enforce deny when a privileged restart path cannot audit.
|
|
|
|
When audit is **not** configured, privileged apply still proceeds (no false
|
|
denials during rollout). When audit **is** configured and the write fails,
|
|
`apply_authorized` is cleared.
|
|
|
|
## Incidents
|
|
|
|
| Kind | Trigger |
|
|
|------|---------|
|
|
| `restart_failed_drain` | Apply denied by drain hard gate / failed proof |
|
|
| `restart_break_glass` | Any authorized break-glass apply |
|
|
| `restart_reconcile_unresolved` | Post-restart reconcile left work unresolved |
|
|
| `restart_unguarded_detected` | Unguarded restart attempt detected |
|
|
|
|
Break-glass **always** creates an incident descriptor (and a Gitea issue when
|
|
the create path is available). Failed drain does the same. Incident bodies
|
|
include correlation id, session, class, scope, proof id, and redacted reasons.
|
|
|
|
Default labels: `mcp-health`, `safety`, `observability`, `status:ready`,
|
|
`type:bug`, `workflow-hardening`.
|
|
|
|
## Tool payload surface
|
|
|
|
`gitea_request_mcp_restart` returns:
|
|
|
|
* `correlation_id` — lifecycle join key
|
|
* `restart_audit.impact_preview_written` — sink success for the preview event
|
|
* `restart_audit.apply_gate_written` — sink success for apply/break-glass (apply only)
|
|
* `restart_audit.incident_result` — materialization outcome when an incident was required
|
|
* `incident` — durable descriptor (when gate requires follow-up)
|
|
|
|
## Security
|
|
|
|
* No secrets in audit payloads or issue bodies.
|
|
* This module never restarts a process.
|
|
* Drain proof verification remains #661; audit only records the decision.
|
|
* Incident creation failures are recorded in `incident_result.reasons` and never
|
|
crash the restart evaluation path (audit write failure still fails closed for
|
|
privileged apply when the sink is enabled).
|
|
|
|
## Tests
|
|
|
|
See `tests/test_restart_audit.py`: schema, redaction, emission, deny policy,
|
|
incident materialization mocks, break-glass / failed-drain selection.
|