Add MCP restart audit events and incident creation #665

New Issue

2026-07-10T15:32:44-05:00

jcwalker3 commented

2026-07-10 15:32:44 -05:00

Problem statement

Restarts and recovery attempts are not consistently audited; failed drains do not always create durable incidents for follow-up.

Operational impact

No forensic trail; repeated unsafe restarts go unnoticed; #606/#607 observability incomplete for restart class.

Scope

Audit events for: impact preview, drain enter/exit, drain proof result, restart apply (by class), break-glass, post-restart reconcile, narrower recovery attempts.
Automatic Gitea incident issue when: failed drain, break-glass, reconcile unresolved, unguarded restart detected (#630).
Redacted payloads; correlation ids across coordinator steps.

Non-goals

Full Sentry product UI (#649).
Shipping metrics dashboards (#651).

Required implementation investigation

gitea_audit.py; #655 children; #606/#607; #633 audit model.

Proposed implementation direction

Structured event schema mcp.restart.*; append-only log + optional Gitea issue creator.

Security / workflow-safety

No secrets in audit; fail closed if audit write fails for privileged restarts.

Failure behavior

Audit failure → deny privileged restart.

Acceptance criteria

All restart lifecycle events audited.
Failed drain creates durable issue.
Break-glass always creates incident.
Tests for event emission + redaction.
Links #652 #653 #655.

Required tests

Event schema tests; incident creation mocks.

Observability and audit

This issue is the audit substrate.

Dependencies

Parent #655 · Soft-depends #658–#664 · Related #633 #606 · Vision #652 · Roadmap #653

Rollout

Enable audit before enforcing deny-on-audit-fail.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement restart audit events + incident creation
NEXT_PROMPT: Author restart audit under #655; PR; stop

Required proof

Events + tests + sample incident body.

Required final response and handoff

Brief PR → reviewer.

## Problem statement Restarts and recovery attempts are not consistently audited; failed drains do not always create durable incidents for follow-up. ## Operational impact No forensic trail; repeated unsafe restarts go unnoticed; #606/#607 observability incomplete for restart class. ## Scope * Audit events for: impact preview, drain enter/exit, drain proof result, restart apply (by class), break-glass, post-restart reconcile, narrower recovery attempts. * Automatic Gitea incident issue when: failed drain, break-glass, reconcile unresolved, unguarded restart detected (#630). * Redacted payloads; correlation ids across coordinator steps. ## Non-goals * Full Sentry product UI (#649). * Shipping metrics dashboards (#651). ## Required implementation investigation `gitea_audit.py`; #655 children; #606/#607; #633 audit model. ## Proposed implementation direction Structured event schema `mcp.restart.*`; append-only log + optional Gitea issue creator. ## Security / workflow-safety No secrets in audit; fail closed if audit write fails for privileged restarts. ## Failure behavior Audit failure → deny privileged restart. ## Acceptance criteria 1. All restart lifecycle events audited. 2. Failed drain creates durable issue. 3. Break-glass always creates incident. 4. Tests for event emission + redaction. 5. Links #652 #653 #655. ## Required tests Event schema tests; incident creation mocks. ## Observability and audit This issue *is* the audit substrate. ## Dependencies * Parent **#655** · Soft-depends **#658–#664** · Related #633 #606 · Vision **#652** · Roadmap **#653** ## Rollout Enable audit before enforcing deny-on-audit-fail. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement restart audit events + incident creation NEXT_PROMPT: Author restart audit under #655; PR; stop ``` ## Required proof Events + tests + sample incident body. ## Required final response and handoff Brief PR → reviewer.