Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e423dd5870 | ||
|
|
c67f39b40e | ||
|
|
4463a300ba | ||
|
|
75794609d1 | ||
|
|
da3294fbe5 | ||
|
|
c1ecadce8e |
@@ -1,93 +0,0 @@
|
||||
# MCP restart audit events and incidents (#665)
|
||||
|
||||
Restarts and recovery attempts leave a forensic trail. Failed drains and
|
||||
break-glass paths also raise durable Gitea incident issues so unsafe restarts
|
||||
cannot be silently repeated.
|
||||
|
||||
Parent umbrella: **#655**. Related: impact coordinator **#658**, drain proof
|
||||
**#661**, restart classes **#663**, break-glass **#664**, post-restart reconcile
|
||||
**#662**, vision **#652**, roadmap **#653**, console recovery **#642**.
|
||||
|
||||
## Components
|
||||
|
||||
| Piece | Where | Responsibility |
|
||||
|-------|-------|----------------|
|
||||
| Event schema + emission | `restart_audit.py` | `mcp.restart.*` vocabulary, redacted payload builder, append-only sink via `gitea_audit` |
|
||||
| Fail-closed privileged gate | `restart_audit.require_audit_or_deny` | When `GITEA_AUDIT_LOG` is set and the write fails, privileged apply is denied |
|
||||
| Incident descriptors | `restart_audit.build_incident_descriptor` | Durable follow-up issues (failed drain, break-glass, reconcile unresolved, unguarded) |
|
||||
| Materializer | `restart_audit.materialize_incident` | Injected `create_issue_fn` (network kept out of pure tests) |
|
||||
| Wiring | `gitea_request_mcp_restart` | Correlation id, impact-preview audit, apply-gate / break-glass audit + incident creation |
|
||||
|
||||
## Event vocabulary
|
||||
|
||||
| Event type | When |
|
||||
|------------|------|
|
||||
| `mcp.restart.impact_preview` | Every `gitea_request_mcp_restart` evaluation |
|
||||
| `mcp.restart.drain_enter` | Drain window starts (schema reserved; emit from drain path) |
|
||||
| `mcp.restart.drain_exit` | Drain window ends |
|
||||
| `mcp.restart.drain_proof` | Drain-proof verification result |
|
||||
| `mcp.restart.apply_gate` | Apply hard gate (`dry_run=False`) |
|
||||
| `mcp.restart.break_glass` | Authorized break-glass bypass |
|
||||
| `mcp.restart.post_restart_reconcile` | Post-restart reconcile outcome |
|
||||
| `mcp.restart.narrower_recovery` | Narrower recovery attempt recorded |
|
||||
| `mcp.restart.unguarded_detected` | Unguarded restart path detected |
|
||||
|
||||
All free text is redacted before sink write or issue body assembly. Emission
|
||||
never raises; callers decide fail-closed policy.
|
||||
|
||||
## Correlation
|
||||
|
||||
Each restart lifecycle mints a short `correlation_id` (`rst-` + 16 hex) shared
|
||||
across impact preview → apply gate → incident descriptors so operators can join
|
||||
the trail.
|
||||
|
||||
## Privileged deny-on-audit-fail
|
||||
|
||||
Rollout policy (issue #665):
|
||||
|
||||
1. Configure `GITEA_AUDIT_LOG` so writes land.
|
||||
2. Only then enforce deny when a privileged restart path cannot audit.
|
||||
|
||||
When audit is **not** configured, privileged apply still proceeds (no false
|
||||
denials during rollout). When audit **is** configured and the write fails,
|
||||
`apply_authorized` is cleared.
|
||||
|
||||
## Incidents
|
||||
|
||||
| Kind | Trigger |
|
||||
|------|---------|
|
||||
| `restart_failed_drain` | Apply denied by drain hard gate / failed proof |
|
||||
| `restart_break_glass` | Any authorized break-glass apply |
|
||||
| `restart_reconcile_unresolved` | Post-restart reconcile left work unresolved |
|
||||
| `restart_unguarded_detected` | Unguarded restart attempt detected |
|
||||
|
||||
Break-glass **always** creates an incident descriptor (and a Gitea issue when
|
||||
the create path is available). Failed drain does the same. Incident bodies
|
||||
include correlation id, session, class, scope, proof id, and redacted reasons.
|
||||
|
||||
Default labels: `mcp-health`, `safety`, `observability`, `status:ready`,
|
||||
`type:bug`, `workflow-hardening`.
|
||||
|
||||
## Tool payload surface
|
||||
|
||||
`gitea_request_mcp_restart` returns:
|
||||
|
||||
* `correlation_id` — lifecycle join key
|
||||
* `restart_audit.impact_preview_written` — sink success for the preview event
|
||||
* `restart_audit.apply_gate_written` — sink success for apply/break-glass (apply only)
|
||||
* `restart_audit.incident_result` — materialization outcome when an incident was required
|
||||
* `incident` — durable descriptor (when gate requires follow-up)
|
||||
|
||||
## Security
|
||||
|
||||
* No secrets in audit payloads or issue bodies.
|
||||
* This module never restarts a process.
|
||||
* Drain proof verification remains #661; audit only records the decision.
|
||||
* Incident creation failures are recorded in `incident_result.reasons` and never
|
||||
crash the restart evaluation path (audit write failure still fails closed for
|
||||
privileged apply when the sink is enabled).
|
||||
|
||||
## Tests
|
||||
|
||||
See `tests/test_restart_audit.py`: schema, redaction, emission, deny policy,
|
||||
incident materialization mocks, break-glass / failed-drain selection.
|
||||
@@ -33,7 +33,6 @@ recovery behavior for all nine classes.
|
||||
| `ControlPlaneDB.list_sessions` | `control_plane_db.py` | Read-only session inventory (the process-level unit a restart kills). |
|
||||
| `gitea_request_mcp_restart` | `gitea_mcp_server.py` | MCP tool: gathers inventory from the #613 DB, calls the coordinator, returns the report, and on `dry_run=False` runs the #661 drain-proof hard gate. Never restarts a process. |
|
||||
| `drain_proof.gate_apply_restart` | `drain_proof.py` | The #661 hard gate: verifies a drain proof against the current impact fingerprint, or records an authorized break-glass bypass. |
|
||||
| `restart_audit` | `restart_audit.py` | #665 forensic trail: `mcp.restart.*` events via `gitea_audit`, correlation ids, durable incidents for failed drain / break-glass. See [`mcp-restart-audit.md`](./mcp-restart-audit.md). |
|
||||
|
||||
## Dimensions evaluated
|
||||
|
||||
@@ -145,6 +144,19 @@ tool argument expresses caller intent and cannot be self-asserted by a worker
|
||||
session. `break_glass_requested` and `break_glass_authorized` are both reported,
|
||||
so a bypass is never silent.
|
||||
|
||||
### Break-glass Restart Workflow (`gitea_break_glass_restart`, #664)
|
||||
|
||||
The dedicated MCP tool `gitea_break_glass_restart` provides the privileged emergency break-glass restart workflow when graceful drain cannot complete:
|
||||
|
||||
- **Authorization (#664 AC1 / B1 / B13 / B15)**: Requires the exact trusted profile `prgs-controller` **and** explicit `runtime.break_glass_restart` and `gitea.issue.create` grants enforced by the real production operation gate (no `gitea.read` fallback). Incident creation is mandatory prior to execution (`gitea.issue.create`), so the deployable production policy for `prgs-controller` includes `allowed_operations`: `["gitea.read", "gitea.pr.close", "gitea.pr.comment", "gitea.issue.comment", "gitea.issue.create", "runtime.break_glass_restart", "gitea.branch.delete", "gitea.decision_lock.irrecoverable_recovery"]`. Ordinary roles, non-controller reconcilers, lookalike profile names (`fake-controller`, …), and env vars cannot authorize. A narrow break-glass capability does **not** redefine the profile's declared global role. Updating a live running `prgs-controller` profile in production requires an operator configuration update and daemon reload post-merge.
|
||||
- **Required Parameters (#664 AC2)**:
|
||||
- `reason`: Mandatory non-empty string (min 10 characters).
|
||||
- `confirmation`: Must equal exactly `"I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"`.
|
||||
- `impact_ack`: Must be `True`.
|
||||
- **Automatic Incident Creation (#664 AC3)**: Creates a Gitea incident issue (`[INCIDENT] [REQUESTED] Break-glass MCP restart invoked by ...`) detailing the reason, timestamp, disrupted sessions, and linking `#652 #653 #655 #630 #658 #662 #664`.
|
||||
- **Immutable Append-Only Audit Entry**: Records immutable pre-execution (REQUESTED) and post-execution (SUCCEEDED/FAILED) audit log entries with correlation identifiers.
|
||||
- **Mandatory Reconciliation (#664 AC4)**: Sets `reconciliation_required=True` requiring post-restart reconciliation via `gitea_reconcile_after_restart` (#662).
|
||||
|
||||
### Fail closed on apply
|
||||
|
||||
A missing, malformed, expired, unclean, tampered, or fingerprint-stale drain
|
||||
@@ -161,3 +173,4 @@ profiles are operational metadata only.
|
||||
|
||||
A representative dry-run report is in
|
||||
[`mcp-restart-impact-sample.json`](./mcp-restart-impact-sample.json).
|
||||
|
||||
|
||||
+99
-10
@@ -27,13 +27,73 @@ ALLOWED = "allowed"
|
||||
BLOCKED = "blocked"
|
||||
FAILED = "failed"
|
||||
SUCCEEDED = "succeeded"
|
||||
REQUESTED = "requested"
|
||||
ACCEPTED = "accepted"
|
||||
PENDING = "pending"
|
||||
|
||||
REDACTED = "[REDACTED]"
|
||||
|
||||
# A dict key containing any of these (case-insensitive) has its value redacted.
|
||||
_SECRET_KEY_HINTS = ("token", "password", "secret", "authorization", "auth")
|
||||
_SECRET_KEY_HINTS = (
|
||||
"token",
|
||||
"password",
|
||||
"passwd",
|
||||
"pwd",
|
||||
"secret",
|
||||
"authorization",
|
||||
"auth",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"access_key",
|
||||
"private_key",
|
||||
"client_secret",
|
||||
"credential",
|
||||
)
|
||||
# A string value starting with one of these has the following run redacted.
|
||||
_SECRET_VALUE_PREFIXES = ("token ", "Basic ", "Bearer ")
|
||||
# Space-terminated scheme prefixes only. Colon forms (``token:`` / ``api_key:``)
|
||||
# and ``Authorization: Bearer …`` are handled by ``_ASSIGNMENT_SECRET_PATTERN``
|
||||
# so policy/docs text that merely *names* a scheme is not itself flagged as a
|
||||
# live secret by console detectors.
|
||||
_SECRET_VALUE_PREFIXES = (
|
||||
"token ",
|
||||
"Basic ",
|
||||
"Bearer ",
|
||||
)
|
||||
|
||||
# Bare token-shaped values only — never a broad ``sec-`` prefix that erases
|
||||
# ordinary words (#664 B8 over-redaction).
|
||||
_BARE_SECRET_PATTERN = re.compile(
|
||||
r'(?i)\b(?:'
|
||||
r'ghp_[A-Za-z0-9_]{16,}'
|
||||
r'|gho_[A-Za-z0-9_]{16,}'
|
||||
r'|ghu_[A-Za-z0-9_]{16,}'
|
||||
r'|ghs_[A-Za-z0-9_]{16,}'
|
||||
r'|ghr_[A-Za-z0-9_]{16,}'
|
||||
r'|sk-live-[A-Za-z0-9_-]{16,}'
|
||||
r'|sk-proj-[A-Za-z0-9_-]{16,}'
|
||||
r'|sk-[A-Za-z0-9_-]{20,}'
|
||||
r'|glpat-[A-Za-z0-9_-]{16,}'
|
||||
r')\b'
|
||||
)
|
||||
|
||||
# Key/value credentials embedded in free text (password=..., api_key: ..., GITEA_TOKEN=..., etc.).
|
||||
# Group 1 captures the key name (e.g. GITEA_TOKEN, password, api_key).
|
||||
# Group 2 captures delimiter/whitespace (=, : ).
|
||||
# Group 3 captures the secret value, stopping at whitespace or non-secret delimiters (&, ;, ,, quotes, closing brackets).
|
||||
_ASSIGNMENT_SECRET_PATTERN = re.compile(
|
||||
r'(?i)\b([A-Za-z0-9_]*?(?:token|password|passwd|pwd|secret|api[_-]?key|access[_-]?key|'
|
||||
r'client[_-]?secret|private[_-]?key|authorization|credential))\b(\s*[:=]\s*)('
|
||||
r'"[^"]*"|\'[^\']*\'|'
|
||||
r'(?:Bearer|Basic|Token)\s+[^\s;&,"\'\)\}\]\>]+|'
|
||||
r'[^\s;&,"\'\)\}\]\>]+'
|
||||
r')'
|
||||
)
|
||||
|
||||
# Connection-string style credentials: Password=...; User ID=...; etc.
|
||||
_CONN_STRING_SECRET_PATTERN = re.compile(
|
||||
r'(?i)\b((?:password|pwd|user\s*id|uid|username|account)\s*=\s*)([^\s;\'"]+)'
|
||||
)
|
||||
|
||||
|
||||
# Known synthetic test-only domains/hostnames to preserve
|
||||
_SYNTHETIC_HOSTS = {
|
||||
@@ -59,7 +119,8 @@ def redact_urls(text: str) -> str:
|
||||
if not isinstance(text, str) or not text:
|
||||
return text
|
||||
|
||||
url_pattern = re.compile(r'(https?://[^\s)>\]}]+)', re.IGNORECASE)
|
||||
# Match any URI scheme (http, https, postgres, mysql, mongodb, redis, etc.)
|
||||
url_pattern = re.compile(r'([a-z0-9\+\.\-]+://[^\s)>\]}]+)', re.IGNORECASE)
|
||||
|
||||
def replace_url(match):
|
||||
url_str = match.group(1)
|
||||
@@ -73,11 +134,11 @@ def redact_urls(text: str) -> str:
|
||||
is_synthetic = True
|
||||
break
|
||||
|
||||
if is_synthetic:
|
||||
# Rebuild synthetic URL to redact any credentials or query secrets
|
||||
if is_synthetic or (parsed.username or parsed.password) or parsed.scheme.lower() not in ("http", "https"):
|
||||
# Rebuild URL to redact any credentials or query secrets
|
||||
new_netloc = parsed.netloc
|
||||
if parsed.username or parsed.password:
|
||||
netloc_clean = parsed.hostname
|
||||
netloc_clean = parsed.hostname or ""
|
||||
if parsed.port:
|
||||
netloc_clean = f"{netloc_clean}:{parsed.port}"
|
||||
new_netloc = f"[REDACTED_USER]:[REDACTED_PASS]@{netloc_clean}"
|
||||
@@ -114,23 +175,51 @@ def redact_urls(text: str) -> str:
|
||||
return out
|
||||
|
||||
|
||||
def _mask_assignment(match: re.Match) -> str:
|
||||
"""Keep the key and separator; replace only the secret value."""
|
||||
val = match.group(3)
|
||||
if val.startswith(REDACTED) or val.startswith("%5BREDACTED") or val.startswith("[REDACTED"):
|
||||
return f"{match.group(1)}{match.group(2)}{val}"
|
||||
return f"{match.group(1)}{match.group(2)}{REDACTED}"
|
||||
|
||||
|
||||
def _mask_conn_secret(match: re.Match) -> str:
|
||||
"""Keep the connection-string key; replace only the credential value."""
|
||||
val = match.group(2)
|
||||
if val.startswith(REDACTED) or val.startswith("%5BREDACTED") or val.startswith("[REDACTED"):
|
||||
return f"{match.group(1)}{val}"
|
||||
return f"{match.group(1)}{REDACTED}"
|
||||
|
||||
|
||||
def _redact_str(text):
|
||||
"""Redact anything that looks like an Authorization credential or raw URL in *text*."""
|
||||
"""Redact credentials, bare token shapes, and raw URLs in *text* (#664 B8).
|
||||
|
||||
Covers key/value credentials, authorization/bearer material, bare
|
||||
token-shaped values, connection-string credentials, and secrets embedded
|
||||
in larger sentences. Deliberately does **not** erase ordinary words that
|
||||
merely begin with a broad ``sec-`` prefix.
|
||||
"""
|
||||
if not isinstance(text, str) or not text:
|
||||
return text
|
||||
out = text
|
||||
out = redact_urls(text)
|
||||
out = _BARE_SECRET_PATTERN.sub(REDACTED, out)
|
||||
out = _ASSIGNMENT_SECRET_PATTERN.sub(_mask_assignment, out)
|
||||
out = _CONN_STRING_SECRET_PATTERN.sub(_mask_conn_secret, out)
|
||||
out_lower = out.lower()
|
||||
for prefix in _SECRET_VALUE_PREFIXES:
|
||||
prefix_lower = prefix.lower()
|
||||
idx = 0
|
||||
while True:
|
||||
i = out.find(prefix, idx)
|
||||
i = out_lower.find(prefix_lower, idx)
|
||||
if i == -1:
|
||||
break
|
||||
j = i + len(prefix)
|
||||
while j < len(out) and not out[j].isspace():
|
||||
j += 1
|
||||
out = out[:i] + prefix + REDACTED + out[j:]
|
||||
out_lower = out.lower()
|
||||
idx = i + len(prefix) + len(REDACTED)
|
||||
return redact_urls(out)
|
||||
return out
|
||||
|
||||
|
||||
def redact(value):
|
||||
|
||||
+32
-3
@@ -97,6 +97,26 @@ GITEA_OPERATION_ALIASES = {
|
||||
_REVIEW_MERGE_OPS = frozenset({"gitea.pr.approve", "gitea.pr.merge"})
|
||||
_AUTHOR_ONLY_OPS = frozenset({"gitea.pr.create", "gitea.branch.push"})
|
||||
|
||||
# First-class operation services that may appear in multi-service profile
|
||||
# allowlists. ``runtime.*`` is the control-plane capability namespace used by
|
||||
# non-Gitea MCP tools such as ``runtime.break_glass_restart`` (#664 B13).
|
||||
# Unknown foreign prefixes (e.g. ``jenkins.*`` under service=gitea) still fail
|
||||
# closed — they are not registered here.
|
||||
KNOWN_OPERATION_SERVICES = frozenset({"gitea", "runtime"})
|
||||
|
||||
|
||||
def service_for_operation(op, default="gitea"):
|
||||
"""Return the registered service prefix for a fully-qualified *op*.
|
||||
|
||||
Unqualified names and unknown prefixes fall back to *default* so callers
|
||||
keep the historical Gitea-centric gate behaviour.
|
||||
"""
|
||||
if isinstance(op, str) and "." in op:
|
||||
prefix = op.split(".", 1)[0]
|
||||
if prefix in KNOWN_OPERATION_SERVICES:
|
||||
return prefix
|
||||
return default
|
||||
|
||||
|
||||
def normalize_operation(op, service="gitea"):
|
||||
"""Return the canonical namespaced name for *op*, or fail closed (#106).
|
||||
@@ -133,6 +153,12 @@ def check_operation(op, allowed, forbidden=(), service="gitea"):
|
||||
Reasons: ``allowed``, ``invalid-operation``, ``invalid-forbidden-entry``,
|
||||
``forbidden``, ``no-allowed-operations``, ``not-allowed``.
|
||||
|
||||
Multi-service profile allowlists (#664 B13): each allow/forbid entry is
|
||||
normalized with its own registered service prefix (``gitea.*`` or
|
||||
``runtime.*``) so a gate defaulting to service=gitea can still enforce an
|
||||
exact ``runtime.break_glass_restart`` grant. Unknown / misspelled
|
||||
operations and foreign service prefixes remain fail-closed.
|
||||
|
||||
Fail-closed rules:
|
||||
- an *op* that cannot be normalized is denied (``invalid-operation``)
|
||||
- a forbidden entry that cannot be normalized denies the request
|
||||
@@ -143,14 +169,16 @@ def check_operation(op, allowed, forbidden=(), service="gitea"):
|
||||
- ``forbidden`` always overrides ``allowed``
|
||||
- an empty or missing allowed list denies everything
|
||||
"""
|
||||
op_service = service_for_operation(op, default=service)
|
||||
try:
|
||||
op_n = normalize_operation(op, service)
|
||||
op_n = normalize_operation(op, op_service)
|
||||
except ConfigError:
|
||||
return (False, "invalid-operation")
|
||||
forbidden_n = set()
|
||||
for entry in (forbidden or ()):
|
||||
try:
|
||||
forbidden_n.add(normalize_operation(entry, service))
|
||||
entry_service = service_for_operation(entry, default=service)
|
||||
forbidden_n.add(normalize_operation(entry, entry_service))
|
||||
except ConfigError:
|
||||
return (False, "invalid-forbidden-entry")
|
||||
if op_n in forbidden_n:
|
||||
@@ -160,7 +188,8 @@ def check_operation(op, allowed, forbidden=(), service="gitea"):
|
||||
allowed_n = set()
|
||||
for entry in allowed:
|
||||
try:
|
||||
allowed_n.add(normalize_operation(entry, service))
|
||||
entry_service = service_for_operation(entry, default=service)
|
||||
allowed_n.add(normalize_operation(entry, entry_service))
|
||||
except ConfigError:
|
||||
continue
|
||||
if op_n in allowed_n:
|
||||
|
||||
+690
-148
@@ -233,28 +233,50 @@ def _effective_workspace_role() -> str:
|
||||
)
|
||||
|
||||
|
||||
# Exact production profile → role mapping used only when no declared role is
|
||||
# present. Substring lookalikes (fake-controller, not-controller, …) never
|
||||
# match (#664 B1/B14).
|
||||
_EXACT_PROFILE_ROLE_NAMES = {
|
||||
"prgs-controller": "controller",
|
||||
"prgs-reconciler": "reconciler",
|
||||
"prgs-author": "author",
|
||||
"prgs-reviewer": "reviewer",
|
||||
"prgs-merger": "merger",
|
||||
"mdcps-author": "author",
|
||||
"mdcps-reviewer": "reviewer",
|
||||
"mdcps-merger": "merger",
|
||||
"controller": "controller",
|
||||
"reconciler": "reconciler",
|
||||
"author": "author",
|
||||
"reviewer": "reviewer",
|
||||
"merger": "merger",
|
||||
}
|
||||
|
||||
# Exact trusted profile that may hold break-glass (#664 B1). Capability
|
||||
# ``runtime.break_glass_restart`` is still required and enforced by the real
|
||||
# operation gate; this set only rejects lookalike profile names.
|
||||
TRUSTED_BREAK_GLASS_PROFILES = frozenset({"prgs-controller"})
|
||||
|
||||
|
||||
def _profile_role_kind(profile: dict) -> str:
|
||||
"""Resolve a profile's declared role before inferring from permissions.
|
||||
|
||||
Declared ``role`` / ``role_kind`` always wins so a controller profile is
|
||||
never reclassified as reconciler from permission inference (#840).
|
||||
never reclassified as reconciler from permission inference (#840). Exact
|
||||
match only — profile-name / role *substrings* never grant controller (or
|
||||
any) authority (#664 B1/B14). Lookalikes such as ``fake-controller``,
|
||||
``not-controller``, or ``xcontrollerx`` do not become controller.
|
||||
"""
|
||||
role = (profile.get("role") or profile.get("role_kind") or "").strip().lower()
|
||||
if role:
|
||||
# Normalize aliases / case.
|
||||
if "control" in role:
|
||||
# Exact aliases only; never ``"control" in role`` (matches
|
||||
# "not-controller" / "control-plane").
|
||||
if role in ("controller", "control"):
|
||||
return "controller"
|
||||
return role
|
||||
profile_name = (profile.get("profile_name") or "").strip().lower()
|
||||
for candidate in (
|
||||
"controller",
|
||||
"reconciler",
|
||||
"merger",
|
||||
"reviewer",
|
||||
"author",
|
||||
):
|
||||
if candidate in profile_name:
|
||||
return candidate
|
||||
if profile_name in _EXACT_PROFILE_ROLE_NAMES:
|
||||
return _EXACT_PROFILE_ROLE_NAMES[profile_name]
|
||||
return _role_kind(
|
||||
profile.get("allowed_operations") or [],
|
||||
profile.get("forbidden_operations") or [],
|
||||
@@ -2100,7 +2122,6 @@ import lease_policy # noqa: E402
|
||||
import workflow_dashboard # noqa: E402 # #605 live queue/lease dashboard
|
||||
import restart_coordinator # noqa: E402 # #658 MCP restart coordinator/impact
|
||||
import drain_proof # noqa: E402 # #661 pre-restart drain proof and hard gate
|
||||
import restart_audit # noqa: E402 # #665 restart audit events + incidents
|
||||
import incident_bridge # noqa: E402
|
||||
import sentry_observability # noqa: E402 (#606 optional Sentry observability)
|
||||
import sentry_incident_bridge # noqa: E402 (#607 Sentry→Gitea incident bridge)
|
||||
@@ -7122,35 +7143,17 @@ def terminal_review_hard_stop_reasons(
|
||||
]
|
||||
|
||||
|
||||
# Patterns scrubbed from any surfaced error text so a credential can never leak.
|
||||
_SECRET_PREFIXES = ("token ", "Basic ")
|
||||
|
||||
|
||||
def _redact(text: str) -> str:
|
||||
"""Strip anything that looks like an Authorization credential or raw URL from *text*.
|
||||
"""Strip credentials, bare token shapes, and raw URLs from *text* (#664 B8).
|
||||
|
||||
Errors raised by ``api_request`` echo the server response body, not the
|
||||
request headers, so a token should never appear — this is defence in depth
|
||||
so a future change can't leak ``token …`` / ``Basic …`` material into a
|
||||
tool result or log line.
|
||||
Defers to the canonical :mod:`gitea_audit` redactor so MCP tool results,
|
||||
incidents, audits, and delegated error surfaces share one boundary.
|
||||
"""
|
||||
if not text:
|
||||
return text
|
||||
out = text
|
||||
for prefix in _SECRET_PREFIXES:
|
||||
idx = 0
|
||||
while True:
|
||||
i = out.find(prefix, idx)
|
||||
if i == -1:
|
||||
break
|
||||
j = i + len(prefix)
|
||||
while j < len(out) and not out[j].isspace():
|
||||
j += 1
|
||||
out = out[:i] + prefix + "[REDACTED]" + out[j:]
|
||||
idx = i + len(prefix) + len("[REDACTED]")
|
||||
# Redact raw URLs, query secrets, hostnames, etc.
|
||||
import gitea_audit
|
||||
return gitea_audit.redact_urls(out)
|
||||
redacted = gitea_audit._redact_str(str(text))
|
||||
return redacted if isinstance(redacted, str) else str(redacted)
|
||||
|
||||
|
||||
# Review states that carry a submitted verdict. Gitea also emits PENDING
|
||||
@@ -23868,38 +23871,6 @@ def gitea_request_mcp_restart(
|
||||
# and a durable incident is raised. Break-glass is the only bypass and its
|
||||
# authorization is read from the environment, never self-asserted.
|
||||
payload["apply_supported"] = False
|
||||
# #665: correlation id threads impact preview → apply gate → incidents.
|
||||
correlation_id = restart_audit.new_correlation_id()
|
||||
payload["correlation_id"] = correlation_id
|
||||
auth_user = None
|
||||
try:
|
||||
# remote-only: host override is for operator diagnostics, not required here
|
||||
auth_user = (gitea_whoami(remote=remote) or {}).get("username")
|
||||
except Exception: # noqa: BLE001 — identity is best-effort for audit
|
||||
auth_user = None
|
||||
preview_audit = restart_audit.record_restart_lifecycle(
|
||||
event_type=restart_audit.EVENT_IMPACT_PREVIEW,
|
||||
outcome=str(report.verdict or "unknown"),
|
||||
correlation_id=correlation_id,
|
||||
remote=remote,
|
||||
org=o,
|
||||
repo=r,
|
||||
requesting_session_id=sid,
|
||||
restart_class=restart_class,
|
||||
profile_name=profile_name,
|
||||
authenticated_username=auth_user,
|
||||
reasons=list(report.reasons or []),
|
||||
details={
|
||||
"dry_run": True,
|
||||
"allow_restart": bool(report.allow_restart),
|
||||
"inventory_complete": inventory_complete,
|
||||
},
|
||||
privileged=False,
|
||||
)
|
||||
payload["restart_audit"] = {
|
||||
"correlation_id": correlation_id,
|
||||
"impact_preview_written": preview_audit["audit_written"],
|
||||
}
|
||||
if not dry_run:
|
||||
proof_obj: dict | None = None
|
||||
proof_parse_error: str | None = None
|
||||
@@ -23952,89 +23923,660 @@ def gitea_request_mcp_restart(
|
||||
)
|
||||
if not gate.allow and gate.incident is not None:
|
||||
payload["incident"] = gate.incident
|
||||
|
||||
# #665: audit apply-gate + materialize durable incidents for failed
|
||||
# drain and break-glass. Privileged apply denies if audit is enabled
|
||||
# and the sink write fails.
|
||||
incident_desc = restart_audit.incident_from_apply_gate(
|
||||
gate_payload={
|
||||
**gate_payload,
|
||||
"incident": gate.incident,
|
||||
"allow": gate.allow,
|
||||
},
|
||||
break_glass=break_glass,
|
||||
correlation_id=correlation_id,
|
||||
requesting_session_id=sid,
|
||||
restart_class=restart_class,
|
||||
remote=remote,
|
||||
org=o,
|
||||
repo=r,
|
||||
)
|
||||
if incident_desc is not None:
|
||||
payload["incident"] = incident_desc
|
||||
|
||||
def _create_restart_incident_issue(
|
||||
*, title, body, labels, org=None, repo=None, **_kw
|
||||
):
|
||||
return gitea_create_issue(
|
||||
title=title,
|
||||
body=body,
|
||||
labels=labels,
|
||||
remote=remote,
|
||||
host=h,
|
||||
org=org or o,
|
||||
repo=repo or r,
|
||||
)
|
||||
|
||||
apply_audit = restart_audit.record_restart_lifecycle(
|
||||
event_type=(
|
||||
restart_audit.EVENT_BREAK_GLASS
|
||||
if break_glass
|
||||
else restart_audit.EVENT_APPLY_GATE
|
||||
),
|
||||
outcome=(
|
||||
"break_glass"
|
||||
if break_glass
|
||||
else ("allow" if payload["apply_authorized"] else "deny")
|
||||
),
|
||||
correlation_id=correlation_id,
|
||||
remote=remote,
|
||||
org=o,
|
||||
repo=r,
|
||||
requesting_session_id=sid,
|
||||
restart_class=restart_class,
|
||||
profile_name=profile_name,
|
||||
authenticated_username=auth_user,
|
||||
reasons=list(gate_payload.get("reasons") or []),
|
||||
details={
|
||||
"apply_authorized": payload["apply_authorized"],
|
||||
"drain_gate_allow": gate_payload.get("drain_gate_allow"),
|
||||
"restart_class_authorized": restart_class_authorized,
|
||||
"break_glass": break_glass,
|
||||
"proof_id": gate_payload.get("proof_id"),
|
||||
},
|
||||
privileged=True,
|
||||
create_incident=incident_desc,
|
||||
create_issue_fn=_create_restart_incident_issue
|
||||
if incident_desc is not None
|
||||
else None,
|
||||
dry_run_incident=False,
|
||||
)
|
||||
payload["restart_audit"] = {
|
||||
"correlation_id": correlation_id,
|
||||
"impact_preview_written": preview_audit["audit_written"],
|
||||
"apply_gate_written": apply_audit["audit_written"],
|
||||
"incident_result": apply_audit.get("incident_result"),
|
||||
}
|
||||
if apply_audit["deny_reasons"]:
|
||||
payload["apply_authorized"] = False
|
||||
payload["reasons"] = list(payload.get("reasons") or []) + list(
|
||||
apply_audit["deny_reasons"]
|
||||
)
|
||||
payload["success"] = True
|
||||
return payload
|
||||
|
||||
|
||||
BREAK_GLASS_CONFIRMATION_PHRASE = "I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"
|
||||
|
||||
# Test-injectable canonical break-glass executor/delegate (#664 B6/B11).
|
||||
# Production default never signals the live MCP cohort.
|
||||
_break_glass_restart_executor = None
|
||||
|
||||
|
||||
def _default_break_glass_restart_executor(request: dict) -> dict:
|
||||
"""Canonical non-dry-run break-glass executor/delegate (#664 B6/B11).
|
||||
|
||||
Never kills, signals, or restarts the live MCP cohort in-process.
|
||||
An environment string alone (``GITEA_SANCTIONED_RESTART_HOOK``) does not
|
||||
prove restart execution without an active confirmed delegate handoff.
|
||||
"""
|
||||
hook = (os.environ.get("GITEA_SANCTIONED_RESTART_HOOK") or "").strip()
|
||||
if hook:
|
||||
return {
|
||||
"success": False,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": True,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"execution_mode": "accepted_not_executed",
|
||||
"host_hook_configured": True,
|
||||
"reasons": [
|
||||
"sanctioned host restart hook reference is configured, but environment text "
|
||||
"cannot prove execution without a confirmed delegate handoff (fail closed) (#664 B6/B11)"
|
||||
],
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"apply_supported": False,
|
||||
"apply_authorized": False,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"execution_mode": "unsupported",
|
||||
"reasons": [
|
||||
"break-glass apply is unsupported: no sanctioned host restart "
|
||||
"executor is configured (#664)"
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _run_break_glass_restart_executor(request: dict) -> dict:
|
||||
"""Invoke the installed or default break-glass executor (test-injectable)."""
|
||||
executor = _break_glass_restart_executor or _default_break_glass_restart_executor
|
||||
result = executor(request)
|
||||
if not isinstance(result, dict):
|
||||
return {
|
||||
"success": False,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": False,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": ["break-glass executor returned a non-dict result (fail closed)"],
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_break_glass_restart(
|
||||
reason: str,
|
||||
confirmation: str,
|
||||
impact_ack: bool = False,
|
||||
restart_class: str = "full_mcp_restart",
|
||||
create_incident_issue: bool = True,
|
||||
dry_run: bool = False,
|
||||
remote: str = "dadeschools",
|
||||
host: str | None = None,
|
||||
org: str | None = None,
|
||||
repo: str | None = None,
|
||||
worktree_path: str | None = None,
|
||||
) -> dict:
|
||||
"""Privileged emergency break-glass MCP restart workflow (#664).
|
||||
|
||||
Break-glass restart permits emergency recovery when graceful drain cannot
|
||||
complete. It requires:
|
||||
1. Exact ``runtime.break_glass_restart`` capability on the trusted
|
||||
``prgs-controller`` profile (ordinary roles, lookalike names, env vars,
|
||||
and non-controller reconcilers fail closed).
|
||||
2. Explicit non-empty reason (minimum 10 characters).
|
||||
3. Exact confirmation string matching 'I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION'.
|
||||
4. Mandatory impact acknowledgement (impact_ack=True).
|
||||
5. Immutable append-only audit entry recorded prior to execution and after terminal completion.
|
||||
6. Automatic incident record created on Gitea prior to execution.
|
||||
7. Truthful execution reporting via the canonical executor/delegate and
|
||||
mandatory post-restart reconciliation (#662).
|
||||
"""
|
||||
# B13: real production gate for the exact canonical operation — never
|
||||
# fall back to gitea.read and never stub this gate in production.
|
||||
capability_block = _profile_operation_gate("runtime.break_glass_restart")
|
||||
if capability_block:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": capability_block,
|
||||
"permission_report": _permission_block_report("runtime.break_glass_restart"),
|
||||
"blocker_kind": "permission_denied",
|
||||
})
|
||||
|
||||
h, o, r = _resolve(remote, host, org, repo)
|
||||
profile = get_profile()
|
||||
active_role = _profile_role_kind(profile)
|
||||
profile_name = (
|
||||
(profile.get("profile_name") or profile.get("execution_profile") or "")
|
||||
.strip()
|
||||
)
|
||||
break_glass_env_auth = bool(
|
||||
(os.environ.get("GITEA_BREAKGLASS_RESTART_AUTHORIZATION") or "").strip()
|
||||
)
|
||||
# Env is never an authorization channel for this endpoint (B2/B1).
|
||||
_ = break_glass_env_auth
|
||||
|
||||
# B1: exact trusted profile only — not role substring, not lookalike names.
|
||||
if profile_name not in TRUSTED_BREAK_GLASS_PROFILES:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"active_role": active_role,
|
||||
"profile_name": profile_name or None,
|
||||
"reasons": [
|
||||
f"profile '{profile_name or '(unset)'}' is not the trusted "
|
||||
f"break-glass profile {sorted(TRUSTED_BREAK_GLASS_PROFILES)}; "
|
||||
"lookalike names, ordinary roles, env vars, and non-controller "
|
||||
"reconcilers cannot authorize break-glass (#664 AC1/B1)"
|
||||
],
|
||||
"blocker_kind": "role_authorization",
|
||||
})
|
||||
|
||||
# 2. Required fields and redaction (B8)
|
||||
raw_reason = (reason or "").strip()
|
||||
clean_reason = _redact(raw_reason)
|
||||
if not raw_reason or len(raw_reason) < 10:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"reason is required and must be at least 10 characters long (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "missing_required_fields",
|
||||
})
|
||||
|
||||
clean_confirmation = (confirmation or "").strip()
|
||||
if clean_confirmation != BREAK_GLASS_CONFIRMATION_PHRASE:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
f"confirmation string mismatch; must equal exactly '{BREAK_GLASS_CONFIRMATION_PHRASE}' (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "confirmation_mismatch",
|
||||
})
|
||||
|
||||
if not impact_ack:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"impact_ack must be True to acknowledge disruption of in-flight sessions (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "impact_ack_required",
|
||||
})
|
||||
|
||||
# Gate incident issue creation on gitea.issue.create permission (B3)
|
||||
if create_incident_issue:
|
||||
create_block = _profile_operation_gate("gitea.issue.create")
|
||||
if create_block:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": create_block,
|
||||
"permission_report": _permission_block_report("gitea.issue.create"),
|
||||
"blocker_kind": "permission_denied",
|
||||
})
|
||||
|
||||
# Evaluate impact / disrupted sessions
|
||||
impact_result = gitea_request_mcp_restart(
|
||||
remote=remote,
|
||||
host=host,
|
||||
org=org,
|
||||
repo=repo,
|
||||
dry_run=True,
|
||||
restart_class=restart_class,
|
||||
request_break_glass=True,
|
||||
)
|
||||
disrupted_sessions = list(impact_result.get("affected_sessions") or [])
|
||||
disrupted_count = len(disrupted_sessions)
|
||||
|
||||
identity = _authenticated_username(h) or profile.get("username") or "unknown"
|
||||
now_iso = datetime.now(timezone.utc).isoformat()
|
||||
ns_ctx = _resolve_namespace_mutation_context(worktree_path)
|
||||
mcp_namespace = ns_ctx.get("mcp_namespace") or profile.get("profile_name") or "gitea-controller"
|
||||
correlation_id = f"bg-{uuid.uuid4().hex[:12]}"
|
||||
|
||||
audit_payload = gitea_audit.redact({
|
||||
"event": "break_glass_mcp_restart",
|
||||
"correlation_id": correlation_id,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"timestamp": now_iso,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"restart_class": restart_class,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": [
|
||||
s.get("session_id") if isinstance(s, dict) else str(s)
|
||||
for s in disrupted_sessions
|
||||
],
|
||||
"dry_run": dry_run,
|
||||
"remote": remote,
|
||||
"org": o,
|
||||
"repo": r,
|
||||
"env_auth_present": break_glass_env_auth,
|
||||
})
|
||||
|
||||
# Dry-run handling (B7: no durable mutation)
|
||||
if dry_run:
|
||||
return gitea_audit.redact({
|
||||
"success": True,
|
||||
"performed": False,
|
||||
"dry_run": True,
|
||||
"break_glass_executed": False,
|
||||
"would_execute": True,
|
||||
"correlation_id": correlation_id,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": disrupted_sessions,
|
||||
"audit_record": audit_payload,
|
||||
"saved_audit": None,
|
||||
"incident_issue": None,
|
||||
"reconciliation_required": True,
|
||||
"reconciliation_tool": "gitea_reconcile_after_restart",
|
||||
"follow_up_issue_required": True,
|
||||
"cross_references": ["#652", "#653", "#655", "#630", "#658", "#662", "#664"],
|
||||
"reasons": ["break-glass restart dry-run evaluated successfully"],
|
||||
})
|
||||
|
||||
# Fail closed if create_incident_issue is False on real execution (B5)
|
||||
if not create_incident_issue:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"create_incident_issue=False is forbidden on real break-glass execution; "
|
||||
"pre-execution incident creation is mandatory (#664 AC3)"
|
||||
],
|
||||
"blocker_kind": "incident_creation_required",
|
||||
})
|
||||
|
||||
# B9: Fail closed if audit backend is disabled
|
||||
if not gitea_audit.audit_enabled():
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"audit recording is disabled or unavailable; break-glass restart requires an enabled audit backend (#664 AC3)"
|
||||
],
|
||||
"blocker_kind": "audit_recording_failed",
|
||||
})
|
||||
|
||||
# Pre-execution recording: Audit record in REQUESTED state (B4, B10)
|
||||
pre_audit_event = gitea_audit.build_event(
|
||||
action="break_glass_mcp_restart_requested",
|
||||
result=gitea_audit.REQUESTED,
|
||||
remote=remote,
|
||||
server=(gitea_url(h, "").rstrip("/") if h else None),
|
||||
repository=r,
|
||||
profile_name=profile.get("profile_name", "unknown"),
|
||||
audit_label=profile.get("audit_label", "unknown"),
|
||||
authenticated_username=identity,
|
||||
reason=clean_reason,
|
||||
mcp_namespace=mcp_namespace,
|
||||
task_role=active_role,
|
||||
operation="break_glass_mcp_restart",
|
||||
now=now_iso,
|
||||
request_metadata={
|
||||
"correlation_id": correlation_id,
|
||||
"confirmation": clean_confirmation,
|
||||
"restart_class": restart_class,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": [
|
||||
s.get("session_id") if isinstance(s, dict) else str(s)
|
||||
for s in disrupted_sessions
|
||||
],
|
||||
},
|
||||
)
|
||||
audit_write_success = gitea_audit.write_event(pre_audit_event)
|
||||
|
||||
if not audit_write_success:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"failed to persist required pre-execution audit event (#664 AC3)"
|
||||
],
|
||||
"blocker_kind": "audit_recording_failed",
|
||||
})
|
||||
|
||||
# Pre-execution recording: Gitea Incident Issue (B5, B8)
|
||||
issue_title = _redact(f"[INCIDENT] [REQUESTED] Break-glass MCP restart invoked by {identity} ({correlation_id})")
|
||||
issue_body = _redact(
|
||||
f"## Break-glass MCP restart incident report (#664)\n\n"
|
||||
f"- **Correlation ID**: `{correlation_id}`\n"
|
||||
f"- **Invoked by**: `{identity}` (role: `{active_role}`, namespace: `{mcp_namespace}`)\n"
|
||||
f"- **Timestamp**: `{now_iso}`\n"
|
||||
f"- **Reason**: {clean_reason}\n"
|
||||
f"- **Confirmation**: `{clean_confirmation}`\n"
|
||||
f"- **Disrupted Sessions Count**: `{disrupted_count}`\n\n"
|
||||
f"### Mandatory Post-Restart Reconciliation (#662)\n"
|
||||
f"Post-restart reconciliation must be executed via `gitea_reconcile_after_restart` "
|
||||
f"to clean up orphaned leases, inspect worktree integrity, and handle disrupted work.\n\n"
|
||||
f"### Cross-references\n"
|
||||
f"Ref #652 #653 #655 #630 #658 #662 #664\n"
|
||||
)
|
||||
incident_issue_result = None
|
||||
try:
|
||||
incident_issue_result = api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues",
|
||||
_auth(h),
|
||||
{
|
||||
"title": issue_title,
|
||||
"body": issue_body,
|
||||
"labels": ["incident", "mcp-health", "break-glass"],
|
||||
},
|
||||
)
|
||||
if not isinstance(incident_issue_result, dict) or "number" not in incident_issue_result:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
f"incident issue creation failed (#664 AC3): {_redact(str(incident_issue_result))}"
|
||||
],
|
||||
"blocker_kind": "incident_creation_failed",
|
||||
"incident_issue": incident_issue_result,
|
||||
})
|
||||
except Exception as exc:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
f"incident issue creation failed with exception (#664 AC3): {_redact(str(exc))}"
|
||||
],
|
||||
"blocker_kind": "incident_creation_failed",
|
||||
"incident_issue": {"error": _redact(str(exc))},
|
||||
})
|
||||
|
||||
incident_number = incident_issue_result.get("number")
|
||||
|
||||
# B6/B11: reach the canonical non-dry-run executor/delegate.
|
||||
# Authorization and apply_authorized do not mean execution occurred.
|
||||
# Dry-run never reaches this path (returned earlier).
|
||||
restart_exec_result = _run_break_glass_restart_executor({
|
||||
"remote": remote,
|
||||
"host": host,
|
||||
"org": o,
|
||||
"repo": r,
|
||||
"restart_class": restart_class,
|
||||
"correlation_id": correlation_id,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"profile_name": profile_name,
|
||||
"active_role": active_role,
|
||||
"incident_number": incident_number,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"request_break_glass": True,
|
||||
"dry_run": False,
|
||||
})
|
||||
|
||||
apply_supported = bool(restart_exec_result.get("apply_supported", False))
|
||||
apply_authorized = bool(restart_exec_result.get("apply_authorized", False))
|
||||
exec_success = bool(restart_exec_result.get("success", False))
|
||||
# break_glass_executed is true only when the executor contract proves
|
||||
# execution (or accepted host delegation) occurred.
|
||||
restart_performed = bool(
|
||||
restart_exec_result.get("restart_performed", False)
|
||||
and restart_exec_result.get("break_glass_executed", False)
|
||||
)
|
||||
|
||||
if not apply_supported:
|
||||
term_audit = gitea_audit.build_event(
|
||||
action="break_glass_mcp_restart_terminal",
|
||||
result=gitea_audit.FAILED,
|
||||
remote=remote,
|
||||
server=(gitea_url(h, "").rstrip("/") if h else None),
|
||||
repository=r,
|
||||
profile_name=profile.get("profile_name", "unknown"),
|
||||
audit_label=profile.get("audit_label", "unknown"),
|
||||
authenticated_username=identity,
|
||||
reason="apply_unsupported",
|
||||
mcp_namespace=mcp_namespace,
|
||||
task_role=active_role,
|
||||
operation="break_glass_mcp_restart",
|
||||
now=datetime.now(timezone.utc).isoformat(),
|
||||
request_metadata={
|
||||
"correlation_id": correlation_id,
|
||||
"incident_number": incident_number,
|
||||
"break_glass_executed": False,
|
||||
"blocker_kind": "apply_unsupported",
|
||||
},
|
||||
)
|
||||
gitea_audit.write_event(term_audit)
|
||||
if incident_number:
|
||||
try:
|
||||
api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues/{incident_number}/comments",
|
||||
_auth(h),
|
||||
{"body": _redact(f"**Terminal Status**: `apply_unsupported` - Restart coordinator does not support apply execution. No restart was performed. ({correlation_id})")},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reasons": [
|
||||
"break-glass apply is unsupported by restart coordinator (apply_supported=False) (#664)",
|
||||
*(restart_exec_result.get("reasons") or []),
|
||||
],
|
||||
"blocker_kind": "apply_unsupported",
|
||||
"restart_result": restart_exec_result,
|
||||
"incident_issue": incident_issue_result,
|
||||
"audit_record": pre_audit_event,
|
||||
})
|
||||
|
||||
if not apply_authorized or not exec_success or not restart_performed:
|
||||
term_audit = gitea_audit.build_event(
|
||||
action="break_glass_mcp_restart_terminal",
|
||||
result=gitea_audit.FAILED,
|
||||
remote=remote,
|
||||
server=(gitea_url(h, "").rstrip("/") if h else None),
|
||||
repository=r,
|
||||
profile_name=profile.get("profile_name", "unknown"),
|
||||
audit_label=profile.get("audit_label", "unknown"),
|
||||
authenticated_username=identity,
|
||||
reason="restart_delegation_failed",
|
||||
mcp_namespace=mcp_namespace,
|
||||
task_role=active_role,
|
||||
operation="break_glass_mcp_restart",
|
||||
now=datetime.now(timezone.utc).isoformat(),
|
||||
request_metadata={
|
||||
"correlation_id": correlation_id,
|
||||
"incident_number": incident_number,
|
||||
"break_glass_executed": False,
|
||||
"blocker_kind": "restart_delegation_failed",
|
||||
},
|
||||
)
|
||||
gitea_audit.write_event(term_audit)
|
||||
if incident_number:
|
||||
try:
|
||||
api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues/{incident_number}/comments",
|
||||
_auth(h),
|
||||
{"body": _redact(f"**Terminal Status**: `restart_delegation_failed` - Restart execution failed or was denied. No restart was performed. ({correlation_id})")},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reasons": [
|
||||
"delegated restart execution failed or was denied by coordinator (#664)",
|
||||
*(restart_exec_result.get("reasons") or []),
|
||||
],
|
||||
"blocker_kind": "restart_delegation_failed",
|
||||
"restart_result": restart_exec_result,
|
||||
"incident_issue": incident_issue_result,
|
||||
"audit_record": pre_audit_event,
|
||||
})
|
||||
|
||||
# Restart occurred! Perform mandatory post-restart reconciliation (B10)
|
||||
recon_result = None
|
||||
try:
|
||||
recon_result = gitea_reconcile_after_restart(
|
||||
remote=remote,
|
||||
host=host,
|
||||
org=org,
|
||||
repo=repo,
|
||||
)
|
||||
except Exception as exc:
|
||||
recon_result = {"success": False, "error": _redact(str(exc))}
|
||||
|
||||
recon_success = bool(recon_result and isinstance(recon_result, dict) and recon_result.get("success", False))
|
||||
|
||||
if not recon_success:
|
||||
term_audit = gitea_audit.build_event(
|
||||
action="break_glass_mcp_restart_terminal",
|
||||
result=gitea_audit.FAILED,
|
||||
remote=remote,
|
||||
server=(gitea_url(h, "").rstrip("/") if h else None),
|
||||
repository=r,
|
||||
profile_name=profile.get("profile_name", "unknown"),
|
||||
audit_label=profile.get("audit_label", "unknown"),
|
||||
authenticated_username=identity,
|
||||
reason="reconciliation_failed",
|
||||
mcp_namespace=mcp_namespace,
|
||||
task_role=active_role,
|
||||
operation="break_glass_mcp_restart",
|
||||
now=datetime.now(timezone.utc).isoformat(),
|
||||
request_metadata={
|
||||
"correlation_id": correlation_id,
|
||||
"incident_number": incident_number,
|
||||
"break_glass_executed": True,
|
||||
"reconciliation_success": False,
|
||||
"blocker_kind": "reconciliation_failed",
|
||||
},
|
||||
)
|
||||
gitea_audit.write_event(term_audit)
|
||||
if incident_number:
|
||||
try:
|
||||
api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues/{incident_number}/comments",
|
||||
_auth(h),
|
||||
{"body": _redact(f"**Terminal Status**: `reconciliation_failed` - Restart was executed but post-restart reconciliation failed. ({correlation_id})")},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": True,
|
||||
"break_glass_executed": True,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reasons": [
|
||||
"break-glass restart executed but post-restart reconciliation failed (#664/#662)"
|
||||
],
|
||||
"blocker_kind": "reconciliation_failed",
|
||||
"restart_result": restart_exec_result,
|
||||
"reconciliation_result": recon_result,
|
||||
"incident_issue": incident_issue_result,
|
||||
"audit_record": pre_audit_event,
|
||||
})
|
||||
|
||||
# Terminal audit append for successful execution + reconciliation
|
||||
term_audit = gitea_audit.build_event(
|
||||
action="break_glass_mcp_restart_terminal",
|
||||
result=gitea_audit.SUCCEEDED,
|
||||
remote=remote,
|
||||
server=(gitea_url(h, "").rstrip("/") if h else None),
|
||||
repository=r,
|
||||
profile_name=profile.get("profile_name", "unknown"),
|
||||
audit_label=profile.get("audit_label", "unknown"),
|
||||
authenticated_username=identity,
|
||||
reason=clean_reason,
|
||||
mcp_namespace=mcp_namespace,
|
||||
task_role=active_role,
|
||||
operation="break_glass_mcp_restart",
|
||||
now=datetime.now(timezone.utc).isoformat(),
|
||||
request_metadata={
|
||||
"correlation_id": correlation_id,
|
||||
"incident_number": incident_number,
|
||||
"break_glass_executed": True,
|
||||
"reconciliation_success": True,
|
||||
},
|
||||
)
|
||||
term_write_success = gitea_audit.write_event(term_audit)
|
||||
|
||||
if incident_number:
|
||||
try:
|
||||
api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues/{incident_number}/comments",
|
||||
_auth(h),
|
||||
{"body": _redact(f"**Terminal Status**: `succeeded` - Break-glass restart executed and reconciled successfully ({correlation_id}).")},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not term_write_success:
|
||||
return gitea_audit.redact({
|
||||
"success": False,
|
||||
"performed": True,
|
||||
"break_glass_executed": True,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reasons": [
|
||||
"break-glass restart executed and reconciled but terminal audit recording failed (#664)"
|
||||
],
|
||||
"blocker_kind": "terminal_audit_failed",
|
||||
"restart_result": restart_exec_result,
|
||||
"reconciliation_result": recon_result,
|
||||
"incident_issue": incident_issue_result,
|
||||
"audit_record": pre_audit_event,
|
||||
})
|
||||
|
||||
return gitea_audit.redact({
|
||||
"success": True,
|
||||
"performed": True,
|
||||
"dry_run": False,
|
||||
"break_glass_executed": True,
|
||||
"would_execute": True,
|
||||
"correlation_id": correlation_id,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"mcp_namespace": mcp_namespace,
|
||||
"restart_class": restart_class,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": disrupted_sessions,
|
||||
"audit_record": term_audit,
|
||||
"saved_audit": term_audit,
|
||||
"incident_issue": incident_issue_result,
|
||||
"reconciliation_result": recon_result,
|
||||
"reconciliation_required": True,
|
||||
"reconciliation_tool": "gitea_reconcile_after_restart",
|
||||
"follow_up_issue_required": True,
|
||||
"cross_references": ["#652", "#653", "#655", "#630", "#658", "#662", "#664"],
|
||||
"reasons": [
|
||||
"break-glass restart executed with incident creation and mandatory reconciliation"
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
|
||||
# --- #662 post-restart reconciliation ---------------------------------------
|
||||
|
||||
_POST_RESTART_LAST_PROOF: dict | None = None
|
||||
|
||||
@@ -37,12 +37,17 @@ def normalize_role_kind(
|
||||
*,
|
||||
profile_name: str | None = None,
|
||||
) -> str:
|
||||
"""Map profile/task role to a workspace namespace key."""
|
||||
"""Map profile/task role to a workspace namespace key.
|
||||
|
||||
Exact profile-name matches only for controller routing (#840 / #664 B1):
|
||||
substring lookalikes such as ``fake-controller`` must not become
|
||||
controller.
|
||||
"""
|
||||
role = (role_kind or "author").strip().lower()
|
||||
profile = (profile_name or "").strip().lower()
|
||||
if role == "reviewer" and "merger" in profile:
|
||||
if role == "reviewer" and profile in ("prgs-merger", "mdcps-merger", "merger"):
|
||||
return "merger"
|
||||
if "controller" in profile or role == "controller":
|
||||
if role == "controller" or profile in ("prgs-controller", "controller"):
|
||||
return "controller"
|
||||
if role in ROLE_WORKTREE_ENVS:
|
||||
return role
|
||||
|
||||
@@ -1,426 +0,0 @@
|
||||
"""MCP restart lifecycle audit events and incident materialization (#665).
|
||||
|
||||
Restarts and recovery attempts must leave a forensic trail: impact previews,
|
||||
drain enter/exit, drain-proof results, apply gate verdicts, break-glass, and
|
||||
post-restart reconcile outcomes. Failed drains and break-glass must also raise
|
||||
durable Gitea incident issues so they cannot be silently repeated.
|
||||
|
||||
This module is the pure + sink layer for that trail:
|
||||
|
||||
* **Schema** — ``mcp.restart.*`` event names and a redacted payload builder.
|
||||
* **Emission** — append-only via :mod:`gitea_audit` (off when ``GITEA_AUDIT_LOG``
|
||||
is unset; privileged apply can still *require* a successful write).
|
||||
* **Incidents** — descriptors for failed drain / break-glass / unguarded restart,
|
||||
plus an optional materializer that creates a Gitea issue through an injected
|
||||
``create_issue_fn`` (keeps this module free of network I/O in tests).
|
||||
|
||||
Design rules:
|
||||
|
||||
* **No secrets.** All free text is redacted before write or issue body assembly.
|
||||
* **Never raises from emission.** ``emit_restart_event`` returns False on sink
|
||||
failure so callers can decide fail-closed policy for privileged restarts.
|
||||
* **Does not restart.** Audit never executes a process restart.
|
||||
* **Drain proof stays #661.** This module records what the gate decided; it
|
||||
does not re-verify proofs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Callable, Mapping, Sequence
|
||||
|
||||
import gitea_audit
|
||||
|
||||
# ── Event vocabulary (stable identifiers for operators + tests) ───────────────
|
||||
|
||||
EVENT_IMPACT_PREVIEW = "mcp.restart.impact_preview"
|
||||
EVENT_DRAIN_ENTER = "mcp.restart.drain_enter"
|
||||
EVENT_DRAIN_EXIT = "mcp.restart.drain_exit"
|
||||
EVENT_DRAIN_PROOF = "mcp.restart.drain_proof"
|
||||
EVENT_APPLY_GATE = "mcp.restart.apply_gate"
|
||||
EVENT_BREAK_GLASS = "mcp.restart.break_glass"
|
||||
EVENT_POST_RESTART_RECONCILE = "mcp.restart.post_restart_reconcile"
|
||||
EVENT_NARROWER_RECOVERY = "mcp.restart.narrower_recovery"
|
||||
EVENT_UNGUARDED_DETECTED = "mcp.restart.unguarded_detected"
|
||||
|
||||
RESTART_EVENT_TYPES: frozenset[str] = frozenset(
|
||||
{
|
||||
EVENT_IMPACT_PREVIEW,
|
||||
EVENT_DRAIN_ENTER,
|
||||
EVENT_DRAIN_EXIT,
|
||||
EVENT_DRAIN_PROOF,
|
||||
EVENT_APPLY_GATE,
|
||||
EVENT_BREAK_GLASS,
|
||||
EVENT_POST_RESTART_RECONCILE,
|
||||
EVENT_NARROWER_RECOVERY,
|
||||
EVENT_UNGUARDED_DETECTED,
|
||||
}
|
||||
)
|
||||
|
||||
# Incident kinds (durable Gitea issues).
|
||||
INCIDENT_FAILED_DRAIN = "restart_failed_drain"
|
||||
INCIDENT_BREAK_GLASS = "restart_break_glass"
|
||||
INCIDENT_RECONCILE_UNRESOLVED = "restart_reconcile_unresolved"
|
||||
INCIDENT_UNGUARDED = "restart_unguarded_detected"
|
||||
|
||||
DEFAULT_INCIDENT_LABELS: tuple[str, ...] = (
|
||||
"mcp-health",
|
||||
"safety",
|
||||
"observability",
|
||||
"status:ready",
|
||||
"type:bug",
|
||||
"workflow-hardening",
|
||||
)
|
||||
|
||||
CreateIssueFn = Callable[..., dict[str, Any]]
|
||||
|
||||
|
||||
def _utc_now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def new_correlation_id() -> str:
|
||||
"""Mint a short correlation id shared across a restart lifecycle."""
|
||||
return f"rst-{uuid.uuid4().hex[:16]}"
|
||||
|
||||
|
||||
def build_restart_event(
|
||||
*,
|
||||
event_type: str,
|
||||
outcome: str,
|
||||
correlation_id: str | None = None,
|
||||
remote: str | None = None,
|
||||
org: str | None = None,
|
||||
repo: str | None = None,
|
||||
requesting_session_id: str | None = None,
|
||||
restart_class: str | None = None,
|
||||
profile_name: str | None = None,
|
||||
authenticated_username: str | None = None,
|
||||
reasons: Sequence[str] | None = None,
|
||||
details: Mapping[str, Any] | None = None,
|
||||
now: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a redacted ``mcp.restart.*`` audit event.
|
||||
|
||||
Raises ``ValueError`` on unknown event types so a typo cannot silently land
|
||||
under a free-form action name.
|
||||
"""
|
||||
name = str(event_type or "").strip()
|
||||
if name not in RESTART_EVENT_TYPES:
|
||||
raise ValueError(
|
||||
f"unknown restart audit event_type {name!r}; expected one of "
|
||||
f"{sorted(RESTART_EVENT_TYPES)}"
|
||||
)
|
||||
redacted_reasons = [
|
||||
gitea_audit.redact(str(r)) for r in (reasons or []) if str(r).strip()
|
||||
]
|
||||
redacted_details = gitea_audit.redact(dict(details or {}))
|
||||
if not isinstance(redacted_details, dict):
|
||||
redacted_details = {"value": redacted_details}
|
||||
|
||||
event = gitea_audit.build_event(
|
||||
action=name,
|
||||
result=str(outcome or "unknown"),
|
||||
remote=remote,
|
||||
repository=f"{org}/{repo}" if org and repo else None,
|
||||
profile_name=profile_name,
|
||||
authenticated_username=authenticated_username,
|
||||
reason="; ".join(redacted_reasons) if redacted_reasons else None,
|
||||
request_metadata={
|
||||
"event_family": "mcp.restart",
|
||||
"correlation_id": correlation_id or new_correlation_id(),
|
||||
"restart_class": restart_class,
|
||||
"requesting_session_id": requesting_session_id,
|
||||
"org": org,
|
||||
"repo": repo,
|
||||
"details": redacted_details,
|
||||
"reasons": redacted_reasons,
|
||||
},
|
||||
now=now or _utc_now_iso(),
|
||||
operation=name,
|
||||
)
|
||||
event["action_type"] = "restart_lifecycle"
|
||||
event["event_type"] = name
|
||||
event["correlation_id"] = (event.get("request_metadata") or {}).get(
|
||||
"correlation_id"
|
||||
)
|
||||
return event
|
||||
|
||||
|
||||
def emit_restart_event(event: Mapping[str, Any], *, path: str | None = None) -> bool:
|
||||
"""Append *event* to the audit sink. Never raises. Returns write success."""
|
||||
try:
|
||||
return bool(gitea_audit.write_event(dict(event), path=path))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def require_audit_or_deny(
|
||||
*,
|
||||
privileged: bool,
|
||||
written: bool,
|
||||
audit_enabled: bool | None = None,
|
||||
) -> list[str]:
|
||||
"""Return deny reasons when a privileged restart path fails to audit.
|
||||
|
||||
When audit is not configured (``GITEA_AUDIT_LOG`` unset), privileged apply
|
||||
still proceeds under the rollout policy "enable audit before enforcing
|
||||
deny-on-audit-fail" — but *if* audit is enabled and the write fails,
|
||||
privileged apply is denied (fail closed).
|
||||
"""
|
||||
enabled = (
|
||||
gitea_audit.audit_enabled() if audit_enabled is None else bool(audit_enabled)
|
||||
)
|
||||
if not privileged:
|
||||
return []
|
||||
if not enabled:
|
||||
return []
|
||||
if written:
|
||||
return []
|
||||
return [
|
||||
"privileged restart path requires a successful audit write; "
|
||||
"audit sink failed (fail closed, #665)"
|
||||
]
|
||||
|
||||
|
||||
# ── Incident descriptors ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_incident_descriptor(
|
||||
*,
|
||||
kind: str,
|
||||
reasons: Sequence[str],
|
||||
correlation_id: str | None = None,
|
||||
requesting_session_id: str | None = None,
|
||||
restart_class: str | None = None,
|
||||
remote: str | None = None,
|
||||
org: str | None = None,
|
||||
repo: str | None = None,
|
||||
proof_id: str | None = None,
|
||||
at: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a durable incident descriptor (no network)."""
|
||||
titles = {
|
||||
INCIDENT_FAILED_DRAIN: "Restart denied: drain proof failed the hard gate",
|
||||
INCIDENT_BREAK_GLASS: "Break-glass MCP restart authorized",
|
||||
INCIDENT_RECONCILE_UNRESOLVED: "Post-restart reconcile left unresolved work",
|
||||
INCIDENT_UNGUARDED: "Unguarded MCP restart attempt detected",
|
||||
}
|
||||
title = titles.get(kind, f"MCP restart incident ({kind})")
|
||||
redacted_reasons = [
|
||||
gitea_audit.redact(str(r)) for r in reasons if str(r).strip()
|
||||
]
|
||||
return {
|
||||
"kind": kind,
|
||||
"title": title,
|
||||
"labels": list(DEFAULT_INCIDENT_LABELS),
|
||||
"reasons": redacted_reasons,
|
||||
"correlation_id": correlation_id,
|
||||
"requesting_session_id": requesting_session_id,
|
||||
"restart_class": restart_class,
|
||||
"remote": remote,
|
||||
"org": org,
|
||||
"repo": repo,
|
||||
"proof_id": proof_id,
|
||||
"at": at or _utc_now_iso(),
|
||||
"source": "restart_audit#665",
|
||||
}
|
||||
|
||||
|
||||
def incident_body(descriptor: Mapping[str, Any]) -> str:
|
||||
"""Render a redacted markdown body for a Gitea incident issue."""
|
||||
reasons = descriptor.get("reasons") or []
|
||||
reason_lines = "\n".join(f"- {gitea_audit.redact(str(r))}" for r in reasons) or (
|
||||
"- (no reasons recorded)"
|
||||
)
|
||||
return "\n".join(
|
||||
[
|
||||
"<!-- mcp-restart-incident:v1 -->",
|
||||
f"## MCP restart incident (`{descriptor.get('kind')}`)",
|
||||
"",
|
||||
f"**Correlation:** `{descriptor.get('correlation_id') or 'none'}`",
|
||||
f"**Session:** `{descriptor.get('requesting_session_id') or 'none'}`",
|
||||
f"**Class:** `{descriptor.get('restart_class') or 'none'}`",
|
||||
f"**Scope:** `{descriptor.get('remote')}/{descriptor.get('org')}/"
|
||||
f"{descriptor.get('repo')}`",
|
||||
f"**At:** `{descriptor.get('at')}`",
|
||||
f"**Proof id:** `{descriptor.get('proof_id') or 'none'}`",
|
||||
"",
|
||||
"### Reasons",
|
||||
reason_lines,
|
||||
"",
|
||||
"### Operator next steps",
|
||||
"- Treat this as durable follow-up work under the restart-governance umbrella (#655).",
|
||||
"- Do not invent a second restart path; use sanctioned coordinator tools only.",
|
||||
"- Raw secrets must never appear in this issue (already redacted).",
|
||||
"",
|
||||
f"_Source: {descriptor.get('source')}_",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def materialize_incident(
|
||||
descriptor: Mapping[str, Any],
|
||||
*,
|
||||
create_issue_fn: CreateIssueFn | None,
|
||||
dry_run: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Create a Gitea issue from *descriptor* when *create_issue_fn* is provided.
|
||||
|
||||
Returns a result dict with ``created`` / ``issue_number`` / ``dry_run`` /
|
||||
``reasons``. Never raises.
|
||||
"""
|
||||
base: dict[str, Any] = {
|
||||
"created": False,
|
||||
"dry_run": bool(dry_run),
|
||||
"issue_number": None,
|
||||
"kind": descriptor.get("kind"),
|
||||
"reasons": [],
|
||||
"descriptor": dict(descriptor),
|
||||
}
|
||||
if dry_run:
|
||||
base["reasons"] = ["dry-run only; no Gitea issue created"]
|
||||
return base
|
||||
if create_issue_fn is None:
|
||||
base["reasons"] = [
|
||||
"create_issue_fn not provided; incident descriptor retained only"
|
||||
]
|
||||
return base
|
||||
try:
|
||||
result = create_issue_fn(
|
||||
title=str(descriptor.get("title") or "MCP restart incident"),
|
||||
body=incident_body(descriptor),
|
||||
labels=list(descriptor.get("labels") or DEFAULT_INCIDENT_LABELS),
|
||||
org=descriptor.get("org"),
|
||||
repo=descriptor.get("repo"),
|
||||
)
|
||||
number = None
|
||||
if isinstance(result, dict):
|
||||
number = result.get("number") or result.get("issue_number")
|
||||
if number is not None:
|
||||
base["created"] = True
|
||||
base["issue_number"] = int(number)
|
||||
base["reasons"] = [f"created incident issue #{int(number)}"]
|
||||
else:
|
||||
base["reasons"] = ["create_issue_fn returned no issue number"]
|
||||
except Exception as exc: # noqa: BLE001 — never break restart path here
|
||||
base["reasons"] = [
|
||||
f"incident issue creation failed: {gitea_audit.redact(str(exc))}"
|
||||
]
|
||||
return base
|
||||
|
||||
|
||||
def record_restart_lifecycle(
|
||||
*,
|
||||
event_type: str,
|
||||
outcome: str,
|
||||
correlation_id: str,
|
||||
remote: str | None = None,
|
||||
org: str | None = None,
|
||||
repo: str | None = None,
|
||||
requesting_session_id: str | None = None,
|
||||
restart_class: str | None = None,
|
||||
profile_name: str | None = None,
|
||||
authenticated_username: str | None = None,
|
||||
reasons: Sequence[str] | None = None,
|
||||
details: Mapping[str, Any] | None = None,
|
||||
privileged: bool = False,
|
||||
create_incident: Mapping[str, Any] | None = None,
|
||||
create_issue_fn: CreateIssueFn | None = None,
|
||||
dry_run_incident: bool = False,
|
||||
audit_path: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Emit one restart audit event and optionally materialize an incident.
|
||||
|
||||
Returns ``{event, audit_written, deny_reasons, incident_result}``.
|
||||
"""
|
||||
event = build_restart_event(
|
||||
event_type=event_type,
|
||||
outcome=outcome,
|
||||
correlation_id=correlation_id,
|
||||
remote=remote,
|
||||
org=org,
|
||||
repo=repo,
|
||||
requesting_session_id=requesting_session_id,
|
||||
restart_class=restart_class,
|
||||
profile_name=profile_name,
|
||||
authenticated_username=authenticated_username,
|
||||
reasons=reasons,
|
||||
details=details,
|
||||
)
|
||||
written = emit_restart_event(event, path=audit_path)
|
||||
deny = require_audit_or_deny(privileged=privileged, written=written)
|
||||
incident_result = None
|
||||
if create_incident is not None:
|
||||
incident_result = materialize_incident(
|
||||
create_incident,
|
||||
create_issue_fn=create_issue_fn,
|
||||
dry_run=dry_run_incident,
|
||||
)
|
||||
return {
|
||||
"event": event,
|
||||
"audit_written": written,
|
||||
"deny_reasons": deny,
|
||||
"incident_result": incident_result,
|
||||
"correlation_id": correlation_id,
|
||||
}
|
||||
|
||||
|
||||
def incident_from_apply_gate(
|
||||
*,
|
||||
gate_payload: Mapping[str, Any],
|
||||
break_glass: bool,
|
||||
correlation_id: str,
|
||||
requesting_session_id: str | None,
|
||||
restart_class: str | None,
|
||||
remote: str | None,
|
||||
org: str | None,
|
||||
repo: str | None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Choose an incident descriptor from an apply-gate payload, if required."""
|
||||
reasons = list(gate_payload.get("reasons") or [])
|
||||
proof_id = gate_payload.get("proof_id")
|
||||
if break_glass:
|
||||
return build_incident_descriptor(
|
||||
kind=INCIDENT_BREAK_GLASS,
|
||||
reasons=reasons
|
||||
or ["break-glass restart path used; durable incident required (#665)"],
|
||||
correlation_id=correlation_id,
|
||||
requesting_session_id=requesting_session_id,
|
||||
restart_class=restart_class,
|
||||
remote=remote,
|
||||
org=org,
|
||||
repo=repo,
|
||||
proof_id=proof_id if isinstance(proof_id, str) else None,
|
||||
)
|
||||
# Failed drain / deny path.
|
||||
incident = gate_payload.get("incident")
|
||||
if isinstance(incident, Mapping) and incident:
|
||||
# Normalize gate-provided descriptor into our schema.
|
||||
return build_incident_descriptor(
|
||||
kind=INCIDENT_FAILED_DRAIN,
|
||||
reasons=list(incident.get("reasons") or reasons),
|
||||
correlation_id=correlation_id,
|
||||
requesting_session_id=requesting_session_id
|
||||
or incident.get("requesting_session_id"),
|
||||
restart_class=restart_class,
|
||||
remote=remote,
|
||||
org=org,
|
||||
repo=repo,
|
||||
proof_id=incident.get("proof_id") or proof_id,
|
||||
at=incident.get("at"),
|
||||
)
|
||||
if not gate_payload.get("allow") and not gate_payload.get("drain_gate_allow", True):
|
||||
return build_incident_descriptor(
|
||||
kind=INCIDENT_FAILED_DRAIN,
|
||||
reasons=reasons or ["restart apply denied"],
|
||||
correlation_id=correlation_id,
|
||||
requesting_session_id=requesting_session_id,
|
||||
restart_class=restart_class,
|
||||
remote=remote,
|
||||
org=org,
|
||||
repo=repo,
|
||||
proof_id=proof_id if isinstance(proof_id, str) else None,
|
||||
)
|
||||
return None
|
||||
@@ -576,6 +576,15 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
|
||||
"permission": "runtime.record_analytics_usage",
|
||||
"role": "author",
|
||||
},
|
||||
# #664: emergency break-glass MCP restart workflow (privileged controller role).
|
||||
"break_glass_restart": {
|
||||
"permission": "runtime.break_glass_restart",
|
||||
"role": "controller",
|
||||
},
|
||||
"gitea_break_glass_restart": {
|
||||
"permission": "runtime.break_glass_restart",
|
||||
"role": "controller",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,982 @@
|
||||
"""Tests for emergency break-glass MCP restart workflow (#664).
|
||||
|
||||
Regression suite for review #641 remediation: B13, B1, B14, B6/B11, B8, and
|
||||
preservation of previously accepted B2/B3/B5/B7/B9/B10 corrections.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import gitea_audit
|
||||
import gitea_config
|
||||
import gitea_mcp_server
|
||||
|
||||
|
||||
def _controller_profile(**extra) -> dict:
|
||||
base = {
|
||||
"profile_name": "prgs-controller",
|
||||
"execution_profile": "prgs-controller",
|
||||
"role": "reconciler", # declared role must not be redefined by capability
|
||||
"allowed_operations": [
|
||||
"gitea.read",
|
||||
"gitea.issue.create",
|
||||
"gitea.branch.delete",
|
||||
"gitea.pr.close",
|
||||
"gitea.pr.comment",
|
||||
"gitea.issue.comment",
|
||||
"runtime.break_glass_restart",
|
||||
],
|
||||
"forbidden_operations": [
|
||||
"gitea.pr.approve",
|
||||
"gitea.pr.merge",
|
||||
"gitea.pr.create",
|
||||
"gitea.branch.push",
|
||||
],
|
||||
}
|
||||
base.update(extra)
|
||||
return base
|
||||
|
||||
|
||||
def _gate_open_patches():
|
||||
"""Keep master-parity / runtime-mode blocks out of unit tests."""
|
||||
return (
|
||||
patch.object(gitea_mcp_server, "_master_parity_block", return_value=[]),
|
||||
patch.object(gitea_mcp_server, "_runtime_mode_block", return_value=[]),
|
||||
patch.object(gitea_mcp_server, "_try_auto_switch_for_operation", return_value=False),
|
||||
)
|
||||
|
||||
|
||||
class TestBreakGlassRestart(unittest.TestCase):
|
||||
"""Test suite for gitea_break_glass_restart tool and guardrails (#664)."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self.env_patcher = patch.dict(os.environ, {}, clear=False)
|
||||
self.env_patcher.start()
|
||||
os.environ.pop("GITEA_BREAKGLASS_RESTART_AUTHORIZATION", None)
|
||||
os.environ.pop("GITEA_SANCTIONED_RESTART_HOOK", None)
|
||||
os.environ.pop("GITEA_AUDIT_LOG", None)
|
||||
# Reset injectable executor between tests.
|
||||
gitea_mcp_server._break_glass_restart_executor = None
|
||||
|
||||
def tearDown(self) -> None:
|
||||
gitea_mcp_server._break_glass_restart_executor = None
|
||||
self.env_patcher.stop()
|
||||
|
||||
# ── B13: operation registration / real gate ───────────────────────────
|
||||
|
||||
def test_normalize_operation_accepts_canonical_break_glass_op(self) -> None:
|
||||
"""B13: production normalizer accepts exact runtime.break_glass_restart."""
|
||||
self.assertEqual(
|
||||
gitea_config.normalize_operation(
|
||||
"runtime.break_glass_restart", service="runtime"
|
||||
),
|
||||
"runtime.break_glass_restart",
|
||||
)
|
||||
ok, reason = gitea_config.check_operation(
|
||||
"runtime.break_glass_restart",
|
||||
["gitea.read", "runtime.break_glass_restart"],
|
||||
)
|
||||
self.assertTrue(ok, reason)
|
||||
self.assertEqual(reason, "allowed")
|
||||
|
||||
def test_normalize_unknown_and_misspelled_ops_fail_closed(self) -> None:
|
||||
"""B13: unknown / misspelled operations fail closed (no gitea.read fallback)."""
|
||||
# Well-formed but misspelled runtime op normalizes, then is not allowed.
|
||||
ok, reason = gitea_config.check_operation(
|
||||
"runtime.break_glass_restar", # misspelled
|
||||
["gitea.read", "runtime.break_glass_restart"],
|
||||
)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(reason, "not-allowed")
|
||||
|
||||
ok2, reason2 = gitea_config.check_operation(
|
||||
"runtime.break_glass_restart",
|
||||
["gitea.read"], # capability not granted
|
||||
)
|
||||
self.assertFalse(ok2)
|
||||
self.assertEqual(reason2, "not-allowed")
|
||||
|
||||
ok3, reason3 = gitea_config.check_operation(
|
||||
"frobnicate",
|
||||
["gitea.read", "runtime.break_glass_restart"],
|
||||
)
|
||||
self.assertFalse(ok3)
|
||||
self.assertEqual(reason3, "invalid-operation")
|
||||
|
||||
# gitea.read grant alone never authorizes break-glass.
|
||||
ok4, reason4 = gitea_config.check_operation(
|
||||
"runtime.break_glass_restart",
|
||||
["gitea.read"],
|
||||
)
|
||||
self.assertFalse(ok4)
|
||||
self.assertNotEqual(reason4, "allowed")
|
||||
|
||||
def test_real_profile_operation_gate_without_stubbing(self) -> None:
|
||||
"""B13: exercise real _profile_operation_gate (not stubbed)."""
|
||||
allowed = _controller_profile()
|
||||
denied = _controller_profile(
|
||||
allowed_operations=["gitea.read", "gitea.issue.create"]
|
||||
)
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=allowed):
|
||||
self.assertEqual(
|
||||
gitea_mcp_server._profile_operation_gate(
|
||||
"runtime.break_glass_restart"
|
||||
),
|
||||
[],
|
||||
)
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=denied):
|
||||
reasons = gitea_mcp_server._profile_operation_gate(
|
||||
"runtime.break_glass_restart"
|
||||
)
|
||||
self.assertTrue(reasons)
|
||||
self.assertTrue(
|
||||
any("runtime.break_glass_restart" in r or "not allowed" in r
|
||||
for r in reasons)
|
||||
)
|
||||
|
||||
def test_entry_point_uses_same_operation_as_gate(self) -> None:
|
||||
"""B13: entry point enforces runtime.break_glass_restart, not gitea.read."""
|
||||
# Profile has gitea.read but not the break-glass capability.
|
||||
prof = _controller_profile(
|
||||
allowed_operations=["gitea.read", "gitea.issue.create"]
|
||||
)
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart required due to deadlock in worker pool",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "permission_denied")
|
||||
self.assertNotIn("gitea.read", " ".join(res.get("reasons") or []))
|
||||
|
||||
# ── B1 / B14: exact profile auth, no substring, role preservation ─────
|
||||
|
||||
def test_trusted_prgs_controller_authorized(self) -> None:
|
||||
"""B1: exact trusted prgs-controller with capability is authorized."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart required due to deadlock in worker pool",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"], res)
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
|
||||
def test_fabricated_controller_like_profile_names_denied(self) -> None:
|
||||
"""B1: lookalike profile names never become authorized."""
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
for name in (
|
||||
"fake-controller",
|
||||
"controller-copy",
|
||||
"not-controller",
|
||||
"xcontrollerx",
|
||||
"CONTROLLER",
|
||||
"prgs-controller-copy",
|
||||
):
|
||||
prof = _controller_profile(profile_name=name, execution_profile=name)
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart required due to deadlock in worker pool",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"], name)
|
||||
self.assertEqual(res["blocker_kind"], "role_authorization", name)
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
|
||||
def test_ordinary_and_non_controller_reconciler_denied(self) -> None:
|
||||
"""B1: ordinary roles and non-controller reconcilers are denied."""
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
denied = [
|
||||
{"profile_name": "prgs-author", "role": "author",
|
||||
"allowed_operations": ["gitea.read", "runtime.break_glass_restart"],
|
||||
"forbidden_operations": []},
|
||||
{"profile_name": "prgs-reviewer", "role": "reviewer",
|
||||
"allowed_operations": ["gitea.read", "runtime.break_glass_restart"],
|
||||
"forbidden_operations": []},
|
||||
{"profile_name": "prgs-merger", "role": "merger",
|
||||
"allowed_operations": ["gitea.read", "runtime.break_glass_restart"],
|
||||
"forbidden_operations": []},
|
||||
{"profile_name": "prgs-reconciler", "role": "reconciler",
|
||||
"allowed_operations": [
|
||||
"gitea.read", "gitea.branch.delete", "runtime.break_glass_restart"
|
||||
],
|
||||
"forbidden_operations": []},
|
||||
{"profile_name": "prgs-controller", "role": "reconciler",
|
||||
"allowed_operations": ["gitea.read"],
|
||||
"forbidden_operations": []}, # no capability
|
||||
]
|
||||
for prof in denied:
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart required due to deadlock in worker pool",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"], prof)
|
||||
self.assertFalse(res["break_glass_executed"], prof)
|
||||
self.assertIn(
|
||||
res["blocker_kind"],
|
||||
("role_authorization", "permission_denied"),
|
||||
prof,
|
||||
)
|
||||
|
||||
def test_env_var_cannot_grant_authorization_or_bypass_denial(self) -> None:
|
||||
"""B2 preserved: env var cannot grant break-glass authorization."""
|
||||
os.environ["GITEA_BREAKGLASS_RESTART_AUTHORIZATION"] = "secret-bypass-token"
|
||||
prof = {
|
||||
"profile_name": "prgs-author",
|
||||
"role": "author",
|
||||
"allowed_operations": [
|
||||
"gitea.read", "gitea.issue.create", "runtime.break_glass_restart"
|
||||
],
|
||||
"forbidden_operations": [],
|
||||
}
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart attempting env var bypass",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "role_authorization")
|
||||
|
||||
def test_profile_role_kind_no_substring_authority(self) -> None:
|
||||
"""B1/B14: substring lookalikes do not become controller."""
|
||||
for name in (
|
||||
"fake-controller",
|
||||
"controller-copy",
|
||||
"not-controller",
|
||||
"xcontrollerx",
|
||||
"myCONTROLLER",
|
||||
):
|
||||
prof = {
|
||||
"profile_name": name,
|
||||
"role": "author",
|
||||
"allowed_operations": ["gitea.read"],
|
||||
}
|
||||
self.assertEqual(
|
||||
gitea_mcp_server._profile_role_kind(prof),
|
||||
"author",
|
||||
name,
|
||||
)
|
||||
# Role substrings must not promote.
|
||||
for role in ("not-controller", "control-plane", "xcontrollerx"):
|
||||
prof = {"profile_name": "other", "role": role, "allowed_operations": ["gitea.read"]}
|
||||
self.assertEqual(
|
||||
gitea_mcp_server._profile_role_kind(prof),
|
||||
role,
|
||||
role,
|
||||
)
|
||||
|
||||
def test_prgs_controller_retains_declared_reconciler_role(self) -> None:
|
||||
"""B14: break-glass capability does not redefine global role."""
|
||||
prof = _controller_profile(role="reconciler")
|
||||
self.assertEqual(gitea_mcp_server._profile_role_kind(prof), "reconciler")
|
||||
# Declared controller still wins when declared.
|
||||
prof2 = _controller_profile(role="controller")
|
||||
self.assertEqual(gitea_mcp_server._profile_role_kind(prof2), "controller")
|
||||
|
||||
def test_cleanup_merged_pr_branch_role_resolution_unchanged(self) -> None:
|
||||
"""B14: prgs-controller with reconciler role still resolves for cleanup."""
|
||||
# When declared reconciler, cleanup gate's role check should see reconciler.
|
||||
prof = _controller_profile(role="reconciler")
|
||||
self.assertEqual(gitea_mcp_server._profile_role_kind(prof), "reconciler")
|
||||
# Fabricated controller-like names with reconciler ops do not become controller.
|
||||
fake = {
|
||||
"profile_name": "fake-controller",
|
||||
"role": "reconciler",
|
||||
"allowed_operations": [
|
||||
"gitea.read", "gitea.branch.delete", "gitea.pr.close"
|
||||
],
|
||||
}
|
||||
self.assertEqual(gitea_mcp_server._profile_role_kind(fake), "reconciler")
|
||||
|
||||
def test_narrow_break_glass_capability_does_not_redefine_role(self) -> None:
|
||||
"""B14: granting runtime.break_glass_restart does not invent controller role."""
|
||||
prof = {
|
||||
"profile_name": "prgs-reconciler",
|
||||
"role": "reconciler",
|
||||
"allowed_operations": [
|
||||
"gitea.read",
|
||||
"gitea.branch.delete",
|
||||
"runtime.break_glass_restart",
|
||||
],
|
||||
}
|
||||
self.assertEqual(gitea_mcp_server._profile_role_kind(prof), "reconciler")
|
||||
|
||||
# ── B2-style input validation (preserved) ─────────────────────────────
|
||||
|
||||
def test_reason_validation(self) -> None:
|
||||
"""AC2: Reason is required and must be at least 10 characters long."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
for invalid_reason in ["", " ", "too short", "123456789"]:
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason=invalid_reason,
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "missing_required_fields")
|
||||
|
||||
def test_confirmation_validation(self) -> None:
|
||||
"""AC2: Confirmation phrase must match exact required string."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to stuck daemon processes",
|
||||
confirmation="wrong_confirmation_phrase",
|
||||
impact_ack=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "confirmation_mismatch")
|
||||
|
||||
def test_impact_ack_validation(self) -> None:
|
||||
"""AC2: impact_ack=True is mandatory."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to stuck daemon processes",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=False,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "impact_ack_required")
|
||||
|
||||
def test_incident_permission_gate(self) -> None:
|
||||
"""B3 preserved: Gate incident creation on gitea.issue.create permission."""
|
||||
prof = _controller_profile(
|
||||
allowed_operations=["gitea.read", "runtime.break_glass_restart"]
|
||||
)
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to hung worker process cohort",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
create_incident_issue=True,
|
||||
dry_run=False,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "permission_denied")
|
||||
|
||||
# ── B8: redaction ─────────────────────────────────────────────────────
|
||||
|
||||
def test_redaction_key_value_and_connection_strings(self) -> None:
|
||||
"""B8: key/value, bearer, connection-string, and embedded secrets."""
|
||||
cases = [
|
||||
("password=hunter2supersecret", ["hunter2supersecret"], "password"),
|
||||
("api_key: sk-live-abcdef1234567890ab", ["sk-live-abcdef1234567890ab"], "api_key"),
|
||||
("Server=db;Password=s3cretValue;Uid=sa", ["s3cretValue"], "password"),
|
||||
(
|
||||
"Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.abc.def",
|
||||
["eyJhbGciOiJIUzI1NiJ9.abc.def", "Bearer eyJ"],
|
||||
"authorization",
|
||||
),
|
||||
(
|
||||
"token ghp_1234567890abcdef12345678 embedded",
|
||||
["ghp_1234567890abcdef12345678"],
|
||||
"token",
|
||||
),
|
||||
("nested note password=letmein12345 end", ["letmein12345"], "password"),
|
||||
]
|
||||
for raw, secrets, key in cases:
|
||||
out = gitea_audit._redact_str(raw)
|
||||
self.assertIn("[REDACTED]", out, raw)
|
||||
for secret in secrets:
|
||||
self.assertNotIn(secret, out, raw)
|
||||
self.assertIn(key, out.lower(), raw)
|
||||
|
||||
nested = gitea_audit.redact({
|
||||
"reason": "password=supersecret99",
|
||||
"items": [{"api_key": "abc123xyz"}, "token ghp_abcdefghijklmnop1234"],
|
||||
"error": RuntimeError("pwd=nestedSecret99"),
|
||||
})
|
||||
# Exception objects pass through redact as non-str/non-container; ensure
|
||||
# string forms are covered via str conversion in _redact_str usage.
|
||||
self.assertEqual(nested["items"][0]["api_key"], gitea_audit.REDACTED)
|
||||
self.assertNotIn("supersecret99", nested["reason"])
|
||||
self.assertNotIn("ghp_abcdefghijklmnop1234", nested["items"][1])
|
||||
|
||||
def test_redaction_preserves_benign_sec_prefix_text(self) -> None:
|
||||
"""B8: ordinary text beginning with sec- must not be erased."""
|
||||
benign = (
|
||||
"Emergency restart in sec-primary-region for sector-planning "
|
||||
"and secondary-health checks"
|
||||
)
|
||||
out = gitea_audit._redact_str(benign)
|
||||
self.assertIn("sec-primary-region", out)
|
||||
self.assertIn("sector-planning", out)
|
||||
self.assertIn("secondary-health", out)
|
||||
self.assertNotIn("[REDACTED]", out)
|
||||
|
||||
def test_redaction_across_break_glass_surfaces(self) -> None:
|
||||
"""B8: operator reason is redacted on result, incident, and audit surfaces."""
|
||||
prof = _controller_profile()
|
||||
raw_reason = (
|
||||
"Emergency restart: password=supersecret99 api_key: "
|
||||
"sk-live-abcdef1234567890ab and url https://user:[email protected]/api"
|
||||
)
|
||||
mock_api_request = MagicMock(return_value={"number": 101, "title": "[INCIDENT]"})
|
||||
fake_exec = {
|
||||
"success": True,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": True,
|
||||
"restart_performed": True,
|
||||
"break_glass_executed": True,
|
||||
}
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: fake_exec
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request", mock_api_request
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_reconcile_after_restart",
|
||||
return_value={"success": True},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason=raw_reason,
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"], res)
|
||||
self.assertNotIn("supersecret99", res["reason"])
|
||||
self.assertNotIn("sk-live-abcdef1234567890ab", res["reason"])
|
||||
self.assertNotIn("user:[email protected]", res["reason"])
|
||||
posted_body = mock_api_request.call_args[0][3]["body"]
|
||||
self.assertNotIn("supersecret99", posted_body)
|
||||
self.assertNotIn("sk-live-abcdef1234567890ab", posted_body)
|
||||
|
||||
# ── B6/B11: reachable executor / truthful flags ───────────────────────
|
||||
|
||||
def test_dry_run_never_executes_and_reports_false(self) -> None:
|
||||
"""B7/B6: dry-run never executes; break_glass_executed always false."""
|
||||
prof = _controller_profile()
|
||||
called = {"n": 0}
|
||||
|
||||
def _should_not_run(_req):
|
||||
called["n"] += 1
|
||||
return {"restart_performed": True, "break_glass_executed": True}
|
||||
|
||||
gitea_mcp_server._break_glass_restart_executor = _should_not_run
|
||||
mock_audit = MagicMock()
|
||||
mock_api = MagicMock()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": [{"session_id": "s1"}]},
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", mock_audit
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request", mock_api
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart preview in dry-run mode",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"])
|
||||
self.assertTrue(res["dry_run"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertTrue(res["would_execute"])
|
||||
self.assertEqual(called["n"], 0)
|
||||
mock_audit.assert_not_called()
|
||||
mock_api.assert_not_called()
|
||||
|
||||
def test_unsupported_apply_truthful(self) -> None:
|
||||
"""B6/B11: unsupported apply returns blocked result, execution false."""
|
||||
prof = _controller_profile()
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: {
|
||||
"success": False,
|
||||
"apply_supported": False,
|
||||
"apply_authorized": False,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": ["no hook"],
|
||||
}
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
return_value={"number": 555, "title": "[INCIDENT]"},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Privileged restart request with unsupported apply",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["performed"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "apply_unsupported")
|
||||
|
||||
def test_delegation_success_rejection_and_failure(self) -> None:
|
||||
"""B6/B11: distinct terminal states for success / rejection / failure."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
|
||||
def _run(exec_result, recon=None):
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: exec_result
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_reconcile_after_restart",
|
||||
return_value=recon or {"success": True},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
return_value={"number": 555, "title": "[INCIDENT]"},
|
||||
):
|
||||
return gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Privileged break-glass restart delegation path",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
|
||||
ok = _run({
|
||||
"success": True,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": True,
|
||||
"restart_performed": True,
|
||||
"break_glass_executed": True,
|
||||
})
|
||||
self.assertTrue(ok["success"], ok)
|
||||
self.assertTrue(ok["break_glass_executed"])
|
||||
self.assertTrue(ok["performed"])
|
||||
|
||||
rejected = _run({
|
||||
"success": False,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": False,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": ["class denied"],
|
||||
})
|
||||
self.assertFalse(rejected["success"])
|
||||
self.assertFalse(rejected["break_glass_executed"])
|
||||
self.assertEqual(rejected["blocker_kind"], "restart_delegation_failed")
|
||||
|
||||
failed = _run({
|
||||
"success": False,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": True,
|
||||
"restart_performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": ["executor error"],
|
||||
})
|
||||
self.assertFalse(failed["success"])
|
||||
self.assertFalse(failed["break_glass_executed"])
|
||||
self.assertEqual(failed["blocker_kind"], "restart_delegation_failed")
|
||||
|
||||
def test_reconciliation_success_and_failure_truthful_flags(self) -> None:
|
||||
"""B10 preserved: recon failure keeps break_glass_executed=true."""
|
||||
prof = _controller_profile()
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: {
|
||||
"success": True,
|
||||
"apply_supported": True,
|
||||
"apply_authorized": True,
|
||||
"restart_performed": True,
|
||||
"break_glass_executed": True,
|
||||
}
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_reconcile_after_restart",
|
||||
return_value={"success": False, "error": "lease cleanup failed"},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
return_value={"number": 555, "title": "[INCIDENT]"},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Privileged restart request with failing reconciliation",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertTrue(res["performed"])
|
||||
self.assertTrue(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "reconciliation_failed")
|
||||
|
||||
def test_authorization_is_not_execution(self) -> None:
|
||||
"""B6: apply_authorized alone never sets break_glass_executed."""
|
||||
# Default executor without hook → unsupported, not executed.
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
return_value={"number": 555, "title": "[INCIDENT]"},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Privileged restart without host hook configured",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "apply_unsupported")
|
||||
|
||||
# ── preserved fail-closed pre-exec (B5/B9) ────────────────────────────
|
||||
|
||||
def test_audit_failure_before_execution_fails_closed(self) -> None:
|
||||
"""B9 preserved: Audit recording failure stops execution fail-closed."""
|
||||
prof = _controller_profile()
|
||||
called = {"n": 0}
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: called.__setitem__("n", called["n"] + 1) or {}
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=False
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request", MagicMock()
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart with failing audit sink",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "audit_recording_failed")
|
||||
self.assertEqual(called["n"], 0)
|
||||
|
||||
def test_incident_creation_failure_before_execution_fails_closed(self) -> None:
|
||||
"""B5 preserved: Incident creation failure stops execution fail-closed."""
|
||||
prof = _controller_profile()
|
||||
called = {"n": 0}
|
||||
gitea_mcp_server._break_glass_restart_executor = lambda req: called.__setitem__("n", called["n"] + 1) or {}
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
side_effect=RuntimeError("Gitea 500 API Error"),
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart with failing incident POST",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "incident_creation_failed")
|
||||
self.assertEqual(called["n"], 0)
|
||||
|
||||
def test_incident_opt_out_on_real_execution_fails_closed(self) -> None:
|
||||
"""B5 preserved: create_incident_issue=False fails closed on real execution."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart trying to skip incident creation",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=False,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "incident_creation_required")
|
||||
|
||||
def test_audit_disabled_fails_closed(self) -> None:
|
||||
"""B9 preserved: Disabling audit recording blocks execution fail-closed."""
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=False
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart with disabled audit logging",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "audit_recording_failed")
|
||||
|
||||
def test_capability_map_registration(self) -> None:
|
||||
"""B12/B13: capability map registers exact runtime.break_glass_restart."""
|
||||
from task_capability_map import TASK_CAPABILITY_MAP
|
||||
entry = TASK_CAPABILITY_MAP.get("gitea_break_glass_restart")
|
||||
self.assertIsNotNone(entry)
|
||||
self.assertEqual(entry["permission"], "runtime.break_glass_restart")
|
||||
self.assertEqual(entry["role"], "controller")
|
||||
entry2 = TASK_CAPABILITY_MAP.get("break_glass_restart")
|
||||
self.assertEqual(entry2["permission"], "runtime.break_glass_restart")
|
||||
|
||||
# ── B8 / B6 / B15 remediation tests ─────────────────────────────────────
|
||||
|
||||
def test_b8_redaction_gitea_token_and_uri_credentials(self) -> None:
|
||||
"""B8: GITEA_TOKEN= and URI userinfo credentials redacted without erasing neighbours."""
|
||||
# GITEA_TOKEN= with underscore key
|
||||
out1 = gitea_audit._redact_str("failed with GITEA_TOKEN=synthetic_tok_123456789")
|
||||
self.assertIn("GITEA_TOKEN=[REDACTED]", out1)
|
||||
self.assertNotIn("synthetic_tok_123456789", out1)
|
||||
|
||||
# Connection string with URI userinfo
|
||||
out2 = gitea_audit._redact_str("conn postgres://user:[email protected]:5432/app")
|
||||
self.assertIn("postgres://[REDACTED_USER]:[REDACTED_PASS]@db.internal:5432/app", out2)
|
||||
self.assertNotIn("s3cr3tpw", out2)
|
||||
|
||||
# Value boundary preserving adjacent audit evidence (correlation_id, incident_number)
|
||||
raw_audit = "password=secret123;correlation_id=bg-7f2a1c;incident_number=4242"
|
||||
out3 = gitea_audit._redact_str(raw_audit)
|
||||
self.assertIn("password=[REDACTED]", out3)
|
||||
self.assertIn("correlation_id=bg-7f2a1c", out3)
|
||||
self.assertIn("incident_number=4242", out3)
|
||||
self.assertNotIn("secret123", out3)
|
||||
|
||||
# Query param boundary in URL preserving adjacent parameters
|
||||
raw_url = "token=abc-123&pr=908&issue=664&head=c67f39b4"
|
||||
out4 = gitea_audit._redact_str(raw_url)
|
||||
self.assertIn("token=[REDACTED]", out4)
|
||||
self.assertIn("pr=908", out4)
|
||||
self.assertIn("issue=664", out4)
|
||||
self.assertIn("head=c67f39b4", out4)
|
||||
|
||||
def test_b6_default_executor_environment_text_cannot_imply_execution(self) -> None:
|
||||
"""B6/B11: GITEA_SANCTIONED_RESTART_HOOK string alone returns break_glass_executed=False."""
|
||||
os.environ["GITEA_SANCTIONED_RESTART_HOOK"] = "this-string-is-never-invoked"
|
||||
prof = _controller_profile()
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prof), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart",
|
||||
return_value={"affected_sessions": []},
|
||||
), patch.object(
|
||||
gitea_audit, "audit_enabled", return_value=True
|
||||
), patch.object(
|
||||
gitea_audit, "write_event", return_value=True
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request",
|
||||
return_value={"number": 555, "title": "[INCIDENT]"},
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Privileged restart request with non-empty hook env var",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["performed"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "restart_delegation_failed")
|
||||
|
||||
def test_b15_production_prgs_controller_grant_set_and_gates(self) -> None:
|
||||
"""B15: Genuine prgs-controller carrying runtime.break_glass_restart and gitea.issue.create passes real gates."""
|
||||
# Full production-shaped prgs-controller profile
|
||||
prod_profile = {
|
||||
"profile_name": "prgs-controller",
|
||||
"execution_profile": "prgs-controller",
|
||||
"role": "reconciler",
|
||||
"allowed_operations": [
|
||||
"gitea.read",
|
||||
"gitea.pr.close",
|
||||
"gitea.pr.comment",
|
||||
"gitea.issue.comment",
|
||||
"gitea.issue.create",
|
||||
"runtime.break_glass_restart",
|
||||
"gitea.branch.delete",
|
||||
],
|
||||
"forbidden_operations": [
|
||||
"gitea.pr.approve",
|
||||
"gitea.pr.merge",
|
||||
"gitea.pr.create",
|
||||
"gitea.branch.push",
|
||||
],
|
||||
}
|
||||
|
||||
# 1. Real _profile_operation_gate checks
|
||||
p_parity, p_runtime, p_switch = _gate_open_patches()
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=prod_profile):
|
||||
# Both required operations pass the real operation gate (no stubs)
|
||||
self.assertEqual(
|
||||
gitea_mcp_server._profile_operation_gate("runtime.break_glass_restart"),
|
||||
[],
|
||||
)
|
||||
self.assertEqual(
|
||||
gitea_mcp_server._profile_operation_gate("gitea.issue.create"),
|
||||
[],
|
||||
)
|
||||
|
||||
# 2. Missing gitea.issue.create fails incident creation gate
|
||||
no_issue_create = dict(prod_profile)
|
||||
no_issue_create["allowed_operations"] = [
|
||||
"gitea.read", "gitea.pr.close", "runtime.break_glass_restart"
|
||||
]
|
||||
with p_parity, p_runtime, p_switch:
|
||||
with patch.object(gitea_mcp_server, "get_profile", return_value=no_issue_create):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Restart testing missing gitea.issue.create permission",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "permission_denied")
|
||||
self.assertIn("gitea.issue.create", " ".join(res.get("reasons") or []))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,342 +0,0 @@
|
||||
"""Tests for MCP restart lifecycle audit events and incidents (#665)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import gitea_audit
|
||||
import restart_audit as ra
|
||||
|
||||
|
||||
class TestEventSchema(unittest.TestCase):
|
||||
def test_all_lifecycle_event_types_are_named(self):
|
||||
expected = {
|
||||
"mcp.restart.impact_preview",
|
||||
"mcp.restart.drain_enter",
|
||||
"mcp.restart.drain_exit",
|
||||
"mcp.restart.drain_proof",
|
||||
"mcp.restart.apply_gate",
|
||||
"mcp.restart.break_glass",
|
||||
"mcp.restart.post_restart_reconcile",
|
||||
"mcp.restart.narrower_recovery",
|
||||
"mcp.restart.unguarded_detected",
|
||||
}
|
||||
self.assertEqual(set(ra.RESTART_EVENT_TYPES), expected)
|
||||
|
||||
def test_build_restart_event_core_fields(self):
|
||||
event = ra.build_restart_event(
|
||||
event_type=ra.EVENT_IMPACT_PREVIEW,
|
||||
outcome="safe",
|
||||
correlation_id="rst-abc123",
|
||||
remote="prgs",
|
||||
org="Scaled-Tech-Consulting",
|
||||
repo="Gitea-Tools",
|
||||
requesting_session_id="sess-1",
|
||||
restart_class="full_mcp_restart",
|
||||
profile_name="prgs-author",
|
||||
authenticated_username="bot",
|
||||
reasons=["inventory complete"],
|
||||
details={"allow_restart": True},
|
||||
now="2026-07-25T12:00:00+00:00",
|
||||
)
|
||||
self.assertEqual(event["event_type"], ra.EVENT_IMPACT_PREVIEW)
|
||||
self.assertEqual(event["action"], ra.EVENT_IMPACT_PREVIEW)
|
||||
self.assertEqual(event["action_type"], "restart_lifecycle")
|
||||
self.assertEqual(event["result"], "safe")
|
||||
self.assertEqual(event["correlation_id"], "rst-abc123")
|
||||
self.assertEqual(event["profile_name"], "prgs-author")
|
||||
self.assertEqual(event["authenticated_username"], "bot")
|
||||
meta = event["request_metadata"]
|
||||
self.assertEqual(meta["event_family"], "mcp.restart")
|
||||
self.assertEqual(meta["correlation_id"], "rst-abc123")
|
||||
self.assertEqual(meta["restart_class"], "full_mcp_restart")
|
||||
self.assertEqual(meta["details"]["allow_restart"], True)
|
||||
|
||||
def test_unknown_event_type_raises(self):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
ra.build_restart_event(
|
||||
event_type="mcp.restart.not_a_real_event",
|
||||
outcome="x",
|
||||
correlation_id="rst-1",
|
||||
)
|
||||
self.assertIn("unknown restart audit event_type", str(ctx.exception))
|
||||
|
||||
def test_reasons_and_details_are_redacted(self):
|
||||
event = ra.build_restart_event(
|
||||
event_type=ra.EVENT_APPLY_GATE,
|
||||
outcome="deny",
|
||||
correlation_id="rst-sec",
|
||||
reasons=["token secret-xyz rejected", "ok"],
|
||||
details={"token": "leak-token", "status": "denied"},
|
||||
)
|
||||
self.assertNotIn("secret-xyz", event.get("reason") or "")
|
||||
meta = event["request_metadata"]
|
||||
self.assertEqual(meta["details"]["token"], gitea_audit.REDACTED)
|
||||
self.assertEqual(meta["details"]["status"], "denied")
|
||||
for reason in meta["reasons"]:
|
||||
self.assertNotIn("secret-xyz", reason)
|
||||
|
||||
def test_new_correlation_id_shape(self):
|
||||
cid = ra.new_correlation_id()
|
||||
self.assertTrue(cid.startswith("rst-"))
|
||||
self.assertEqual(len(cid), len("rst-") + 16)
|
||||
|
||||
|
||||
class TestEmitAndRequire(unittest.TestCase):
|
||||
def test_emit_appends_json_line(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = os.path.join(d, "audit.log")
|
||||
event = ra.build_restart_event(
|
||||
event_type=ra.EVENT_DRAIN_PROOF,
|
||||
outcome="pass",
|
||||
correlation_id="rst-write",
|
||||
)
|
||||
self.assertTrue(ra.emit_restart_event(event, path=path))
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
self.assertEqual(len(lines), 1)
|
||||
loaded = json.loads(lines[0])
|
||||
self.assertEqual(loaded["event_type"], ra.EVENT_DRAIN_PROOF)
|
||||
self.assertEqual(loaded["correlation_id"], "rst-write")
|
||||
|
||||
def test_emit_never_raises(self):
|
||||
self.assertFalse(
|
||||
ra.emit_restart_event({"action": "x"}, path="/no/such/dir/audit.log")
|
||||
)
|
||||
|
||||
def test_require_audit_denies_privileged_when_write_fails_and_enabled(self):
|
||||
deny = ra.require_audit_or_deny(
|
||||
privileged=True, written=False, audit_enabled=True
|
||||
)
|
||||
self.assertEqual(len(deny), 1)
|
||||
self.assertIn("fail closed", deny[0])
|
||||
|
||||
def test_require_audit_allows_when_audit_disabled(self):
|
||||
# Rollout policy: enable audit before enforcing deny-on-audit-fail.
|
||||
deny = ra.require_audit_or_deny(
|
||||
privileged=True, written=False, audit_enabled=False
|
||||
)
|
||||
self.assertEqual(deny, [])
|
||||
|
||||
def test_require_audit_noop_for_non_privileged(self):
|
||||
deny = ra.require_audit_or_deny(
|
||||
privileged=False, written=False, audit_enabled=True
|
||||
)
|
||||
self.assertEqual(deny, [])
|
||||
|
||||
def test_require_audit_allows_when_written(self):
|
||||
deny = ra.require_audit_or_deny(
|
||||
privileged=True, written=True, audit_enabled=True
|
||||
)
|
||||
self.assertEqual(deny, [])
|
||||
|
||||
|
||||
class TestIncidents(unittest.TestCase):
|
||||
def test_break_glass_descriptor(self):
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_BREAK_GLASS,
|
||||
reasons=["break-glass authorized"],
|
||||
correlation_id="rst-bg",
|
||||
requesting_session_id="s1",
|
||||
restart_class="full_mcp_restart",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
self.assertEqual(desc["kind"], ra.INCIDENT_BREAK_GLASS)
|
||||
self.assertIn("Break-glass", desc["title"])
|
||||
self.assertIn("mcp-health", desc["labels"])
|
||||
self.assertEqual(desc["source"], "restart_audit#665")
|
||||
|
||||
def test_incident_body_redacts_and_includes_correlation(self):
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||
reasons=["token secret-xyz failed proof"],
|
||||
correlation_id="rst-body",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
proof_id="proof-1",
|
||||
)
|
||||
body = ra.incident_body(desc)
|
||||
self.assertIn("rst-body", body)
|
||||
self.assertIn("proof-1", body)
|
||||
self.assertIn("mcp-restart-incident:v1", body)
|
||||
self.assertNotIn("secret-xyz", body)
|
||||
|
||||
def test_materialize_dry_run(self):
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||
reasons=["denied"],
|
||||
correlation_id="rst-dr",
|
||||
)
|
||||
result = ra.materialize_incident(desc, create_issue_fn=lambda **k: {}, dry_run=True)
|
||||
self.assertFalse(result["created"])
|
||||
self.assertTrue(result["dry_run"])
|
||||
self.assertIn("dry-run", result["reasons"][0])
|
||||
|
||||
def test_materialize_without_create_fn(self):
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||
reasons=["denied"],
|
||||
correlation_id="rst-nfn",
|
||||
)
|
||||
result = ra.materialize_incident(desc, create_issue_fn=None)
|
||||
self.assertFalse(result["created"])
|
||||
self.assertIn("create_issue_fn not provided", result["reasons"][0])
|
||||
|
||||
def test_materialize_creates_issue(self):
|
||||
created = {}
|
||||
|
||||
def _create(*, title, body, labels, org=None, repo=None, **_kw):
|
||||
created["title"] = title
|
||||
created["body"] = body
|
||||
created["labels"] = labels
|
||||
created["org"] = org
|
||||
created["repo"] = repo
|
||||
return {"number": 999}
|
||||
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_BREAK_GLASS,
|
||||
reasons=["break-glass"],
|
||||
correlation_id="rst-create",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
result = ra.materialize_incident(desc, create_issue_fn=_create)
|
||||
self.assertTrue(result["created"])
|
||||
self.assertEqual(result["issue_number"], 999)
|
||||
self.assertIn("Break-glass", created["title"])
|
||||
self.assertIn("rst-create", created["body"])
|
||||
self.assertEqual(created["org"], "O")
|
||||
|
||||
def test_materialize_never_raises_on_create_failure(self):
|
||||
def _boom(**_kw):
|
||||
raise RuntimeError("token secret-xyz network")
|
||||
|
||||
desc = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||
reasons=["x"],
|
||||
correlation_id="rst-boom",
|
||||
)
|
||||
result = ra.materialize_incident(desc, create_issue_fn=_boom)
|
||||
self.assertFalse(result["created"])
|
||||
self.assertIn("failed", result["reasons"][0])
|
||||
self.assertNotIn("secret-xyz", result["reasons"][0])
|
||||
|
||||
|
||||
class TestIncidentFromApplyGate(unittest.TestCase):
|
||||
def test_break_glass_always_incident(self):
|
||||
desc = ra.incident_from_apply_gate(
|
||||
gate_payload={"allow": True, "reasons": [], "proof_id": None},
|
||||
break_glass=True,
|
||||
correlation_id="rst-bg2",
|
||||
requesting_session_id="s",
|
||||
restart_class="full_mcp_restart",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
self.assertIsNotNone(desc)
|
||||
self.assertEqual(desc["kind"], ra.INCIDENT_BREAK_GLASS)
|
||||
|
||||
def test_failed_drain_from_gate_incident(self):
|
||||
desc = ra.incident_from_apply_gate(
|
||||
gate_payload={
|
||||
"allow": False,
|
||||
"drain_gate_allow": False,
|
||||
"reasons": ["proof expired"],
|
||||
"incident": {
|
||||
"reasons": ["proof expired"],
|
||||
"proof_id": "p1",
|
||||
},
|
||||
},
|
||||
break_glass=False,
|
||||
correlation_id="rst-fd",
|
||||
requesting_session_id="s",
|
||||
restart_class="full_mcp_restart",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
self.assertIsNotNone(desc)
|
||||
self.assertEqual(desc["kind"], ra.INCIDENT_FAILED_DRAIN)
|
||||
self.assertEqual(desc["proof_id"], "p1")
|
||||
|
||||
def test_allow_without_break_glass_no_incident(self):
|
||||
desc = ra.incident_from_apply_gate(
|
||||
gate_payload={
|
||||
"allow": True,
|
||||
"drain_gate_allow": True,
|
||||
"reasons": [],
|
||||
},
|
||||
break_glass=False,
|
||||
correlation_id="rst-ok",
|
||||
requesting_session_id="s",
|
||||
restart_class="full_mcp_restart",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
self.assertIsNone(desc)
|
||||
|
||||
|
||||
class TestRecordLifecycle(unittest.TestCase):
|
||||
def test_record_emits_and_materializes(self):
|
||||
created = []
|
||||
|
||||
def _create(**kwargs):
|
||||
created.append(kwargs)
|
||||
return {"number": 42}
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = os.path.join(d, "audit.log")
|
||||
with patch.dict(os.environ, {"GITEA_AUDIT_LOG": path}, clear=False):
|
||||
incident = ra.build_incident_descriptor(
|
||||
kind=ra.INCIDENT_BREAK_GLASS,
|
||||
reasons=["bg"],
|
||||
correlation_id="rst-lc",
|
||||
org="O",
|
||||
repo="R",
|
||||
)
|
||||
out = ra.record_restart_lifecycle(
|
||||
event_type=ra.EVENT_BREAK_GLASS,
|
||||
outcome="break_glass",
|
||||
correlation_id="rst-lc",
|
||||
remote="prgs",
|
||||
org="O",
|
||||
repo="R",
|
||||
privileged=True,
|
||||
create_incident=incident,
|
||||
create_issue_fn=_create,
|
||||
audit_path=path,
|
||||
)
|
||||
self.assertTrue(out["audit_written"])
|
||||
self.assertEqual(out["deny_reasons"], [])
|
||||
self.assertTrue(out["incident_result"]["created"])
|
||||
self.assertEqual(out["incident_result"]["issue_number"], 42)
|
||||
self.assertEqual(len(created), 1)
|
||||
|
||||
def test_privileged_deny_when_audit_write_fails(self):
|
||||
with patch.dict(
|
||||
os.environ, {"GITEA_AUDIT_LOG": "/no/such/dir/a.log"}, clear=False
|
||||
):
|
||||
with patch("restart_audit.emit_restart_event", return_value=False):
|
||||
with patch("gitea_audit.audit_enabled", return_value=True):
|
||||
out = ra.record_restart_lifecycle(
|
||||
event_type=ra.EVENT_APPLY_GATE,
|
||||
outcome="deny",
|
||||
correlation_id="rst-deny",
|
||||
privileged=True,
|
||||
audit_path="/no/such/dir/a.log",
|
||||
)
|
||||
self.assertFalse(out["audit_written"])
|
||||
self.assertEqual(len(out["deny_reasons"]), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user