Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22e0a41bd5 |
@@ -0,0 +1,93 @@
|
|||||||
|
# MCP restart audit events and incidents (#665)
|
||||||
|
|
||||||
|
Restarts and recovery attempts leave a forensic trail. Failed drains and
|
||||||
|
break-glass paths also raise durable Gitea incident issues so unsafe restarts
|
||||||
|
cannot be silently repeated.
|
||||||
|
|
||||||
|
Parent umbrella: **#655**. Related: impact coordinator **#658**, drain proof
|
||||||
|
**#661**, restart classes **#663**, break-glass **#664**, post-restart reconcile
|
||||||
|
**#662**, vision **#652**, roadmap **#653**, console recovery **#642**.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Piece | Where | Responsibility |
|
||||||
|
|-------|-------|----------------|
|
||||||
|
| Event schema + emission | `restart_audit.py` | `mcp.restart.*` vocabulary, redacted payload builder, append-only sink via `gitea_audit` |
|
||||||
|
| Fail-closed privileged gate | `restart_audit.require_audit_or_deny` | When `GITEA_AUDIT_LOG` is set and the write fails, privileged apply is denied |
|
||||||
|
| Incident descriptors | `restart_audit.build_incident_descriptor` | Durable follow-up issues (failed drain, break-glass, reconcile unresolved, unguarded) |
|
||||||
|
| Materializer | `restart_audit.materialize_incident` | Injected `create_issue_fn` (network kept out of pure tests) |
|
||||||
|
| Wiring | `gitea_request_mcp_restart` | Correlation id, impact-preview audit, apply-gate / break-glass audit + incident creation |
|
||||||
|
|
||||||
|
## Event vocabulary
|
||||||
|
|
||||||
|
| Event type | When |
|
||||||
|
|------------|------|
|
||||||
|
| `mcp.restart.impact_preview` | Every `gitea_request_mcp_restart` evaluation |
|
||||||
|
| `mcp.restart.drain_enter` | Drain window starts (schema reserved; emit from drain path) |
|
||||||
|
| `mcp.restart.drain_exit` | Drain window ends |
|
||||||
|
| `mcp.restart.drain_proof` | Drain-proof verification result |
|
||||||
|
| `mcp.restart.apply_gate` | Apply hard gate (`dry_run=False`) |
|
||||||
|
| `mcp.restart.break_glass` | Authorized break-glass bypass |
|
||||||
|
| `mcp.restart.post_restart_reconcile` | Post-restart reconcile outcome |
|
||||||
|
| `mcp.restart.narrower_recovery` | Narrower recovery attempt recorded |
|
||||||
|
| `mcp.restart.unguarded_detected` | Unguarded restart path detected |
|
||||||
|
|
||||||
|
All free text is redacted before sink write or issue body assembly. Emission
|
||||||
|
never raises; callers decide fail-closed policy.
|
||||||
|
|
||||||
|
## Correlation
|
||||||
|
|
||||||
|
Each restart lifecycle mints a short `correlation_id` (`rst-` + 16 hex) shared
|
||||||
|
across impact preview → apply gate → incident descriptors so operators can join
|
||||||
|
the trail.
|
||||||
|
|
||||||
|
## Privileged deny-on-audit-fail
|
||||||
|
|
||||||
|
Rollout policy (issue #665):
|
||||||
|
|
||||||
|
1. Configure `GITEA_AUDIT_LOG` so writes land.
|
||||||
|
2. Only then enforce deny when a privileged restart path cannot audit.
|
||||||
|
|
||||||
|
When audit is **not** configured, privileged apply still proceeds (no false
|
||||||
|
denials during rollout). When audit **is** configured and the write fails,
|
||||||
|
`apply_authorized` is cleared.
|
||||||
|
|
||||||
|
## Incidents
|
||||||
|
|
||||||
|
| Kind | Trigger |
|
||||||
|
|------|---------|
|
||||||
|
| `restart_failed_drain` | Apply denied by drain hard gate / failed proof |
|
||||||
|
| `restart_break_glass` | Any authorized break-glass apply |
|
||||||
|
| `restart_reconcile_unresolved` | Post-restart reconcile left work unresolved |
|
||||||
|
| `restart_unguarded_detected` | Unguarded restart attempt detected |
|
||||||
|
|
||||||
|
Break-glass **always** creates an incident descriptor (and a Gitea issue when
|
||||||
|
the create path is available). Failed drain does the same. Incident bodies
|
||||||
|
include correlation id, session, class, scope, proof id, and redacted reasons.
|
||||||
|
|
||||||
|
Default labels: `mcp-health`, `safety`, `observability`, `status:ready`,
|
||||||
|
`type:bug`, `workflow-hardening`.
|
||||||
|
|
||||||
|
## Tool payload surface
|
||||||
|
|
||||||
|
`gitea_request_mcp_restart` returns:
|
||||||
|
|
||||||
|
* `correlation_id` — lifecycle join key
|
||||||
|
* `restart_audit.impact_preview_written` — sink success for the preview event
|
||||||
|
* `restart_audit.apply_gate_written` — sink success for apply/break-glass (apply only)
|
||||||
|
* `restart_audit.incident_result` — materialization outcome when an incident was required
|
||||||
|
* `incident` — durable descriptor (when gate requires follow-up)
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
* No secrets in audit payloads or issue bodies.
|
||||||
|
* This module never restarts a process.
|
||||||
|
* Drain proof verification remains #661; audit only records the decision.
|
||||||
|
* Incident creation failures are recorded in `incident_result.reasons` and never
|
||||||
|
crash the restart evaluation path (audit write failure still fails closed for
|
||||||
|
privileged apply when the sink is enabled).
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
See `tests/test_restart_audit.py`: schema, redaction, emission, deny policy,
|
||||||
|
incident materialization mocks, break-glass / failed-drain selection.
|
||||||
@@ -33,6 +33,7 @@ recovery behavior for all nine classes.
|
|||||||
| `ControlPlaneDB.list_sessions` | `control_plane_db.py` | Read-only session inventory (the process-level unit a restart kills). |
|
| `ControlPlaneDB.list_sessions` | `control_plane_db.py` | Read-only session inventory (the process-level unit a restart kills). |
|
||||||
| `gitea_request_mcp_restart` | `gitea_mcp_server.py` | MCP tool: gathers inventory from the #613 DB, calls the coordinator, returns the report, and on `dry_run=False` runs the #661 drain-proof hard gate. Never restarts a process. |
|
| `gitea_request_mcp_restart` | `gitea_mcp_server.py` | MCP tool: gathers inventory from the #613 DB, calls the coordinator, returns the report, and on `dry_run=False` runs the #661 drain-proof hard gate. Never restarts a process. |
|
||||||
| `drain_proof.gate_apply_restart` | `drain_proof.py` | The #661 hard gate: verifies a drain proof against the current impact fingerprint, or records an authorized break-glass bypass. |
|
| `drain_proof.gate_apply_restart` | `drain_proof.py` | The #661 hard gate: verifies a drain proof against the current impact fingerprint, or records an authorized break-glass bypass. |
|
||||||
|
| `restart_audit` | `restart_audit.py` | #665 forensic trail: `mcp.restart.*` events via `gitea_audit`, correlation ids, durable incidents for failed drain / break-glass. See [`mcp-restart-audit.md`](./mcp-restart-audit.md). |
|
||||||
|
|
||||||
## Dimensions evaluated
|
## Dimensions evaluated
|
||||||
|
|
||||||
|
|||||||
@@ -1,102 +0,0 @@
|
|||||||
# Web Console: restart status, impact preview, and approval state (#667)
|
|
||||||
|
|
||||||
Phase 1 of the console restart surface. It consumes the #655 coordinator
|
|
||||||
substrate and displays it. It performs no restart, reload, drain, approval, or
|
|
||||||
process action, and it registers no write endpoint.
|
|
||||||
|
|
||||||
Issue #667's rollout is explicit — *status views first, write approval after the
|
|
||||||
backend gates are green* — and this change delivers only the status half.
|
|
||||||
|
|
||||||
## Surfaces
|
|
||||||
|
|
||||||
| Path | Method | Purpose |
|
|
||||||
|------|--------|---------|
|
|
||||||
| `/runtime/restart` | GET | Restart status page |
|
|
||||||
| `/api/v1/system/restart/status` | GET | Same snapshot as JSON |
|
|
||||||
|
|
||||||
Both accept an optional `restart_class` query parameter (default
|
|
||||||
`full_mcp_restart`). An unrecognised class is not an error: the coordinator
|
|
||||||
resolves it as unknown and fails closed, and the page shows the resulting deny.
|
|
||||||
|
|
||||||
Neither path accepts `POST`; a write attempt returns `405`, and a test asserts
|
|
||||||
it.
|
|
||||||
|
|
||||||
## What it shows
|
|
||||||
|
|
||||||
* **Impact preview (#658)** — verdict, blast radius, affected sessions, leases,
|
|
||||||
critical sections, mutations, and the counts behind them, evaluated
|
|
||||||
`dry_run=True` against live control-plane state.
|
|
||||||
* **Drain proof (#661)** — verification of a supplied proof: valid, clean,
|
|
||||||
expired, tampered, and the reasons behind a refusal.
|
|
||||||
* **Post-restart reconcile (#662)** — the most recent completion proof, its
|
|
||||||
overall status, and which dimensions still require follow-up.
|
|
||||||
* **Restart classes (#663)** — the least-privilege matrix, with *you may
|
|
||||||
request* and *you may execute* computed for the viewing role rather than for a
|
|
||||||
generic operator.
|
|
||||||
* **Approval controls (#633)** — the authorization state of
|
|
||||||
`system.restart_namespace` and `system.reload_namespace`.
|
|
||||||
* **Break-glass (#664)** — declared and marked unavailable; see below.
|
|
||||||
|
|
||||||
## Three rules this surface holds itself to
|
|
||||||
|
|
||||||
A status page that is wrong is worse than one that is missing, because an
|
|
||||||
operator acts on it. Three properties are enforced by tests, and each was
|
|
||||||
verified by reverting the guard and watching a test fail.
|
|
||||||
|
|
||||||
### An unreadable source reports unavailable, never green
|
|
||||||
|
|
||||||
Every source carries its own `SourceStatus`. Nothing substitutes a default,
|
|
||||||
placeholder, or self-comparison for a reading that failed. An unreadable
|
|
||||||
control-plane database yields `inventory_complete: false`, which the coordinator
|
|
||||||
itself turns into a fail-closed verdict, and the page says the blast radius is
|
|
||||||
unknown rather than showing an empty affected-sessions table.
|
|
||||||
|
|
||||||
An absent drain proof is reported as absent — not as a pass. The #661 gate
|
|
||||||
authorizes a restart only against a valid, unexpired, clean proof, so no proof
|
|
||||||
is precisely the state that gate denies on.
|
|
||||||
|
|
||||||
### Authorization is asked the way execution would ask it
|
|
||||||
|
|
||||||
Every probe passes `for_execution=True`.
|
|
||||||
|
|
||||||
Asked without it, an admin is `allowed` for `system.restart_namespace`. On a
|
|
||||||
control surface that reads as a live button. Asked the way an execution attempt
|
|
||||||
would ask, the same principal is refused `phase_not_active`, because the console
|
|
||||||
is in Phase 1 and the action is Phase 2. This surface reports the second answer.
|
|
||||||
|
|
||||||
`execution_enabled` is therefore `false` for every action and every role today,
|
|
||||||
and a test asserts that across the whole role matrix.
|
|
||||||
|
|
||||||
### The control-plane database is opened read-only
|
|
||||||
|
|
||||||
`ControlPlaneDB()` creates directories and runs migrations on construction — a
|
|
||||||
write. This surface never constructs one. It opens the sqlite file with
|
|
||||||
`mode=ro`, exactly as `webui/inventory.py` does, and treats a missing file as
|
|
||||||
missing authority rather than as an empty inventory.
|
|
||||||
|
|
||||||
The test that protects this points at a path inside a directory that already
|
|
||||||
exists, so a read-write `connect` would really create the file. A nested
|
|
||||||
missing-directory path would have passed for the wrong reason.
|
|
||||||
|
|
||||||
## Break-glass is declared, not offered
|
|
||||||
|
|
||||||
The break-glass workflow (#664) is not available on this branch's base. The
|
|
||||||
panel is rendered to operator-class roles as **unavailable**, naming the issue
|
|
||||||
that tracks it. It is not silently omitted, because an operator who has been
|
|
||||||
told a governance path exists needs to see that it is not wired here; and it is
|
|
||||||
not rendered as a control, because there is nothing behind it.
|
|
||||||
|
|
||||||
Unprivileged viewers see only a note that the surface is operator-class.
|
|
||||||
|
|
||||||
## Redaction and escaping
|
|
||||||
|
|
||||||
Every interpolated value passes through `_esc` (`html.escape(..., quote=True)`).
|
|
||||||
Free-form text and anything that can carry a filesystem path additionally passes
|
|
||||||
through `webui.inventory.scrub_text`, which redacts credential-shaped tokens
|
|
||||||
inside a string rather than only at its start. The impact payload is passed
|
|
||||||
through `webui.inventory.scrub` before rendering.
|
|
||||||
|
|
||||||
## Linkage
|
|
||||||
|
|
||||||
Parent #655 · extends #642 · consumes #658, #661, #662, #663 · RBAC #633 ·
|
|
||||||
console #631 · vision #652 · roadmap #653 · break-glass #664.
|
|
||||||
@@ -2069,6 +2069,7 @@ import lease_policy # noqa: E402
|
|||||||
import workflow_dashboard # noqa: E402 # #605 live queue/lease dashboard
|
import workflow_dashboard # noqa: E402 # #605 live queue/lease dashboard
|
||||||
import restart_coordinator # noqa: E402 # #658 MCP restart coordinator/impact
|
import restart_coordinator # noqa: E402 # #658 MCP restart coordinator/impact
|
||||||
import drain_proof # noqa: E402 # #661 pre-restart drain proof and hard gate
|
import drain_proof # noqa: E402 # #661 pre-restart drain proof and hard gate
|
||||||
|
import restart_audit # noqa: E402 # #665 restart audit events + incidents
|
||||||
import incident_bridge # noqa: E402
|
import incident_bridge # noqa: E402
|
||||||
import sentry_observability # noqa: E402 (#606 optional Sentry observability)
|
import sentry_observability # noqa: E402 (#606 optional Sentry observability)
|
||||||
import sentry_incident_bridge # noqa: E402 (#607 Sentry→Gitea incident bridge)
|
import sentry_incident_bridge # noqa: E402 (#607 Sentry→Gitea incident bridge)
|
||||||
@@ -22750,6 +22751,38 @@ def gitea_request_mcp_restart(
|
|||||||
# and a durable incident is raised. Break-glass is the only bypass and its
|
# and a durable incident is raised. Break-glass is the only bypass and its
|
||||||
# authorization is read from the environment, never self-asserted.
|
# authorization is read from the environment, never self-asserted.
|
||||||
payload["apply_supported"] = False
|
payload["apply_supported"] = False
|
||||||
|
# #665: correlation id threads impact preview → apply gate → incidents.
|
||||||
|
correlation_id = restart_audit.new_correlation_id()
|
||||||
|
payload["correlation_id"] = correlation_id
|
||||||
|
auth_user = None
|
||||||
|
try:
|
||||||
|
# remote-only: host override is for operator diagnostics, not required here
|
||||||
|
auth_user = (gitea_whoami(remote=remote) or {}).get("username")
|
||||||
|
except Exception: # noqa: BLE001 — identity is best-effort for audit
|
||||||
|
auth_user = None
|
||||||
|
preview_audit = restart_audit.record_restart_lifecycle(
|
||||||
|
event_type=restart_audit.EVENT_IMPACT_PREVIEW,
|
||||||
|
outcome=str(report.verdict or "unknown"),
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
remote=remote,
|
||||||
|
org=o,
|
||||||
|
repo=r,
|
||||||
|
requesting_session_id=sid,
|
||||||
|
restart_class=restart_class,
|
||||||
|
profile_name=profile_name,
|
||||||
|
authenticated_username=auth_user,
|
||||||
|
reasons=list(report.reasons or []),
|
||||||
|
details={
|
||||||
|
"dry_run": True,
|
||||||
|
"allow_restart": bool(report.allow_restart),
|
||||||
|
"inventory_complete": inventory_complete,
|
||||||
|
},
|
||||||
|
privileged=False,
|
||||||
|
)
|
||||||
|
payload["restart_audit"] = {
|
||||||
|
"correlation_id": correlation_id,
|
||||||
|
"impact_preview_written": preview_audit["audit_written"],
|
||||||
|
}
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
proof_obj: dict | None = None
|
proof_obj: dict | None = None
|
||||||
proof_parse_error: str | None = None
|
proof_parse_error: str | None = None
|
||||||
@@ -22809,6 +22842,86 @@ def gitea_request_mcp_restart(
|
|||||||
)
|
)
|
||||||
if not gate.allow and gate.incident is not None:
|
if not gate.allow and gate.incident is not None:
|
||||||
payload["incident"] = gate.incident
|
payload["incident"] = gate.incident
|
||||||
|
|
||||||
|
# #665: audit apply-gate + materialize durable incidents for failed
|
||||||
|
# drain and break-glass. Privileged apply denies if audit is enabled
|
||||||
|
# and the sink write fails.
|
||||||
|
incident_desc = restart_audit.incident_from_apply_gate(
|
||||||
|
gate_payload={
|
||||||
|
**gate_payload,
|
||||||
|
"incident": gate.incident,
|
||||||
|
"allow": gate.allow,
|
||||||
|
},
|
||||||
|
break_glass=break_glass,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
requesting_session_id=sid,
|
||||||
|
restart_class=restart_class,
|
||||||
|
remote=remote,
|
||||||
|
org=o,
|
||||||
|
repo=r,
|
||||||
|
)
|
||||||
|
if incident_desc is not None:
|
||||||
|
payload["incident"] = incident_desc
|
||||||
|
|
||||||
|
def _create_restart_incident_issue(
|
||||||
|
*, title, body, labels, org=None, repo=None, **_kw
|
||||||
|
):
|
||||||
|
return gitea_create_issue(
|
||||||
|
title=title,
|
||||||
|
body=body,
|
||||||
|
labels=labels,
|
||||||
|
remote=remote,
|
||||||
|
host=h,
|
||||||
|
org=org or o,
|
||||||
|
repo=repo or r,
|
||||||
|
)
|
||||||
|
|
||||||
|
apply_audit = restart_audit.record_restart_lifecycle(
|
||||||
|
event_type=(
|
||||||
|
restart_audit.EVENT_BREAK_GLASS
|
||||||
|
if break_glass
|
||||||
|
else restart_audit.EVENT_APPLY_GATE
|
||||||
|
),
|
||||||
|
outcome=(
|
||||||
|
"break_glass"
|
||||||
|
if break_glass
|
||||||
|
else ("allow" if payload["apply_authorized"] else "deny")
|
||||||
|
),
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
remote=remote,
|
||||||
|
org=o,
|
||||||
|
repo=r,
|
||||||
|
requesting_session_id=sid,
|
||||||
|
restart_class=restart_class,
|
||||||
|
profile_name=profile_name,
|
||||||
|
authenticated_username=auth_user,
|
||||||
|
reasons=list(gate_payload.get("reasons") or []),
|
||||||
|
details={
|
||||||
|
"apply_authorized": payload["apply_authorized"],
|
||||||
|
"drain_gate_allow": gate_payload.get("drain_gate_allow"),
|
||||||
|
"restart_class_authorized": restart_class_authorized,
|
||||||
|
"break_glass": break_glass,
|
||||||
|
"proof_id": gate_payload.get("proof_id"),
|
||||||
|
},
|
||||||
|
privileged=True,
|
||||||
|
create_incident=incident_desc,
|
||||||
|
create_issue_fn=_create_restart_incident_issue
|
||||||
|
if incident_desc is not None
|
||||||
|
else None,
|
||||||
|
dry_run_incident=False,
|
||||||
|
)
|
||||||
|
payload["restart_audit"] = {
|
||||||
|
"correlation_id": correlation_id,
|
||||||
|
"impact_preview_written": preview_audit["audit_written"],
|
||||||
|
"apply_gate_written": apply_audit["audit_written"],
|
||||||
|
"incident_result": apply_audit.get("incident_result"),
|
||||||
|
}
|
||||||
|
if apply_audit["deny_reasons"]:
|
||||||
|
payload["apply_authorized"] = False
|
||||||
|
payload["reasons"] = list(payload.get("reasons") or []) + list(
|
||||||
|
apply_audit["deny_reasons"]
|
||||||
|
)
|
||||||
|
payload["success"] = True
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,426 @@
|
|||||||
|
"""MCP restart lifecycle audit events and incident materialization (#665).
|
||||||
|
|
||||||
|
Restarts and recovery attempts must leave a forensic trail: impact previews,
|
||||||
|
drain enter/exit, drain-proof results, apply gate verdicts, break-glass, and
|
||||||
|
post-restart reconcile outcomes. Failed drains and break-glass must also raise
|
||||||
|
durable Gitea incident issues so they cannot be silently repeated.
|
||||||
|
|
||||||
|
This module is the pure + sink layer for that trail:
|
||||||
|
|
||||||
|
* **Schema** — ``mcp.restart.*`` event names and a redacted payload builder.
|
||||||
|
* **Emission** — append-only via :mod:`gitea_audit` (off when ``GITEA_AUDIT_LOG``
|
||||||
|
is unset; privileged apply can still *require* a successful write).
|
||||||
|
* **Incidents** — descriptors for failed drain / break-glass / unguarded restart,
|
||||||
|
plus an optional materializer that creates a Gitea issue through an injected
|
||||||
|
``create_issue_fn`` (keeps this module free of network I/O in tests).
|
||||||
|
|
||||||
|
Design rules:
|
||||||
|
|
||||||
|
* **No secrets.** All free text is redacted before write or issue body assembly.
|
||||||
|
* **Never raises from emission.** ``emit_restart_event`` returns False on sink
|
||||||
|
failure so callers can decide fail-closed policy for privileged restarts.
|
||||||
|
* **Does not restart.** Audit never executes a process restart.
|
||||||
|
* **Drain proof stays #661.** This module records what the gate decided; it
|
||||||
|
does not re-verify proofs.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Callable, Mapping, Sequence
|
||||||
|
|
||||||
|
import gitea_audit
|
||||||
|
|
||||||
|
# ── Event vocabulary (stable identifiers for operators + tests) ───────────────
|
||||||
|
|
||||||
|
EVENT_IMPACT_PREVIEW = "mcp.restart.impact_preview"
|
||||||
|
EVENT_DRAIN_ENTER = "mcp.restart.drain_enter"
|
||||||
|
EVENT_DRAIN_EXIT = "mcp.restart.drain_exit"
|
||||||
|
EVENT_DRAIN_PROOF = "mcp.restart.drain_proof"
|
||||||
|
EVENT_APPLY_GATE = "mcp.restart.apply_gate"
|
||||||
|
EVENT_BREAK_GLASS = "mcp.restart.break_glass"
|
||||||
|
EVENT_POST_RESTART_RECONCILE = "mcp.restart.post_restart_reconcile"
|
||||||
|
EVENT_NARROWER_RECOVERY = "mcp.restart.narrower_recovery"
|
||||||
|
EVENT_UNGUARDED_DETECTED = "mcp.restart.unguarded_detected"
|
||||||
|
|
||||||
|
RESTART_EVENT_TYPES: frozenset[str] = frozenset(
|
||||||
|
{
|
||||||
|
EVENT_IMPACT_PREVIEW,
|
||||||
|
EVENT_DRAIN_ENTER,
|
||||||
|
EVENT_DRAIN_EXIT,
|
||||||
|
EVENT_DRAIN_PROOF,
|
||||||
|
EVENT_APPLY_GATE,
|
||||||
|
EVENT_BREAK_GLASS,
|
||||||
|
EVENT_POST_RESTART_RECONCILE,
|
||||||
|
EVENT_NARROWER_RECOVERY,
|
||||||
|
EVENT_UNGUARDED_DETECTED,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Incident kinds (durable Gitea issues).
|
||||||
|
INCIDENT_FAILED_DRAIN = "restart_failed_drain"
|
||||||
|
INCIDENT_BREAK_GLASS = "restart_break_glass"
|
||||||
|
INCIDENT_RECONCILE_UNRESOLVED = "restart_reconcile_unresolved"
|
||||||
|
INCIDENT_UNGUARDED = "restart_unguarded_detected"
|
||||||
|
|
||||||
|
DEFAULT_INCIDENT_LABELS: tuple[str, ...] = (
|
||||||
|
"mcp-health",
|
||||||
|
"safety",
|
||||||
|
"observability",
|
||||||
|
"status:ready",
|
||||||
|
"type:bug",
|
||||||
|
"workflow-hardening",
|
||||||
|
)
|
||||||
|
|
||||||
|
CreateIssueFn = Callable[..., dict[str, Any]]
|
||||||
|
|
||||||
|
|
||||||
|
def _utc_now_iso() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def new_correlation_id() -> str:
|
||||||
|
"""Mint a short correlation id shared across a restart lifecycle."""
|
||||||
|
return f"rst-{uuid.uuid4().hex[:16]}"
|
||||||
|
|
||||||
|
|
||||||
|
def build_restart_event(
|
||||||
|
*,
|
||||||
|
event_type: str,
|
||||||
|
outcome: str,
|
||||||
|
correlation_id: str | None = None,
|
||||||
|
remote: str | None = None,
|
||||||
|
org: str | None = None,
|
||||||
|
repo: str | None = None,
|
||||||
|
requesting_session_id: str | None = None,
|
||||||
|
restart_class: str | None = None,
|
||||||
|
profile_name: str | None = None,
|
||||||
|
authenticated_username: str | None = None,
|
||||||
|
reasons: Sequence[str] | None = None,
|
||||||
|
details: Mapping[str, Any] | None = None,
|
||||||
|
now: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Build a redacted ``mcp.restart.*`` audit event.
|
||||||
|
|
||||||
|
Raises ``ValueError`` on unknown event types so a typo cannot silently land
|
||||||
|
under a free-form action name.
|
||||||
|
"""
|
||||||
|
name = str(event_type or "").strip()
|
||||||
|
if name not in RESTART_EVENT_TYPES:
|
||||||
|
raise ValueError(
|
||||||
|
f"unknown restart audit event_type {name!r}; expected one of "
|
||||||
|
f"{sorted(RESTART_EVENT_TYPES)}"
|
||||||
|
)
|
||||||
|
redacted_reasons = [
|
||||||
|
gitea_audit.redact(str(r)) for r in (reasons or []) if str(r).strip()
|
||||||
|
]
|
||||||
|
redacted_details = gitea_audit.redact(dict(details or {}))
|
||||||
|
if not isinstance(redacted_details, dict):
|
||||||
|
redacted_details = {"value": redacted_details}
|
||||||
|
|
||||||
|
event = gitea_audit.build_event(
|
||||||
|
action=name,
|
||||||
|
result=str(outcome or "unknown"),
|
||||||
|
remote=remote,
|
||||||
|
repository=f"{org}/{repo}" if org and repo else None,
|
||||||
|
profile_name=profile_name,
|
||||||
|
authenticated_username=authenticated_username,
|
||||||
|
reason="; ".join(redacted_reasons) if redacted_reasons else None,
|
||||||
|
request_metadata={
|
||||||
|
"event_family": "mcp.restart",
|
||||||
|
"correlation_id": correlation_id or new_correlation_id(),
|
||||||
|
"restart_class": restart_class,
|
||||||
|
"requesting_session_id": requesting_session_id,
|
||||||
|
"org": org,
|
||||||
|
"repo": repo,
|
||||||
|
"details": redacted_details,
|
||||||
|
"reasons": redacted_reasons,
|
||||||
|
},
|
||||||
|
now=now or _utc_now_iso(),
|
||||||
|
operation=name,
|
||||||
|
)
|
||||||
|
event["action_type"] = "restart_lifecycle"
|
||||||
|
event["event_type"] = name
|
||||||
|
event["correlation_id"] = (event.get("request_metadata") or {}).get(
|
||||||
|
"correlation_id"
|
||||||
|
)
|
||||||
|
return event
|
||||||
|
|
||||||
|
|
||||||
|
def emit_restart_event(event: Mapping[str, Any], *, path: str | None = None) -> bool:
|
||||||
|
"""Append *event* to the audit sink. Never raises. Returns write success."""
|
||||||
|
try:
|
||||||
|
return bool(gitea_audit.write_event(dict(event), path=path))
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def require_audit_or_deny(
|
||||||
|
*,
|
||||||
|
privileged: bool,
|
||||||
|
written: bool,
|
||||||
|
audit_enabled: bool | None = None,
|
||||||
|
) -> list[str]:
|
||||||
|
"""Return deny reasons when a privileged restart path fails to audit.
|
||||||
|
|
||||||
|
When audit is not configured (``GITEA_AUDIT_LOG`` unset), privileged apply
|
||||||
|
still proceeds under the rollout policy "enable audit before enforcing
|
||||||
|
deny-on-audit-fail" — but *if* audit is enabled and the write fails,
|
||||||
|
privileged apply is denied (fail closed).
|
||||||
|
"""
|
||||||
|
enabled = (
|
||||||
|
gitea_audit.audit_enabled() if audit_enabled is None else bool(audit_enabled)
|
||||||
|
)
|
||||||
|
if not privileged:
|
||||||
|
return []
|
||||||
|
if not enabled:
|
||||||
|
return []
|
||||||
|
if written:
|
||||||
|
return []
|
||||||
|
return [
|
||||||
|
"privileged restart path requires a successful audit write; "
|
||||||
|
"audit sink failed (fail closed, #665)"
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ── Incident descriptors ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def build_incident_descriptor(
|
||||||
|
*,
|
||||||
|
kind: str,
|
||||||
|
reasons: Sequence[str],
|
||||||
|
correlation_id: str | None = None,
|
||||||
|
requesting_session_id: str | None = None,
|
||||||
|
restart_class: str | None = None,
|
||||||
|
remote: str | None = None,
|
||||||
|
org: str | None = None,
|
||||||
|
repo: str | None = None,
|
||||||
|
proof_id: str | None = None,
|
||||||
|
at: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Build a durable incident descriptor (no network)."""
|
||||||
|
titles = {
|
||||||
|
INCIDENT_FAILED_DRAIN: "Restart denied: drain proof failed the hard gate",
|
||||||
|
INCIDENT_BREAK_GLASS: "Break-glass MCP restart authorized",
|
||||||
|
INCIDENT_RECONCILE_UNRESOLVED: "Post-restart reconcile left unresolved work",
|
||||||
|
INCIDENT_UNGUARDED: "Unguarded MCP restart attempt detected",
|
||||||
|
}
|
||||||
|
title = titles.get(kind, f"MCP restart incident ({kind})")
|
||||||
|
redacted_reasons = [
|
||||||
|
gitea_audit.redact(str(r)) for r in reasons if str(r).strip()
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
"kind": kind,
|
||||||
|
"title": title,
|
||||||
|
"labels": list(DEFAULT_INCIDENT_LABELS),
|
||||||
|
"reasons": redacted_reasons,
|
||||||
|
"correlation_id": correlation_id,
|
||||||
|
"requesting_session_id": requesting_session_id,
|
||||||
|
"restart_class": restart_class,
|
||||||
|
"remote": remote,
|
||||||
|
"org": org,
|
||||||
|
"repo": repo,
|
||||||
|
"proof_id": proof_id,
|
||||||
|
"at": at or _utc_now_iso(),
|
||||||
|
"source": "restart_audit#665",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def incident_body(descriptor: Mapping[str, Any]) -> str:
|
||||||
|
"""Render a redacted markdown body for a Gitea incident issue."""
|
||||||
|
reasons = descriptor.get("reasons") or []
|
||||||
|
reason_lines = "\n".join(f"- {gitea_audit.redact(str(r))}" for r in reasons) or (
|
||||||
|
"- (no reasons recorded)"
|
||||||
|
)
|
||||||
|
return "\n".join(
|
||||||
|
[
|
||||||
|
"<!-- mcp-restart-incident:v1 -->",
|
||||||
|
f"## MCP restart incident (`{descriptor.get('kind')}`)",
|
||||||
|
"",
|
||||||
|
f"**Correlation:** `{descriptor.get('correlation_id') or 'none'}`",
|
||||||
|
f"**Session:** `{descriptor.get('requesting_session_id') or 'none'}`",
|
||||||
|
f"**Class:** `{descriptor.get('restart_class') or 'none'}`",
|
||||||
|
f"**Scope:** `{descriptor.get('remote')}/{descriptor.get('org')}/"
|
||||||
|
f"{descriptor.get('repo')}`",
|
||||||
|
f"**At:** `{descriptor.get('at')}`",
|
||||||
|
f"**Proof id:** `{descriptor.get('proof_id') or 'none'}`",
|
||||||
|
"",
|
||||||
|
"### Reasons",
|
||||||
|
reason_lines,
|
||||||
|
"",
|
||||||
|
"### Operator next steps",
|
||||||
|
"- Treat this as durable follow-up work under the restart-governance umbrella (#655).",
|
||||||
|
"- Do not invent a second restart path; use sanctioned coordinator tools only.",
|
||||||
|
"- Raw secrets must never appear in this issue (already redacted).",
|
||||||
|
"",
|
||||||
|
f"_Source: {descriptor.get('source')}_",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def materialize_incident(
|
||||||
|
descriptor: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
create_issue_fn: CreateIssueFn | None,
|
||||||
|
dry_run: bool = False,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Create a Gitea issue from *descriptor* when *create_issue_fn* is provided.
|
||||||
|
|
||||||
|
Returns a result dict with ``created`` / ``issue_number`` / ``dry_run`` /
|
||||||
|
``reasons``. Never raises.
|
||||||
|
"""
|
||||||
|
base: dict[str, Any] = {
|
||||||
|
"created": False,
|
||||||
|
"dry_run": bool(dry_run),
|
||||||
|
"issue_number": None,
|
||||||
|
"kind": descriptor.get("kind"),
|
||||||
|
"reasons": [],
|
||||||
|
"descriptor": dict(descriptor),
|
||||||
|
}
|
||||||
|
if dry_run:
|
||||||
|
base["reasons"] = ["dry-run only; no Gitea issue created"]
|
||||||
|
return base
|
||||||
|
if create_issue_fn is None:
|
||||||
|
base["reasons"] = [
|
||||||
|
"create_issue_fn not provided; incident descriptor retained only"
|
||||||
|
]
|
||||||
|
return base
|
||||||
|
try:
|
||||||
|
result = create_issue_fn(
|
||||||
|
title=str(descriptor.get("title") or "MCP restart incident"),
|
||||||
|
body=incident_body(descriptor),
|
||||||
|
labels=list(descriptor.get("labels") or DEFAULT_INCIDENT_LABELS),
|
||||||
|
org=descriptor.get("org"),
|
||||||
|
repo=descriptor.get("repo"),
|
||||||
|
)
|
||||||
|
number = None
|
||||||
|
if isinstance(result, dict):
|
||||||
|
number = result.get("number") or result.get("issue_number")
|
||||||
|
if number is not None:
|
||||||
|
base["created"] = True
|
||||||
|
base["issue_number"] = int(number)
|
||||||
|
base["reasons"] = [f"created incident issue #{int(number)}"]
|
||||||
|
else:
|
||||||
|
base["reasons"] = ["create_issue_fn returned no issue number"]
|
||||||
|
except Exception as exc: # noqa: BLE001 — never break restart path here
|
||||||
|
base["reasons"] = [
|
||||||
|
f"incident issue creation failed: {gitea_audit.redact(str(exc))}"
|
||||||
|
]
|
||||||
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
def record_restart_lifecycle(
|
||||||
|
*,
|
||||||
|
event_type: str,
|
||||||
|
outcome: str,
|
||||||
|
correlation_id: str,
|
||||||
|
remote: str | None = None,
|
||||||
|
org: str | None = None,
|
||||||
|
repo: str | None = None,
|
||||||
|
requesting_session_id: str | None = None,
|
||||||
|
restart_class: str | None = None,
|
||||||
|
profile_name: str | None = None,
|
||||||
|
authenticated_username: str | None = None,
|
||||||
|
reasons: Sequence[str] | None = None,
|
||||||
|
details: Mapping[str, Any] | None = None,
|
||||||
|
privileged: bool = False,
|
||||||
|
create_incident: Mapping[str, Any] | None = None,
|
||||||
|
create_issue_fn: CreateIssueFn | None = None,
|
||||||
|
dry_run_incident: bool = False,
|
||||||
|
audit_path: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Emit one restart audit event and optionally materialize an incident.
|
||||||
|
|
||||||
|
Returns ``{event, audit_written, deny_reasons, incident_result}``.
|
||||||
|
"""
|
||||||
|
event = build_restart_event(
|
||||||
|
event_type=event_type,
|
||||||
|
outcome=outcome,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
remote=remote,
|
||||||
|
org=org,
|
||||||
|
repo=repo,
|
||||||
|
requesting_session_id=requesting_session_id,
|
||||||
|
restart_class=restart_class,
|
||||||
|
profile_name=profile_name,
|
||||||
|
authenticated_username=authenticated_username,
|
||||||
|
reasons=reasons,
|
||||||
|
details=details,
|
||||||
|
)
|
||||||
|
written = emit_restart_event(event, path=audit_path)
|
||||||
|
deny = require_audit_or_deny(privileged=privileged, written=written)
|
||||||
|
incident_result = None
|
||||||
|
if create_incident is not None:
|
||||||
|
incident_result = materialize_incident(
|
||||||
|
create_incident,
|
||||||
|
create_issue_fn=create_issue_fn,
|
||||||
|
dry_run=dry_run_incident,
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"event": event,
|
||||||
|
"audit_written": written,
|
||||||
|
"deny_reasons": deny,
|
||||||
|
"incident_result": incident_result,
|
||||||
|
"correlation_id": correlation_id,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def incident_from_apply_gate(
|
||||||
|
*,
|
||||||
|
gate_payload: Mapping[str, Any],
|
||||||
|
break_glass: bool,
|
||||||
|
correlation_id: str,
|
||||||
|
requesting_session_id: str | None,
|
||||||
|
restart_class: str | None,
|
||||||
|
remote: str | None,
|
||||||
|
org: str | None,
|
||||||
|
repo: str | None,
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
"""Choose an incident descriptor from an apply-gate payload, if required."""
|
||||||
|
reasons = list(gate_payload.get("reasons") or [])
|
||||||
|
proof_id = gate_payload.get("proof_id")
|
||||||
|
if break_glass:
|
||||||
|
return build_incident_descriptor(
|
||||||
|
kind=INCIDENT_BREAK_GLASS,
|
||||||
|
reasons=reasons
|
||||||
|
or ["break-glass restart path used; durable incident required (#665)"],
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
requesting_session_id=requesting_session_id,
|
||||||
|
restart_class=restart_class,
|
||||||
|
remote=remote,
|
||||||
|
org=org,
|
||||||
|
repo=repo,
|
||||||
|
proof_id=proof_id if isinstance(proof_id, str) else None,
|
||||||
|
)
|
||||||
|
# Failed drain / deny path.
|
||||||
|
incident = gate_payload.get("incident")
|
||||||
|
if isinstance(incident, Mapping) and incident:
|
||||||
|
# Normalize gate-provided descriptor into our schema.
|
||||||
|
return build_incident_descriptor(
|
||||||
|
kind=INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=list(incident.get("reasons") or reasons),
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
requesting_session_id=requesting_session_id
|
||||||
|
or incident.get("requesting_session_id"),
|
||||||
|
restart_class=restart_class,
|
||||||
|
remote=remote,
|
||||||
|
org=org,
|
||||||
|
repo=repo,
|
||||||
|
proof_id=incident.get("proof_id") or proof_id,
|
||||||
|
at=incident.get("at"),
|
||||||
|
)
|
||||||
|
if not gate_payload.get("allow") and not gate_payload.get("drain_gate_allow", True):
|
||||||
|
return build_incident_descriptor(
|
||||||
|
kind=INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=reasons or ["restart apply denied"],
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
requesting_session_id=requesting_session_id,
|
||||||
|
restart_class=restart_class,
|
||||||
|
remote=remote,
|
||||||
|
org=org,
|
||||||
|
repo=repo,
|
||||||
|
proof_id=proof_id if isinstance(proof_id, str) else None,
|
||||||
|
)
|
||||||
|
return None
|
||||||
@@ -0,0 +1,342 @@
|
|||||||
|
"""Tests for MCP restart lifecycle audit events and incidents (#665)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import gitea_audit
|
||||||
|
import restart_audit as ra
|
||||||
|
|
||||||
|
|
||||||
|
class TestEventSchema(unittest.TestCase):
|
||||||
|
def test_all_lifecycle_event_types_are_named(self):
|
||||||
|
expected = {
|
||||||
|
"mcp.restart.impact_preview",
|
||||||
|
"mcp.restart.drain_enter",
|
||||||
|
"mcp.restart.drain_exit",
|
||||||
|
"mcp.restart.drain_proof",
|
||||||
|
"mcp.restart.apply_gate",
|
||||||
|
"mcp.restart.break_glass",
|
||||||
|
"mcp.restart.post_restart_reconcile",
|
||||||
|
"mcp.restart.narrower_recovery",
|
||||||
|
"mcp.restart.unguarded_detected",
|
||||||
|
}
|
||||||
|
self.assertEqual(set(ra.RESTART_EVENT_TYPES), expected)
|
||||||
|
|
||||||
|
def test_build_restart_event_core_fields(self):
|
||||||
|
event = ra.build_restart_event(
|
||||||
|
event_type=ra.EVENT_IMPACT_PREVIEW,
|
||||||
|
outcome="safe",
|
||||||
|
correlation_id="rst-abc123",
|
||||||
|
remote="prgs",
|
||||||
|
org="Scaled-Tech-Consulting",
|
||||||
|
repo="Gitea-Tools",
|
||||||
|
requesting_session_id="sess-1",
|
||||||
|
restart_class="full_mcp_restart",
|
||||||
|
profile_name="prgs-author",
|
||||||
|
authenticated_username="bot",
|
||||||
|
reasons=["inventory complete"],
|
||||||
|
details={"allow_restart": True},
|
||||||
|
now="2026-07-25T12:00:00+00:00",
|
||||||
|
)
|
||||||
|
self.assertEqual(event["event_type"], ra.EVENT_IMPACT_PREVIEW)
|
||||||
|
self.assertEqual(event["action"], ra.EVENT_IMPACT_PREVIEW)
|
||||||
|
self.assertEqual(event["action_type"], "restart_lifecycle")
|
||||||
|
self.assertEqual(event["result"], "safe")
|
||||||
|
self.assertEqual(event["correlation_id"], "rst-abc123")
|
||||||
|
self.assertEqual(event["profile_name"], "prgs-author")
|
||||||
|
self.assertEqual(event["authenticated_username"], "bot")
|
||||||
|
meta = event["request_metadata"]
|
||||||
|
self.assertEqual(meta["event_family"], "mcp.restart")
|
||||||
|
self.assertEqual(meta["correlation_id"], "rst-abc123")
|
||||||
|
self.assertEqual(meta["restart_class"], "full_mcp_restart")
|
||||||
|
self.assertEqual(meta["details"]["allow_restart"], True)
|
||||||
|
|
||||||
|
def test_unknown_event_type_raises(self):
|
||||||
|
with self.assertRaises(ValueError) as ctx:
|
||||||
|
ra.build_restart_event(
|
||||||
|
event_type="mcp.restart.not_a_real_event",
|
||||||
|
outcome="x",
|
||||||
|
correlation_id="rst-1",
|
||||||
|
)
|
||||||
|
self.assertIn("unknown restart audit event_type", str(ctx.exception))
|
||||||
|
|
||||||
|
def test_reasons_and_details_are_redacted(self):
|
||||||
|
event = ra.build_restart_event(
|
||||||
|
event_type=ra.EVENT_APPLY_GATE,
|
||||||
|
outcome="deny",
|
||||||
|
correlation_id="rst-sec",
|
||||||
|
reasons=["token secret-xyz rejected", "ok"],
|
||||||
|
details={"token": "leak-token", "status": "denied"},
|
||||||
|
)
|
||||||
|
self.assertNotIn("secret-xyz", event.get("reason") or "")
|
||||||
|
meta = event["request_metadata"]
|
||||||
|
self.assertEqual(meta["details"]["token"], gitea_audit.REDACTED)
|
||||||
|
self.assertEqual(meta["details"]["status"], "denied")
|
||||||
|
for reason in meta["reasons"]:
|
||||||
|
self.assertNotIn("secret-xyz", reason)
|
||||||
|
|
||||||
|
def test_new_correlation_id_shape(self):
|
||||||
|
cid = ra.new_correlation_id()
|
||||||
|
self.assertTrue(cid.startswith("rst-"))
|
||||||
|
self.assertEqual(len(cid), len("rst-") + 16)
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmitAndRequire(unittest.TestCase):
|
||||||
|
def test_emit_appends_json_line(self):
|
||||||
|
with tempfile.TemporaryDirectory() as d:
|
||||||
|
path = os.path.join(d, "audit.log")
|
||||||
|
event = ra.build_restart_event(
|
||||||
|
event_type=ra.EVENT_DRAIN_PROOF,
|
||||||
|
outcome="pass",
|
||||||
|
correlation_id="rst-write",
|
||||||
|
)
|
||||||
|
self.assertTrue(ra.emit_restart_event(event, path=path))
|
||||||
|
with open(path, encoding="utf-8") as fh:
|
||||||
|
lines = fh.read().splitlines()
|
||||||
|
self.assertEqual(len(lines), 1)
|
||||||
|
loaded = json.loads(lines[0])
|
||||||
|
self.assertEqual(loaded["event_type"], ra.EVENT_DRAIN_PROOF)
|
||||||
|
self.assertEqual(loaded["correlation_id"], "rst-write")
|
||||||
|
|
||||||
|
def test_emit_never_raises(self):
|
||||||
|
self.assertFalse(
|
||||||
|
ra.emit_restart_event({"action": "x"}, path="/no/such/dir/audit.log")
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_require_audit_denies_privileged_when_write_fails_and_enabled(self):
|
||||||
|
deny = ra.require_audit_or_deny(
|
||||||
|
privileged=True, written=False, audit_enabled=True
|
||||||
|
)
|
||||||
|
self.assertEqual(len(deny), 1)
|
||||||
|
self.assertIn("fail closed", deny[0])
|
||||||
|
|
||||||
|
def test_require_audit_allows_when_audit_disabled(self):
|
||||||
|
# Rollout policy: enable audit before enforcing deny-on-audit-fail.
|
||||||
|
deny = ra.require_audit_or_deny(
|
||||||
|
privileged=True, written=False, audit_enabled=False
|
||||||
|
)
|
||||||
|
self.assertEqual(deny, [])
|
||||||
|
|
||||||
|
def test_require_audit_noop_for_non_privileged(self):
|
||||||
|
deny = ra.require_audit_or_deny(
|
||||||
|
privileged=False, written=False, audit_enabled=True
|
||||||
|
)
|
||||||
|
self.assertEqual(deny, [])
|
||||||
|
|
||||||
|
def test_require_audit_allows_when_written(self):
|
||||||
|
deny = ra.require_audit_or_deny(
|
||||||
|
privileged=True, written=True, audit_enabled=True
|
||||||
|
)
|
||||||
|
self.assertEqual(deny, [])
|
||||||
|
|
||||||
|
|
||||||
|
class TestIncidents(unittest.TestCase):
|
||||||
|
def test_break_glass_descriptor(self):
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_BREAK_GLASS,
|
||||||
|
reasons=["break-glass authorized"],
|
||||||
|
correlation_id="rst-bg",
|
||||||
|
requesting_session_id="s1",
|
||||||
|
restart_class="full_mcp_restart",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
self.assertEqual(desc["kind"], ra.INCIDENT_BREAK_GLASS)
|
||||||
|
self.assertIn("Break-glass", desc["title"])
|
||||||
|
self.assertIn("mcp-health", desc["labels"])
|
||||||
|
self.assertEqual(desc["source"], "restart_audit#665")
|
||||||
|
|
||||||
|
def test_incident_body_redacts_and_includes_correlation(self):
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=["token secret-xyz failed proof"],
|
||||||
|
correlation_id="rst-body",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
proof_id="proof-1",
|
||||||
|
)
|
||||||
|
body = ra.incident_body(desc)
|
||||||
|
self.assertIn("rst-body", body)
|
||||||
|
self.assertIn("proof-1", body)
|
||||||
|
self.assertIn("mcp-restart-incident:v1", body)
|
||||||
|
self.assertNotIn("secret-xyz", body)
|
||||||
|
|
||||||
|
def test_materialize_dry_run(self):
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=["denied"],
|
||||||
|
correlation_id="rst-dr",
|
||||||
|
)
|
||||||
|
result = ra.materialize_incident(desc, create_issue_fn=lambda **k: {}, dry_run=True)
|
||||||
|
self.assertFalse(result["created"])
|
||||||
|
self.assertTrue(result["dry_run"])
|
||||||
|
self.assertIn("dry-run", result["reasons"][0])
|
||||||
|
|
||||||
|
def test_materialize_without_create_fn(self):
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=["denied"],
|
||||||
|
correlation_id="rst-nfn",
|
||||||
|
)
|
||||||
|
result = ra.materialize_incident(desc, create_issue_fn=None)
|
||||||
|
self.assertFalse(result["created"])
|
||||||
|
self.assertIn("create_issue_fn not provided", result["reasons"][0])
|
||||||
|
|
||||||
|
def test_materialize_creates_issue(self):
|
||||||
|
created = {}
|
||||||
|
|
||||||
|
def _create(*, title, body, labels, org=None, repo=None, **_kw):
|
||||||
|
created["title"] = title
|
||||||
|
created["body"] = body
|
||||||
|
created["labels"] = labels
|
||||||
|
created["org"] = org
|
||||||
|
created["repo"] = repo
|
||||||
|
return {"number": 999}
|
||||||
|
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_BREAK_GLASS,
|
||||||
|
reasons=["break-glass"],
|
||||||
|
correlation_id="rst-create",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
result = ra.materialize_incident(desc, create_issue_fn=_create)
|
||||||
|
self.assertTrue(result["created"])
|
||||||
|
self.assertEqual(result["issue_number"], 999)
|
||||||
|
self.assertIn("Break-glass", created["title"])
|
||||||
|
self.assertIn("rst-create", created["body"])
|
||||||
|
self.assertEqual(created["org"], "O")
|
||||||
|
|
||||||
|
def test_materialize_never_raises_on_create_failure(self):
|
||||||
|
def _boom(**_kw):
|
||||||
|
raise RuntimeError("token secret-xyz network")
|
||||||
|
|
||||||
|
desc = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_FAILED_DRAIN,
|
||||||
|
reasons=["x"],
|
||||||
|
correlation_id="rst-boom",
|
||||||
|
)
|
||||||
|
result = ra.materialize_incident(desc, create_issue_fn=_boom)
|
||||||
|
self.assertFalse(result["created"])
|
||||||
|
self.assertIn("failed", result["reasons"][0])
|
||||||
|
self.assertNotIn("secret-xyz", result["reasons"][0])
|
||||||
|
|
||||||
|
|
||||||
|
class TestIncidentFromApplyGate(unittest.TestCase):
|
||||||
|
def test_break_glass_always_incident(self):
|
||||||
|
desc = ra.incident_from_apply_gate(
|
||||||
|
gate_payload={"allow": True, "reasons": [], "proof_id": None},
|
||||||
|
break_glass=True,
|
||||||
|
correlation_id="rst-bg2",
|
||||||
|
requesting_session_id="s",
|
||||||
|
restart_class="full_mcp_restart",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
self.assertIsNotNone(desc)
|
||||||
|
self.assertEqual(desc["kind"], ra.INCIDENT_BREAK_GLASS)
|
||||||
|
|
||||||
|
def test_failed_drain_from_gate_incident(self):
|
||||||
|
desc = ra.incident_from_apply_gate(
|
||||||
|
gate_payload={
|
||||||
|
"allow": False,
|
||||||
|
"drain_gate_allow": False,
|
||||||
|
"reasons": ["proof expired"],
|
||||||
|
"incident": {
|
||||||
|
"reasons": ["proof expired"],
|
||||||
|
"proof_id": "p1",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
break_glass=False,
|
||||||
|
correlation_id="rst-fd",
|
||||||
|
requesting_session_id="s",
|
||||||
|
restart_class="full_mcp_restart",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
self.assertIsNotNone(desc)
|
||||||
|
self.assertEqual(desc["kind"], ra.INCIDENT_FAILED_DRAIN)
|
||||||
|
self.assertEqual(desc["proof_id"], "p1")
|
||||||
|
|
||||||
|
def test_allow_without_break_glass_no_incident(self):
|
||||||
|
desc = ra.incident_from_apply_gate(
|
||||||
|
gate_payload={
|
||||||
|
"allow": True,
|
||||||
|
"drain_gate_allow": True,
|
||||||
|
"reasons": [],
|
||||||
|
},
|
||||||
|
break_glass=False,
|
||||||
|
correlation_id="rst-ok",
|
||||||
|
requesting_session_id="s",
|
||||||
|
restart_class="full_mcp_restart",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
self.assertIsNone(desc)
|
||||||
|
|
||||||
|
|
||||||
|
class TestRecordLifecycle(unittest.TestCase):
|
||||||
|
def test_record_emits_and_materializes(self):
|
||||||
|
created = []
|
||||||
|
|
||||||
|
def _create(**kwargs):
|
||||||
|
created.append(kwargs)
|
||||||
|
return {"number": 42}
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as d:
|
||||||
|
path = os.path.join(d, "audit.log")
|
||||||
|
with patch.dict(os.environ, {"GITEA_AUDIT_LOG": path}, clear=False):
|
||||||
|
incident = ra.build_incident_descriptor(
|
||||||
|
kind=ra.INCIDENT_BREAK_GLASS,
|
||||||
|
reasons=["bg"],
|
||||||
|
correlation_id="rst-lc",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
)
|
||||||
|
out = ra.record_restart_lifecycle(
|
||||||
|
event_type=ra.EVENT_BREAK_GLASS,
|
||||||
|
outcome="break_glass",
|
||||||
|
correlation_id="rst-lc",
|
||||||
|
remote="prgs",
|
||||||
|
org="O",
|
||||||
|
repo="R",
|
||||||
|
privileged=True,
|
||||||
|
create_incident=incident,
|
||||||
|
create_issue_fn=_create,
|
||||||
|
audit_path=path,
|
||||||
|
)
|
||||||
|
self.assertTrue(out["audit_written"])
|
||||||
|
self.assertEqual(out["deny_reasons"], [])
|
||||||
|
self.assertTrue(out["incident_result"]["created"])
|
||||||
|
self.assertEqual(out["incident_result"]["issue_number"], 42)
|
||||||
|
self.assertEqual(len(created), 1)
|
||||||
|
|
||||||
|
def test_privileged_deny_when_audit_write_fails(self):
|
||||||
|
with patch.dict(
|
||||||
|
os.environ, {"GITEA_AUDIT_LOG": "/no/such/dir/a.log"}, clear=False
|
||||||
|
):
|
||||||
|
with patch("restart_audit.emit_restart_event", return_value=False):
|
||||||
|
with patch("gitea_audit.audit_enabled", return_value=True):
|
||||||
|
out = ra.record_restart_lifecycle(
|
||||||
|
event_type=ra.EVENT_APPLY_GATE,
|
||||||
|
outcome="deny",
|
||||||
|
correlation_id="rst-deny",
|
||||||
|
privileged=True,
|
||||||
|
audit_path="/no/such/dir/a.log",
|
||||||
|
)
|
||||||
|
self.assertFalse(out["audit_written"])
|
||||||
|
self.assertEqual(len(out["deny_reasons"]), 1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -1,452 +0,0 @@
|
|||||||
"""Read-only restart console: views, gates, and honesty rules (#667).
|
|
||||||
|
|
||||||
The console consumes the #655 substrate. These tests hold it to the three
|
|
||||||
properties that make a status surface trustworthy:
|
|
||||||
|
|
||||||
* an unreadable source is reported unavailable, never rendered as green;
|
|
||||||
* authorization is probed the way execution would probe it, so an allow is
|
|
||||||
never shown for something that could not run;
|
|
||||||
* the surface performs no mutation, including no write to the control-plane DB.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from starlette.testclient import TestClient # noqa: E402
|
|
||||||
|
|
||||||
import restart_coordinator # noqa: E402
|
|
||||||
from webui import console_authz, restart_console, restart_views # noqa: E402
|
|
||||||
from webui.app import create_app # noqa: E402
|
|
||||||
|
|
||||||
NOW = datetime(2026, 7, 25, 21, 0, 0, tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
def _principal(role: str) -> console_authz.Principal:
|
|
||||||
return console_authz.Principal(
|
|
||||||
subject="[email protected]",
|
|
||||||
role=role,
|
|
||||||
identity_source=console_authz.IDENTITY_LOCAL_DEV,
|
|
||||||
authenticated=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _inventory(*, complete: bool = True, sessions=(), leases=()):
|
|
||||||
def _read(**_kwargs):
|
|
||||||
return {
|
|
||||||
"sessions": list(sessions),
|
|
||||||
"leases": list(leases),
|
|
||||||
"terminal_lock": None,
|
|
||||||
"prior_recovery_attempts": [],
|
|
||||||
"inventory_complete": complete,
|
|
||||||
"incomplete_reasons": (
|
|
||||||
[] if complete else ["fixture: inventory withheld"]
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
return _read
|
|
||||||
|
|
||||||
|
|
||||||
def _live_session(session_id: str = "prgs-author-1234-abcd") -> dict:
|
|
||||||
return {
|
|
||||||
"session_id": session_id,
|
|
||||||
"role": "author",
|
|
||||||
"profile": "prgs-author",
|
|
||||||
"pid": os.getpid(),
|
|
||||||
"status": "active",
|
|
||||||
"last_heartbeat_at": (NOW - timedelta(seconds=30)).isoformat(),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def drain_proof_fixture() -> dict:
|
|
||||||
"""A structurally complete but unsigned drain proof."""
|
|
||||||
return {
|
|
||||||
"version": "drain-proof/v1",
|
|
||||||
"proof_id": "deadbeef" * 8,
|
|
||||||
"clean": True,
|
|
||||||
"issued_at": (NOW - timedelta(minutes=1)).isoformat(),
|
|
||||||
"expires_at": (NOW + timedelta(minutes=5)).isoformat(),
|
|
||||||
"requesting_session_id": "s-live",
|
|
||||||
"impact_fingerprint": "f" * 64,
|
|
||||||
"checks": [],
|
|
||||||
"failed_checks": [],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class RestartClassMatrixTest(unittest.TestCase):
|
|
||||||
def test_every_policy_class_is_rendered(self) -> None:
|
|
||||||
views = restart_console.build_restart_class_views("operator")
|
|
||||||
self.assertEqual(len(views), len(restart_coordinator.RESTART_CLASS_POLICIES))
|
|
||||||
|
|
||||||
def test_viewer_capability_is_role_scoped_not_generic(self) -> None:
|
|
||||||
"""A worker role must not be shown as able to request a full restart."""
|
|
||||||
author = {
|
|
||||||
v.restart_class: v
|
|
||||||
for v in restart_console.build_restart_class_views("author")
|
|
||||||
}
|
|
||||||
operator = {
|
|
||||||
v.restart_class: v
|
|
||||||
for v in restart_console.build_restart_class_views("operator")
|
|
||||||
}
|
|
||||||
full = restart_coordinator.RestartClass.FULL_MCP_RESTART.value
|
|
||||||
|
|
||||||
self.assertFalse(author[full].viewer_may_request)
|
|
||||||
self.assertFalse(author[full].viewer_may_execute)
|
|
||||||
self.assertTrue(operator[full].viewer_may_request)
|
|
||||||
self.assertTrue(operator[full].viewer_may_execute)
|
|
||||||
|
|
||||||
def test_unknown_role_may_do_nothing(self) -> None:
|
|
||||||
views = restart_console.build_restart_class_views("not-a-role")
|
|
||||||
self.assertTrue(all(not v.viewer_may_request for v in views))
|
|
||||||
self.assertTrue(all(not v.viewer_may_execute for v in views))
|
|
||||||
|
|
||||||
|
|
||||||
class AuthorizationProbeTest(unittest.TestCase):
|
|
||||||
def test_probe_asks_for_execution_so_phase_gate_is_reported(self) -> None:
|
|
||||||
"""An admin clears the role bar and still cannot execute in Phase 1.
|
|
||||||
|
|
||||||
This is the case that distinguishes the two probes. Asked without
|
|
||||||
``for_execution`` an admin is *allowed* for ``system.restart_namespace``,
|
|
||||||
which on a control surface reads as a live button. Asked the way
|
|
||||||
execution asks, the same principal is refused ``phase_not_active``. The
|
|
||||||
console must report the second answer.
|
|
||||||
"""
|
|
||||||
by_id = {
|
|
||||||
a.action_id: a
|
|
||||||
for a in restart_console.build_action_authorizations(
|
|
||||||
_principal(console_authz.ADMIN)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
restart = by_id["system.restart_namespace"]
|
|
||||||
|
|
||||||
self.assertFalse(restart.execution_enabled)
|
|
||||||
self.assertEqual(restart.reason_code, console_authz.DENY_PHASE_NOT_ACTIVE)
|
|
||||||
|
|
||||||
permissive = console_authz.authorize(
|
|
||||||
"system.restart_namespace", _principal(console_authz.ADMIN)
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
permissive.allowed,
|
|
||||||
"guard precondition: without for_execution an admin is allowed, "
|
|
||||||
"which is exactly why the console must not probe that way",
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_operator_is_refused_the_admin_only_restart_action(self) -> None:
|
|
||||||
"""Role refusal precedes the phase gate and is reported as such."""
|
|
||||||
by_id = {
|
|
||||||
a.action_id: a
|
|
||||||
for a in restart_console.build_action_authorizations(
|
|
||||||
_principal(console_authz.OPERATOR)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
self.assertEqual(
|
|
||||||
by_id["system.restart_namespace"].reason_code,
|
|
||||||
console_authz.DENY_INSUFFICIENT_ROLE,
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_anonymous_is_denied_unauthenticated(self) -> None:
|
|
||||||
by_id = {
|
|
||||||
a.action_id: a for a in restart_console.build_action_authorizations(None)
|
|
||||||
}
|
|
||||||
self.assertEqual(
|
|
||||||
by_id["system.restart_namespace"].reason_code,
|
|
||||||
console_authz.DENY_UNAUTHENTICATED,
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_no_authorization_ever_reports_execution_enabled(self) -> None:
|
|
||||||
for role in (
|
|
||||||
console_authz.VIEWER,
|
|
||||||
console_authz.OPERATOR,
|
|
||||||
console_authz.CONTROLLER,
|
|
||||||
console_authz.ADMIN,
|
|
||||||
):
|
|
||||||
for auth in restart_console.build_action_authorizations(_principal(role)):
|
|
||||||
self.assertFalse(
|
|
||||||
auth.execution_enabled,
|
|
||||||
f"{role} reported execution_enabled for {auth.action_id}",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class ImpactPreviewTest(unittest.TestCase):
|
|
||||||
def test_impact_renders_from_coordinator_dto(self) -> None:
|
|
||||||
impact, source = restart_console.load_impact_report(
|
|
||||||
principal=_principal(console_authz.OPERATOR),
|
|
||||||
read_inventory=_inventory(sessions=[_live_session()]),
|
|
||||||
now=NOW,
|
|
||||||
)
|
|
||||||
self.assertTrue(source.available)
|
|
||||||
self.assertIsNotNone(impact)
|
|
||||||
self.assertEqual(
|
|
||||||
impact["restart_class"],
|
|
||||||
restart_coordinator.RestartClass.FULL_MCP_RESTART.value,
|
|
||||||
)
|
|
||||||
self.assertIn("verdict", impact)
|
|
||||||
self.assertFalse(impact["restart_performed"])
|
|
||||||
self.assertTrue(impact["dry_run"])
|
|
||||||
|
|
||||||
def test_incomplete_inventory_is_surfaced_and_denies(self) -> None:
|
|
||||||
impact, source = restart_console.load_impact_report(
|
|
||||||
principal=_principal(console_authz.OPERATOR),
|
|
||||||
read_inventory=_inventory(complete=False),
|
|
||||||
now=NOW,
|
|
||||||
)
|
|
||||||
self.assertFalse(impact["inventory_complete"])
|
|
||||||
self.assertFalse(impact["allow_restart"])
|
|
||||||
self.assertTrue(source.detail, "incomplete inventory must explain itself")
|
|
||||||
|
|
||||||
def test_inventory_reader_failure_is_unavailable_not_empty(self) -> None:
|
|
||||||
"""A reader that raises must not be rendered as 'no sessions affected'."""
|
|
||||||
|
|
||||||
def _boom(**_kwargs):
|
|
||||||
raise RuntimeError("control-plane unreachable")
|
|
||||||
|
|
||||||
impact, source = restart_console.load_impact_report(
|
|
||||||
principal=_principal(console_authz.OPERATOR),
|
|
||||||
read_inventory=_boom,
|
|
||||||
now=NOW,
|
|
||||||
)
|
|
||||||
self.assertIsNone(impact)
|
|
||||||
self.assertFalse(source.available)
|
|
||||||
self.assertIn("control-plane unreachable", source.detail)
|
|
||||||
|
|
||||||
|
|
||||||
class ControlPlaneReadTest(unittest.TestCase):
|
|
||||||
def test_missing_database_is_incomplete_not_empty(self) -> None:
|
|
||||||
inventory = restart_console.read_control_plane_inventory(
|
|
||||||
db_path="/nonexistent/control-plane.sqlite3"
|
|
||||||
)
|
|
||||||
self.assertFalse(inventory["inventory_complete"])
|
|
||||||
self.assertEqual(inventory["sessions"], [])
|
|
||||||
self.assertTrue(inventory["incomplete_reasons"])
|
|
||||||
|
|
||||||
def test_reader_never_creates_the_database(self) -> None:
|
|
||||||
"""Reading status must not bring a control-plane DB into existence.
|
|
||||||
|
|
||||||
The path deliberately sits in a directory that already exists: a
|
|
||||||
read-write ``sqlite3.connect`` would happily create the file there, so
|
|
||||||
this fails if the reader ever stops opening the database ``mode=ro``.
|
|
||||||
A nested-missing-directory path would pass for the wrong reason,
|
|
||||||
because sqlite cannot create the parent directory either way.
|
|
||||||
"""
|
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
|
||||||
path = os.path.join(tmp, "control_plane.sqlite3")
|
|
||||||
self.assertTrue(os.path.isdir(os.path.dirname(path)))
|
|
||||||
|
|
||||||
inventory = restart_console.read_control_plane_inventory(db_path=path)
|
|
||||||
|
|
||||||
self.assertFalse(
|
|
||||||
os.path.exists(path),
|
|
||||||
"reading restart status created a control-plane database",
|
|
||||||
)
|
|
||||||
self.assertFalse(inventory["inventory_complete"])
|
|
||||||
|
|
||||||
def test_reads_active_sessions_from_a_real_database(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
|
||||||
path = os.path.join(tmp, "cp.sqlite3")
|
|
||||||
conn = sqlite3.connect(path)
|
|
||||||
conn.execute(
|
|
||||||
"CREATE TABLE sessions (session_id TEXT, role TEXT, profile TEXT,"
|
|
||||||
" pid INTEGER, status TEXT, last_heartbeat_at TEXT)"
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"CREATE TABLE work_items (work_item_id INTEGER, kind TEXT,"
|
|
||||||
" number INTEGER)"
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"CREATE TABLE leases (lease_id TEXT, session_id TEXT, role TEXT,"
|
|
||||||
" phase TEXT, status TEXT, worktree_path TEXT,"
|
|
||||||
" work_item_id INTEGER, expires_at TEXT)"
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO sessions VALUES (?,?,?,?,?,?)",
|
|
||||||
("s-live", "author", "prgs-author", 4242, "active", NOW.isoformat()),
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO sessions VALUES (?,?,?,?,?,?)",
|
|
||||||
("s-done", "author", "prgs-author", 11, "closed", NOW.isoformat()),
|
|
||||||
)
|
|
||||||
conn.execute("INSERT INTO work_items VALUES (1, 'issue', 667)")
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO leases VALUES (?,?,?,?,?,?,?,?)",
|
|
||||||
(
|
|
||||||
"l-1",
|
|
||||||
"s-live",
|
|
||||||
"author",
|
|
||||||
"allocated",
|
|
||||||
"active",
|
|
||||||
None,
|
|
||||||
1,
|
|
||||||
NOW.isoformat(),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
inventory = restart_console.read_control_plane_inventory(db_path=path)
|
|
||||||
|
|
||||||
self.assertTrue(inventory["inventory_complete"])
|
|
||||||
self.assertEqual([s["session_id"] for s in inventory["sessions"]], ["s-live"])
|
|
||||||
self.assertEqual(inventory["leases"][0]["work_number"], 667)
|
|
||||||
|
|
||||||
|
|
||||||
class DrainAndReconcileTest(unittest.TestCase):
|
|
||||||
def test_absent_drain_proof_is_not_a_pass(self) -> None:
|
|
||||||
drain, source = restart_console.load_drain_status(proof=None, now=NOW)
|
|
||||||
self.assertIsNone(drain)
|
|
||||||
self.assertFalse(source.available)
|
|
||||||
self.assertIn("denies", source.detail)
|
|
||||||
|
|
||||||
def test_tampered_drain_proof_is_reported_invalid(self) -> None:
|
|
||||||
proof = drain_proof_fixture()
|
|
||||||
proof["clean"] = True
|
|
||||||
proof["proof_id"] = "0" * 64
|
|
||||||
drain, source = restart_console.load_drain_status(proof=proof, now=NOW)
|
|
||||||
self.assertTrue(source.available)
|
|
||||||
self.assertFalse(drain["valid"])
|
|
||||||
|
|
||||||
def test_absent_reconcile_proof_is_unavailable(self) -> None:
|
|
||||||
reconcile, source = restart_console.load_reconcile_status(load_proof=None)
|
|
||||||
self.assertIsNone(reconcile)
|
|
||||||
self.assertFalse(source.available)
|
|
||||||
|
|
||||||
def test_reconcile_proof_is_rendered_when_supplied(self) -> None:
|
|
||||||
payload = {
|
|
||||||
"overall_status": "degraded",
|
|
||||||
"mode": "log_only",
|
|
||||||
"resolved_count": 3,
|
|
||||||
"unresolved_count": 2,
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"dimension": "leases",
|
|
||||||
"status": "unresolved",
|
|
||||||
"summary": "2 orphaned leases",
|
|
||||||
"follow_up_required": True,
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
reconcile, source = restart_console.load_reconcile_status(
|
|
||||||
load_proof=lambda: payload
|
|
||||||
)
|
|
||||||
self.assertTrue(source.available)
|
|
||||||
self.assertEqual(reconcile["unresolved_count"], 2)
|
|
||||||
|
|
||||||
|
|
||||||
class RenderingTest(unittest.TestCase):
|
|
||||||
def _snapshot(self, **kwargs):
|
|
||||||
params = {
|
|
||||||
"principal": _principal(console_authz.OPERATOR),
|
|
||||||
"read_inventory": _inventory(sessions=[_live_session()]),
|
|
||||||
"now": NOW,
|
|
||||||
}
|
|
||||||
params.update(kwargs)
|
|
||||||
return restart_console.load_restart_console_snapshot(**params)
|
|
||||||
|
|
||||||
def test_page_renders_every_section(self) -> None:
|
|
||||||
html = restart_views.render_restart_console_page(self._snapshot())
|
|
||||||
for heading in (
|
|
||||||
"Impact preview",
|
|
||||||
"Drain proof",
|
|
||||||
"Post-restart reconcile",
|
|
||||||
"Restart classes",
|
|
||||||
"Approval controls",
|
|
||||||
"Break-glass",
|
|
||||||
):
|
|
||||||
self.assertIn(heading, html)
|
|
||||||
|
|
||||||
def test_hostile_session_id_is_escaped(self) -> None:
|
|
||||||
hostile = "<script>alert('x')</script>"
|
|
||||||
html = restart_views.render_restart_console_page(
|
|
||||||
self._snapshot(read_inventory=_inventory(sessions=[_live_session(hostile)]))
|
|
||||||
)
|
|
||||||
self.assertNotIn("<script>alert", html)
|
|
||||||
self.assertIn("<script>", html)
|
|
||||||
|
|
||||||
def test_unavailable_impact_says_unsafe_rather_than_clean(self) -> None:
|
|
||||||
def _boom(**_kwargs):
|
|
||||||
raise RuntimeError("nope")
|
|
||||||
|
|
||||||
snapshot = self._snapshot(read_inventory=_boom)
|
|
||||||
html = restart_views.render_restart_console_page(snapshot)
|
|
||||||
self.assertIn("blast radius of a restart is unknown", html)
|
|
||||||
self.assertIn("unavailable", html)
|
|
||||||
|
|
||||||
def test_break_glass_is_hidden_from_unprivileged_viewers(self) -> None:
|
|
||||||
viewer_html = restart_views.render_restart_console_page(
|
|
||||||
self._snapshot(principal=_principal(console_authz.VIEWER))
|
|
||||||
)
|
|
||||||
self.assertIn("visible to operator-class", viewer_html)
|
|
||||||
self.assertNotIn(
|
|
||||||
f"#{restart_console.BREAK_GLASS_ISSUE}", viewer_html
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_break_glass_shown_to_operator_is_marked_unavailable(self) -> None:
|
|
||||||
html = restart_views.render_restart_console_page(self._snapshot())
|
|
||||||
self.assertIn("unavailable", html)
|
|
||||||
self.assertIn(f"#{restart_console.BREAK_GLASS_ISSUE}", html)
|
|
||||||
|
|
||||||
def test_snapshot_always_declares_itself_read_only(self) -> None:
|
|
||||||
self.assertTrue(self._snapshot().read_only)
|
|
||||||
|
|
||||||
|
|
||||||
class RestartConsoleRouteTest(unittest.TestCase):
|
|
||||||
def setUp(self) -> None:
|
|
||||||
self.client = TestClient(create_app())
|
|
||||||
|
|
||||||
def test_page_route_renders(self) -> None:
|
|
||||||
res = self.client.get("/runtime/restart")
|
|
||||||
self.assertEqual(res.status_code, 200)
|
|
||||||
self.assertIn("Restart status and impact", res.text)
|
|
||||||
|
|
||||||
def test_api_route_exports_snapshot(self) -> None:
|
|
||||||
res = self.client.get("/api/v1/system/restart/status")
|
|
||||||
self.assertEqual(res.status_code, 200)
|
|
||||||
payload = res.json()
|
|
||||||
self.assertTrue(payload["read_only"])
|
|
||||||
self.assertEqual(payload["links"]["issue"], 667)
|
|
||||||
self.assertEqual(
|
|
||||||
len(payload["restart_classes"]),
|
|
||||||
len(restart_coordinator.RESTART_CLASS_POLICIES),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_restart_class_is_selectable(self) -> None:
|
|
||||||
res = self.client.get(
|
|
||||||
"/api/v1/system/restart/status?restart_class=client_reconnect"
|
|
||||||
)
|
|
||||||
self.assertEqual(res.status_code, 200)
|
|
||||||
self.assertEqual(res.json()["impact"]["restart_class"], "client_reconnect")
|
|
||||||
|
|
||||||
def test_unknown_restart_class_fails_closed(self) -> None:
|
|
||||||
res = self.client.get(
|
|
||||||
"/api/v1/system/restart/status?restart_class=obliterate-everything"
|
|
||||||
)
|
|
||||||
self.assertEqual(res.status_code, 200)
|
|
||||||
impact = res.json()["impact"]
|
|
||||||
self.assertFalse(impact["allow_restart"])
|
|
||||||
|
|
||||||
def test_anonymous_api_reader_gets_no_execution_grant(self) -> None:
|
|
||||||
payload = self.client.get("/api/v1/system/restart/status").json()
|
|
||||||
self.assertFalse(payload["break_glass"]["available"])
|
|
||||||
for auth in payload["authorizations"]:
|
|
||||||
self.assertFalse(auth["execution_enabled"])
|
|
||||||
|
|
||||||
def test_route_is_registered_in_nav(self) -> None:
|
|
||||||
from webui.nav import nav_hrefs
|
|
||||||
|
|
||||||
self.assertIn("/runtime/restart", nav_hrefs())
|
|
||||||
|
|
||||||
def test_no_write_method_is_exposed(self) -> None:
|
|
||||||
"""The surface is read-only: nothing accepts a POST."""
|
|
||||||
for path in ("/runtime/restart", "/api/v1/system/restart/status"):
|
|
||||||
self.assertEqual(self.client.post(path).status_code, 405, path)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -47,9 +47,6 @@ from webui.traffic_views import render_traffic_page
|
|||||||
from webui.worktree_scanner import load_hygiene_snapshot, snapshot_to_dict as worktree_snapshot_to_dict
|
from webui.worktree_scanner import load_hygiene_snapshot, snapshot_to_dict as worktree_snapshot_to_dict
|
||||||
from webui.worktree_views import render_worktrees_page
|
from webui.worktree_views import render_worktrees_page
|
||||||
from webui.runtime_health import load_runtime_snapshot, snapshot_to_dict as runtime_snapshot_to_dict
|
from webui.runtime_health import load_runtime_snapshot, snapshot_to_dict as runtime_snapshot_to_dict
|
||||||
import restart_coordinator
|
|
||||||
from webui.restart_console import load_restart_console_snapshot
|
|
||||||
from webui.restart_views import render_restart_console_page
|
|
||||||
from webui.runtime_views import render_runtime_page
|
from webui.runtime_views import render_runtime_page
|
||||||
from webui.session_loader import (
|
from webui.session_loader import (
|
||||||
load_session_view_snapshot,
|
load_session_view_snapshot,
|
||||||
@@ -334,33 +331,6 @@ async def api_runtime(_request: Request) -> JSONResponse:
|
|||||||
return JSONResponse(runtime_snapshot_to_dict(load_runtime_snapshot()))
|
return JSONResponse(runtime_snapshot_to_dict(load_runtime_snapshot()))
|
||||||
|
|
||||||
|
|
||||||
def _restart_console_snapshot(request: Request):
|
|
||||||
"""Build the read-only restart snapshot for the requesting principal (#667)."""
|
|
||||||
principal = resolve_principal(request.headers)
|
|
||||||
restart_class = (
|
|
||||||
request.query_params.get("restart_class")
|
|
||||||
or restart_coordinator.RestartClass.FULL_MCP_RESTART.value
|
|
||||||
)
|
|
||||||
return load_restart_console_snapshot(
|
|
||||||
principal=principal, restart_class=restart_class
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def restart_console_page(request: Request) -> HTMLResponse:
|
|
||||||
"""Restart status, impact preview, and approval state (#667). Read-only."""
|
|
||||||
snapshot = _restart_console_snapshot(request)
|
|
||||||
return HTMLResponse(
|
|
||||||
render_page(
|
|
||||||
title="Restart", body_html=render_restart_console_page(snapshot)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def api_restart_status(request: Request) -> JSONResponse:
|
|
||||||
"""JSON export of the read-only restart console snapshot (#667)."""
|
|
||||||
return JSONResponse(_restart_console_snapshot(request).as_dict())
|
|
||||||
|
|
||||||
|
|
||||||
async def sessions(_request: Request) -> HTMLResponse:
|
async def sessions(_request: Request) -> HTMLResponse:
|
||||||
"""Runtime and session view (#641) — read-only composition of health + inventory."""
|
"""Runtime and session view (#641) — read-only composition of health + inventory."""
|
||||||
snapshot = load_session_view_snapshot()
|
snapshot = load_session_view_snapshot()
|
||||||
@@ -811,13 +781,6 @@ def create_app(*, bind_host: str | None = None) -> Starlette:
|
|||||||
Route("/api/prompts", api_prompts, methods=["GET"]),
|
Route("/api/prompts", api_prompts, methods=["GET"]),
|
||||||
Route("/runtime", runtime, methods=["GET"]),
|
Route("/runtime", runtime, methods=["GET"]),
|
||||||
Route("/api/runtime", api_runtime, methods=["GET"]),
|
Route("/api/runtime", api_runtime, methods=["GET"]),
|
||||||
# #667 read-only restart status / impact preview / approval state.
|
|
||||||
Route("/runtime/restart", restart_console_page, methods=["GET"]),
|
|
||||||
Route(
|
|
||||||
"/api/v1/system/restart/status",
|
|
||||||
api_restart_status,
|
|
||||||
methods=["GET"],
|
|
||||||
),
|
|
||||||
Route("/sessions", sessions, methods=["GET"]),
|
Route("/sessions", sessions, methods=["GET"]),
|
||||||
Route("/api/sessions", api_sessions, methods=["GET"]),
|
Route("/api/sessions", api_sessions, methods=["GET"]),
|
||||||
Route("/api/v1/sessions", api_sessions, methods=["GET"]),
|
Route("/api/v1/sessions", api_sessions, methods=["GET"]),
|
||||||
|
|||||||
@@ -48,7 +48,6 @@ NAV_GROUPS: tuple[NavGroup, ...] = (
|
|||||||
)),
|
)),
|
||||||
NavGroup("Runtime/Sessions", (
|
NavGroup("Runtime/Sessions", (
|
||||||
NavItem("/runtime", "Runtime health"),
|
NavItem("/runtime", "Runtime health"),
|
||||||
NavItem("/runtime/restart", "Restart status"),
|
|
||||||
NavItem("/sessions", "Sessions"),
|
NavItem("/sessions", "Sessions"),
|
||||||
)),
|
)),
|
||||||
NavGroup("Projects", (
|
NavGroup("Projects", (
|
||||||
|
|||||||
@@ -1,579 +0,0 @@
|
|||||||
"""Read-only restart status, impact preview, and approval state (#667).
|
|
||||||
|
|
||||||
Phase 1 of the console restart surface. It *consumes* the #655 coordinator
|
|
||||||
substrate and renders it; it never restarts, reloads, drains, approves, or kills
|
|
||||||
anything. There is no apply path in this module, so there is no execution gate
|
|
||||||
here to arm incorrectly — the only writes the console could perform are the ones
|
|
||||||
it does not implement.
|
|
||||||
|
|
||||||
Sources, each independently fail-soft and each reported with its own
|
|
||||||
:class:`SourceStatus`:
|
|
||||||
|
|
||||||
* :mod:`restart_coordinator` — restart-class policy matrix (#663) and the
|
|
||||||
blast-radius impact report (#658).
|
|
||||||
* :mod:`drain_proof` — drain checklist and gate verdict (#661), verified
|
|
||||||
read-only against a caller-supplied proof.
|
|
||||||
* :mod:`post_restart_reconcile` — post-restart completion proof (#662).
|
|
||||||
* :mod:`webui.console_authz` — role authorization for the approval controls
|
|
||||||
(#633).
|
|
||||||
|
|
||||||
Three rules this module holds itself to, because a status surface that lies is
|
|
||||||
worse than one that is absent:
|
|
||||||
|
|
||||||
**A source that could not be read is reported unavailable, never green.** No
|
|
||||||
default, placeholder, or self-comparison is substituted for a reading that
|
|
||||||
failed. An unreadable control-plane DB yields ``inventory_complete=False``,
|
|
||||||
which the coordinator itself turns into a fail-closed verdict.
|
|
||||||
|
|
||||||
**Authorization is asked the way execution would ask it.** Every authorization
|
|
||||||
probe passes ``for_execution=True``, so the console reports whether the action
|
|
||||||
could actually run rather than the weaker "this principal is the right role".
|
|
||||||
While the console is in Phase 1 that answer is ``phase_not_active`` for every
|
|
||||||
phase-2 action, and the surface says so plainly instead of showing an allow.
|
|
||||||
|
|
||||||
**The database is opened read-only.** ``ControlPlaneDB()`` creates directories
|
|
||||||
and runs migrations on construction, which is a write; this module opens the
|
|
||||||
sqlite file with ``mode=ro`` exactly as :mod:`webui.inventory` does, and treats
|
|
||||||
a missing file as missing authority rather than an empty inventory.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sqlite3
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Any, Callable, Mapping
|
|
||||||
|
|
||||||
import control_plane_db
|
|
||||||
import drain_proof
|
|
||||||
import restart_coordinator
|
|
||||||
from webui import console_authz
|
|
||||||
from webui.inventory import redact_path, scrub
|
|
||||||
|
|
||||||
# --- Source status ----------------------------------------------------------
|
|
||||||
|
|
||||||
STATUS_OK = "ok"
|
|
||||||
STATUS_UNAVAILABLE = "unavailable"
|
|
||||||
|
|
||||||
#: Console actions whose authorization state this surface reports. Both are
|
|
||||||
#: pre-existing #642 actions; this module adds no new console action because it
|
|
||||||
#: performs no console action.
|
|
||||||
REPORTED_ACTIONS: tuple[str, ...] = (
|
|
||||||
"system.restart_namespace",
|
|
||||||
"system.reload_namespace",
|
|
||||||
)
|
|
||||||
|
|
||||||
#: The break-glass workflow (#664) is not consumed here. It is declared so the
|
|
||||||
#: surface is honest about the gap rather than silently omitting a governance
|
|
||||||
#: path the operator has been told exists.
|
|
||||||
BREAK_GLASS_ISSUE = 664
|
|
||||||
BREAK_GLASS_PENDING_REASON = (
|
|
||||||
"The break-glass workflow (#664) is not yet available on this branch's "
|
|
||||||
"base; no break-glass control is offered and none is implied."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class SourceStatus:
|
|
||||||
"""Whether one backing source could be read, and why not when it could not."""
|
|
||||||
|
|
||||||
name: str
|
|
||||||
status: str
|
|
||||||
detail: str = ""
|
|
||||||
|
|
||||||
@property
|
|
||||||
def available(self) -> bool:
|
|
||||||
return self.status == STATUS_OK
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"name": self.name,
|
|
||||||
"status": self.status,
|
|
||||||
"available": self.available,
|
|
||||||
"detail": self.detail,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RestartClassView:
|
|
||||||
"""One row of the #663 restart-class matrix, scoped to the viewer's role."""
|
|
||||||
|
|
||||||
restart_class: str
|
|
||||||
required_permission: str
|
|
||||||
expected_blast_radius: str
|
|
||||||
drain_requirement: str
|
|
||||||
full_drain_required: bool
|
|
||||||
approval_requirement: str
|
|
||||||
request_roles: tuple[str, ...]
|
|
||||||
execution_roles: tuple[str, ...]
|
|
||||||
viewer_may_request: bool
|
|
||||||
viewer_may_execute: bool
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"restart_class": self.restart_class,
|
|
||||||
"required_permission": self.required_permission,
|
|
||||||
"expected_blast_radius": self.expected_blast_radius,
|
|
||||||
"drain_requirement": self.drain_requirement,
|
|
||||||
"full_drain_required": self.full_drain_required,
|
|
||||||
"approval_requirement": self.approval_requirement,
|
|
||||||
"request_roles": list(self.request_roles),
|
|
||||||
"execution_roles": list(self.execution_roles),
|
|
||||||
"viewer_may_request": self.viewer_may_request,
|
|
||||||
"viewer_may_execute": self.viewer_may_execute,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ActionAuthorization:
|
|
||||||
"""Authorization state for one console action, asked as execution would."""
|
|
||||||
|
|
||||||
action_id: str
|
|
||||||
summary: str
|
|
||||||
required_role: str
|
|
||||||
allowed: bool
|
|
||||||
execution_enabled: bool
|
|
||||||
reason_code: str
|
|
||||||
detail: str
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"action_id": self.action_id,
|
|
||||||
"summary": self.summary,
|
|
||||||
"required_role": self.required_role,
|
|
||||||
"allowed": self.allowed,
|
|
||||||
"execution_enabled": self.execution_enabled,
|
|
||||||
"reason_code": self.reason_code,
|
|
||||||
"detail": self.detail,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BreakGlassSurface:
|
|
||||||
"""Declared-but-unavailable break-glass panel (#664 is not on this base)."""
|
|
||||||
|
|
||||||
available: bool
|
|
||||||
issue: int
|
|
||||||
reason: str
|
|
||||||
viewer_is_privileged: bool
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"available": self.available,
|
|
||||||
"issue": self.issue,
|
|
||||||
"reason": self.reason,
|
|
||||||
"viewer_is_privileged": self.viewer_is_privileged,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RestartConsoleSnapshot:
|
|
||||||
"""Everything the read-only restart console renders."""
|
|
||||||
|
|
||||||
generated_at: str
|
|
||||||
viewer_role: str
|
|
||||||
viewer_authenticated: bool
|
|
||||||
read_only: bool
|
|
||||||
impact: dict[str, Any] | None
|
|
||||||
impact_source: SourceStatus
|
|
||||||
drain: dict[str, Any] | None
|
|
||||||
drain_source: SourceStatus
|
|
||||||
reconcile: dict[str, Any] | None
|
|
||||||
reconcile_source: SourceStatus
|
|
||||||
restart_classes: tuple[RestartClassView, ...]
|
|
||||||
authorizations: tuple[ActionAuthorization, ...]
|
|
||||||
break_glass: BreakGlassSurface
|
|
||||||
notes: tuple[str, ...] = field(default_factory=tuple)
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"generated_at": self.generated_at,
|
|
||||||
"viewer_role": self.viewer_role,
|
|
||||||
"viewer_authenticated": self.viewer_authenticated,
|
|
||||||
"read_only": self.read_only,
|
|
||||||
"impact": self.impact,
|
|
||||||
"impact_source": self.impact_source.as_dict(),
|
|
||||||
"drain": self.drain,
|
|
||||||
"drain_source": self.drain_source.as_dict(),
|
|
||||||
"reconcile": self.reconcile,
|
|
||||||
"reconcile_source": self.reconcile_source.as_dict(),
|
|
||||||
"restart_classes": [c.as_dict() for c in self.restart_classes],
|
|
||||||
"authorizations": [a.as_dict() for a in self.authorizations],
|
|
||||||
"break_glass": self.break_glass.as_dict(),
|
|
||||||
"notes": list(self.notes),
|
|
||||||
"links": {
|
|
||||||
"issue": 667,
|
|
||||||
"extends": 642,
|
|
||||||
"umbrella": 655,
|
|
||||||
"coordinator": 658,
|
|
||||||
"drain_proof": 661,
|
|
||||||
"reconcile": 662,
|
|
||||||
"restart_classes": 663,
|
|
||||||
"break_glass": BREAK_GLASS_ISSUE,
|
|
||||||
"vision": 652,
|
|
||||||
"roadmap": 653,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _utc_now() -> datetime:
|
|
||||||
return datetime.now(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
# --- Control-plane inventory (read-only) ------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def read_control_plane_inventory(
|
|
||||||
*,
|
|
||||||
db_path: str | None = None,
|
|
||||||
limit: int = 200,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Read sessions and leases for an impact evaluation, read-only.
|
|
||||||
|
|
||||||
Returns the inventory mapping
|
|
||||||
:func:`restart_coordinator.evaluate_restart_impact` expects.
|
|
||||||
``inventory_complete`` is True only when every read succeeded, so a partial
|
|
||||||
read denies rather than under-reporting the blast radius.
|
|
||||||
|
|
||||||
The database is never created, migrated, or written: a missing file means
|
|
||||||
the console has no session authority, which is not the same as there being
|
|
||||||
no sessions.
|
|
||||||
"""
|
|
||||||
|
|
||||||
path = (db_path or control_plane_db.default_db_path() or "").strip()
|
|
||||||
incomplete: list[str] = []
|
|
||||||
|
|
||||||
def _incomplete(reason: str) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"sessions": [],
|
|
||||||
"leases": [],
|
|
||||||
"terminal_lock": None,
|
|
||||||
"prior_recovery_attempts": [],
|
|
||||||
"inventory_complete": False,
|
|
||||||
"incomplete_reasons": [reason],
|
|
||||||
}
|
|
||||||
|
|
||||||
if not path:
|
|
||||||
return _incomplete("control-plane database path is not configured")
|
|
||||||
if not os.path.exists(path):
|
|
||||||
return _incomplete(
|
|
||||||
f"control-plane database not present at {redact_path(path)}; "
|
|
||||||
"no session or lease authority available"
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=5)
|
|
||||||
conn.row_factory = sqlite3.Row
|
|
||||||
except sqlite3.Error as exc:
|
|
||||||
return _incomplete(f"control-plane database could not be opened: {exc}")
|
|
||||||
|
|
||||||
sessions: list[dict[str, Any]] = []
|
|
||||||
leases: list[dict[str, Any]] = []
|
|
||||||
capped = max(1, int(limit))
|
|
||||||
try:
|
|
||||||
tables = {
|
|
||||||
str(row[0])
|
|
||||||
for row in conn.execute(
|
|
||||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
|
||||||
).fetchall()
|
|
||||||
}
|
|
||||||
if "sessions" not in tables:
|
|
||||||
incomplete.append("control-plane database has no sessions table")
|
|
||||||
else:
|
|
||||||
sessions = [
|
|
||||||
dict(row)
|
|
||||||
for row in conn.execute(
|
|
||||||
"SELECT session_id, role, profile, pid, status,"
|
|
||||||
" last_heartbeat_at FROM sessions"
|
|
||||||
" WHERE status = 'active'"
|
|
||||||
" ORDER BY last_heartbeat_at DESC LIMIT ?",
|
|
||||||
(capped,),
|
|
||||||
).fetchall()
|
|
||||||
]
|
|
||||||
|
|
||||||
if "leases" not in tables:
|
|
||||||
incomplete.append("control-plane database has no leases table")
|
|
||||||
elif "work_items" not in tables:
|
|
||||||
incomplete.append(
|
|
||||||
"control-plane database has no work_items table; lease work "
|
|
||||||
"identity cannot be resolved"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
leases = [
|
|
||||||
dict(row)
|
|
||||||
for row in conn.execute(
|
|
||||||
"SELECT l.lease_id, l.session_id, l.role, l.phase,"
|
|
||||||
" l.status AS freshness, l.worktree_path,"
|
|
||||||
" w.kind AS work_kind, w.number AS work_number"
|
|
||||||
" FROM leases l"
|
|
||||||
" JOIN work_items w ON w.work_item_id = l.work_item_id"
|
|
||||||
" WHERE l.status = 'active'"
|
|
||||||
" ORDER BY l.expires_at DESC LIMIT ?",
|
|
||||||
(capped,),
|
|
||||||
).fetchall()
|
|
||||||
]
|
|
||||||
except sqlite3.Error as exc:
|
|
||||||
return _incomplete(f"control-plane database read failed: {exc}")
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
return {
|
|
||||||
"sessions": sessions,
|
|
||||||
"leases": leases,
|
|
||||||
"terminal_lock": None,
|
|
||||||
"prior_recovery_attempts": [],
|
|
||||||
"inventory_complete": not incomplete,
|
|
||||||
"incomplete_reasons": incomplete,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# --- Composition ------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def build_restart_class_views(viewer_role: str | None) -> tuple[RestartClassView, ...]:
|
|
||||||
"""Render the #663 class matrix, marking what this viewer may request."""
|
|
||||||
|
|
||||||
normalized = str(viewer_role or "").strip().lower()
|
|
||||||
views: list[RestartClassView] = []
|
|
||||||
for policy in restart_coordinator.RESTART_CLASS_POLICIES.values():
|
|
||||||
views.append(
|
|
||||||
RestartClassView(
|
|
||||||
restart_class=policy.restart_class.value,
|
|
||||||
required_permission=policy.required_permission,
|
|
||||||
expected_blast_radius=policy.expected_blast_radius,
|
|
||||||
drain_requirement=policy.drain_requirement,
|
|
||||||
full_drain_required=policy.full_drain_required,
|
|
||||||
approval_requirement=policy.approval_requirement,
|
|
||||||
request_roles=tuple(policy.request_roles),
|
|
||||||
execution_roles=tuple(policy.execution_roles),
|
|
||||||
viewer_may_request=normalized in policy.request_roles,
|
|
||||||
viewer_may_execute=normalized in policy.execution_roles,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(views)
|
|
||||||
|
|
||||||
|
|
||||||
def build_action_authorizations(
|
|
||||||
principal: console_authz.Principal | None,
|
|
||||||
) -> tuple[ActionAuthorization, ...]:
|
|
||||||
"""Authorization state for the approval controls, asked as execution.
|
|
||||||
|
|
||||||
``for_execution=True`` is deliberate. Asking without it answers "is this
|
|
||||||
principal senior enough", which is not the question an operator looking at a
|
|
||||||
control needs answered; asking with it answers "would this run", and while
|
|
||||||
the console is in Phase 1 the honest answer is no.
|
|
||||||
"""
|
|
||||||
|
|
||||||
results: list[ActionAuthorization] = []
|
|
||||||
for action_id in REPORTED_ACTIONS:
|
|
||||||
action = console_authz.get_action(action_id)
|
|
||||||
decision = console_authz.authorize(action_id, principal, for_execution=True)
|
|
||||||
results.append(
|
|
||||||
ActionAuthorization(
|
|
||||||
action_id=action_id,
|
|
||||||
summary=action.summary if action else "",
|
|
||||||
required_role=(
|
|
||||||
action.minimum_role if action else console_authz.OPERATOR
|
|
||||||
),
|
|
||||||
allowed=bool(decision.allowed),
|
|
||||||
execution_enabled=bool(decision.execution_enabled),
|
|
||||||
reason_code=str(decision.reason_code or ""),
|
|
||||||
detail=str(decision.detail or ""),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(results)
|
|
||||||
|
|
||||||
|
|
||||||
def viewer_is_privileged(principal: console_authz.Principal | None) -> bool:
|
|
||||||
"""True when the viewer holds at least the operator role."""
|
|
||||||
|
|
||||||
who = principal if principal is not None else console_authz.ANONYMOUS
|
|
||||||
if not who.authenticated:
|
|
||||||
return False
|
|
||||||
return who.rank >= console_authz.ROLE_ORDER.index(console_authz.OPERATOR)
|
|
||||||
|
|
||||||
|
|
||||||
def load_impact_report(
|
|
||||||
*,
|
|
||||||
principal: console_authz.Principal | None = None,
|
|
||||||
restart_class: str = restart_coordinator.RestartClass.FULL_MCP_RESTART.value,
|
|
||||||
db_path: str | None = None,
|
|
||||||
limit: int = 200,
|
|
||||||
read_inventory: Callable[..., Mapping[str, Any]] | None = None,
|
|
||||||
now: datetime | None = None,
|
|
||||||
) -> tuple[dict[str, Any] | None, SourceStatus]:
|
|
||||||
"""Evaluate the blast radius for *restart_class*, always dry-run."""
|
|
||||||
|
|
||||||
reader = read_inventory or read_control_plane_inventory
|
|
||||||
try:
|
|
||||||
inventory = dict(reader(db_path=db_path, limit=limit))
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return None, SourceStatus(
|
|
||||||
"impact",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
f"control-plane inventory failed: {type(exc).__name__}: {exc}",
|
|
||||||
)
|
|
||||||
|
|
||||||
who = principal if principal is not None else console_authz.ANONYMOUS
|
|
||||||
viewer_role = str(who.role or "").strip().lower()
|
|
||||||
try:
|
|
||||||
report = restart_coordinator.evaluate_restart_impact(
|
|
||||||
inventory,
|
|
||||||
now=now,
|
|
||||||
dry_run=True,
|
|
||||||
restart_class=restart_class,
|
|
||||||
requester_role=viewer_role,
|
|
||||||
requester_permissions=restart_coordinator.permissions_for_role(
|
|
||||||
viewer_role
|
|
||||||
),
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return None, SourceStatus(
|
|
||||||
"impact",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
f"impact evaluation failed: {type(exc).__name__}: {exc}",
|
|
||||||
)
|
|
||||||
|
|
||||||
payload = scrub(report.as_dict())
|
|
||||||
detail = ""
|
|
||||||
if not report.inventory_complete:
|
|
||||||
detail = "; ".join(report.incomplete_reasons) or "inventory incomplete"
|
|
||||||
return payload, SourceStatus("impact", STATUS_OK, detail)
|
|
||||||
|
|
||||||
|
|
||||||
def load_drain_status(
|
|
||||||
*,
|
|
||||||
proof: Mapping[str, Any] | None = None,
|
|
||||||
now: datetime | None = None,
|
|
||||||
expected_impact_fingerprint: str | None = None,
|
|
||||||
) -> tuple[dict[str, Any] | None, SourceStatus]:
|
|
||||||
"""Verify a supplied drain proof read-only and report the verdict.
|
|
||||||
|
|
||||||
No proof supplied is not a failure and not a pass: it is reported as the
|
|
||||||
absence of a proof, which is exactly what the #661 gate would deny on.
|
|
||||||
"""
|
|
||||||
|
|
||||||
if proof is None:
|
|
||||||
return None, SourceStatus(
|
|
||||||
"drain",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
"no drain proof supplied; the #661 gate denies a restart without a "
|
|
||||||
"valid unexpired clean proof",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
verified = drain_proof.verify_drain_proof(
|
|
||||||
proof,
|
|
||||||
now=now,
|
|
||||||
expected_impact_fingerprint=expected_impact_fingerprint,
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return None, SourceStatus(
|
|
||||||
"drain",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
f"drain proof verification failed: {type(exc).__name__}: {exc}",
|
|
||||||
)
|
|
||||||
return scrub(verified.as_dict()), SourceStatus("drain", STATUS_OK)
|
|
||||||
|
|
||||||
|
|
||||||
def load_reconcile_status(
|
|
||||||
*,
|
|
||||||
load_proof: Callable[[], Any] | None = None,
|
|
||||||
) -> tuple[dict[str, Any] | None, SourceStatus]:
|
|
||||||
"""Report the most recent post-restart completion proof (#662)."""
|
|
||||||
|
|
||||||
if load_proof is None:
|
|
||||||
return None, SourceStatus(
|
|
||||||
"reconcile",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
"no post-restart completion proof source is wired into this view",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
proof = load_proof()
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
return None, SourceStatus(
|
|
||||||
"reconcile",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
f"reconcile proof unavailable: {type(exc).__name__}: {exc}",
|
|
||||||
)
|
|
||||||
if proof is None:
|
|
||||||
return None, SourceStatus(
|
|
||||||
"reconcile",
|
|
||||||
STATUS_UNAVAILABLE,
|
|
||||||
"no post-restart reconcile has been recorded",
|
|
||||||
)
|
|
||||||
payload = proof.as_dict() if hasattr(proof, "as_dict") else dict(proof)
|
|
||||||
return scrub(payload), SourceStatus("reconcile", STATUS_OK)
|
|
||||||
|
|
||||||
|
|
||||||
def load_restart_console_snapshot(
|
|
||||||
*,
|
|
||||||
principal: console_authz.Principal | None = None,
|
|
||||||
restart_class: str = restart_coordinator.RestartClass.FULL_MCP_RESTART.value,
|
|
||||||
db_path: str | None = None,
|
|
||||||
limit: int = 200,
|
|
||||||
drain_proof_payload: Mapping[str, Any] | None = None,
|
|
||||||
read_inventory: Callable[..., Mapping[str, Any]] | None = None,
|
|
||||||
load_reconcile_proof: Callable[[], Any] | None = None,
|
|
||||||
now: datetime | None = None,
|
|
||||||
) -> RestartConsoleSnapshot:
|
|
||||||
"""Compose the read-only restart console snapshot."""
|
|
||||||
|
|
||||||
who = principal if principal is not None else console_authz.ANONYMOUS
|
|
||||||
moment = now or _utc_now()
|
|
||||||
|
|
||||||
impact, impact_source = load_impact_report(
|
|
||||||
principal=who,
|
|
||||||
restart_class=restart_class,
|
|
||||||
db_path=db_path,
|
|
||||||
limit=limit,
|
|
||||||
read_inventory=read_inventory,
|
|
||||||
now=moment,
|
|
||||||
)
|
|
||||||
fingerprint = None
|
|
||||||
if impact is not None:
|
|
||||||
try:
|
|
||||||
fingerprint = drain_proof.impact_fingerprint(impact)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
fingerprint = None
|
|
||||||
|
|
||||||
drain, drain_source = load_drain_status(
|
|
||||||
proof=drain_proof_payload,
|
|
||||||
now=moment,
|
|
||||||
expected_impact_fingerprint=fingerprint,
|
|
||||||
)
|
|
||||||
reconcile, reconcile_source = load_reconcile_status(
|
|
||||||
load_proof=load_reconcile_proof
|
|
||||||
)
|
|
||||||
|
|
||||||
notes: list[str] = [
|
|
||||||
"This surface is read-only: it evaluates and displays, and performs no "
|
|
||||||
"restart, reload, drain, approval, or process action.",
|
|
||||||
]
|
|
||||||
if not impact_source.available:
|
|
||||||
notes.append(
|
|
||||||
"Impact preview unavailable — a restart decision must not be made "
|
|
||||||
"from this page while the blast radius is unknown."
|
|
||||||
)
|
|
||||||
|
|
||||||
return RestartConsoleSnapshot(
|
|
||||||
generated_at=moment.isoformat(),
|
|
||||||
viewer_role=str(who.role or "anonymous"),
|
|
||||||
viewer_authenticated=bool(who.authenticated),
|
|
||||||
read_only=True,
|
|
||||||
impact=impact,
|
|
||||||
impact_source=impact_source,
|
|
||||||
drain=drain,
|
|
||||||
drain_source=drain_source,
|
|
||||||
reconcile=reconcile,
|
|
||||||
reconcile_source=reconcile_source,
|
|
||||||
restart_classes=build_restart_class_views(who.role),
|
|
||||||
authorizations=build_action_authorizations(who),
|
|
||||||
break_glass=BreakGlassSurface(
|
|
||||||
available=False,
|
|
||||||
issue=BREAK_GLASS_ISSUE,
|
|
||||||
reason=BREAK_GLASS_PENDING_REASON,
|
|
||||||
viewer_is_privileged=viewer_is_privileged(who),
|
|
||||||
),
|
|
||||||
notes=tuple(notes),
|
|
||||||
)
|
|
||||||
@@ -1,299 +0,0 @@
|
|||||||
"""HTML views for the read-only restart console (#667).
|
|
||||||
|
|
||||||
Every interpolated value passes through :func:`_esc`. Values that can carry a
|
|
||||||
filesystem path or free-form operator text additionally pass through
|
|
||||||
:func:`webui.inventory.scrub_text`, which redacts credential-shaped tokens
|
|
||||||
*inside* a string rather than only at its start.
|
|
||||||
|
|
||||||
The page renders state and never offers a control that would mutate anything:
|
|
||||||
the approval and break-glass panels report authorization and availability, and
|
|
||||||
there is no form, button, or endpoint behind them.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import html
|
|
||||||
|
|
||||||
from webui.inventory import scrub_text
|
|
||||||
from webui.restart_console import RestartConsoleSnapshot, SourceStatus
|
|
||||||
|
|
||||||
|
|
||||||
def _esc(value: object) -> str:
|
|
||||||
"""Escape any value for HTML text or a quoted attribute."""
|
|
||||||
if value is None:
|
|
||||||
return ""
|
|
||||||
return html.escape(str(value), quote=True)
|
|
||||||
|
|
||||||
|
|
||||||
def _esc_text(value: object) -> str:
|
|
||||||
"""Escape free-form text after redacting secrets embedded inside it."""
|
|
||||||
if value is None:
|
|
||||||
return ""
|
|
||||||
return _esc(scrub_text(str(value)))
|
|
||||||
|
|
||||||
|
|
||||||
def _bool_badge(
|
|
||||||
value: bool, *, true_label: str = "yes", false_label: str = "no"
|
|
||||||
) -> str:
|
|
||||||
css = "badge-ok" if value else "badge-blocked"
|
|
||||||
label = true_label if value else false_label
|
|
||||||
return f'<span class="badge {css}">{_esc(label)}</span>'
|
|
||||||
|
|
||||||
|
|
||||||
def _source_badge(source: SourceStatus) -> str:
|
|
||||||
css = "badge-ok" if source.available else "badge-blocked"
|
|
||||||
badge = f'<span class="badge {css}">{_esc(source.status)}</span>'
|
|
||||||
if source.detail:
|
|
||||||
badge += f' <span class="muted">{_esc_text(source.detail)}</span>'
|
|
||||||
return badge
|
|
||||||
|
|
||||||
|
|
||||||
def _notes_block(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
if not snapshot.notes:
|
|
||||||
return ""
|
|
||||||
items = "".join(f"<li>{_esc_text(note)}</li>" for note in snapshot.notes)
|
|
||||||
return f"<ul class='reasons'>{items}</ul>"
|
|
||||||
|
|
||||||
|
|
||||||
def _impact_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
head = (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
f"<h3>Impact preview {_source_badge(snapshot.impact_source)}</h3>"
|
|
||||||
)
|
|
||||||
impact = snapshot.impact
|
|
||||||
if impact is None:
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<p class='muted'>No impact preview is available, so the blast "
|
|
||||||
"radius of a restart is unknown. Treat this as unsafe.</p></section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
counts = impact.get("counts") or {}
|
|
||||||
verdict = str(impact.get("verdict") or "unknown")
|
|
||||||
verdict_css = "badge-ok" if verdict == "safe" else "badge-blocked"
|
|
||||||
rows = "".join(
|
|
||||||
f"<tr><th>{_esc(key.replace('_', ' '))}</th><td>{_esc(value)}</td></tr>"
|
|
||||||
for key, value in sorted(counts.items())
|
|
||||||
)
|
|
||||||
reasons = "".join(
|
|
||||||
f"<li>{_esc_text(reason)}</li>" for reason in (impact.get("reasons") or [])
|
|
||||||
)
|
|
||||||
incomplete = ""
|
|
||||||
if not impact.get("inventory_complete", False):
|
|
||||||
detail = "; ".join(str(r) for r in (impact.get("incomplete_reasons") or []))
|
|
||||||
incomplete = (
|
|
||||||
"<p class='error'><strong>Inventory incomplete:</strong> "
|
|
||||||
f"{_esc_text(detail or 'unspecified')}. The coordinator fails "
|
|
||||||
"closed on an incomplete inventory.</p>"
|
|
||||||
)
|
|
||||||
|
|
||||||
sessions = impact.get("affected_sessions") or []
|
|
||||||
session_rows = "".join(
|
|
||||||
"<tr>"
|
|
||||||
f"<td><code>{_esc(s.get('session_id'))}</code></td>"
|
|
||||||
f"<td>{_esc(s.get('role'))}</td>"
|
|
||||||
f"<td>{_esc(s.get('pid'))}</td>"
|
|
||||||
f"<td>{_bool_badge(bool(s.get('live')), true_label='live', false_label='idle')}</td>"
|
|
||||||
f"<td>{_bool_badge(not s.get('heartbeat_stale'), true_label='fresh', false_label='stale')}</td>"
|
|
||||||
"</tr>"
|
|
||||||
for s in sessions[:50]
|
|
||||||
)
|
|
||||||
session_table = (
|
|
||||||
"<h4>Sessions a restart would terminate</h4>"
|
|
||||||
"<div class='table-scroll'><table class='registry'><thead><tr>"
|
|
||||||
"<th>Session</th><th>Role</th><th>PID</th><th>State</th>"
|
|
||||||
"<th>Heartbeat</th></tr></thead><tbody>"
|
|
||||||
f"{session_rows}</tbody></table></div>"
|
|
||||||
if session_rows
|
|
||||||
else "<p class='muted'>No affected sessions reported.</p>"
|
|
||||||
)
|
|
||||||
truncated = (
|
|
||||||
f"<p class='muted'>Showing the first 50 of {_esc(len(sessions))} "
|
|
||||||
"affected sessions.</p>"
|
|
||||||
if len(sessions) > 50
|
|
||||||
else ""
|
|
||||||
)
|
|
||||||
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<p class='health-headline'>Verdict "
|
|
||||||
f"<span class='badge {verdict_css}'>{_esc(verdict)}</span> · "
|
|
||||||
f"blast radius <code>{_esc(impact.get('blast_radius'))}</code> · "
|
|
||||||
f"class <code>{_esc(impact.get('restart_class'))}</code></p>"
|
|
||||||
+ incomplete
|
|
||||||
+ (f"<ul class='reasons'>{reasons}</ul>" if reasons else "")
|
|
||||||
+ (f"<table class='registry'><tbody>{rows}</tbody></table>" if rows else "")
|
|
||||||
+ session_table
|
|
||||||
+ truncated
|
|
||||||
+ "</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _drain_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
head = (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
f"<h3>Drain proof {_source_badge(snapshot.drain_source)}</h3>"
|
|
||||||
)
|
|
||||||
drain = snapshot.drain
|
|
||||||
if drain is None:
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<p class='muted'>No drain proof has been presented to this view. "
|
|
||||||
"The #661 gate authorizes a restart only against a valid, unexpired, "
|
|
||||||
"clean proof, so the absence of one is a denial, not a pass.</p>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
reasons = "".join(
|
|
||||||
f"<li>{_esc_text(reason)}</li>" for reason in (drain.get("reasons") or [])
|
|
||||||
)
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<table class='registry'><tbody>"
|
|
||||||
f"<tr><th>Valid</th><td>{_bool_badge(bool(drain.get('valid')))}</td></tr>"
|
|
||||||
f"<tr><th>Clean</th><td>{_bool_badge(bool(drain.get('clean')))}</td></tr>"
|
|
||||||
f"<tr><th>Expired</th><td>{_bool_badge(not drain.get('expired'), true_label='no', false_label='yes')}</td></tr>"
|
|
||||||
f"<tr><th>Tampered</th><td>{_bool_badge(not drain.get('tampered'), true_label='no', false_label='yes')}</td></tr>"
|
|
||||||
f"<tr><th>Proof id</th><td><code>{_esc(drain.get('proof_id'))}</code></td></tr>"
|
|
||||||
"</tbody></table>"
|
|
||||||
+ (f"<ul class='reasons'>{reasons}</ul>" if reasons else "")
|
|
||||||
+ "</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _reconcile_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
head = (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
f"<h3>Post-restart reconcile {_source_badge(snapshot.reconcile_source)}</h3>"
|
|
||||||
)
|
|
||||||
proof = snapshot.reconcile
|
|
||||||
if proof is None:
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<p class='muted'>No post-restart completion proof is recorded. "
|
|
||||||
"Until one is, the last restart's recovery state is unproven.</p>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
items = "".join(
|
|
||||||
"<tr>"
|
|
||||||
f"<td>{_esc(item.get('dimension'))}</td>"
|
|
||||||
f"<td>{_esc(item.get('status'))}</td>"
|
|
||||||
f"<td>{_esc_text(item.get('summary'))}</td>"
|
|
||||||
f"<td>{_bool_badge(not item.get('follow_up_required'), true_label='no', false_label='yes')}</td>"
|
|
||||||
"</tr>"
|
|
||||||
for item in (proof.get("items") or [])
|
|
||||||
)
|
|
||||||
return (
|
|
||||||
head
|
|
||||||
+ "<p class='health-headline'>Status "
|
|
||||||
f"<code>{_esc(proof.get('overall_status'))}</code> · mode "
|
|
||||||
f"<code>{_esc(proof.get('mode'))}</code> · resolved "
|
|
||||||
f"{_esc(proof.get('resolved_count'))} · unresolved "
|
|
||||||
f"{_esc(proof.get('unresolved_count'))}</p>"
|
|
||||||
+ (
|
|
||||||
"<div class='table-scroll'><table class='registry'><thead><tr>"
|
|
||||||
"<th>Dimension</th><th>Status</th><th>Summary</th>"
|
|
||||||
"<th>Follow-up required</th></tr></thead><tbody>"
|
|
||||||
f"{items}</tbody></table></div>"
|
|
||||||
if items
|
|
||||||
else "<p class='muted'>No reconcile dimensions reported.</p>"
|
|
||||||
)
|
|
||||||
+ "</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _class_matrix_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
rows = "".join(
|
|
||||||
"<tr>"
|
|
||||||
f"<td><code>{_esc(view.restart_class)}</code></td>"
|
|
||||||
f"<td><code>{_esc(view.required_permission)}</code></td>"
|
|
||||||
f"<td>{_esc(view.expected_blast_radius)}</td>"
|
|
||||||
f"<td>{_esc(view.drain_requirement)}</td>"
|
|
||||||
f"<td>{_esc(view.approval_requirement)}</td>"
|
|
||||||
f"<td>{_bool_badge(view.viewer_may_request)}</td>"
|
|
||||||
f"<td>{_bool_badge(view.viewer_may_execute)}</td>"
|
|
||||||
"</tr>"
|
|
||||||
for view in snapshot.restart_classes
|
|
||||||
)
|
|
||||||
return (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
"<h3>Restart classes</h3>"
|
|
||||||
"<p class='muted'>The least-privilege matrix each restart request is "
|
|
||||||
"resolved against. “You may request” and “you may "
|
|
||||||
"execute” are computed for the current viewer role, not for a "
|
|
||||||
"generic operator.</p>"
|
|
||||||
"<div class='table-scroll'><table class='registry'><thead><tr>"
|
|
||||||
"<th>Class</th><th>Permission</th><th>Blast radius</th>"
|
|
||||||
"<th>Drain</th><th>Approval</th><th>You may request</th>"
|
|
||||||
"<th>You may execute</th></tr></thead><tbody>"
|
|
||||||
f"{rows}</tbody></table></div>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _approval_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
rows = "".join(
|
|
||||||
"<tr>"
|
|
||||||
f"<td><code>{_esc(a.action_id)}</code></td>"
|
|
||||||
f"<td>{_esc(a.required_role)}</td>"
|
|
||||||
f"<td>{_bool_badge(a.allowed)}</td>"
|
|
||||||
f"<td>{_bool_badge(a.execution_enabled)}</td>"
|
|
||||||
f"<td><code>{_esc(a.reason_code)}</code></td>"
|
|
||||||
f"<td>{_esc_text(a.detail)}</td>"
|
|
||||||
"</tr>"
|
|
||||||
for a in snapshot.authorizations
|
|
||||||
)
|
|
||||||
return (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
"<h3>Approval controls</h3>"
|
|
||||||
"<p class='muted'>Authorization is probed the way execution would probe "
|
|
||||||
"it, so “execution enabled” answers whether the action would "
|
|
||||||
"actually run — not merely whether this role outranks the requirement. "
|
|
||||||
"No control on this page performs the action.</p>"
|
|
||||||
"<div class='table-scroll'><table class='registry'><thead><tr>"
|
|
||||||
"<th>Action</th><th>Required role</th><th>Authorized</th>"
|
|
||||||
"<th>Execution enabled</th><th>Reason</th><th>Detail</th>"
|
|
||||||
"</tr></thead><tbody>"
|
|
||||||
f"{rows}</tbody></table></div>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _break_glass_section(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
bg = snapshot.break_glass
|
|
||||||
if not bg.viewer_is_privileged:
|
|
||||||
return (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
"<h3>Break-glass</h3>"
|
|
||||||
"<p class='muted'>Break-glass status is visible to operator-class "
|
|
||||||
"roles only. Your role does not carry that authority, so no "
|
|
||||||
"emergency surface is shown.</p>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
return (
|
|
||||||
"<section class='health-card'>"
|
|
||||||
"<h3>Break-glass "
|
|
||||||
f"{_bool_badge(bg.available, true_label='available', false_label='unavailable')}"
|
|
||||||
"</h3>"
|
|
||||||
f"<p class='muted'>{_esc_text(bg.reason)}</p>"
|
|
||||||
f"<p class='meta'>Tracked by issue #{_esc(bg.issue)}.</p>"
|
|
||||||
"</section>"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def render_restart_console_page(snapshot: RestartConsoleSnapshot) -> str:
|
|
||||||
"""Render the whole read-only restart console body."""
|
|
||||||
|
|
||||||
return (
|
|
||||||
"<h2>Restart status and impact</h2>"
|
|
||||||
f"<p class='meta'>Generated <code>{_esc(snapshot.generated_at)}</code> · "
|
|
||||||
f"viewer role <code>{_esc(snapshot.viewer_role)}</code> · "
|
|
||||||
f"authenticated {_bool_badge(snapshot.viewer_authenticated)} · "
|
|
||||||
f"read-only {_bool_badge(snapshot.read_only)}</p>"
|
|
||||||
+ _notes_block(snapshot)
|
|
||||||
+ _impact_section(snapshot)
|
|
||||||
+ _drain_section(snapshot)
|
|
||||||
+ _reconcile_section(snapshot)
|
|
||||||
+ _class_matrix_section(snapshot)
|
|
||||||
+ _approval_section(snapshot)
|
|
||||||
+ _break_glass_section(snapshot)
|
|
||||||
)
|
|
||||||
Reference in New Issue
Block a user