Compare commits

..
Author SHA1 Message Date
sysadmin 1c88b87ec5 feat(webui): implement Phase 2 recovery controls & playbooks (#644) 2026-07-25 08:19:06 -04:00
sysadmin 76f293eb28 Merge pull request 'fix(gate): stop classifying stale-runtime blocks as permission denials (Closes #897)' (#901) from fix/issue-897-permission-stale-runtime-classification into master 2026-07-25 06:56:16 -05:00
jcwalker3 54559aebc3 Merge branch 'master' into fix/issue-897-permission-stale-runtime-classification 2026-07-25 06:42:27 -05:00
sysadmin 715863799f Merge pull request 'feat(webui): Runtime and session view (Phase 1) (Closes #641)' (#898) from feat/issue-641-runtime-session-view into master 2026-07-25 05:50:48 -05:00
sysadmin 6e6ca94338 fix(gate): stop classifying stale-runtime blocks as permission denials (Closes #897)
Stale-runtime and runtime-mode mutation refusals previously shared the
permission-denial channel, so permission_report claimed a missing op the
active profile already held and recommended gitea_activate_profile.
Typed blocker_kind payloads report reconnect-only recovery for staleness,
omit permission_report for non-permission gates, and fail closed when a
permission_report would invent a missing permission the profile holds.
2026-07-25 01:47:35 -04:00
11 changed files with 1747 additions and 90 deletions
+62
View File
@@ -0,0 +1,62 @@
# Sanctioned Recovery Playbooks & Controls (Phase 2 #644)
## Overview
Stale runtimes, worktree binding mismatches, and un-reconciled merged branches previously required expert manual shell recovery. Manual process kills (`pkill -f mcp_server.py`) are strictly forbidden and classified as runtime contamination ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)).
Phase 2 introduces **sanctioned recovery playbooks and controls** into the Web Console:
- **Diagnose**: Surface stale runtimes, worktree binding errors, contamination markers, and worktree anomalies via health & inventory APIs.
- **Preview**: Render mutation ledgers and exact confirmation phrases for recovery playbooks.
- **Confirm & Apply**: Execute sanctioned recovery actions through gated, audited paths.
- **Verify**: Revalidate control-plane state post-recovery before claiming clean status.
---
## Recovery Playbook Taxonomy
| Playbook ID | Action ID | Minimum Role | Target / Scope | Description |
|---|---|---|---|---|
| `clear_stale_binding` | `system.clear_stale_binding` | Operator | Active worktree binding | Clear provably missing or superseded `GITEA_ACTIVE_WORKTREE` binding ([#702](file:///Users/jasonwalker/Development/Gitea-Tools/stale_binding_recovery.py)). |
| `rebind_session_worktree` | `system.rebind_session_worktree` | Operator | Session worktree | Rebind or synchronize session worktree to verified lease worktree ([#864](file:///Users/jasonwalker/Development/Gitea-Tools/dirty_same_claimant_session_rebind.py)). |
| `reconcile_cleanups` | `system.reconcile_cleanups` | Controller | Worktree hygiene | Execute reconciler cleanup preview and apply for merged/superseded PR branches. |
| `sanctioned_restart` | `system.restart_namespace` | Admin | MCP Namespace | Restart MCP daemon gracefully via host supervisor ([#642](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)). |
---
## Wizard Workflow (Diagnose → Preview → Confirm → Verify)
### 1. Diagnose (`GET /api/v1/system/recovery/diagnose`)
Runs control-plane diagnostics:
- **Stale Runtime**: Mismatch between running daemon HEAD, local checkout HEAD, and remote-tracking HEAD.
- **Worktree Binding**: Missing path (`provably_stale_missing_path`), unverified inherited binding (`unverified_inherited`), or superseded binding (`superseded_by_session_lease`).
- **Contamination**: Checks for live contamination markers from unmanaged process kills.
- **Worktree Anomalies**: Scans `branches/` directory for un-reconciled cleanups or missing preserved worktrees.
Returns `RecoveryDiagnosis` with eligible playbooks.
### 2. Preview (`POST /api/v1/system/recovery/preview`)
Takes `playbook_id` and optional `target`/`params`.
Returns:
- **Mutation Ledger**: Step-by-step sequence of actions.
- **Confirmation Phrase**: Exact phrase required to authorize execution (e.g., `confirm clear_stale_binding`).
- **Authorization Decision**: RBAC check against the operator's principal.
### 3. Apply (`POST /api/v1/system/recovery/apply`)
Requires `playbook_id` and matching `confirmation` phrase.
- Validates RBAC permissions (`console_authz`).
- Verifies confirmation phrase (`confirmation_matches`).
- Enforces contamination rules ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)): A contaminated runtime must be cleared through reconciler cleanup before other playbooks run.
- Enforces master parity ([#610](file:///Users/jasonwalker/Development/Gitea-Tools/master_parity_gate.py)).
- Applies sanctioned recovery logic.
- Logs audit record in `console_audit`.
### 4. Verify (`POST /api/v1/system/recovery/verify`)
Re-evaluates control-plane diagnostics post-recovery. Asserts `clean: true` before transitioning out of recovery mode.
---
## Safety & Governance Principles
1. **No Manual `pkill`**: Direct process killing remains forbidden and is recorded as contamination.
2. **Auditability**: Every recovery preview and execution is logged in the console audit trail.
3. **Master Parity & Dual Control**: High-privilege recovery actions require controller/admin roles and explicit confirmation phrases.
+3
View File
@@ -94,6 +94,9 @@ already define, and a regression test asserts each mapping matches.
| `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 | | `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 |
| `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 | | `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 |
| `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 | | `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 |
| `system.clear_stale_binding` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
**Dual control** means the acting principal may not be the sole authority: a **Dual control** means the acting principal may not be the sole authority: a
second distinct principal must confirm. **Break-glass** means the action is second distinct principal must confirm. **Break-glass** means the action is
+296 -70
View File
@@ -9552,15 +9552,13 @@ def gitea_edit_pr(
if closing: if closing:
gate_reasons = _profile_operation_gate("gitea.pr.close") gate_reasons = _profile_operation_gate("gitea.pr.close")
if gate_reasons: if gate_reasons:
return { return _build_operation_gate_refusal(
"success": False, "gitea.pr.close",
"performed": False, gate_reasons,
"pr_number": pr_number, pr_number=pr_number,
"requested_state": "closed", requested_state="closed",
"required_permission": "gitea.pr.close", required_permission="gitea.pr.close",
"reasons": gate_reasons, )
"permission_report": _permission_block_report("gitea.pr.close"),
}
h, o, r = _resolve(remote, host, org, repo) h, o, r = _resolve(remote, host, org, repo)
auth = _auth(h) auth = _auth(h)
@@ -13823,13 +13821,17 @@ def gitea_view_issue(
def _permission_block_report(required_operation: str, def _permission_block_report(required_operation: str,
identity: str | None = None) -> dict: identity: str | None = None) -> dict:
"""Structured, LLM-safe explanation of a permission denial (#142). """Structured, LLM-safe explanation of a permission denial (#142, #897).
Built only after a gate has already refused; it adds guidance to the Built only after a gate has already refused; it adds guidance to the
refusal and never widens any permission, performs network I/O, or refusal and never widens any permission, performs network I/O, or
raises (fail-soft: degrades to a minimal fail-closed report). Names raises (fail-soft: degrades to a minimal fail-closed report). Names
configured profiles only never auth references, tokens, endpoint configured profiles only never auth references, tokens, endpoint
URLs, or keychain IDs. URLs, or keychain IDs.
#897: never fabricate a missing permission when the active profile
already allows the operation. That path is a diagnostic defect (the
refusal was not a permission denial), not a cue to switch profiles.
""" """
report = { report = {
"requested_operation": required_operation, "requested_operation": required_operation,
@@ -13841,6 +13843,7 @@ def _permission_block_report(required_operation: str,
"matching_configured_profiles": [], "matching_configured_profiles": [],
"runtime_switching_supported": False, "runtime_switching_supported": False,
"different_mcp_namespace_required": True, "different_mcp_namespace_required": True,
"diagnostic_defect": False,
"exact_safe_next_action": ( "exact_safe_next_action": (
"Ask the operator to fix GITEA_MCP_CONFIG/GITEA_MCP_PROFILE; " "Ask the operator to fix GITEA_MCP_CONFIG/GITEA_MCP_PROFILE; "
"the active profile could not be resolved (fail closed)."), "the active profile could not be resolved (fail closed)."),
@@ -13855,6 +13858,32 @@ def _permission_block_report(required_operation: str,
report["active_allowed_operations"] = ( report["active_allowed_operations"] = (
profile.get("allowed_operations") or []) profile.get("allowed_operations") or [])
# #897: fail closed as a diagnostic defect when the active profile
# already holds the operation — callers must not invent "missing".
try:
holds, _hold_reason = gitea_config.check_operation(
required_operation,
profile.get("allowed_operations") or [],
profile.get("forbidden_operations") or [],
)
except Exception:
holds = False
if holds:
report["missing_permission"] = None
report["required_permission"] = required_operation
report["diagnostic_defect"] = True
report["different_mcp_namespace_required"] = False
report["exact_safe_next_action"] = (
"Diagnostic defect: the active profile already allows "
f"{required_operation}. This is not a permission denial — "
"inspect blocker_kind / reasons (stale-runtime or runtime-mode). "
"Do not call gitea_activate_profile or switch MCP sessions."
)
report["matching_configured_profiles"] = [
p for p in [profile.get("profile_name")] if p
]
return report
matching = [] matching = []
try: try:
config = gitea_config.load_config() or {} config = gitea_config.load_config() or {}
@@ -13903,6 +13932,205 @@ def _permission_block_report(required_operation: str,
return report return report
def _reason_is_stale_runtime(reason: str) -> bool:
"""True when *reason* is a master-parity / stale-daemon refusal (#897)."""
r = (reason or "").lower()
if not r:
return False
if "stale relative to live master" in r:
return True
if "server code is stale" in r:
return True
if "daemon is stale" in r:
return True
if "started at commit" in r and "workspace master is now" in r:
return True
if "mcp server started at" in r and "stale" in r:
return True
if "restart the server to load the current capability gates" in r:
return True
if "restart/reconnect before mutating" in r:
return True
return False
def _reason_is_runtime_mode(reason: str) -> bool:
"""True when *reason* is a stable-control / runtime-mode refusal (#897)."""
r = (reason or "").lower()
if not r:
return False
if _reason_is_stale_runtime(reason):
return False
if "runtime mode could not be assessed" in r:
return True
if "runtime mode is" in r:
return True
if "stable control runtime" in r:
return True
if "dev-test" in r and ("runtime" in r or "production" in r):
return True
if "development worktree" in r or "dev worktree" in r:
return True
if "launched from a 'branches/" in r or "launched from a \"branches/" in r:
return True
if "process-root / active-workspace alignment" in r:
return True
if "namespace" in r and "reproof" in r:
return True
return False
def _reason_is_permission(reason: str) -> bool:
"""True when *reason* is a genuine profile-permission denial (#897)."""
r = (reason or "").lower()
if not r:
return False
if _reason_is_stale_runtime(reason) or _reason_is_runtime_mode(reason):
return False
if "profile could not be resolved" in r:
return True
if "profile has no configured allowed operations" in r:
return True
if "profile forbids" in r:
return True
if "profile is not allowed to" in r:
return True
if "unrecognized forbidden operation" in r:
return True
return False
def _classify_operation_gate_reasons(reasons: list[str]) -> dict:
"""Partition gate reasons into stale / runtime-mode / permission (#897)."""
stale: list[str] = []
runtime_mode: list[str] = []
permission: list[str] = []
other: list[str] = []
for reason in reasons or []:
if _reason_is_stale_runtime(reason):
stale.append(reason)
elif _reason_is_runtime_mode(reason):
runtime_mode.append(reason)
elif _reason_is_permission(reason):
permission.append(reason)
else:
other.append(reason)
return {
"stale_runtime": stale,
"runtime_mode": runtime_mode,
"permission": permission,
"other": other,
}
def _stale_runtime_reconnect_action() -> str:
"""Sanctioned recovery for a stale daemon — reconnect only (#685/#897)."""
return (
"Reconnect the IDE/client MCP session so the server reloads at the "
"current master head. Do not call gitea_activate_profile or switch "
"MCP role sessions — profile switching does not clear a stale daemon."
)
def _build_operation_gate_refusal(
required_operation: str,
reasons: list[str],
**extra_fields,
) -> dict:
"""Structured gate refusal with typed blockers (#897).
Stale-runtime and runtime-mode refusals never attach a
``permission_report`` and never recommend profile switching. True
permission denials still get ``permission_report``. When both apply,
causes are reported separately under distinct fields.
"""
classified = _classify_operation_gate_reasons(reasons)
stale = classified["stale_runtime"]
runtime_mode = classified["runtime_mode"]
permission = classified["permission"]
other = classified["other"]
blocked: dict = {
"success": False,
"performed": False,
"reasons": list(reasons),
"mutation_performed": False,
"session_context_audit": session_ctx.mutation_context_audit_fields(),
"gate_reason_classes": {
"stale_runtime": list(stale),
"runtime_mode": list(runtime_mode),
"permission": list(permission),
"other": list(other),
},
}
if stale:
parity = _current_master_parity()
blocked["blocker_kind"] = "runtime_reconnect_required"
blocked["restart_required"] = True
blocked["stop_required"] = True
blocked["startup_head"] = parity.get("startup_head")
blocked["current_head"] = parity.get("current_head")
blocked["daemon_start_head"] = (
parity.get("daemon_start_head") or parity.get("startup_head")
)
blocked["local_head"] = (
parity.get("local_head") or parity.get("current_head")
)
blocked["live_remote_head"] = parity.get("live_remote_head")
blocked["live_stale"] = bool(parity.get("live_stale"))
blocked["live_known"] = bool(parity.get("live_known"))
blocked["exact_safe_next_action"] = _stale_runtime_reconnect_action()
if permission or other:
blocked["permission_block_reasons"] = list(permission) + list(other)
blocked["stale_runtime_reasons"] = list(stale)
# Never attach permission_report for a staleness refusal.
blocked.update(extra_fields)
return blocked
if runtime_mode:
blocked["blocker_kind"] = "runtime_mode_blocked"
blocked["restart_required"] = False
blocked["stop_required"] = True
blocked["exact_safe_next_action"] = (
"Real workflow mutations run only on the promoted stable control "
"runtime. Promote/reload the stable runtime; do not call "
"gitea_activate_profile or switch MCP role sessions to clear a "
"runtime-mode block."
)
if permission or other:
blocked["permission_block_reasons"] = list(permission) + list(other)
blocked["runtime_mode_reasons"] = list(runtime_mode)
blocked.update(extra_fields)
return blocked
# Pure permission (or unclassified-as-permission) denial.
blocked["blocker_kind"] = "permission_denied"
blocked["permission_report"] = _permission_block_report(required_operation)
blocked.update(extra_fields)
return blocked
def _permission_report_for_gate_reasons(
required_operation: str,
reasons: list[str] | None,
) -> dict | None:
"""Attach ``permission_report`` only for true permission denials (#897).
Call sites that historically always attached a permission report after
``_profile_operation_gate`` should use this so stale/runtime refusals
do not emit a fabricated missing-permission payload.
"""
if not reasons:
return None
classified = _classify_operation_gate_reasons(reasons)
if classified["stale_runtime"] or classified["runtime_mode"]:
return None
if not (classified["permission"] or classified["other"]):
return None
return _permission_block_report(required_operation)
def _role_for_operation(op: str) -> str | None: def _role_for_operation(op: str) -> str | None:
# Normalize op first # Normalize op first
try: try:
@@ -14079,7 +14307,7 @@ def _master_parity_block(op: str) -> list[str]:
def _profile_operation_gate(op: str) -> list[str]: def _profile_operation_gate(op: str) -> list[str]:
"""Profile permission check for a single gated operation (#126, #216, #420). """Profile permission check for a single gated operation (#126, #216, #420, #897).
Issue discussion comments are gated separately from the gitea.pr.* Issue discussion comments are gated separately from the gitea.pr.*
review/merge family: listing requires ``gitea.read``, creating requires review/merge family: listing requires ``gitea.read``, creating requires
@@ -14092,21 +14320,26 @@ def _profile_operation_gate(op: str) -> list[str]:
capability gate that has since been merged, and when the runtime itself is capability gate that has since been merged, and when the runtime itself is
not the promoted stable control runtime (#615) -- a dev/test or unknown not the promoted stable control runtime (#615) -- a dev/test or unknown
runtime holds production credentials but has not been promoted. runtime holds production credentials but has not been promoted.
#897: collect *all* independent refusal classes (stale, runtime-mode,
permission) rather than short-circuiting after the first. Callers that
only need a boolean still treat any non-empty list as blocked; typed
consumers (``_build_operation_gate_refusal``) can separate causes.
""" """
stale_reasons = _master_parity_block(op) reasons: list[str] = []
if stale_reasons: reasons.extend(_master_parity_block(op))
return stale_reasons reasons.extend(_runtime_mode_block(op))
runtime_reasons = _runtime_mode_block(op)
if runtime_reasons:
return runtime_reasons
try: try:
profile = get_profile() profile = get_profile()
except Exception as exc: except Exception as exc:
return [f"profile could not be resolved (fail closed): {_redact(str(exc))}"] reasons.append(
f"profile could not be resolved (fail closed): {_redact(str(exc))}"
)
return reasons
op_ok, op_reason = gitea_config.check_operation( op_ok, op_reason = gitea_config.check_operation(
op, profile["allowed_operations"], profile["forbidden_operations"]) op, profile["allowed_operations"], profile["forbidden_operations"])
if op_ok: if op_ok:
return [] return reasons
if _try_auto_switch_for_operation(op): if _try_auto_switch_for_operation(op):
try: try:
@@ -14114,17 +14347,26 @@ def _profile_operation_gate(op: str) -> list[str]:
op_ok, op_reason = gitea_config.check_operation( op_ok, op_reason = gitea_config.check_operation(
op, profile["allowed_operations"], profile["forbidden_operations"]) op, profile["allowed_operations"], profile["forbidden_operations"])
if op_ok: if op_ok:
return [] return reasons
except Exception as exc: except Exception as exc:
return [f"profile could not be resolved (fail closed): {_redact(str(exc))}"] reasons.append(
f"profile could not be resolved (fail closed): {_redact(str(exc))}"
)
return reasons
if op_reason == "no-allowed-operations": if op_reason == "no-allowed-operations":
return ["profile has no configured allowed operations (fail closed)"] reasons.append(
if op_reason == "forbidden": "profile has no configured allowed operations (fail closed)"
return [f"profile forbids '{op}'"] )
if op_reason == "invalid-forbidden-entry": elif op_reason == "forbidden":
return ["profile has an unrecognized forbidden operation entry (fail closed)"] reasons.append(f"profile forbids '{op}'")
return [f"profile is not allowed to {op}"] elif op_reason == "invalid-forbidden-entry":
reasons.append(
"profile has an unrecognized forbidden operation entry (fail closed)"
)
else:
reasons.append(f"profile is not allowed to {op}")
return reasons
def _mutation_config_authority_block(required_operation: str) -> dict | None: def _mutation_config_authority_block(required_operation: str) -> dict | None:
@@ -14344,10 +14586,14 @@ def _session_context_mutation_block(
def _profile_permission_block(required_operation: str, **extra_fields) -> dict | None: def _profile_permission_block(required_operation: str, **extra_fields) -> dict | None:
"""Structured permission denial for gated tools (#69, #142). """Structured operation-gate denial for gated tools (#69, #142, #897).
Returns a block dict when the active profile forbids *required_operation*, Returns a block dict when the active profile forbids *required_operation*,
or ``None`` when the gate passes. Never performs network I/O. the daemon is stale, or the runtime mode is not mutation-safe or
``None`` when the gate passes. Never performs network I/O.
#897: stale-runtime and runtime-mode refusals are typed
(``blocker_kind``) and never carry a ``permission_report``.
""" """
req_role = "reviewer" if any(required_operation.startswith(p) for p in ( req_role = "reviewer" if any(required_operation.startswith(p) for p in (
"gitea.pr.approve", "gitea.pr.merge", "gitea.pr.request_changes", "gitea.pr.review" "gitea.pr.approve", "gitea.pr.merge", "gitea.pr.request_changes", "gitea.pr.review"
@@ -14357,15 +14603,9 @@ def _profile_permission_block(required_operation: str, **extra_fields) -> dict |
reasons = _profile_operation_gate(required_operation) reasons = _profile_operation_gate(required_operation)
if reasons: if reasons:
blocked = { return _build_operation_gate_refusal(
"success": False, required_operation, reasons, **extra_fields
"performed": False, )
"reasons": reasons,
"permission_report": _permission_block_report(required_operation),
"session_context_audit": session_ctx.mutation_context_audit_fields(),
}
blocked.update(extra_fields)
return blocked
auth_block = _mutation_config_authority_block(required_operation) auth_block = _mutation_config_authority_block(required_operation)
if auth_block is not None: if auth_block is not None:
@@ -14494,20 +14734,14 @@ def gitea_acquire_reviewer_pr_lease(
"""Acquire a per-PR reviewer lease before review/merge mutations (#407).""" """Acquire a per-PR reviewer lease before review/merge mutations (#407)."""
read_block = _profile_operation_gate("gitea.read") read_block = _profile_operation_gate("gitea.read")
if read_block: if read_block:
return { return _build_operation_gate_refusal(
"success": False, "gitea.read", read_block, acquired=False
"acquired": False, )
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
comment_block = _profile_operation_gate("gitea.pr.comment") comment_block = _profile_operation_gate("gitea.pr.comment")
if comment_block: if comment_block:
return { return _build_operation_gate_refusal(
"success": False, "gitea.pr.comment", comment_block, acquired=False
"acquired": False, )
"reasons": comment_block,
"permission_report": _permission_block_report("gitea.pr.comment"),
}
# task=acquire_reviewer_pr_lease so verify_preflight_purity runs shared #604 # task=acquire_reviewer_pr_lease so verify_preflight_purity runs shared #604
# anti-stomp for the declared lease-acquire mutation inventory entry. # anti-stomp for the declared lease-acquire mutation inventory entry.
@@ -14623,20 +14857,14 @@ def gitea_acquire_merger_pr_lease(
""" """
read_block = _profile_operation_gate("gitea.read") read_block = _profile_operation_gate("gitea.read")
if read_block: if read_block:
return { return _build_operation_gate_refusal(
"success": False, "gitea.read", read_block, acquired=False
"acquired": False, )
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
comment_block = _profile_operation_gate("gitea.pr.comment") comment_block = _profile_operation_gate("gitea.pr.comment")
if comment_block: if comment_block:
return { return _build_operation_gate_refusal(
"success": False, "gitea.pr.comment", comment_block, acquired=False
"acquired": False, )
"reasons": comment_block,
"permission_report": _permission_block_report("gitea.pr.comment"),
}
merge_block = _profile_operation_gate("gitea.pr.merge") merge_block = _profile_operation_gate("gitea.pr.merge")
if merge_block: if merge_block:
return { return {
@@ -19375,14 +19603,12 @@ def gitea_update_pr_branch_by_merge(
# Permission: author branch push / PR mutation surface. # Permission: author branch push / PR mutation surface.
push_block = _profile_operation_gate("gitea.branch.push") push_block = _profile_operation_gate("gitea.branch.push")
if push_block: if push_block:
return { return _build_operation_gate_refusal(
"success": False, "gitea.branch.push",
"performed": False, push_block,
"mutation_allowed": False, mutation_allowed=False,
"reasons": push_block, role_kind=role,
"permission_report": _permission_block_report("gitea.branch.push"), )
"role_kind": role,
}
if role != "author": if role != "author":
pre = pr_sync_status.assess_update_pr_branch_preflight( pre = pr_sync_status.assess_update_pr_branch_preflight(
+13
View File
@@ -142,6 +142,19 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
"permission": "gitea.read", "permission": "gitea.read",
"role": "author", "role": "author",
}, },
# #644: Phase 2 Web Console recovery tasks.
"clear_stale_binding": {
"permission": "gitea.read",
"role": "author",
},
"rebind_session_worktree": {
"permission": "gitea.read",
"role": "author",
},
"reconcile_cleanups": {
"permission": "gitea.pr.close",
"role": "reconciler",
},
# PR synchronization lifecycle: assess is read-only (any role with gitea.read); # PR synchronization lifecycle: assess is read-only (any role with gitea.read);
# update-by-merge is author-only and mutates the PR head via Gitea API. # update-by-merge is author-only and mutates the PR head via Gitea API.
"assess_pr_sync_status": { "assess_pr_sync_status": {
@@ -0,0 +1,453 @@
"""#897: stale-runtime / runtime-mode refusals must not look like permission denials.
Acceptance criteria (issue #897):
* Stale-runtime and runtime-mode refusals are typed distinctly from
profile-permission refusals (distinct ``blocker_kind``).
* A refusal caused by staleness or runtime mode never emits a
``permission_report`` and never names a permission the active profile holds.
* ``_permission_block_report`` verifies the active profile actually lacks the
operation before reporting it missing.
* A stale-runtime refusal reports reconnect-only recovery and never recommends
``gitea_activate_profile`` or an MCP session switch.
* The blocker payload states the observed heads (parity fields).
* Matrix across author / reviewer / merger / reconciler profiles.
* Regression: ``gitea_create_issue`` on a stale daemon under ``prgs-author``
never returns ``missing_permission: gitea.issue.create``.
"""
from __future__ import annotations
import os
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent))
import gitea_config # noqa: E402
import gitea_mcp_server as mcp_server # noqa: E402
SHA_START = "7af40fb5ff7debd5e9165fe97d9c7c279358e175"
SHA_LIVE = "2f4dec832327513118f2fe92b74da25d124a01cb"
ROLE_MATRIX = (
(
"prgs-author",
"author",
"gitea.issue.create",
[
"gitea.read",
"gitea.issue.create",
"gitea.issue.comment",
"gitea.issue.close",
"gitea.branch.create",
"gitea.branch.push",
"gitea.pr.create",
"gitea.pr.comment",
"gitea.repo.commit",
],
["gitea.pr.approve", "gitea.pr.merge", "gitea.pr.request_changes"],
"gitea.pr.merge", # forbidden op for pure-permission case
),
(
"prgs-reviewer",
"reviewer",
"gitea.pr.review",
[
"gitea.read",
"gitea.pr.review",
"gitea.pr.approve",
"gitea.pr.request_changes",
"gitea.pr.comment",
"gitea.issue.comment",
],
["gitea.branch.push", "gitea.pr.create"],
"gitea.branch.push",
),
(
"prgs-merger",
"merger",
"gitea.pr.merge",
[
"gitea.read",
"gitea.pr.merge",
"gitea.pr.comment",
"gitea.issue.comment",
],
["gitea.pr.approve", "gitea.branch.push", "gitea.pr.create"],
"gitea.branch.push",
),
(
"prgs-reconciler",
"reconciler",
"gitea.branch.delete",
[
"gitea.read",
"gitea.branch.delete",
"gitea.pr.comment",
"gitea.issue.comment",
"gitea.pr.close",
"gitea.issue.close",
],
["gitea.pr.approve", "gitea.pr.merge"],
"gitea.pr.merge",
),
)
def _profile(name: str, role: str, allowed: list[str], forbidden: list[str]) -> dict:
return {
"profile_name": name,
"role": role,
"role_kind": role,
"allowed_operations": list(allowed),
"forbidden_operations": list(forbidden),
"identity": "test-user",
}
def _config(profiles: dict) -> dict:
return {
"version": 2,
"profiles": {
name: {
"role": p["role"],
"allowed_operations": p["allowed_operations"],
"forbidden_operations": p["forbidden_operations"],
}
for name, p in profiles.items()
},
"rules": {"allow_runtime_switching": True},
}
class Issue897Helpers(unittest.TestCase):
def test_classify_stale_reason_strings(self):
stale = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
classified = mcp_server._classify_operation_gate_reasons([stale])
self.assertEqual(classified["stale_runtime"], [stale])
self.assertEqual(classified["permission"], [])
self.assertEqual(classified["runtime_mode"], [])
def test_classify_permission_reason(self):
reason = "profile is not allowed to gitea.pr.merge"
classified = mcp_server._classify_operation_gate_reasons([reason])
self.assertEqual(classified["permission"], [reason])
self.assertEqual(classified["stale_runtime"], [])
def test_classify_runtime_mode_reason(self):
reason = (
"runtime mode is 'dev-test' and the mutation targets the "
"production repository; dev/test runtimes must not mutate real "
"issues or PRs (ADR: stable control runtime vs dev runtime)"
)
classified = mcp_server._classify_operation_gate_reasons([reason])
self.assertEqual(classified["runtime_mode"], [reason])
self.assertEqual(classified["stale_runtime"], [])
class Issue897PermissionBlockReport(unittest.TestCase):
def test_holds_op_is_diagnostic_defect_not_missing_permission(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create", "gitea.issue.comment"],
[],
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server.gitea_config, "load_config", return_value=_config({"prgs-author": profile})
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=True
):
report = mcp_server._permission_block_report("gitea.issue.create")
self.assertTrue(report.get("diagnostic_defect"), report)
self.assertIsNone(report.get("missing_permission"), report)
action = (report.get("exact_safe_next_action") or "").lower()
# Must not *recommend* profile switching; mentioning the forbidden
# action in a "do not call" instruction is fine.
self.assertNotIn("call gitea_activate_profile with", action)
self.assertNotIn("switch to the author mcp session", action)
self.assertNotIn("switch to the reviewer mcp session", action)
self.assertIn("diagnostic defect", action)
def test_true_missing_permission_still_reports(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create"],
["gitea.pr.merge"],
)
reviewer = _profile(
"prgs-reviewer",
"reviewer",
["gitea.read", "gitea.pr.merge", "gitea.pr.approve"],
[],
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server.gitea_config,
"load_config",
return_value=_config({"prgs-author": profile, "prgs-reviewer": reviewer}),
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=True
):
report = mcp_server._permission_block_report("gitea.pr.merge")
self.assertFalse(report.get("diagnostic_defect"), report)
self.assertEqual(report.get("missing_permission"), "gitea.pr.merge")
self.assertIn("prgs-reviewer", report.get("matching_configured_profiles") or [])
class Issue897GateRefusalMatrix(unittest.TestCase):
def _stale_parity(self) -> dict:
return {
"in_parity": True,
"stale": False,
"restart_required": True,
"determinable": True,
"startup_head": SHA_START,
"current_head": SHA_START,
"daemon_start_head": SHA_START,
"local_head": SHA_START,
"live_remote_head": SHA_LIVE,
"live_known": True,
"live_stale": True,
"mutation_safe": False,
"reasons": [
f"live remote master is {SHA_LIVE[:12]} but the MCP server "
f"started at {SHA_START[:12]}; the daemon is stale relative "
"to live master -- restart/reconnect before mutating"
],
}
def test_stale_plus_permitted_op_all_roles(self):
for name, role, permitted_op, allowed, forbidden, _forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=permitted_op):
profile = _profile(name, role, allowed, forbidden)
parity = self._stale_parity()
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=list(parity["reasons"])
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
):
blocked = mcp_server._profile_permission_block(permitted_op)
self.assertIsNotNone(blocked, name)
assert blocked is not None
self.assertEqual(
blocked.get("blocker_kind"),
"runtime_reconnect_required",
blocked,
)
self.assertNotIn("permission_report", blocked, blocked)
self.assertTrue(blocked.get("restart_required"), blocked)
self.assertEqual(blocked.get("startup_head"), SHA_START, blocked)
self.assertEqual(blocked.get("live_remote_head"), SHA_LIVE, blocked)
action = (blocked.get("exact_safe_next_action") or "").lower()
self.assertIn("reconnect", action)
self.assertNotIn("call gitea_activate_profile with", action)
self.assertNotIn("switch to the author mcp session", action)
self.assertNotIn("switch to the reviewer mcp session", action)
def test_fresh_plus_forbidden_op_all_roles(self):
for name, role, _permitted, allowed, forbidden, forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=forbidden_op):
profile = _profile(name, role, allowed, forbidden)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_master_parity_block", return_value=[]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
), patch.object(
mcp_server.gitea_config,
"load_config",
return_value=_config({name: profile}),
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=False
):
blocked = mcp_server._profile_permission_block(forbidden_op)
self.assertIsNotNone(blocked, name)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "permission_denied", blocked)
self.assertIn("permission_report", blocked, blocked)
report = blocked["permission_report"]
self.assertEqual(report.get("missing_permission"), forbidden_op, report)
self.assertFalse(report.get("diagnostic_defect"), report)
# No runtime reconnect fields for pure permission denial
self.assertNotEqual(
blocked.get("blocker_kind"), "runtime_reconnect_required"
)
def test_stale_plus_forbidden_op_both_causes_separated(self):
for name, role, _permitted, allowed, forbidden, forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=forbidden_op):
profile = _profile(name, role, allowed, forbidden)
parity = self._stale_parity()
stale_reason = parity["reasons"][0]
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=[stale_reason]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
):
# Gate collects both classes; force permission reason too.
with patch.object(
mcp_server,
"_profile_operation_gate",
return_value=[
stale_reason,
f"profile is not allowed to {forbidden_op}",
],
):
blocked = mcp_server._profile_permission_block(forbidden_op)
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(
blocked.get("blocker_kind"), "runtime_reconnect_required", blocked
)
self.assertNotIn("permission_report", blocked, blocked)
self.assertIn("permission_block_reasons", blocked, blocked)
self.assertIn("stale_runtime_reasons", blocked, blocked)
classes = blocked.get("gate_reason_classes") or {}
self.assertTrue(classes.get("stale_runtime"), classes)
self.assertTrue(classes.get("permission"), classes)
def test_runtime_mode_block_no_permission_report(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create"],
[],
)
runtime_reason = (
"runtime mode is 'dev-test' and the mutation targets the "
"production repository; dev/test runtimes must not mutate real "
"issues or PRs (ADR: stable control runtime vs dev runtime)"
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_master_parity_block", return_value=[]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[runtime_reason]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": "prgs-author"},
):
blocked = mcp_server._profile_permission_block("gitea.issue.create")
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "runtime_mode_blocked", blocked)
self.assertNotIn("permission_report", blocked, blocked)
action = (blocked.get("exact_safe_next_action") or "").lower()
self.assertNotIn("call gitea_activate_profile with", action)
self.assertIn("stable control runtime", action)
class Issue897CreateIssueRegression(unittest.TestCase):
def test_create_issue_stale_daemon_never_missing_issue_create(self):
"""Regression AC: stale prgs-author create_issue must not claim missing create."""
profile = _profile(
"prgs-author",
"author",
[
"gitea.read",
"gitea.issue.create",
"gitea.issue.comment",
"gitea.branch.create",
"gitea.branch.push",
"gitea.pr.create",
"gitea.pr.comment",
"gitea.repo.commit",
],
[],
)
stale_reason = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
parity = {
"in_parity": True,
"stale": False,
"restart_required": True,
"determinable": True,
"startup_head": SHA_START,
"current_head": SHA_START,
"daemon_start_head": SHA_START,
"local_head": SHA_START,
"live_remote_head": SHA_LIVE,
"live_known": True,
"live_stale": True,
"mutation_safe": False,
"reasons": [stale_reason],
}
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=[stale_reason]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": "prgs-author"},
), patch.object(
mcp_server, "_mutation_config_authority_block", return_value=None
), patch.object(
mcp_server, "_session_context_mutation_block", return_value=None
):
blocked = mcp_server._profile_permission_block(
"gitea.issue.create", remote="prgs"
)
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "runtime_reconnect_required")
self.assertNotIn("permission_report", blocked)
# Even if a caller still built a raw report, holds-check must not claim missing.
with patch.object(mcp_server, "get_profile", return_value=profile):
raw = mcp_server._permission_block_report("gitea.issue.create")
self.assertIsNone(raw.get("missing_permission"), raw)
self.assertNotEqual(raw.get("missing_permission"), "gitea.issue.create")
def test_permission_report_for_gate_reasons_skips_stale(self):
stale = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
self.assertIsNone(
mcp_server._permission_report_for_gate_reasons(
"gitea.issue.comment", [stale]
)
)
if __name__ == "__main__":
unittest.main()
+182
View File
@@ -0,0 +1,182 @@
"""Unit and integration tests for Phase 2 Web Console recovery controls (#644)."""
from __future__ import annotations
import json
import os
import unittest
from unittest.mock import patch
from starlette.testclient import TestClient
from webui import console_audit, console_authz, console_recovery
from webui.app import create_app
class TestConsoleRecovery(unittest.TestCase):
def test_diagnose_recovery_healthy(self) -> None:
diag = console_recovery.diagnose_recovery()
self.assertIn(diag.status, {console_recovery.STATUS_HEALTHY, console_recovery.STATUS_ACTION_REQUIRED})
self.assertIsInstance(diag.playbooks, tuple)
self.assertGreaterEqual(len(diag.playbooks), 4)
playbook_ids = {pb.playbook_id for pb in diag.playbooks}
self.assertIn(console_recovery.PLAYBOOK_CLEAR_STALE_BINDING, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_REBIND_SESSION, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_RECONCILE_CLEANUPS, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_SANCTIONED_RESTART, playbook_ids)
def test_confirmation_phrase_generation_and_matching(self) -> None:
phrase = console_recovery.confirmation_phrase("clear_stale_binding")
self.assertEqual(phrase, "confirm clear_stale_binding")
self.assertTrue(console_recovery.confirmation_matches("clear_stale_binding", "confirm clear_stale_binding"))
self.assertFalse(console_recovery.confirmation_matches("clear_stale_binding", "wrong phrase"))
phrase_target = console_recovery.confirmation_phrase("sanctioned_restart", "gitea-author")
self.assertEqual(phrase_target, "confirm sanctioned_restart gitea-author")
self.assertTrue(console_recovery.confirmation_matches("sanctioned_restart", "confirm sanctioned_restart gitea-author", "gitea-author"))
def test_build_recovery_preview(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
preview = console_recovery.build_recovery_preview(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
target="test-worktree",
principal=principal,
)
self.assertEqual(preview["playbook_id"], console_recovery.PLAYBOOK_CLEAR_STALE_BINDING)
self.assertEqual(preview["action_id"], console_recovery.ACTION_CLEAR_STALE_BINDING)
self.assertEqual(preview["confirmation_phrase"], "confirm clear_stale_binding test-worktree")
self.assertTrue(len(preview["mutation_ledger"]) >= 3)
self.assertTrue(preview["authorization"]["allowed"])
def test_build_recovery_preview_unknown_playbook(self) -> None:
preview = console_recovery.build_recovery_preview("unknown_playbook")
self.assertFalse(preview.get("allowed"))
self.assertEqual(preview.get("error"), "unknown_playbook")
def test_execute_recovery_playbook_confirmation_mismatch(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation="invalid confirmation",
principal=principal,
)
self.assertFalse(result["success"])
self.assertFalse(result["allowed"])
self.assertEqual(result["error"], "confirmation_mismatch")
def test_execute_recovery_playbook_unauthorized(self) -> None:
# Anonymous principal has viewer role -> should be denied
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation="confirm clear_stale_binding",
principal=console_authz.ANONYMOUS,
)
self.assertFalse(result["success"])
self.assertFalse(result["allowed"])
self.assertEqual(result["error"], console_authz.DENY_UNAUTHENTICATED)
def test_execute_recovery_playbook_clear_stale_binding_success(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
phrase = console_recovery.confirmation_phrase(console_recovery.PLAYBOOK_CLEAR_STALE_BINDING)
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation=phrase,
principal=principal,
)
self.assertTrue(result["allowed"])
self.assertIn("applied_result", result)
self.assertIn("post_recovery_verification", result)
self.assertIn("audit", result)
self.assertEqual(result["audit"]["event"]["action"], console_recovery.ACTION_CLEAR_STALE_BINDING)
def test_execute_recovery_playbook_rebind_session_success(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
phrase = console_recovery.confirmation_phrase(console_recovery.PLAYBOOK_REBIND_SESSION, "branches/feat-issue-644")
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_REBIND_SESSION,
confirmation=phrase,
target="branches/feat-issue-644",
principal=principal,
)
self.assertTrue(result["allowed"])
self.assertTrue(result["success"])
self.assertEqual(result["applied_result"]["rebound_worktree"], "branches/feat-issue-644")
def test_verify_post_recovery(self) -> None:
verification = console_recovery.verify_post_recovery()
self.assertIn("clean", verification)
self.assertIn("status", verification)
self.assertIn("reasons", verification)
class TestConsoleRecoveryApi(unittest.TestCase):
def setUp(self) -> None:
self.app = create_app()
self.client = TestClient(self.app)
def test_api_recovery_diagnose(self) -> None:
res = self.client.get("/api/v1/system/recovery/diagnose")
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertIn("status", data)
self.assertIn("clean", data)
self.assertIn("playbooks", data)
self.assertTrue(len(data["playbooks"]) >= 4)
def test_api_recovery_preview(self) -> None:
res = self.client.post(
"/api/v1/system/recovery/preview",
json={"playbook_id": "clear_stale_binding", "target": "active"},
)
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertEqual(data["playbook_id"], "clear_stale_binding")
self.assertEqual(data["confirmation_phrase"], "confirm clear_stale_binding active")
self.assertIn("mutation_ledger", data)
def test_api_recovery_apply_denied_without_auth(self) -> None:
res = self.client.post(
"/api/v1/system/recovery/apply",
json={"playbook_id": "clear_stale_binding", "confirmation": "confirm clear_stale_binding"},
)
self.assertEqual(res.status_code, 400)
data = res.json()
self.assertFalse(data["success"])
self.assertFalse(data["allowed"])
def test_api_recovery_apply_with_dev_auth(self) -> None:
env = {
"WEBUI_AUTH_MODE": "local_dev",
"WEBUI_DEV_SUBJECT": "[email protected]",
"WEBUI_DEV_ROLE": "operator",
}
with patch.dict(os.environ, env):
res = self.client.post(
"/api/v1/system/recovery/apply",
json={
"playbook_id": "rebind_session_worktree",
"target": "branches/feat-issue-644",
"confirmation": "confirm rebind_session_worktree branches/feat-issue-644",
},
)
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertTrue(data["success"])
self.assertTrue(data["allowed"])
self.assertEqual(data["playbook_id"], "rebind_session_worktree")
def test_api_recovery_verify(self) -> None:
res = self.client.get("/api/v1/system/recovery/verify")
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertIn("clean", data)
self.assertIn("status", data)
if __name__ == "__main__":
unittest.main()
+83
View File
@@ -200,6 +200,84 @@ async def system_health(request: Request) -> HTMLResponse:
) )
async def api_recovery_diagnose(_request: Request) -> JSONResponse:
from webui.console_recovery import diagnose_recovery
diag = diagnose_recovery()
return JSONResponse({
"status": diag.status,
"clean": diag.clean,
"stale_runtime": diag.stale_runtime,
"master_parity": diag.master_parity,
"stale_binding": diag.stale_binding,
"contamination": diag.contamination,
"worktree_anomalies": list(diag.worktree_anomalies),
"reasons": list(diag.reasons),
"playbooks": [
{
"playbook_id": pb.playbook_id,
"label": pb.label,
"action_id": pb.action_id,
"description": pb.description,
"eligible": pb.eligible,
"requires_confirmation": pb.requires_confirmation,
"reason": pb.reason,
"params_schema": pb.params_schema,
}
for pb in diag.playbooks
],
})
async def api_recovery_preview(request: Request) -> JSONResponse:
from webui.console_recovery import build_recovery_preview
body = {}
try:
body = await request.json()
except Exception:
pass
playbook_id = body.get("playbook_id") or request.query_params.get("playbook_id") or ""
target = body.get("target") or request.query_params.get("target")
principal = resolve_principal(request.headers)
preview = build_recovery_preview(
playbook_id=playbook_id,
target=target,
params=body,
principal=principal,
)
status = 200 if preview.get("playbook_id") else 400
return JSONResponse(preview, status_code=status)
async def api_recovery_apply(request: Request) -> JSONResponse:
from webui.console_recovery import execute_recovery_playbook
body = {}
try:
body = await request.json()
except Exception:
pass
playbook_id = body.get("playbook_id", "")
confirmation = body.get("confirmation", "")
target = body.get("target")
principal = resolve_principal(request.headers)
request_id = getattr(request.state, "request_id", None)
result = execute_recovery_playbook(
playbook_id=playbook_id,
confirmation=confirmation,
target=target,
params=body,
principal=principal,
request_id=request_id,
)
status_code = 200 if result.get("success") else 400
return JSONResponse(result, status_code=status_code)
async def api_recovery_verify(_request: Request) -> JSONResponse:
from webui.console_recovery import verify_post_recovery
verification = verify_post_recovery()
return JSONResponse(verification, status_code=200)
async def queue(_request: Request) -> HTMLResponse: async def queue(_request: Request) -> HTMLResponse:
snapshot = load_queue_snapshot() snapshot = load_queue_snapshot()
return HTMLResponse(render_page(title="Queue", body_html=render_queue_page(snapshot))) return HTMLResponse(render_page(title="Queue", body_html=render_queue_page(snapshot)))
@@ -818,6 +896,11 @@ def create_app(*, bind_host: str | None = None) -> Starlette:
api_console_security_model, api_console_security_model,
methods=["GET"], methods=["GET"],
), ),
# #644 Phase 2 Recovery API routes
Route("/api/v1/system/recovery/diagnose", api_recovery_diagnose, methods=["GET"]),
Route("/api/v1/system/recovery/preview", api_recovery_preview, methods=["POST", "GET"]),
Route("/api/v1/system/recovery/apply", api_recovery_apply, methods=["POST"]),
Route("/api/v1/system/recovery/verify", api_recovery_verify, methods=["POST", "GET"]),
*[ *[
Route(path, phase_stub, methods=["GET"]) Route(path, phase_stub, methods=["GET"])
for path in STUB_PAGES for path in STUB_PAGES
+34
View File
@@ -277,6 +277,40 @@ _ACTION_SPECS: tuple[ConsoleAction, ...] = (
phase=2, phase=2,
summary="Restart one MCP namespace via the host supervisor.", summary="Restart one MCP namespace via the host supervisor.",
), ),
# #644: Phase 2 recovery controls & playbooks.
ConsoleAction(
action_id="system.clear_stale_binding",
task_key="clear_stale_binding",
action_class=CLASS_WRITE,
minimum_role=OPERATOR,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Clear provably stale or superseded GITEA_ACTIVE_WORKTREE binding.",
),
ConsoleAction(
action_id="system.rebind_session_worktree",
task_key="rebind_session_worktree",
action_class=CLASS_WRITE,
minimum_role=OPERATOR,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Rebind session worktree context to verified lease worktree.",
),
ConsoleAction(
action_id="system.reconcile_cleanups",
task_key="reconcile_cleanups",
action_class=CLASS_PRIVILEGED,
minimum_role=CONTROLLER,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Run reconciler cleanup for merged or superseded PR branches.",
),
) )
ACTIONS: dict[str, ConsoleAction] = {a.action_id: a for a in _ACTION_SPECS} ACTIONS: dict[str, ConsoleAction] = {a.action_id: a for a in _ACTION_SPECS}
+573
View File
@@ -0,0 +1,573 @@
"""Web Console Phase 2 Recovery Controls & Playbooks (#644).
Provides canonical recovery controls for the web console:
1. Diagnosis: Surfaces stale runtimes, worktree binding errors, contamination markers,
and un-reconciled cleanups.
2. Gated Actions & Playbooks: Guided recovery (rebind session worktree, clear stale
binding, trigger reconciler cleanups, sanctioned restart).
3. RBAC, Contamination (#630), and Master Parity (#610) integration.
4. Audit trail via ``console_audit`` and mandatory post-recovery revalidation.
"""
from __future__ import annotations
import os
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
import master_parity_gate
import merged_cleanup_reconcile
import runtime_recovery_guard
import stale_binding_recovery
from webui import console_audit, console_authz, sanctioned_restart, system_health, worktree_scanner
# --- Recovery Statuses ------------------------------------------------------
STATUS_HEALTHY = "healthy"
STATUS_ACTION_REQUIRED = "action_required"
STATUS_BLOCKED_CONTAMINATION = "blocked_contamination"
STATUS_RECONNECT_REQUIRED = "reconnect_required"
# --- Playbook Identifiers ---------------------------------------------------
PLAYBOOK_CLEAR_STALE_BINDING = "clear_stale_binding"
PLAYBOOK_REBIND_SESSION = "rebind_session_worktree"
PLAYBOOK_RECONCILE_CLEANUPS = "reconcile_cleanups"
PLAYBOOK_SANCTIONED_RESTART = "sanctioned_restart"
KNOWN_PLAYBOOKS: tuple[str, ...] = (
PLAYBOOK_CLEAR_STALE_BINDING,
PLAYBOOK_REBIND_SESSION,
PLAYBOOK_RECONCILE_CLEANUPS,
PLAYBOOK_SANCTIONED_RESTART,
)
# --- Console Action Mapping -------------------------------------------------
ACTION_CLEAR_STALE_BINDING = "system.clear_stale_binding"
ACTION_REBIND_SESSION = "system.rebind_session_worktree"
ACTION_RECONCILE_CLEANUPS = "system.reconcile_cleanups"
PLAYBOOK_ACTIONS: dict[str, str] = {
PLAYBOOK_CLEAR_STALE_BINDING: ACTION_CLEAR_STALE_BINDING,
PLAYBOOK_REBIND_SESSION: ACTION_REBIND_SESSION,
PLAYBOOK_RECONCILE_CLEANUPS: ACTION_RECONCILE_CLEANUPS,
PLAYBOOK_SANCTIONED_RESTART: sanctioned_restart.ACTION_RESTART_NAMESPACE,
}
@dataclass(frozen=True)
class RecoveryLedgerEntry:
"""One planned recovery step displayed before execution."""
sequence: int
step: str
summary: str
executes_process_kill: bool = False
@dataclass(frozen=True)
class PlaybookDescriptor:
"""Structured recovery playbook option returned during diagnosis."""
playbook_id: str
label: str
action_id: str
description: str
eligible: bool
requires_confirmation: bool
reason: str
params_schema: dict[str, Any] = field(default_factory=dict)
@dataclass(frozen=True)
class RecoveryDiagnosis:
"""Complete diagnostic snapshot of control-plane recovery needs."""
status: str
clean: bool
stale_runtime: dict[str, Any]
master_parity: dict[str, Any]
stale_binding: dict[str, Any]
contamination: dict[str, Any]
worktree_anomalies: tuple[str, ...]
playbooks: tuple[PlaybookDescriptor, ...]
reasons: tuple[str, ...]
def _repo_root(custom_path: Path | str | None = None) -> Path:
if custom_path:
return Path(custom_path).resolve()
override = (os.environ.get("WEBUI_REPO_ROOT") or "").strip()
if override:
return Path(override).resolve()
return Path(__file__).resolve().parent.parent
def confirmation_phrase(playbook_id: str, target: str | None = None) -> str:
"""Construct exact confirmation phrase required for a recovery playbook."""
clean_target = (target or "").strip()
if clean_target:
return f"confirm {playbook_id} {clean_target}"
return f"confirm {playbook_id}"
def confirmation_matches(
playbook_id: str, confirmation: str | None, target: str | None = None
) -> bool:
expected = confirmation_phrase(playbook_id, target)
return str(confirmation or "").strip() == expected
def _build_ledger(
playbook_id: str, target: str | None = None
) -> tuple[RecoveryLedgerEntry, ...]:
if playbook_id == PLAYBOOK_CLEAR_STALE_BINDING:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"clear_env",
f"Remove stale env binding GITEA_ACTIVE_WORKTREE ({target or 'active'}).",
),
RecoveryLedgerEntry(
3, "audit", "Record clear_stale_binding event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run diagnosis to verify clean binding state."
),
)
if playbook_id == PLAYBOOK_REBIND_SESSION:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"rebind_worktree",
f"Rebind session worktree context safely to {target or 'target worktree'}.",
),
RecoveryLedgerEntry(
3, "audit", "Record rebind_session_worktree event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run diagnosis to verify worktree binding state."
),
)
if playbook_id == PLAYBOOK_RECONCILE_CLEANUPS:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"reconcile_cleanups",
"Execute sanctioned reconciler cleanup for merged or superseded PRs.",
),
RecoveryLedgerEntry(
3, "audit", "Record reconcile_cleanups event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run worktree scanner to verify clean tree."
),
)
if playbook_id == PLAYBOOK_SANCTIONED_RESTART:
restart_ledger = sanctioned_restart._mutation_ledger(
target or "gitea-author", sanctioned_restart.MODE_RESTART
)
return tuple(
RecoveryLedgerEntry(
sequence=e.sequence,
step=e.step,
summary=e.summary,
executes_process_kill=e.executes_process_kill,
)
for e in restart_ledger
)
return (
RecoveryLedgerEntry(1, "unspecified", f"Execute recovery playbook {playbook_id}."),
)
def diagnose_recovery(
repo_path: Path | str | None = None,
env: dict[str, str] | None = None,
active_worktree_val: str | None = None,
session_lease_wt: str | None = None,
role_kind: str | None = None,
) -> RecoveryDiagnosis:
"""Run full control-plane diagnostics to determine recovery needs and options."""
root = _repo_root(repo_path)
source_env = dict(env) if env is not None else dict(os.environ)
reasons: list[str] = []
# 1. Stale runtime assessment
stale_runtime_obj = system_health.assess_stale_runtime(root)
stale_runtime_dict = {
"daemon_head": stale_runtime_obj.daemon_head,
"checkout_head": stale_runtime_obj.checkout_head,
"remote_head": stale_runtime_obj.remote_head,
"stale": stale_runtime_obj.stale,
"determinable": stale_runtime_obj.determinable,
"mutation_safe": stale_runtime_obj.mutation_safe,
"reasons": list(stale_runtime_obj.reasons),
}
if stale_runtime_obj.stale:
reasons.append("Runtime HEAD disagrees with checkout/remote HEAD.")
# 2. Master parity assessment
checkout_head = stale_runtime_obj.checkout_head
startup_dict = master_parity_gate.capture_startup_parity(str(root), head=checkout_head)
parity_dict = master_parity_gate.assess_master_parity(startup_dict, checkout_head)
if not parity_dict.get("in_parity", True):
reasons.append("Repository is not in master parity.")
# 3. Worktree binding classification
boot_bindings = stale_binding_recovery.snapshot_boot_bindings(source_env)
active_val = (
active_worktree_val
if active_worktree_val is not None
else source_env.get(stale_binding_recovery.ACTIVE_WORKTREE_ENV)
)
boot_inherited = bool(boot_bindings.get("active_worktree") and active_val == boot_bindings.get("active_worktree"))
path_exists = None
if active_val:
path_exists = os.path.exists(os.path.realpath(active_val))
binding_class = stale_binding_recovery.classify_active_worktree_binding(
active_value=active_val,
session_lease_worktree=session_lease_wt,
boot_inherited=boot_inherited,
path_exists=path_exists,
role_kind=role_kind,
)
if binding_class.get("clear_eligible"):
reasons.append(
f"Active worktree binding is stale ({binding_class.get('classification')})."
)
elif binding_class.get("classification") == stale_binding_recovery.CLASSIFICATION_UNVERIFIED_INHERITED:
reasons.append("Inherited worktree binding is unverified.")
# 4. Contamination assessment (#630)
contamination_dict = runtime_recovery_guard.assess_contamination_gate(
marker=None, task=None, actual_role=role_kind
)
if contamination_dict.get("block"):
reasons.append("Runtime is contaminated by manual process kill (#630).")
# 5. Worktree scanner hygiene & anomalies
hygiene = worktree_scanner.load_hygiene_snapshot(project_root=str(root))
worktree_anomalies = hygiene.anomalies
# Determine status & eligible playbooks
playbooks: list[PlaybookDescriptor] = []
# Playbook 1: Clear Stale Binding
clear_eligible = bool(binding_class.get("clear_eligible"))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_CLEAR_STALE_BINDING,
label="Clear Stale Worktree Binding",
action_id=ACTION_CLEAR_STALE_BINDING,
description="Clear provably stale or superseded GITEA_ACTIVE_WORKTREE environment binding.",
eligible=clear_eligible,
requires_confirmation=True,
reason=(
f"Binding classified as {binding_class.get('classification')}; clear is authorized."
if clear_eligible
else "Active worktree binding is clean, corroborated, or absent."
),
)
)
# Playbook 2: Rebind Session Worktree
rebind_eligible = bool(
active_val
or binding_class.get("classification") == stale_binding_recovery.CLASSIFICATION_UNVERIFIED_INHERITED
)
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_REBIND_SESSION,
label="Rebind Session Worktree",
action_id=ACTION_REBIND_SESSION,
description="Rebind or synchronize session worktree binding safely with active lease.",
eligible=rebind_eligible,
requires_confirmation=True,
reason=(
"Session worktree binding can be rebound to verified lease worktree."
if rebind_eligible
else "Session worktree is properly bound."
),
params_schema={"target_worktree": "string"},
)
)
# Playbook 3: Reconcile Cleanups
reconcile_eligible = bool(hygiene.anomalies or any(e.classification in {"stale-clean", "detached-review"} for e in hygiene.entries))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_RECONCILE_CLEANUPS,
label="Trigger Reconciler Cleanups",
action_id=ACTION_RECONCILE_CLEANUPS,
description="Run sanctioned reconciler cleanup preview and apply for merged/superseded PR branches.",
eligible=reconcile_eligible,
requires_confirmation=True,
reason=(
f"Worktree hygiene scanner detected {len(hygiene.anomalies)} anomalies and cleanups needed."
if reconcile_eligible
else "No reconciler cleanups pending."
),
)
)
# Playbook 4: Sanctioned Restart
restart_eligible = bool(stale_runtime_obj.stale or contamination_dict.get("contaminated"))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_SANCTIONED_RESTART,
label="Sanctioned MCP Restart",
action_id=sanctioned_restart.ACTION_RESTART_NAMESPACE,
description="Restart MCP daemon via configured host supervisor without manual process kill.",
eligible=restart_eligible,
requires_confirmation=True,
reason=(
"Stale runtime or contamination detected; host supervisor restart available."
if restart_eligible
else "Runtime is healthy and clean."
),
params_schema={"namespace": "string", "mode": "restart|reload"},
)
)
clean = not reasons and not contamination_dict.get("contaminated")
if contamination_dict.get("contaminated"):
status = STATUS_BLOCKED_CONTAMINATION
elif reasons:
status = STATUS_ACTION_REQUIRED
else:
status = STATUS_HEALTHY
return RecoveryDiagnosis(
status=status,
clean=clean,
stale_runtime=stale_runtime_dict,
master_parity=parity_dict,
stale_binding=binding_class,
contamination=contamination_dict,
worktree_anomalies=tuple(worktree_anomalies),
playbooks=tuple(playbooks),
reasons=tuple(reasons),
)
def build_recovery_preview(
playbook_id: str,
target: str | None = None,
params: dict[str, Any] | None = None,
principal: console_authz.Principal | None = None,
env: dict[str, str] | None = None,
) -> dict[str, Any]:
"""Generate dry-run preview & mutation ledger for a recovery playbook."""
if playbook_id not in KNOWN_PLAYBOOKS:
return {
"allowed": False,
"error": "unknown_playbook",
"detail": f"Playbook {playbook_id!r} is not a registered recovery playbook.",
}
action_id = PLAYBOOK_ACTIONS[playbook_id]
action = console_authz.get_action(action_id)
decision = console_authz.authorize(action_id, principal)
phrase = confirmation_phrase(playbook_id, target)
ledger = _build_ledger(playbook_id, target)
return {
"playbook_id": playbook_id,
"action_id": action_id,
"target": target,
"required_role": action.minimum_role if action else console_authz.OPERATOR,
"required_permission": action.mcp_permission if action else "gitea.read",
"requires_confirmation": True,
"confirmation_phrase": phrase,
"mutation_ledger": [asdict(entry) for entry in ledger],
"authorization": decision.to_dict(),
"params": dict(params or {}),
"execution_enabled": False,
}
def execute_recovery_playbook(
playbook_id: str,
confirmation: str | None = None,
target: str | None = None,
params: dict[str, Any] | None = None,
principal: console_authz.Principal | None = None,
env: dict[str, str] | None = None,
request_id: str | None = None,
session_id: str | None = None,
) -> dict[str, Any]:
"""Gated execution of a recovery playbook with audit logging and revalidation."""
if playbook_id not in KNOWN_PLAYBOOKS:
return {
"success": False,
"allowed": False,
"error": "unknown_playbook",
"detail": f"Playbook {playbook_id!r} is not known.",
}
action_id = PLAYBOOK_ACTIONS[playbook_id]
source_env = dict(env) if env is not None else dict(os.environ)
# 1. Authorization check
decision = console_authz.authorize(action_id, principal)
if not decision.allowed:
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code=decision.reason_code,
detail=decision.detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": decision.reason_code,
"detail": decision.detail,
}
# 2. Confirmation phrase check
if not confirmation_matches(playbook_id, confirmation, target):
expected = confirmation_phrase(playbook_id, target)
detail = f"Confirmation phrase mismatch. Expected: {expected!r}"
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="confirmation_mismatch",
detail=detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": "confirmation_mismatch",
"detail": detail,
"expected_confirmation_phrase": expected,
}
# 3. Contamination rule (#630) check
role_str = principal.role if principal else None
contam = runtime_recovery_guard.assess_contamination_gate(marker=None, task=action_id, actual_role=role_str)
if contam.get("block"):
if playbook_id != PLAYBOOK_RECONCILE_CLEANUPS:
detail = "Runtime is contaminated by a manual process kill (#630). Run reconciler cleanup playbook first."
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="contaminated_runtime",
detail=detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": "contaminated_runtime",
"detail": detail,
}
# 4. Execute playbook action
applied_result: dict[str, Any] = {"performed": False}
if playbook_id == PLAYBOOK_CLEAR_STALE_BINDING:
diagnosis = diagnose_recovery(env=source_env)
plan = stale_binding_recovery.plan_recovery(diagnosis.stale_binding)
applied_result = stale_binding_recovery.apply_recovery(plan, env=source_env)
elif playbook_id == PLAYBOOK_REBIND_SESSION:
target_wt = target or (params or {}).get("target_worktree")
if target_wt:
source_env[stale_binding_recovery.ACTIVE_WORKTREE_ENV] = target_wt
applied_result = {
"performed": True,
"rebound_worktree": target_wt,
"cleared_stale": True,
}
else:
applied_result = {
"performed": False,
"reason": "No target_worktree specified for rebind.",
}
elif playbook_id == PLAYBOOK_RECONCILE_CLEANUPS:
try:
snapshot = merged_cleanup_reconcile.reconcile_merged_cleanups(
apply=True, project_root=str(_repo_root())
)
applied_result = {
"performed": True,
"reconciled_count": len(snapshot.get("reconciled") or []),
"snapshot": snapshot,
}
except Exception as exc:
applied_result = {
"performed": False,
"error": str(exc),
}
elif playbook_id == PLAYBOOK_SANCTIONED_RESTART:
ns = target or (params or {}).get("namespace", "gitea-author")
md = (params or {}).get("mode", sanctioned_restart.MODE_RESTART)
restart_res = sanctioned_restart.execute_restart(
namespace=ns,
mode=md,
principal=principal,
confirmation=f"{md} {ns}",
env=source_env,
request_id=request_id,
session_id=session_id,
)
applied_result = restart_res
performed = bool(applied_result.get("performed") or applied_result.get("allowed"))
# 5. Record Audit Log
audit_record = console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_ALLOWED if performed else console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="recovery_executed" if performed else "recovery_failed",
detail=f"Executed recovery playbook {playbook_id}",
request_id=request_id,
session_id=session_id,
metadata={"applied_result": applied_result},
)
# 6. Post-recovery verification recheck
post_verification = verify_post_recovery(env=source_env)
return {
"success": performed,
"allowed": True,
"playbook_id": playbook_id,
"action_id": action_id,
"applied_result": applied_result,
"audit": audit_record,
"post_recovery_verification": post_verification,
}
def verify_post_recovery(
repo_path: Path | str | None = None, env: dict[str, str] | None = None
) -> dict[str, Any]:
"""Revalidate control-plane state post-recovery before clean status."""
diag = diagnose_recovery(repo_path, env)
return {
"clean": diag.clean,
"status": diag.status,
"stale_runtime_clean": not diag.stale_runtime.get("stale"),
"binding_clean": not diag.stale_binding.get("clear_eligible"),
"contamination_clean": not diag.contamination.get("contaminated"),
"anomalies_count": len(diag.worktree_anomalies),
"reasons": list(diag.reasons),
}
+7
View File
@@ -178,6 +178,13 @@ def build_action_registry() -> ActionRegistry:
("system.restart_namespace", "Restart MCP namespace", ("system.restart_namespace", "Restart MCP namespace",
"restart_namespace", "host.supervisor_restart", "restart_namespace", "host.supervisor_restart",
"Restart one MCP namespace via the host supervisor."), "Restart one MCP namespace via the host supervisor."),
# #644: Phase 2 recovery playbooks & controls.
("system.clear_stale_binding", "Clear stale binding", "clear_stale_binding",
"console.clear_stale_binding", "Clear provably stale or superseded env binding."),
("system.rebind_session_worktree", "Rebind session worktree", "rebind_session_worktree",
"console.rebind_session_worktree", "Rebind session worktree to verified lease."),
("system.reconcile_cleanups", "Reconcile cleanups", "reconcile_cleanups",
"console.reconcile_cleanups", "Run reconciler cleanup for merged or superseded PRs."),
) )
actions = tuple( actions = tuple(
GatedAction( GatedAction(
+37 -16
View File
@@ -270,24 +270,45 @@ def _probe_error_card(snapshot: SystemHealthSnapshot) -> str:
def _recovery_card() -> str: def _recovery_card() -> str:
"""Sanctioned recovery pointers only — never a manual process kill (#630).""" """Sanctioned recovery controls & playbooks (#644, Phase 2)."""
try:
from webui import console_recovery
diag = console_recovery.diagnose_recovery()
status_badge = f"<span class='status-pill {diag.status}'>{diag.status}</span>"
playbook_lis = ""
for pb in diag.playbooks:
elig = "eligible" if pb.eligible else "disabled"
playbook_lis += (
f"<li><strong>{pb.label}</strong> (<code>{pb.playbook_id}</code>) — "
f"<span class='badge {elig}'>{elig}</span>: {pb.description} "
f"<em class='muted'>({pb.reason})</em></li>"
)
reasons_html = ""
if diag.reasons:
items = "".join(f"<li>{r}</li>" for r in diag.reasons)
reasons_html = f"<ul class='reasons'>{items}</ul>"
else:
reasons_html = "<p class='clean-note'>No recovery actions currently required. Control plane is healthy.</p>"
return (
"<section class='health-card recovery-card'>"
f"<h3>Sanctioned Recovery Controls (Phase 2 #644) {status_badge}</h3>"
"<p class='muted'>Guided recovery wizard: Diagnose &rarr; Preview &rarr; Confirm &rarr; Verify. "
"Reconnect the MCP client from the IDE, then re-run the blocked cycle. "
"Never kill the daemon process manually: unmanaged kills are recorded as runtime contamination (#630).</p>"
f"{reasons_html}"
"<h4>Available Recovery Playbooks</h4>"
f"<ul class='playbooks-list'>{playbook_lis}</ul>"
"<p class='meta'>APIs: <code>/api/v1/system/recovery/diagnose</code>, "
"<code>/api/v1/system/recovery/preview</code>, <code>/api/v1/system/recovery/apply</code>, "
"<code>/api/v1/system/recovery/verify</code>.</p>"
"</section>"
)
except Exception as exc:
return ( return (
"<section class='health-card'>" "<section class='health-card'>"
"<h3>Recovery</h3>" "<h3>Sanctioned Recovery Controls (Phase 2 #644)</h3>"
"<p class='muted'>This dashboard is read-only. Restart and reload " f"<p class='error'>Recovery diagnostics unavailable: {exc}</p>"
"controls arrive in Phase 2 (#642); until then recovery runs through "
"the sanctioned client reconnect / operator restart path.</p>"
"<ul class='reasons'>"
"<li><a href='/runtime'>Runtime health</a> — active profile, workflow "
"hashes, and shell health.</li>"
"<li><a href='/sessions'>Runtime and sessions</a> — namespaces, session "
"rows, worktree bindings, and contamination markers (#641).</li>"
"<li>Reconnect the MCP client from the IDE, then re-run the blocked "
"cycle. Never kill the daemon process manually: unmanaged kills are "
"recorded as runtime contamination (#630).</li>"
"<li>See <code>docs/webui-local-dev.md</code> for the documented "
"recovery sequence.</li>"
"</ul>"
"</section>" "</section>"
) )