Compare commits

...
Author SHA1 Message Date
sysadmin 1c88b87ec5 feat(webui): implement Phase 2 recovery controls & playbooks (#644) 2026-07-25 08:19:06 -04:00
sysadmin 76f293eb28 Merge pull request 'fix(gate): stop classifying stale-runtime blocks as permission denials (Closes #897)' (#901) from fix/issue-897-permission-stale-runtime-classification into master 2026-07-25 06:56:16 -05:00
jcwalker3 54559aebc3 Merge branch 'master' into fix/issue-897-permission-stale-runtime-classification 2026-07-25 06:42:27 -05:00
sysadmin 715863799f Merge pull request 'feat(webui): Runtime and session view (Phase 1) (Closes #641)' (#898) from feat/issue-641-runtime-session-view into master 2026-07-25 05:50:48 -05:00
sysadmin 6e6ca94338 fix(gate): stop classifying stale-runtime blocks as permission denials (Closes #897)
Stale-runtime and runtime-mode mutation refusals previously shared the
permission-denial channel, so permission_report claimed a missing op the
active profile already held and recommended gitea_activate_profile.
Typed blocker_kind payloads report reconnect-only recovery for staleness,
omit permission_report for non-permission gates, and fail closed when a
permission_report would invent a missing permission the profile holds.
2026-07-25 01:47:35 -04:00
11 changed files with 1747 additions and 90 deletions
+62
View File
@@ -0,0 +1,62 @@
# Sanctioned Recovery Playbooks & Controls (Phase 2 #644)
## Overview
Stale runtimes, worktree binding mismatches, and un-reconciled merged branches previously required expert manual shell recovery. Manual process kills (`pkill -f mcp_server.py`) are strictly forbidden and classified as runtime contamination ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)).
Phase 2 introduces **sanctioned recovery playbooks and controls** into the Web Console:
- **Diagnose**: Surface stale runtimes, worktree binding errors, contamination markers, and worktree anomalies via health & inventory APIs.
- **Preview**: Render mutation ledgers and exact confirmation phrases for recovery playbooks.
- **Confirm & Apply**: Execute sanctioned recovery actions through gated, audited paths.
- **Verify**: Revalidate control-plane state post-recovery before claiming clean status.
---
## Recovery Playbook Taxonomy
| Playbook ID | Action ID | Minimum Role | Target / Scope | Description |
|---|---|---|---|---|
| `clear_stale_binding` | `system.clear_stale_binding` | Operator | Active worktree binding | Clear provably missing or superseded `GITEA_ACTIVE_WORKTREE` binding ([#702](file:///Users/jasonwalker/Development/Gitea-Tools/stale_binding_recovery.py)). |
| `rebind_session_worktree` | `system.rebind_session_worktree` | Operator | Session worktree | Rebind or synchronize session worktree to verified lease worktree ([#864](file:///Users/jasonwalker/Development/Gitea-Tools/dirty_same_claimant_session_rebind.py)). |
| `reconcile_cleanups` | `system.reconcile_cleanups` | Controller | Worktree hygiene | Execute reconciler cleanup preview and apply for merged/superseded PR branches. |
| `sanctioned_restart` | `system.restart_namespace` | Admin | MCP Namespace | Restart MCP daemon gracefully via host supervisor ([#642](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)). |
---
## Wizard Workflow (Diagnose → Preview → Confirm → Verify)
### 1. Diagnose (`GET /api/v1/system/recovery/diagnose`)
Runs control-plane diagnostics:
- **Stale Runtime**: Mismatch between running daemon HEAD, local checkout HEAD, and remote-tracking HEAD.
- **Worktree Binding**: Missing path (`provably_stale_missing_path`), unverified inherited binding (`unverified_inherited`), or superseded binding (`superseded_by_session_lease`).
- **Contamination**: Checks for live contamination markers from unmanaged process kills.
- **Worktree Anomalies**: Scans `branches/` directory for un-reconciled cleanups or missing preserved worktrees.
Returns `RecoveryDiagnosis` with eligible playbooks.
### 2. Preview (`POST /api/v1/system/recovery/preview`)
Takes `playbook_id` and optional `target`/`params`.
Returns:
- **Mutation Ledger**: Step-by-step sequence of actions.
- **Confirmation Phrase**: Exact phrase required to authorize execution (e.g., `confirm clear_stale_binding`).
- **Authorization Decision**: RBAC check against the operator's principal.
### 3. Apply (`POST /api/v1/system/recovery/apply`)
Requires `playbook_id` and matching `confirmation` phrase.
- Validates RBAC permissions (`console_authz`).
- Verifies confirmation phrase (`confirmation_matches`).
- Enforces contamination rules ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)): A contaminated runtime must be cleared through reconciler cleanup before other playbooks run.
- Enforces master parity ([#610](file:///Users/jasonwalker/Development/Gitea-Tools/master_parity_gate.py)).
- Applies sanctioned recovery logic.
- Logs audit record in `console_audit`.
### 4. Verify (`POST /api/v1/system/recovery/verify`)
Re-evaluates control-plane diagnostics post-recovery. Asserts `clean: true` before transitioning out of recovery mode.
---
## Safety & Governance Principles
1. **No Manual `pkill`**: Direct process killing remains forbidden and is recorded as contamination.
2. **Auditability**: Every recovery preview and execution is logged in the console audit trail.
3. **Master Parity & Dual Control**: High-privilege recovery actions require controller/admin roles and explicit confirmation phrases.
+3
View File
@@ -94,6 +94,9 @@ already define, and a regression test asserts each mapping matches.
| `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 |
| `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 |
| `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 |
| `system.clear_stale_binding` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
**Dual control** means the acting principal may not be the sole authority: a
second distinct principal must confirm. **Break-glass** means the action is
+296 -70
View File
@@ -9552,15 +9552,13 @@ def gitea_edit_pr(
if closing:
gate_reasons = _profile_operation_gate("gitea.pr.close")
if gate_reasons:
return {
"success": False,
"performed": False,
"pr_number": pr_number,
"requested_state": "closed",
"required_permission": "gitea.pr.close",
"reasons": gate_reasons,
"permission_report": _permission_block_report("gitea.pr.close"),
}
return _build_operation_gate_refusal(
"gitea.pr.close",
gate_reasons,
pr_number=pr_number,
requested_state="closed",
required_permission="gitea.pr.close",
)
h, o, r = _resolve(remote, host, org, repo)
auth = _auth(h)
@@ -13823,13 +13821,17 @@ def gitea_view_issue(
def _permission_block_report(required_operation: str,
identity: str | None = None) -> dict:
"""Structured, LLM-safe explanation of a permission denial (#142).
"""Structured, LLM-safe explanation of a permission denial (#142, #897).
Built only after a gate has already refused; it adds guidance to the
refusal and never widens any permission, performs network I/O, or
raises (fail-soft: degrades to a minimal fail-closed report). Names
configured profiles only never auth references, tokens, endpoint
URLs, or keychain IDs.
#897: never fabricate a missing permission when the active profile
already allows the operation. That path is a diagnostic defect (the
refusal was not a permission denial), not a cue to switch profiles.
"""
report = {
"requested_operation": required_operation,
@@ -13841,6 +13843,7 @@ def _permission_block_report(required_operation: str,
"matching_configured_profiles": [],
"runtime_switching_supported": False,
"different_mcp_namespace_required": True,
"diagnostic_defect": False,
"exact_safe_next_action": (
"Ask the operator to fix GITEA_MCP_CONFIG/GITEA_MCP_PROFILE; "
"the active profile could not be resolved (fail closed)."),
@@ -13855,6 +13858,32 @@ def _permission_block_report(required_operation: str,
report["active_allowed_operations"] = (
profile.get("allowed_operations") or [])
# #897: fail closed as a diagnostic defect when the active profile
# already holds the operation — callers must not invent "missing".
try:
holds, _hold_reason = gitea_config.check_operation(
required_operation,
profile.get("allowed_operations") or [],
profile.get("forbidden_operations") or [],
)
except Exception:
holds = False
if holds:
report["missing_permission"] = None
report["required_permission"] = required_operation
report["diagnostic_defect"] = True
report["different_mcp_namespace_required"] = False
report["exact_safe_next_action"] = (
"Diagnostic defect: the active profile already allows "
f"{required_operation}. This is not a permission denial — "
"inspect blocker_kind / reasons (stale-runtime or runtime-mode). "
"Do not call gitea_activate_profile or switch MCP sessions."
)
report["matching_configured_profiles"] = [
p for p in [profile.get("profile_name")] if p
]
return report
matching = []
try:
config = gitea_config.load_config() or {}
@@ -13903,6 +13932,205 @@ def _permission_block_report(required_operation: str,
return report
def _reason_is_stale_runtime(reason: str) -> bool:
"""True when *reason* is a master-parity / stale-daemon refusal (#897)."""
r = (reason or "").lower()
if not r:
return False
if "stale relative to live master" in r:
return True
if "server code is stale" in r:
return True
if "daemon is stale" in r:
return True
if "started at commit" in r and "workspace master is now" in r:
return True
if "mcp server started at" in r and "stale" in r:
return True
if "restart the server to load the current capability gates" in r:
return True
if "restart/reconnect before mutating" in r:
return True
return False
def _reason_is_runtime_mode(reason: str) -> bool:
"""True when *reason* is a stable-control / runtime-mode refusal (#897)."""
r = (reason or "").lower()
if not r:
return False
if _reason_is_stale_runtime(reason):
return False
if "runtime mode could not be assessed" in r:
return True
if "runtime mode is" in r:
return True
if "stable control runtime" in r:
return True
if "dev-test" in r and ("runtime" in r or "production" in r):
return True
if "development worktree" in r or "dev worktree" in r:
return True
if "launched from a 'branches/" in r or "launched from a \"branches/" in r:
return True
if "process-root / active-workspace alignment" in r:
return True
if "namespace" in r and "reproof" in r:
return True
return False
def _reason_is_permission(reason: str) -> bool:
"""True when *reason* is a genuine profile-permission denial (#897)."""
r = (reason or "").lower()
if not r:
return False
if _reason_is_stale_runtime(reason) or _reason_is_runtime_mode(reason):
return False
if "profile could not be resolved" in r:
return True
if "profile has no configured allowed operations" in r:
return True
if "profile forbids" in r:
return True
if "profile is not allowed to" in r:
return True
if "unrecognized forbidden operation" in r:
return True
return False
def _classify_operation_gate_reasons(reasons: list[str]) -> dict:
"""Partition gate reasons into stale / runtime-mode / permission (#897)."""
stale: list[str] = []
runtime_mode: list[str] = []
permission: list[str] = []
other: list[str] = []
for reason in reasons or []:
if _reason_is_stale_runtime(reason):
stale.append(reason)
elif _reason_is_runtime_mode(reason):
runtime_mode.append(reason)
elif _reason_is_permission(reason):
permission.append(reason)
else:
other.append(reason)
return {
"stale_runtime": stale,
"runtime_mode": runtime_mode,
"permission": permission,
"other": other,
}
def _stale_runtime_reconnect_action() -> str:
"""Sanctioned recovery for a stale daemon — reconnect only (#685/#897)."""
return (
"Reconnect the IDE/client MCP session so the server reloads at the "
"current master head. Do not call gitea_activate_profile or switch "
"MCP role sessions — profile switching does not clear a stale daemon."
)
def _build_operation_gate_refusal(
required_operation: str,
reasons: list[str],
**extra_fields,
) -> dict:
"""Structured gate refusal with typed blockers (#897).
Stale-runtime and runtime-mode refusals never attach a
``permission_report`` and never recommend profile switching. True
permission denials still get ``permission_report``. When both apply,
causes are reported separately under distinct fields.
"""
classified = _classify_operation_gate_reasons(reasons)
stale = classified["stale_runtime"]
runtime_mode = classified["runtime_mode"]
permission = classified["permission"]
other = classified["other"]
blocked: dict = {
"success": False,
"performed": False,
"reasons": list(reasons),
"mutation_performed": False,
"session_context_audit": session_ctx.mutation_context_audit_fields(),
"gate_reason_classes": {
"stale_runtime": list(stale),
"runtime_mode": list(runtime_mode),
"permission": list(permission),
"other": list(other),
},
}
if stale:
parity = _current_master_parity()
blocked["blocker_kind"] = "runtime_reconnect_required"
blocked["restart_required"] = True
blocked["stop_required"] = True
blocked["startup_head"] = parity.get("startup_head")
blocked["current_head"] = parity.get("current_head")
blocked["daemon_start_head"] = (
parity.get("daemon_start_head") or parity.get("startup_head")
)
blocked["local_head"] = (
parity.get("local_head") or parity.get("current_head")
)
blocked["live_remote_head"] = parity.get("live_remote_head")
blocked["live_stale"] = bool(parity.get("live_stale"))
blocked["live_known"] = bool(parity.get("live_known"))
blocked["exact_safe_next_action"] = _stale_runtime_reconnect_action()
if permission or other:
blocked["permission_block_reasons"] = list(permission) + list(other)
blocked["stale_runtime_reasons"] = list(stale)
# Never attach permission_report for a staleness refusal.
blocked.update(extra_fields)
return blocked
if runtime_mode:
blocked["blocker_kind"] = "runtime_mode_blocked"
blocked["restart_required"] = False
blocked["stop_required"] = True
blocked["exact_safe_next_action"] = (
"Real workflow mutations run only on the promoted stable control "
"runtime. Promote/reload the stable runtime; do not call "
"gitea_activate_profile or switch MCP role sessions to clear a "
"runtime-mode block."
)
if permission or other:
blocked["permission_block_reasons"] = list(permission) + list(other)
blocked["runtime_mode_reasons"] = list(runtime_mode)
blocked.update(extra_fields)
return blocked
# Pure permission (or unclassified-as-permission) denial.
blocked["blocker_kind"] = "permission_denied"
blocked["permission_report"] = _permission_block_report(required_operation)
blocked.update(extra_fields)
return blocked
def _permission_report_for_gate_reasons(
required_operation: str,
reasons: list[str] | None,
) -> dict | None:
"""Attach ``permission_report`` only for true permission denials (#897).
Call sites that historically always attached a permission report after
``_profile_operation_gate`` should use this so stale/runtime refusals
do not emit a fabricated missing-permission payload.
"""
if not reasons:
return None
classified = _classify_operation_gate_reasons(reasons)
if classified["stale_runtime"] or classified["runtime_mode"]:
return None
if not (classified["permission"] or classified["other"]):
return None
return _permission_block_report(required_operation)
def _role_for_operation(op: str) -> str | None:
# Normalize op first
try:
@@ -14079,7 +14307,7 @@ def _master_parity_block(op: str) -> list[str]:
def _profile_operation_gate(op: str) -> list[str]:
"""Profile permission check for a single gated operation (#126, #216, #420).
"""Profile permission check for a single gated operation (#126, #216, #420, #897).
Issue discussion comments are gated separately from the gitea.pr.*
review/merge family: listing requires ``gitea.read``, creating requires
@@ -14092,21 +14320,26 @@ def _profile_operation_gate(op: str) -> list[str]:
capability gate that has since been merged, and when the runtime itself is
not the promoted stable control runtime (#615) -- a dev/test or unknown
runtime holds production credentials but has not been promoted.
#897: collect *all* independent refusal classes (stale, runtime-mode,
permission) rather than short-circuiting after the first. Callers that
only need a boolean still treat any non-empty list as blocked; typed
consumers (``_build_operation_gate_refusal``) can separate causes.
"""
stale_reasons = _master_parity_block(op)
if stale_reasons:
return stale_reasons
runtime_reasons = _runtime_mode_block(op)
if runtime_reasons:
return runtime_reasons
reasons: list[str] = []
reasons.extend(_master_parity_block(op))
reasons.extend(_runtime_mode_block(op))
try:
profile = get_profile()
except Exception as exc:
return [f"profile could not be resolved (fail closed): {_redact(str(exc))}"]
reasons.append(
f"profile could not be resolved (fail closed): {_redact(str(exc))}"
)
return reasons
op_ok, op_reason = gitea_config.check_operation(
op, profile["allowed_operations"], profile["forbidden_operations"])
if op_ok:
return []
return reasons
if _try_auto_switch_for_operation(op):
try:
@@ -14114,17 +14347,26 @@ def _profile_operation_gate(op: str) -> list[str]:
op_ok, op_reason = gitea_config.check_operation(
op, profile["allowed_operations"], profile["forbidden_operations"])
if op_ok:
return []
return reasons
except Exception as exc:
return [f"profile could not be resolved (fail closed): {_redact(str(exc))}"]
reasons.append(
f"profile could not be resolved (fail closed): {_redact(str(exc))}"
)
return reasons
if op_reason == "no-allowed-operations":
return ["profile has no configured allowed operations (fail closed)"]
if op_reason == "forbidden":
return [f"profile forbids '{op}'"]
if op_reason == "invalid-forbidden-entry":
return ["profile has an unrecognized forbidden operation entry (fail closed)"]
return [f"profile is not allowed to {op}"]
reasons.append(
"profile has no configured allowed operations (fail closed)"
)
elif op_reason == "forbidden":
reasons.append(f"profile forbids '{op}'")
elif op_reason == "invalid-forbidden-entry":
reasons.append(
"profile has an unrecognized forbidden operation entry (fail closed)"
)
else:
reasons.append(f"profile is not allowed to {op}")
return reasons
def _mutation_config_authority_block(required_operation: str) -> dict | None:
@@ -14344,10 +14586,14 @@ def _session_context_mutation_block(
def _profile_permission_block(required_operation: str, **extra_fields) -> dict | None:
"""Structured permission denial for gated tools (#69, #142).
"""Structured operation-gate denial for gated tools (#69, #142, #897).
Returns a block dict when the active profile forbids *required_operation*,
or ``None`` when the gate passes. Never performs network I/O.
the daemon is stale, or the runtime mode is not mutation-safe or
``None`` when the gate passes. Never performs network I/O.
#897: stale-runtime and runtime-mode refusals are typed
(``blocker_kind``) and never carry a ``permission_report``.
"""
req_role = "reviewer" if any(required_operation.startswith(p) for p in (
"gitea.pr.approve", "gitea.pr.merge", "gitea.pr.request_changes", "gitea.pr.review"
@@ -14357,15 +14603,9 @@ def _profile_permission_block(required_operation: str, **extra_fields) -> dict |
reasons = _profile_operation_gate(required_operation)
if reasons:
blocked = {
"success": False,
"performed": False,
"reasons": reasons,
"permission_report": _permission_block_report(required_operation),
"session_context_audit": session_ctx.mutation_context_audit_fields(),
}
blocked.update(extra_fields)
return blocked
return _build_operation_gate_refusal(
required_operation, reasons, **extra_fields
)
auth_block = _mutation_config_authority_block(required_operation)
if auth_block is not None:
@@ -14494,20 +14734,14 @@ def gitea_acquire_reviewer_pr_lease(
"""Acquire a per-PR reviewer lease before review/merge mutations (#407)."""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"acquired": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
return _build_operation_gate_refusal(
"gitea.read", read_block, acquired=False
)
comment_block = _profile_operation_gate("gitea.pr.comment")
if comment_block:
return {
"success": False,
"acquired": False,
"reasons": comment_block,
"permission_report": _permission_block_report("gitea.pr.comment"),
}
return _build_operation_gate_refusal(
"gitea.pr.comment", comment_block, acquired=False
)
# task=acquire_reviewer_pr_lease so verify_preflight_purity runs shared #604
# anti-stomp for the declared lease-acquire mutation inventory entry.
@@ -14623,20 +14857,14 @@ def gitea_acquire_merger_pr_lease(
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"acquired": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
return _build_operation_gate_refusal(
"gitea.read", read_block, acquired=False
)
comment_block = _profile_operation_gate("gitea.pr.comment")
if comment_block:
return {
"success": False,
"acquired": False,
"reasons": comment_block,
"permission_report": _permission_block_report("gitea.pr.comment"),
}
return _build_operation_gate_refusal(
"gitea.pr.comment", comment_block, acquired=False
)
merge_block = _profile_operation_gate("gitea.pr.merge")
if merge_block:
return {
@@ -19375,14 +19603,12 @@ def gitea_update_pr_branch_by_merge(
# Permission: author branch push / PR mutation surface.
push_block = _profile_operation_gate("gitea.branch.push")
if push_block:
return {
"success": False,
"performed": False,
"mutation_allowed": False,
"reasons": push_block,
"permission_report": _permission_block_report("gitea.branch.push"),
"role_kind": role,
}
return _build_operation_gate_refusal(
"gitea.branch.push",
push_block,
mutation_allowed=False,
role_kind=role,
)
if role != "author":
pre = pr_sync_status.assess_update_pr_branch_preflight(
+13
View File
@@ -142,6 +142,19 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
"permission": "gitea.read",
"role": "author",
},
# #644: Phase 2 Web Console recovery tasks.
"clear_stale_binding": {
"permission": "gitea.read",
"role": "author",
},
"rebind_session_worktree": {
"permission": "gitea.read",
"role": "author",
},
"reconcile_cleanups": {
"permission": "gitea.pr.close",
"role": "reconciler",
},
# PR synchronization lifecycle: assess is read-only (any role with gitea.read);
# update-by-merge is author-only and mutates the PR head via Gitea API.
"assess_pr_sync_status": {
@@ -0,0 +1,453 @@
"""#897: stale-runtime / runtime-mode refusals must not look like permission denials.
Acceptance criteria (issue #897):
* Stale-runtime and runtime-mode refusals are typed distinctly from
profile-permission refusals (distinct ``blocker_kind``).
* A refusal caused by staleness or runtime mode never emits a
``permission_report`` and never names a permission the active profile holds.
* ``_permission_block_report`` verifies the active profile actually lacks the
operation before reporting it missing.
* A stale-runtime refusal reports reconnect-only recovery and never recommends
``gitea_activate_profile`` or an MCP session switch.
* The blocker payload states the observed heads (parity fields).
* Matrix across author / reviewer / merger / reconciler profiles.
* Regression: ``gitea_create_issue`` on a stale daemon under ``prgs-author``
never returns ``missing_permission: gitea.issue.create``.
"""
from __future__ import annotations
import os
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent))
import gitea_config # noqa: E402
import gitea_mcp_server as mcp_server # noqa: E402
SHA_START = "7af40fb5ff7debd5e9165fe97d9c7c279358e175"
SHA_LIVE = "2f4dec832327513118f2fe92b74da25d124a01cb"
ROLE_MATRIX = (
(
"prgs-author",
"author",
"gitea.issue.create",
[
"gitea.read",
"gitea.issue.create",
"gitea.issue.comment",
"gitea.issue.close",
"gitea.branch.create",
"gitea.branch.push",
"gitea.pr.create",
"gitea.pr.comment",
"gitea.repo.commit",
],
["gitea.pr.approve", "gitea.pr.merge", "gitea.pr.request_changes"],
"gitea.pr.merge", # forbidden op for pure-permission case
),
(
"prgs-reviewer",
"reviewer",
"gitea.pr.review",
[
"gitea.read",
"gitea.pr.review",
"gitea.pr.approve",
"gitea.pr.request_changes",
"gitea.pr.comment",
"gitea.issue.comment",
],
["gitea.branch.push", "gitea.pr.create"],
"gitea.branch.push",
),
(
"prgs-merger",
"merger",
"gitea.pr.merge",
[
"gitea.read",
"gitea.pr.merge",
"gitea.pr.comment",
"gitea.issue.comment",
],
["gitea.pr.approve", "gitea.branch.push", "gitea.pr.create"],
"gitea.branch.push",
),
(
"prgs-reconciler",
"reconciler",
"gitea.branch.delete",
[
"gitea.read",
"gitea.branch.delete",
"gitea.pr.comment",
"gitea.issue.comment",
"gitea.pr.close",
"gitea.issue.close",
],
["gitea.pr.approve", "gitea.pr.merge"],
"gitea.pr.merge",
),
)
def _profile(name: str, role: str, allowed: list[str], forbidden: list[str]) -> dict:
return {
"profile_name": name,
"role": role,
"role_kind": role,
"allowed_operations": list(allowed),
"forbidden_operations": list(forbidden),
"identity": "test-user",
}
def _config(profiles: dict) -> dict:
return {
"version": 2,
"profiles": {
name: {
"role": p["role"],
"allowed_operations": p["allowed_operations"],
"forbidden_operations": p["forbidden_operations"],
}
for name, p in profiles.items()
},
"rules": {"allow_runtime_switching": True},
}
class Issue897Helpers(unittest.TestCase):
def test_classify_stale_reason_strings(self):
stale = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
classified = mcp_server._classify_operation_gate_reasons([stale])
self.assertEqual(classified["stale_runtime"], [stale])
self.assertEqual(classified["permission"], [])
self.assertEqual(classified["runtime_mode"], [])
def test_classify_permission_reason(self):
reason = "profile is not allowed to gitea.pr.merge"
classified = mcp_server._classify_operation_gate_reasons([reason])
self.assertEqual(classified["permission"], [reason])
self.assertEqual(classified["stale_runtime"], [])
def test_classify_runtime_mode_reason(self):
reason = (
"runtime mode is 'dev-test' and the mutation targets the "
"production repository; dev/test runtimes must not mutate real "
"issues or PRs (ADR: stable control runtime vs dev runtime)"
)
classified = mcp_server._classify_operation_gate_reasons([reason])
self.assertEqual(classified["runtime_mode"], [reason])
self.assertEqual(classified["stale_runtime"], [])
class Issue897PermissionBlockReport(unittest.TestCase):
def test_holds_op_is_diagnostic_defect_not_missing_permission(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create", "gitea.issue.comment"],
[],
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server.gitea_config, "load_config", return_value=_config({"prgs-author": profile})
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=True
):
report = mcp_server._permission_block_report("gitea.issue.create")
self.assertTrue(report.get("diagnostic_defect"), report)
self.assertIsNone(report.get("missing_permission"), report)
action = (report.get("exact_safe_next_action") or "").lower()
# Must not *recommend* profile switching; mentioning the forbidden
# action in a "do not call" instruction is fine.
self.assertNotIn("call gitea_activate_profile with", action)
self.assertNotIn("switch to the author mcp session", action)
self.assertNotIn("switch to the reviewer mcp session", action)
self.assertIn("diagnostic defect", action)
def test_true_missing_permission_still_reports(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create"],
["gitea.pr.merge"],
)
reviewer = _profile(
"prgs-reviewer",
"reviewer",
["gitea.read", "gitea.pr.merge", "gitea.pr.approve"],
[],
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server.gitea_config,
"load_config",
return_value=_config({"prgs-author": profile, "prgs-reviewer": reviewer}),
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=True
):
report = mcp_server._permission_block_report("gitea.pr.merge")
self.assertFalse(report.get("diagnostic_defect"), report)
self.assertEqual(report.get("missing_permission"), "gitea.pr.merge")
self.assertIn("prgs-reviewer", report.get("matching_configured_profiles") or [])
class Issue897GateRefusalMatrix(unittest.TestCase):
def _stale_parity(self) -> dict:
return {
"in_parity": True,
"stale": False,
"restart_required": True,
"determinable": True,
"startup_head": SHA_START,
"current_head": SHA_START,
"daemon_start_head": SHA_START,
"local_head": SHA_START,
"live_remote_head": SHA_LIVE,
"live_known": True,
"live_stale": True,
"mutation_safe": False,
"reasons": [
f"live remote master is {SHA_LIVE[:12]} but the MCP server "
f"started at {SHA_START[:12]}; the daemon is stale relative "
"to live master -- restart/reconnect before mutating"
],
}
def test_stale_plus_permitted_op_all_roles(self):
for name, role, permitted_op, allowed, forbidden, _forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=permitted_op):
profile = _profile(name, role, allowed, forbidden)
parity = self._stale_parity()
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=list(parity["reasons"])
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
):
blocked = mcp_server._profile_permission_block(permitted_op)
self.assertIsNotNone(blocked, name)
assert blocked is not None
self.assertEqual(
blocked.get("blocker_kind"),
"runtime_reconnect_required",
blocked,
)
self.assertNotIn("permission_report", blocked, blocked)
self.assertTrue(blocked.get("restart_required"), blocked)
self.assertEqual(blocked.get("startup_head"), SHA_START, blocked)
self.assertEqual(blocked.get("live_remote_head"), SHA_LIVE, blocked)
action = (blocked.get("exact_safe_next_action") or "").lower()
self.assertIn("reconnect", action)
self.assertNotIn("call gitea_activate_profile with", action)
self.assertNotIn("switch to the author mcp session", action)
self.assertNotIn("switch to the reviewer mcp session", action)
def test_fresh_plus_forbidden_op_all_roles(self):
for name, role, _permitted, allowed, forbidden, forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=forbidden_op):
profile = _profile(name, role, allowed, forbidden)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_master_parity_block", return_value=[]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
), patch.object(
mcp_server.gitea_config,
"load_config",
return_value=_config({name: profile}),
), patch.object(
mcp_server.gitea_config, "is_runtime_switching_enabled", return_value=False
):
blocked = mcp_server._profile_permission_block(forbidden_op)
self.assertIsNotNone(blocked, name)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "permission_denied", blocked)
self.assertIn("permission_report", blocked, blocked)
report = blocked["permission_report"]
self.assertEqual(report.get("missing_permission"), forbidden_op, report)
self.assertFalse(report.get("diagnostic_defect"), report)
# No runtime reconnect fields for pure permission denial
self.assertNotEqual(
blocked.get("blocker_kind"), "runtime_reconnect_required"
)
def test_stale_plus_forbidden_op_both_causes_separated(self):
for name, role, _permitted, allowed, forbidden, forbidden_op in ROLE_MATRIX:
with self.subTest(profile=name, op=forbidden_op):
profile = _profile(name, role, allowed, forbidden)
parity = self._stale_parity()
stale_reason = parity["reasons"][0]
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=[stale_reason]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": name},
):
# Gate collects both classes; force permission reason too.
with patch.object(
mcp_server,
"_profile_operation_gate",
return_value=[
stale_reason,
f"profile is not allowed to {forbidden_op}",
],
):
blocked = mcp_server._profile_permission_block(forbidden_op)
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(
blocked.get("blocker_kind"), "runtime_reconnect_required", blocked
)
self.assertNotIn("permission_report", blocked, blocked)
self.assertIn("permission_block_reasons", blocked, blocked)
self.assertIn("stale_runtime_reasons", blocked, blocked)
classes = blocked.get("gate_reason_classes") or {}
self.assertTrue(classes.get("stale_runtime"), classes)
self.assertTrue(classes.get("permission"), classes)
def test_runtime_mode_block_no_permission_report(self):
profile = _profile(
"prgs-author",
"author",
["gitea.read", "gitea.issue.create"],
[],
)
runtime_reason = (
"runtime mode is 'dev-test' and the mutation targets the "
"production repository; dev/test runtimes must not mutate real "
"issues or PRs (ADR: stable control runtime vs dev runtime)"
)
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_master_parity_block", return_value=[]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[runtime_reason]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": "prgs-author"},
):
blocked = mcp_server._profile_permission_block("gitea.issue.create")
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "runtime_mode_blocked", blocked)
self.assertNotIn("permission_report", blocked, blocked)
action = (blocked.get("exact_safe_next_action") or "").lower()
self.assertNotIn("call gitea_activate_profile with", action)
self.assertIn("stable control runtime", action)
class Issue897CreateIssueRegression(unittest.TestCase):
def test_create_issue_stale_daemon_never_missing_issue_create(self):
"""Regression AC: stale prgs-author create_issue must not claim missing create."""
profile = _profile(
"prgs-author",
"author",
[
"gitea.read",
"gitea.issue.create",
"gitea.issue.comment",
"gitea.branch.create",
"gitea.branch.push",
"gitea.pr.create",
"gitea.pr.comment",
"gitea.repo.commit",
],
[],
)
stale_reason = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
parity = {
"in_parity": True,
"stale": False,
"restart_required": True,
"determinable": True,
"startup_head": SHA_START,
"current_head": SHA_START,
"daemon_start_head": SHA_START,
"local_head": SHA_START,
"live_remote_head": SHA_LIVE,
"live_known": True,
"live_stale": True,
"mutation_safe": False,
"reasons": [stale_reason],
}
with patch.object(mcp_server, "get_profile", return_value=profile), patch.object(
mcp_server, "_current_master_parity", return_value=parity
), patch.object(
mcp_server, "_master_parity_block", return_value=[stale_reason]
), patch.object(
mcp_server, "_runtime_mode_block", return_value=[]
), patch.object(
mcp_server, "_ensure_matching_profile", return_value=None
), patch.object(
mcp_server.session_ctx,
"mutation_context_audit_fields",
return_value={"session_profile": "prgs-author"},
), patch.object(
mcp_server, "_mutation_config_authority_block", return_value=None
), patch.object(
mcp_server, "_session_context_mutation_block", return_value=None
):
blocked = mcp_server._profile_permission_block(
"gitea.issue.create", remote="prgs"
)
self.assertIsNotNone(blocked)
assert blocked is not None
self.assertEqual(blocked.get("blocker_kind"), "runtime_reconnect_required")
self.assertNotIn("permission_report", blocked)
# Even if a caller still built a raw report, holds-check must not claim missing.
with patch.object(mcp_server, "get_profile", return_value=profile):
raw = mcp_server._permission_block_report("gitea.issue.create")
self.assertIsNone(raw.get("missing_permission"), raw)
self.assertNotEqual(raw.get("missing_permission"), "gitea.issue.create")
def test_permission_report_for_gate_reasons_skips_stale(self):
stale = (
f"live remote master is {SHA_LIVE[:12]} but the MCP server started "
f"at {SHA_START[:12]}; the daemon is stale relative to live master "
"-- restart/reconnect before mutating"
)
self.assertIsNone(
mcp_server._permission_report_for_gate_reasons(
"gitea.issue.comment", [stale]
)
)
if __name__ == "__main__":
unittest.main()
+182
View File
@@ -0,0 +1,182 @@
"""Unit and integration tests for Phase 2 Web Console recovery controls (#644)."""
from __future__ import annotations
import json
import os
import unittest
from unittest.mock import patch
from starlette.testclient import TestClient
from webui import console_audit, console_authz, console_recovery
from webui.app import create_app
class TestConsoleRecovery(unittest.TestCase):
def test_diagnose_recovery_healthy(self) -> None:
diag = console_recovery.diagnose_recovery()
self.assertIn(diag.status, {console_recovery.STATUS_HEALTHY, console_recovery.STATUS_ACTION_REQUIRED})
self.assertIsInstance(diag.playbooks, tuple)
self.assertGreaterEqual(len(diag.playbooks), 4)
playbook_ids = {pb.playbook_id for pb in diag.playbooks}
self.assertIn(console_recovery.PLAYBOOK_CLEAR_STALE_BINDING, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_REBIND_SESSION, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_RECONCILE_CLEANUPS, playbook_ids)
self.assertIn(console_recovery.PLAYBOOK_SANCTIONED_RESTART, playbook_ids)
def test_confirmation_phrase_generation_and_matching(self) -> None:
phrase = console_recovery.confirmation_phrase("clear_stale_binding")
self.assertEqual(phrase, "confirm clear_stale_binding")
self.assertTrue(console_recovery.confirmation_matches("clear_stale_binding", "confirm clear_stale_binding"))
self.assertFalse(console_recovery.confirmation_matches("clear_stale_binding", "wrong phrase"))
phrase_target = console_recovery.confirmation_phrase("sanctioned_restart", "gitea-author")
self.assertEqual(phrase_target, "confirm sanctioned_restart gitea-author")
self.assertTrue(console_recovery.confirmation_matches("sanctioned_restart", "confirm sanctioned_restart gitea-author", "gitea-author"))
def test_build_recovery_preview(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
preview = console_recovery.build_recovery_preview(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
target="test-worktree",
principal=principal,
)
self.assertEqual(preview["playbook_id"], console_recovery.PLAYBOOK_CLEAR_STALE_BINDING)
self.assertEqual(preview["action_id"], console_recovery.ACTION_CLEAR_STALE_BINDING)
self.assertEqual(preview["confirmation_phrase"], "confirm clear_stale_binding test-worktree")
self.assertTrue(len(preview["mutation_ledger"]) >= 3)
self.assertTrue(preview["authorization"]["allowed"])
def test_build_recovery_preview_unknown_playbook(self) -> None:
preview = console_recovery.build_recovery_preview("unknown_playbook")
self.assertFalse(preview.get("allowed"))
self.assertEqual(preview.get("error"), "unknown_playbook")
def test_execute_recovery_playbook_confirmation_mismatch(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation="invalid confirmation",
principal=principal,
)
self.assertFalse(result["success"])
self.assertFalse(result["allowed"])
self.assertEqual(result["error"], "confirmation_mismatch")
def test_execute_recovery_playbook_unauthorized(self) -> None:
# Anonymous principal has viewer role -> should be denied
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation="confirm clear_stale_binding",
principal=console_authz.ANONYMOUS,
)
self.assertFalse(result["success"])
self.assertFalse(result["allowed"])
self.assertEqual(result["error"], console_authz.DENY_UNAUTHENTICATED)
def test_execute_recovery_playbook_clear_stale_binding_success(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
phrase = console_recovery.confirmation_phrase(console_recovery.PLAYBOOK_CLEAR_STALE_BINDING)
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_CLEAR_STALE_BINDING,
confirmation=phrase,
principal=principal,
)
self.assertTrue(result["allowed"])
self.assertIn("applied_result", result)
self.assertIn("post_recovery_verification", result)
self.assertIn("audit", result)
self.assertEqual(result["audit"]["event"]["action"], console_recovery.ACTION_CLEAR_STALE_BINDING)
def test_execute_recovery_playbook_rebind_session_success(self) -> None:
principal = console_authz.Principal("[email protected]", console_authz.OPERATOR, console_authz.IDENTITY_LOCAL_DEV, True)
phrase = console_recovery.confirmation_phrase(console_recovery.PLAYBOOK_REBIND_SESSION, "branches/feat-issue-644")
result = console_recovery.execute_recovery_playbook(
playbook_id=console_recovery.PLAYBOOK_REBIND_SESSION,
confirmation=phrase,
target="branches/feat-issue-644",
principal=principal,
)
self.assertTrue(result["allowed"])
self.assertTrue(result["success"])
self.assertEqual(result["applied_result"]["rebound_worktree"], "branches/feat-issue-644")
def test_verify_post_recovery(self) -> None:
verification = console_recovery.verify_post_recovery()
self.assertIn("clean", verification)
self.assertIn("status", verification)
self.assertIn("reasons", verification)
class TestConsoleRecoveryApi(unittest.TestCase):
def setUp(self) -> None:
self.app = create_app()
self.client = TestClient(self.app)
def test_api_recovery_diagnose(self) -> None:
res = self.client.get("/api/v1/system/recovery/diagnose")
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertIn("status", data)
self.assertIn("clean", data)
self.assertIn("playbooks", data)
self.assertTrue(len(data["playbooks"]) >= 4)
def test_api_recovery_preview(self) -> None:
res = self.client.post(
"/api/v1/system/recovery/preview",
json={"playbook_id": "clear_stale_binding", "target": "active"},
)
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertEqual(data["playbook_id"], "clear_stale_binding")
self.assertEqual(data["confirmation_phrase"], "confirm clear_stale_binding active")
self.assertIn("mutation_ledger", data)
def test_api_recovery_apply_denied_without_auth(self) -> None:
res = self.client.post(
"/api/v1/system/recovery/apply",
json={"playbook_id": "clear_stale_binding", "confirmation": "confirm clear_stale_binding"},
)
self.assertEqual(res.status_code, 400)
data = res.json()
self.assertFalse(data["success"])
self.assertFalse(data["allowed"])
def test_api_recovery_apply_with_dev_auth(self) -> None:
env = {
"WEBUI_AUTH_MODE": "local_dev",
"WEBUI_DEV_SUBJECT": "[email protected]",
"WEBUI_DEV_ROLE": "operator",
}
with patch.dict(os.environ, env):
res = self.client.post(
"/api/v1/system/recovery/apply",
json={
"playbook_id": "rebind_session_worktree",
"target": "branches/feat-issue-644",
"confirmation": "confirm rebind_session_worktree branches/feat-issue-644",
},
)
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertTrue(data["success"])
self.assertTrue(data["allowed"])
self.assertEqual(data["playbook_id"], "rebind_session_worktree")
def test_api_recovery_verify(self) -> None:
res = self.client.get("/api/v1/system/recovery/verify")
self.assertEqual(res.status_code, 200)
data = res.json()
self.assertIn("clean", data)
self.assertIn("status", data)
if __name__ == "__main__":
unittest.main()
+83
View File
@@ -200,6 +200,84 @@ async def system_health(request: Request) -> HTMLResponse:
)
async def api_recovery_diagnose(_request: Request) -> JSONResponse:
from webui.console_recovery import diagnose_recovery
diag = diagnose_recovery()
return JSONResponse({
"status": diag.status,
"clean": diag.clean,
"stale_runtime": diag.stale_runtime,
"master_parity": diag.master_parity,
"stale_binding": diag.stale_binding,
"contamination": diag.contamination,
"worktree_anomalies": list(diag.worktree_anomalies),
"reasons": list(diag.reasons),
"playbooks": [
{
"playbook_id": pb.playbook_id,
"label": pb.label,
"action_id": pb.action_id,
"description": pb.description,
"eligible": pb.eligible,
"requires_confirmation": pb.requires_confirmation,
"reason": pb.reason,
"params_schema": pb.params_schema,
}
for pb in diag.playbooks
],
})
async def api_recovery_preview(request: Request) -> JSONResponse:
from webui.console_recovery import build_recovery_preview
body = {}
try:
body = await request.json()
except Exception:
pass
playbook_id = body.get("playbook_id") or request.query_params.get("playbook_id") or ""
target = body.get("target") or request.query_params.get("target")
principal = resolve_principal(request.headers)
preview = build_recovery_preview(
playbook_id=playbook_id,
target=target,
params=body,
principal=principal,
)
status = 200 if preview.get("playbook_id") else 400
return JSONResponse(preview, status_code=status)
async def api_recovery_apply(request: Request) -> JSONResponse:
from webui.console_recovery import execute_recovery_playbook
body = {}
try:
body = await request.json()
except Exception:
pass
playbook_id = body.get("playbook_id", "")
confirmation = body.get("confirmation", "")
target = body.get("target")
principal = resolve_principal(request.headers)
request_id = getattr(request.state, "request_id", None)
result = execute_recovery_playbook(
playbook_id=playbook_id,
confirmation=confirmation,
target=target,
params=body,
principal=principal,
request_id=request_id,
)
status_code = 200 if result.get("success") else 400
return JSONResponse(result, status_code=status_code)
async def api_recovery_verify(_request: Request) -> JSONResponse:
from webui.console_recovery import verify_post_recovery
verification = verify_post_recovery()
return JSONResponse(verification, status_code=200)
async def queue(_request: Request) -> HTMLResponse:
snapshot = load_queue_snapshot()
return HTMLResponse(render_page(title="Queue", body_html=render_queue_page(snapshot)))
@@ -818,6 +896,11 @@ def create_app(*, bind_host: str | None = None) -> Starlette:
api_console_security_model,
methods=["GET"],
),
# #644 Phase 2 Recovery API routes
Route("/api/v1/system/recovery/diagnose", api_recovery_diagnose, methods=["GET"]),
Route("/api/v1/system/recovery/preview", api_recovery_preview, methods=["POST", "GET"]),
Route("/api/v1/system/recovery/apply", api_recovery_apply, methods=["POST"]),
Route("/api/v1/system/recovery/verify", api_recovery_verify, methods=["POST", "GET"]),
*[
Route(path, phase_stub, methods=["GET"])
for path in STUB_PAGES
+34
View File
@@ -277,6 +277,40 @@ _ACTION_SPECS: tuple[ConsoleAction, ...] = (
phase=2,
summary="Restart one MCP namespace via the host supervisor.",
),
# #644: Phase 2 recovery controls & playbooks.
ConsoleAction(
action_id="system.clear_stale_binding",
task_key="clear_stale_binding",
action_class=CLASS_WRITE,
minimum_role=OPERATOR,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Clear provably stale or superseded GITEA_ACTIVE_WORKTREE binding.",
),
ConsoleAction(
action_id="system.rebind_session_worktree",
task_key="rebind_session_worktree",
action_class=CLASS_WRITE,
minimum_role=OPERATOR,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Rebind session worktree context to verified lease worktree.",
),
ConsoleAction(
action_id="system.reconcile_cleanups",
task_key="reconcile_cleanups",
action_class=CLASS_PRIVILEGED,
minimum_role=CONTROLLER,
requires_confirmation=True,
dual_control=False,
break_glass=False,
phase=2,
summary="Run reconciler cleanup for merged or superseded PR branches.",
),
)
ACTIONS: dict[str, ConsoleAction] = {a.action_id: a for a in _ACTION_SPECS}
+573
View File
@@ -0,0 +1,573 @@
"""Web Console Phase 2 Recovery Controls & Playbooks (#644).
Provides canonical recovery controls for the web console:
1. Diagnosis: Surfaces stale runtimes, worktree binding errors, contamination markers,
and un-reconciled cleanups.
2. Gated Actions & Playbooks: Guided recovery (rebind session worktree, clear stale
binding, trigger reconciler cleanups, sanctioned restart).
3. RBAC, Contamination (#630), and Master Parity (#610) integration.
4. Audit trail via ``console_audit`` and mandatory post-recovery revalidation.
"""
from __future__ import annotations
import os
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
import master_parity_gate
import merged_cleanup_reconcile
import runtime_recovery_guard
import stale_binding_recovery
from webui import console_audit, console_authz, sanctioned_restart, system_health, worktree_scanner
# --- Recovery Statuses ------------------------------------------------------
STATUS_HEALTHY = "healthy"
STATUS_ACTION_REQUIRED = "action_required"
STATUS_BLOCKED_CONTAMINATION = "blocked_contamination"
STATUS_RECONNECT_REQUIRED = "reconnect_required"
# --- Playbook Identifiers ---------------------------------------------------
PLAYBOOK_CLEAR_STALE_BINDING = "clear_stale_binding"
PLAYBOOK_REBIND_SESSION = "rebind_session_worktree"
PLAYBOOK_RECONCILE_CLEANUPS = "reconcile_cleanups"
PLAYBOOK_SANCTIONED_RESTART = "sanctioned_restart"
KNOWN_PLAYBOOKS: tuple[str, ...] = (
PLAYBOOK_CLEAR_STALE_BINDING,
PLAYBOOK_REBIND_SESSION,
PLAYBOOK_RECONCILE_CLEANUPS,
PLAYBOOK_SANCTIONED_RESTART,
)
# --- Console Action Mapping -------------------------------------------------
ACTION_CLEAR_STALE_BINDING = "system.clear_stale_binding"
ACTION_REBIND_SESSION = "system.rebind_session_worktree"
ACTION_RECONCILE_CLEANUPS = "system.reconcile_cleanups"
PLAYBOOK_ACTIONS: dict[str, str] = {
PLAYBOOK_CLEAR_STALE_BINDING: ACTION_CLEAR_STALE_BINDING,
PLAYBOOK_REBIND_SESSION: ACTION_REBIND_SESSION,
PLAYBOOK_RECONCILE_CLEANUPS: ACTION_RECONCILE_CLEANUPS,
PLAYBOOK_SANCTIONED_RESTART: sanctioned_restart.ACTION_RESTART_NAMESPACE,
}
@dataclass(frozen=True)
class RecoveryLedgerEntry:
"""One planned recovery step displayed before execution."""
sequence: int
step: str
summary: str
executes_process_kill: bool = False
@dataclass(frozen=True)
class PlaybookDescriptor:
"""Structured recovery playbook option returned during diagnosis."""
playbook_id: str
label: str
action_id: str
description: str
eligible: bool
requires_confirmation: bool
reason: str
params_schema: dict[str, Any] = field(default_factory=dict)
@dataclass(frozen=True)
class RecoveryDiagnosis:
"""Complete diagnostic snapshot of control-plane recovery needs."""
status: str
clean: bool
stale_runtime: dict[str, Any]
master_parity: dict[str, Any]
stale_binding: dict[str, Any]
contamination: dict[str, Any]
worktree_anomalies: tuple[str, ...]
playbooks: tuple[PlaybookDescriptor, ...]
reasons: tuple[str, ...]
def _repo_root(custom_path: Path | str | None = None) -> Path:
if custom_path:
return Path(custom_path).resolve()
override = (os.environ.get("WEBUI_REPO_ROOT") or "").strip()
if override:
return Path(override).resolve()
return Path(__file__).resolve().parent.parent
def confirmation_phrase(playbook_id: str, target: str | None = None) -> str:
"""Construct exact confirmation phrase required for a recovery playbook."""
clean_target = (target or "").strip()
if clean_target:
return f"confirm {playbook_id} {clean_target}"
return f"confirm {playbook_id}"
def confirmation_matches(
playbook_id: str, confirmation: str | None, target: str | None = None
) -> bool:
expected = confirmation_phrase(playbook_id, target)
return str(confirmation or "").strip() == expected
def _build_ledger(
playbook_id: str, target: str | None = None
) -> tuple[RecoveryLedgerEntry, ...]:
if playbook_id == PLAYBOOK_CLEAR_STALE_BINDING:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"clear_env",
f"Remove stale env binding GITEA_ACTIVE_WORKTREE ({target or 'active'}).",
),
RecoveryLedgerEntry(
3, "audit", "Record clear_stale_binding event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run diagnosis to verify clean binding state."
),
)
if playbook_id == PLAYBOOK_REBIND_SESSION:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"rebind_worktree",
f"Rebind session worktree context safely to {target or 'target worktree'}.",
),
RecoveryLedgerEntry(
3, "audit", "Record rebind_session_worktree event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run diagnosis to verify worktree binding state."
),
)
if playbook_id == PLAYBOOK_RECONCILE_CLEANUPS:
return (
RecoveryLedgerEntry(1, "quiesce", "Stop admitting new gated mutations."),
RecoveryLedgerEntry(
2,
"reconcile_cleanups",
"Execute sanctioned reconciler cleanup for merged or superseded PRs.",
),
RecoveryLedgerEntry(
3, "audit", "Record reconcile_cleanups event in console audit log."
),
RecoveryLedgerEntry(
4, "revalidate", "Re-run worktree scanner to verify clean tree."
),
)
if playbook_id == PLAYBOOK_SANCTIONED_RESTART:
restart_ledger = sanctioned_restart._mutation_ledger(
target or "gitea-author", sanctioned_restart.MODE_RESTART
)
return tuple(
RecoveryLedgerEntry(
sequence=e.sequence,
step=e.step,
summary=e.summary,
executes_process_kill=e.executes_process_kill,
)
for e in restart_ledger
)
return (
RecoveryLedgerEntry(1, "unspecified", f"Execute recovery playbook {playbook_id}."),
)
def diagnose_recovery(
repo_path: Path | str | None = None,
env: dict[str, str] | None = None,
active_worktree_val: str | None = None,
session_lease_wt: str | None = None,
role_kind: str | None = None,
) -> RecoveryDiagnosis:
"""Run full control-plane diagnostics to determine recovery needs and options."""
root = _repo_root(repo_path)
source_env = dict(env) if env is not None else dict(os.environ)
reasons: list[str] = []
# 1. Stale runtime assessment
stale_runtime_obj = system_health.assess_stale_runtime(root)
stale_runtime_dict = {
"daemon_head": stale_runtime_obj.daemon_head,
"checkout_head": stale_runtime_obj.checkout_head,
"remote_head": stale_runtime_obj.remote_head,
"stale": stale_runtime_obj.stale,
"determinable": stale_runtime_obj.determinable,
"mutation_safe": stale_runtime_obj.mutation_safe,
"reasons": list(stale_runtime_obj.reasons),
}
if stale_runtime_obj.stale:
reasons.append("Runtime HEAD disagrees with checkout/remote HEAD.")
# 2. Master parity assessment
checkout_head = stale_runtime_obj.checkout_head
startup_dict = master_parity_gate.capture_startup_parity(str(root), head=checkout_head)
parity_dict = master_parity_gate.assess_master_parity(startup_dict, checkout_head)
if not parity_dict.get("in_parity", True):
reasons.append("Repository is not in master parity.")
# 3. Worktree binding classification
boot_bindings = stale_binding_recovery.snapshot_boot_bindings(source_env)
active_val = (
active_worktree_val
if active_worktree_val is not None
else source_env.get(stale_binding_recovery.ACTIVE_WORKTREE_ENV)
)
boot_inherited = bool(boot_bindings.get("active_worktree") and active_val == boot_bindings.get("active_worktree"))
path_exists = None
if active_val:
path_exists = os.path.exists(os.path.realpath(active_val))
binding_class = stale_binding_recovery.classify_active_worktree_binding(
active_value=active_val,
session_lease_worktree=session_lease_wt,
boot_inherited=boot_inherited,
path_exists=path_exists,
role_kind=role_kind,
)
if binding_class.get("clear_eligible"):
reasons.append(
f"Active worktree binding is stale ({binding_class.get('classification')})."
)
elif binding_class.get("classification") == stale_binding_recovery.CLASSIFICATION_UNVERIFIED_INHERITED:
reasons.append("Inherited worktree binding is unverified.")
# 4. Contamination assessment (#630)
contamination_dict = runtime_recovery_guard.assess_contamination_gate(
marker=None, task=None, actual_role=role_kind
)
if contamination_dict.get("block"):
reasons.append("Runtime is contaminated by manual process kill (#630).")
# 5. Worktree scanner hygiene & anomalies
hygiene = worktree_scanner.load_hygiene_snapshot(project_root=str(root))
worktree_anomalies = hygiene.anomalies
# Determine status & eligible playbooks
playbooks: list[PlaybookDescriptor] = []
# Playbook 1: Clear Stale Binding
clear_eligible = bool(binding_class.get("clear_eligible"))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_CLEAR_STALE_BINDING,
label="Clear Stale Worktree Binding",
action_id=ACTION_CLEAR_STALE_BINDING,
description="Clear provably stale or superseded GITEA_ACTIVE_WORKTREE environment binding.",
eligible=clear_eligible,
requires_confirmation=True,
reason=(
f"Binding classified as {binding_class.get('classification')}; clear is authorized."
if clear_eligible
else "Active worktree binding is clean, corroborated, or absent."
),
)
)
# Playbook 2: Rebind Session Worktree
rebind_eligible = bool(
active_val
or binding_class.get("classification") == stale_binding_recovery.CLASSIFICATION_UNVERIFIED_INHERITED
)
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_REBIND_SESSION,
label="Rebind Session Worktree",
action_id=ACTION_REBIND_SESSION,
description="Rebind or synchronize session worktree binding safely with active lease.",
eligible=rebind_eligible,
requires_confirmation=True,
reason=(
"Session worktree binding can be rebound to verified lease worktree."
if rebind_eligible
else "Session worktree is properly bound."
),
params_schema={"target_worktree": "string"},
)
)
# Playbook 3: Reconcile Cleanups
reconcile_eligible = bool(hygiene.anomalies or any(e.classification in {"stale-clean", "detached-review"} for e in hygiene.entries))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_RECONCILE_CLEANUPS,
label="Trigger Reconciler Cleanups",
action_id=ACTION_RECONCILE_CLEANUPS,
description="Run sanctioned reconciler cleanup preview and apply for merged/superseded PR branches.",
eligible=reconcile_eligible,
requires_confirmation=True,
reason=(
f"Worktree hygiene scanner detected {len(hygiene.anomalies)} anomalies and cleanups needed."
if reconcile_eligible
else "No reconciler cleanups pending."
),
)
)
# Playbook 4: Sanctioned Restart
restart_eligible = bool(stale_runtime_obj.stale or contamination_dict.get("contaminated"))
playbooks.append(
PlaybookDescriptor(
playbook_id=PLAYBOOK_SANCTIONED_RESTART,
label="Sanctioned MCP Restart",
action_id=sanctioned_restart.ACTION_RESTART_NAMESPACE,
description="Restart MCP daemon via configured host supervisor without manual process kill.",
eligible=restart_eligible,
requires_confirmation=True,
reason=(
"Stale runtime or contamination detected; host supervisor restart available."
if restart_eligible
else "Runtime is healthy and clean."
),
params_schema={"namespace": "string", "mode": "restart|reload"},
)
)
clean = not reasons and not contamination_dict.get("contaminated")
if contamination_dict.get("contaminated"):
status = STATUS_BLOCKED_CONTAMINATION
elif reasons:
status = STATUS_ACTION_REQUIRED
else:
status = STATUS_HEALTHY
return RecoveryDiagnosis(
status=status,
clean=clean,
stale_runtime=stale_runtime_dict,
master_parity=parity_dict,
stale_binding=binding_class,
contamination=contamination_dict,
worktree_anomalies=tuple(worktree_anomalies),
playbooks=tuple(playbooks),
reasons=tuple(reasons),
)
def build_recovery_preview(
playbook_id: str,
target: str | None = None,
params: dict[str, Any] | None = None,
principal: console_authz.Principal | None = None,
env: dict[str, str] | None = None,
) -> dict[str, Any]:
"""Generate dry-run preview & mutation ledger for a recovery playbook."""
if playbook_id not in KNOWN_PLAYBOOKS:
return {
"allowed": False,
"error": "unknown_playbook",
"detail": f"Playbook {playbook_id!r} is not a registered recovery playbook.",
}
action_id = PLAYBOOK_ACTIONS[playbook_id]
action = console_authz.get_action(action_id)
decision = console_authz.authorize(action_id, principal)
phrase = confirmation_phrase(playbook_id, target)
ledger = _build_ledger(playbook_id, target)
return {
"playbook_id": playbook_id,
"action_id": action_id,
"target": target,
"required_role": action.minimum_role if action else console_authz.OPERATOR,
"required_permission": action.mcp_permission if action else "gitea.read",
"requires_confirmation": True,
"confirmation_phrase": phrase,
"mutation_ledger": [asdict(entry) for entry in ledger],
"authorization": decision.to_dict(),
"params": dict(params or {}),
"execution_enabled": False,
}
def execute_recovery_playbook(
playbook_id: str,
confirmation: str | None = None,
target: str | None = None,
params: dict[str, Any] | None = None,
principal: console_authz.Principal | None = None,
env: dict[str, str] | None = None,
request_id: str | None = None,
session_id: str | None = None,
) -> dict[str, Any]:
"""Gated execution of a recovery playbook with audit logging and revalidation."""
if playbook_id not in KNOWN_PLAYBOOKS:
return {
"success": False,
"allowed": False,
"error": "unknown_playbook",
"detail": f"Playbook {playbook_id!r} is not known.",
}
action_id = PLAYBOOK_ACTIONS[playbook_id]
source_env = dict(env) if env is not None else dict(os.environ)
# 1. Authorization check
decision = console_authz.authorize(action_id, principal)
if not decision.allowed:
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code=decision.reason_code,
detail=decision.detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": decision.reason_code,
"detail": decision.detail,
}
# 2. Confirmation phrase check
if not confirmation_matches(playbook_id, confirmation, target):
expected = confirmation_phrase(playbook_id, target)
detail = f"Confirmation phrase mismatch. Expected: {expected!r}"
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="confirmation_mismatch",
detail=detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": "confirmation_mismatch",
"detail": detail,
"expected_confirmation_phrase": expected,
}
# 3. Contamination rule (#630) check
role_str = principal.role if principal else None
contam = runtime_recovery_guard.assess_contamination_gate(marker=None, task=action_id, actual_role=role_str)
if contam.get("block"):
if playbook_id != PLAYBOOK_RECONCILE_CLEANUPS:
detail = "Runtime is contaminated by a manual process kill (#630). Run reconciler cleanup playbook first."
console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="contaminated_runtime",
detail=detail,
request_id=request_id,
session_id=session_id,
)
return {
"success": False,
"allowed": False,
"error": "contaminated_runtime",
"detail": detail,
}
# 4. Execute playbook action
applied_result: dict[str, Any] = {"performed": False}
if playbook_id == PLAYBOOK_CLEAR_STALE_BINDING:
diagnosis = diagnose_recovery(env=source_env)
plan = stale_binding_recovery.plan_recovery(diagnosis.stale_binding)
applied_result = stale_binding_recovery.apply_recovery(plan, env=source_env)
elif playbook_id == PLAYBOOK_REBIND_SESSION:
target_wt = target or (params or {}).get("target_worktree")
if target_wt:
source_env[stale_binding_recovery.ACTIVE_WORKTREE_ENV] = target_wt
applied_result = {
"performed": True,
"rebound_worktree": target_wt,
"cleared_stale": True,
}
else:
applied_result = {
"performed": False,
"reason": "No target_worktree specified for rebind.",
}
elif playbook_id == PLAYBOOK_RECONCILE_CLEANUPS:
try:
snapshot = merged_cleanup_reconcile.reconcile_merged_cleanups(
apply=True, project_root=str(_repo_root())
)
applied_result = {
"performed": True,
"reconciled_count": len(snapshot.get("reconciled") or []),
"snapshot": snapshot,
}
except Exception as exc:
applied_result = {
"performed": False,
"error": str(exc),
}
elif playbook_id == PLAYBOOK_SANCTIONED_RESTART:
ns = target or (params or {}).get("namespace", "gitea-author")
md = (params or {}).get("mode", sanctioned_restart.MODE_RESTART)
restart_res = sanctioned_restart.execute_restart(
namespace=ns,
mode=md,
principal=principal,
confirmation=f"{md} {ns}",
env=source_env,
request_id=request_id,
session_id=session_id,
)
applied_result = restart_res
performed = bool(applied_result.get("performed") or applied_result.get("allowed"))
# 5. Record Audit Log
audit_record = console_audit.record_event(
action_id=action_id,
result=console_audit.RESULT_ALLOWED if performed else console_audit.RESULT_DENIED,
principal=principal,
target={"playbook_id": playbook_id, "target": target},
reason_code="recovery_executed" if performed else "recovery_failed",
detail=f"Executed recovery playbook {playbook_id}",
request_id=request_id,
session_id=session_id,
metadata={"applied_result": applied_result},
)
# 6. Post-recovery verification recheck
post_verification = verify_post_recovery(env=source_env)
return {
"success": performed,
"allowed": True,
"playbook_id": playbook_id,
"action_id": action_id,
"applied_result": applied_result,
"audit": audit_record,
"post_recovery_verification": post_verification,
}
def verify_post_recovery(
repo_path: Path | str | None = None, env: dict[str, str] | None = None
) -> dict[str, Any]:
"""Revalidate control-plane state post-recovery before clean status."""
diag = diagnose_recovery(repo_path, env)
return {
"clean": diag.clean,
"status": diag.status,
"stale_runtime_clean": not diag.stale_runtime.get("stale"),
"binding_clean": not diag.stale_binding.get("clear_eligible"),
"contamination_clean": not diag.contamination.get("contaminated"),
"anomalies_count": len(diag.worktree_anomalies),
"reasons": list(diag.reasons),
}
+7
View File
@@ -178,6 +178,13 @@ def build_action_registry() -> ActionRegistry:
("system.restart_namespace", "Restart MCP namespace",
"restart_namespace", "host.supervisor_restart",
"Restart one MCP namespace via the host supervisor."),
# #644: Phase 2 recovery playbooks & controls.
("system.clear_stale_binding", "Clear stale binding", "clear_stale_binding",
"console.clear_stale_binding", "Clear provably stale or superseded env binding."),
("system.rebind_session_worktree", "Rebind session worktree", "rebind_session_worktree",
"console.rebind_session_worktree", "Rebind session worktree to verified lease."),
("system.reconcile_cleanups", "Reconcile cleanups", "reconcile_cleanups",
"console.reconcile_cleanups", "Run reconciler cleanup for merged or superseded PRs."),
)
actions = tuple(
GatedAction(
+37 -16
View File
@@ -270,24 +270,45 @@ def _probe_error_card(snapshot: SystemHealthSnapshot) -> str:
def _recovery_card() -> str:
"""Sanctioned recovery pointers only — never a manual process kill (#630)."""
"""Sanctioned recovery controls & playbooks (#644, Phase 2)."""
try:
from webui import console_recovery
diag = console_recovery.diagnose_recovery()
status_badge = f"<span class='status-pill {diag.status}'>{diag.status}</span>"
playbook_lis = ""
for pb in diag.playbooks:
elig = "eligible" if pb.eligible else "disabled"
playbook_lis += (
f"<li><strong>{pb.label}</strong> (<code>{pb.playbook_id}</code>) — "
f"<span class='badge {elig}'>{elig}</span>: {pb.description} "
f"<em class='muted'>({pb.reason})</em></li>"
)
reasons_html = ""
if diag.reasons:
items = "".join(f"<li>{r}</li>" for r in diag.reasons)
reasons_html = f"<ul class='reasons'>{items}</ul>"
else:
reasons_html = "<p class='clean-note'>No recovery actions currently required. Control plane is healthy.</p>"
return (
"<section class='health-card recovery-card'>"
f"<h3>Sanctioned Recovery Controls (Phase 2 #644) {status_badge}</h3>"
"<p class='muted'>Guided recovery wizard: Diagnose &rarr; Preview &rarr; Confirm &rarr; Verify. "
"Reconnect the MCP client from the IDE, then re-run the blocked cycle. "
"Never kill the daemon process manually: unmanaged kills are recorded as runtime contamination (#630).</p>"
f"{reasons_html}"
"<h4>Available Recovery Playbooks</h4>"
f"<ul class='playbooks-list'>{playbook_lis}</ul>"
"<p class='meta'>APIs: <code>/api/v1/system/recovery/diagnose</code>, "
"<code>/api/v1/system/recovery/preview</code>, <code>/api/v1/system/recovery/apply</code>, "
"<code>/api/v1/system/recovery/verify</code>.</p>"
"</section>"
)
except Exception as exc:
return (
"<section class='health-card'>"
"<h3>Recovery</h3>"
"<p class='muted'>This dashboard is read-only. Restart and reload "
"controls arrive in Phase 2 (#642); until then recovery runs through "
"the sanctioned client reconnect / operator restart path.</p>"
"<ul class='reasons'>"
"<li><a href='/runtime'>Runtime health</a> — active profile, workflow "
"hashes, and shell health.</li>"
"<li><a href='/sessions'>Runtime and sessions</a> — namespaces, session "
"rows, worktree bindings, and contamination markers (#641).</li>"
"<li>Reconnect the MCP client from the IDE, then re-run the blocked "
"cycle. Never kill the daemon process manually: unmanaged kills are "
"recorded as runtime contamination (#630).</li>"
"<li>See <code>docs/webui-local-dev.md</code> for the documented "
"recovery sequence.</li>"
"</ul>"
"<h3>Sanctioned Recovery Controls (Phase 2 #644)</h3>"
f"<p class='error'>Recovery diagnostics unavailable: {exc}</p>"
"</section>"
)