Merge branch 'master' into feat/issue-648-notifications-console

2026-07-25 17:29:57 -05:00
parent a64ba08e27 26f54851d1
commit 71031c812e
17 changed files with 2491 additions and 36 deletions
@@ -0,0 +1,94 @@
+# MCP scoped recovery playbook (#669)
+
+**Parent:** [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655)  
+**Vision / roadmap:** [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) · [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653)  
+**Class matrix:** [#663](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/663) · `docs/mcp-restart-classes.md`  
+**Coordinator:** [#658](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/658) · `restart_coordinator.py`  
+**Audit lineage:** [#665](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/665)
+
+## Decision
+
+Full-server MCP reset is a **last resort**. Prefer the narrowest recovery that
+can clear the symptom. The coordinator **refuses** `rolling_mcp_restart`,
+`full_mcp_restart`, and `host_restart` unless:
+
+1. The inventory carries a prior **attempt log** of at least one *insufficient*
+   narrower recovery, **or**
+2. **Break-glass** is authorized
+   (`request_break_glass` + `GITEA_BREAKGLASS_RESTART_AUTHORIZATION`).
+
+Break-glass still never bypasses the #663 class matrix (role/permission).
+
+## Ladder (narrow → broad)
+
+| Rank | Action | Self-service | Implementation / delegation |
+|---:|---|---|---|
+| 0 | `client_reconnect` | yes | Host auto-reconnect / client reconnect · [#584](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/584) · `docs/mcp-namespace-eof-recovery.md` |
+| 1 | `capability_refresh` | yes | `gitea_resolve_task_capability` + `gitea_whoami` · [#610](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/610) · [#685](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/685) |
+| 2 | `session_reconnect` | yes | Runtime rebind + explicit `worktree_path` · [#543](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/543) · [#618](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/618) |
+| 3 | `configuration_reload` | no | Class `configuration_reload` · console reload · [#642](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/642) |
+| 4 | `lease_recovery` | no | Lock/lease recovery paths · [#702](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/702) · [#753](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/753) · [#790](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/790) |
+| 5 | `worker_restart` | no | Class `worker_restart` · #663 |
+| 6 | `role_runtime_restart` | no | Class `role_runtime_restart` · console restart · #642/#663 |
+| 7 | `connector_restart` | no | Class `connector_restart` · #663 |
+| 8 | `rolling_mcp_restart` | no | Class `rolling_mcp_restart` · design [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668) · **attempt log required** |
+| 9 | `full_mcp_restart` | no | Class `full_mcp_restart` · **attempt log required** |
+| 10 | `host_restart` | no | Class `host_restart` · **attempt log required** |
+
+Machine-readable source of truth: `recovery_playbook.RECOVERY_LADDER` and
+`recovery_playbook.ladder_document()`.
+
+## Attempt log shape
+
+Each prior attempt is a mapping:
+
+```json
+{
+  "action": "client_reconnect",
+  "outcome": "insufficient",
+  "reason": "transport still closed after IDE reconnect",
+  "actor": "prgs-controller-12345",
+  "recorded_at": "2026-07-25T21:00:00+00:00"
+}
+```
+
+Outcomes that count toward escalation: `failed`, `insufficient`, `denied`,
+`unresolved`, `timeout`, `error`.
+
+Pass attempts into the coordinator via inventory
+`prior_recovery_attempts` or the MCP tool argument
+`prior_recovery_attempts_json` on `gitea_request_mcp_restart`.
+
+Helper: `recovery_playbook.build_attempt_record(...)`.
+
+## Symptom → first rung
+
+`recovery_playbook.recommend_actions(symptoms=[...])` maps symptoms such as
+`transport_eof`, `stale_capability`, `stale_lease`, `daemon_corrupt` to the
+narrowest recommended action, then walks the ladder. Soft recommendations
+never replace the hard gate on broad restarts.
+
+## Enforcement points
+
+1. **`recovery_playbook.assess_escalation`** — pure gate.
+2. **`restart_coordinator.evaluate_restart_impact`** — when `restart_class` is
+   set (policy-enforced path), broad classes require the gate; report fields
+   `attempt_log_satisfied`, `playbook_escalation`, `break_glass`.
+3. **`gitea_request_mcp_restart`** — accepts attempt JSON and env-authorized
+   break-glass; never restarts a process.
+
+## Metrics
+
+`recovery_playbook.recovery_metrics(attempts)` reports the fraction of
+successful recoveries that avoided full/host restart
+(`fraction_avoided_full_restart`).
+
+## Non-goals
+
+* HA multi-instance execution (#668 design only here).
+* Normalizing `pkill` (#630 contamination stays forbidden).
+* Silent mutation of leases or processes from the playbook itself.
+
+## Manual process kills
+
+Remain forbidden and contaminating (#630). The playbook never recommends them.
@@ -95,12 +95,18 @@ gitea_request_mcp_restart(remote, host, org, repo,
                          target_session_id=None, target_role=None,
                          target_connector=None,
                          drain_proof_json=None,
-                          request_break_glass=False)
+                          request_break_glass=False,
+                          prior_recovery_attempts_json=None)
 ```

 It **never restarts anything**: `apply_supported` is always `false` and
 `restart_performed` is always `false`.

+`prior_recovery_attempts_json` (#669) is an optional JSON array of prior
+narrow recovery attempts. Rolling / full / host classes require at least one
+*insufficient* narrower attempt (or authorized break-glass). See
+`docs/mcp-recovery-playbook.md`.
+
 ### Dry-run versus apply

 | Call | Behavior |
@@ -112,9 +118,10 @@ It **never restarts anything**: `apply_supported` is always `false` and

 An apply requires **both** authorizations, and they are independent:

-1. **Restart-class authorization** (#663) — the requester's role and permissions
-   must allow the requested class, the class's approval requirement must be
-   satisfied, and any target-scoped class must name its target. Failing any of
+1. **Restart-class authorization** (#663 / #669) — the requester's role and
+   permissions must allow the requested class, the class's approval requirement
+   must be satisfied, any target-scoped class must name its target, and broad
+   classes must satisfy the recovery-playbook attempt-log gate. Failing any of
   these makes `allow_restart` `false`.
 2. **Drain-proof gate** (#661) — a valid, unexpired, clean proof bound to the
   current impact fingerprint, or an authorized break-glass.
@@ -127,7 +134,10 @@ the authorization that produced it.

 ### Break-glass

-Break-glass bypasses the **drain proof only** — never the restart-class matrix.
+Break-glass bypasses the **drain proof only** — never the restart-class matrix
+(role/permission). Separately, authorized break-glass also satisfies the #669
+attempt-log requirement for broad restarts (rolling/full/host), because that
+gate is not a class-matrix permission check.
 It is honoured solely when `request_break_glass` is set *and* the environment
 carries `GITEA_BREAKGLASS_RESTART_AUTHORIZATION`; like operator override, the
 tool argument expresses caller intent and cannot be self-asserted by a worker
@@ -0,0 +1,64 @@
+# Sanctioned Recovery Playbooks & Controls (Phase 2 #644)
+
+## Overview
+
+Stale runtimes, worktree binding mismatches, and un-reconciled merged branches previously required expert manual shell recovery. Manual process kills (`pkill -f mcp_server.py`) are strictly forbidden and classified as runtime contamination ([#630](sanctioned-restart-controls.md)).
+
+Phase 2 introduces **sanctioned recovery playbooks and controls** into the Web Console:
+- **Diagnose**: Surface stale runtimes, worktree binding errors, contamination markers, and worktree anomalies via health & inventory APIs.
+- **Preview**: Render mutation ledgers and exact confirmation phrases for recovery playbooks.
+- **Confirm & Apply**: Execute sanctioned recovery actions through gated, audited paths.
+- **Verify**: Revalidate control-plane state post-recovery before claiming clean status.
+
+---
+
+## Recovery Playbook Taxonomy
+
+| Playbook ID | Action ID | Minimum Role | Target / Scope | Description |
+|---|---|---|---|---|
+| `clear_stale_binding` | `system.clear_stale_binding` | Operator | Active worktree binding | Clear provably missing or superseded `GITEA_ACTIVE_WORKTREE` binding ([#702](../stale_binding_recovery.py)). |
+| `rebind_session_worktree` | `system.rebind_session_worktree` | Operator | Session worktree | Rebind or synchronize session worktree to verified lease worktree ([#864](../dirty_same_claimant_session_rebind.py)). |
+| `reconcile_cleanups` | `system.reconcile_cleanups` | Controller | Worktree hygiene | Execute reconciler cleanup preview and apply for merged/superseded PR branches. |
+| `sanctioned_restart` | `system.restart_namespace` | Admin | MCP Namespace | Restart MCP daemon gracefully via host supervisor ([#642](sanctioned-restart-controls.md)). |
+
+---
+
+## Wizard Workflow (Diagnose &rarr; Preview &rarr; Confirm &rarr; Verify)
+
+### 1. Diagnose (`GET /api/v1/system/recovery/diagnose`)
+Runs control-plane diagnostics:
+- **Stale Runtime**: Mismatch between running daemon HEAD, local checkout HEAD, and remote-tracking HEAD.
+- **Worktree Binding**: Missing path (`provably_stale_missing_path`), unverified inherited binding (`unverified_inherited`), or superseded binding (`superseded_by_session_lease`).
+- **Contamination**: Checks for live contamination markers from unmanaged process kills.
+- **Worktree Anomalies**: Scans `branches/` directory for un-reconciled cleanups or missing preserved worktrees.
+
+Returns `RecoveryDiagnosis` with eligible playbooks.
+
+### 2. Preview (`POST /api/v1/system/recovery/preview`)
+Takes `playbook_id` and optional `target`/`params`.
+Returns:
+- **Mutation Ledger**: Step-by-step sequence of actions.
+- **Confirmation Phrase**: Exact phrase required to authorize execution (e.g., `confirm clear_stale_binding`).
+- **Authorization Decision**: RBAC check against the operator's principal.
+
+### 3. Apply (`POST /api/v1/system/recovery/apply`)
+Requires `playbook_id` and matching `confirmation` phrase. Gates run in this order, and each fails closed before anything is mutated:
+
+1. **RBAC and execution phase** (`console_authz.authorize(..., for_execution=True)`). The phase branch only applies when `for_execution` is set. While `ACTIVE_PHASE` is `1`, every phase-2 recovery action is refused with `phase_not_active`, so no recovery playbook writes yet. Preview reports the same decision under `execution_authorization` / `execution_blocked_reason`.
+2. **Confirmation phrase** (`confirmation_matches`).
+3. **Contamination rules** ([#630](sanctioned-restart-controls.md)): the live marker is read from the session inventory and assessed under the gated task key `console_recovery_apply`. A contaminated runtime must be cleared through the reconciler cleanup playbook, which is the one playbook exempted from this gate because it is the designated remedy. The marker is also forwarded to `sanctioned_restart.execute_restart`, so a restart cannot launder a contaminated runtime.
+
+Apply then executes the sanctioned recovery logic against the **live** process environment — not a copy — and records an audit entry in `console_audit`. A playbook that leaves the binding unchanged reports `performed: false`; `binding_before`, `binding_after`, and `binding_changed` are returned so a no-op cannot read as success.
+
+Apply does **not** enforce master parity. Parity is reported by Diagnose ([#610](../master_parity_gate.py)) as evidence for the operator; it is not a precondition of this endpoint.
+
+### 4. Verify (`POST /api/v1/system/recovery/verify`)
+Re-evaluates control-plane diagnostics post-recovery and **reports** `clean`, `stale_runtime_clean`, `binding_clean`, `binding_classification`, and `contamination_clean`. It reports; it does not assert or block. State is read fresh rather than from the mapping a mutation just wrote. An `unverified_inherited` binding is reported as not clean, because unproven is not clean.
+
+---
+
+## Safety & Governance Principles
+
+1. **No Manual `pkill`**: Direct process killing remains forbidden and is recorded as contamination.
+2. **Auditability**: Every recovery preview and execution is logged in the console audit trail.
+3. **Master Parity & Dual Control**: High-privilege recovery actions require controller/admin roles and explicit confirmation phrases.
@@ -94,6 +94,9 @@ already define, and a regression test asserts each mapping matches.
 | `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 |
 | `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 |
 | `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 |
+| `system.clear_stale_binding` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
+| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
+| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
 | `initiate_workflow` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |

 **Dual control** means the acting principal may not be the sole authority: a