63 lines
4.1 KiB
Markdown
63 lines
4.1 KiB
Markdown
# Sanctioned Recovery Playbooks & Controls (Phase 2 #644)
|
|
|
|
## Overview
|
|
|
|
Stale runtimes, worktree binding mismatches, and un-reconciled merged branches previously required expert manual shell recovery. Manual process kills (`pkill -f mcp_server.py`) are strictly forbidden and classified as runtime contamination ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)).
|
|
|
|
Phase 2 introduces **sanctioned recovery playbooks and controls** into the Web Console:
|
|
- **Diagnose**: Surface stale runtimes, worktree binding errors, contamination markers, and worktree anomalies via health & inventory APIs.
|
|
- **Preview**: Render mutation ledgers and exact confirmation phrases for recovery playbooks.
|
|
- **Confirm & Apply**: Execute sanctioned recovery actions through gated, audited paths.
|
|
- **Verify**: Revalidate control-plane state post-recovery before claiming clean status.
|
|
|
|
---
|
|
|
|
## Recovery Playbook Taxonomy
|
|
|
|
| Playbook ID | Action ID | Minimum Role | Target / Scope | Description |
|
|
|---|---|---|---|---|
|
|
| `clear_stale_binding` | `system.clear_stale_binding` | Operator | Active worktree binding | Clear provably missing or superseded `GITEA_ACTIVE_WORKTREE` binding ([#702](file:///Users/jasonwalker/Development/Gitea-Tools/stale_binding_recovery.py)). |
|
|
| `rebind_session_worktree` | `system.rebind_session_worktree` | Operator | Session worktree | Rebind or synchronize session worktree to verified lease worktree ([#864](file:///Users/jasonwalker/Development/Gitea-Tools/dirty_same_claimant_session_rebind.py)). |
|
|
| `reconcile_cleanups` | `system.reconcile_cleanups` | Controller | Worktree hygiene | Execute reconciler cleanup preview and apply for merged/superseded PR branches. |
|
|
| `sanctioned_restart` | `system.restart_namespace` | Admin | MCP Namespace | Restart MCP daemon gracefully via host supervisor ([#642](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)). |
|
|
|
|
---
|
|
|
|
## Wizard Workflow (Diagnose → Preview → Confirm → Verify)
|
|
|
|
### 1. Diagnose (`GET /api/v1/system/recovery/diagnose`)
|
|
Runs control-plane diagnostics:
|
|
- **Stale Runtime**: Mismatch between running daemon HEAD, local checkout HEAD, and remote-tracking HEAD.
|
|
- **Worktree Binding**: Missing path (`provably_stale_missing_path`), unverified inherited binding (`unverified_inherited`), or superseded binding (`superseded_by_session_lease`).
|
|
- **Contamination**: Checks for live contamination markers from unmanaged process kills.
|
|
- **Worktree Anomalies**: Scans `branches/` directory for un-reconciled cleanups or missing preserved worktrees.
|
|
|
|
Returns `RecoveryDiagnosis` with eligible playbooks.
|
|
|
|
### 2. Preview (`POST /api/v1/system/recovery/preview`)
|
|
Takes `playbook_id` and optional `target`/`params`.
|
|
Returns:
|
|
- **Mutation Ledger**: Step-by-step sequence of actions.
|
|
- **Confirmation Phrase**: Exact phrase required to authorize execution (e.g., `confirm clear_stale_binding`).
|
|
- **Authorization Decision**: RBAC check against the operator's principal.
|
|
|
|
### 3. Apply (`POST /api/v1/system/recovery/apply`)
|
|
Requires `playbook_id` and matching `confirmation` phrase.
|
|
- Validates RBAC permissions (`console_authz`).
|
|
- Verifies confirmation phrase (`confirmation_matches`).
|
|
- Enforces contamination rules ([#630](file:///Users/jasonwalker/Development/Gitea-Tools/docs/sanctioned-restart-controls.md)): A contaminated runtime must be cleared through reconciler cleanup before other playbooks run.
|
|
- Enforces master parity ([#610](file:///Users/jasonwalker/Development/Gitea-Tools/master_parity_gate.py)).
|
|
- Applies sanctioned recovery logic.
|
|
- Logs audit record in `console_audit`.
|
|
|
|
### 4. Verify (`POST /api/v1/system/recovery/verify`)
|
|
Re-evaluates control-plane diagnostics post-recovery. Asserts `clean: true` before transitioning out of recovery mode.
|
|
|
|
---
|
|
|
|
## Safety & Governance Principles
|
|
|
|
1. **No Manual `pkill`**: Direct process killing remains forbidden and is recorded as contamination.
|
|
2. **Auditability**: Every recovery preview and execution is logged in the console audit trail.
|
|
3. **Master Parity & Dual Control**: High-privilege recovery actions require controller/admin roles and explicit confirmation phrases.
|