Merge remote-tracking branch 'prgs/master' into feat/issue-659-maintenance-drain-mode

docs/architecture/mcp-ha-rolling-restart.md

@@ -0,0 +1,167 @@
+# ADR: High-availability and rolling-restart architecture for Gitea MCP control plane
+
+- **Status:** Proposed (Design ADR under [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668))
+- **Date:** 2026-07-25
+- **Tracking Issue:** [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668)
+- **Policy Version:** `mcp-ha-rolling-restart/v1`
+- **Related:**
+  - Parent: [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655) — Governed MCP restart coordination and zero-disruption recovery
+  - Governance Policy: [#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656) / `docs/architecture/mcp-restart-governance.md`
+  - Control-Plane DB Substrate: [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) / `docs/architecture/control-plane-db-substrate.md`
+  - Runtime Policy: [#615](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/615) / `docs/architecture/mcp-stable-control-runtime-policy-adr.md`
+  - Product Vision: [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) (Phase 5 Maturity)
+  - Delivery Roadmap: [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653)
+
+---
+
+## 1. Context & Problem Statement
+
+The Gitea MCP server operates as the authoritative **control plane** for managing issues, Pull Requests, code mutations, formal reviews, and workflow reconciliations. Under single-process governance ([#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656)), process restarts are strictly controlled using pre-flight checks, drain phases, and operator approvals.
+
+However, a single-instance control plane inherently presents fundamental constraints:
+
+1. **Downtime during updates:** Even a perfectly executed single-process drain requires a window where incoming client requests must be paused or rejected while the server binary or python environment reloads.
+2. **Single point of failure:** Infrastructure issues, process crashes, or unhandled host-level terminations immediately disconnect active LLM sessions and leave transient workflows incomplete.
+3. **Multi-agent concurrency bottlenecks:** High volumes of concurrent multi-LLM tasks put all lock management, lease allocation, and Gitea API interactions through a single process event loop.
+
+To achieve true zero-disruption operation and seamless rolling deployments without stopping active work, the system requires a high-availability (HA), multi-instance MCP architecture.
+
+---
+
+## 2. Architectural Principles & Non-Goals
+
+### 2.1 Core Architectural Principles
+* **Gitea as Canonical Work SoT:** Gitea remains the ultimate System of Record (SoT) for issue states, pull requests, labels, and audit comments. The MCP control plane does not duplicate domain entities.
+* **Control-Plane DB as Multi-Instance State Substrate:** The control-plane SQLite/durable database ([#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613)) acts as the single source of truth for workflow leases, session tokens, assignment records, and lock fences across all MCP nodes.
+* **Stateless Worker Nodes:** MCP role server processes (`gitea-author`, `gitea-reviewer`, `gitea-merger`, `gitea-reconciler`, `gitea-controller`) maintain no unique in-memory state; any node can handle any request given a valid session resume token.
+* **Fail-Closed Split-Brain Defense:** In any network partition or quorum loss scenario, nodes must fail closed rather than risk double-mutations or conflicting Gitea states.
+
+### 2.2 Non-Goals
+* **Replacing Gitea:** We do not replace Gitea issue/PR tracking with an independent database.
+* **Immediate Multi-Node Cluster Execution in v1:** This ADR defines the target architecture and phased roadmap; immediate implementation occurs incrementally post-[#655] v1.
+
+---
+
+## 3. High-Availability & Rolling-Restart Architecture
+
+### 3.1 Architecture Overview
+
+```
+                        +----------------------------+
+                        | LLM Clients / IDE Sessions |
+                        +--------------+-------------+
+                                       |
+                                       v
+                        +----------------------------+
+                        |    HA Proxy / Router       |
+                        | (Health-based & Affinity)  |
+                        +------+--------------+------+
+                               |              |
+                +--------------+              +--------------+
+                v                                            v
+     +--------------------+                        +--------------------+
+     | MCP Instance Node A|                        | MCP Instance Node B|
+     | (Version N)        |                        | (Version N+1)      |
+     +---------+----------+                        +---------+----------+
+               |                                             |
+               +----------------------+----------------------+
+                                      |
+                                      v
+                        +----------------------------+
+                        |  Control-Plane DB Substrate|
+                        |  (Shared Lease & Locks)    |
+                        +--------------+-------------+
+                                       |
+                                       v
+                        +----------------------------+
+                        |        Gitea API           |
+                        +----------------------------+
+```
+
+---
+
+### 3.2 Key System Components
+
+#### A. Multiple MCP Instance Cohorts
+* The control plane runs across $N \ge 2$ redundant process nodes.
+* Dual-namespace deployment allows running the old version (Node A) alongside a updated version (Node B) during rolling upgrades.
+
+#### B. Shared Durable Session Storage & Resume Tokens
+* Session context, preflight verification proofs, and capability resolution states are stored in the shared control-plane database.
+* Client requests carry an explicit `session_id` and `resume_token`. If an MCP instance restarts or a request routes to a different instance, the target node validates the token against the database without requiring full session re-initialization.
+
+#### C. Shared Lease Authority & Fencing Counters
+* Workflow leases (`gitea_allocate_next_work`, `gitea_adopt_workflow_lease`) use monotonic fencing tokens (`lease_generation_id`).
+* When Node B acquires or renews a lease, it increments the generation counter. Any delayed or out-of-order write attempt from Node A using an older generation token is rejected by database constraints.
+
+#### D. Leader Election & Coordinated Drain
+* Node clusters elect a primary coordinator node for administrative background tasks (such as stale lease cleanup or incident Watchdogs).
+* During a rolling deployment:
+  1. Node B (new version) is launched and registers as healthy.
+  2. Router directs new session creations to Node B.
+  3. Node A enters `MAINTENANCE_DRAIN` status ([#659](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/659)), completing in-flight mutations while refusing new tasks.
+  4. Once all active sessions migrate or complete, Node A shuts down cleanly.
+
+#### E. Idempotent Mutations & Failover Safety
+* All state-changing tool executions (PR creation, review submission, merge operations, label changes) carry a deterministic `idempotency_key`.
+* If a network connection flaps or a node fails mid-mutation, the re-issued request with the same `idempotency_key` is recognized by the control-plane substrate, returning the existing recorded result without repeating side effects on Gitea.
+
+#### F. Schema Version Compatibility
+* Database migrations follow non-breaking additive patterns.
+* During rolling upgrades where Node A (Version $N$) and Node B (Version $N+1$) run concurrently, both versions operate against the shared schema without structural conflicts.
+
+---
+
+## 4. Split-Brain & Failure Behavior
+
+### 4.1 Split-Brain Risk Scenarios & Mitigation
+
+| Scenario | Risk | Mitigation Strategy |
+|---|---|---|
+| **Network Partition between Nodes** | Both Node A and Node B attempt to process operations for the same issue/PR. | **Generation Fencing:** Lease renewal requires updating the DB generation counter. The node isolated from the DB fails closed immediately. |
+| **Stale Node Recovery** | Node A recovers after a long pause and executes a queued mutation. | **Lease Expiry & TTL Fencing:** Transactions verify that `expires_at > NOW()` within the atomic SQLite transaction boundaries. |
+| **Database Connection Loss** | Node loses access to shared control-plane DB substrate. | **Strict Fail-Closed:** The node immediately marks all task capabilities as `blocked` and rejects mutation tools until DB connectivity is re-established. |
+
+---
+
+## 5. Phased Implementation Milestones
+
+```mermaid
+flowchart TD
+    M1[Milestone 1: Shared Control-Plane DB Schema & Resume Tokens] --> M2[Milestone 2: Idempotent Mutation Layer]
+    M2 --> M3[Milestone 3: Health Routing & Standby Failover]
+    M3 --> M4[Milestone 4: Active-Active Rolling Deployment & Auto-Drain]
+```
+
+### Milestone 1: Shared Control-Plane DB Schema & Resume Tokens (Post-#655)
+* Extend [#613] Control-Plane DB schema to store multi-instance node heartbeat records and session resume tokens.
+* Enable session lookup across instances via `session_id`.
+
+### Milestone 2: Idempotent Mutation Layer & Lease Fencing
+* Add mandatory `idempotency_key` tracking to all Gitea mutation tools.
+* Implement monotonic lease fencing counters in `gitea_allocate_next_work` and `gitea_adopt_workflow_lease`.
+
+### Milestone 3: Health-Based Routing & Active-Passive Standby
+* Introduce lightweight proxy/router capable of checking node health endpoints.
+* Implement active-standby failover where standby node automatically assumes work if active node fails health checks.
+
+### Milestone 4: Active-Active Horizontal Deployment & Rolling Upgrade Automation
+* Enable true active-active multi-instance execution.
+* Integrate automated zero-downtime rolling upgrades coordinated with `gitea_request_mcp_restart` maintenance drain.
+
+---
+
+## 6. Observability & Audit Requirements
+
+High-availability control plane operations must expose clear telemetry and audit trails:
+
+* **Node Registry Telemetry:** Active nodes, version numbers, uptime, and heartbeat timestamps reported via `gitea_get_runtime_context`.
+* **Lease Fencing Metrics:** Tracking lease acquire latency, fence rejection counts, and lease handoff durations.
+* **Failover & Re-route Audit Logs:** Durable logging of session migrations between nodes, drain initiation, and process retirement events.
+
+---
+
+## 7. Tradeoffs & Accepted Risks
+
+* **Increased Architectural Complexity:** Moving from a single process to a multi-instance control plane requires robust DB locking, proxy routing, and migration governance.
+* **Database Dependency:** The control-plane database substrate becomes a critical shared dependency for multi-node deployments. High availability for the underlying SQLite file system / DB must be guaranteed.

docs/incidents/670-direct-master-commit-2fa97c26.md

@@ -0,0 +1,83 @@
+# Incident #670: bare direct-to-master commit `2fa97c26` (retroactive audit)
+
+Status: verified; disposition recommendation: **accept as-is, no revert** (final
+disposition owned by controller per issue #670).
+
+## Summary
+
+Commit `2fa97c26fbda555a1a83930ca5fdcea9d8e47b50`
+(`fix(mcp): load dotenv relative to project root`) landed on `prgs/master`
+as a single-parent commit with no PR wrapper and no review record, bypassing
+the sanctioned issue → branch → PR → review → merge workflow. It was
+discovered during the PR #654 post-merge audit. PR #654 itself merged
+cleanly via the Gitea API and did **not** introduce this commit.
+
+## Verification evidence (acceptance criteria 1–3)
+
+- **AC1 — present on `prgs/master`: yes.**
+  `git merge-base --is-ancestor 2fa97c26fbda555a1a83930ca5fdcea9d8e47b50 prgs/master` → true.
+- **AC2 — no PR or review record: confirmed.**
+  The commit is a single-parent, non-merge commit sitting directly on
+  first-parent master between the #629 merge (`5ab5fe85`) and the #654
+  merge (`ec903b0d`). A PR landing on master produces a merge commit (or a
+  PR-linked head); neither exists here. The controller audit at issue-create
+  time also found no PR wrapper and no review record for this SHA.
+- **AC3 — changed files and diff summary: confirmed.**
+  `gitea_auth.py | 5 +++--` (+3/−2). Single parent
+  `5ab5fe8583c07134d55dadf09381aecb67df246e`. The change moves
+  `PROJECT_ROOT` derivation above `load_dotenv()` and loads
+  `.env` relative to the project root instead of the process CWD.
+
+## AC4 — why no immediate revert
+
+- The dotenv fix is intentional and required for correct runtime behavior:
+  without it, `load_dotenv()` resolves `.env` against the process working
+  directory, which breaks MCP server launches whose CWD is not the project
+  root.
+- The change is small (+3/−2), self-contained in `gitea_auth.py`, and has
+  been running on master without incident since 2026-07-10.
+- Reverting would re-introduce a real bug to remove a provenance defect —
+  the wrong trade. Provenance is repaired retroactively by this document,
+  issue #670, and the hardening landed under #671.
+- If the controller later judges the change unsafe, a separate
+  revert/repair issue is the sanctioned path (issue #670, recommended
+  disposition option 4).
+
+## AC5 — workflow-hardening linkage
+
+Prevention already landed: **issue #671** (closed)
+*“Block direct pushes to stable branches from MCP workflow sessions”*,
+implemented by commit `5933d87647656643a67a50331c4c7b06ea751dad`
+(`feat(guard): block direct stable-branch pushes from MCP workflow sessions`).
+
+Shipped guardrails include:
+
+- `gitea_record_stable_branch_push_attempt` — classifies proposed commands
+  for direct stable-branch push intent (`git push <remote> master`,
+  refspecs, `HEAD:master`, `--force`, dry-run intent, `:master` delete),
+  plus root/control-checkout local commits not carried by an issue branch,
+  and writes a durable `stable_branch_contamination` marker.
+- `gitea_audit_stable_branch_contamination` — reconciler-only audit/clear
+  path; a contaminated worker session cannot self-clear.
+- Review/merge/close/completion mutations fail closed while a
+  contamination marker is active.
+
+## AC6 — PR #654 was not the source
+
+- `2fa97c26` is the **first parent** of the #654 merge commit
+  `ec903b0d619e7a27d24aed272a890f4e5d381411`; it predates the #654 merge.
+- First-parent history `5ab5fe8..ec903b0`:
+  `2fa97c2 fix(mcp): load dotenv relative to project root` followed by
+  `ec903b0 Merge pull request 'feat: lifecycle role/hazard labels ... (#603)' (#654)`.
+- The #654 merger audit confirmed `ec903b0d` was a valid Gitea-API merge,
+  the `git push prgs master` attempt during that run was a no-op, and the
+  net change `2fa97c2..ec903b0` contained only the reviewed #603
+  lifecycle-label files.
+- Conclusion: #654 merged reviewed content only; the unauthorized-path
+  defect is solely the earlier bare commit `2fa97c26`.
+
+## Explicit non-actions (unchanged by this audit)
+
+- No revert of `2fa97c26`.
+- No force-push or history rewrite.
+- No master mutation from the audit session.

docs/mcp-recovery-playbook.md

@@ -0,0 +1,94 @@
+# MCP scoped recovery playbook (#669)
+
+**Parent:** [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655)  
+**Vision / roadmap:** [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) · [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653)  
+**Class matrix:** [#663](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/663) · `docs/mcp-restart-classes.md`  
+**Coordinator:** [#658](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/658) · `restart_coordinator.py`  
+**Audit lineage:** [#665](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/665)
+
+## Decision
+
+Full-server MCP reset is a **last resort**. Prefer the narrowest recovery that
+can clear the symptom. The coordinator **refuses** `rolling_mcp_restart`,
+`full_mcp_restart`, and `host_restart` unless:
+
+1. The inventory carries a prior **attempt log** of at least one *insufficient*
+   narrower recovery, **or**
+2. **Break-glass** is authorized
+   (`request_break_glass` + `GITEA_BREAKGLASS_RESTART_AUTHORIZATION`).
+
+Break-glass still never bypasses the #663 class matrix (role/permission).
+
+## Ladder (narrow → broad)
+
+| Rank | Action | Self-service | Implementation / delegation |
+|---:|---|---|---|
+| 0 | `client_reconnect` | yes | Host auto-reconnect / client reconnect · [#584](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/584) · `docs/mcp-namespace-eof-recovery.md` |
+| 1 | `capability_refresh` | yes | `gitea_resolve_task_capability` + `gitea_whoami` · [#610](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/610) · [#685](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/685) |
+| 2 | `session_reconnect` | yes | Runtime rebind + explicit `worktree_path` · [#543](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/543) · [#618](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/618) |
+| 3 | `configuration_reload` | no | Class `configuration_reload` · console reload · [#642](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/642) |
+| 4 | `lease_recovery` | no | Lock/lease recovery paths · [#702](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/702) · [#753](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/753) · [#790](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/790) |
+| 5 | `worker_restart` | no | Class `worker_restart` · #663 |
+| 6 | `role_runtime_restart` | no | Class `role_runtime_restart` · console restart · #642/#663 |
+| 7 | `connector_restart` | no | Class `connector_restart` · #663 |
+| 8 | `rolling_mcp_restart` | no | Class `rolling_mcp_restart` · design [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668) · **attempt log required** |
+| 9 | `full_mcp_restart` | no | Class `full_mcp_restart` · **attempt log required** |
+| 10 | `host_restart` | no | Class `host_restart` · **attempt log required** |
+
+Machine-readable source of truth: `recovery_playbook.RECOVERY_LADDER` and
+`recovery_playbook.ladder_document()`.
+
+## Attempt log shape
+
+Each prior attempt is a mapping:
+
+```json
+{
+  "action": "client_reconnect",
+  "outcome": "insufficient",
+  "reason": "transport still closed after IDE reconnect",
+  "actor": "prgs-controller-12345",
+  "recorded_at": "2026-07-25T21:00:00+00:00"
+}
+```
+
+Outcomes that count toward escalation: `failed`, `insufficient`, `denied`,
+`unresolved`, `timeout`, `error`.
+
+Pass attempts into the coordinator via inventory
+`prior_recovery_attempts` or the MCP tool argument
+`prior_recovery_attempts_json` on `gitea_request_mcp_restart`.
+
+Helper: `recovery_playbook.build_attempt_record(...)`.
+
+## Symptom → first rung
+
+`recovery_playbook.recommend_actions(symptoms=[...])` maps symptoms such as
+`transport_eof`, `stale_capability`, `stale_lease`, `daemon_corrupt` to the
+narrowest recommended action, then walks the ladder. Soft recommendations
+never replace the hard gate on broad restarts.
+
+## Enforcement points
+
+1. **`recovery_playbook.assess_escalation`** — pure gate.
+2. **`restart_coordinator.evaluate_restart_impact`** — when `restart_class` is
+   set (policy-enforced path), broad classes require the gate; report fields
+   `attempt_log_satisfied`, `playbook_escalation`, `break_glass`.
+3. **`gitea_request_mcp_restart`** — accepts attempt JSON and env-authorized
+   break-glass; never restarts a process.
+
+## Metrics
+
+`recovery_playbook.recovery_metrics(attempts)` reports the fraction of
+successful recoveries that avoided full/host restart
+(`fraction_avoided_full_restart`).
+
+## Non-goals
+
+* HA multi-instance execution (#668 design only here).
+* Normalizing `pkill` (#630 contamination stays forbidden).
+* Silent mutation of leases or processes from the playbook itself.
+
+## Manual process kills
+
+Remain forbidden and contaminating (#630). The playbook never recommends them.

docs/mcp-restart-coordinator.md

@@ -95,12 +95,18 @@ gitea_request_mcp_restart(remote, host, org, repo,
                           target_session_id=None, target_role=None,
                           target_connector=None,
                           drain_proof_json=None,
-                          request_break_glass=False)
+                          request_break_glass=False,
+                          prior_recovery_attempts_json=None)
 ```
 
 It **never restarts anything**: `apply_supported` is always `false` and
 `restart_performed` is always `false`.
 
+`prior_recovery_attempts_json` (#669) is an optional JSON array of prior
+narrow recovery attempts. Rolling / full / host classes require at least one
+*insufficient* narrower attempt (or authorized break-glass). See
+`docs/mcp-recovery-playbook.md`.
+
 ### Dry-run versus apply
 
 | Call | Behavior |
@@ -112,9 +118,10 @@ It **never restarts anything**: `apply_supported` is always `false` and
 
 An apply requires **both** authorizations, and they are independent:
 
-1. **Restart-class authorization** (#663) — the requester's role and permissions
-   must allow the requested class, the class's approval requirement must be
-   satisfied, and any target-scoped class must name its target. Failing any of
+1. **Restart-class authorization** (#663 / #669) — the requester's role and
+   permissions must allow the requested class, the class's approval requirement
+   must be satisfied, any target-scoped class must name its target, and broad
+   classes must satisfy the recovery-playbook attempt-log gate. Failing any of
    these makes `allow_restart` `false`.
 2. **Drain-proof gate** (#661) — a valid, unexpired, clean proof bound to the
    current impact fingerprint, or an authorized break-glass.
@@ -127,7 +134,10 @@ the authorization that produced it.
 
 ### Break-glass
 
-Break-glass bypasses the **drain proof only** — never the restart-class matrix.
+Break-glass bypasses the **drain proof only** — never the restart-class matrix
+(role/permission). Separately, authorized break-glass also satisfies the #669
+attempt-log requirement for broad restarts (rolling/full/host), because that
+gate is not a class-matrix permission check.
 It is honoured solely when `request_break_glass` is set *and* the environment
 carries `GITEA_BREAKGLASS_RESTART_AUTHORIZATION`; like operator override, the
 tool argument expresses caller intent and cannot be self-asserted by a worker

docs/sanctioned-recovery-playbooks.md

@@ -0,0 +1,64 @@
+# Sanctioned Recovery Playbooks & Controls (Phase 2 #644)
+
+## Overview
+
+Stale runtimes, worktree binding mismatches, and un-reconciled merged branches previously required expert manual shell recovery. Manual process kills (`pkill -f mcp_server.py`) are strictly forbidden and classified as runtime contamination ([#630](sanctioned-restart-controls.md)).
+
+Phase 2 introduces **sanctioned recovery playbooks and controls** into the Web Console:
+- **Diagnose**: Surface stale runtimes, worktree binding errors, contamination markers, and worktree anomalies via health & inventory APIs.
+- **Preview**: Render mutation ledgers and exact confirmation phrases for recovery playbooks.
+- **Confirm & Apply**: Execute sanctioned recovery actions through gated, audited paths.
+- **Verify**: Revalidate control-plane state post-recovery before claiming clean status.
+
+---
+
+## Recovery Playbook Taxonomy
+
+| Playbook ID | Action ID | Minimum Role | Target / Scope | Description |
+|---|---|---|---|---|
+| `clear_stale_binding` | `system.clear_stale_binding` | Operator | Active worktree binding | Clear provably missing or superseded `GITEA_ACTIVE_WORKTREE` binding ([#702](../stale_binding_recovery.py)). |
+| `rebind_session_worktree` | `system.rebind_session_worktree` | Operator | Session worktree | Rebind or synchronize session worktree to verified lease worktree ([#864](../dirty_same_claimant_session_rebind.py)). |
+| `reconcile_cleanups` | `system.reconcile_cleanups` | Controller | Worktree hygiene | Execute reconciler cleanup preview and apply for merged/superseded PR branches. |
+| `sanctioned_restart` | `system.restart_namespace` | Admin | MCP Namespace | Restart MCP daemon gracefully via host supervisor ([#642](sanctioned-restart-controls.md)). |
+
+---
+
+## Wizard Workflow (Diagnose → Preview → Confirm → Verify)
+
+### 1. Diagnose (`GET /api/v1/system/recovery/diagnose`)
+Runs control-plane diagnostics:
+- **Stale Runtime**: Mismatch between running daemon HEAD, local checkout HEAD, and remote-tracking HEAD.
+- **Worktree Binding**: Missing path (`provably_stale_missing_path`), unverified inherited binding (`unverified_inherited`), or superseded binding (`superseded_by_session_lease`).
+- **Contamination**: Checks for live contamination markers from unmanaged process kills.
+- **Worktree Anomalies**: Scans `branches/` directory for un-reconciled cleanups or missing preserved worktrees.
+
+Returns `RecoveryDiagnosis` with eligible playbooks.
+
+### 2. Preview (`POST /api/v1/system/recovery/preview`)
+Takes `playbook_id` and optional `target`/`params`.
+Returns:
+- **Mutation Ledger**: Step-by-step sequence of actions.
+- **Confirmation Phrase**: Exact phrase required to authorize execution (e.g., `confirm clear_stale_binding`).
+- **Authorization Decision**: RBAC check against the operator's principal.
+
+### 3. Apply (`POST /api/v1/system/recovery/apply`)
+Requires `playbook_id` and matching `confirmation` phrase. Gates run in this order, and each fails closed before anything is mutated:
+
+1. **RBAC and execution phase** (`console_authz.authorize(..., for_execution=True)`). The phase branch only applies when `for_execution` is set. While `ACTIVE_PHASE` is `1`, every phase-2 recovery action is refused with `phase_not_active`, so no recovery playbook writes yet. Preview reports the same decision under `execution_authorization` / `execution_blocked_reason`.
+2. **Confirmation phrase** (`confirmation_matches`).
+3. **Contamination rules** ([#630](sanctioned-restart-controls.md)): the live marker is read from the session inventory and assessed under the gated task key `console_recovery_apply`. A contaminated runtime must be cleared through the reconciler cleanup playbook, which is the one playbook exempted from this gate because it is the designated remedy. The marker is also forwarded to `sanctioned_restart.execute_restart`, so a restart cannot launder a contaminated runtime.
+
+Apply then executes the sanctioned recovery logic against the **live** process environment — not a copy — and records an audit entry in `console_audit`. A playbook that leaves the binding unchanged reports `performed: false`; `binding_before`, `binding_after`, and `binding_changed` are returned so a no-op cannot read as success.
+
+Apply does **not** enforce master parity. Parity is reported by Diagnose ([#610](../master_parity_gate.py)) as evidence for the operator; it is not a precondition of this endpoint.
+
+### 4. Verify (`POST /api/v1/system/recovery/verify`)
+Re-evaluates control-plane diagnostics post-recovery and **reports** `clean`, `stale_runtime_clean`, `binding_clean`, `binding_classification`, and `contamination_clean`. It reports; it does not assert or block. State is read fresh rather than from the mapping a mutation just wrote. An `unverified_inherited` binding is reported as not clean, because unproven is not clean.
+
+---
+
+## Safety & Governance Principles
+
+1. **No Manual `pkill`**: Direct process killing remains forbidden and is recorded as contamination.
+2. **Auditability**: Every recovery preview and execution is logged in the console audit trail.
+3. **Master Parity & Dual Control**: High-privilege recovery actions require controller/admin roles and explicit confirmation phrases.

docs/webui-authz-audit.md

@@ -94,6 +94,9 @@ already define, and a regression test asserts each mapping matches.
 | `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 |
 | `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 |
 | `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 |
+| `system.clear_stale_binding` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
+| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
+| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
 | `initiate_workflow` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
 
 **Dual control** means the acting principal may not be the sole authority: a

docs/webui-notifications.md

@@ -0,0 +1,81 @@
+# Web Console: Notifications & Human-Attention Routing (#648)
+
+- **Status:** Phase 3 Live
+- **Tracking Issue:** [#648](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/648)
+- **Parent Epic:** [#631](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/631)
+- **Attention Boundary Reference:** [#628](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/628)
+
+---
+
+## 1. Overview
+
+The **Notifications & Human-Attention Console** (`/notifications`, `/api/v1/notifications`) provides intelligent event classification and human-attention routing for autonomous workflow operations.
+
+To prevent alert fatigue while ensuring critical escalation boundaries are never missed, events are classified into three distinct **Attention Classes**:
+
+1. **`human-required`** (Urgent Escalation Boundary):
+   - Items requiring immediate human intervention or business decisions.
+   - Triggers: Auth failures, hard stops, irrecoverable state, decision locks, failed report validations, critical probe errors.
+   - Display: Highlighted in red (`badge-blocked`) with a `HUMAN REQUIRED` badge.
+
+2. **`operator`** (Operational Inbox):
+   - Items requiring controller or operator review/triage during routine execution.
+   - Triggers: Blocked PRs (merge conflicts), stale leases, duplicate PRs on issues, unassigned ready work.
+   - Display: Displayed in orange/yellow (`badge-claimed`).
+
+3. **`routine`** (Background Workflow Transitions):
+   - Normal, healthy workflow transitions and state progressions.
+   - Triggers: Active PRs/issues in standard state, clean branch creation, routine heartbeats.
+   - Display: Filtered out of default inbox views to eliminate notification spam; viewable on demand via the "Routine" or "All" tab.
+
+---
+
+## 2. API Endpoints
+
+### `GET /api/v1/notifications`
+*Compatibility Alias:* `GET /api/notifications`
+
+#### Query Parameters:
+- `project_id` (optional): Filter notifications by project ID.
+- `attention_class` (optional): `inbox` (default: human-required + operator), `human-required`, `operator`, `routine`, `all`.
+
+#### Example JSON Response:
+```json
+{
+  "project_id": "gitea-tools",
+  "repo_label": "Scaled-Tech-Consulting/Gitea-Tools",
+  "human_required_count": 0,
+  "operator_count": 2,
+  "routine_count": 5,
+  "total_count": 7,
+  "fetch_error": null,
+  "inbox_items": [
+    {
+      "id": "notif-pr-block-742",
+      "attention_class": "operator",
+      "category": "blocker",
+      "title": "Blocked PR #742",
+      "summary": "PR #742 requires merge conflict resolution.",
+      "work_kind": "pr",
+      "work_number": 742,
+      "project_id": "gitea-tools",
+      "repo_label": "Scaled-Tech-Consulting/Gitea-Tools",
+      "created_at": "2026-07-25T16:39:47Z",
+      "deep_link": "/traffic",
+      "requires_human": false,
+      "extra": {}
+    }
+  ],
+  "all_items": [...]
+}
+```
+
+---
+
+## 3. UI Navigation
+
+- Access via the **Traffic** navigation menu: **Traffic → Notifications**.
+- The main view displays:
+  - **Metrics Summary Bar**: Highlighting counts for Human Required, Operator Inbox, and Routine items.
+  - **Attention Filter Tabs**: Toggle between Inbox (Human + Operator), Human Required, Operator, Routine, and All.
+  - **Structured Event Table**: Displays category, title, summary, work item links, and timestamps.

docs/webui-restart-console.md

@@ -0,0 +1,102 @@
+# Web Console: restart status, impact preview, and approval state (#667)
+
+Phase 1 of the console restart surface. It consumes the #655 coordinator
+substrate and displays it. It performs no restart, reload, drain, approval, or
+process action, and it registers no write endpoint.
+
+Issue #667's rollout is explicit — *status views first, write approval after the
+backend gates are green* — and this change delivers only the status half.
+
+## Surfaces
+
+| Path | Method | Purpose |
+|------|--------|---------|
+| `/runtime/restart` | GET | Restart status page |
+| `/api/v1/system/restart/status` | GET | Same snapshot as JSON |
+
+Both accept an optional `restart_class` query parameter (default
+`full_mcp_restart`). An unrecognised class is not an error: the coordinator
+resolves it as unknown and fails closed, and the page shows the resulting deny.
+
+Neither path accepts `POST`; a write attempt returns `405`, and a test asserts
+it.
+
+## What it shows
+
+* **Impact preview (#658)** — verdict, blast radius, affected sessions, leases,
+  critical sections, mutations, and the counts behind them, evaluated
+  `dry_run=True` against live control-plane state.
+* **Drain proof (#661)** — verification of a supplied proof: valid, clean,
+  expired, tampered, and the reasons behind a refusal.
+* **Post-restart reconcile (#662)** — the most recent completion proof, its
+  overall status, and which dimensions still require follow-up.
+* **Restart classes (#663)** — the least-privilege matrix, with *you may
+  request* and *you may execute* computed for the viewing role rather than for a
+  generic operator.
+* **Approval controls (#633)** — the authorization state of
+  `system.restart_namespace` and `system.reload_namespace`.
+* **Break-glass (#664)** — declared and marked unavailable; see below.
+
+## Three rules this surface holds itself to
+
+A status page that is wrong is worse than one that is missing, because an
+operator acts on it. Three properties are enforced by tests, and each was
+verified by reverting the guard and watching a test fail.
+
+### An unreadable source reports unavailable, never green
+
+Every source carries its own `SourceStatus`. Nothing substitutes a default,
+placeholder, or self-comparison for a reading that failed. An unreadable
+control-plane database yields `inventory_complete: false`, which the coordinator
+itself turns into a fail-closed verdict, and the page says the blast radius is
+unknown rather than showing an empty affected-sessions table.
+
+An absent drain proof is reported as absent — not as a pass. The #661 gate
+authorizes a restart only against a valid, unexpired, clean proof, so no proof
+is precisely the state that gate denies on.
+
+### Authorization is asked the way execution would ask it
+
+Every probe passes `for_execution=True`.
+
+Asked without it, an admin is `allowed` for `system.restart_namespace`. On a
+control surface that reads as a live button. Asked the way an execution attempt
+would ask, the same principal is refused `phase_not_active`, because the console
+is in Phase 1 and the action is Phase 2. This surface reports the second answer.
+
+`execution_enabled` is therefore `false` for every action and every role today,
+and a test asserts that across the whole role matrix.
+
+### The control-plane database is opened read-only
+
+`ControlPlaneDB()` creates directories and runs migrations on construction — a
+write. This surface never constructs one. It opens the sqlite file with
+`mode=ro`, exactly as `webui/inventory.py` does, and treats a missing file as
+missing authority rather than as an empty inventory.
+
+The test that protects this points at a path inside a directory that already
+exists, so a read-write `connect` would really create the file. A nested
+missing-directory path would have passed for the wrong reason.
+
+## Break-glass is declared, not offered
+
+The break-glass workflow (#664) is not available on this branch's base. The
+panel is rendered to operator-class roles as **unavailable**, naming the issue
+that tracks it. It is not silently omitted, because an operator who has been
+told a governance path exists needs to see that it is not wired here; and it is
+not rendered as a control, because there is nothing behind it.
+
+Unprivileged viewers see only a note that the surface is operator-class.
+
+## Redaction and escaping
+
+Every interpolated value passes through `_esc` (`html.escape(..., quote=True)`).
+Free-form text and anything that can carry a filesystem path additionally passes
+through `webui.inventory.scrub_text`, which redacts credential-shaped tokens
+inside a string rather than only at its start. The impact payload is passed
+through `webui.inventory.scrub` before rendering.
+
+## Linkage
+
+Parent #655 · extends #642 · consumes #658, #661, #662, #663 · RBAC #633 ·
+console #631 · vision #652 · roadmap #653 · break-glass #664.