diff --git a/docs/architecture/mcp-ha-rolling-restart.md b/docs/architecture/mcp-ha-rolling-restart.md new file mode 100644 index 0000000..fc82f99 --- /dev/null +++ b/docs/architecture/mcp-ha-rolling-restart.md @@ -0,0 +1,167 @@ +# ADR: High-availability and rolling-restart architecture for Gitea MCP control plane + +- **Status:** Proposed (Design ADR under [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668)) +- **Date:** 2026-07-25 +- **Tracking Issue:** [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668) +- **Policy Version:** `mcp-ha-rolling-restart/v1` +- **Related:** + - Parent: [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655) — Governed MCP restart coordination and zero-disruption recovery + - Governance Policy: [#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656) / `docs/architecture/mcp-restart-governance.md` + - Control-Plane DB Substrate: [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) / `docs/architecture/control-plane-db-substrate.md` + - Runtime Policy: [#615](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/615) / `docs/architecture/mcp-stable-control-runtime-policy-adr.md` + - Product Vision: [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) (Phase 5 Maturity) + - Delivery Roadmap: [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653) + +--- + +## 1. Context & Problem Statement + +The Gitea MCP server operates as the authoritative **control plane** for managing issues, Pull Requests, code mutations, formal reviews, and workflow reconciliations. Under single-process governance ([#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656)), process restarts are strictly controlled using pre-flight checks, drain phases, and operator approvals. + +However, a single-instance control plane inherently presents fundamental constraints: + +1. **Downtime during updates:** Even a perfectly executed single-process drain requires a window where incoming client requests must be paused or rejected while the server binary or python environment reloads. +2. **Single point of failure:** Infrastructure issues, process crashes, or unhandled host-level terminations immediately disconnect active LLM sessions and leave transient workflows incomplete. +3. **Multi-agent concurrency bottlenecks:** High volumes of concurrent multi-LLM tasks put all lock management, lease allocation, and Gitea API interactions through a single process event loop. + +To achieve true zero-disruption operation and seamless rolling deployments without stopping active work, the system requires a high-availability (HA), multi-instance MCP architecture. + +--- + +## 2. Architectural Principles & Non-Goals + +### 2.1 Core Architectural Principles +* **Gitea as Canonical Work SoT:** Gitea remains the ultimate System of Record (SoT) for issue states, pull requests, labels, and audit comments. The MCP control plane does not duplicate domain entities. +* **Control-Plane DB as Multi-Instance State Substrate:** The control-plane SQLite/durable database ([#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613)) acts as the single source of truth for workflow leases, session tokens, assignment records, and lock fences across all MCP nodes. +* **Stateless Worker Nodes:** MCP role server processes (`gitea-author`, `gitea-reviewer`, `gitea-merger`, `gitea-reconciler`, `gitea-controller`) maintain no unique in-memory state; any node can handle any request given a valid session resume token. +* **Fail-Closed Split-Brain Defense:** In any network partition or quorum loss scenario, nodes must fail closed rather than risk double-mutations or conflicting Gitea states. + +### 2.2 Non-Goals +* **Replacing Gitea:** We do not replace Gitea issue/PR tracking with an independent database. +* **Immediate Multi-Node Cluster Execution in v1:** This ADR defines the target architecture and phased roadmap; immediate implementation occurs incrementally post-[#655] v1. + +--- + +## 3. High-Availability & Rolling-Restart Architecture + +### 3.1 Architecture Overview + +``` + +----------------------------+ + | LLM Clients / IDE Sessions | + +--------------+-------------+ + | + v + +----------------------------+ + | HA Proxy / Router | + | (Health-based & Affinity) | + +------+--------------+------+ + | | + +--------------+ +--------------+ + v v + +--------------------+ +--------------------+ + | MCP Instance Node A| | MCP Instance Node B| + | (Version N) | | (Version N+1) | + +---------+----------+ +---------+----------+ + | | + +----------------------+----------------------+ + | + v + +----------------------------+ + | Control-Plane DB Substrate| + | (Shared Lease & Locks) | + +--------------+-------------+ + | + v + +----------------------------+ + | Gitea API | + +----------------------------+ +``` + +--- + +### 3.2 Key System Components + +#### A. Multiple MCP Instance Cohorts +* The control plane runs across $N \ge 2$ redundant process nodes. +* Dual-namespace deployment allows running the old version (Node A) alongside a updated version (Node B) during rolling upgrades. + +#### B. Shared Durable Session Storage & Resume Tokens +* Session context, preflight verification proofs, and capability resolution states are stored in the shared control-plane database. +* Client requests carry an explicit `session_id` and `resume_token`. If an MCP instance restarts or a request routes to a different instance, the target node validates the token against the database without requiring full session re-initialization. + +#### C. Shared Lease Authority & Fencing Counters +* Workflow leases (`gitea_allocate_next_work`, `gitea_adopt_workflow_lease`) use monotonic fencing tokens (`lease_generation_id`). +* When Node B acquires or renews a lease, it increments the generation counter. Any delayed or out-of-order write attempt from Node A using an older generation token is rejected by database constraints. + +#### D. Leader Election & Coordinated Drain +* Node clusters elect a primary coordinator node for administrative background tasks (such as stale lease cleanup or incident Watchdogs). +* During a rolling deployment: + 1. Node B (new version) is launched and registers as healthy. + 2. Router directs new session creations to Node B. + 3. Node A enters `MAINTENANCE_DRAIN` status ([#659](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/659)), completing in-flight mutations while refusing new tasks. + 4. Once all active sessions migrate or complete, Node A shuts down cleanly. + +#### E. Idempotent Mutations & Failover Safety +* All state-changing tool executions (PR creation, review submission, merge operations, label changes) carry a deterministic `idempotency_key`. +* If a network connection flaps or a node fails mid-mutation, the re-issued request with the same `idempotency_key` is recognized by the control-plane substrate, returning the existing recorded result without repeating side effects on Gitea. + +#### F. Schema Version Compatibility +* Database migrations follow non-breaking additive patterns. +* During rolling upgrades where Node A (Version $N$) and Node B (Version $N+1$) run concurrently, both versions operate against the shared schema without structural conflicts. + +--- + +## 4. Split-Brain & Failure Behavior + +### 4.1 Split-Brain Risk Scenarios & Mitigation + +| Scenario | Risk | Mitigation Strategy | +|---|---|---| +| **Network Partition between Nodes** | Both Node A and Node B attempt to process operations for the same issue/PR. | **Generation Fencing:** Lease renewal requires updating the DB generation counter. The node isolated from the DB fails closed immediately. | +| **Stale Node Recovery** | Node A recovers after a long pause and executes a queued mutation. | **Lease Expiry & TTL Fencing:** Transactions verify that `expires_at > NOW()` within the atomic SQLite transaction boundaries. | +| **Database Connection Loss** | Node loses access to shared control-plane DB substrate. | **Strict Fail-Closed:** The node immediately marks all task capabilities as `blocked` and rejects mutation tools until DB connectivity is re-established. | + +--- + +## 5. Phased Implementation Milestones + +```mermaid +flowchart TD + M1[Milestone 1: Shared Control-Plane DB Schema & Resume Tokens] --> M2[Milestone 2: Idempotent Mutation Layer] + M2 --> M3[Milestone 3: Health Routing & Standby Failover] + M3 --> M4[Milestone 4: Active-Active Rolling Deployment & Auto-Drain] +``` + +### Milestone 1: Shared Control-Plane DB Schema & Resume Tokens (Post-#655) +* Extend [#613] Control-Plane DB schema to store multi-instance node heartbeat records and session resume tokens. +* Enable session lookup across instances via `session_id`. + +### Milestone 2: Idempotent Mutation Layer & Lease Fencing +* Add mandatory `idempotency_key` tracking to all Gitea mutation tools. +* Implement monotonic lease fencing counters in `gitea_allocate_next_work` and `gitea_adopt_workflow_lease`. + +### Milestone 3: Health-Based Routing & Active-Passive Standby +* Introduce lightweight proxy/router capable of checking node health endpoints. +* Implement active-standby failover where standby node automatically assumes work if active node fails health checks. + +### Milestone 4: Active-Active Horizontal Deployment & Rolling Upgrade Automation +* Enable true active-active multi-instance execution. +* Integrate automated zero-downtime rolling upgrades coordinated with `gitea_request_mcp_restart` maintenance drain. + +--- + +## 6. Observability & Audit Requirements + +High-availability control plane operations must expose clear telemetry and audit trails: + +* **Node Registry Telemetry:** Active nodes, version numbers, uptime, and heartbeat timestamps reported via `gitea_get_runtime_context`. +* **Lease Fencing Metrics:** Tracking lease acquire latency, fence rejection counts, and lease handoff durations. +* **Failover & Re-route Audit Logs:** Durable logging of session migrations between nodes, drain initiation, and process retirement events. + +--- + +## 7. Tradeoffs & Accepted Risks + +* **Increased Architectural Complexity:** Moving from a single process to a multi-instance control plane requires robust DB locking, proxy routing, and migration governance. +* **Database Dependency:** The control-plane database substrate becomes a critical shared dependency for multi-node deployments. High availability for the underlying SQLite file system / DB must be guaranteed.