Design rolling-restart and high-availability MCP architecture #668

New Issue

2026-07-10T15:33:26-05:00

jcwalker3 commented

2026-07-10 15:33:26 -05:00

Problem statement

Even perfect single-process drain cannot eliminate all disruption. There is no durable architecture for replacing MCP instances without stopping active work.

Operational impact

Upgrades still require coordinated stop; dogfooding zero-disruption incomplete.

Scope (design-first; preserve on roadmap)

Investigate and document:

Multiple MCP instances
Shared durable session storage
Shared lease authority
Stateless workers
Leader election
Rolling deployment
Health-based routing
Resume tokens
Idempotent mutations
Connection failover
Session affinity
Version compatibility during rolling upgrades

Produce ADR with phased path from single-node governed restart (#655) to multi-instance HA. Do not delete this work if deferred—keep linked from #652/#653 preserved lists.

Non-goals

Implement multi-instance cluster in this issue.
Change Gitea as durable work SoT.

Required implementation investigation

#655 #613 control-plane DB; dual-namespace deployment; #615 runtime ADR; #652 Phase 5 maturity.

Proposed implementation direction

docs/architecture/mcp-ha-rolling-restart.md ADR with options, risks, milestones.

Security / workflow-safety

Shared lease authority must not split-brain; fail closed on quorum loss.

Failure behavior

Split-brain scenarios analyzed; preferred fail-closed behavior specified.

Acceptance criteria

ADR merged covering all investigation bullets.
Explicit milestones after #655 v1.
Risks/tradeoffs recorded.
Linked #652 #653 #655; preserved if unscheduled.
No code required beyond docs unless tiny stubs.

Required tests

Doc presence / link checks if used.

Observability and audit

HA control plane observability sketched.

Dependencies

Parent #655 · Soft-depends #656 · Vision #652 · Roadmap #653 Phase 5

Rollout

Design only until controller prioritizes implementation epics.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Write HA/rolling-restart ADR
NEXT_PROMPT: Author HA ADR under #655; PR; stop

Required proof

ADR PR.

Required final response and handoff

Brief PR → controller/architect review.

## Problem statement Even perfect single-process drain cannot eliminate all disruption. There is no durable architecture for replacing MCP instances without stopping active work. ## Operational impact Upgrades still require coordinated stop; dogfooding zero-disruption incomplete. ## Scope (design-first; preserve on roadmap) Investigate and document: * Multiple MCP instances * Shared durable session storage * Shared lease authority * Stateless workers * Leader election * Rolling deployment * Health-based routing * Resume tokens * Idempotent mutations * Connection failover * Session affinity * Version compatibility during rolling upgrades Produce ADR with phased path from single-node governed restart (#655) to multi-instance HA. **Do not delete this work if deferred**—keep linked from #652/#653 preserved lists. ## Non-goals * Implement multi-instance cluster in this issue. * Change Gitea as durable work SoT. ## Required implementation investigation #655 #613 control-plane DB; dual-namespace deployment; #615 runtime ADR; #652 Phase 5 maturity. ## Proposed implementation direction `docs/architecture/mcp-ha-rolling-restart.md` ADR with options, risks, milestones. ## Security / workflow-safety Shared lease authority must not split-brain; fail closed on quorum loss. ## Failure behavior Split-brain scenarios analyzed; preferred fail-closed behavior specified. ## Acceptance criteria 1. ADR merged covering all investigation bullets. 2. Explicit milestones after #655 v1. 3. Risks/tradeoffs recorded. 4. Linked #652 #653 #655; preserved if unscheduled. 5. No code required beyond docs unless tiny stubs. ## Required tests Doc presence / link checks if used. ## Observability and audit HA control plane observability sketched. ## Dependencies * Parent **#655** · Soft-depends **#656** · Vision **#652** · Roadmap **#653** Phase 5 ## Rollout Design only until controller prioritizes implementation epics. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Write HA/rolling-restart ADR NEXT_PROMPT: Author HA ADR under #655; PR; stop ``` ## Required proof ADR PR. ## Required final response and handoff Brief PR → controller/architect review.