Umbrella: Governed MCP restart coordination and zero-disruption recovery #655

New Issue

2026-07-10T15:30:24-05:00

jcwalker3 commented

2026-07-10 15:30:24 -05:00

Product principle

No single LLM or ordinary operator may restart the MCP server while other active work may be affected.

Restart is a last resort. Uncoordinated restarts interrupt unrelated author/reviewer/controller/merger/reconciler/monitoring work, drop tool sessions, orphan leases/locks, interrupt mutations, create duplicates, and leave workflow state inconsistent.

Dogfooding: Gitea-Tools must use these controls for its own development.

Problem statement

Today, recovery often means touching configs, IDE reconnect, auto-restart helpers (_trigger_mcp_auto_restart in gitea_mcp_server.py), or forbidden pkill (#630). There is no central coordinator evaluating blast radius, no mandatory graceful drain, no durable session checkpoint gate, and no post-restart reconciliation proof. Manual restarts are evidence of missing product capability—not a normal path to preserve.

Operational impact

Concurrent LLMs lose mid-flight mutations and leases.
Contaminated sessions claim clean completion after host process kills.
Operators reconstruct context from chat instead of durable checkpoints.
Control Plane vision (#652) maintenance mode / restart controls cannot ship safely without this substrate.

Scope (umbrella)

This issue owns the canonical restart-governance program. Implementation is via linked children only.

Restart must be last resort

Prefer, and record attempts for:

Client reconnect · session reconnect · capability refresh · namespace rebind · transport recreation · configuration reload · individual worker restart · role-runtime restart · connector restart · stale lease recovery · lock recovery · session quarantine · runtime reconciliation

Full restart only after narrower recovery is insufficient, with durable rationale.

Central restart coordinator

All restart requests evaluate: active sessions/projects/issue-PR work, leases/locks, in-progress mutations, critical sections, worktrees, connected clients, pending tool calls, heartbeats, checkpoint completeness, blast radius.

Direct bypass paths removed, restricted, or guarded.

Consensus and authorization

Restart only if: no affected sessions; or all affected sessions drained; or controller approval after safety checks; or defined quorum; or emergency break-glass.

Investigation: first version may be controller approval + automated gates (preferred MVP) vs full distributed consensus (later).

No author/reviewer/merger/reconciler/ordinary LLM session may independently restart.

Graceful drain protocol

Maintenance-drain state: stop new assignment; reject/defer mutations; safe reads only; notify sessions; finish critical ops; checkpoint; record next actions; resolve/transfer/preserve leases; verify handoffs; drain proof; restart only after gate passes.

Durable session checkpointing

Checkpoint schema (session/runtime/LLM identity, role, project/repo, issue/PR, worktree, branch/commit, capabilities, lease/lock, stage, last/pending actions, mutations, evidence, blocker, next action, recovery instructions, timestamp/version). Post-start reconcile against live state—never blind restore.

Restart impact preview

Machine-readable report for console (#642/#652): affected sessions/projects/issues/PRs, mutations, leases at risk, ack/non-ack, critical sections, recovery actions, blast radius, safe/unsafe/override-required.

Restart classes

Client reconnect · session reconnect · worker · role-runtime · connector · config reload · rolling MCP · full MCP · host — each with permission, blast radius, drain, approval, audit, recovery.

Break-glass

Privileged auth, reason, impact ack, immutable audit, incident record, disrupted session list, mandatory post-restart reconcile, durable follow-up issue. Not normal path.

Post-restart reconciliation

Health, clients, sessions, leases/locks, capabilities, worktrees, interrupted mutations, duplicates, queue restore, next-action notify, completion proof, unresolved recovery work.

HA / rolling-restart roadmap (preserve)

Multi-instance MCP, shared durable session store, shared lease authority, stateless workers, leader election, rolling deploy, health routing, resume tokens, idempotent mutations, failover, affinity, version compatibility.

Explicit non-goals

Do not normalize unilateral LLM restarts or pkill as recovery.
Do not re-implement #630 (contamination detection) or #642 (console UI) wholesale—integrate.
Do not implement full HA multi-instance in v1—keep as explicit roadmap children.
Do not implement unrelated web console features here.

Related issues (not duplicates)

#630 — block manual pkill contamination (complementary fail-closed)
#642 — web console sanctioned restart UI (consumes this substrate)
#591 — auto-restart on master advance (closed; must be brought under coordinator)
#584 — host auto-reconnect (sanctioned narrow recovery)
#610/#615/#531 — stale runtime detection/ADR
#559 — persist proof across daemon processes (closed; checkpoint related)
#652 vision · #653 roadmap · #631 console umbrella

Child issues (create/link)

Restart governance and authorization policy
Inventory and guard all existing restart paths
Restart coordinator and impact analysis
Graceful maintenance-drain mode
Durable session checkpoint schema
Pre-restart drain proof and hard gate
Post-restart reconciliation
Restart classes and blast-radius permissions
Emergency break-glass restart workflow
Restart audit events and incident creation
Concurrent-session restart safety tests
Restart status/approval in Control Plane web console (extends #642)
Rolling-restart and HA architecture design
Scoped component recovery replacing full-server resets

Dogfooding acceptance

Concurrent-session tests must prove: no unilateral restart; drain stops new work; safe work finishes; unsafe mutations block restart; checkpoints durable; leases not silently orphaned; resume/next-action; failed drain → durable incident; scoped restarts spare unrelated work; no chat reconstruction required.

Security and workflow-safety

Fail closed; RBAC; audit immutability; redaction; controller/break-glass separation; never bypass capability gates during drain.

Acceptance criteria (umbrella)

All children linked to this umbrella, #652, #653.
Policy: restart last resort + coordinator mandatory.
v1 path defined (controller+gates vs quorum).
Dogfooding tests specified in test child.
HA roadmap preserved even if deferred.
No product feature claimed complete on this issue alone.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement children in dependency order starting with policy + path inventory

## Product principle > **No single LLM or ordinary operator may restart the MCP server while other active work may be affected.** Restart is a **last resort**. Uncoordinated restarts interrupt unrelated author/reviewer/controller/merger/reconciler/monitoring work, drop tool sessions, orphan leases/locks, interrupt mutations, create duplicates, and leave workflow state inconsistent. **Dogfooding:** Gitea-Tools must use these controls for its own development. ## Problem statement Today, recovery often means touching configs, IDE reconnect, auto-restart helpers (`_trigger_mcp_auto_restart` in `gitea_mcp_server.py`), or forbidden `pkill` (#630). There is no central coordinator evaluating blast radius, no mandatory graceful drain, no durable session checkpoint gate, and no post-restart reconciliation proof. Manual restarts are evidence of missing product capability—not a normal path to preserve. ## Operational impact * Concurrent LLMs lose mid-flight mutations and leases. * Contaminated sessions claim clean completion after host process kills. * Operators reconstruct context from chat instead of durable checkpoints. * Control Plane vision (#652) maintenance mode / restart controls cannot ship safely without this substrate. ## Scope (umbrella) This issue owns the **canonical restart-governance program**. Implementation is via linked children only. ### Restart must be last resort Prefer, and record attempts for: * Client reconnect · session reconnect · capability refresh · namespace rebind · transport recreation · configuration reload · individual worker restart · role-runtime restart · connector restart · stale lease recovery · lock recovery · session quarantine · runtime reconciliation Full restart only after narrower recovery is insufficient, with durable rationale. ### Central restart coordinator All restart requests evaluate: active sessions/projects/issue-PR work, leases/locks, in-progress mutations, critical sections, worktrees, connected clients, pending tool calls, heartbeats, checkpoint completeness, blast radius. Direct bypass paths removed, restricted, or guarded. ### Consensus and authorization Restart only if: no affected sessions; or all affected sessions drained; or controller approval after safety checks; or defined quorum; or emergency break-glass. **Investigation:** first version may be controller approval + automated gates (preferred MVP) vs full distributed consensus (later). No author/reviewer/merger/reconciler/ordinary LLM session may independently restart. ### Graceful drain protocol Maintenance-drain state: stop new assignment; reject/defer mutations; safe reads only; notify sessions; finish critical ops; checkpoint; record next actions; resolve/transfer/preserve leases; verify handoffs; drain proof; restart only after gate passes. ### Durable session checkpointing Checkpoint schema (session/runtime/LLM identity, role, project/repo, issue/PR, worktree, branch/commit, capabilities, lease/lock, stage, last/pending actions, mutations, evidence, blocker, next action, recovery instructions, timestamp/version). Post-start reconcile against live state—never blind restore. ### Restart impact preview Machine-readable report for console (#642/#652): affected sessions/projects/issues/PRs, mutations, leases at risk, ack/non-ack, critical sections, recovery actions, blast radius, safe/unsafe/override-required. ### Restart classes Client reconnect · session reconnect · worker · role-runtime · connector · config reload · rolling MCP · full MCP · host — each with permission, blast radius, drain, approval, audit, recovery. ### Break-glass Privileged auth, reason, impact ack, immutable audit, incident record, disrupted session list, mandatory post-restart reconcile, durable follow-up issue. Not normal path. ### Post-restart reconciliation Health, clients, sessions, leases/locks, capabilities, worktrees, interrupted mutations, duplicates, queue restore, next-action notify, completion proof, unresolved recovery work. ### HA / rolling-restart roadmap (preserve) Multi-instance MCP, shared durable session store, shared lease authority, stateless workers, leader election, rolling deploy, health routing, resume tokens, idempotent mutations, failover, affinity, version compatibility. ## Explicit non-goals * Do not normalize unilateral LLM restarts or pkill as recovery. * Do not re-implement #630 (contamination detection) or #642 (console UI) wholesale—integrate. * Do not implement full HA multi-instance in v1—keep as explicit roadmap children. * Do not implement unrelated web console features here. ## Related issues (not duplicates) * **#630** — block manual pkill contamination (complementary fail-closed) * **#642** — web console sanctioned restart UI (consumes this substrate) * **#591** — auto-restart on master advance (closed; must be brought under coordinator) * **#584** — host auto-reconnect (sanctioned narrow recovery) * **#610/#615/#531** — stale runtime detection/ADR * **#559** — persist proof across daemon processes (closed; checkpoint related) * **#652** vision · **#653** roadmap · **#631** console umbrella ## Child issues (create/link) 1. Restart governance and authorization policy 2. Inventory and guard all existing restart paths 3. Restart coordinator and impact analysis 4. Graceful maintenance-drain mode 5. Durable session checkpoint schema 6. Pre-restart drain proof and hard gate 7. Post-restart reconciliation 8. Restart classes and blast-radius permissions 9. Emergency break-glass restart workflow 10. Restart audit events and incident creation 11. Concurrent-session restart safety tests 12. Restart status/approval in Control Plane web console (extends #642) 13. Rolling-restart and HA architecture design 14. Scoped component recovery replacing full-server resets ## Dogfooding acceptance Concurrent-session tests must prove: no unilateral restart; drain stops new work; safe work finishes; unsafe mutations block restart; checkpoints durable; leases not silently orphaned; resume/next-action; failed drain → durable incident; scoped restarts spare unrelated work; no chat reconstruction required. ## Security and workflow-safety Fail closed; RBAC; audit immutability; redaction; controller/break-glass separation; never bypass capability gates during drain. ## Acceptance criteria (umbrella) 1. All children linked to this umbrella, #652, #653. 2. Policy: restart last resort + coordinator mandatory. 3. v1 path defined (controller+gates vs quorum). 4. Dogfooding tests specified in test child. 5. HA roadmap preserved even if deferred. 6. No product feature claimed complete on this issue alone. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement children in dependency order starting with policy + path inventory ```

jcwalker3 added the type:guardrail status:ready workflow-hardening anti-stomp safety mcp-health stale-runtime labels 2026-07-10 15:30:25 -05:00

Umbrella: Governed MCP restart coordination and zero-disruption recovery #655