Implement graceful MCP maintenance-drain mode #659

New Issue

2026-07-10T15:31:23-05:00

jcwalker3 commented

2026-07-10 15:31:23 -05:00

Problem statement

MCP has no visible maintenance-drain state that stops new work, defers mutations, notifies sessions, finishes critical ops, and checkpoints before restart.

Operational impact

Restarts cut mid-mutation; new work is still assigned during recovery chaos.

Scope

Implement drain protocol steps:

Stop new assignment
Reject/defer new mutations
Permit only explicitly safe reads
Notify connected sessions
Allow critical ops to finish
Require durable checkpointing
Record each session state + next action
Resolve/transfer/preserve leases and locks
Verify canonical handoff completeness
Confirm no unsafe in-flight ops
Produce machine-verifiable drain proof
Permit restart only after drain gate (see drain-proof child)

Non-goals

Break-glass bypass implementation details (separate).
Full HA rolling restart.

Required implementation investigation

#655 #658; allocator; mutation preflight; session registry; #628 handoffs.

Proposed implementation direction

Global maintenance_drain flag in control-plane + MCP process; preflight hooks deny mutations; allocator returns WAIT; status tool for clients.

Security / workflow-safety

Fail closed: if drain incomplete, no restart; reads redacted.

Failure behavior

Drain timeout → fail drain proof → durable incident issue.

Acceptance criteria

Enter/exit drain is capability-gated and audited.
New work assignment stops during drain.
Mutations deferred/rejected except allowlisted safety ops.
Sessions can observe drain state.
Tests for assignment stop + mutation deny.
Links #652 #653 #655.

Required tests

Concurrent assign during drain fails; mutation fail closed; read still works.

Observability and audit

Drain enter/exit/progress events.

Dependencies

Parent #655 · Soft-depends #658 · Pairs with checkpoint + drain-proof children · Vision #652 · Roadmap #653

Rollout

Dogfood on Gitea-Tools MCP namespaces first.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement maintenance-drain mode
NEXT_PROMPT: Author drain mode under #655; PR; stop

Required proof

Code + tests + drain status sample.

Required final response and handoff

Brief PR → reviewer.

## Problem statement MCP has no visible maintenance-drain state that stops new work, defers mutations, notifies sessions, finishes critical ops, and checkpoints before restart. ## Operational impact Restarts cut mid-mutation; new work is still assigned during recovery chaos. ## Scope Implement drain protocol steps: 1. Stop new assignment 2. Reject/defer new mutations 3. Permit only explicitly safe reads 4. Notify connected sessions 5. Allow critical ops to finish 6. Require durable checkpointing 7. Record each session state + next action 8. Resolve/transfer/preserve leases and locks 9. Verify canonical handoff completeness 10. Confirm no unsafe in-flight ops 11. Produce machine-verifiable drain proof 12. Permit restart only after drain gate (see drain-proof child) ## Non-goals * Break-glass bypass implementation details (separate). * Full HA rolling restart. ## Required implementation investigation #655 #658; allocator; mutation preflight; session registry; #628 handoffs. ## Proposed implementation direction Global `maintenance_drain` flag in control-plane + MCP process; preflight hooks deny mutations; allocator returns WAIT; status tool for clients. ## Security / workflow-safety Fail closed: if drain incomplete, no restart; reads redacted. ## Failure behavior Drain timeout → fail drain proof → durable incident issue. ## Acceptance criteria 1. Enter/exit drain is capability-gated and audited. 2. New work assignment stops during drain. 3. Mutations deferred/rejected except allowlisted safety ops. 4. Sessions can observe drain state. 5. Tests for assignment stop + mutation deny. 6. Links #652 #653 #655. ## Required tests Concurrent assign during drain fails; mutation fail closed; read still works. ## Observability and audit Drain enter/exit/progress events. ## Dependencies * Parent **#655** · Soft-depends **#658** · Pairs with checkpoint + drain-proof children · Vision **#652** · Roadmap **#653** ## Rollout Dogfood on Gitea-Tools MCP namespaces first. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement maintenance-drain mode NEXT_PROMPT: Author drain mode under #655; PR; stop ``` ## Required proof Code + tests + drain status sample. ## Required final response and handoff Brief PR → reviewer.