Scaled-Tech-Consulting/Gitea-Tools
4
0
Fork 0
Code Issues 110 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

Implement graceful MCP maintenance-drain mode #659

New Issue
Open
opened 2026-07-10 15:31:23 -05:00 by jcwalker3 · 2 comments
jcwalker3 commented 2026-07-10 15:31:23 -05:00
Owner
Copy Link
Copy Source

Problem statement

MCP has no visible maintenance-drain state that stops new work, defers mutations, notifies sessions, finishes critical ops, and checkpoints before restart.

Operational impact

Restarts cut mid-mutation; new work is still assigned during recovery chaos.

Scope

Implement drain protocol steps:

  1. Stop new assignment
  2. Reject/defer new mutations
  3. Permit only explicitly safe reads
  4. Notify connected sessions
  5. Allow critical ops to finish
  6. Require durable checkpointing
  7. Record each session state + next action
  8. Resolve/transfer/preserve leases and locks
  9. Verify canonical handoff completeness
  10. Confirm no unsafe in-flight ops
  11. Produce machine-verifiable drain proof
  12. Permit restart only after drain gate (see drain-proof child)

Non-goals

  • Break-glass bypass implementation details (separate).
  • Full HA rolling restart.

Required implementation investigation

#655 #658; allocator; mutation preflight; session registry; #628 handoffs.

Proposed implementation direction

Global maintenance_drain flag in control-plane + MCP process; preflight hooks deny mutations; allocator returns WAIT; status tool for clients.

Security / workflow-safety

Fail closed: if drain incomplete, no restart; reads redacted.

Failure behavior

Drain timeout → fail drain proof → durable incident issue.

Acceptance criteria

  1. Enter/exit drain is capability-gated and audited.
  2. New work assignment stops during drain.
  3. Mutations deferred/rejected except allowlisted safety ops.
  4. Sessions can observe drain state.
  5. Tests for assignment stop + mutation deny.
  6. Links #652 #653 #655.

Required tests

Concurrent assign during drain fails; mutation fail closed; read still works.

Observability and audit

Drain enter/exit/progress events.

Dependencies

  • Parent #655 · Soft-depends #658 · Pairs with checkpoint + drain-proof children · Vision #652 · Roadmap #653

Rollout

Dogfood on Gitea-Tools MCP namespaces first.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement maintenance-drain mode
NEXT_PROMPT: Author drain mode under #655; PR; stop

Required proof

Code + tests + drain status sample.

Required final response and handoff

Brief PR → reviewer.

## Problem statement MCP has no visible maintenance-drain state that stops new work, defers mutations, notifies sessions, finishes critical ops, and checkpoints before restart. ## Operational impact Restarts cut mid-mutation; new work is still assigned during recovery chaos. ## Scope Implement drain protocol steps: 1. Stop new assignment 2. Reject/defer new mutations 3. Permit only explicitly safe reads 4. Notify connected sessions 5. Allow critical ops to finish 6. Require durable checkpointing 7. Record each session state + next action 8. Resolve/transfer/preserve leases and locks 9. Verify canonical handoff completeness 10. Confirm no unsafe in-flight ops 11. Produce machine-verifiable drain proof 12. Permit restart only after drain gate (see drain-proof child) ## Non-goals * Break-glass bypass implementation details (separate). * Full HA rolling restart. ## Required implementation investigation #655 #658; allocator; mutation preflight; session registry; #628 handoffs. ## Proposed implementation direction Global `maintenance_drain` flag in control-plane + MCP process; preflight hooks deny mutations; allocator returns WAIT; status tool for clients. ## Security / workflow-safety Fail closed: if drain incomplete, no restart; reads redacted. ## Failure behavior Drain timeout → fail drain proof → durable incident issue. ## Acceptance criteria 1. Enter/exit drain is capability-gated and audited. 2. New work assignment stops during drain. 3. Mutations deferred/rejected except allowlisted safety ops. 4. Sessions can observe drain state. 5. Tests for assignment stop + mutation deny. 6. Links #652 #653 #655. ## Required tests Concurrent assign during drain fails; mutation fail closed; read still works. ## Observability and audit Drain enter/exit/progress events. ## Dependencies * Parent **#655** · Soft-depends **#658** · Pairs with checkpoint + drain-proof children · Vision **#652** · Roadmap **#653** ## Rollout Dogfood on Gitea-Tools MCP namespaces first. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement maintenance-drain mode NEXT_PROMPT: Author drain mode under #655; PR; stop ``` ## Required proof Code + tests + drain status sample. ## Required final response and handoff Brief PR → reviewer.
jcwalker3 added status:in-progress and removed status:ready labels 2026-07-25 16:02:20 -05:00
jcwalker3 commented 2026-07-25 16:02:20 -05:00
Author
Owner
Copy Link
Copy Source

Issue claim heartbeat

<!-- gitea-issue-claim-heartbeat:v1 --> **Issue claim heartbeat** - kind: claim - issue: #659 - branch: feat/issue-659-maintenance-drain-mode - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation
jcwalker3 added status:pr-open and removed status:in-progress labels 2026-07-25 16:02:41 -05:00
sysadmin commented 2026-07-25 17:54:52 -05:00
Owner
Copy Link
Copy Source

Canonical Issue State

STATE: PR-open
WHO_IS_NEXT: author
NEXT_ACTION: Address the three blockers recorded in review 611 on PR #907, then hand back for re-review.
NEXT_PROMPT:

Address the REQUEST_CHANGES verdict (server-side review 611) on PR #907
(Closes #659), prgs / Scaled-Tech-Consulting / Gitea-Tools, reviewed at head
f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9.

B1 (maintenance_drain.py:59-77 vs gitea_mcp_server.py:15493) — the drain safety
allowlist is keyed on task names the mutation chokepoint never passes. Two entries
(write_session_checkpoint, checkpoint_session) name no MCP task at all; seven name
tools that make zero verify_preflight_purity calls; of the two that do reach the
gate, only post_heartbeat matches its own literal, while
gitea_heartbeat_reviewer_pr_lease calls
verify_preflight_purity(remote, task="review_pr"), which is not on the list and is
therefore deferred. A reviewer cannot heartbeat a PR lease during a drain, so the
lease expires across the very restart the drain exists to make safe. Do not simply
add review_pr to the allowlist: that literal is shared with review-verdict
submission, which must stay deferred. Give the reviewer lease-lifecycle calls their
own task literal and key the allowlist on it; drop or fix the entries that name
ungated tools so the list stops implying coverage it does not have.

B2 (gitea_mcp_server.py:2032-2035) — the gate catches every exception from _resolve
and falls back to (org or ""), (repo or ""). For a call site that passes no org or
repo, the drain lookup then queries scope ("", ""), finds no row, and permits the
mutation while a drain is active. This fails open inside a gate whose docstring at
lines 2022-2023 promises the opposite, and whose two neighboring failure paths both
raise. Make scope-resolution failure raise as well.

B3 (tests/test_maintenance_drain.py:43-54, and line 80) — the allowlist test feeds
classify_mutation three of the allowlist's own constants, so it asserts only that
the frozenset contains its own members and cannot fail while B1 is present. Replace
it with a test driving the real task literal from the reviewer heartbeat path,
asserting it is permitted during drain, paired with a negative asserting a review
verdict under the same path is deferred.

Add a regression test per blocker.

WHAT_HAPPENED: PR #907 for this issue received its first formal review decision — REQUEST_CHANGES, recorded server-side as review 611 at head f0c6255d7d by sysadmin under profile prgs-reviewer. The full findings, per-blocker evidence, and the author remediation prompt are in that review body on PR #907.
WHY: AC3 of this issue promises a defined set of quiesce operations keeps working while draining, so in-flight sessions can finish and hand their claims back before a restart. As implemented the allowlist cannot deliver that: nine of its eleven entries name strings the chokepoint never receives, and the one entry corresponding to a real gated call — the reviewer lease heartbeat — is refused because the tool passes the literal review_pr. The observable consequence is a reviewer lease expiring during the very drain entered to avoid stranding it. AC1, AC2, AC4, AC5 and AC6 are satisfied by the current implementation; AC3 is not.
ISSUE: #659
RELATED_PRS: #907
REVIEW_STATUS: REQUEST_CHANGES
MERGE_READY: false
HEAD_SHA: f0c6255d7d
BLOCKERS: B1 the drain safety allowlist is keyed on task literals the mutation chokepoint never passes, so the reviewer lease heartbeat is refused during a drain; B2 the gate's scope resolution fails open on a _resolve exception inside a gate documented as fail-closed; B3 the allowlist test asserts the constant against itself and cannot detect B1.
VALIDATION: Static review at head f0c6255d7d against base branch master. The head advanced from 77d808e7d4 during the review cycle by a base-integration commit only (first parent 77d808e7, second parent master d7ad2838ec); maintenance_drain.py, tests/test_maintenance_drain.py, allocator_service.py and control_plane_db.py are byte-identical across that advance, and every cited line in gitea_mcp_server.py was re-read at f0c6255d. Author changes to the feature during validation: none. No test suite was executed at this head in this session and none is claimed.
LAST_UPDATED_BY: sysadmin (prgs-reviewer)

NATIVE_REVIEW_PROOF: gitea_submit_pr_review via native MCP namespace gitea-reviewer, profile prgs-reviewer, identity sysadmin, expected_head_sha f0c6255d7d, server-side review id 611. This issue comment was posted via gitea_create_issue_comment on the same native namespace. No offline, import, or helper path was used.

[THREAD STATE LEDGER]

what is true now

Issue #659 is open with PR #907 open against master at head f0c6255d7d.

Server-side decision state: review 611, REQUEST_CHANGES, recorded at the current head and not stale; it is the first review decision of any kind on PR #907.
Local verdict/state: REQUEST_CHANGES, three blockers, verified statically against the live head.

what changed

The review verdict for this issue's PR moved from a locally-held result carried across earlier sessions to a recorded server-side decision. The PR head advanced from 77d808e7 to f0c6255d during that interval by a base-integration commit only; the blocker evidence was re-verified at f0c6255d before the verdict was recorded.

what is blocked

Blocker classification: code blocker

B1 — maintenance_drain.py:59-77 allowlists eleven task names, but the gate at gitea_mcp_server.py:2029 compares them against the literal passed to verify_preflight_purity. Two entries name nothing in the tree, seven name tools that never call the chokepoint, one (post_heartbeat) matches correctly, and gitea_heartbeat_reviewer_pr_lease reaches it as review_pr (line 15493) and is deferred. B2 — gitea_mcp_server.py:2032-2035 degrades to scope ("", "") on any _resolve exception and permits the mutation, contradicting the fail-closed contract stated at lines 2022-2023. B3 — tests/test_maintenance_drain.py:43-54 asserts allowlist membership using the allowlist's own constants and cannot fail while B1 is present.

who/what acts next

Next actor: author
Required action: Give the reviewer lease-lifecycle calls a task literal distinct from review_pr and key the allowlist on it; reconcile or drop the allowlist entries that name ungated tools or nonexistent symbols; make the gate's scope-resolution failure raise instead of defaulting to an empty scope; replace the tautological allowlist test with one driving real task literals, plus a regression test per blocker.
Do not do: Do not resolve B1 by adding review_pr to the allowlist — that literal is shared with review-verdict submission, which must stay deferred during a drain, so that change would open a hole rather than close one.

## Canonical Issue State STATE: PR-open WHO_IS_NEXT: author NEXT_ACTION: Address the three blockers recorded in review 611 on PR #907, then hand back for re-review. NEXT_PROMPT: ```text Address the REQUEST_CHANGES verdict (server-side review 611) on PR #907 (Closes #659), prgs / Scaled-Tech-Consulting / Gitea-Tools, reviewed at head f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9. B1 (maintenance_drain.py:59-77 vs gitea_mcp_server.py:15493) — the drain safety allowlist is keyed on task names the mutation chokepoint never passes. Two entries (write_session_checkpoint, checkpoint_session) name no MCP task at all; seven name tools that make zero verify_preflight_purity calls; of the two that do reach the gate, only post_heartbeat matches its own literal, while gitea_heartbeat_reviewer_pr_lease calls verify_preflight_purity(remote, task="review_pr"), which is not on the list and is therefore deferred. A reviewer cannot heartbeat a PR lease during a drain, so the lease expires across the very restart the drain exists to make safe. Do not simply add review_pr to the allowlist: that literal is shared with review-verdict submission, which must stay deferred. Give the reviewer lease-lifecycle calls their own task literal and key the allowlist on it; drop or fix the entries that name ungated tools so the list stops implying coverage it does not have. B2 (gitea_mcp_server.py:2032-2035) — the gate catches every exception from _resolve and falls back to (org or ""), (repo or ""). For a call site that passes no org or repo, the drain lookup then queries scope ("", ""), finds no row, and permits the mutation while a drain is active. This fails open inside a gate whose docstring at lines 2022-2023 promises the opposite, and whose two neighboring failure paths both raise. Make scope-resolution failure raise as well. B3 (tests/test_maintenance_drain.py:43-54, and line 80) — the allowlist test feeds classify_mutation three of the allowlist's own constants, so it asserts only that the frozenset contains its own members and cannot fail while B1 is present. Replace it with a test driving the real task literal from the reviewer heartbeat path, asserting it is permitted during drain, paired with a negative asserting a review verdict under the same path is deferred. Add a regression test per blocker. ``` WHAT_HAPPENED: PR #907 for this issue received its first formal review decision — REQUEST_CHANGES, recorded server-side as review 611 at head f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9 by sysadmin under profile prgs-reviewer. The full findings, per-blocker evidence, and the author remediation prompt are in that review body on PR #907. WHY: AC3 of this issue promises a defined set of quiesce operations keeps working while draining, so in-flight sessions can finish and hand their claims back before a restart. As implemented the allowlist cannot deliver that: nine of its eleven entries name strings the chokepoint never receives, and the one entry corresponding to a real gated call — the reviewer lease heartbeat — is refused because the tool passes the literal `review_pr`. The observable consequence is a reviewer lease expiring during the very drain entered to avoid stranding it. AC1, AC2, AC4, AC5 and AC6 are satisfied by the current implementation; AC3 is not. ISSUE: #659 RELATED_PRS: #907 REVIEW_STATUS: REQUEST_CHANGES MERGE_READY: false HEAD_SHA: f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9 BLOCKERS: B1 the drain safety allowlist is keyed on task literals the mutation chokepoint never passes, so the reviewer lease heartbeat is refused during a drain; B2 the gate's scope resolution fails open on a `_resolve` exception inside a gate documented as fail-closed; B3 the allowlist test asserts the constant against itself and cannot detect B1. VALIDATION: Static review at head f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9 against base branch master. The head advanced from 77d808e7d4793c8f3b9f1b6e8f76f666221caed0 during the review cycle by a base-integration commit only (first parent 77d808e7, second parent master d7ad2838ec6f13a4a8009a800be9b715882e308e); `maintenance_drain.py`, `tests/test_maintenance_drain.py`, `allocator_service.py` and `control_plane_db.py` are byte-identical across that advance, and every cited line in `gitea_mcp_server.py` was re-read at f0c6255d. Author changes to the feature during validation: none. No test suite was executed at this head in this session and none is claimed. LAST_UPDATED_BY: sysadmin (prgs-reviewer) NATIVE_REVIEW_PROOF: gitea_submit_pr_review via native MCP namespace gitea-reviewer, profile prgs-reviewer, identity sysadmin, expected_head_sha f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9, server-side review id 611. This issue comment was posted via gitea_create_issue_comment on the same native namespace. No offline, import, or helper path was used. [THREAD STATE LEDGER] ### what is true now Issue #659 is open with PR #907 open against master at head f0c6255d7d7efe7c1d37a4a450a47b0bc8fe0fa9. Server-side decision state: review 611, REQUEST_CHANGES, recorded at the current head and not stale; it is the first review decision of any kind on PR #907. Local verdict/state: REQUEST_CHANGES, three blockers, verified statically against the live head. ### what changed The review verdict for this issue's PR moved from a locally-held result carried across earlier sessions to a recorded server-side decision. The PR head advanced from 77d808e7 to f0c6255d during that interval by a base-integration commit only; the blocker evidence was re-verified at f0c6255d before the verdict was recorded. ### what is blocked Blocker classification: code blocker B1 — `maintenance_drain.py:59-77` allowlists eleven task names, but the gate at `gitea_mcp_server.py:2029` compares them against the literal passed to `verify_preflight_purity`. Two entries name nothing in the tree, seven name tools that never call the chokepoint, one (`post_heartbeat`) matches correctly, and `gitea_heartbeat_reviewer_pr_lease` reaches it as `review_pr` (line 15493) and is deferred. B2 — `gitea_mcp_server.py:2032-2035` degrades to scope `("", "")` on any `_resolve` exception and permits the mutation, contradicting the fail-closed contract stated at lines 2022-2023. B3 — `tests/test_maintenance_drain.py:43-54` asserts allowlist membership using the allowlist's own constants and cannot fail while B1 is present. ### who/what acts next Next actor: author Required action: Give the reviewer lease-lifecycle calls a task literal distinct from `review_pr` and key the allowlist on it; reconcile or drop the allowlist entries that name ungated tools or nonexistent symbols; make the gate's scope-resolution failure raise instead of defaulting to an empty scope; replace the tautological allowlist test with one driving real task literals, plus a regression test per blocker. Do not do: Do not resolve B1 by adding `review_pr` to the allowlist — that literal is shared with review-verdict submission, which must stay deferred during a drain, so that change would open a hole rather than close one.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels anti-stomp mcp-health status:pr-open type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 jwalker (Regular User Account) superadmin (Admin Account) sysadmin (Regular User Account)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#659