Supervise MCP daemon cohort lifecycle: one eligible cohort per profile, drain and reap superseded cohorts #900

New Issue

2026-07-24T21:58:07-05:00

jcwalker3 commented

2026-07-24 21:58:07 -05:00

Related: epic #887, #686, #655, #659, #669, #678

Problem

Nothing owns the lifetime of an MCP daemon cohort. Reconnects create new cohorts, superseded cohorts are never drained or reaped, and the surviving population is only ever inspected — never bounded.

Read-only process inventory on this host found roughly forty live mcp_server.py processes in at least five distinct cohorts, spawned across four separate days and still resident: cohorts at Wed06AM, Thu03AM, 5:45AM, 10:32PM, and 10:33PM. Every one of them holds production credentials and reads and writes the same session, lease, lock, and decision stores. A comparable inventory was already recorded on #686 during the 2026-07-12 incident, which found approximately fifty-six such processes. The population has never been bounded in the intervening time, which is the defect: the accumulation is structural, not incidental.

The existing issues address neighbouring concerns and stop short of this one. #686 detects manually launched duplicates by provenance and fails closed on them. #659 drains work before a restart. #669 prefers narrow recovery over full reset. None of them establishes how many cohorts may exist, retires the ones that have been superseded, or bounds growth across repeated reconnects.

Coverage gap this issue closes

Assessed against the daemon lifecycle requirement set at master a4c73766f4b0cc32f7c3808688eceeb6fee74335:

Requirement	Prior state
Exactly one eligible cohort per profile and configuration	PARTIAL — #686 AC1 inventories duplicates per profile and AC3 stops them masking stale runtimes; comment 10662 requirement 5 fails closed when duplicates exist. All are detection; none establishes a single eligible cohort, and configuration is not a dimension of eligibility
Superseded cohorts drained and reaped	UNCOVERED — #659 drains work ahead of a restart, which is a different lifecycle event; no issue retires a superseded cohort
Reconnects do not accumulate eligible daemons	UNCOVERED — no issue bounds process growth across reconnects
Status exposes PID, cohort ID, startup SHA, age, endpoint, config fingerprint	PARTIAL — #678 AC2 covers PID/session, startup SHA, current master SHA, boundary status; #686 AC1 adds provenance and per-profile duplicate inventory. Cohort ID, age, endpoint, and configuration fingerprint are absent
Regression proves repeated reconnects leave one eligible cohort	UNCOVERED

Acceptance criteria

A cohort has an explicit identity covering profile, configuration fingerprint, and startup SHA, and that identity is stable for the cohort's lifetime.
At most one cohort per profile and configuration is eligible to serve; any other is ineligible by construction rather than by heuristic.
When a new cohort becomes eligible, the superseded cohort is drained and then reaped; reaping is a supervised transition, never a manual kill, which #630 forbids.
Repeated reconnects do not increase the number of eligible daemons; total resident processes converge rather than grow.
Runtime status exposes, for every cohort: PID, cohort ID, startup SHA, age, endpoint, and configuration fingerprint, and marks exactly which cohort is eligible.
Ineligible-but-resident cohorts are reported as such rather than being silently omitted or counted as healthy.
A cohort that cannot be drained within bounds is reported as an incident rather than left resident and unreported.
Regression: N successive reconnects leave exactly one eligible cohort per profile, with the resident count bounded and asserted.
Regression: a superseded cohort holding an in-flight mutation is drained before reaping, and the mutation is not lost.

Non-goals

Restart authorization and blast radius, owned by #655.
Maintenance-drain of work ahead of a restart, owned by #659.
Recovery-ladder selection, owned by #669.
Provenance of manually launched servers, owned by #686; this issue governs sanctioned cohorts and consumes #686's provenance signal rather than restating it.
Client-side attachment choice, tracked separately as the deterministic-attachment defect.

Duplicate verdict

NOT a duplicate. Verified by reading the bodies, comments, and acceptance criteria of #686, #655, #659, #669, and #678. #686 is the closest and remains detection-only: its criteria inventory and reject untrusted processes but never bound the sanctioned population or retire a superseded cohort.

Related: epic #887, #686, #655, #659, #669, #678 ## Problem Nothing owns the lifetime of an MCP daemon cohort. Reconnects create new cohorts, superseded cohorts are never drained or reaped, and the surviving population is only ever inspected — never bounded. Read-only process inventory on this host found roughly forty live `mcp_server.py` processes in at least five distinct cohorts, spawned across four separate days and still resident: cohorts at `Wed06AM`, `Thu03AM`, `5:45AM`, `10:32PM`, and `10:33PM`. Every one of them holds production credentials and reads and writes the same session, lease, lock, and decision stores. A comparable inventory was already recorded on #686 during the 2026-07-12 incident, which found approximately fifty-six such processes. The population has never been bounded in the intervening time, which is the defect: the accumulation is structural, not incidental. The existing issues address neighbouring concerns and stop short of this one. #686 detects manually launched duplicates by provenance and fails closed on them. #659 drains work before a restart. #669 prefers narrow recovery over full reset. None of them establishes how many cohorts may exist, retires the ones that have been superseded, or bounds growth across repeated reconnects. ## Coverage gap this issue closes Assessed against the daemon lifecycle requirement set at master `a4c73766f4b0cc32f7c3808688eceeb6fee74335`: | Requirement | Prior state | | --- | --- | | Exactly one eligible cohort per profile and configuration | PARTIAL — #686 AC1 inventories duplicates per profile and AC3 stops them masking stale runtimes; comment 10662 requirement 5 fails closed when duplicates exist. All are detection; none establishes a single eligible cohort, and configuration is not a dimension of eligibility | | Superseded cohorts drained and reaped | UNCOVERED — #659 drains work ahead of a restart, which is a different lifecycle event; no issue retires a superseded cohort | | Reconnects do not accumulate eligible daemons | UNCOVERED — no issue bounds process growth across reconnects | | Status exposes PID, cohort ID, startup SHA, age, endpoint, config fingerprint | PARTIAL — #678 AC2 covers PID/session, startup SHA, current master SHA, boundary status; #686 AC1 adds provenance and per-profile duplicate inventory. Cohort ID, age, endpoint, and configuration fingerprint are absent | | Regression proves repeated reconnects leave one eligible cohort | UNCOVERED | ## Acceptance criteria - A cohort has an explicit identity covering profile, configuration fingerprint, and startup SHA, and that identity is stable for the cohort's lifetime. - At most one cohort per profile and configuration is eligible to serve; any other is ineligible by construction rather than by heuristic. - When a new cohort becomes eligible, the superseded cohort is drained and then reaped; reaping is a supervised transition, never a manual kill, which #630 forbids. - Repeated reconnects do not increase the number of eligible daemons; total resident processes converge rather than grow. - Runtime status exposes, for every cohort: PID, cohort ID, startup SHA, age, endpoint, and configuration fingerprint, and marks exactly which cohort is eligible. - Ineligible-but-resident cohorts are reported as such rather than being silently omitted or counted as healthy. - A cohort that cannot be drained within bounds is reported as an incident rather than left resident and unreported. - Regression: N successive reconnects leave exactly one eligible cohort per profile, with the resident count bounded and asserted. - Regression: a superseded cohort holding an in-flight mutation is drained before reaping, and the mutation is not lost. ## Non-goals - Restart authorization and blast radius, owned by #655. - Maintenance-drain of work ahead of a restart, owned by #659. - Recovery-ladder selection, owned by #669. - Provenance of manually launched servers, owned by #686; this issue governs sanctioned cohorts and consumes #686's provenance signal rather than restating it. - Client-side attachment choice, tracked separately as the deterministic-attachment defect. ## Duplicate verdict NOT a duplicate. Verified by reading the bodies, comments, and acceptance criteria of #686, #655, #659, #669, and #678. #686 is the closest and remains detection-only: its criteria inventory and reject untrusted processes but never bound the sanctioned population or retire a superseded cohort.