Detect and reject manually launched duplicate MCP role servers as unsupported recovery #686

Open
opened 2026-07-12 18:25:02 -05:00 by jcwalker3 · 2 comments
Owner

Problem

During the 2026-07-12 #584/#672 EOF incident, a diagnostic session manually launched a duplicate role server from a terminal:

GITEA_MCP_CONFIG=/Users/jasonwalker/.config/gitea-tools/profiles.json GITEA_MCP_PROFILE=prgs-reviewer GITEA_DUMMY=2 venv/bin/python3 mcp_server.py

Nothing in the toolchain detected or rejected this. There is currently no guard distinguishing a client-managed role server (spawned by the IDE/MCP client, bound to its stdio transport) from an ad hoc terminal launch.

Why this is dangerous

  1. Never namespace recovery. A terminal-launched mcp_server.py holds its own stdio; it can never bind to the IDE client's pipes. It cannot restore a dropped namespace, but a session may believe it did.
  2. Poisons staleness detection. _check_mcp_runtimes_diagnostics keeps only the newest process per GITEA_MCP_PROFILE (gitea_mcp_server.py:10986 at 22698c1). A freshly launched manual duplicate masks a stale client-managed runtime for that profile, defeating the stale-runtime gate.
  3. Unsupported env overrides go unrejected. GITEA_DUMMY has zero references in the codebase — it is silently ignored rather than refused, so ad hoc experimentation looks accepted.
  4. Shared mutable state. A duplicate role server reads/writes the same session, lease, lock, and decision stores as the supported servers, from an unaudited provenance.

Evidence (2026-07-12)

  • Read-only process inventory at ~19:15 local found ~56 mcp_server.py processes across cohorts (hourly orphan pairs plus the live client-managed quads at 19:06:30/19:07:40); none still carried GITEA_DUMMY in its environment — the manual process had already terminated (lifetime unknown; PID not captured before exit).
  • It was not accepted as namespace recovery: the healthy namespaces answering after the incident are the client-managed quads (verified via gitea_whoami + gitea_get_runtime_context on each).
  • Full violation record: see incident comments on #584 and #672.

Required behavior

  • Detection: a role server can prove its provenance (e.g. launch marker/env stamped only by the sanctioned client-managed launch path, parent-process identity, or an explicit provenance file). Diagnostics (gitea_get_runtime_context, namespace-health tools) report provenance for the serving process and inventory duplicates per profile.
  • Rejection: a server lacking client-managed provenance refuses mutating tools fail-closed with a typed blocker naming the supported path (IDE/client reconnect); unknown env overrides matching GITEA_* that the server does not consume are reported, not silently ignored.
  • Duplicate handling: staleness/runtime diagnostics must not let a non-client-managed process represent a profile's runtime freshness (fixes the running_profiles masking above, coordinated with the resolver contract fix in #685).
  • Docs: supported recovery is IDE/client-managed reconnect only; manual mcp_server.py / gitea_mcp_server.py launches, GITEA_DUMMY-style ad hoc overrides, and mcp_config.json hand-edits are classified as workflow contamination (same class as #630 manual kills).

Acceptance criteria

  1. A manually launched role server is detectable and identified in runtime diagnostics (provenance field + duplicate inventory per profile).
  2. Mutating tools on a server without client-managed provenance fail closed with a typed blocker and exact next action.
  3. Staleness detection ignores (or flags) non-client-managed duplicates instead of letting them mask stale supported runtimes.
  4. Unsupported GITEA_* env overrides are surfaced in diagnostics rather than silently ignored.
  5. Tests cover: manual duplicate present + supported runtime stale; manual-only server attempting mutation; clean client-managed baseline.
  6. Docs distinguish sanctioned reconnect from manual launch, cross-linked with #630 (manual kill) and #657 (restart-path inventory).

Linkage

  • Incident: #584, #672 (violation record comments there).
  • Resolver contract defect that provoked the manual launch: #685.
  • Siblings: #630 (block manual process killing; this issue blocks manual launching), #657 (restart-path inventory), #669 (scoped recovery), #678 (sanctioned reconnect exposure).

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement provenance detection + fail-closed rejection of manually launched duplicate role servers per ACs
NEXT_PROMPT:
Role: AUTHOR
Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools
Issue: this issue

1. gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind branches/ worktree.
2. Design provenance marker for client-managed launches; expose provenance + per-profile duplicate inventory in runtime diagnostics.
3. Fail-close mutating tools when serving process lacks client-managed provenance (typed blocker, exact next action = IDE/client reconnect).
4. Coordinate with #685 so staleness checks ignore/flag non-client-managed duplicates.
5. Surface unconsumed GITEA_* env overrides in diagnostics.
6. Tests per ACs; docs runbook update; PR; stop; hand off to independent reviewer.
## Problem During the 2026-07-12 #584/#672 EOF incident, a diagnostic session manually launched a duplicate role server from a terminal: ``` GITEA_MCP_CONFIG=/Users/jasonwalker/.config/gitea-tools/profiles.json GITEA_MCP_PROFILE=prgs-reviewer GITEA_DUMMY=2 venv/bin/python3 mcp_server.py ``` Nothing in the toolchain detected or rejected this. There is currently no guard distinguishing a client-managed role server (spawned by the IDE/MCP client, bound to its stdio transport) from an ad hoc terminal launch. ## Why this is dangerous 1. **Never namespace recovery.** A terminal-launched `mcp_server.py` holds its own stdio; it can never bind to the IDE client's pipes. It cannot restore a dropped namespace, but a session may believe it did. 2. **Poisons staleness detection.** `_check_mcp_runtimes_diagnostics` keeps only the newest process per `GITEA_MCP_PROFILE` (`gitea_mcp_server.py:10986` at `22698c1`). A freshly launched manual duplicate masks a stale client-managed runtime for that profile, defeating the stale-runtime gate. 3. **Unsupported env overrides go unrejected.** `GITEA_DUMMY` has zero references in the codebase — it is silently ignored rather than refused, so ad hoc experimentation looks accepted. 4. **Shared mutable state.** A duplicate role server reads/writes the same session, lease, lock, and decision stores as the supported servers, from an unaudited provenance. ## Evidence (2026-07-12) * Read-only process inventory at ~19:15 local found ~56 `mcp_server.py` processes across cohorts (hourly orphan pairs plus the live client-managed quads at 19:06:30/19:07:40); none still carried `GITEA_DUMMY` in its environment — the manual process had already terminated (lifetime unknown; PID not captured before exit). * It was not accepted as namespace recovery: the healthy namespaces answering after the incident are the client-managed quads (verified via `gitea_whoami` + `gitea_get_runtime_context` on each). * Full violation record: see incident comments on #584 and #672. ## Required behavior * **Detection:** a role server can prove its provenance (e.g. launch marker/env stamped only by the sanctioned client-managed launch path, parent-process identity, or an explicit provenance file). Diagnostics (`gitea_get_runtime_context`, namespace-health tools) report provenance for the serving process and inventory duplicates per profile. * **Rejection:** a server lacking client-managed provenance refuses mutating tools fail-closed with a typed blocker naming the supported path (IDE/client reconnect); unknown env overrides matching `GITEA_*` that the server does not consume are reported, not silently ignored. * **Duplicate handling:** staleness/runtime diagnostics must not let a non-client-managed process represent a profile's runtime freshness (fixes the `running_profiles` masking above, coordinated with the resolver contract fix in #685). * **Docs:** supported recovery is IDE/client-managed reconnect only; manual `mcp_server.py` / `gitea_mcp_server.py` launches, `GITEA_DUMMY`-style ad hoc overrides, and `mcp_config.json` hand-edits are classified as workflow contamination (same class as #630 manual kills). ## Acceptance criteria 1. A manually launched role server is detectable and identified in runtime diagnostics (provenance field + duplicate inventory per profile). 2. Mutating tools on a server without client-managed provenance fail closed with a typed blocker and exact next action. 3. Staleness detection ignores (or flags) non-client-managed duplicates instead of letting them mask stale supported runtimes. 4. Unsupported `GITEA_*` env overrides are surfaced in diagnostics rather than silently ignored. 5. Tests cover: manual duplicate present + supported runtime stale; manual-only server attempting mutation; clean client-managed baseline. 6. Docs distinguish sanctioned reconnect from manual launch, cross-linked with #630 (manual kill) and #657 (restart-path inventory). ## Linkage * Incident: #584, #672 (violation record comments there). * Resolver contract defect that provoked the manual launch: #685. * Siblings: #630 (block manual process **killing**; this issue blocks manual **launching**), #657 (restart-path inventory), #669 (scoped recovery), #678 (sanctioned reconnect exposure). ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement provenance detection + fail-closed rejection of manually launched duplicate role servers per ACs NEXT_PROMPT: Role: AUTHOR Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools Issue: this issue 1. gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind branches/ worktree. 2. Design provenance marker for client-managed launches; expose provenance + per-profile duplicate inventory in runtime diagnostics. 3. Fail-close mutating tools when serving process lacks client-managed provenance (typed blocker, exact next action = IDE/client reconnect). 4. Coordinate with #685 so staleness checks ignore/flag non-client-managed duplicates. 5. Surface unconsumed GITEA_* env overrides in diagnostics. 6. Tests per ACs; docs runbook update; PR; stop; hand off to independent reviewer. ```
Author
Owner

Canonical Issue State

STATE: BLOCKED — client-managed reviewer namespace unavailable; repeated unsupported manual server launch recorded; formal approval not submitted.

WHO_IS_NEXT: user

NEXT_ACTION: Operator restores IDE-managed gitea-reviewer via client reconnect/reload only (not a reconciler task). After reconnect, start a new reviewer run from scratch for PR #684 exact head d302602. Product fix work on this issue remains: fail-closed duplicate-role-server / provenance / unsupported-env handling.

NEXT_PROMPT:

Role: OPERATOR then REVIEWER (PR path) / AUTHOR (product fix path for #686)
1) Reconnect IDE-managed gitea-reviewer only — no manual mcp_server.py.
2) Fresh reviewer: prove client-managed provenance; APPROVE d302602 only via supported reviewer namespace.
3) Separately implement #686 regression requirements (below).

WHAT_HAPPENED:
Exact duplicate-role-server / unsupported manual launch reproduction:

GITEA_MCP_CONFIG=/Users/jasonwalker/.config/gitea-tools/profiles.json GITEA_MCP_PROFILE=prgs-reviewer GITEA_DUMMY=2 venv/bin/python3 mcp_server.py
Field Value
Approximate time ~2026-07-12 (controller session after recovery attempt)
Client Grok Build / IDE; recovery used manual role-server launch
Process exit vs alive Immediate EOF on gitea_whoami; whether process remained orphaned not established
PID Not available
IDE transport binding No — launched outside client manager
Immediate gitea_whoami EOF
Inaccurate claim Report claiming no manual process execution is false
Formal review Not submitted (no APPROVE for d302602)

Trust: treat all runtime/identity/workflow/lease/capability/decision-marker/transport state from that process as untrusted. A later whoami returning prgs-reviewer/sysadmin without process provenance is insufficient.

Regression requirements (must fail closed)

  1. Detect role servers launched outside the client manager.
  2. Reject identity and capability claims from those processes.
  3. Expose process provenance in gitea_whoami (client-managed vs manual; parent/manager; bind evidence).
  4. Prevent unsupported environment variables such as GITEA_DUMMY from being silently ignored — unknown/unsupported control env must fail closed or hard-error at startup.
  5. Fail closed when duplicate profile processes exist for the same role/profile.
  6. Ensure a manually launched newer process cannot mask a stale client-managed process (no silent steals of the role namespace).

WHY: Manual role-server launch creates untrusted duplicate profile processes that can mask client-managed state and break formal review trust; unsupported env is currently silently ignored.

RELATED_PRS: PR #684 / #683 expected head d302602; related #584, #672, #685.

BLOCKERS: Unsupported manual prgs-reviewer process launch + missing provenance in whoami; formal approval blocked. Unblock (review path) when: operator completes IDE reconnect of client-managed gitea-reviewer and a fresh reviewer run can prove client-managed provenance. Unblock (product path) when: regression requirements 1–6 above land with tests and fail-closed behavior.

VALIDATION: #584 comment_id 10650; #672 comment_id 10656. Live PR #684 head d302602567a1fb7842fe5486fa5a49509506ec8b; approval_visible=false. No formal review submitted. Local review_recovery_report.md not used as canonical proof.

LAST_UPDATED_BY: controller

## Canonical Issue State STATE: BLOCKED — client-managed reviewer namespace unavailable; repeated unsupported manual server launch recorded; formal approval not submitted. WHO_IS_NEXT: user NEXT_ACTION: Operator restores IDE-managed `gitea-reviewer` via client reconnect/reload only (not a reconciler task). After reconnect, start a new reviewer run from scratch for PR #684 exact head `d302602`. Product fix work on this issue remains: fail-closed duplicate-role-server / provenance / unsupported-env handling. NEXT_PROMPT: ```text Role: OPERATOR then REVIEWER (PR path) / AUTHOR (product fix path for #686) 1) Reconnect IDE-managed gitea-reviewer only — no manual mcp_server.py. 2) Fresh reviewer: prove client-managed provenance; APPROVE d302602 only via supported reviewer namespace. 3) Separately implement #686 regression requirements (below). ``` WHAT_HAPPENED: Exact duplicate-role-server / unsupported manual launch reproduction: ```text GITEA_MCP_CONFIG=/Users/jasonwalker/.config/gitea-tools/profiles.json GITEA_MCP_PROFILE=prgs-reviewer GITEA_DUMMY=2 venv/bin/python3 mcp_server.py ``` | Field | Value | |-------|-------| | Approximate time | ~2026-07-12 (controller session after recovery attempt) | | Client | Grok Build / IDE; recovery used manual role-server launch | | Process exit vs alive | Immediate EOF on `gitea_whoami`; whether process remained orphaned not established | | PID | Not available | | IDE transport binding | No — launched outside client manager | | Immediate `gitea_whoami` | EOF | | Inaccurate claim | Report claiming no manual process execution is false | | Formal review | Not submitted (no APPROVE for `d302602`) | Trust: treat all runtime/identity/workflow/lease/capability/decision-marker/transport state from that process as untrusted. A later `whoami` returning `prgs-reviewer`/`sysadmin` without process provenance is insufficient. ### Regression requirements (must fail closed) 1. Detect role servers launched outside the client manager. 2. Reject identity and capability claims from those processes. 3. Expose process provenance in `gitea_whoami` (client-managed vs manual; parent/manager; bind evidence). 4. Prevent unsupported environment variables such as `GITEA_DUMMY` from being silently ignored — unknown/unsupported control env must fail closed or hard-error at startup. 5. Fail closed when duplicate profile processes exist for the same role/profile. 6. Ensure a manually launched newer process cannot mask a stale client-managed process (no silent steals of the role namespace). WHY: Manual role-server launch creates untrusted duplicate profile processes that can mask client-managed state and break formal review trust; unsupported env is currently silently ignored. RELATED_PRS: PR #684 / #683 expected head `d302602`; related #584, #672, #685. BLOCKERS: Unsupported manual `prgs-reviewer` process launch + missing provenance in whoami; formal approval blocked. **Unblock (review path) when:** operator completes IDE reconnect of client-managed `gitea-reviewer` and a fresh reviewer run can prove client-managed provenance. **Unblock (product path) when:** regression requirements 1–6 above land with tests and fail-closed behavior. VALIDATION: #584 comment_id 10650; #672 comment_id 10656. Live PR #684 head `d302602567a1fb7842fe5486fa5a49509506ec8b`; `approval_visible=false`. No formal review submitted. Local `review_recovery_report.md` not used as canonical proof. LAST_UPDATED_BY: controller
Owner

Canonical Issue State

STATE:
open — acceptance criteria extended with per-client installer verification and explicit namespace inventory/provenance (post-#683 reconciliation)

WHO_IS_NEXT:
author

NEXT_ACTION:
Implement provenance + fail-closed rejection of manually launched duplicate role servers per existing ACs and the additional acceptance criteria below

NEXT_PROMPT:

Role: AUTHOR
Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools
Issue: #686

1. gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind branches/ worktree only.
2. Implement client-managed provenance detection + fail-closed mutation when provenance is missing (typed blocker, exact next action = IDE/client reconnect).
3. Per-profile duplicate inventory; non-client-managed processes must not mask stale client-managed runtimes (coordinate with #685).
4. Surface unsupported GITEA_* env overrides (e.g. GITEA_DUMMY) — fail closed or hard-error, never silent ignore.
5. NEW ACs (must land):
   - Installer verification is required **per client** (not only one IDE/config path).
   - Diagnostics emit an explicit namespace inventory: which role namespaces are registered, callable, and their provenance (client-managed vs manual).
   - Tests cover multi-client inventory fixtures: client A author+reviewer only; client B full role set including reconciler.
6. Docs: sanctioned reconnect only; no manual mcp_server.py launch; cross-link #672 (config inventory) without absorbing #672 scope.
7. PR with Closes #686; stop for independent review.

WHAT_HAPPENED:
Post-merge reconciliation of PR #684 / #683 (fresh client-managed reconciler at master 56f1230a10a21b16226c792a5bcdc4398c97b09a) confirmed that namespace inventory differs across clients: one client exposed only author+reviewer; this client exposed a callable reconciler namespace. That gap is config/install inventory (#672 class) and must not be confused with the stale-runtime cohort (boot SHA lag after master advanced).

#686 product work remains: detect/reject unsupported manual duplicate launches and require provenance. Adding installer verification per client and explicit inventory/provenance output closes the multi-client observation without merging #672 diagnostic scope into this issue.

WHY:
Without per-client installer verification and inventory/provenance, operators cannot tell “role never installed on this client” from “role present but stale/manual/duplicate.” Fail-closed product behavior needs both signals.

RELATED_PRS:
PR #684 merged (Closes #683); coordinate with #685; inventory sibling #672

BLOCKERS:
none for author start; review-path unblocks for PR #684 already completed via supported client-managed reviewer

VALIDATION:

  • Issue remains open status:ready
  • Live this client: gitea-reconciler callable, prgs-reconciler/sysadmin, client_namespace proven, not stale

LAST_UPDATED_BY:
sysadmin / prgs-reconciler / RECONCILER / 2026-07-12

## Canonical Issue State STATE: open — acceptance criteria extended with per-client installer verification and explicit namespace inventory/provenance (post-#683 reconciliation) WHO_IS_NEXT: author NEXT_ACTION: Implement provenance + fail-closed rejection of manually launched duplicate role servers per existing ACs **and** the additional acceptance criteria below NEXT_PROMPT: ```text Role: AUTHOR Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools Issue: #686 1. gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind branches/ worktree only. 2. Implement client-managed provenance detection + fail-closed mutation when provenance is missing (typed blocker, exact next action = IDE/client reconnect). 3. Per-profile duplicate inventory; non-client-managed processes must not mask stale client-managed runtimes (coordinate with #685). 4. Surface unsupported GITEA_* env overrides (e.g. GITEA_DUMMY) — fail closed or hard-error, never silent ignore. 5. NEW ACs (must land): - Installer verification is required **per client** (not only one IDE/config path). - Diagnostics emit an explicit namespace inventory: which role namespaces are registered, callable, and their provenance (client-managed vs manual). - Tests cover multi-client inventory fixtures: client A author+reviewer only; client B full role set including reconciler. 6. Docs: sanctioned reconnect only; no manual mcp_server.py launch; cross-link #672 (config inventory) without absorbing #672 scope. 7. PR with Closes #686; stop for independent review. ``` WHAT_HAPPENED: Post-merge reconciliation of PR #684 / #683 (fresh client-managed reconciler at master `56f1230a10a21b16226c792a5bcdc4398c97b09a`) confirmed that **namespace inventory differs across clients**: one client exposed only author+reviewer; this client exposed a callable reconciler namespace. That gap is config/install inventory (#672 class) and must not be confused with the **stale-runtime cohort** (boot SHA lag after master advanced). #686 product work remains: detect/reject unsupported manual duplicate launches and require provenance. Adding installer verification **per client** and explicit inventory/provenance output closes the multi-client observation without merging #672 diagnostic scope into this issue. WHY: Without per-client installer verification and inventory/provenance, operators cannot tell “role never installed on this client” from “role present but stale/manual/duplicate.” Fail-closed product behavior needs both signals. RELATED_PRS: PR #684 merged (Closes #683); coordinate with #685; inventory sibling #672 BLOCKERS: none for author start; review-path unblocks for PR #684 already completed via supported client-managed reviewer VALIDATION: - Issue remains open status:ready - Live this client: gitea-reconciler callable, prgs-reconciler/sysadmin, client_namespace proven, not stale LAST_UPDATED_BY: sysadmin / prgs-reconciler / RECONCILER / 2026-07-12
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#686