Make MCP runtime generation ownership client/session-aware so multiple LLMs can work concurrently #948

New Issue

2026-07-27T03:44:40-05:00

jcwalker3 commented

2026-07-27 03:44:40 -05:00

Problem

The control plane currently treats legitimate MCP generations launched by different active LLM clients as globally conflicting duplicates. Claude, Codex, Gemini, Grok, Antigravity, or other clients cannot simultaneously use the control plane, even when they are working on different issues.

The global “exactly one MCP generation” mutation gate is the defect. Concurrency ownership should be scoped to authenticated client/session/generation identity, while issue ownership should be enforced independently through atomic leases, heartbeats, expiration, and fencing.

Required behavior

Current behavior: a second legitimate client generation triggers a fleet-wide stale-runtime mutation wall.
Desired behavior: multiple active LLM clients can use MCP concurrently.
Different clients may author different issues simultaneously.
Each issue is atomically locked to one author workflow without manual assignment.
Lock heartbeats renew ownership automatically.
When heartbeats cease, the lock expires after the configured timeout.
Takeover creates a new fencing epoch/token.
An expired or superseded session cannot mutate if it resumes.
The Gitea assignee field is visibility metadata, not ownership authority.
Reviewer identity remains independent from the author.
Merge still requires current-head approval and no unresolved change request.
Restart, reconnect, and stale-runtime diagnostics must be scoped to the affected client generation instead of globally blocking unrelated healthy clients.

Do not prematurely require either a shared fleet or one fleet per client. The implementation may choose the repository-appropriate architecture, but it must satisfy the behavioral contract.

Acceptance criteria

Every client connection has a stable, non-LLM-forgeable client_instance_id.
Every client session has a distinct session_id.
Every daemon launch has a generation_id or boot epoch.
Runtime ownership records include client, session, generation, profile/namespace, process, start time, liveness, and applicable repository binding.
Multiple healthy generations owned by different authenticated client sessions do not trigger a global mutation wall.
An unintended duplicate within the same ownership scope remains blocked or is resolved through an explicit fenced replacement protocol.
Abandoned generations are detected through authoritative liveness evidence, not PID comparison alone.
Stale or superseded generations cannot mutate after a replacement generation becomes authoritative.
work_issue atomically acquires an issue lease; no operator assignment step is required.
The lease records repository, issue, claimant identity, client, session, branch, registered worktree, lease epoch/fencing token, and expiration.
Automatic heartbeats renew only the owning lease.
Two clients racing for the same issue produce exactly one winner.
Different clients can mutate different leased issues concurrently.
After expiration, a new author may acquire the issue with a higher fencing epoch.
The prior session remains fenced if it reconnects or resumes.
All issue-scoped mutations validate the current fencing token immediately before mutation.
A stale generation affects only its own ownership scope unless authoritative evidence proves a wider safety problem.
Reviewer independence is checked by identity, not merely client or session ID.
Approval applies only to the exact reviewed PR head.
Any unresolved current-head REQUEST_CHANGES prevents merging.
Valid current-head approval permits the sanctioned LLM merger to complete the merge without returning it to the operator.
Migration from legacy runtime records fails closed without permanently preventing legitimate clients from reconnecting.
Diagnostics name the conflicting owner, affected scope, liveness evidence, recoverability, and exact sanctioned next action.
End-to-end tests run multiple simulated clients and their role namespaces concurrently.
Regression tests cover:
- four healthy LLM clients working on four different issues;
- two clients racing for one issue;
- heartbeat renewal;
- lease expiration and takeover;
- stale-session resurrection;
- same-owner duplicate generation;
- healthy generations owned by different clients;
- one stale generation alongside unrelated healthy generations;
- author/reviewer independence;
- stale approval after a head change;
- blocking change requests;
- successful sanctioned LLM merge.
worker_identity is independent of role and profile. Neither a role nor a profile is ever used as, or derived into, the unique identity of a worker.
Each LLM instance establishes a collision-resistant worker_identity.
The supported identity format carries the LLM name, a UTC timestamp, and a collision-resistant digest, for example <llm-name>-<UTC-timestamp>-<short-sha>.
The digest incorporates sufficient entropy — session identity, high-resolution time, and a random nonce, for example sha256(llm_name + session_id + timestamp_ns + random_nonce) — or an equivalently collision-resistant construction.
Worker registration is atomic.
Registering a worker_identity that is already active fails closed, without replacing, adopting, merging with, or corrupting the existing worker registration.
After a collision, the new worker generates a different identity and atomically registers that replacement.
A genuinely restarted worker receives a new worker_identity.
A restarted worker cannot silently inherit the previous worker's leases or fencing tokens.
Automated tests deliberately force an identity collision and verify rejection of the colliding registration, safe regeneration, and successful registration of the replacement identity.
worker_identity is unique.
Role is not unique and may be shared concurrently by multiple healthy workers.
Profile is not unique and may be shared concurrently by multiple healthy workers.
Exclusivity applies to an active lease target, never to a role or a profile.
Multiple healthy workers sharing the same role or profile do not trigger a profile-wide duplicate-process block, stale-runtime block, reconnect requirement, or mutation block.
Stale-runtime and reconnect failures are scoped only to the affected worker_identity or generation.
Issue #900 is reconciled: cohort lifecycle supervision detects and reaps abandoned or genuinely superseded instances without blocking or terminating legitimate, independently identified workers that share a profile.
The behavior introduced by issue #686 is reconciled: duplicate detection distinguishes reuse of the same worker_identity from valid concurrent workers holding different identities under the same role/profile.
Terminating all additional processes that share a role or profile is explicitly rejected as a valid implementation of this issue.

Identity invariants

Roles and profiles remain reusable capability definitions, not singleton worker identities:

UNIQUE: worker_identity
NOT UNIQUE: role
NOT UNIQUE: profile
EXCLUSIVE: active lease target

Criteria 26–44 and these invariants are recorded from scope-clarification comment 17651.

Non-goals and prohibited fixes

Do not require manual issue assignment.
Do not globally serialize all LLM clients.
Do not identify ownership using PID alone.
Do not trust an LLM-supplied identity without authenticated runtime binding.
Do not weaken issue locks, duplicate-work protection, approval rules, or merge gates.
Do not special-case Claude, Codex, Gemini, Grok, Antigravity, or particular issue numbers.
Do not use prompt-only workarounds.
Do not require operators to close unrelated healthy clients.
Do not use raw Git or direct API fallbacks.

## Problem The control plane currently treats legitimate MCP generations launched by different active LLM clients as globally conflicting duplicates. Claude, Codex, Gemini, Grok, Antigravity, or other clients cannot simultaneously use the control plane, even when they are working on different issues. The global “exactly one MCP generation” mutation gate is the defect. Concurrency ownership should be scoped to authenticated client/session/generation identity, while issue ownership should be enforced independently through atomic leases, heartbeats, expiration, and fencing. ## Required behavior * Current behavior: a second legitimate client generation triggers a fleet-wide stale-runtime mutation wall. * Desired behavior: multiple active LLM clients can use MCP concurrently. * Different clients may author different issues simultaneously. * Each issue is atomically locked to one author workflow without manual assignment. * Lock heartbeats renew ownership automatically. * When heartbeats cease, the lock expires after the configured timeout. * Takeover creates a new fencing epoch/token. * An expired or superseded session cannot mutate if it resumes. * The Gitea assignee field is visibility metadata, not ownership authority. * Reviewer identity remains independent from the author. * Merge still requires current-head approval and no unresolved change request. * Restart, reconnect, and stale-runtime diagnostics must be scoped to the affected client generation instead of globally blocking unrelated healthy clients. Do not prematurely require either a shared fleet or one fleet per client. The implementation may choose the repository-appropriate architecture, but it must satisfy the behavioral contract. ## Acceptance criteria 1. Every client connection has a stable, non-LLM-forgeable `client_instance_id`. 2. Every client session has a distinct `session_id`. 3. Every daemon launch has a `generation_id` or boot epoch. 4. Runtime ownership records include client, session, generation, profile/namespace, process, start time, liveness, and applicable repository binding. 5. Multiple healthy generations owned by different authenticated client sessions do not trigger a global mutation wall. 6. An unintended duplicate within the same ownership scope remains blocked or is resolved through an explicit fenced replacement protocol. 7. Abandoned generations are detected through authoritative liveness evidence, not PID comparison alone. 8. Stale or superseded generations cannot mutate after a replacement generation becomes authoritative. 9. `work_issue` atomically acquires an issue lease; no operator assignment step is required. 10. The lease records repository, issue, claimant identity, client, session, branch, registered worktree, lease epoch/fencing token, and expiration. 11. Automatic heartbeats renew only the owning lease. 12. Two clients racing for the same issue produce exactly one winner. 13. Different clients can mutate different leased issues concurrently. 14. After expiration, a new author may acquire the issue with a higher fencing epoch. 15. The prior session remains fenced if it reconnects or resumes. 16. All issue-scoped mutations validate the current fencing token immediately before mutation. 17. A stale generation affects only its own ownership scope unless authoritative evidence proves a wider safety problem. 18. Reviewer independence is checked by identity, not merely client or session ID. 19. Approval applies only to the exact reviewed PR head. 20. Any unresolved current-head `REQUEST_CHANGES` prevents merging. 21. Valid current-head approval permits the sanctioned LLM merger to complete the merge without returning it to the operator. 22. Migration from legacy runtime records fails closed without permanently preventing legitimate clients from reconnecting. 23. Diagnostics name the conflicting owner, affected scope, liveness evidence, recoverability, and exact sanctioned next action. 24. End-to-end tests run multiple simulated clients and their role namespaces concurrently. 25. Regression tests cover: * four healthy LLM clients working on four different issues; * two clients racing for one issue; * heartbeat renewal; * lease expiration and takeover; * stale-session resurrection; * same-owner duplicate generation; * healthy generations owned by different clients; * one stale generation alongside unrelated healthy generations; * author/reviewer independence; * stale approval after a head change; * blocking change requests; * successful sanctioned LLM merge. 26. `worker_identity` is independent of role and profile. Neither a role nor a profile is ever used as, or derived into, the unique identity of a worker. 27. Each LLM instance establishes a collision-resistant `worker_identity`. 28. The supported identity format carries the LLM name, a UTC timestamp, and a collision-resistant digest, for example `<llm-name>-<UTC-timestamp>-<short-sha>`. 29. The digest incorporates sufficient entropy — session identity, high-resolution time, and a random nonce, for example `sha256(llm_name + session_id + timestamp_ns + random_nonce)` — or an equivalently collision-resistant construction. 30. Worker registration is atomic. 31. Registering a `worker_identity` that is already active fails closed, without replacing, adopting, merging with, or corrupting the existing worker registration. 32. After a collision, the new worker generates a different identity and atomically registers that replacement. 33. A genuinely restarted worker receives a new `worker_identity`. 34. A restarted worker cannot silently inherit the previous worker's leases or fencing tokens. 35. Automated tests deliberately force an identity collision and verify rejection of the colliding registration, safe regeneration, and successful registration of the replacement identity. 36. `worker_identity` is unique. 37. Role is not unique and may be shared concurrently by multiple healthy workers. 38. Profile is not unique and may be shared concurrently by multiple healthy workers. 39. Exclusivity applies to an active lease target, never to a role or a profile. 40. Multiple healthy workers sharing the same role or profile do not trigger a profile-wide duplicate-process block, stale-runtime block, reconnect requirement, or mutation block. 41. Stale-runtime and reconnect failures are scoped only to the affected `worker_identity` or generation. 42. Issue #900 is reconciled: cohort lifecycle supervision detects and reaps abandoned or genuinely superseded instances without blocking or terminating legitimate, independently identified workers that share a profile. 43. The behavior introduced by issue #686 is reconciled: duplicate detection distinguishes reuse of the same `worker_identity` from valid concurrent workers holding different identities under the same role/profile. 44. Terminating all additional processes that share a role or profile is explicitly rejected as a valid implementation of this issue. ### Identity invariants Roles and profiles remain reusable capability definitions, not singleton worker identities: ```text UNIQUE: worker_identity NOT UNIQUE: role NOT UNIQUE: profile EXCLUSIVE: active lease target ``` Criteria 26–44 and these invariants are recorded from scope-clarification comment 17651. ## Non-goals and prohibited fixes * Do not require manual issue assignment. * Do not globally serialize all LLM clients. * Do not identify ownership using PID alone. * Do not trust an LLM-supplied identity without authenticated runtime binding. * Do not weaken issue locks, duplicate-work protection, approval rules, or merge gates. * Do not special-case Claude, Codex, Gemini, Grok, Antigravity, or particular issue numbers. * Do not use prompt-only workarounds. * Do not require operators to close unrelated healthy clients. * Do not use raw Git or direct API fallbacks.

jcwalker3 added the type:feature status:ready labels 2026-07-27 03:44:41 -05:00

jcwalker3 commented

2026-07-27 16:07:01 -05:00

P0 scope clarification: worker identity collision handling

This issue is the top-priority concurrency blocker limiting how many independent LLM workers can operate simultaneously.

Please make the following requirements explicit in the implementation and acceptance criteria:

worker_identity must remain independent of role and profile.
Each LLM instance must establish a collision-resistant identity, for example:

<llm-name>-<UTC-timestamp>-<short-sha>
The digest should incorporate sufficient entropy, such as:

sha256(llm_name + session_id + timestamp_ns + random_nonce)
Worker registration must be atomic.
A registration collision must be detected and rejected without replacing, adopting, or corrupting the existing worker.
After a collision, the new worker must generate and atomically register a different identity.
A restarted worker must receive a new identity and must not silently inherit the previous worker's leases or fencing tokens.
Automated tests must deliberately force an identity collision and verify safe regeneration and registration.
Roles and profiles must remain reusable capability definitions:
- UNIQUE: worker_identity
- NOT UNIQUE: role
- NOT UNIQUE: profile
- EXCLUSIVE: active lease target
Multiple healthy workers sharing the same role/profile must never trigger a profile-wide duplicate-process or stale-runtime block.

Required issue reconciliation

Issue #900 appears to require one eligible runtime cohort per profile. That requirement conflicts with #948 if "cohort" or profile eligibility permits only one LLM worker.

Implementation of #948 must explicitly reconcile #900 so that lifecycle supervision may detect abandoned or superseded instances without terminating or blocking legitimate, independently identified workers sharing the same profile.

Issue #686 should also be revisited because its duplicate-role-server guardrail must distinguish:

accidental duplicate registration of the same worker identity; from
valid concurrent workers with different identities using the same role/profile.

Killing every additional process using a shared profile is not an acceptable resolution.

Canonical Issue State

STATE: ready
WHO_IS_NEXT: author
NEXT_ACTION: Amend the #948 acceptance criteria to add the ten worker-identity collision requirements above as explicit numbered criteria, add the forced-collision regression test to AC25, and record the #900 and #686 reconciliation decisions in the issue body before implementation starts.
NEXT_PROMPT:

Amend Gitea-Tools issue #948 acceptance criteria per the scope-clarification comment.

Repository: Scaled-Tech-Consulting/Gitea-Tools
Profile: prgs-author
Identity: jcwalker3
Target issue: #948

Add as explicit acceptance criteria:
1. worker_identity is independent of role and profile.
2. Identity format <llm-name>-<UTC-timestamp>-<short-sha>.
3. Digest entropy sha256(llm_name + session_id + timestamp_ns + random_nonce).
4. Worker registration is atomic.
5. A registration collision is detected and rejected without replacing, adopting, or corrupting the existing worker.
6. After a collision the new worker generates and atomically registers a different identity.
7. A restarted worker receives a new identity and inherits no prior leases or fencing tokens.
8. A regression test deliberately forces an identity collision and verifies safe regeneration.
9. UNIQUE worker_identity; NOT UNIQUE role; NOT UNIQUE profile; EXCLUSIVE active lease target.
10. Multiple healthy workers sharing a role/profile never trigger a profile-wide duplicate-process or stale-runtime block.

Also record in the issue body how #900 (one eligible cohort per profile) and #686
(duplicate-role-server guardrail) are reconciled with #948.

Do not implement code, create a branch, or open a PR in this task.

WHAT_HAPPENED: A duplicate-issue scan for a proposed new P0 issue titled "Support concurrent per-LLM workers independently of shared roles and profiles" found #948 already covers that architectural defect. No duplicate issue was created. This comment records the one requirement gap found during that comparison plus two cross-issue conflicts.
WHY: #948 AC1 requires a stable, non-LLM-forgeable client identity but never states what happens when registration collides. Without an explicit collision requirement an implementation may satisfy AC1 by adopting or overwriting an existing registration, which silently transfers ownership and defeats the lease and fencing guarantees in AC10, AC14, and AC16. #900 and #686 are named because both encode the one-worker-per-profile assumption that #948 exists to remove; leaving them unreconciled would let a correct #948 implementation be reverted by the guardrails that remain.
RELATED_PRS: none
BLOCKERS: none
VALIDATION: Live state re-read at comment time. #948 open, labels status:ready and type:feature, comment thread empty before this comment. Native duplicate scan over all open issues and 120 closed issues found no equivalent clarification. Related issues confirmed open: #900, #887, #937, #797. Related issues confirmed closed: #686, #613. Runtime parity verified in agreement at ed9414ebda with live_stale false, mutation_safe true, restart_required false. No code, branch, commit, or pull request was produced by this task.
LAST_UPDATED_BY: jcwalker3 (prgs-author)

## P0 scope clarification: worker identity collision handling This issue is the top-priority concurrency blocker limiting how many independent LLM workers can operate simultaneously. Please make the following requirements explicit in the implementation and acceptance criteria: 1. `worker_identity` must remain independent of role and profile. 2. Each LLM instance must establish a collision-resistant identity, for example: `<llm-name>-<UTC-timestamp>-<short-sha>` 3. The digest should incorporate sufficient entropy, such as: `sha256(llm_name + session_id + timestamp_ns + random_nonce)` 4. Worker registration must be atomic. 5. A registration collision must be detected and rejected without replacing, adopting, or corrupting the existing worker. 6. After a collision, the new worker must generate and atomically register a different identity. 7. A restarted worker must receive a new identity and must not silently inherit the previous worker's leases or fencing tokens. 8. Automated tests must deliberately force an identity collision and verify safe regeneration and registration. 9. Roles and profiles must remain reusable capability definitions: - `UNIQUE: worker_identity` - `NOT UNIQUE: role` - `NOT UNIQUE: profile` - `EXCLUSIVE: active lease target` 10. Multiple healthy workers sharing the same role/profile must never trigger a profile-wide duplicate-process or stale-runtime block. ## Required issue reconciliation Issue #900 appears to require one eligible runtime cohort per profile. That requirement conflicts with #948 if "cohort" or profile eligibility permits only one LLM worker. Implementation of #948 must explicitly reconcile #900 so that lifecycle supervision may detect abandoned or superseded instances without terminating or blocking legitimate, independently identified workers sharing the same profile. Issue #686 should also be revisited because its duplicate-role-server guardrail must distinguish: - accidental duplicate registration of the same worker identity; from - valid concurrent workers with different identities using the same role/profile. Killing every additional process using a shared profile is not an acceptable resolution. ## Canonical Issue State STATE: ready WHO_IS_NEXT: author NEXT_ACTION: Amend the #948 acceptance criteria to add the ten worker-identity collision requirements above as explicit numbered criteria, add the forced-collision regression test to AC25, and record the #900 and #686 reconciliation decisions in the issue body before implementation starts. NEXT_PROMPT: ```text Amend Gitea-Tools issue #948 acceptance criteria per the scope-clarification comment. Repository: Scaled-Tech-Consulting/Gitea-Tools Profile: prgs-author Identity: jcwalker3 Target issue: #948 Add as explicit acceptance criteria: 1. worker_identity is independent of role and profile. 2. Identity format <llm-name>-<UTC-timestamp>-<short-sha>. 3. Digest entropy sha256(llm_name + session_id + timestamp_ns + random_nonce). 4. Worker registration is atomic. 5. A registration collision is detected and rejected without replacing, adopting, or corrupting the existing worker. 6. After a collision the new worker generates and atomically registers a different identity. 7. A restarted worker receives a new identity and inherits no prior leases or fencing tokens. 8. A regression test deliberately forces an identity collision and verifies safe regeneration. 9. UNIQUE worker_identity; NOT UNIQUE role; NOT UNIQUE profile; EXCLUSIVE active lease target. 10. Multiple healthy workers sharing a role/profile never trigger a profile-wide duplicate-process or stale-runtime block. Also record in the issue body how #900 (one eligible cohort per profile) and #686 (duplicate-role-server guardrail) are reconciled with #948. Do not implement code, create a branch, or open a PR in this task. ``` WHAT_HAPPENED: A duplicate-issue scan for a proposed new P0 issue titled "Support concurrent per-LLM workers independently of shared roles and profiles" found #948 already covers that architectural defect. No duplicate issue was created. This comment records the one requirement gap found during that comparison plus two cross-issue conflicts. WHY: #948 AC1 requires a stable, non-LLM-forgeable client identity but never states what happens when registration collides. Without an explicit collision requirement an implementation may satisfy AC1 by adopting or overwriting an existing registration, which silently transfers ownership and defeats the lease and fencing guarantees in AC10, AC14, and AC16. #900 and #686 are named because both encode the one-worker-per-profile assumption that #948 exists to remove; leaving them unreconciled would let a correct #948 implementation be reverted by the guardrails that remain. RELATED_PRS: none BLOCKERS: none VALIDATION: Live state re-read at comment time. #948 open, labels status:ready and type:feature, comment thread empty before this comment. Native duplicate scan over all open issues and 120 closed issues found no equivalent clarification. Related issues confirmed open: #900, #887, #937, #797. Related issues confirmed closed: #686, #613. Runtime parity verified in agreement at ed9414ebda9034ca87b36a9fce1c1ff7f98090f6 with live_stale false, mutation_safe true, restart_required false. No code, branch, commit, or pull request was produced by this task. LAST_UPDATED_BY: jcwalker3 (prgs-author)

jcwalker3 referenced this issue

2026-07-27 21:04:33 -05:00

feat(control-plane): add authoritative native MCP fleet inventory and duplicate-process detection #949

jcwalker3 referenced this issue

2026-07-27 21:23:50 -05:00

feat(control-plane): persist authoritative fleet synchronization and restart receipts #951

jcwalker3 referenced this issue

2026-07-28 02:52:04 -05:00

Remote-MCP: deny-by-default server-side authorization over the full request tuple, read-only by default #957

jcwalker3 referenced this issue

2026-07-28 02:56:38 -05:00

Canary: five concurrent LLM clients with a negative concurrency suite #963