Canary: five concurrent LLM clients with a negative concurrency suite #963

New Issue

2026-07-28T02:56:38-05:00

jcwalker3 commented

2026-07-28 02:56:38 -05:00

Parent epic: #887.
Cross-link: #929, #797, #955 (certification umbrella).
Depends on: #948, #949, #890, #891, #896, #916.
Satisfied prerequisite: #953 / PR #954 (merged at aad5c8b42361d380a8eeb07b94b90815e594c2c5).

Goal supported: multi-LLM.

Problem

#948 AC24–25 run simulated clients inside pytest. #887 and #929 require "two different LLM clients" as epic completion gates — prose in an epic body, owned by no issue, testable by no command, and two is not five. #797 operates five real workers (Claude, Grok, Codex, AGY, Kimi K) with no concurrent-safety proof. Nothing tests repository crossover as a concurrency failure.

Current evidence

#948 AC24 says "multiple simulated clients".
#887's and #929's completion gates, verbatim, each name two clients.
#797's problem statement names the five workers.

Scope

A supervised canary running five real clients simultaneously against distinct issues, each acquiring its own worker_identity, lease, branch, and worktree, through author, reviewer, and merger roles.
A negative suite proving clients cannot cross repositories, reuse another's identity, steal or resume a fenced lease, or mutate the same target.

The implementation must not reference client names. Clients are configuration, and #948 forbids special-casing any of them.

Non-goals

Implementing the concurrency fixes themselves — #948 and the #887 children.
Replacing #948's unit-level tests.
Performance benchmarking.

Security and concurrency boundaries

Each client must be independently identified. Exclusivity applies to an active lease target, never to a role or a profile.

Acceptance criteria

Five clients complete concurrent end-to-end cycles with no identity reuse, crossover, lease theft, duplicate work, invalid review, or unsafe recovery.
Each client's worker_identity is unique and independent of role and profile.
Zero orphaned assignments, locks, branches, worktrees, PRs, or leases remain afterwards, verified natively.
The canary is parameterized by client configuration and contains no client-name literal.
The canary is a required gate before unattended scheduling.

Required positive tests

Five concurrent successful cycles.
Two clients on different repositories concurrently.

Required negative and adversarial tests

Two clients racing one issue produce exactly one winner.
A client attempting a mutation in a repository it is not bound to is refused.
A client presenting another's worker_identity is refused.
A fenced session resuming after takeover is refused.
A reviewer approving its own authored PR is refused.
A merge attempted at a head other than the approved one is refused.
A client killed mid-transaction: recovery leaves no orphan.

Deployment and migration requirements

Supervised and recorded, with per-client evidence naming client, role, issue or PR, and commit.

Rollback and recovery requirements

Every canary artifact is disposable and cleaned up through sanctioned paths.

Observability and audit requirements

Results are retained as certification evidence.

Definition of done

Green, with all negative cases refused.

Parent epic: #887. Cross-link: #929, #797, #955 (certification umbrella). Depends on: #948, #949, #890, #891, #896, #916. Satisfied prerequisite: #953 / PR #954 (merged at `aad5c8b42361d380a8eeb07b94b90815e594c2c5`). **Goal supported:** multi-LLM. ## Problem #948 AC24–25 run *simulated* clients inside pytest. #887 and #929 require "two different LLM clients" as **epic completion gates** — prose in an epic body, owned by no issue, testable by no command, and two is not five. #797 operates five real workers (Claude, Grok, Codex, AGY, Kimi K) with no concurrent-safety proof. Nothing tests repository crossover as a concurrency failure. ## Current evidence - #948 AC24 says "multiple simulated clients". - #887's and #929's completion gates, verbatim, each name two clients. - #797's problem statement names the five workers. ## Scope - A supervised canary running five real clients simultaneously against distinct issues, each acquiring its own `worker_identity`, lease, branch, and worktree, through author, reviewer, and merger roles. - A negative suite proving clients cannot cross repositories, reuse another's identity, steal or resume a fenced lease, or mutate the same target. **The implementation must not reference client names.** Clients are configuration, and #948 forbids special-casing any of them. ## Non-goals - Implementing the concurrency fixes themselves — #948 and the #887 children. - Replacing #948's unit-level tests. - Performance benchmarking. ## Security and concurrency boundaries Each client must be independently identified. Exclusivity applies to an active lease target, never to a role or a profile. ## Acceptance criteria - Five clients complete concurrent end-to-end cycles with no identity reuse, crossover, lease theft, duplicate work, invalid review, or unsafe recovery. - Each client's `worker_identity` is unique and independent of role and profile. - Zero orphaned assignments, locks, branches, worktrees, PRs, or leases remain afterwards, verified natively. - The canary is parameterized by client configuration and contains no client-name literal. - The canary is a required gate before unattended scheduling. ## Required positive tests - Five concurrent successful cycles. - Two clients on different repositories concurrently. ## Required negative and adversarial tests - Two clients racing one issue produce exactly one winner. - A client attempting a mutation in a repository it is not bound to is refused. - A client presenting another's `worker_identity` is refused. - A fenced session resuming after takeover is refused. - A reviewer approving its own authored PR is refused. - A merge attempted at a head other than the approved one is refused. - A client killed mid-transaction: recovery leaves no orphan. ## Deployment and migration requirements Supervised and recorded, with per-client evidence naming client, role, issue or PR, and commit. ## Rollback and recovery requirements Every canary artifact is disposable and cleaned up through sanctioned paths. ## Observability and audit requirements Results are retained as certification evidence. ## Definition of done Green, with all negative cases refused.

jcwalker3 added the testing integration type:guardrail status:blocked workflow-hardening anti-stomp concurrency safety labels 2026-07-28 02:56:39 -05:00

jcwalker3 referenced this issue

2026-07-28 02:57:23 -05:00

Gate: block unattended scheduling until the concurrency and exfiltration canaries both certify #964

jcwalker3 referenced this issue

2026-07-28 03:02:56 -05:00

Epic: Make author workflows transactional and safe for multiple LLM clients #887

jcwalker3 referenced this issue

2026-07-28 03:07:51 -05:00

Epic: certify safe multi-LLM execution through securely isolated remote MCP #955

jcwalker3 referenced this issue

2026-07-29 19:42:27 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #975

jcwalker3 referenced this issue

2026-07-29 20:08:51 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin referenced this issue

2026-07-29 20:37:15 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

jcwalker3 referenced this issue

2026-07-29 22:27:51 -05:00

feat(config): add six-client MCP configuration doctor and safe repair tool #977

jcwalker3 referenced this issue

2026-07-29 22:45:22 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin referenced this issue

2026-07-29 23:05:52 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

jcwalker3 commented

2026-07-30 00:29:42 -05:00

Canonical Issue State

STATE: open / status:blocked — operator scope clarified; configuration remediation in progress; canary not started
WHO_IS_NEXT: author
NEXT_ACTION: finish local MCP config remediation (Kimi kimi_k, Gemini CLI dedupe, Antigravity identity, Grok precedence, Codex reverify); then hand off four-client restart; do not start canary
NEXT_PROMPT:

Complete configuration remediation for #963 per operator scope, then hand off four-client restart (Codex, Gemini CLI, Grok, Kimi K). Do not launch canary. Do not disturb healthy Claude.

WHAT_HAPPENED: Operator scope decision recorded for #963 five-client concurrency canary.

Exact five-client fleet: Claude Code, Codex, Gemini CLI, Grok, Kimi K.

Identities (operator decisions):

Gemini CLI participant = Gemini CLI (not Antigravity/AGY); GITEA_MCP_CLIENT=gemini
Kimi K: GITEA_MCP_CLIENT=kimi_k
Antigravity: GITEA_MCP_CLIENT=antigravity (identity correction only; excluded from canary)

Excluded: Antigravity/AGY is not a sixth participant and is not the Gemini participant.

WHY: Canary parameterization and client identity must match operator decisions before any restart/canary; prior ambiguity between Gemini CLI and Antigravity/AGY blocked safe fleet prep.
RELATED_PRS: #976 merged (deployed master 6596b259fcc00d4a58668b4bae813a08e843d4b4); #975 closed; Claude Code already healthy on deployed revision.
BLOCKERS: status:blocked until all five clients are live on deployed revision and canary passes. Config remediation still required for Kimi identity, Gemini CLI duplication, Antigravity wrong identity, Grok precedence. Canary not started.
VALIDATION: gitea-author profile prgs-author, identity jcwalker3 verified; remote prgs; org/repo Scaled-Tech-Consulting/Gitea-Tools; gitea.issue.comment allowed; issue remains open with status:blocked; no canary launch; no label/state change.
LAST_UPDATED_BY: jcwalker3 via gitea-author (prgs-author) configuration-remediation session

## Canonical Issue State STATE: open / status:blocked — operator scope clarified; configuration remediation in progress; canary not started WHO_IS_NEXT: author NEXT_ACTION: finish local MCP config remediation (Kimi `kimi_k`, Gemini CLI dedupe, Antigravity identity, Grok precedence, Codex reverify); then hand off four-client restart; do not start canary NEXT_PROMPT: ```text Complete configuration remediation for #963 per operator scope, then hand off four-client restart (Codex, Gemini CLI, Grok, Kimi K). Do not launch canary. Do not disturb healthy Claude. ``` WHAT_HAPPENED: Operator scope decision recorded for #963 five-client concurrency canary. **Exact five-client fleet:** Claude Code, Codex, Gemini CLI, Grok, Kimi K. **Identities (operator decisions):** - Gemini CLI participant = Gemini CLI (not Antigravity/AGY); `GITEA_MCP_CLIENT=gemini` - Kimi K: `GITEA_MCP_CLIENT=kimi_k` - Antigravity: `GITEA_MCP_CLIENT=antigravity` (identity correction only; excluded from canary) **Excluded:** Antigravity/AGY is not a sixth participant and is not the Gemini participant. WHY: Canary parameterization and client identity must match operator decisions before any restart/canary; prior ambiguity between Gemini CLI and Antigravity/AGY blocked safe fleet prep. RELATED_PRS: #976 merged (deployed master `6596b259fcc00d4a58668b4bae813a08e843d4b4`); #975 closed; Claude Code already healthy on deployed revision. BLOCKERS: status:blocked until all five clients are live on deployed revision and canary passes. Config remediation still required for Kimi identity, Gemini CLI duplication, Antigravity wrong identity, Grok precedence. Canary not started. VALIDATION: gitea-author profile `prgs-author`, identity `jcwalker3` verified; remote prgs; org/repo Scaled-Tech-Consulting/Gitea-Tools; `gitea.issue.comment` allowed; issue remains open with status:blocked; no canary launch; no label/state change. LAST_UPDATED_BY: jcwalker3 via gitea-author (prgs-author) configuration-remediation session

jcwalker3 referenced this issue

2026-07-30 01:57:00 -05:00

feat(controller): expose native instance-level fleet identity and health snapshots #978

jcwalker3 referenced this issue

2026-07-30 02:16:21 -05:00

feat(controller): expose instance-level fleet identity and health snapshots (Closes #978) #979

sysadmin referenced this issue

2026-07-30 06:24:01 -05:00

feat(controller): expose instance-level fleet identity and health snapshots (Closes #978) #979

jcwalker3 referenced this issue

2026-07-31 10:14:52 -05:00

MCP Control Plane: charter-compliant autonomous task selection (umbrella) #989