Canary: five concurrent LLM clients with a negative concurrency suite #963

New Issue

2026-07-28T02:56:38-05:00

jcwalker3 commented

2026-07-28 02:56:38 -05:00

Parent epic: #887.
Cross-link: #929, #797, #955 (certification umbrella).
Depends on: #948, #949, #890, #891, #896, #916.
Satisfied prerequisite: #953 / PR #954 (merged at aad5c8b42361d380a8eeb07b94b90815e594c2c5).

Goal supported: multi-LLM.

Problem

#948 AC24–25 run simulated clients inside pytest. #887 and #929 require "two different LLM clients" as epic completion gates — prose in an epic body, owned by no issue, testable by no command, and two is not five. #797 operates five real workers (Claude, Grok, Codex, AGY, Kimi K) with no concurrent-safety proof. Nothing tests repository crossover as a concurrency failure.

Current evidence

#948 AC24 says "multiple simulated clients".
#887's and #929's completion gates, verbatim, each name two clients.
#797's problem statement names the five workers.

Scope

A supervised canary running five real clients simultaneously against distinct issues, each acquiring its own worker_identity, lease, branch, and worktree, through author, reviewer, and merger roles.
A negative suite proving clients cannot cross repositories, reuse another's identity, steal or resume a fenced lease, or mutate the same target.

The implementation must not reference client names. Clients are configuration, and #948 forbids special-casing any of them.

Non-goals

Implementing the concurrency fixes themselves — #948 and the #887 children.
Replacing #948's unit-level tests.
Performance benchmarking.

Security and concurrency boundaries

Each client must be independently identified. Exclusivity applies to an active lease target, never to a role or a profile.

Acceptance criteria

Five clients complete concurrent end-to-end cycles with no identity reuse, crossover, lease theft, duplicate work, invalid review, or unsafe recovery.
Each client's worker_identity is unique and independent of role and profile.
Zero orphaned assignments, locks, branches, worktrees, PRs, or leases remain afterwards, verified natively.
The canary is parameterized by client configuration and contains no client-name literal.
The canary is a required gate before unattended scheduling.

Required positive tests

Five concurrent successful cycles.
Two clients on different repositories concurrently.

Required negative and adversarial tests

Two clients racing one issue produce exactly one winner.
A client attempting a mutation in a repository it is not bound to is refused.
A client presenting another's worker_identity is refused.
A fenced session resuming after takeover is refused.
A reviewer approving its own authored PR is refused.
A merge attempted at a head other than the approved one is refused.
A client killed mid-transaction: recovery leaves no orphan.

Deployment and migration requirements

Supervised and recorded, with per-client evidence naming client, role, issue or PR, and commit.

Rollback and recovery requirements

Every canary artifact is disposable and cleaned up through sanctioned paths.

Observability and audit requirements

Results are retained as certification evidence.

Definition of done

Green, with all negative cases refused.

Parent epic: #887. Cross-link: #929, #797, #955 (certification umbrella). Depends on: #948, #949, #890, #891, #896, #916. Satisfied prerequisite: #953 / PR #954 (merged at `aad5c8b42361d380a8eeb07b94b90815e594c2c5`). **Goal supported:** multi-LLM. ## Problem #948 AC24–25 run *simulated* clients inside pytest. #887 and #929 require "two different LLM clients" as **epic completion gates** — prose in an epic body, owned by no issue, testable by no command, and two is not five. #797 operates five real workers (Claude, Grok, Codex, AGY, Kimi K) with no concurrent-safety proof. Nothing tests repository crossover as a concurrency failure. ## Current evidence - #948 AC24 says "multiple simulated clients". - #887's and #929's completion gates, verbatim, each name two clients. - #797's problem statement names the five workers. ## Scope - A supervised canary running five real clients simultaneously against distinct issues, each acquiring its own `worker_identity`, lease, branch, and worktree, through author, reviewer, and merger roles. - A negative suite proving clients cannot cross repositories, reuse another's identity, steal or resume a fenced lease, or mutate the same target. **The implementation must not reference client names.** Clients are configuration, and #948 forbids special-casing any of them. ## Non-goals - Implementing the concurrency fixes themselves — #948 and the #887 children. - Replacing #948's unit-level tests. - Performance benchmarking. ## Security and concurrency boundaries Each client must be independently identified. Exclusivity applies to an active lease target, never to a role or a profile. ## Acceptance criteria - Five clients complete concurrent end-to-end cycles with no identity reuse, crossover, lease theft, duplicate work, invalid review, or unsafe recovery. - Each client's `worker_identity` is unique and independent of role and profile. - Zero orphaned assignments, locks, branches, worktrees, PRs, or leases remain afterwards, verified natively. - The canary is parameterized by client configuration and contains no client-name literal. - The canary is a required gate before unattended scheduling. ## Required positive tests - Five concurrent successful cycles. - Two clients on different repositories concurrently. ## Required negative and adversarial tests - Two clients racing one issue produce exactly one winner. - A client attempting a mutation in a repository it is not bound to is refused. - A client presenting another's `worker_identity` is refused. - A fenced session resuming after takeover is refused. - A reviewer approving its own authored PR is refused. - A merge attempted at a head other than the approved one is refused. - A client killed mid-transaction: recovery leaves no orphan. ## Deployment and migration requirements Supervised and recorded, with per-client evidence naming client, role, issue or PR, and commit. ## Rollback and recovery requirements Every canary artifact is disposable and cleaned up through sanctioned paths. ## Observability and audit requirements Results are retained as certification evidence. ## Definition of done Green, with all negative cases refused.