fix(runtime): recognize client identity environment and refresh worker registrations #975

New Issue

2026-07-29T19:42:27-05:00

jcwalker3 commented

2026-07-29 19:42:27 -05:00

Related: #963 (blocked by this), #948 (closed, introduced the incomplete lifecycle), #949, #969, #887, #929

Problem

Two defects in the #948 client/session ownership lifecycle make sanctioned
multi-client operation impossible. They are inseparable: fixing either alone
still leaves the four-client canary unable to run, and the second cannot even be
authored while the first stands.

1. Production client-identity environment keys are not recognized

gitea_mcp_server.py reads three environment variables as the authoritative
client-identity inputs for worker registration:

CLIENT_NAME_ENV     = "GITEA_MCP_CLIENT"
CLIENT_INSTANCE_ENV = "GITEA_MCP_CLIENT_INSTANCE"
CLIENT_SESSION_ENV  = "GITEA_MCP_CLIENT_SESSION"

None of the three is present in gitea_config.RECOGNIZED_GITEA_ENV_KEYS, and
none matches RECOGNIZED_GITEA_ENV_PREFIXES. The runtime diagnostic scans every
peer mcp_server.py process environment and classifies any unlisted GITEA_*
key as an unsupported override. That reason is then raised as a runtime blocker,
so gitea_resolve_task_capability returns:

blocker_kind:     runtime_reconnect_required
restart_required: true
stop_required:    true
reason:           "unsupported-env: Unsupported GITEA_* environment variable
                   override(s) detected: GITEA_MCP_CLIENT=<client>.
                   Unknown env overrides are unsupported."

The capability resolver is the mandatory preflight for every author, reviewer,
and merger mutation, so setting the very variable #948 requires for client
identity closes the mutation gate for the whole fleet. The refusal is also
misdiagnosed as a staleness problem: reconnecting cannot clear it, because the
variable is re-exported from the client's server definition on every launch. It
was reached with the server in full parity (in_parity: true, stale: false,
mutation_safe: true).

A second, smaller inconsistency in the same path: the diagnostic reasons are
raised as a RuntimeError that is re-raised only when the message contains
"stale-runtime:", so an unsupported-env: reason is silently swallowed on that
branch while still failing the resolver.

2. `WorkerRegistry.heartbeat()` has no production caller

WorkerRegistry.heartbeat() was delivered by #948 but is called only from
tests/test_issue_948_client_session_provenance.py. The single production writer
registers exactly once per process, behind a module-level attempted-once flag,
and register() stamps the same timestamp into both started_at and
last_heartbeat_at. Nothing ever advances the heartbeat afterwards: there is no
lifespan hook, no background task, and no atexit handler in the server process,
which blocks in mcp.run(...) until exit.

Liveness is computed as age = now - last_heartbeat_at against
heartbeat_ttl_seconds (900.0). So heartbeat_ttl_seconds is not a liveness
window at all in production — it is a hard cap on how long any client may remain
attached. At 900 seconds a completely healthy, connected, client-managed process
becomes:

session_ownership: unowned
blocker_kind:      session_attachment_missing

and can no longer perform sanctioned mutations.

Observed evidence

Read from a live, healthy, client-managed author namespace at control revision
324a0c8a8d76137b8466fe621b983c6ccf135be6:

worker_identity:       <client>-20260730T001758Z-<digest>
client_name:           <client>
pid:                   6111
fencing_epoch:         1
status:                active
started_at:            2026-07-30T00:17:58Z
last_heartbeat_at:     2026-07-30T00:17:58Z      <-- identical, never advances
heartbeat_ttl_seconds: 900.0
unconsumed_gitea_env:  { "GITEA_MCP_CLIENT": "<client>", ... }

The identical timestamps are the defect for every production worker row, not a
property of this one process.

Required behavior

Environment recognition

Add GITEA_MCP_CLIENT, GITEA_MCP_CLIENT_INSTANCE, and
GITEA_MCP_CLIENT_SESSION to the authoritative recognized-key set, so the
keys production consumes are the keys the gate accepts.
Do not broaden the gate: no new wildcard prefix, no arbitrary GITEA_*
acceptance. An unrecognized override must still be refused exactly as it is
today.

Heartbeat lifecycle

Registration happens once for the current worker identity; heartbeating must
not create a second registration or a second worker-identity system.
A production heartbeat runs periodically for as long as the server process
remains healthy, at an interval safely below heartbeat_ttl_seconds.
A heartbeat renews only the row matching the exact current client name, worker
identity, session id, generation id, and process identity. One worker can never
refresh another worker's row, and a superseded or fenced session can never
renew its way back into ownership.
Several namespaces belonging to one application stay distinguishable from
separate clients, and separate legitimate applications stay distinct.
Heartbeating stops on orderly shutdown.
A dead or disconnected worker still becomes stale on the configured TTL.
Heartbeat failures are observable through a sanctioned read-only surface and
fail safely: a failure never grants ownership and never crashes a tool call.
Provenance, parity, role, repository, and mutation gates keep their current
behavior. No mutation that is refused today becomes permitted.

Acceptance criteria

Each of the three client-identity variables is accepted by the recognized-key
set, individually asserted.
An unknown GITEA_* variable is still reported as an unsupported override.
Setting the client-identity variables no longer produces
blocker_kind=runtime_reconnect_required on a server that is otherwise in
parity.
A healthy registered worker remains session_ownership: owned beyond two
complete TTL periods of simulated time.
last_heartbeat_at advances while the worker stays healthy, and diverges from
started_at.
Repeated heartbeats create no duplicate registry rows for the same worker.
Two distinct client names remain distinct rows and distinct identities.
Two different sessions, and two different generations, cannot refresh each
other's rows.
Orderly shutdown stops the heartbeat.
A stopped or dead worker becomes unowned once the configured TTL elapses.
One worker expiring does not stale other healthy workers.
Existing session-ownership and stale-runtime behavior remains enforced.
Tests use controlled time; no test waits for a real TTL to elapse.

Non-goals

Running the #963 canary, or any part of its package.
Closing or modifying #963. This issue unblocks it and nothing more.
Retiring or reaping stale session rows — #969.
Fleet inventory and duplicate-process detection — #949.
Cohort lifecycle supervision, draining, and reaping — #900.
Client-side namespace attachment determinism — #689.
Changing any lock or lease TTL, including the author issue-lock TTL.
Any other runtime or concurrency hardening.

Duplicate verdict

NOT a duplicate. Verified by reading the bodies and acceptance criteria of #949,
#963, #969, #890, #900, #689, and #950, and the full open-issue list.

Issue	State	Why it does not cover this
#948	closed	Defined the ownership model and delivered `WorkerRegistry.heartbeat()` with no production caller. This issue is the regression that omission left behind; a closed issue cannot own the repair.
#963	open	Owns the multi-client acceptance gate and is blocked by this defect, but its non-goals explicitly exclude "implementing the concurrency fixes themselves".
#949	open	Strictly read-only fleet inventory; a periodic registry writer is a mutation and contradicts its stated scope and non-goals.
#969	open	Retires session rows whose owners are no longer live — the opposite direction, and its criteria require that live client-managed sessions are never retired. Already has an open PR.
#950	open	Aligns controller/reconciler role metadata across identity surfaces and restructures the capability-resolution outcome taxonomy. Neither recognized-env-key classification nor worker-registration freshness appears in its required behavior, acceptance criteria, or non-goals.
#890	open	Unifies competing author issue ownership stores (locks, leases, session pointer), not MCP worker-registry session ownership.
#900	open	Bounds the cohort population — one eligible cohort per profile, draining and reaping superseded ones. Not registration freshness.
#689	open	Client-side namespace attachment determinism. Already has an open PR.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Recognize the three client-identity environment keys and connect WorkerRegistry.heartbeat() to the production MCP server lifecycle, with focused tests
NEXT_PROMPT:
Role: AUTHOR
Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools
Issue: this issue

1. Native MCP only; gitea_whoami then gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/.
2. Add the three client-identity keys to the authoritative recognized-key set without broadening the gate.
3. Connect the worker heartbeat to the server process lifecycle: register once, refresh below the TTL, refresh only the exact current worker, stop on orderly shutdown, let dead workers expire.
4. Add focused deterministic tests using controlled time for every acceptance criterion.
5. Run the focused tests and the repository-required regression suite from inside the branches worktree.
6. Open one PR with Closes for this issue only; do not close #963; do not start the canary; hand off to an independent reviewer.

Related: #963 (blocked by this), #948 (closed, introduced the incomplete lifecycle), #949, #969, #887, #929 ## Problem Two defects in the #948 client/session ownership lifecycle make sanctioned multi-client operation impossible. They are inseparable: fixing either alone still leaves the four-client canary unable to run, and the second cannot even be authored while the first stands. ### 1. Production client-identity environment keys are not recognized `gitea_mcp_server.py` reads three environment variables as the authoritative client-identity inputs for worker registration: ``` CLIENT_NAME_ENV = "GITEA_MCP_CLIENT" CLIENT_INSTANCE_ENV = "GITEA_MCP_CLIENT_INSTANCE" CLIENT_SESSION_ENV = "GITEA_MCP_CLIENT_SESSION" ``` None of the three is present in `gitea_config.RECOGNIZED_GITEA_ENV_KEYS`, and none matches `RECOGNIZED_GITEA_ENV_PREFIXES`. The runtime diagnostic scans every peer `mcp_server.py` process environment and classifies any unlisted `GITEA_*` key as an unsupported override. That reason is then raised as a runtime blocker, so `gitea_resolve_task_capability` returns: ``` blocker_kind: runtime_reconnect_required restart_required: true stop_required: true reason: "unsupported-env: Unsupported GITEA_* environment variable override(s) detected: GITEA_MCP_CLIENT=<client>. Unknown env overrides are unsupported." ``` The capability resolver is the mandatory preflight for every author, reviewer, and merger mutation, so setting the very variable #948 requires for client identity closes the mutation gate for the whole fleet. The refusal is also misdiagnosed as a staleness problem: reconnecting cannot clear it, because the variable is re-exported from the client's server definition on every launch. It was reached with the server in full parity (`in_parity: true`, `stale: false`, `mutation_safe: true`). A second, smaller inconsistency in the same path: the diagnostic reasons are raised as a `RuntimeError` that is re-raised only when the message contains `"stale-runtime:"`, so an `unsupported-env:` reason is silently swallowed on that branch while still failing the resolver. ### 2. `WorkerRegistry.heartbeat()` has no production caller `WorkerRegistry.heartbeat()` was delivered by #948 but is called only from `tests/test_issue_948_client_session_provenance.py`. The single production writer registers exactly once per process, behind a module-level attempted-once flag, and `register()` stamps the same timestamp into both `started_at` and `last_heartbeat_at`. Nothing ever advances the heartbeat afterwards: there is no lifespan hook, no background task, and no `atexit` handler in the server process, which blocks in `mcp.run(...)` until exit. Liveness is computed as `age = now - last_heartbeat_at` against `heartbeat_ttl_seconds` (900.0). So `heartbeat_ttl_seconds` is not a liveness window at all in production — it is a hard cap on how long any client may remain attached. At 900 seconds a completely healthy, connected, client-managed process becomes: ``` session_ownership: unowned blocker_kind: session_attachment_missing ``` and can no longer perform sanctioned mutations. ## Observed evidence Read from a live, healthy, client-managed author namespace at control revision `324a0c8a8d76137b8466fe621b983c6ccf135be6`: ``` worker_identity: <client>-20260730T001758Z-<digest> client_name: <client> pid: 6111 fencing_epoch: 1 status: active started_at: 2026-07-30T00:17:58Z last_heartbeat_at: 2026-07-30T00:17:58Z <-- identical, never advances heartbeat_ttl_seconds: 900.0 unconsumed_gitea_env: { "GITEA_MCP_CLIENT": "<client>", ... } ``` The identical timestamps are the defect for every production worker row, not a property of this one process. ## Required behavior ### Environment recognition - Add `GITEA_MCP_CLIENT`, `GITEA_MCP_CLIENT_INSTANCE`, and `GITEA_MCP_CLIENT_SESSION` to the authoritative recognized-key set, so the keys production consumes are the keys the gate accepts. - Do not broaden the gate: no new wildcard prefix, no arbitrary `GITEA_*` acceptance. An unrecognized override must still be refused exactly as it is today. ### Heartbeat lifecycle - Registration happens once for the current worker identity; heartbeating must not create a second registration or a second worker-identity system. - A production heartbeat runs periodically for as long as the server process remains healthy, at an interval safely below `heartbeat_ttl_seconds`. - A heartbeat renews only the row matching the exact current client name, worker identity, session id, generation id, and process identity. One worker can never refresh another worker's row, and a superseded or fenced session can never renew its way back into ownership. - Several namespaces belonging to one application stay distinguishable from separate clients, and separate legitimate applications stay distinct. - Heartbeating stops on orderly shutdown. - A dead or disconnected worker still becomes stale on the configured TTL. - Heartbeat failures are observable through a sanctioned read-only surface and fail safely: a failure never grants ownership and never crashes a tool call. - Provenance, parity, role, repository, and mutation gates keep their current behavior. No mutation that is refused today becomes permitted. ## Acceptance criteria 1. Each of the three client-identity variables is accepted by the recognized-key set, individually asserted. 2. An unknown `GITEA_*` variable is still reported as an unsupported override. 3. Setting the client-identity variables no longer produces `blocker_kind=runtime_reconnect_required` on a server that is otherwise in parity. 4. A healthy registered worker remains `session_ownership: owned` beyond two complete TTL periods of simulated time. 5. `last_heartbeat_at` advances while the worker stays healthy, and diverges from `started_at`. 6. Repeated heartbeats create no duplicate registry rows for the same worker. 7. Two distinct client names remain distinct rows and distinct identities. 8. Two different sessions, and two different generations, cannot refresh each other's rows. 9. Orderly shutdown stops the heartbeat. 10. A stopped or dead worker becomes unowned once the configured TTL elapses. 11. One worker expiring does not stale other healthy workers. 12. Existing session-ownership and stale-runtime behavior remains enforced. 13. Tests use controlled time; no test waits for a real TTL to elapse. ## Non-goals - Running the #963 canary, or any part of its package. - Closing or modifying #963. This issue unblocks it and nothing more. - Retiring or reaping stale session rows — #969. - Fleet inventory and duplicate-process detection — #949. - Cohort lifecycle supervision, draining, and reaping — #900. - Client-side namespace attachment determinism — #689. - Changing any lock or lease TTL, including the author issue-lock TTL. - Any other runtime or concurrency hardening. ## Duplicate verdict NOT a duplicate. Verified by reading the bodies and acceptance criteria of #949, #963, #969, #890, #900, #689, and #950, and the full open-issue list. | Issue | State | Why it does not cover this | | --- | --- | --- | | #948 | closed | Defined the ownership model and delivered `WorkerRegistry.heartbeat()` with no production caller. This issue is the regression that omission left behind; a closed issue cannot own the repair. | | #963 | open | Owns the multi-client acceptance gate and is blocked by this defect, but its non-goals explicitly exclude "implementing the concurrency fixes themselves". | | #949 | open | Strictly read-only fleet inventory; a periodic registry writer is a mutation and contradicts its stated scope and non-goals. | | #969 | open | Retires session rows whose owners are no longer live — the opposite direction, and its criteria require that live client-managed sessions are never retired. Already has an open PR. | | #950 | open | Aligns controller/reconciler *role metadata* across identity surfaces and restructures the capability-resolution outcome taxonomy. Neither recognized-env-key classification nor worker-registration freshness appears in its required behavior, acceptance criteria, or non-goals. | | #890 | open | Unifies competing *author issue* ownership stores (locks, leases, session pointer), not MCP worker-registry session ownership. | | #900 | open | Bounds the cohort *population* — one eligible cohort per profile, draining and reaping superseded ones. Not registration freshness. | | #689 | open | Client-side namespace attachment determinism. Already has an open PR. | ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Recognize the three client-identity environment keys and connect WorkerRegistry.heartbeat() to the production MCP server lifecycle, with focused tests NEXT_PROMPT: Role: AUTHOR Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools Issue: this issue 1. Native MCP only; gitea_whoami then gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/. 2. Add the three client-identity keys to the authoritative recognized-key set without broadening the gate. 3. Connect the worker heartbeat to the server process lifecycle: register once, refresh below the TTL, refresh only the exact current worker, stop on orderly shutdown, let dead workers expire. 4. Add focused deterministic tests using controlled time for every acceptance criterion. 5. Run the focused tests and the repository-required regression suite from inside the branches worktree. 6. Open one PR with Closes for this issue only; do not close #963; do not start the canary; hand off to an independent reviewer. ```

jcwalker3 added the mcp status:ready workflow-hardening safety mcp-health control-plane type:bug labels 2026-07-29 19:42:28 -05:00

jcwalker3 referenced this issue from a commit

2026-07-29 20:06:13 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations

jcwalker3 referenced a pull request that will close this issue

2026-07-29 20:08:51 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

jcwalker3 added status:pr-open and removed status:ready labels 2026-07-29 20:08:52 -05:00

sysadmin referenced this issue

2026-07-29 20:18:05 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin referenced this issue

2026-07-29 20:31:38 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin referenced this issue

2026-07-29 20:37:43 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

jcwalker3 referenced this issue from a commit

2026-07-29 22:42:09 -05:00

fix(runtime): recognize heartbeat interval env and wire production lifecycle tests

jcwalker3 commented

2026-07-29 22:44:51 -05:00

Issue claim heartbeat

kind: progress
issue: fix(runtime): recognize client identity environment and refresh worker registrations (#975)
branch: fix/issue-975-client-identity-heartbeat
phase: ready-for-review
profile: prgs-author
pr: 976
blocker: none
next_action: Independent reviewer re-reviews at head b0868be6b3

**Issue claim heartbeat** - kind: progress - issue: #975 - branch: fix/issue-975-client-identity-heartbeat - phase: ready-for-review - profile: prgs-author - pr: 976 - blocker: none - next_action: Independent reviewer re-reviews at head b0868be6b34f03338dd9fb3c3eda65f42b0edca3

sysadmin referenced this issue

2026-07-29 23:05:52 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin referenced this issue

2026-07-29 23:25:13 -05:00

fix(runtime): recognize client identity environment and refresh worker registrations #976

sysadmin closed this issue

2026-07-29 23:26:34 -05:00

sysadmin removed the status:pr-open label 2026-07-29 23:26:40 -05:00

jcwalker3 referenced this issue

2026-07-30 00:29:42 -05:00

Canary: five concurrent LLM clients with a negative concurrency suite #963