e344e68a1345cb21db06776eef839f863b81f3e5
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 sysadmin
|
e344e68a13 |
fix(fleet): dedicated retirement capability, identity proof, external fencing
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.
B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.
Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.
B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.
New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.
B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.
Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.
Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.
Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
|
||
 sysadminandClaude Opus 4.8
|
c0c6d14b73 |
feat(fleet): add CAS-protected stale worker retirement capability
Adds a sanctioned controller/reconciler capability that retires conclusively stale MCP worker-registry rows through a dry-run-first, compare-and-swap protected workflow (#980). The #978 fleet snapshot is read-only, and its registry_revision is seeded with snapshot_at at second precision, so a CAS gated on it can never pass. That token is left unchanged; retirement gets its own stable registry_fingerprint derived exclusively from canonical retirement-relevant registry content, plus a candidate_fingerprint pinning the approved set. Identical contents observed at different times produce the same token; row order never affects it; any retirement-relevant change moves it. WorkerRegistry.retire_stale_workers performs the whole decision inside one BEGIN IMMEDIATE transaction: re-read rows, recompute the fingerprint from those rows, recompute the eligibility plan from those rows (re-reading active workflow leases), compare the candidate fingerprint, revalidate every target, then retire each survivor with a guarded UPDATE asserting its status, heartbeat, generation, session, fencing epoch, and pid are unchanged. Drift retires zero workers and reports registry_revision_moved or candidate_set_moved; any exception rolls back and reports transaction_failed, so a partial write is never reported as success. Retirement requires a full conjunction: active status, complete registry fields, parsable heartbeat, not live, pid_alive false, expired heartbeat, stale ownership, canonical repository binding, no identity evidence shared with a live or unprobeable worker, and no active workflow-lease ownership. Everything else is preserved with a structured reason code. Retired rows become historical rather than stale and keep their history; nothing is deleted. Retirement does not repair untrusted live identity: live workers on legacy instance identities are preserved and keep their legacy_incomplete_identity blockers, so the result never claims the fleet became safe. Exposed as gitea_plan_stale_worker_retirement and gitea_apply_stale_worker_retirement, restricted to controller/reconciler role kinds. The Gitea operation gate stays gitea.read because the mutation lands in the local control-plane registry, matching the #601 lease lifecycle; no new Gitea write permission is introduced and no author permission is broadened. Tests: tests/test_issue_980_stale_worker_retirement.py (40 passed, 23 subtests), including the regression test proving the old snapshot_at derivation moved the token one second apart while the new registry CAS token does not. Full suite from the branch worktree matches the master baseline exactly: 28 failed / 6206 passed vs 28 failed / 6166 passed, identical failure set. Closes #980 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
sysadminandClaude Opus 4.8
|
4a1cc63e94 |
feat(controller): expose instance-level fleet identity and health snapshots
Add a pure fleet snapshot assessor and a read-only controller/reconciler MCP tool so multi-instance fleets can be enumerated by client_instance_id without treating shared client_type as duplication. Trusted launcher instance IDs, classification (missing/unmanifested/collisions/historical), registry schema extensions, docs, and regression tests for #978. Closes #978 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
 jcwalker3andClaude Opus 4.8
|
1a38ef95e3 |
fix(runtime): recognize client identity environment and refresh worker registrations
Two defects left behind by #948 made sanctioned multi-client operation
impossible. They are inseparable: fixing either alone still leaves the
multi-client canary unable to run.
1. Client-identity environment keys were not recognized.
gitea_mcp_server reads GITEA_MCP_CLIENT, GITEA_MCP_CLIENT_INSTANCE and
GITEA_MCP_CLIENT_SESSION as the authoritative client-identity inputs for
worker registration, but none of the three appeared in
RECOGNIZED_GITEA_ENV_KEYS or matched a recognized prefix. The runtime
diagnostic scans every peer mcp_server.py process environment and classifies
any unlisted GITEA_* key as an unsupported override, which is raised as a
runtime blocker, so gitea_resolve_task_capability returned
blocker_kind=runtime_reconnect_required with stop_required=true. Because that
resolver is the mandatory preflight for every author, reviewer and merger
mutation, setting the very variable #948 requires closed the mutation gate
for the whole fleet, and reconnecting could not clear it: the variable is
re-exported from the client's server definition on every launch.
The three keys are now named individually in the recognized-key set. No
prefix is added, so an unrecognized GITEA_* override is still refused
exactly as before.
A related inconsistency in the same path is also fixed. The diagnostic
reasons are raised as one RuntimeError, but the preflight re-raise
recognized only "stale-runtime:", so an "unsupported-env:" reason was
silently swallowed there while still failing the resolver. Both reason
families now live in RUNTIME_DIAGNOSTIC_HARD_PREFIXES beside the function
that produces them, and both propagate identically. This only widens what is
refused, never what is permitted.
2. WorkerRegistry.heartbeat() had no production caller.
#948 delivered heartbeat() but only tests called it. The single production
writer registers once per process behind an attempted-once flag, and
register() stamps the same timestamp into both started_at and
last_heartbeat_at. Nothing advanced it afterwards: no lifespan hook, no
background task, no atexit handler in a process that blocks in mcp.run().
Since liveness is age against heartbeat_ttl_seconds, that TTL was not a
liveness window at all but a hard cap on how long any client could stay
attached; at 900 seconds a healthy, connected, client-managed process became
session_ownership=unowned with blocker_kind=session_attachment_missing.
WorkerHeartbeatSupervisor in mcp_worker_identity is the missing caller,
started from _active_worker_identity() at the moment register() succeeds,
because that is the only point where identity and fencing_epoch are both
known. It is a daemon thread rather than an asyncio task or a request-driven
refresh because renewal must survive an idle session, and because the
registry performs blocking BEGIN IMMEDIATE sqlite writes that must not run on
the server's event loop. daemon=True is deliberate: a hard kill takes the
thread with it, so a dead worker still goes stale on the normal TTL.
heartbeat_interval_for() returns one third of the TTL, hard-capped at one
half, so two consecutive beats can be lost without the row expiring and no
override can produce an interval that outlives the registration it renews.
heartbeat() gains optional keyword-only expectations (session, generation,
client name, pid); each supplied one must match the recorded row or the
renewal is refused with the existing BLOCKER_FENCED literal rather than a new
blocker_kind, since consumers switch on that value. Omitting them preserves
the pre-existing behavior exactly. Client names are compared normalized, so
several namespaces of one application stay one client while separate
applications stay distinct.
A terminal refusal stops the supervisor permanently and records why, so a
fenced session can never beat its way back into ownership. A transient
failure is counted and beating continues. An atexit hook stops it on orderly
shutdown. status() is surfaced read-only as worker_heartbeat on
gitea_get_runtime_context so a stopped heartbeat is diagnosable before the
TTL turns it into session_attachment_missing; it grants nothing.
claim_generation() still has no production caller. It bumps fencing_epoch,
which would fence the supervisor's cached epoch, and the strict refusal is
left in place deliberately: auto-re-adopting a bumped epoch would defeat
fencing.
No lock or lease TTL is changed, including the author issue-lock TTL, and no
mutation refused today becomes permitted.
Tests: tests/test_issue_975_client_identity_heartbeat.py adds 40 focused tests
covering all 13 acceptance criteria. Every TTL assertion uses an injected
clock; no test waits for a real TTL. The thread-loop tests use a
millisecond-scale interval with bounded polling.
Focused: 40 passed, 15 subtests passed.
Full suite from inside the branches worktree: 28 failed, 6105 passed, 6 skipped,
1105 subtests — an identical failure set to the
|
||
|
sysadminandClaude Opus 5
|
1dd30ecb15 |
feat(mcp): client/session-aware runtime ownership and provenance (#948)
Two surfaces reported different provenance for one process.
`gitea_get_runtime_context` read the live environment and reported
`client_managed`; `mcp_namespace_health.classify_namespace_probe` derived
provenance from `_safe_env_summary()`, whose `SAFE_ENV_KEYS` allowlist never
contained `GITEA_CLIENT_MANAGED`, `GITEA_MCP_CLIENT_MANAGED`, or
`GITEA_SERVER_PROVENANCE`. That lookup could only ever miss, so the health
surface was structurally incapable of returning anything but `manual_launch`.
Neither model could name which client or which session owned a runtime, so a
healthy daemon serving a second client was indistinguishable from a duplicate,
and the profile-wide duplicate scan walled the whole fleet.
Introduce `mcp_worker_identity` as the one authority, splitting two claims the
old code ran together:
* launch provenance — was this hand-launched from a terminal? Answered from the
environment, which is legitimate because the launcher sets it. Preserves the
#686 wall unchanged.
* session ownership — which live client session owns this runtime now? Answered
only from a live attachment record; no environment flag can establish it.
The module provides collision-resistant worker identities
(`<llm-name>-<UTC-timestamp>-<short-sha>`), an atomic SQLite registry with
fencing epochs, heartbeat-based liveness, generation takeover that supersedes
only a non-live claimant, cohort classification, and failure scoping.
Behaviour changes:
* Registering an existing worker identity fails closed; it is never replaced,
adopted, or merged with. The caller mints a different identity instead.
* A generation held by a live session cannot be claimed by a second one. A
generation whose claimant is not live is taken over with a higher fencing
epoch, so stale ownership cannot permanently strand a healthy daemon.
* A superseded session presenting an old epoch is refused and performs no write.
* Liveness comes from heartbeat freshness; a live PID cannot resurrect an
expired record, and a dead PID withdraws liveness.
* Workers sharing a role or profile no longer trigger a profile-wide duplicate
block, provided each carries a distinct identity. Processes with no identity
evidence remain classified as duplicates, so the #686 wall still holds.
* Runtime failures are scoped to a worker identity or generation, never to a
profile or the fleet.
* Reconnect guidance no longer defaults to Codex. An unidentified client gets
host-agnostic steps; `gitea_request_mcp_reconnect(client=...)` defaults to
resolving the client from the live attachment record.
* `resolve_bound_remote` keeps a bound namespace on its remote instead of
falling through to the `dadeschools` library default.
Absence of proof is now reported as `unproven` rather than asserted as
`manual_launch`. Both still fail closed — `is_client_managed` is unchanged, so
nothing previously refused is now permitted — but remediation names the proof
that is actually missing instead of describing a terminal launch it cannot
evidence. The #686 test is updated for that vocabulary and keeps every
wall-preserving assertion.
Threat-model anchors and their citations in docs/remote-mcp/threat-model.md are
restamped for the line movement in gitea_mcp_server.py.
Tests: tests/test_issue_948_client_session_provenance.py adds 43 cases covering
Codex/Gemini/Antigravity/Claude attachment, same-client new session, cross-client
takeover after a session ends, two live conflicting sessions, stale records,
missing attachment proof, environment flags without attachment, mixed
generations, duplicate cohorts, the hardcoded-client regression, explicit PRGS
selection, default-remote host drift, cross-surface agreement, and fail-closed
handling without false reconnect loops. Synthetic identifiers throughout.
Full suite from a branches/ worktree: 28F/5953P/6S at head vs 28F/5910P/6S at
merge base
|