Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 103 Pull Requests 11 Actions Packages Projects Releases Wiki Activity

fix(fleet): dedicated retirement capability, identity proof, external fencing

Browse Source 
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.

B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.

Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.

B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.

New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.

B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.

Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.

Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.

Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
108cbfa173 from branches/baseline-980-108cbfa 28 failed, 6166 passed, 6
skipped, 1106 subtests. The failing sets are identical under comm, so zero
regressions and zero masked pre-existing failures; the +87 passing delta is
this branch's tests.

No live worker-registry row was retired — every test uses a throwaway SQLite
database. BAA, issue #981, PR #906, issue #650, review 624, unrelated branches
and worktrees, profiles, configuration, credentials, sessions, and running
processes were not touched.

Refs #980
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
sysadmin
2026-07-30 18:01:20 -04:00
parent c0c6d14b73
commit e344e68a13
7 changed files with 1783 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp_worker_identity.py
+153 -6
View File
@@ -322,6 +322,14 @@ _SCHEMA_OPTIONAL_COLUMNS: tuple[tuple[str, str], ...] = (
("parity_revision", "TEXT"),
("live_revision", "TEXT"),
("instance_id_provenance", "TEXT"),
# #980 review 657 B2/B3: fencing evidence that turns a bare pid into a
# statement about one specific process. A registration written before these
# columns existed carries NULL and can never satisfy the retirement identity
# proof, which is the intended fail-closed outcome — absence of evidence is
# not evidence of staleness.
("host_id", "TEXT"),
("boot_id", "TEXT"),
("process_start_time", "TEXT"),
# #980 retirement bookkeeping. Deliberately outside the CAS fingerprint
# field set: they record *that* a retirement happened, and are written only
# by the retirement transaction itself.
@@ -331,10 +339,36 @@ _SCHEMA_OPTIONAL_COLUMNS: tuple[tuple[str, str], ...] = (
)
def _default_process_fencing(pid: int | None) -> dict[str, str | None]:
"""Fencing triple for *pid*, degrading to unknowns rather than raising.
Imported lazily so this storage module keeps no import-time dependency on
the probe layer, and so a platform where the probes are unavailable still
registers workers — it simply records no fencing evidence, and those rows
are then permanently ineligible for retirement.
"""
try:
import mcp_process_fencing
return mcp_process_fencing.current_process_fencing(pid)
except Exception:
return {"host_id": None, "boot_id": None, "process_start_time": None}
class WorkerRegistryError(RuntimeError):
"""Raised for registry misuse that is a programming error, not a refusal."""
class _ExternalStateMoved(RuntimeError):
"""External safety evidence changed inside the retirement transaction.
Raised so the surrounding ``with self._tx()`` rolls back: once lease state
or process liveness has moved, every remaining per-row decision was computed
against a world that no longer exists, so the whole attempt is abandoned
rather than partially applied.
"""
def _utc_now() -> datetime:
return datetime.now(timezone.utc)
@@ -804,6 +838,9 @@ class WorkerRegistry:
parity_revision: str | None = None,
live_revision: str | None = None,
instance_id_provenance: str | None = None,
host_id: str | None = None,
boot_id: str | None = None,
process_start_time: str | None = None,
) -> dict[str, Any]:
"""Atomically register one worker identity.
@@ -836,6 +873,20 @@ class WorkerRegistry:
proc_id = process_identity or (
f"pid-{int(pid)}" if pid is not None else None
)
# #980 B2/B3: capture the fencing triple for the pid being registered.
# A caller may supply it (tests, or a launcher that already probed);
# otherwise it is probed here, at the only moment the process is known
# to be the one that owns this registration. Any probe that cannot
# answer stores NULL, which permanently withholds retirement eligibility
# from the row rather than granting it on absent evidence.
fencing = _default_process_fencing(pid)
host_id = host_id if host_id is not None else fencing["host_id"]
boot_id = boot_id if boot_id is not None else fencing["boot_id"]
process_start_time = (
process_start_time
if process_start_time is not None
else fencing["process_start_time"]
)
with self._tx() as conn:
existing = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
@@ -918,8 +969,9 @@ class WorkerRegistry:
fencing_epoch, status,
fleet_run_id, authenticated_account, process_identity,
startup_revision, loaded_revision, parity_revision,
live_revision, instance_id_provenance
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
live_revision, instance_id_provenance,
host_id, boot_id, process_start_time
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
""",
(
worker_identity,
@@ -948,6 +1000,9 @@ class WorkerRegistry:
(parity_revision or "").strip() or None,
(live_revision or "").strip() or None,
(instance_id_provenance or "").strip() or None,
(host_id or "").strip() or None,
(boot_id or "").strip() or None,
(process_start_time or "").strip() or None,
),
)
row = conn.execute(
@@ -1241,6 +1296,8 @@ class WorkerRegistry:
retired_by: str | None = None,
retirement_reason: str = "",
now: datetime | None = None,
external_fence_fn: Callable[[], str] | None = None,
liveness_fn: Callable[[dict[str, Any]], dict[str, Any]] | None = None,
) -> dict[str, Any]:
"""Compare-and-swap retirement of conclusively stale registrations (#980).
@@ -1265,6 +1322,33 @@ class WorkerRegistry:
back and is reported as ``transaction_failed`` with zero retirements
and ``mutation_performed`` false, so a partial write can never be
reported as success.
**External state (#980 review 657 B3).** ``BEGIN IMMEDIATE`` locks this
registry and nothing else, so two critical inputs live outside the
transaction's isolation domain: the workflow-lease table in a separate
control-plane database, and OS process liveness. Re-reading them once
during revalidation is not enough — the per-target loop takes time, so a
lease acquired (or a pid revived) after ``plan_fn`` returned but before
*this* row's ``UPDATE`` would go unnoticed, and the registry-column guard
cannot catch it because no registry column changed.
Two mechanisms close that gap, both applied per target and immediately
before its own write:
``external_fence_fn``
A version token over all external state the decision consumed. It is
captured inside the transaction before revalidation and re-read
before every guarded ``UPDATE``; any movement aborts the whole
transaction rather than retiring against evidence that has changed.
``liveness_fn``
A per-row re-probe of process liveness and fencing identity (host,
boot, start time). It runs immediately before the row's write and
must affirmatively re-establish that this exact process is gone.
Both default to ``None`` only so the storage layer stays independent of
the decision and control-plane layers; production always supplies them,
and a caller that omits ``liveness_fn`` gets no retirement at all rather
than an unfenced one.
"""
requested = [str(w) for w in (worker_identities or []) if str(w).strip()]
stamp = _ts(now or _utc_now())
@@ -1295,6 +1379,10 @@ class WorkerRegistry:
try:
with self._tx() as conn:
# Captured *after* BEGIN IMMEDIATE and *before* the
# authoritative read, so every later comparison is against the
# external state this decision was actually built on.
fence_at_plan = external_fence_fn() if external_fence_fn else None
rows = [
self._row_to_record(r)
for r in conn.execute(
@@ -1413,6 +1501,53 @@ class WorkerRegistry:
)
continue
# --- External-state fence, immediately before this write ---
#
# Lease state and OS liveness are outside this transaction,
# so they are re-checked here rather than trusted from
# revalidation. Movement aborts the whole transaction: a
# changed world invalidates every remaining decision, not
# only this row's.
if external_fence_fn is not None:
fence_now = external_fence_fn()
if fence_now != fence_at_plan:
raise _ExternalStateMoved(
"external safety state (workflow leases or "
"process liveness) changed inside the retirement "
"transaction; rolling back and retiring nothing"
)
if liveness_fn is None:
preserved.append(
{
"worker_identity": wid,
"reason_code": "liveness_reprobe_unavailable",
"detail": (
"no immediate pre-write liveness re-probe was "
"supplied; refusing to retire on revalidation "
"evidence alone (fail closed)"
),
}
)
continue
verdict = liveness_fn(dict(row))
if not verdict.get("safe"):
preserved.append(
{
"worker_identity": wid,
"reason_code": verdict.get("reason_code")
or "liveness_reprobe_refused",
"detail": verdict.get("detail")
or (
"immediate pre-write re-probe could not "
"re-establish that this process is gone"
),
"evidence": verdict.get("evidence"),
}
)
continue
cursor = conn.execute(
"UPDATE worker_registrations "
"SET status = ?, retired_at = ?, retired_by = ?, "
@@ -1423,7 +1558,10 @@ class WorkerRegistry:
" AND generation_id = ? "
" AND session_id = ? "
" AND fencing_epoch = ? "
" AND IFNULL(pid, -1) = IFNULL(?, -1)",
" AND IFNULL(pid, -1) = IFNULL(?, -1) "
" AND IFNULL(host_id, '') = IFNULL(?, '') "
" AND IFNULL(boot_id, '') = IFNULL(?, '') "
" AND IFNULL(process_start_time, '') = IFNULL(?, '')",
(
STATUS_RETIRED,
stamp,
@@ -1437,6 +1575,9 @@ class WorkerRegistry:
row.get("session_id"),
row.get("fencing_epoch"),
row.get("pid"),
row.get("host_id"),
row.get("boot_id"),
row.get("process_start_time"),
),
)
if cursor.rowcount == 1:
@@ -1475,17 +1616,23 @@ class WorkerRegistry:
current_candidate_fingerprint
)
result["retired_at"] = stamp if retired else None
result["external_fence"] = fence_at_plan
except Exception as exc: # rolled back by _tx; report, never half-claim
failure = _base("transaction_failed")
moved = isinstance(exc, _ExternalStateMoved)
failure = _base("external_state_moved" if moved else "transaction_failed")
failure["success"] = False
failure["reasons"] = [
"retirement transaction failed and was rolled back; zero "
str(exc)
if moved
else "retirement transaction failed and was rolled back; zero "
f"registrations were retired: {type(exc).__name__}: {exc}"
]
failure["preserved"] = [
{
"worker_identity": wid,
"reason_code": "transaction_failed",
"reason_code": (
"external_state_moved" if moved else "transaction_failed"
),
"detail": "transaction rolled back before any commit",
}
for wid in requested