Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 104 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

fix(fleet): dedicated retirement capability, identity proof, external fencing

Browse Source 
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.

B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.

Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.

B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.

New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.

B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.

Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.

Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.

Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
108cbfa173 from branches/baseline-980-108cbfa 28 failed, 6166 passed, 6
skipped, 1106 subtests. The failing sets are identical under comm, so zero
regressions and zero masked pre-existing failures; the +87 passing delta is
this branch's tests.

No live worker-registry row was retired — every test uses a throwaway SQLite
database. BAA, issue #981, PR #906, issue #650, review 624, unrelated branches
and worktrees, profiles, configuration, credentials, sessions, and running
processes were not touched.

Refs #980
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
sysadmin
2026-07-30 18:01:20 -04:00
parent c0c6d14b73
commit e344e68a13
7 changed files with 1783 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp_fleet_retirement.py
+297
View File
@@ -84,8 +84,43 @@ REASON_ROW_MISSING = "registration_missing"
REASON_UNPARSABLE_HEARTBEAT = "unparsable_heartbeat"
REASON_WORKER_LIVE = "worker_live"
# --- #980 review 657 B2: affirmative identity/liveness proof ---------------
#: The registration's instance identity is not launcher-minted (``inst-…``),
#: so nothing proves which application launch this row belongs to.
REASON_UNTRUSTED_PROVENANCE = "untrusted_identity_provenance"
#: The row does not record which host its pid belongs to, or records a
#: different host than the one probing. A local pid probe cannot speak for a
#: process on another machine.
REASON_HOST_UNPROVEN = "host_binding_unproven"
#: Boot identity is missing on the row or unobtainable here, so a recorded pid
#: cannot be compared against a live pid at all.
REASON_BOOT_UNKNOWN = "boot_identity_unknown"
#: The pid is alive but belongs to a different process incarnation than the one
#: registered — reported distinctly from a plain live worker.
REASON_PID_REUSED = "pid_reuse_detected"
#: Two active registrations claim one client instance within one namespace.
REASON_INSTANCE_CONFLICT = "client_instance_conflict"
#: The immediate pre-write re-probe could not re-establish death (#980 B3).
REASON_LIVENESS_REPROBE = "liveness_reprobe_refused"
#: Instance-identity prefix minted by the trusted launcher. Kept in sync with
#: ``mcp_fleet_snapshot._TRUSTED_INSTANCE_PREFIX`` through
#: :func:`mcp_fleet_snapshot.assess_instance_identity`, which stays the single
#: authority on what "trusted" means — this module never re-implements it.
TRUSTED_INSTANCE_PREFIX = "inst-"
#: Registry columns that must carry a usable value before the eligibility
#: conjunction can even be evaluated. Absence is ambiguity, not permission.
#:
#: #980 review 657 B2 added the fencing triple. Before it, "complete identity"
#: meant only that the pre-existing columns were non-null, which a legacy
#: ``legacy-pid-…`` row satisfies trivially — so a row that proved nothing about
#: *which* process it described was retireable. The triple is what makes a
#: recorded pid interpretable: which machine it ran on, which boot of that
#: machine, and which incarnation of that pid number. A registration written
#: before these columns existed carries NULL and is therefore preserved
#: permanently, which is the intended fail-closed outcome.
REQUIRED_IDENTITY_FIELDS: tuple[str, ...] = (
"worker_identity",
"client_instance_id",
@@ -96,6 +131,9 @@ REQUIRED_IDENTITY_FIELDS: tuple[str, ...] = (
"last_heartbeat_at",
"heartbeat_ttl_seconds",
"pid",
"host_id",
"boot_id",
"process_start_time",
)
#: Canonical retirement-relevant content. Ordering here is fixed and part of
@@ -128,6 +166,13 @@ FINGERPRINT_FIELDS: tuple[str, ...] = (
"fleet_run_id",
"authenticated_account",
"instance_id_provenance",
# #980 review 657 B3: fencing evidence is a retirement input, so moving it
# must move the CAS token. Without these, a row whose host, boot, or
# process incarnation changed would hash identically to the row the plan
# approved.
"host_id",
"boot_id",
"process_start_time",
)
_FINGERPRINT_VERSION = "registryfp-v1"
@@ -209,6 +254,48 @@ def _probe_pid(
return None if result is None else bool(result)
def _reuse_detected(
row: Mapping[str, Any],
live_start_time: str | None,
current_host_id: str | None,
current_boot_id: str | None,
) -> bool:
"""Is the pid occupied by a *different* incarnation than the one recorded?
Only meaningful when the recorded pid is comparable to the live one — same
machine, same boot. Across hosts or boots the number is unrelated by
construction and reuse is not the interesting question.
"""
recorded_host = (row.get("host_id") or "").strip()
recorded_boot = (row.get("boot_id") or "").strip()
if not current_host_id or recorded_host != current_host_id:
return False
if not current_boot_id or recorded_boot != current_boot_id:
return False
recorded_start = (row.get("process_start_time") or "").strip()
return bool(live_start_time) and live_start_time != recorded_start
def _instance_key(row: Mapping[str, Any]) -> tuple[str, str] | None:
"""The (instance, namespace) pair #978 requires to be unique among live rows."""
instance = _canon(row.get("client_instance_id"))
namespace = _canon(row.get("namespace"))
if not instance or not namespace:
return None
return (instance, namespace)
def _probe_start_time(
pid: Any, start_time_probe: Callable[[Any], str | None] | None
) -> str | None:
if start_time_probe is None or pid is None:
return None
try:
return start_time_probe(pid)
except Exception:
return None
def _evidence(
row: Mapping[str, Any],
snapshot_row: Mapping[str, Any],
@@ -229,6 +316,9 @@ def _evidence(
"process_identity": snapshot_row.get("process_identity"),
"pid": row.get("pid"),
"pid_alive": pid_alive,
"host_id": row.get("host_id"),
"boot_id": row.get("boot_id"),
"process_start_time": row.get("process_start_time"),
"repository_binding": row.get("repository_binding"),
"foreign_repository": bool(snapshot_row.get("foreign_repository")),
"status": row.get("status"),
@@ -262,6 +352,138 @@ def _conflict_keys(
return keys
def external_state_fingerprint(
leases: Iterable[Mapping[str, Any]],
*,
liveness: Iterable[tuple[Any, Any]] = (),
) -> str:
"""Version token over the external state a retirement decision consumed.
#980 review 657 B3: ``BEGIN IMMEDIATE`` on the worker registry does not
cover the control-plane lease table or the OS process table, so those
inputs can move while the transaction is open. This token lets the
transaction detect that movement: it is captured before the authoritative
read and re-compared immediately before every guarded write, and any
difference aborts rather than retiring against evidence that has changed.
Only ownership-relevant lease fields participate, so unrelated churn (a
heartbeat timestamp advancing on an unrelated lease) does not cause
spurious aborts, while an acquire, release, or owner change always does.
"""
lease_units: list[str] = []
for lease in leases:
lease_units.append(
_UNIT.join(
f"{name}={_canon(lease.get(name))}"
for name in (
"lease_id",
"role",
"target",
"status",
"session_id",
"owner_session_id",
"owner_pid",
"session_pid",
"generation",
)
)
)
for pid, alive in liveness:
lease_units.append(f"liveness{_UNIT}pid={_canon(pid)}{_UNIT}alive={_canon(alive)}")
return _digest("externalfp-v1", lease_units, "externalfp")
def assess_retirement_identity_proof(
row: Mapping[str, Any],
snapshot_row: Mapping[str, Any],
*,
current_host_id: str | None,
current_boot_id: str | None,
live_start_time: str | None,
pid_alive: bool | None,
) -> tuple[str, str] | None:
"""Affirmative proof that this row names one specific, now-dead process.
Returns ``None`` when the proof holds, or ``(reason_code, detail)`` naming
the first thing that could not be established. #980 review 657 B2: absence
of evidence is never read as staleness, so every branch here refuses on
*missing* information exactly as firmly as on contradictory information.
The proof has two independent halves and needs both:
* **Attribution** — a launcher-minted ``inst-…`` instance identity, so the
row is known to belong to one specific application launch rather than
having been inferred from pid proximity.
* **Fencing** — the row's host matches the host doing the probing, boot
identity is known on both sides, and the recorded process incarnation
agrees with whatever currently occupies that pid number.
Requiring trusted attribution means pre-#978 ``legacy-pid-…`` rows are
preserved permanently. That is deliberate. The reviewer specifically
rejected the argument that legacy rows "would remain forever" as grounds
for a weaker proof, and #980 lists backfilling trusted identity for legacy
workers as a non-goal — so those rows are retired only after their worker
re-registers under a trusted identity, never on weaker evidence.
"""
if not snapshot_row.get("instance_identity_trusted"):
return (
REASON_UNTRUSTED_PROVENANCE,
"client_instance_id "
f"{row.get('client_instance_id')!r} is not launcher-minted "
f"({snapshot_row.get('instance_id_provenance')!r}); nothing proves "
"which application launch this registration belongs to",
)
recorded_host = (row.get("host_id") or "").strip()
if not current_host_id:
return (
REASON_HOST_UNPROVEN,
"this process cannot establish its own host identity, so a local "
"pid probe cannot be attributed to any machine",
)
if recorded_host != current_host_id:
return (
REASON_HOST_UNPROVEN,
f"registration is bound to host {recorded_host!r} but retirement is "
f"running on {current_host_id!r}; a local pid probe says nothing "
"about a process on another machine",
)
recorded_boot = (row.get("boot_id") or "").strip()
if not current_boot_id:
return (
REASON_BOOT_UNKNOWN,
"the current boot identity could not be determined, so a recorded "
"pid cannot be compared against a live pid",
)
recorded_start = (row.get("process_start_time") or "").strip()
if recorded_boot != current_boot_id:
# A different boot is the strongest possible death evidence: every pid
# from a previous boot is gone, and pid numbers restart, so whatever
# occupies this number now is unrelated by construction.
return None
# Same boot: the pid number is directly comparable, so the recorded
# incarnation must still agree with whatever holds that number.
if pid_alive and live_start_time and live_start_time != recorded_start:
return (
REASON_PID_REUSED,
f"pid {row.get('pid')!r} is alive but started at "
f"{live_start_time!r}, not the registered {recorded_start!r}; the "
"number was reused by an unrelated process and this registration's "
"own liveness is therefore unproven",
)
if pid_alive:
return (
REASON_PID_ALIVE,
f"recorded pid {row.get('pid')!r} is still running on this host and "
"boot",
)
return None
def _missing_identity_fields(row: Mapping[str, Any]) -> list[str]:
missing: list[str] = []
for name in REQUIRED_IDENTITY_FIELDS:
@@ -280,6 +502,9 @@ def plan_stale_worker_retirement(
protected_worker_identities: Iterable[str] | None = None,
protected_session_ids: Iterable[str] | None = None,
protected_pids: Iterable[Any] | None = None,
current_host_id: str | None = None,
current_boot_id: str | None = None,
start_time_probe: Callable[[Any], str | None] | None = None,
) -> dict[str, Any]:
"""Decide, without mutating anything, which registrations may be retired.
@@ -294,9 +519,13 @@ def plan_stale_worker_retirement(
snapshots: dict[int, dict[str, Any]] = {}
pid_alive_by_index: dict[int, bool | None] = {}
start_time_by_index: dict[int, str | None] = {}
for index, row in enumerate(all_rows):
pid_alive = _probe_pid(row.get("pid"), pid_alive_probe)
pid_alive_by_index[index] = pid_alive
start_time_by_index[index] = _probe_start_time(
row.get("pid"), start_time_probe
)
snapshots[index] = fleet.build_worker_snapshot_row(
row,
now=now,
@@ -323,6 +552,37 @@ def plan_stale_worker_retirement(
):
ambiguous_keys.update(_conflict_keys(row, snapshot_row))
# Two active registrations claiming one (client_instance_id, namespace)
# violate the #978 uniqueness invariant — but only when one of them might
# still be running. Several *dead* rows accumulating on one slot across
# restarts is ordinary history and every one of them is safely retirable;
# a slot shared with a live or unprobeable worker is genuinely ambiguous,
# because which row that process belongs to cannot be settled from the
# registry alone.
#
# The key is deliberately the (instance, namespace) pair, not the instance
# alone: one legitimate cohort is exactly one instance spread across
# distinct namespaces, so keying on the instance would make every cohort
# look self-conflicting and preserve the whole fleet forever.
instance_members: dict[tuple[str, str], list[int]] = {}
for index, row in enumerate(all_rows):
if str(row.get("status") or "") != mwi.STATUS_ACTIVE:
continue
key = _instance_key(row)
if key is None:
continue
instance_members.setdefault(key, []).append(index)
instance_conflicts: dict[tuple[str, str], bool] = {}
for key, members in instance_members.items():
if len(members) < 2:
continue
contested = any(
snapshots[i].get("live") or pid_alive_by_index[i] is None
for i in members
)
if contested:
instance_conflicts[key] = True
candidates: list[dict[str, Any]] = []
candidate_rows: list[Mapping[str, Any]] = []
preserved: list[dict[str, Any]] = []
@@ -378,6 +638,19 @@ def plan_stale_worker_retirement(
REASON_PID_UNKNOWN,
f"pid {row.get('pid')!r} could not be probed; liveness is unproven",
)
elif pid_alive and _reuse_detected(
row, start_time_by_index[index], current_host_id, current_boot_id
):
# Reported before the generic live-pid branch so the operator sees
# *why* the number is occupied: an unrelated process inherited it,
# which means this registration's own liveness is unproven rather
# than positively established.
blocked = (
REASON_PID_REUSED,
f"pid {row.get('pid')!r} is alive but started at "
f"{start_time_by_index[index]!r}, not the registered "
f"{row.get('process_start_time')!r}; the number was reused",
)
elif pid_alive:
blocked = (
REASON_PID_ALIVE,
@@ -412,6 +685,25 @@ def plan_stale_worker_retirement(
"worker still owns active workflow state requiring separate "
f"reconciliation: {protected_hits}",
)
elif instance_conflicts.get(_instance_key(row)):
blocked = (
REASON_INSTANCE_CONFLICT,
"another active registration claims client_instance_id "
f"{row.get('client_instance_id')!r} in namespace "
f"{row.get('namespace')!r}; instance ownership is ambiguous",
)
else:
# Affirmative identity + fencing proof runs last: everything above
# establishes the row is *inert*, and this establishes it is
# unambiguously *this* worker (#980 review 657 B2).
blocked = assess_retirement_identity_proof(
row,
snapshot_row,
current_host_id=current_host_id,
current_boot_id=current_boot_id,
live_start_time=start_time_by_index[index],
pid_alive=pid_alive,
)
if blocked is not None:
preserved.append(
@@ -462,6 +754,11 @@ def plan_stale_worker_retirement(
"pids": sorted(protected_pid_set),
},
"canonical_repository": canonical_repository,
"fencing_context": {
"current_host_id": current_host_id,
"current_boot_id": current_boot_id,
"start_time_probe_available": start_time_probe is not None,
},
}