fix(fleet): dedicated retirement capability, identity proof, external fencing
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.
B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.
Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.
B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.
New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.
B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.
Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.
Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.
Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
108cbfa173 from branches/baseline-980-108cbfa 28 failed, 6166 passed, 6
skipped, 1106 subtests. The failing sets are identical under comm, so zero
regressions and zero masked pre-existing failures; the +87 passing delta is
this branch's tests.
No live worker-registry row was retired — every test uses a throwaway SQLite
database. BAA, issue #981, PR #906, issue #650, review 624, unrelated branches
and worktrees, profiles, configuration, credentials, sessions, and running
processes were not touched.
Refs #980
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
+142
-9
@@ -19626,13 +19626,23 @@ def gitea_snapshot_instance_fleet(
|
||||
# --- #980 CAS-protected stale worker retirement ---------------------------
|
||||
|
||||
|
||||
#: #980 review 657 B1. Apply is a mutation and must not be authorized by the
|
||||
#: observational ``gitea.read`` class. Declared once here and consumed by both
|
||||
#: the entry gate and the immediately-pre-mutation re-check so the two can
|
||||
#: never drift apart; ``task_capability_map`` maps the apply tasks to the same
|
||||
#: literal.
|
||||
RETIREMENT_MUTATION_PERMISSION = "gitea.worker_registry.retire"
|
||||
|
||||
|
||||
def _retirement_role_block(task: str) -> dict | None:
|
||||
"""Refuse the retirement surface to any non-controller/reconciler role.
|
||||
|
||||
Mirrors ``gitea_snapshot_instance_fleet``: ``gitea.read`` is the operation
|
||||
gate (the mutation lands in the local worker registry, not in Gitea), and
|
||||
the role restriction is what actually keeps author, reviewer, and merger
|
||||
profiles out. No unrelated permission is granted to anyone.
|
||||
Role is a *secondary* control. The primary authority for apply is the
|
||||
dedicated ``gitea.worker_registry.retire`` capability (#980 review 657 B1);
|
||||
this check additionally pins the surface to the two roles that own fleet
|
||||
reconciliation, so a profile mistakenly granted the permission still cannot
|
||||
reach it from an author, reviewer, or merger role. Plan remains
|
||||
``gitea.read`` and is observational.
|
||||
"""
|
||||
profile = get_profile()
|
||||
role = _profile_role_kind(profile)
|
||||
@@ -19755,6 +19765,7 @@ def _retirement_protected_owners() -> tuple[dict | None, dict]:
|
||||
def _retirement_plan(records, canonical_repository: str, protected: dict) -> dict:
|
||||
"""The single planning path shared by dry run and in-transaction revalidation."""
|
||||
import mcp_fleet_retirement
|
||||
import mcp_process_fencing
|
||||
|
||||
return mcp_fleet_retirement.plan_stale_worker_retirement(
|
||||
records,
|
||||
@@ -19762,9 +19773,119 @@ def _retirement_plan(records, canonical_repository: str, protected: dict) -> dic
|
||||
canonical_repository=canonical_repository,
|
||||
protected_session_ids=(protected or {}).get("session_ids"),
|
||||
protected_pids=(protected or {}).get("pids"),
|
||||
# #980 review 657 B2: the fencing evidence that makes a recorded pid
|
||||
# interpretable. Probed live so a plan produced on one host can never
|
||||
# authorize a retirement carried out on another.
|
||||
current_host_id=mcp_process_fencing.current_host_id(),
|
||||
current_boot_id=mcp_process_fencing.current_boot_id(),
|
||||
start_time_probe=mcp_process_fencing.process_start_time,
|
||||
)
|
||||
|
||||
|
||||
def _retirement_external_fence() -> str:
|
||||
"""Version token over lease + liveness state consumed by a retirement (#980 B3).
|
||||
|
||||
Recomputed inside the retirement transaction immediately before every
|
||||
guarded write. Because neither the control-plane lease database nor the OS
|
||||
process table is covered by the registry's ``BEGIN IMMEDIATE``, this token
|
||||
is the only thing that can detect either of them moving mid-transaction.
|
||||
|
||||
A failure to read leases raises rather than returning a token: an
|
||||
unreadable external world is indistinguishable from a changed one, so it
|
||||
must abort the transaction rather than silently compare equal.
|
||||
"""
|
||||
import mcp_fleet_retirement
|
||||
|
||||
db, errs = _control_plane_db_or_error()
|
||||
if db is None:
|
||||
raise RuntimeError(
|
||||
"active workflow leases could not be read while fencing the "
|
||||
f"retirement transaction: {errs}"
|
||||
)
|
||||
leases = db.list_leases(statuses=["active"], limit=1000)
|
||||
return mcp_fleet_retirement.external_state_fingerprint(leases)
|
||||
|
||||
|
||||
def _retirement_liveness_reprobe(row: dict) -> dict:
|
||||
"""Immediate pre-write re-establishment that one specific process is gone.
|
||||
|
||||
#980 review 657 B3: revalidation happens once per transaction, but the
|
||||
per-target loop runs afterwards, so this re-probe closes the remaining
|
||||
window between "this row was judged safe" and "this row is written". It
|
||||
re-reads OS state rather than trusting the plan, and it compares the
|
||||
fencing triple so a pid number reused since the plan cannot pass.
|
||||
"""
|
||||
import mcp_fleet_retirement
|
||||
import mcp_process_fencing
|
||||
|
||||
pid = row.get("pid")
|
||||
try:
|
||||
alive = issue_lock_store.is_process_alive(pid)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"safe": False,
|
||||
"reason_code": mcp_fleet_retirement.REASON_PID_UNKNOWN,
|
||||
"detail": (
|
||||
f"pid {pid!r} could not be re-probed immediately before the "
|
||||
f"write: {_redact(str(exc))}"
|
||||
),
|
||||
}
|
||||
if alive:
|
||||
live_start = mcp_process_fencing.process_start_time(pid)
|
||||
recorded_start = (row.get("process_start_time") or "").strip()
|
||||
reused = bool(live_start) and live_start != recorded_start
|
||||
return {
|
||||
"safe": False,
|
||||
"reason_code": (
|
||||
mcp_fleet_retirement.REASON_PID_REUSED
|
||||
if reused
|
||||
else mcp_fleet_retirement.REASON_PID_ALIVE
|
||||
),
|
||||
"detail": (
|
||||
f"pid {pid!r} is alive at write time"
|
||||
+ (
|
||||
f" but is a different incarnation ({live_start!r} != "
|
||||
f"{recorded_start!r})"
|
||||
if reused
|
||||
else ""
|
||||
)
|
||||
),
|
||||
"evidence": {
|
||||
"pid": pid,
|
||||
"pid_alive": True,
|
||||
"live_process_start_time": live_start,
|
||||
"recorded_process_start_time": recorded_start,
|
||||
},
|
||||
}
|
||||
|
||||
host_now = mcp_process_fencing.current_host_id()
|
||||
boot_now = mcp_process_fencing.current_boot_id()
|
||||
if not host_now or (row.get("host_id") or "").strip() != host_now:
|
||||
return {
|
||||
"safe": False,
|
||||
"reason_code": mcp_fleet_retirement.REASON_HOST_UNPROVEN,
|
||||
"detail": (
|
||||
"host identity no longer agrees with the registration at write "
|
||||
f"time (row {row.get('host_id')!r}, now {host_now!r})"
|
||||
),
|
||||
}
|
||||
if not boot_now:
|
||||
return {
|
||||
"safe": False,
|
||||
"reason_code": mcp_fleet_retirement.REASON_BOOT_UNKNOWN,
|
||||
"detail": "boot identity became unobtainable before the write",
|
||||
}
|
||||
return {
|
||||
"safe": True,
|
||||
"evidence": {
|
||||
"pid": pid,
|
||||
"pid_alive": False,
|
||||
"host_id": host_now,
|
||||
"boot_id": boot_now,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _retirement_revalidation_plan(records, canonical_repository: str) -> dict:
|
||||
"""Revalidation planner used *inside* the retirement transaction.
|
||||
|
||||
@@ -19937,14 +20058,22 @@ def gitea_apply_stale_worker_retirement(
|
||||
|
||||
import mcp_fleet_retirement
|
||||
|
||||
read_block = _profile_operation_gate("gitea.read")
|
||||
if read_block:
|
||||
# #980 review 657 B1: the dedicated mutation capability, not gitea.read.
|
||||
# A profile holding only the observational permission cannot get past this
|
||||
# line, and the role check below is defence in depth rather than the sole
|
||||
# authority.
|
||||
mutation_block = _profile_operation_gate(RETIREMENT_MUTATION_PERMISSION)
|
||||
if mutation_block:
|
||||
return {
|
||||
"success": False,
|
||||
"mutation_performed": False,
|
||||
"retired_count": 0,
|
||||
"reasons": read_block,
|
||||
"permission_report": _permission_block_report("gitea.read"),
|
||||
"requested_task": "apply_stale_worker_retirement",
|
||||
"required_operation_permission": RETIREMENT_MUTATION_PERMISSION,
|
||||
"reasons": mutation_block,
|
||||
"permission_report": _permission_block_report(
|
||||
RETIREMENT_MUTATION_PERMISSION
|
||||
),
|
||||
}
|
||||
|
||||
role_block = _retirement_role_block("gitea_apply_stale_worker_retirement")
|
||||
@@ -20081,6 +20210,8 @@ def gitea_apply_stale_worker_retirement(
|
||||
plan_fn=lambda rows: _retirement_revalidation_plan(rows, canon),
|
||||
retired_by=acting,
|
||||
retirement_reason=mcp_fleet_retirement.REASON_ELIGIBLE,
|
||||
external_fence_fn=_retirement_external_fence,
|
||||
liveness_fn=_retirement_liveness_reprobe,
|
||||
)
|
||||
|
||||
result["role_kind"] = role_kind
|
||||
@@ -20089,7 +20220,9 @@ def gitea_apply_stale_worker_retirement(
|
||||
result["repository"] = {"org": org, "repo": repo, "canonical_repository": canon}
|
||||
result["protected_active_workflow_owners"] = protected
|
||||
result["permission_scope"] = {
|
||||
"granted_operations": ["gitea.read"],
|
||||
"granted_operations": [RETIREMENT_MUTATION_PERMISSION],
|
||||
"mutation_capability": RETIREMENT_MUTATION_PERMISSION,
|
||||
"plan_capability": "gitea.read",
|
||||
"control_plane_mutation": "worker_registrations.status -> retired",
|
||||
"denied_unrelated_mutations": True,
|
||||
"note": (
|
||||
|
||||
Reference in New Issue
Block a user