fix(fleet): dedicated retirement capability, identity proof, external fencing
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.
B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.
Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.
B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.
New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.
B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.
Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.
Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.
Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
108cbfa173 from branches/baseline-980-108cbfa 28 failed, 6166 passed, 6
skipped, 1106 subtests. The failing sets are identical under comm, so zero
regressions and zero masked pre-existing failures; the +87 passing delta is
this branch's tests.
No live worker-registry row was retired — every test uses a throwaway SQLite
database. BAA, issue #981, PR #906, issue #650, review 624, unrelated branches
and worktrees, profiles, configuration, credentials, sessions, and running
processes were not touched.
Refs #980
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
@@ -0,0 +1,205 @@
|
||||
"""Host, boot, and process-start fencing evidence for retirement safety (#980).
|
||||
|
||||
Review 657 B2/B3 established that a local ``os.kill(pid, 0)`` probe is not, on
|
||||
its own, evidence that a *particular registered worker* is gone:
|
||||
|
||||
* **Host ambiguity.** A registry row written on host A records pid 1234. Probing
|
||||
pid 1234 on host B answers a question nobody asked. "Not running here" is not
|
||||
"not running".
|
||||
* **PID reuse.** Pid 1234 may be alive and belong to an unrelated process that
|
||||
the kernel handed the number to after the original exited. The naive probe
|
||||
reads that as "the worker is live" (safe, over-preserving) — but the converse
|
||||
matters too: evidence captured about pid 1234 at time T must not be honoured
|
||||
at time T+n if the process behind that number changed in between.
|
||||
* **Boot boundaries.** Every pid from a previous boot is conclusively gone, and
|
||||
pid numbers restart, so a recorded pid is only comparable to a live pid when
|
||||
both belong to the same boot.
|
||||
|
||||
This module supplies the three pieces of evidence that turn a bare pid into a
|
||||
statement about one specific process:
|
||||
|
||||
``host_id``
|
||||
Which machine the pid belongs to.
|
||||
``boot_id``
|
||||
Which boot of that machine the pid belongs to. Pids are only comparable
|
||||
within a single boot.
|
||||
``process_start_time``
|
||||
Which *incarnation* of that pid number. Two processes on the same host and
|
||||
boot sharing a pid number cannot share a start time, so comparing start
|
||||
times defeats reuse.
|
||||
|
||||
Every probe here is read-only, never raises, and returns ``None`` when the
|
||||
evidence cannot be established. ``None`` means *unknown*, and the retirement
|
||||
conjunction is required to treat unknown as "preserve", never as "safe".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import platform
|
||||
import subprocess
|
||||
|
||||
# --- Host identity --------------------------------------------------------
|
||||
|
||||
|
||||
def current_host_id() -> str | None:
|
||||
"""Stable identifier for the machine this process runs on.
|
||||
|
||||
Deliberately the kernel node name rather than anything network-derived: it
|
||||
does not change when an interface goes down or a VPN reassigns an address,
|
||||
and retirement must not become unsafe because DNS moved.
|
||||
"""
|
||||
try:
|
||||
node = (platform.node() or "").strip()
|
||||
except Exception:
|
||||
return None
|
||||
return node or None
|
||||
|
||||
|
||||
# --- Boot identity --------------------------------------------------------
|
||||
|
||||
|
||||
def _linux_boot_id() -> str | None:
|
||||
try:
|
||||
with open("/proc/sys/kernel/random/boot_id", encoding="utf-8") as handle:
|
||||
value = handle.read().strip()
|
||||
except Exception:
|
||||
return None
|
||||
return value or None
|
||||
|
||||
|
||||
def _darwin_boot_id() -> str | None:
|
||||
"""macOS boot identity, derived from ``kern.boottime``.
|
||||
|
||||
``sysctl`` prints e.g. ``{ sec = 1785400000, usec = 123456 } Wed Jul 30 ...``.
|
||||
Only the integer seconds are kept: the trailing human-readable date is
|
||||
locale-dependent and would make the identifier unstable across environments
|
||||
for the very same boot.
|
||||
"""
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["/usr/sbin/sysctl", "-n", "kern.boottime"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
text = (completed.stdout or "").strip()
|
||||
marker = "sec = "
|
||||
start = text.find(marker)
|
||||
if start < 0:
|
||||
return None
|
||||
tail = text[start + len(marker) :]
|
||||
digits = ""
|
||||
for char in tail:
|
||||
if char.isdigit():
|
||||
digits += char
|
||||
else:
|
||||
break
|
||||
return f"boot-{digits}" if digits else None
|
||||
|
||||
|
||||
def current_boot_id() -> str | None:
|
||||
"""Identifier for the current boot, or ``None`` when it cannot be proven."""
|
||||
system = ""
|
||||
try:
|
||||
system = (platform.system() or "").strip().lower()
|
||||
except Exception:
|
||||
system = ""
|
||||
if system == "linux":
|
||||
return _linux_boot_id()
|
||||
if system == "darwin":
|
||||
return _darwin_boot_id()
|
||||
# An unrecognised platform yields no boot evidence rather than a guess.
|
||||
return None
|
||||
|
||||
|
||||
# --- Process start time ---------------------------------------------------
|
||||
|
||||
|
||||
def _linux_process_start_time(pid: int) -> str | None:
|
||||
"""Field 22 of ``/proc/<pid>/stat`` — start time in clock ticks since boot.
|
||||
|
||||
The executable name in field 2 is parenthesised and may itself contain
|
||||
spaces and parentheses, so the fields are located from the *last* ``)``
|
||||
rather than by splitting the whole line.
|
||||
"""
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat", encoding="utf-8") as handle:
|
||||
raw = handle.read()
|
||||
except Exception:
|
||||
return None
|
||||
close = raw.rfind(")")
|
||||
if close < 0:
|
||||
return None
|
||||
fields = raw[close + 1 :].split()
|
||||
# After the ')' the next field is state (field 3), so field 22 is index 19.
|
||||
if len(fields) < 20:
|
||||
return None
|
||||
value = fields[19].strip()
|
||||
return f"ticks-{value}" if value else None
|
||||
|
||||
|
||||
def _darwin_process_start_time(pid: int) -> str | None:
|
||||
"""macOS process start, from ``ps -o lstart=``.
|
||||
|
||||
``lstart`` is the absolute wall-clock start of that pid's current
|
||||
incarnation. Two processes reusing one pid number report different values,
|
||||
which is exactly the discrimination reuse detection needs.
|
||||
"""
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["/bin/ps", "-o", "lstart=", "-p", str(int(pid))],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
value = " ".join((completed.stdout or "").split())
|
||||
return f"lstart-{value}" if value else None
|
||||
|
||||
|
||||
def process_start_time(pid: int | None) -> str | None:
|
||||
"""Start-time token for *pid*'s current incarnation, or ``None``.
|
||||
|
||||
``None`` is returned both when the pid does not exist and when the platform
|
||||
cannot answer. Callers must not read either case as evidence of death — a
|
||||
dead pid is established by the liveness probe, and this value only ever
|
||||
*withdraws* a retirement that pid-level evidence would otherwise allow.
|
||||
"""
|
||||
if pid is None:
|
||||
return None
|
||||
try:
|
||||
numeric = int(pid)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if numeric <= 0:
|
||||
return None
|
||||
system = ""
|
||||
try:
|
||||
system = (platform.system() or "").strip().lower()
|
||||
except Exception:
|
||||
system = ""
|
||||
if system == "linux":
|
||||
return _linux_process_start_time(numeric)
|
||||
if system == "darwin":
|
||||
return _darwin_process_start_time(numeric)
|
||||
return None
|
||||
|
||||
|
||||
def current_process_fencing(pid: int | None = None) -> dict[str, str | None]:
|
||||
"""The full fencing triple for *pid* (defaults to this process)."""
|
||||
target = os.getpid() if pid is None else pid
|
||||
return {
|
||||
"host_id": current_host_id(),
|
||||
"boot_id": current_boot_id(),
|
||||
"process_start_time": process_start_time(target),
|
||||
}
|
||||
Reference in New Issue
Block a user