sysadmin
e344e68a13
Addresses the three blocking findings in review 657 on PR #982 and nothing
else.
B1 — apply was authorized by gitea.read
---------------------------------------
Plan and apply shared the observational permission class, so any profile that
could look could also destroy; the mutation landing in the local control-plane
registry rather than in Gitea makes it no less a mutation.
Apply now requires its own capability, gitea.worker_registry.retire, checked at
entry and re-resolved immediately before the registry mutation. Plan stays on
gitea.read. No profile holds the new permission by default, so author,
reviewer, merger, and read-only profiles fail closed on the permission itself
rather than on the role check alone; the controller/reconciler role restriction
remains as defence in depth. Granting it is a deliberate operator edit to
profiles.json, and removing it revokes apply completely. No new Gitea write
permission is introduced and no author permission is broadened.
B2 — complete legacy rows could be retired without identity proof
-----------------------------------------------------------------
"Incomplete identity" previously meant only that pre-existing columns were
null, which a legacy-pid row satisfies trivially. Retirement now requires an
affirmative two-part proof: launcher-minted inst- attribution, plus fencing
evidence (host_id, boot_id, process_start_time) that turns a bare pid into a
statement about one process incarnation.
New mcp_process_fencing supplies those probes; every one returns None rather
than guessing, and None always preserves. Rows written before these columns
existed, and rows on legacy instance identities, are preserved permanently —
they are retired only after re-registering under a trusted identity. That
legacy rows would otherwise remain outstanding is explicitly not treated as
grounds for a weaker proof. A live pid stays an absolute block even across a
boot boundary, and pid reuse is reported distinctly from a live worker.
B3 — lease and OS liveness sat outside the registry transaction
---------------------------------------------------------------
BEGIN IMMEDIATE locks the worker registry only, and the per-target loop runs
after revalidation, so a lease acquired or a pid revived in between would go
unnoticed — the registry-column guard cannot catch it because no registry
column changed.
Two mechanisms now close that window, both applied per target immediately
before its own write: an external_fence_fn version token over active leases,
captured inside the transaction before the authoritative read and re-compared
before every guarded UPDATE (movement aborts the whole transaction; an
unreadable lease store raises rather than comparing equal), and a liveness_fn
re-probe that must affirmatively re-establish that this exact process is gone,
comparing process_start_time so a reused pid is refused. Omitting the re-probe
retires nothing rather than proceeding unfenced. The guarded UPDATE also
asserts the fencing triple is unchanged, and all three columns participate in
the CAS token.
Preserved from the accepted work: registry_fingerprint still excludes
observation time and is order- and numeric-typing stable, plan still mutates
nothing, drift still retires zero, retired rows stay historical, and ordinary
author operations remain ungated by fleet state.
Tests: tests/test_issue_980_stale_worker_retirement.py 87 passed, 46 subtests
(was 40 passed, 23 subtests), covering all three blockers' required
regressions. Adjacent suites (#980, #978, #975, #948, task-capability role
invariants) 242 passed, 156 subtests. Full suite from the branch worktree
28 failed, 6253 passed, 6 skipped, 1152 subtests; master baseline at
108cbfa173 from branches/baseline-980-108cbfa 28 failed, 6166 passed, 6
skipped, 1106 subtests. The failing sets are identical under comm, so zero
regressions and zero masked pre-existing failures; the +87 passing delta is
this branch's tests.
No live worker-registry row was retired — every test uses a throwaway SQLite
database. BAA, issue #981, PR #906, issue #650, review 624, unrelated branches
and worktrees, profiles, configuration, credentials, sessions, and running
processes were not touched.
Refs #980
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1674 lines
68 KiB
Python
1674 lines
68 KiB
Python
"""CAS-protected stale worker retirement (#980).
|
|
|
|
Covers the acceptance criteria: the registry CAS token is stable across time
|
|
and row order but moves on any retirement-relevant change, plan mutates
|
|
nothing, apply fails closed on drift, every target is revalidated inside the
|
|
retirement transaction, and live / ambiguous / incomplete rows are preserved.
|
|
|
|
The regression test that matters most is
|
|
``test_snapshot_at_would_have_moved_the_token``: it demonstrates the exact
|
|
defect the R3-C review found — ``mcp_fleet_snapshot._consistency_token``
|
|
changes when only the observation time changed — and proves the new token does
|
|
not.
|
|
|
|
All identifiers are synthetic.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import os
|
|
import sqlite3
|
|
import tempfile
|
|
import unittest
|
|
from datetime import datetime, timedelta, timezone
|
|
from typing import Any
|
|
from unittest import mock
|
|
|
|
import mcp_fleet_retirement as retire
|
|
import mcp_fleet_snapshot as fleet
|
|
import mcp_worker_identity as mwi
|
|
|
|
|
|
NOW = datetime(2026, 7, 30, 7, 0, 0, tzinfo=timezone.utc)
|
|
TTL = 900.0
|
|
REPO = "/synthetic/repo/Gitea-Tools"
|
|
|
|
#: Synthetic fencing identity (#980 review 657 B2/B3). Retirement is only ever
|
|
#: safe relative to a host and a boot, so the fixtures name both explicitly
|
|
#: rather than inheriting whatever machine the suite happens to run on.
|
|
HOST = "synthetic-host-a"
|
|
OTHER_HOST = "synthetic-host-b"
|
|
BOOT = "boot-1000"
|
|
OTHER_BOOT = "boot-2000"
|
|
|
|
|
|
def _start(pid: int | None) -> str | None:
|
|
"""Deterministic start-time token for one pid incarnation."""
|
|
return None if pid is None else f"start-{pid}"
|
|
|
|
_ROW_COLUMNS = (
|
|
"worker_identity",
|
|
"client_name",
|
|
"client_instance_id",
|
|
"session_id",
|
|
"generation_id",
|
|
"role",
|
|
"profile",
|
|
"namespace",
|
|
"remote",
|
|
"repository_binding",
|
|
"pid",
|
|
"process_identity",
|
|
"transport",
|
|
"token_fingerprint",
|
|
"started_at",
|
|
"last_heartbeat_at",
|
|
"heartbeat_ttl_seconds",
|
|
"fencing_epoch",
|
|
"status",
|
|
"fleet_run_id",
|
|
"authenticated_account",
|
|
"instance_id_provenance",
|
|
"host_id",
|
|
"boot_id",
|
|
"process_start_time",
|
|
)
|
|
|
|
|
|
def _registry() -> mwi.WorkerRegistry:
|
|
handle, path = tempfile.mkstemp(suffix=".sqlite3")
|
|
os.close(handle)
|
|
os.unlink(path)
|
|
return mwi.WorkerRegistry(path)
|
|
|
|
|
|
def _instance_for(identity: str) -> str:
|
|
"""A launcher-minted instance id unique to one synthetic worker.
|
|
|
|
Independent workers are independent *application launches*, so they get
|
|
distinct instance ids by default. Sharing one id across several rows is the
|
|
#978 cohort/uniqueness situation and tests that mean it say so explicitly
|
|
by passing ``instance=``.
|
|
"""
|
|
suffix = hashlib.sha256(identity.encode("utf-8")).hexdigest()[:12]
|
|
return f"inst-codex-20260730T070000Z-{suffix}"
|
|
|
|
|
|
def _row(
|
|
*,
|
|
identity: str,
|
|
pid: int | None,
|
|
instance: str | None = None,
|
|
session: str = "sess-a",
|
|
generation: str = "gen-a",
|
|
namespace: str = "author",
|
|
heartbeat: datetime | None = None,
|
|
status: str = mwi.STATUS_ACTIVE,
|
|
repository_binding: str | None = REPO,
|
|
ttl: float = TTL,
|
|
**overrides: Any,
|
|
) -> dict[str, Any]:
|
|
"""One synthetic ``worker_registrations`` row."""
|
|
record = {
|
|
"worker_identity": identity,
|
|
"client_name": "codex",
|
|
"client_instance_id": instance or _instance_for(identity),
|
|
"session_id": session,
|
|
"generation_id": generation,
|
|
"role": namespace,
|
|
"profile": f"prgs-{namespace}",
|
|
"namespace": namespace,
|
|
"remote": "prgs",
|
|
"repository_binding": repository_binding,
|
|
"pid": pid,
|
|
"process_identity": f"pid-{pid}" if pid is not None else None,
|
|
"transport": "stdio",
|
|
"token_fingerprint": None,
|
|
"started_at": "2026-07-30T06:00:00Z",
|
|
"last_heartbeat_at": mwi._ts(heartbeat or (NOW - timedelta(seconds=7200))),
|
|
"heartbeat_ttl_seconds": ttl,
|
|
"fencing_epoch": 1,
|
|
"status": status,
|
|
"fleet_run_id": "run-canary",
|
|
"authenticated_account": "synthetic-user",
|
|
"instance_id_provenance": fleet.INSTANCE_ID_PROVENANCE_TRUSTED,
|
|
# Complete fencing evidence by default, so a test that wants an
|
|
# incomplete row has to say so explicitly rather than getting one by
|
|
# omission (#980 review 657 B2).
|
|
"host_id": HOST,
|
|
"boot_id": BOOT,
|
|
"process_start_time": _start(pid),
|
|
}
|
|
record.update(overrides)
|
|
return record
|
|
|
|
|
|
def _dead(_pid: int | None) -> bool:
|
|
return False
|
|
|
|
|
|
def _alive(_pid: int | None) -> bool:
|
|
return True
|
|
|
|
|
|
def _selective(alive_pids: set[int]):
|
|
def probe(pid: int | None) -> bool:
|
|
return pid in alive_pids
|
|
|
|
return probe
|
|
|
|
|
|
def _plan(rows, *, probe=_dead, now=NOW, **kwargs):
|
|
kwargs.setdefault("current_host_id", HOST)
|
|
kwargs.setdefault("current_boot_id", BOOT)
|
|
kwargs.setdefault("start_time_probe", _start)
|
|
return retire.plan_stale_worker_retirement(
|
|
rows,
|
|
now=now,
|
|
pid_alive_probe=probe,
|
|
canonical_repository=REPO,
|
|
**kwargs,
|
|
)
|
|
|
|
|
|
def _safe_liveness(row: dict[str, Any]) -> dict[str, Any]:
|
|
"""Pre-write re-probe that agrees the process is gone."""
|
|
return {"safe": True, "evidence": {"pid": row.get("pid"), "pid_alive": False}}
|
|
|
|
|
|
def _register(registry: mwi.WorkerRegistry, row: dict[str, Any]) -> None:
|
|
"""Insert a synthetic row directly, bypassing register()'s live-now stamps."""
|
|
columns = [name for name in _ROW_COLUMNS if name in row]
|
|
placeholders = ", ".join("?" for _ in columns)
|
|
with registry._tx() as conn:
|
|
conn.execute(
|
|
f"INSERT INTO worker_registrations ({', '.join(columns)}) "
|
|
f"VALUES ({placeholders})",
|
|
[row[name] for name in columns],
|
|
)
|
|
|
|
|
|
class RegistryFingerprintStabilityTests(unittest.TestCase):
|
|
"""AC: identical contents observed at different times produce one token."""
|
|
|
|
def test_same_contents_different_observation_times_same_token(self):
|
|
rows = [_row(identity="w-1", pid=101), _row(identity="w-2", pid=102)]
|
|
self.assertEqual(
|
|
retire.registry_fingerprint(rows), retire.registry_fingerprint(rows)
|
|
)
|
|
# Recompute after the clock has moved a full hour: the token is derived
|
|
# from content only, so it cannot notice.
|
|
plan_early = _plan(rows, now=NOW)
|
|
plan_late = _plan(rows, now=NOW + timedelta(hours=1))
|
|
self.assertEqual(
|
|
plan_early["registry_fingerprint"], plan_late["registry_fingerprint"]
|
|
)
|
|
|
|
def test_snapshot_at_would_have_moved_the_token(self):
|
|
"""Regression: the old time-seeded derivation moved, the new one does not."""
|
|
rows = [_row(identity="w-1", pid=101)]
|
|
old_early = fleet._consistency_token(rows, "2026-07-30T07:00:00Z")
|
|
old_late = fleet._consistency_token(rows, "2026-07-30T07:00:01Z")
|
|
self.assertNotEqual(
|
|
old_early,
|
|
old_late,
|
|
"the #980 defect: one second of observation drift changed the token",
|
|
)
|
|
self.assertEqual(
|
|
retire.registry_fingerprint(rows), retire.registry_fingerprint(rows)
|
|
)
|
|
|
|
def test_row_order_does_not_change_the_token(self):
|
|
rows = [
|
|
_row(identity="w-1", pid=101),
|
|
_row(identity="w-2", pid=102),
|
|
_row(identity="w-3", pid=103),
|
|
]
|
|
self.assertEqual(
|
|
retire.registry_fingerprint(rows),
|
|
retire.registry_fingerprint(list(reversed(rows))),
|
|
)
|
|
|
|
def test_numeric_typing_does_not_change_the_token(self):
|
|
as_float = [_row(identity="w-1", pid=101, heartbeat_ttl_seconds=900.0)]
|
|
as_int = [_row(identity="w-1", pid=101, heartbeat_ttl_seconds=900)]
|
|
self.assertEqual(
|
|
retire.registry_fingerprint(as_float),
|
|
retire.registry_fingerprint(as_int),
|
|
)
|
|
|
|
def test_retirement_relevant_changes_move_the_token(self):
|
|
base = [_row(identity="w-1", pid=101)]
|
|
baseline = retire.registry_fingerprint(base)
|
|
mutations = {
|
|
"row added": base + [_row(identity="w-2", pid=102)],
|
|
"row removed": [],
|
|
"status changed": [_row(identity="w-1", pid=101, status="released")],
|
|
"identity changed": [
|
|
_row(
|
|
identity="w-1",
|
|
pid=101,
|
|
instance="inst-codex-20260730T070000Z-ffffffffffff",
|
|
)
|
|
],
|
|
"ownership changed": [_row(identity="w-1", pid=101, session="sess-other")],
|
|
"generation changed": [
|
|
_row(identity="w-1", pid=101, generation="gen-other")
|
|
],
|
|
"pid changed": [_row(identity="w-1", pid=999)],
|
|
"repository binding changed": [
|
|
_row(identity="w-1", pid=101, repository_binding="/elsewhere")
|
|
],
|
|
}
|
|
for label, rows in mutations.items():
|
|
with self.subTest(change=label):
|
|
self.assertNotEqual(baseline, retire.registry_fingerprint(rows))
|
|
|
|
def test_heartbeat_change_moves_the_token(self):
|
|
base = [_row(identity="w-1", pid=101)]
|
|
moved = [_row(identity="w-1", pid=101, heartbeat=NOW - timedelta(seconds=30))]
|
|
self.assertNotEqual(
|
|
retire.registry_fingerprint(base), retire.registry_fingerprint(moved)
|
|
)
|
|
|
|
def test_ttl_change_moves_the_token(self):
|
|
base = [_row(identity="w-1", pid=101)]
|
|
moved = [_row(identity="w-1", pid=101, ttl=60.0)]
|
|
self.assertNotEqual(
|
|
retire.registry_fingerprint(base), retire.registry_fingerprint(moved)
|
|
)
|
|
|
|
|
|
class PlanEligibilityTests(unittest.TestCase):
|
|
"""AC: only conclusively stale orphans are selected; everything else stays."""
|
|
|
|
def test_plan_selects_dead_stale_orphan(self):
|
|
plan = _plan([_row(identity="w-1", pid=101)])
|
|
self.assertEqual(plan["candidate_count"], 1)
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
self.assertEqual(plan["candidates"][0]["reason_code"], retire.REASON_ELIGIBLE)
|
|
self.assertFalse(plan["mutation_performed"])
|
|
self.assertTrue(plan["read_only"])
|
|
|
|
def test_plan_performs_no_mutation(self):
|
|
registry = _registry()
|
|
_register(registry, _row(identity="w-1", pid=101))
|
|
before = registry.list_workers(status=None)
|
|
plan = _plan(before)
|
|
after = registry.list_workers(status=None)
|
|
self.assertEqual(plan["candidate_count"], 1)
|
|
self.assertEqual(before, after)
|
|
self.assertEqual([r["status"] for r in after], [mwi.STATUS_ACTIVE])
|
|
self.assertFalse(plan["mutation_performed"])
|
|
|
|
def test_live_worker_is_preserved(self):
|
|
row = _row(identity="w-live", pid=101, heartbeat=NOW - timedelta(seconds=10))
|
|
plan = _plan([row], probe=_alive)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_WORKER_LIVE
|
|
)
|
|
|
|
def test_fresh_heartbeat_with_dead_pid_still_fails_closed(self):
|
|
"""A dead pid withdraws liveness; the unexpired heartbeat still preserves."""
|
|
row = _row(identity="w-fresh", pid=101, heartbeat=NOW - timedelta(seconds=10))
|
|
plan = _plan([row], probe=_dead)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_HEARTBEAT_FRESH
|
|
)
|
|
|
|
def test_unprobeable_pid_is_preserved(self):
|
|
plan = _plan([_row(identity="w-1", pid=101)], probe=lambda _pid: None)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_PID_UNKNOWN
|
|
)
|
|
|
|
def test_incomplete_legacy_identity_is_preserved(self):
|
|
"""AC: incomplete legacy identities remain fail-closed."""
|
|
rows = [
|
|
_row(identity="w-nopid", pid=None, instance="legacy-pid-27833"),
|
|
_row(identity="w-nosession", pid=102, session=""),
|
|
]
|
|
plan = _plan(rows)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
{p["reason_code"] for p in plan["preserved"]},
|
|
{retire.REASON_INCOMPLETE_IDENTITY},
|
|
)
|
|
detail = next(
|
|
p["detail"] for p in plan["preserved"] if p["worker_identity"] == "w-nopid"
|
|
)
|
|
self.assertIn("pid", detail)
|
|
|
|
def test_unparsable_heartbeat_is_preserved(self):
|
|
rows = [_row(identity="w-1", pid=101, last_heartbeat_at="not-a-stamp")]
|
|
plan = _plan(rows)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_UNPARSABLE_HEARTBEAT
|
|
)
|
|
|
|
def test_identity_shared_with_live_worker_is_preserved(self):
|
|
"""Two rows, one live: the dead one shares session evidence, so it stays."""
|
|
rows = [
|
|
_row(
|
|
identity="w-live",
|
|
pid=101,
|
|
session="sess-shared",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
_row(identity="w-dead", pid=102, session="sess-shared"),
|
|
]
|
|
plan = _plan(rows, probe=_selective({101}))
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
codes = {p["reason_code"] for p in plan["preserved"]}
|
|
self.assertIn(retire.REASON_CONFLICTING_IDENTITY, codes)
|
|
|
|
def test_foreign_or_missing_repository_binding_is_preserved(self):
|
|
rows = [
|
|
_row(identity="w-foreign", pid=101, repository_binding="/other/repo"),
|
|
_row(identity="w-unbound", pid=102, repository_binding=None),
|
|
]
|
|
plan = _plan(rows)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
{p["reason_code"] for p in plan["preserved"]},
|
|
{retire.REASON_FOREIGN_REPOSITORY},
|
|
)
|
|
|
|
def test_active_workflow_owner_is_preserved(self):
|
|
rows = [_row(identity="w-1", pid=101, session="sess-leased")]
|
|
plan = _plan(rows, protected_session_ids=["sess-leased"])
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_PROTECTED_OWNER
|
|
)
|
|
|
|
def test_terminal_rows_are_not_retired_again(self):
|
|
rows = [_row(identity="w-1", pid=101, status=mwi.STATUS_RELEASED)]
|
|
plan = _plan(rows)
|
|
self.assertEqual(plan["candidate_count"], 0)
|
|
self.assertEqual(
|
|
plan["preserved"][0]["reason_code"], retire.REASON_ALREADY_TERMINAL
|
|
)
|
|
|
|
def test_mixed_fleet_produces_correct_per_worker_outcomes(self):
|
|
rows = [
|
|
_row(identity="w-stale-1", pid=101, session="s1", generation="g1"),
|
|
_row(identity="w-stale-2", pid=102, session="s2", generation="g2"),
|
|
_row(
|
|
# An independent worker, so a distinct application launch and
|
|
# therefore a distinct instance: sharing one instance *and*
|
|
# namespace with a live worker is the #978 uniqueness
|
|
# violation, which is covered separately.
|
|
identity="w-live",
|
|
pid=103,
|
|
session="s3",
|
|
generation="g3",
|
|
instance="inst-codex-20260730T070000Z-ffffffffffff",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
_row(identity="w-nopid", pid=None, session="s4", generation="g4"),
|
|
_row(
|
|
identity="w-foreign",
|
|
pid=105,
|
|
session="s5",
|
|
generation="g5",
|
|
repository_binding="/other",
|
|
),
|
|
_row(
|
|
identity="w-terminal",
|
|
pid=106,
|
|
session="s6",
|
|
generation="g6",
|
|
status=mwi.STATUS_SUPERSEDED,
|
|
),
|
|
]
|
|
plan = _plan(rows, probe=_selective({103}))
|
|
self.assertEqual(
|
|
sorted(plan["candidate_worker_identities"]), ["w-stale-1", "w-stale-2"]
|
|
)
|
|
by_identity = {
|
|
p["worker_identity"]: p["reason_code"] for p in plan["preserved"]
|
|
}
|
|
self.assertEqual(by_identity["w-live"], retire.REASON_WORKER_LIVE)
|
|
self.assertEqual(by_identity["w-nopid"], retire.REASON_INCOMPLETE_IDENTITY)
|
|
self.assertEqual(by_identity["w-foreign"], retire.REASON_FOREIGN_REPOSITORY)
|
|
self.assertEqual(by_identity["w-terminal"], retire.REASON_ALREADY_TERMINAL)
|
|
self.assertEqual(plan["assessed_count"], 6)
|
|
self.assertEqual(plan["preserved_count"], 4)
|
|
|
|
def test_plan_is_deterministic_for_identical_contents(self):
|
|
rows = [_row(identity="w-1", pid=101), _row(identity="w-2", pid=102)]
|
|
first = _plan(rows)
|
|
second = _plan(list(reversed(rows)), now=NOW + timedelta(minutes=5))
|
|
self.assertEqual(first["registry_fingerprint"], second["registry_fingerprint"])
|
|
self.assertEqual(
|
|
first["candidate_fingerprint"], second["candidate_fingerprint"]
|
|
)
|
|
self.assertEqual(
|
|
sorted(first["candidate_worker_identities"]),
|
|
sorted(second["candidate_worker_identities"]),
|
|
)
|
|
|
|
|
|
class ApplyCasTests(unittest.TestCase):
|
|
"""AC: apply is compare-and-swap protected and revalidates every target."""
|
|
|
|
def setUp(self) -> None:
|
|
self.registry = _registry()
|
|
self.probe = _dead
|
|
|
|
def _rows(self):
|
|
return self.registry.list_workers(status=None)
|
|
|
|
def _apply(
|
|
self,
|
|
plan,
|
|
*,
|
|
probe=None,
|
|
identities=None,
|
|
liveness_fn=_safe_liveness,
|
|
external_fence_fn=None,
|
|
**kwargs,
|
|
):
|
|
chosen = probe or self.probe
|
|
return self.registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=(
|
|
identities
|
|
if identities is not None
|
|
else plan["candidate_worker_identities"]
|
|
),
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
plan_fn=lambda rows: _plan(rows, probe=chosen, **kwargs),
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
liveness_fn=liveness_fn,
|
|
external_fence_fn=external_fence_fn,
|
|
)
|
|
|
|
def test_matching_token_retires_the_planned_set(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-2", pid=102, session="s2"))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
result = self._apply(plan)
|
|
self.assertEqual(result["outcome"], "applied")
|
|
self.assertTrue(result["mutation_performed"])
|
|
self.assertEqual(result["retired_count"], 2)
|
|
statuses = {r["worker_identity"]: r["status"] for r in self._rows()}
|
|
self.assertEqual(
|
|
statuses, {"w-1": mwi.STATUS_RETIRED, "w-2": mwi.STATUS_RETIRED}
|
|
)
|
|
retired_row = self.registry.get("w-1")
|
|
self.assertEqual(retired_row["retired_at"], mwi._ts(NOW))
|
|
self.assertEqual(retired_row["retired_by"], "synthetic-user/prgs-reconciler")
|
|
|
|
def test_moved_registry_token_retires_zero_workers(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
# An unrelated registration lands between plan and apply.
|
|
_register(self.registry, _row(identity="w-2", pid=102, session="s2"))
|
|
result = self._apply(plan)
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_REGISTRY_MOVED)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
self.assertTrue(all(r["status"] == mwi.STATUS_ACTIVE for r in self._rows()))
|
|
|
|
def test_moved_candidate_set_retires_zero_workers(self):
|
|
"""Registry unchanged, but the eligibility verdict is no longer the same."""
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
# The row did not change; the process came back (pid probes alive), so
|
|
# revalidation inside the transaction finds no candidates at all.
|
|
result = self._apply(plan, probe=_alive)
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_CANDIDATES_MOVED)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
self.assertEqual(self.registry.get("w-1")["status"], mwi.STATUS_ACTIVE)
|
|
|
|
def test_worker_that_becomes_live_between_plan_and_apply_is_preserved(self):
|
|
_register(self.registry, _row(identity="w-dead", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-back", pid=102, session="s2"))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
self.assertEqual(len(plan["candidate_worker_identities"]), 2)
|
|
# w-back's process is alive by the time apply runs, so the candidate
|
|
# fingerprint moves and nothing at all is retired.
|
|
result = self._apply(plan, probe=_selective({102}))
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_CANDIDATES_MOVED)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertEqual(self.registry.get("w-back")["status"], mwi.STATUS_ACTIVE)
|
|
self.assertEqual(self.registry.get("w-dead")["status"], mwi.STATUS_ACTIVE)
|
|
|
|
def test_worker_that_becomes_ambiguous_between_plan_and_apply_is_preserved(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
result = self._apply(plan, probe=lambda _pid: None)
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_CANDIDATES_MOVED)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertEqual(self.registry.get("w-1")["status"], mwi.STATUS_ACTIVE)
|
|
|
|
def test_apply_revalidates_and_will_not_narrow_the_approved_set(self):
|
|
"""A protection appearing after plan moves the CAS, so nothing is retired."""
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
_register(
|
|
self.registry, _row(identity="w-protected", pid=102, session="sess-leased")
|
|
)
|
|
unprotected_plan = _plan(self._rows(), probe=self.probe)
|
|
self.assertEqual(unprotected_plan["candidate_count"], 2)
|
|
result = self._apply(
|
|
unprotected_plan, protected_session_ids=["sess-leased"]
|
|
)
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_CANDIDATES_MOVED)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertEqual(
|
|
self.registry.get("w-protected")["status"], mwi.STATUS_ACTIVE
|
|
)
|
|
|
|
def test_target_outside_the_current_plan_is_preserved(self):
|
|
_register(
|
|
self.registry,
|
|
_row(identity="w-1", pid=101, session="s1", generation="g1"),
|
|
)
|
|
_register(
|
|
self.registry,
|
|
_row(
|
|
identity="w-live",
|
|
pid=102,
|
|
session="s2",
|
|
generation="g2",
|
|
instance="inst-codex-20260730T070000Z-ffffffffffff",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
)
|
|
probe = _selective({102})
|
|
plan = _plan(self._rows(), probe=probe)
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
# A caller that appends a live identity to the approved list gets it
|
|
# preserved with the live reason code; only the real candidate retires.
|
|
result = self._apply(plan, probe=probe, identities=["w-1", "w-live"])
|
|
self.assertEqual(result["outcome"], "applied")
|
|
self.assertEqual(result["retired_count"], 1)
|
|
self.assertEqual(result["retired"][0]["worker_identity"], "w-1")
|
|
preserved = {
|
|
p["worker_identity"]: p["reason_code"] for p in result["preserved"]
|
|
}
|
|
self.assertEqual(preserved["w-live"], retire.REASON_WORKER_LIVE)
|
|
self.assertEqual(self.registry.get("w-live")["status"], mwi.STATUS_ACTIVE)
|
|
|
|
def test_missing_registration_is_reported_not_invented(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
result = self._apply(plan, identities=["w-1", "w-ghost"])
|
|
self.assertEqual(result["retired_count"], 1)
|
|
preserved = {
|
|
p["worker_identity"]: p["reason_code"] for p in result["preserved"]
|
|
}
|
|
self.assertEqual(preserved["w-ghost"], "registration_missing")
|
|
|
|
def test_reapplying_a_completed_plan_is_a_safe_no_op(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
first = self._apply(plan)
|
|
self.assertEqual(first["retired_count"], 1)
|
|
second = self._apply(plan)
|
|
self.assertEqual(second["outcome"], retire.OUTCOME_ALREADY_RETIRED)
|
|
self.assertTrue(second["idempotent"])
|
|
self.assertEqual(second["retired_count"], 0)
|
|
self.assertFalse(second["mutation_performed"])
|
|
self.assertEqual(self.registry.get("w-1")["status"], mwi.STATUS_RETIRED)
|
|
|
|
def test_empty_target_list_reports_nothing_requested(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
result = self._apply(plan, identities=[])
|
|
self.assertEqual(result["outcome"], retire.OUTCOME_NOTHING_REQUESTED)
|
|
self.assertFalse(result["mutation_performed"])
|
|
self.assertEqual(self.registry.get("w-1")["status"], mwi.STATUS_ACTIVE)
|
|
|
|
def test_transaction_failure_cannot_report_partial_success(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-2", pid=102, session="s2"))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
calls = {"n": 0}
|
|
|
|
def exploding_plan(rows):
|
|
calls["n"] += 1
|
|
raise RuntimeError("synthetic revalidation failure")
|
|
|
|
result = self.registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=plan["candidate_worker_identities"],
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
plan_fn=exploding_plan,
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
liveness_fn=_safe_liveness,
|
|
)
|
|
self.assertEqual(calls["n"], 1)
|
|
self.assertFalse(result["success"])
|
|
self.assertEqual(result["outcome"], "transaction_failed")
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
self.assertTrue(
|
|
all(r["status"] == mwi.STATUS_ACTIVE for r in self._rows()),
|
|
"a failed transaction must roll back every retirement",
|
|
)
|
|
|
|
def test_structured_result_fields_are_accurate(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows(), probe=self.probe)
|
|
blocked = self._apply(plan, probe=_alive)
|
|
self.assertFalse(blocked["mutation_performed"])
|
|
self.assertEqual(blocked["retired"], [])
|
|
self.assertEqual(blocked["requested_count"], 1)
|
|
self.assertEqual(
|
|
blocked["expected_registry_fingerprint"], plan["registry_fingerprint"]
|
|
)
|
|
applied = self._apply(plan)
|
|
self.assertTrue(applied["mutation_performed"])
|
|
self.assertEqual(applied["acting_identity"], "synthetic-user/prgs-reconciler")
|
|
self.assertEqual(applied["retired"][0]["reason_code"], retire.REASON_ELIGIBLE)
|
|
self.assertIn("evidence", applied["retired"][0])
|
|
|
|
|
|
class PostRetirementFleetCompatibilityTests(unittest.TestCase):
|
|
"""AC: retired rows leave the stale count and never fake fleet safety."""
|
|
|
|
def test_retired_rows_are_historical_not_stale(self):
|
|
registry = _registry()
|
|
_register(
|
|
registry, _row(identity="w-1", pid=101, session="s1", generation="g1")
|
|
)
|
|
_register(
|
|
registry,
|
|
_row(
|
|
identity="w-live",
|
|
pid=102,
|
|
session="s2",
|
|
generation="g2",
|
|
instance="legacy-pid-27833",
|
|
instance_id_provenance=fleet.INSTANCE_ID_PROVENANCE_LEGACY,
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
)
|
|
probe = _selective({102})
|
|
before = fleet.snapshot_instance_fleet(
|
|
registry.list_workers(status=None),
|
|
now=NOW,
|
|
pid_alive_probe=probe,
|
|
canonical_repository=REPO,
|
|
)
|
|
self.assertEqual(before["stale_worker_count"], 1)
|
|
|
|
plan = _plan(registry.list_workers(status=None), probe=probe)
|
|
result = registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=plan["candidate_worker_identities"],
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
plan_fn=lambda rows: _plan(rows, probe=probe),
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
liveness_fn=_safe_liveness,
|
|
)
|
|
self.assertEqual(result["retired_count"], 1)
|
|
|
|
after = fleet.snapshot_instance_fleet(
|
|
registry.list_workers(status=None),
|
|
now=NOW,
|
|
pid_alive_probe=probe,
|
|
canonical_repository=REPO,
|
|
)
|
|
self.assertEqual(after["stale_worker_count"], 0)
|
|
self.assertEqual(after["live_worker_count"], 1)
|
|
self.assertEqual(after["historical_worker_count"], 1)
|
|
# The surviving live worker still carries a legacy instance identity, so
|
|
# the fleet must not be declared safe.
|
|
self.assertFalse(after["live_fleet_safe"])
|
|
self.assertIn(
|
|
fleet.CLASS_LEGACY_INCOMPLETE,
|
|
{f["classification"] for f in after["active_blockers"]},
|
|
)
|
|
|
|
def test_existing_snapshot_shape_is_unchanged(self):
|
|
rows = [_row(identity="w-1", pid=101)]
|
|
snapshot = fleet.snapshot_instance_fleet(
|
|
rows, now=NOW, pid_alive_probe=_dead, canonical_repository=REPO
|
|
)
|
|
for key in (
|
|
"consistency_token",
|
|
"registry_revision",
|
|
"snapshot_at",
|
|
"live_workers",
|
|
"stale_workers",
|
|
"historical_workers",
|
|
"findings",
|
|
"live_fleet_safe",
|
|
):
|
|
self.assertIn(key, snapshot)
|
|
self.assertTrue(snapshot["registry_revision"].startswith("fleetrev-"))
|
|
self.assertTrue(snapshot["read_only"])
|
|
self.assertFalse(snapshot.get("mutation_performed", False))
|
|
|
|
|
|
class CapabilityExposureTests(unittest.TestCase):
|
|
"""AC: only controller/reconciler reach the surface; author policy unchanged."""
|
|
|
|
def test_capability_map_declares_controller_role(self):
|
|
import task_capability_map as tcm
|
|
|
|
for task in (
|
|
"plan_stale_worker_retirement",
|
|
"gitea_plan_stale_worker_retirement",
|
|
"apply_stale_worker_retirement",
|
|
"gitea_apply_stale_worker_retirement",
|
|
):
|
|
with self.subTest(task=task):
|
|
self.assertEqual(tcm.required_role(task), "controller")
|
|
|
|
def test_plan_is_observational_but_apply_needs_its_own_capability(self):
|
|
"""#980 review 657 B1: apply must not share plan's read permission."""
|
|
import task_capability_map as tcm
|
|
|
|
for task in (
|
|
"plan_stale_worker_retirement",
|
|
"gitea_plan_stale_worker_retirement",
|
|
):
|
|
with self.subTest(task=task):
|
|
self.assertEqual(tcm.required_permission(task), "gitea.read")
|
|
|
|
for task in (
|
|
"apply_stale_worker_retirement",
|
|
"gitea_apply_stale_worker_retirement",
|
|
):
|
|
with self.subTest(task=task):
|
|
self.assertEqual(
|
|
tcm.required_permission(task),
|
|
"gitea.worker_registry.retire",
|
|
"apply is a mutation and cannot be authorized by gitea.read",
|
|
)
|
|
self.assertNotEqual(tcm.required_permission(task), "gitea.read")
|
|
|
|
def test_ordinary_author_capability_resolution_is_unaffected(self):
|
|
import task_capability_map as tcm
|
|
|
|
self.assertEqual(tcm.required_permission("work_issue"), "gitea.pr.create")
|
|
self.assertEqual(tcm.required_role("work_issue"), "author")
|
|
self.assertEqual(tcm.required_permission("create_pr"), "gitea.pr.create")
|
|
self.assertEqual(tcm.required_role("create_pr"), "author")
|
|
self.assertEqual(tcm.required_permission("commit_files"), "gitea.repo.commit")
|
|
self.assertEqual(tcm.required_permission("push_branch"), "gitea.branch.push")
|
|
self.assertEqual(
|
|
tcm.required_permission("snapshot_instance_fleet"), "gitea.read"
|
|
)
|
|
|
|
def test_no_new_gitea_write_permission_is_introduced(self):
|
|
import task_capability_map as tcm
|
|
|
|
for task in ("plan_stale_worker_retirement", "apply_stale_worker_retirement"):
|
|
with self.subTest(task=task):
|
|
self.assertNotIn(
|
|
tcm.required_permission(task),
|
|
{
|
|
"gitea.pr.approve",
|
|
"gitea.pr.merge",
|
|
"gitea.pr.review",
|
|
"gitea.branch.push",
|
|
"gitea.repo.commit",
|
|
},
|
|
)
|
|
|
|
|
|
class SurfaceRegistrationTests(unittest.TestCase):
|
|
"""AC: the dry-run and apply modes are exposed as sanctioned native tools."""
|
|
|
|
def test_tools_are_registered_and_role_restricted(self):
|
|
import inspect
|
|
|
|
import gitea_mcp_server as server
|
|
|
|
for name in (
|
|
"gitea_plan_stale_worker_retirement",
|
|
"gitea_apply_stale_worker_retirement",
|
|
):
|
|
with self.subTest(tool=name):
|
|
tool = getattr(server, name)
|
|
self.assertTrue(callable(tool))
|
|
source = inspect.getsource(tool)
|
|
# Both tools route their role check through the shared guard.
|
|
self.assertIn(f'_retirement_role_block("{name}")', source)
|
|
|
|
guard = inspect.getsource(server._retirement_role_block)
|
|
self.assertIn('{"controller", "reconciler"}', guard)
|
|
self.assertIn("required_roles", guard)
|
|
|
|
apply_source = inspect.getsource(server.gitea_apply_stale_worker_retirement)
|
|
self.assertIn("registry_fingerprint", apply_source)
|
|
self.assertIn("candidate_fingerprint", apply_source)
|
|
self.assertIn("_retirement_runtime_block", apply_source)
|
|
self.assertIn("classify_cohort", apply_source)
|
|
# Revalidation inside the transaction re-reads lease protection rather
|
|
# than reusing the pre-transaction snapshot.
|
|
self.assertIn("_retirement_revalidation_plan", apply_source)
|
|
revalidation = inspect.getsource(server._retirement_revalidation_plan)
|
|
self.assertIn("_retirement_protected_owners", revalidation)
|
|
self.assertIn("raise RuntimeError", revalidation)
|
|
|
|
def test_plan_tool_declares_itself_read_only(self):
|
|
import inspect
|
|
|
|
import gitea_mcp_server as server
|
|
|
|
source = inspect.getsource(server.gitea_plan_stale_worker_retirement)
|
|
self.assertIn("read_only", source)
|
|
self.assertNotIn("retire_stale_workers", source)
|
|
|
|
def test_docs_exist(self):
|
|
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
|
path = os.path.join(root, "docs", "stale-worker-retirement.md")
|
|
self.assertTrue(os.path.isfile(path))
|
|
with open(path, encoding="utf-8") as handle:
|
|
text = handle.read()
|
|
for token in (
|
|
"registry_fingerprint",
|
|
"candidate_fingerprint",
|
|
"gitea_plan_stale_worker_retirement",
|
|
"gitea_apply_stale_worker_retirement",
|
|
"eligible_stale_orphan",
|
|
"registry_revision_moved",
|
|
"does not repair untrusted live identity",
|
|
):
|
|
with self.subTest(token=token):
|
|
self.assertIn(token, text)
|
|
|
|
|
|
RETIRE_PERMISSION = "gitea.worker_registry.retire"
|
|
|
|
|
|
def _profile(name: str, role: str, operations: list[str]) -> dict[str, Any]:
|
|
return {
|
|
"profile_name": name,
|
|
"role": role,
|
|
"role_kind": role,
|
|
"allowed_operations": list(operations),
|
|
"forbidden_operations": [],
|
|
}
|
|
|
|
|
|
class DedicatedMutationCapabilityTests(unittest.TestCase):
|
|
"""#980 review 657 B1: apply requires its own mutation capability.
|
|
|
|
These exercise the *real* permission gate. Only the two environment gates
|
|
that ``_profile_operation_gate`` also consults — master parity and runtime
|
|
mode — are neutralised, because they are unrelated to the capability being
|
|
proven and would otherwise make the result depend on the machine the suite
|
|
runs on.
|
|
"""
|
|
|
|
def _apply_as(self, profile: dict[str, Any], **patches):
|
|
import gitea_mcp_server as server
|
|
|
|
with mock.patch.object(server, "_master_parity_block", return_value=[]), \
|
|
mock.patch.object(server, "_runtime_mode_block", return_value=[]), \
|
|
mock.patch.object(server, "get_profile", return_value=profile), \
|
|
mock.patch.object(
|
|
server, "_retirement_runtime_block", return_value=[]
|
|
):
|
|
with mock.patch.multiple(server, **patches) if patches else _null():
|
|
return server.gitea_apply_stale_worker_retirement(
|
|
registry_fingerprint="registryfp-synthetic",
|
|
candidate_fingerprint="candidatefp-synthetic",
|
|
worker_identities=["w-1"],
|
|
remote="prgs",
|
|
)
|
|
|
|
def _assert_denied_without_mutation(self, result):
|
|
self.assertFalse(result.get("success"))
|
|
self.assertEqual(result.get("retired_count"), 0)
|
|
self.assertFalse(result.get("mutation_performed", True))
|
|
|
|
def test_read_only_capability_is_insufficient(self):
|
|
result = self._apply_as(
|
|
_profile("prgs-controller", "controller", ["gitea.read"])
|
|
)
|
|
self._assert_denied_without_mutation(result)
|
|
self.assertEqual(
|
|
result.get("required_operation_permission"), RETIRE_PERMISSION
|
|
)
|
|
|
|
def test_author_reviewer_and_merger_are_denied(self):
|
|
cases = {
|
|
"prgs-author": ("author", ["gitea.read", "gitea.pr.create"]),
|
|
"prgs-reviewer": ("reviewer", ["gitea.read", "gitea.pr.review"]),
|
|
"prgs-merger": ("merger", ["gitea.read", "gitea.pr.merge"]),
|
|
}
|
|
for name, (role, operations) in cases.items():
|
|
with self.subTest(profile=name):
|
|
result = self._apply_as(_profile(name, role, operations))
|
|
self._assert_denied_without_mutation(result)
|
|
|
|
def test_author_is_denied_even_if_granted_the_capability(self):
|
|
"""Role remains defence in depth behind the capability."""
|
|
result = self._apply_as(
|
|
_profile("prgs-author", "author", ["gitea.read", RETIRE_PERMISSION])
|
|
)
|
|
self._assert_denied_without_mutation(result)
|
|
self.assertEqual(result.get("denied_role"), "author")
|
|
|
|
def test_removing_the_capability_fails_closed(self):
|
|
granted = _profile(
|
|
"prgs-controller", "controller", ["gitea.read", RETIRE_PERMISSION]
|
|
)
|
|
revoked = _profile("prgs-controller", "controller", ["gitea.read"])
|
|
self._assert_denied_without_mutation(self._apply_as(revoked))
|
|
# The same profile *with* the capability gets a different refusal —
|
|
# proving the denial above came from the capability, not from something
|
|
# incidental that would deny either way.
|
|
with_capability = self._apply_as(granted)
|
|
self.assertNotEqual(
|
|
with_capability.get("required_operation_permission"),
|
|
RETIRE_PERMISSION,
|
|
"a granted profile must not be refused on the mutation capability",
|
|
)
|
|
|
|
def test_granted_controller_reaches_the_isolated_apply_path(self):
|
|
import gitea_mcp_server as server
|
|
|
|
registry = _registry()
|
|
_register(registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(registry.list_workers(status=None))
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
|
|
with mock.patch.object(server, "_master_parity_block", return_value=[]), \
|
|
mock.patch.object(server, "_runtime_mode_block", return_value=[]), \
|
|
mock.patch.object(
|
|
server,
|
|
"get_profile",
|
|
return_value=_profile(
|
|
"prgs-controller",
|
|
"controller",
|
|
["gitea.read", RETIRE_PERMISSION],
|
|
),
|
|
), \
|
|
mock.patch.object(
|
|
server, "_retirement_runtime_block", return_value=[]
|
|
), \
|
|
mock.patch.object(
|
|
server, "_worker_registry", return_value=registry
|
|
), \
|
|
mock.patch.object(
|
|
server,
|
|
"_retirement_protected_owners",
|
|
return_value=(None, {"session_ids": [], "pids": []}),
|
|
), \
|
|
mock.patch.object(
|
|
server, "_retirement_external_fence", return_value="fence-1"
|
|
), \
|
|
mock.patch.object(
|
|
server, "_retirement_liveness_reprobe", _safe_liveness
|
|
):
|
|
result = server.gitea_apply_stale_worker_retirement(
|
|
registry_fingerprint=plan["registry_fingerprint"],
|
|
candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=["w-1"],
|
|
remote="prgs",
|
|
canonical_repository=REPO,
|
|
)
|
|
|
|
# It reached the CAS/registry layer rather than a permission refusal.
|
|
self.assertNotEqual(
|
|
result.get("required_operation_permission"), RETIRE_PERMISSION
|
|
)
|
|
self.assertIn("outcome", result)
|
|
self.assertEqual(
|
|
result.get("permission_scope", {}).get("mutation_capability"),
|
|
RETIRE_PERMISSION,
|
|
)
|
|
|
|
def test_plan_stays_on_the_observational_capability(self):
|
|
import gitea_mcp_server as server
|
|
|
|
registry = _registry()
|
|
_register(registry, _row(identity="w-1", pid=101))
|
|
with mock.patch.object(server, "_master_parity_block", return_value=[]), \
|
|
mock.patch.object(server, "_runtime_mode_block", return_value=[]), \
|
|
mock.patch.object(
|
|
server,
|
|
"get_profile",
|
|
return_value=_profile(
|
|
"prgs-controller", "controller", ["gitea.read"]
|
|
),
|
|
), \
|
|
mock.patch.object(
|
|
server, "_worker_registry", return_value=registry
|
|
), \
|
|
mock.patch.object(
|
|
server,
|
|
"_retirement_protected_owners",
|
|
return_value=(None, {"session_ids": [], "pids": []}),
|
|
):
|
|
result = server.gitea_plan_stale_worker_retirement(
|
|
remote="prgs", canonical_repository=REPO
|
|
)
|
|
self.assertTrue(result.get("read_only"))
|
|
self.assertFalse(result.get("mutation_performed", True))
|
|
|
|
|
|
class _null:
|
|
def __enter__(self):
|
|
return None
|
|
|
|
def __exit__(self, *exc):
|
|
return False
|
|
|
|
|
|
class AffirmativeIdentityProofTests(unittest.TestCase):
|
|
"""#980 review 657 B2: retirement needs proof, not merely absent evidence.
|
|
|
|
Every case here is a registration whose *registry columns are all present*.
|
|
Before the correction that alone made a row eligible, which is exactly the
|
|
defect: non-null columns say nothing about which process a row describes.
|
|
"""
|
|
|
|
def _only(self, plan, key="preserved"):
|
|
self.assertEqual(len(plan[key]), 1, plan[key])
|
|
return plan[key][0]
|
|
|
|
def test_complete_trusted_identity_that_is_safely_stale_is_eligible(self):
|
|
plan = _plan([_row(identity="w-1", pid=101)])
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
self.assertEqual(
|
|
self._only(plan, "candidates")["reason_code"], retire.REASON_ELIGIBLE
|
|
)
|
|
|
|
def test_missing_instance_identity_is_preserved(self):
|
|
plan = _plan([_row(identity="w-1", pid=101, client_instance_id=None)])
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(
|
|
self._only(plan)["reason_code"], retire.REASON_INCOMPLETE_IDENTITY
|
|
)
|
|
|
|
def test_legacy_row_without_trusted_provenance_is_preserved(self):
|
|
"""The headline B2 case: complete columns, untrustworthy identity."""
|
|
plan = _plan(
|
|
[
|
|
_row(
|
|
identity="w-1",
|
|
pid=101,
|
|
client_instance_id="legacy-pid-80287",
|
|
instance_id_provenance=fleet.INSTANCE_ID_PROVENANCE_LEGACY,
|
|
)
|
|
]
|
|
)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(
|
|
self._only(plan)["reason_code"], retire.REASON_UNTRUSTED_PROVENANCE
|
|
)
|
|
|
|
def test_missing_fencing_columns_are_preserved(self):
|
|
for field in ("host_id", "boot_id", "process_start_time"):
|
|
with self.subTest(missing=field):
|
|
plan = _plan([_row(identity="w-1", pid=101, **{field: None})])
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(
|
|
self._only(plan)["reason_code"],
|
|
retire.REASON_INCOMPLETE_IDENTITY,
|
|
)
|
|
|
|
def test_pid_reuse_is_detected_and_preserved(self):
|
|
"""Same pid, same host and boot, but a different process incarnation."""
|
|
plan = _plan(
|
|
[_row(identity="w-1", pid=101, process_start_time="start-ORIGINAL")],
|
|
probe=_alive,
|
|
)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_PID_REUSED)
|
|
|
|
def test_same_pid_on_a_different_host_is_preserved(self):
|
|
plan = _plan([_row(identity="w-1", pid=101, host_id=OTHER_HOST)])
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_HOST_UNPROVEN)
|
|
|
|
def test_host_reuse_without_a_provable_local_host_is_preserved(self):
|
|
plan = _plan([_row(identity="w-1", pid=101)], current_host_id=None)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_HOST_UNPROVEN)
|
|
|
|
def test_unknown_current_boot_identity_is_preserved(self):
|
|
plan = _plan([_row(identity="w-1", pid=101)], current_boot_id=None)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_BOOT_UNKNOWN)
|
|
|
|
def test_a_prior_boot_satisfies_the_fencing_proof_when_the_pid_is_dead(self):
|
|
"""Pids do not survive a reboot, so the recorded process is gone."""
|
|
plan = _plan([_row(identity="w-1", pid=101, boot_id=OTHER_BOOT)])
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
|
|
def test_a_live_pid_always_blocks_even_across_a_boot_boundary(self):
|
|
"""Deliberately conservative: an occupied pid number is never retired.
|
|
|
|
A number occupied after a reboot cannot belong to the registered
|
|
process, so retiring would arguably be safe — but "the pid is alive"
|
|
stays an absolute block rather than something the fencing proof can
|
|
argue away.
|
|
"""
|
|
plan = _plan(
|
|
[_row(identity="w-1", pid=101, boot_id=OTHER_BOOT)], probe=_alive
|
|
)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_PID_ALIVE)
|
|
|
|
def test_conflicting_session_ownership_is_preserved(self):
|
|
rows = [
|
|
_row(identity="w-dead", pid=101, session="shared-session"),
|
|
_row(
|
|
identity="w-live",
|
|
pid=102,
|
|
session="shared-session",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
]
|
|
plan = _plan(rows, probe=_selective({102}))
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
reasons = {p["worker_identity"]: p["reason_code"] for p in plan["preserved"]}
|
|
self.assertEqual(reasons["w-dead"], retire.REASON_CONFLICTING_IDENTITY)
|
|
|
|
def test_conflicting_client_instance_identity_is_preserved(self):
|
|
shared = _instance_for("shared-launch")
|
|
rows = [
|
|
_row(
|
|
identity="w-dead",
|
|
pid=101,
|
|
session="s1",
|
|
generation="g1",
|
|
instance=shared,
|
|
namespace="author",
|
|
),
|
|
_row(
|
|
identity="w-live",
|
|
pid=102,
|
|
session="s2",
|
|
generation="g2",
|
|
instance=shared,
|
|
namespace="author",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
]
|
|
plan = _plan(rows, probe=_selective({102}))
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
reasons = {p["worker_identity"]: p["reason_code"] for p in plan["preserved"]}
|
|
self.assertEqual(reasons["w-dead"], retire.REASON_INSTANCE_CONFLICT)
|
|
|
|
def test_conflicting_generation_and_fencing_evidence_is_preserved(self):
|
|
rows = [
|
|
_row(identity="w-dead", pid=101, session="s1", generation="shared-gen"),
|
|
_row(
|
|
identity="w-live",
|
|
pid=102,
|
|
session="s2",
|
|
generation="shared-gen",
|
|
heartbeat=NOW - timedelta(seconds=5),
|
|
),
|
|
]
|
|
plan = _plan(rows, probe=_selective({102}))
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
reasons = {p["worker_identity"]: p["reason_code"] for p in plan["preserved"]}
|
|
self.assertEqual(reasons["w-dead"], retire.REASON_CONFLICTING_IDENTITY)
|
|
|
|
def test_live_lease_ownership_is_preserved(self):
|
|
plan = _plan(
|
|
[_row(identity="w-1", pid=101, session="s-leased")],
|
|
protected_session_ids=["s-leased"],
|
|
)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(
|
|
self._only(plan)["reason_code"], retire.REASON_PROTECTED_OWNER
|
|
)
|
|
|
|
def test_unknown_lease_state_aborts_rather_than_assuming_none(self):
|
|
"""An unreadable lease store must not degrade to an empty protection set."""
|
|
import gitea_mcp_server as server
|
|
|
|
with mock.patch.object(
|
|
server, "_control_plane_db_or_error", return_value=(None, ["db down"])
|
|
):
|
|
with self.assertRaises(Exception):
|
|
server._retirement_external_fence()
|
|
|
|
def test_active_worker_is_preserved(self):
|
|
plan = _plan(
|
|
[_row(identity="w-1", pid=101, heartbeat=NOW - timedelta(seconds=5))],
|
|
probe=_alive,
|
|
)
|
|
self.assertEqual(plan["candidate_worker_identities"], [])
|
|
self.assertEqual(self._only(plan)["reason_code"], retire.REASON_WORKER_LIVE)
|
|
|
|
def test_multiple_processes_in_one_cohort_are_distinguished_from_conflicts(self):
|
|
"""One launch, five namespaces: a cohort, not five conflicting claims."""
|
|
shared = _instance_for("one-cohort")
|
|
rows = [
|
|
_row(
|
|
identity=f"w-{ns}",
|
|
pid=200 + index,
|
|
session=f"s-{ns}",
|
|
generation="gen-cohort",
|
|
namespace=ns,
|
|
instance=shared,
|
|
)
|
|
for index, ns in enumerate(
|
|
("author", "reviewer", "merger", "controller", "reconciler")
|
|
)
|
|
]
|
|
plan = _plan(rows)
|
|
self.assertEqual(len(plan["candidate_worker_identities"]), 5)
|
|
|
|
def test_multiple_independent_workers_are_each_assessed_alone(self):
|
|
rows = [
|
|
_row(identity="w-1", pid=101, session="s1", generation="g1"),
|
|
_row(identity="w-2", pid=102, session="s2", generation="g2"),
|
|
]
|
|
plan = _plan(rows)
|
|
self.assertEqual(
|
|
sorted(plan["candidate_worker_identities"]), ["w-1", "w-2"]
|
|
)
|
|
|
|
def test_mixed_eligible_and_uncertain_candidates_split_correctly(self):
|
|
rows = [
|
|
_row(identity="w-ok", pid=101, session="s1", generation="g1"),
|
|
_row(
|
|
identity="w-legacy",
|
|
pid=102,
|
|
session="s2",
|
|
generation="g2",
|
|
client_instance_id="legacy-pid-777",
|
|
instance_id_provenance=fleet.INSTANCE_ID_PROVENANCE_LEGACY,
|
|
),
|
|
_row(
|
|
identity="w-nohost",
|
|
pid=103,
|
|
session="s3",
|
|
generation="g3",
|
|
host_id=None,
|
|
),
|
|
_row(
|
|
identity="w-otherhost",
|
|
pid=104,
|
|
session="s4",
|
|
generation="g4",
|
|
host_id=OTHER_HOST,
|
|
),
|
|
]
|
|
plan = _plan(rows)
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-ok"])
|
|
reasons = {p["worker_identity"]: p["reason_code"] for p in plan["preserved"]}
|
|
self.assertEqual(reasons["w-legacy"], retire.REASON_UNTRUSTED_PROVENANCE)
|
|
self.assertEqual(reasons["w-nohost"], retire.REASON_INCOMPLETE_IDENTITY)
|
|
self.assertEqual(reasons["w-otherhost"], retire.REASON_HOST_UNPROVEN)
|
|
|
|
def test_every_uncertain_case_reports_a_structured_reason(self):
|
|
rows = [
|
|
_row(identity="w-legacy", pid=101, client_instance_id="legacy-pid-1",
|
|
instance_id_provenance=fleet.INSTANCE_ID_PROVENANCE_LEGACY),
|
|
_row(identity="w-nohost", pid=102, session="s2", host_id=None),
|
|
_row(identity="w-otherhost", pid=103, session="s3", host_id=OTHER_HOST),
|
|
]
|
|
plan = _plan(rows)
|
|
for entry in plan["preserved"]:
|
|
with self.subTest(worker=entry["worker_identity"]):
|
|
self.assertTrue(entry["reason_code"])
|
|
self.assertTrue(entry["detail"])
|
|
self.assertIn("evidence", entry)
|
|
|
|
|
|
class ExternalStateFencingTests(unittest.TestCase):
|
|
"""#980 review 657 B3: evidence outside the registry TX cannot go stale.
|
|
|
|
``BEGIN IMMEDIATE`` locks the worker registry and nothing else. The lease
|
|
table lives in a different database and process liveness lives in the
|
|
kernel, so both can move while the transaction is open. These prove the
|
|
fence catches that movement and that nothing is ever committed against
|
|
evidence that changed.
|
|
"""
|
|
|
|
def setUp(self) -> None:
|
|
self.registry = _registry()
|
|
|
|
def _rows(self):
|
|
return self.registry.list_workers(status=None)
|
|
|
|
def _apply(self, plan, **kwargs):
|
|
kwargs.setdefault("liveness_fn", _safe_liveness)
|
|
kwargs.setdefault("plan_fn", lambda rows: _plan(rows, probe=_dead))
|
|
return self.registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=plan["candidate_worker_identities"],
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
**kwargs,
|
|
)
|
|
|
|
def _assert_nothing_committed(self, result):
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
self.assertTrue(
|
|
all(r["status"] == mwi.STATUS_ACTIVE for r in self._rows()),
|
|
"no row may be left retired after an aborted attempt",
|
|
)
|
|
|
|
# --- lease movement ---------------------------------------------------
|
|
|
|
def test_lease_acquired_between_plan_and_apply_preserves_its_worker(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s-1"))
|
|
plan = _plan(self._rows())
|
|
self.assertEqual(plan["candidate_worker_identities"], ["w-1"])
|
|
# A lease appears before apply: revalidation re-reads leases and the
|
|
# candidate set no longer matches what was approved.
|
|
result = self._apply(
|
|
plan,
|
|
plan_fn=lambda rows: _plan(
|
|
rows, probe=_dead, protected_session_ids=["s-1"]
|
|
),
|
|
)
|
|
self.assertEqual(result["outcome"], "candidate_set_moved")
|
|
self._assert_nothing_committed(result)
|
|
|
|
def test_lease_changing_during_apply_validation_aborts(self):
|
|
"""The fence moves *after* revalidation, mid per-target loop."""
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-2", pid=102, session="s2"))
|
|
plan = _plan(self._rows())
|
|
self.assertEqual(len(plan["candidate_worker_identities"]), 2)
|
|
|
|
fences = iter(["fence-1", "fence-1", "fence-CHANGED", "fence-CHANGED"])
|
|
|
|
result = self._apply(plan, external_fence_fn=lambda: next(fences))
|
|
self.assertEqual(result["outcome"], "external_state_moved")
|
|
self.assertFalse(result["success"])
|
|
self._assert_nothing_committed(result)
|
|
|
|
def test_unchanged_external_fence_allows_the_retirement(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
result = self._apply(plan, external_fence_fn=lambda: "stable-fence")
|
|
self.assertEqual(result["outcome"], "applied")
|
|
self.assertEqual(result["retired_count"], 1)
|
|
self.assertTrue(result["mutation_performed"])
|
|
|
|
def test_unreadable_external_state_aborts_rather_than_comparing_equal(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
|
|
def exploding_fence():
|
|
raise RuntimeError("lease store unavailable")
|
|
|
|
result = self._apply(plan, external_fence_fn=exploding_fence)
|
|
self.assertFalse(result["success"])
|
|
self._assert_nothing_committed(result)
|
|
|
|
# --- liveness movement ------------------------------------------------
|
|
|
|
def test_worker_becoming_active_before_the_write_is_preserved(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
|
|
def revived(row):
|
|
return {
|
|
"safe": False,
|
|
"reason_code": retire.REASON_PID_ALIVE,
|
|
"detail": "pid came back to life before the write",
|
|
}
|
|
|
|
result = self._apply(plan, liveness_fn=revived)
|
|
self.assertEqual(result["outcome"], "applied")
|
|
self._assert_nothing_committed(result)
|
|
self.assertEqual(
|
|
result["preserved"][0]["reason_code"], retire.REASON_PID_ALIVE
|
|
)
|
|
|
|
def test_pid_reuse_before_the_write_is_preserved(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
|
|
def reused(row):
|
|
return {
|
|
"safe": False,
|
|
"reason_code": retire.REASON_PID_REUSED,
|
|
"detail": "pid number reused by an unrelated process",
|
|
}
|
|
|
|
result = self._apply(plan, liveness_fn=reused)
|
|
self._assert_nothing_committed(result)
|
|
self.assertEqual(
|
|
result["preserved"][0]["reason_code"], retire.REASON_PID_REUSED
|
|
)
|
|
|
|
def test_host_or_boot_movement_before_the_write_is_preserved(self):
|
|
for reason in (retire.REASON_HOST_UNPROVEN, retire.REASON_BOOT_UNKNOWN):
|
|
with self.subTest(reason=reason):
|
|
registry = _registry()
|
|
_register(registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(registry.list_workers(status=None))
|
|
result = registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=plan["candidate_worker_identities"],
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
plan_fn=lambda rows: _plan(rows, probe=_dead),
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
liveness_fn=lambda row: {
|
|
"safe": False,
|
|
"reason_code": reason,
|
|
"detail": "fencing identity moved before the write",
|
|
},
|
|
)
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
|
|
def test_omitting_the_reprobe_retires_nothing(self):
|
|
"""A caller that supplies no pre-write probe gets no retirement."""
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
result = self._apply(plan, liveness_fn=None)
|
|
self._assert_nothing_committed(result)
|
|
self.assertEqual(
|
|
result["preserved"][0]["reason_code"], "liveness_reprobe_unavailable"
|
|
)
|
|
|
|
# --- registry movement ------------------------------------------------
|
|
|
|
def test_heartbeat_movement_retires_zero(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
with self.registry._tx() as conn:
|
|
conn.execute(
|
|
"UPDATE worker_registrations SET last_heartbeat_at = ? "
|
|
"WHERE worker_identity = ?",
|
|
(mwi._ts(NOW), "w-1"),
|
|
)
|
|
result = self._apply(plan)
|
|
self.assertEqual(result["outcome"], "registry_revision_moved")
|
|
self._assert_nothing_committed(result)
|
|
|
|
def test_generation_and_fencing_movement_retires_zero(self):
|
|
for column, value in (
|
|
("generation_id", "gen-MOVED"),
|
|
("fencing_epoch", 99),
|
|
("host_id", OTHER_HOST),
|
|
("boot_id", OTHER_BOOT),
|
|
("process_start_time", "start-MOVED"),
|
|
):
|
|
with self.subTest(column=column):
|
|
registry = _registry()
|
|
_register(registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(registry.list_workers(status=None))
|
|
with registry._tx() as conn:
|
|
conn.execute(
|
|
f"UPDATE worker_registrations SET {column} = ? "
|
|
"WHERE worker_identity = ?",
|
|
(value, "w-1"),
|
|
)
|
|
result = registry.retire_stale_workers(
|
|
expected_registry_fingerprint=plan["registry_fingerprint"],
|
|
expected_candidate_fingerprint=plan["candidate_fingerprint"],
|
|
worker_identities=plan["candidate_worker_identities"],
|
|
fingerprint_fn=retire.registry_fingerprint,
|
|
plan_fn=lambda rows: _plan(rows, probe=_dead),
|
|
retired_by="synthetic-user/prgs-reconciler",
|
|
now=NOW,
|
|
liveness_fn=_safe_liveness,
|
|
)
|
|
self.assertEqual(result["outcome"], "registry_revision_moved")
|
|
self.assertEqual(result["retired_count"], 0)
|
|
self.assertFalse(result["mutation_performed"])
|
|
|
|
def test_fencing_columns_are_part_of_the_cas_token(self):
|
|
base = [_row(identity="w-1", pid=101)]
|
|
for column, value in (
|
|
("host_id", OTHER_HOST),
|
|
("boot_id", OTHER_BOOT),
|
|
("process_start_time", "start-OTHER"),
|
|
):
|
|
with self.subTest(column=column):
|
|
moved = [_row(identity="w-1", pid=101, **{column: value})]
|
|
self.assertNotEqual(
|
|
retire.registry_fingerprint(base),
|
|
retire.registry_fingerprint(moved),
|
|
)
|
|
|
|
# --- concurrency and partial-failure ----------------------------------
|
|
|
|
def test_concurrent_applies_cannot_both_report_success(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
first = self._apply(plan)
|
|
second = self._apply(plan)
|
|
self.assertEqual(first["outcome"], "applied")
|
|
self.assertEqual(first["retired_count"], 1)
|
|
self.assertTrue(first["mutation_performed"])
|
|
# The second sees a moved registry: idempotent no-op, never a second
|
|
# claimed mutation.
|
|
self.assertIn(second["outcome"], {"already_retired", "registry_revision_moved"})
|
|
self.assertEqual(second["retired_count"], 0)
|
|
self.assertFalse(second["mutation_performed"])
|
|
|
|
def test_later_candidate_failure_rolls_back_the_earlier_one(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-2", pid=102, session="s2"))
|
|
plan = _plan(self._rows())
|
|
self.assertEqual(len(plan["candidate_worker_identities"]), 2)
|
|
seen = {"n": 0}
|
|
|
|
def fail_on_second(row):
|
|
seen["n"] += 1
|
|
if seen["n"] > 1:
|
|
raise sqlite3.OperationalError("database is locked")
|
|
return {"safe": True}
|
|
|
|
result = self._apply(plan, liveness_fn=fail_on_second)
|
|
self.assertFalse(result["success"])
|
|
self.assertEqual(result["outcome"], "transaction_failed")
|
|
self._assert_nothing_committed(result)
|
|
|
|
def test_database_failure_reports_no_mutation(self):
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
|
|
def exploding_plan(rows):
|
|
raise sqlite3.OperationalError("database is locked")
|
|
|
|
result = self._apply(plan, plan_fn=exploding_plan)
|
|
self.assertFalse(result["success"])
|
|
self.assertEqual(result["outcome"], "transaction_failed")
|
|
self._assert_nothing_committed(result)
|
|
|
|
def test_reported_counts_match_committed_state(self):
|
|
_register(self.registry, _row(identity="w-ok", pid=101, session="s1"))
|
|
_register(self.registry, _row(identity="w-hold", pid=102, session="s2"))
|
|
plan = _plan(self._rows())
|
|
|
|
def selective(row):
|
|
if row.get("worker_identity") == "w-hold":
|
|
return {
|
|
"safe": False,
|
|
"reason_code": retire.REASON_PID_ALIVE,
|
|
"detail": "came back before the write",
|
|
}
|
|
return {"safe": True}
|
|
|
|
result = self._apply(plan, liveness_fn=selective)
|
|
self.assertEqual(result["retired_count"], 1)
|
|
self.assertTrue(result["mutation_performed"])
|
|
committed = {r["worker_identity"]: r["status"] for r in self._rows()}
|
|
self.assertEqual(committed["w-ok"], mwi.STATUS_RETIRED)
|
|
self.assertEqual(committed["w-hold"], mwi.STATUS_ACTIVE)
|
|
self.assertEqual(
|
|
len(result["retired"]), result["retired_count"], "counts must agree"
|
|
)
|
|
|
|
def test_external_fence_is_captured_before_the_authoritative_read(self):
|
|
"""Ordering proof: BEGIN IMMEDIATE, then fence, then read."""
|
|
_register(self.registry, _row(identity="w-1", pid=101))
|
|
plan = _plan(self._rows())
|
|
order: list[str] = []
|
|
|
|
def fence():
|
|
order.append("fence")
|
|
return "stable"
|
|
|
|
def plan_fn(rows):
|
|
order.append("revalidate")
|
|
return _plan(rows, probe=_dead)
|
|
|
|
result = self._apply(plan, external_fence_fn=fence, plan_fn=plan_fn)
|
|
self.assertEqual(result["outcome"], "applied")
|
|
self.assertEqual(order[0], "fence")
|
|
self.assertIn("revalidate", order)
|
|
self.assertGreater(
|
|
order.count("fence"), 1, "the fence must be re-read before writes"
|
|
)
|
|
|
|
|
|
class ExternalStateFingerprintTests(unittest.TestCase):
|
|
"""The lease/liveness version token itself."""
|
|
|
|
def test_identical_lease_state_produces_one_token(self):
|
|
leases = [{"lease_id": "l1", "status": "active", "session_id": "s1"}]
|
|
self.assertEqual(
|
|
retire.external_state_fingerprint(leases),
|
|
retire.external_state_fingerprint(list(leases)),
|
|
)
|
|
|
|
def test_acquiring_a_lease_moves_the_token(self):
|
|
before = retire.external_state_fingerprint([])
|
|
after = retire.external_state_fingerprint(
|
|
[{"lease_id": "l1", "status": "active", "session_id": "s1"}]
|
|
)
|
|
self.assertNotEqual(before, after)
|
|
|
|
def test_owner_change_moves_the_token(self):
|
|
first = retire.external_state_fingerprint(
|
|
[{"lease_id": "l1", "status": "active", "session_id": "s1"}]
|
|
)
|
|
second = retire.external_state_fingerprint(
|
|
[{"lease_id": "l1", "status": "active", "session_id": "s2"}]
|
|
)
|
|
self.assertNotEqual(first, second)
|
|
|
|
def test_liveness_movement_moves_the_token(self):
|
|
first = retire.external_state_fingerprint([], liveness=[(101, False)])
|
|
second = retire.external_state_fingerprint([], liveness=[(101, True)])
|
|
self.assertNotEqual(first, second)
|
|
|
|
|
|
if __name__ == "__main__": # pragma: no cover
|
|
unittest.main()
|