Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 104 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

feat(fleet): add CAS-protected stale worker retirement capability

Browse Source 
Adds a sanctioned controller/reconciler capability that retires conclusively
stale MCP worker-registry rows through a dry-run-first, compare-and-swap
protected workflow (#980).

The #978 fleet snapshot is read-only, and its registry_revision is seeded with
snapshot_at at second precision, so a CAS gated on it can never pass. That
token is left unchanged; retirement gets its own stable registry_fingerprint
derived exclusively from canonical retirement-relevant registry content, plus a
candidate_fingerprint pinning the approved set. Identical contents observed at
different times produce the same token; row order never affects it; any
retirement-relevant change moves it.

WorkerRegistry.retire_stale_workers performs the whole decision inside one
BEGIN IMMEDIATE transaction: re-read rows, recompute the fingerprint from those
rows, recompute the eligibility plan from those rows (re-reading active workflow
leases), compare the candidate fingerprint, revalidate every target, then retire
each survivor with a guarded UPDATE asserting its status, heartbeat, generation,
session, fencing epoch, and pid are unchanged. Drift retires zero workers and
reports registry_revision_moved or candidate_set_moved; any exception rolls back
and reports transaction_failed, so a partial write is never reported as success.

Retirement requires a full conjunction: active status, complete registry fields,
parsable heartbeat, not live, pid_alive false, expired heartbeat, stale
ownership, canonical repository binding, no identity evidence shared with a live
or unprobeable worker, and no active workflow-lease ownership. Everything else
is preserved with a structured reason code. Retired rows become historical
rather than stale and keep their history; nothing is deleted.

Retirement does not repair untrusted live identity: live workers on legacy
instance identities are preserved and keep their legacy_incomplete_identity
blockers, so the result never claims the fleet became safe.

Exposed as gitea_plan_stale_worker_retirement and
gitea_apply_stale_worker_retirement, restricted to controller/reconciler role
kinds. The Gitea operation gate stays gitea.read because the mutation lands in
the local control-plane registry, matching the #601 lease lifecycle; no new
Gitea write permission is introduced and no author permission is broadened.

Tests: tests/test_issue_980_stale_worker_retirement.py (40 passed, 23 subtests),
including the regression test proving the old snapshot_at derivation moved the
token one second apart while the new registry CAS token does not. Full suite
from the branch worktree matches the master baseline exactly: 28 failed / 6206
passed vs 28 failed / 6166 passed, identical failure set.

Closes #980

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
sysadmin
2026-07-30 15:22:30 -04:00
co-authored by Claude Opus 4.8
parent 108cbfa173
commit c0c6d14b73
8 changed files with 2362 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gitea_mcp_server.py
+562
View File
@@ -19623,6 +19623,568 @@ def gitea_snapshot_instance_fleet(
return snapshot
# --- #980 CAS-protected stale worker retirement ---------------------------
def _retirement_role_block(task: str) -> dict | None:
"""Refuse the retirement surface to any non-controller/reconciler role.
Mirrors ``gitea_snapshot_instance_fleet``: ``gitea.read`` is the operation
gate (the mutation lands in the local worker registry, not in Gitea), and
the role restriction is what actually keeps author, reviewer, and merger
profiles out. No unrelated permission is granted to anyone.
"""
profile = get_profile()
role = _profile_role_kind(profile)
if role in {"controller", "reconciler"}:
return None
return {
"success": False,
"allowed": False,
"mutation_performed": False,
"retired_count": 0,
"denied_role": role,
"required_roles": ["controller", "reconciler"],
"requested_task": task,
"reasons": [
f"{task} is restricted to controller and reconciler roles; active "
f"role_kind is {role!r}. Author, reviewer, and merger profiles keep "
"gitea.read for diagnosis elsewhere but never receive worker-"
"registry retirement, and no unrelated mutation permission is "
"granted."
],
"exact_next_action": (
"Re-run from a prgs-controller or prgs-reconciler namespace."
),
}
def _retirement_runtime_block() -> list[str]:
"""Fail-closed runtime reasons that must stop a retirement apply (#980).
``gitea.read`` deliberately bypasses the #420 parity gate and the #615
stable-runtime gate, because a stale server may still be *inspected*. An
apply is a mutation, so both gates are re-asserted explicitly here rather
than inherited.
"""
reasons: list[str] = []
try:
parity = _current_master_parity()
except Exception as exc:
return [
f"master parity could not be assessed (fail closed): {_redact(str(exc))}"
]
if not parity.get("mutation_safe"):
reasons.append(
"runtime parity is not mutation-safe: "
f"{parity.get('summary') or 'stale runtime'}"
)
try:
reasons.extend(
stable_control_runtime.runtime_block_reasons(
_current_runtime_mode_report()
)
)
except Exception as exc:
reasons.append(
f"runtime mode could not be assessed (fail closed): {_redact(str(exc))}"
)
return reasons
def _retirement_protected_owners() -> tuple[dict | None, dict]:
"""Workflow owners that must never be retired as stale workers.
A registration whose process still owns an active control-plane lease is a
live workflow participant needing its own reconciliation, not a stale
orphan. Failure to enumerate leases is ambiguity, so it aborts rather than
proceeding with an empty protection set.
"""
db, errs = _control_plane_db_or_error()
if db is None:
return (
{
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": [
"active workflow leases could not be enumerated, so "
"protected owners are unknown (fail closed)",
*errs,
],
},
{},
)
try:
leases = db.list_leases(statuses=["active"], limit=1000)
except Exception as exc:
return (
{
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": [
"active workflow leases could not be enumerated, so "
"protected owners are unknown (fail closed): "
f"{_redact(str(exc))}"
],
},
{},
)
session_ids: set[str] = set()
pids: set[int] = set()
for lease in leases:
owner = lease.get("session_id") or lease.get("owner_session_id")
if owner:
session_ids.add(str(owner))
for key in ("owner_pid", "session_pid"):
value = lease.get(key)
if value is None:
continue
try:
pids.add(int(value))
except (TypeError, ValueError):
continue
return None, {
"session_ids": sorted(session_ids),
"pids": sorted(pids),
"active_lease_count": len(leases),
}
def _retirement_plan(records, canonical_repository: str, protected: dict) -> dict:
"""The single planning path shared by dry run and in-transaction revalidation."""
import mcp_fleet_retirement
return mcp_fleet_retirement.plan_stale_worker_retirement(
records,
pid_alive_probe=issue_lock_store.is_process_alive,
canonical_repository=canonical_repository,
protected_session_ids=(protected or {}).get("session_ids"),
protected_pids=(protected or {}).get("pids"),
)
def _retirement_revalidation_plan(records, canonical_repository: str) -> dict:
"""Revalidation planner used *inside* the retirement transaction.
Deliberately re-reads the active workflow leases rather than reusing the
set captured before the transaction opened: a lease acquired after planning
must still preserve its worker. An enumeration failure raises, which rolls
the transaction back and retires nothing.
"""
block, protected = _retirement_protected_owners()
if block:
raise RuntimeError(
"active workflow leases could not be re-read inside the retirement "
"transaction; refusing to retire anything"
)
return _retirement_plan(records, canonical_repository, protected)
@mcp.tool()
def gitea_plan_stale_worker_retirement(
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
canonical_repository: str | None = None,
) -> dict:
"""Read-only: authoritative retirement plan for stale worker rows (#980).
Controller and reconciler only. Reads the worker registry, classifies every
registration with the same assessor the #978 fleet snapshot uses, and
returns the exact set of registrations that are conclusively stale orphans
together with a **stable** ``registry_fingerprint`` and an exact
``candidate_fingerprint``.
The fingerprint is derived only from canonical retirement-relevant registry
content — never from ``snapshot_at``, wall-clock, request, or report time —
so two plans over an unchanged registry agree and the apply compare-and-swap
can actually pass. Row order never affects it.
Retires nothing. Live workers, workers whose PID cannot be probed, workers
with unparsable heartbeats, workers sharing identity evidence with a live or
unprobeable worker, foreign or unbound repositories, and workers that still
own an active workflow lease are all preserved with a reason code.
Args:
remote: Known instance — 'dadeschools' or 'prgs'.
host: Optional host override.
org: Optional org override (audit context only).
repo: Optional repo override (audit context only).
canonical_repository: Expected repository binding for
foreign-repository classification (defaults to the process root).
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"read_only": True,
"mutation_performed": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
role_block = _retirement_role_block("gitea_plan_stale_worker_retirement")
if role_block:
role_block["read_only"] = True
return role_block
registry = _worker_registry()
if registry is None:
return {
"success": False,
"read_only": True,
"mutation_performed": False,
"reasons": [
"worker registry is unavailable; cannot produce an authoritative "
"retirement plan (fail closed)"
],
"exact_next_action": (
"Ensure GITEA_WORKER_REGISTRY_DB is writable and re-run after "
"workers have registered."
),
}
protected_block, protected = _retirement_protected_owners()
if protected_block:
protected_block["read_only"] = True
return protected_block
try:
records = registry.list_workers(status=None)
except Exception as exc:
return {
"success": False,
"read_only": True,
"mutation_performed": False,
"reasons": [
f"failed to read worker registry: {type(exc).__name__}: "
f"{_redact(str(exc))}"
],
}
canon = canonical_repository or PROJECT_ROOT
plan = _retirement_plan(records, canon, protected)
profile = get_profile()
plan["role_kind"] = _profile_role_kind(profile)
plan["profile"] = profile.get("profile_name")
plan["remote"] = _effective_remote(remote)
plan["repository"] = {"org": org, "repo": repo, "canonical_repository": canon}
plan["protected_active_workflow_owners"] = protected
plan["apply_tool"] = "gitea_apply_stale_worker_retirement"
plan["permission_scope"] = {
"read_only": True,
"granted_operations": ["gitea.read"],
"denied_unrelated_mutations": True,
"note": (
"Planning is strictly observational. It does not authorize branch, "
"issue, PR, review, merge, or restart mutations, and it retires "
"nothing."
),
}
plan["exact_next_action"] = (
"Pass registry_fingerprint, candidate_fingerprint, and the exact "
"candidate_worker_identities to gitea_apply_stale_worker_retirement."
if plan.get("candidate_count")
else "No registration is conclusively stale; nothing to apply."
)
return plan
@mcp.tool()
def gitea_apply_stale_worker_retirement(
registry_fingerprint: str,
candidate_fingerprint: str,
worker_identities: list | str,
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
canonical_repository: str | None = None,
) -> dict:
"""Retire conclusively stale worker registrations under CAS (#980).
Controller and reconciler only. Requires the exact ``registry_fingerprint``
and ``candidate_fingerprint`` returned by
``gitea_plan_stale_worker_retirement`` plus the exact candidate identity
list. A matching token is necessary but never sufficient: inside a single
``BEGIN IMMEDIATE`` transaction the registry is re-read, the fingerprint is
recomputed from those rows, the eligibility plan is recomputed from those
rows, and every target is independently revalidated immediately before its
own guarded ``UPDATE``. Any drift retires zero workers and reports
``registry_revision_moved`` or ``candidate_set_moved``.
Retiring stale rows does not repair untrusted live identity. Live workers
registered under legacy ``pid-``/``proc-`` instance identities remain
untouched and their ``legacy_incomplete_identity`` blockers remain
outstanding, so the result never claims the fleet became safe.
Args:
registry_fingerprint: Stable token from the plan (CAS expectation).
candidate_fingerprint: Exact candidate-set token from the plan.
worker_identities: The exact candidate identities the plan returned
(list, or a JSON / comma-separated string).
remote: Known instance — 'dadeschools' or 'prgs'.
host: Optional host override.
org: Optional org override (audit context only).
repo: Optional repo override (audit context only).
canonical_repository: Expected repository binding (defaults to the
process root); must match the value the plan used.
"""
import json as _json
import mcp_fleet_retirement
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
role_block = _retirement_role_block("gitea_apply_stale_worker_retirement")
if role_block:
return role_block
runtime_reasons = _retirement_runtime_block()
if runtime_reasons:
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"blocker_kind": "runtime_not_mutation_safe",
"reasons": runtime_reasons,
"exact_next_action": (
"Restore runtime parity on the stable control checkout, then "
"re-plan and re-apply."
),
}
# Fresh identity + capability resolution immediately before mutation.
profile = get_profile()
role_kind = _profile_role_kind(profile)
required_permission = task_capability_map.required_permission(
"apply_stale_worker_retirement"
)
required_role = task_capability_map.required_role(
"apply_stale_worker_retirement"
)
permission_ok, permission_reason = gitea_config.check_operation(
required_permission,
profile.get("allowed_operations") or [],
profile.get("forbidden_operations") or [],
)
if not permission_ok:
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"requested_task": "apply_stale_worker_retirement",
"required_operation_permission": required_permission,
"required_role_kind": required_role,
"reasons": [
"capability resolution immediately before apply refused this "
f"session: {permission_reason}"
],
}
try:
resolved_host = host or REMOTES[_effective_remote(remote)]["host"]
authenticated_username = _authenticated_username(resolved_host)
except Exception:
authenticated_username = None
registry = _worker_registry()
if registry is None:
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": [
"worker registry is unavailable; refusing to retire anything "
"(fail closed)"
],
}
raw = worker_identities
if isinstance(raw, str):
text = raw.strip()
try:
raw = _json.loads(text)
except Exception:
raw = [part.strip() for part in text.split(",") if part.strip()]
if isinstance(raw, str):
raw = [raw]
if not isinstance(raw, list):
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": ["worker_identities must be a list of worker identities"],
}
targets = [str(item).strip() for item in raw if str(item).strip()]
protected_block, protected = _retirement_protected_owners()
if protected_block:
return protected_block
# #948 daemon-cohort uniqueness: a contested generation or a reused worker
# identity means ownership is ambiguous fleet-wide, so retire nothing.
try:
cohort = mcp_worker_identity.classify_cohort(
registry.list_workers(status=mcp_worker_identity.STATUS_ACTIVE),
pid_alive_probe=issue_lock_store.is_process_alive,
)
except Exception as exc:
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"reasons": [
"daemon-cohort uniqueness could not be assessed (fail closed): "
f"{_redact(str(exc))}"
],
}
if cohort.get("blocked"):
return {
"success": False,
"mutation_performed": False,
"retired_count": 0,
"blocker_kind": cohort.get("blocker_kind"),
"reasons": [
"daemon-cohort uniqueness failed; worker ownership is contested",
*(cohort.get("reasons") or []),
],
"cohort": {
"blocked_worker_identities": cohort.get("blocked_worker_identities"),
"duplicate_identities": cohort.get("duplicate_identities"),
"contested_generations": cohort.get("contested_generations"),
},
}
canon = canonical_repository or PROJECT_ROOT
acting = "/".join(
part
for part in (authenticated_username, profile.get("profile_name"))
if part
) or "unknown"
result = registry.retire_stale_workers(
expected_registry_fingerprint=registry_fingerprint,
expected_candidate_fingerprint=candidate_fingerprint,
worker_identities=targets,
fingerprint_fn=mcp_fleet_retirement.registry_fingerprint,
plan_fn=lambda rows: _retirement_revalidation_plan(rows, canon),
retired_by=acting,
retirement_reason=mcp_fleet_retirement.REASON_ELIGIBLE,
)
result["role_kind"] = role_kind
result["profile"] = profile.get("profile_name")
result["remote"] = _effective_remote(remote)
result["repository"] = {"org": org, "repo": repo, "canonical_repository": canon}
result["protected_active_workflow_owners"] = protected
result["permission_scope"] = {
"granted_operations": ["gitea.read"],
"control_plane_mutation": "worker_registrations.status -> retired",
"denied_unrelated_mutations": True,
"note": (
"This capability retires local worker-registry rows only. It grants "
"no branch, issue, PR, review, merge, or restart authority, and it "
"never kills or restarts a process."
),
}
result["post_apply"] = _retirement_post_apply(
registry, result, canonical_repository=canon
)
try:
gitea_audit.write_event(
gitea_audit.build_event(
action="gitea_apply_stale_worker_retirement",
result=(
gitea_audit.SUCCEEDED
if result.get("mutation_performed")
else gitea_audit.BLOCKED
if not result.get("success")
else gitea_audit.ALLOWED
),
remote=_effective_remote(remote),
repository=canon,
profile_name=profile.get("profile_name"),
audit_label=profile.get("audit_label"),
authenticated_username=authenticated_username,
task_role=role_kind,
operation="worker_registry.retire_stale_workers",
reason=result.get("outcome"),
request_metadata=mcp_fleet_retirement.summarize_plan(result),
)
)
except Exception:
pass
return result
def _retirement_post_apply(
registry, result: dict, *, canonical_repository: str
) -> dict:
"""Fresh fleet verification after a retirement attempt (#980 requirement 6)."""
import mcp_fleet_retirement
import mcp_fleet_snapshot as _fleet_after
try:
after_records = registry.list_workers(status=None)
after = _fleet_after.snapshot_instance_fleet(
after_records,
pid_alive_probe=issue_lock_store.is_process_alive,
canonical_repository=canonical_repository,
)
except Exception as exc:
return {
"available": False,
"reasons": [
f"post-apply verification could not be produced: {_redact(str(exc))}"
],
}
retired_ids = {
str(r.get("worker_identity")) for r in result.get("retired") or []
}
return {
"available": True,
"registry_fingerprint": mcp_fleet_retirement.registry_fingerprint(
after_records
),
"live_worker_count": after.get("live_worker_count"),
"stale_worker_count": after.get("stale_worker_count"),
"historical_worker_count": after.get("historical_worker_count"),
"retired_still_counted_live": sorted(
str(w.get("worker_identity"))
for w in after.get("live_workers") or []
if str(w.get("worker_identity")) in retired_ids
),
"retired_still_counted_stale": sorted(
str(w.get("worker_identity"))
for w in after.get("stale_workers") or []
if str(w.get("worker_identity")) in retired_ids
),
"live_fleet_safe": after.get("live_fleet_safe"),
"remaining_blockers": [
{"classification": f.get("classification"), "detail": f.get("detail")}
for f in after.get("active_blockers") or []
],
"note": (
"Stale retirement does not repair untrusted live identity; residual "
"legacy_incomplete_identity blockers keep live_fleet_safe false and "
"that is a truthful result."
),
}
@mcp.tool()
def gitea_get_runtime_context(
remote: str = "dadeschools",