feat(fleet): add CAS-protected stale worker retirement capability

Adds a sanctioned controller/reconciler capability that retires conclusively
stale MCP worker-registry rows through a dry-run-first, compare-and-swap
protected workflow (#980).

The #978 fleet snapshot is read-only, and its registry_revision is seeded with
snapshot_at at second precision, so a CAS gated on it can never pass. That
token is left unchanged; retirement gets its own stable registry_fingerprint
derived exclusively from canonical retirement-relevant registry content, plus a
candidate_fingerprint pinning the approved set. Identical contents observed at
different times produce the same token; row order never affects it; any
retirement-relevant change moves it.

WorkerRegistry.retire_stale_workers performs the whole decision inside one
BEGIN IMMEDIATE transaction: re-read rows, recompute the fingerprint from those
rows, recompute the eligibility plan from those rows (re-reading active workflow
leases), compare the candidate fingerprint, revalidate every target, then retire
each survivor with a guarded UPDATE asserting its status, heartbeat, generation,
session, fencing epoch, and pid are unchanged. Drift retires zero workers and
reports registry_revision_moved or candidate_set_moved; any exception rolls back
and reports transaction_failed, so a partial write is never reported as success.

Retirement requires a full conjunction: active status, complete registry fields,
parsable heartbeat, not live, pid_alive false, expired heartbeat, stale
ownership, canonical repository binding, no identity evidence shared with a live
or unprobeable worker, and no active workflow-lease ownership. Everything else
is preserved with a structured reason code. Retired rows become historical
rather than stale and keep their history; nothing is deleted.

Retirement does not repair untrusted live identity: live workers on legacy
instance identities are preserved and keep their legacy_incomplete_identity
blockers, so the result never claims the fleet became safe.

Exposed as gitea_plan_stale_worker_retirement and
gitea_apply_stale_worker_retirement, restricted to controller/reconciler role
kinds. The Gitea operation gate stays gitea.read because the mutation lands in
the local control-plane registry, matching the #601 lease lifecycle; no new
Gitea write permission is introduced and no author permission is broadened.

Tests: tests/test_issue_980_stale_worker_retirement.py (40 passed, 23 subtests),
including the regression test proving the old snapshot_at derivation moved the
token one second apart while the new registry CAS token does not. Full suite
from the branch worktree matches the master baseline exactly: 28 failed / 6206
passed vs 28 failed / 6166 passed, identical failure set.

Closes #980

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

docs/instance-fleet-identity.md

@@ -189,6 +189,11 @@ mutation. Diagnostic reads remain available where `gitea.read` allows.
 * `mcp_fleet_snapshot` — pure snapshot + classification (#978).
 * `gitea_snapshot_instance_fleet` — sanctioned MCP tool (#978).
 * `gitea_get_runtime_context` — single-process view (not fleet-wide).
+* [stale-worker-retirement.md](stale-worker-retirement.md) — CAS-protected
+  retirement of conclusively stale registrations (#980). Note that the
+  `registry_revision` this snapshot returns is **time-seeded** and unsuitable
+  for compare-and-swap; retirement derives its own stable
+  `registry_fingerprint` from registry content alone.
 
 ## Non-goals
 

docs/mcp-tool-inventory.md

@@ -51,6 +51,7 @@ that gates each call, not which tools exist.
 - `gitea_adopt_merger_pr_lease`
 - `gitea_adopt_workflow_lease`
 - `gitea_allocate_next_work`
+- `gitea_apply_stale_worker_retirement`
 - `gitea_assess_already_landed_reconciliation`
 - `gitea_assess_conflict_fix_classification`
 - `gitea_assess_conflict_fix_push`
@@ -124,6 +125,7 @@ that gates each call, not which tools exist.
 - `gitea_observability_link_issue`
 - `gitea_observability_list_projects`
 - `gitea_observability_reconcile_incident`
+- `gitea_plan_stale_worker_retirement`
 - `gitea_post_heartbeat`
 - `gitea_publish_unpublished_issue_branch`
 - `gitea_quarantine_contaminated_review`

docs/stale-worker-retirement.md

@@ -0,0 +1,203 @@
+# Stale worker retirement (#980)
+
+The #978 instance-fleet snapshot made registry accuracy observable but
+deliberately read-only: a registry full of rows whose owning processes are long
+gone stays full. This document describes the sanctioned way to retire those
+rows — a dry-run-first, compare-and-swap-protected workflow available only to
+controller and reconciler namespaces.
+
+Related: [instance-fleet-identity.md](instance-fleet-identity.md) (#978),
+[post-restart-reconcile.md](post-restart-reconcile.md) (#662).
+
+## Why a dedicated registry token
+
+`mcp_fleet_snapshot._consistency_token` seeds its digest with `snapshot_at`,
+formatted at second precision. Its output — surfaced as `registry_revision` and
+`consistency_token` on the snapshot — therefore changes on **every call**, even
+when no registry row changed. Any compare-and-swap gated on it can never pass:
+a dry-run/apply cycle spanning more than one second aborts unconditionally.
+
+That token remains useful as an observation stamp and is unchanged. #980 adds a
+separate, *stable* token instead:
+
+| Token | Module | Derived from | Stable across time? |
+| --- | --- | --- | --- |
+| `registry_revision` / `consistency_token` | `mcp_fleet_snapshot` | `snapshot_at` + a subset of row fields | **No** — moves every second |
+| `registry_fingerprint` | `mcp_fleet_retirement` | canonical retirement-relevant row content only | **Yes** |
+| `candidate_fingerprint` | `mcp_fleet_retirement` | canonical content of the selected candidate rows | **Yes** |
+
+`registry_fingerprint` guarantees:
+
+* identical canonical registry contents always produce the same token, whenever
+  they are observed;
+* row iteration order never affects the token (serialized rows are sorted);
+* any retirement-relevant change moves it — row creation or deletion, identity
+  change, heartbeat or TTL change, ownership change, registration-state change,
+  PID change, or repository-binding change.
+
+The exact field set is `mcp_fleet_retirement.FINGERPRINT_FIELDS`. Deliberately
+excluded: `token_fingerprint` (credential-adjacent, never a retirement input),
+the four `*_revision` columns (revision drift is an independent restart concern
+and is not part of the eligibility conjunction), and the `retired_*` bookkeeping
+columns this feature adds. Numeric values are canonicalised, so a TTL that
+round-trips through SQLite as `900.0` hashes identically to `900`.
+
+## Eligibility — the conjunction
+
+A registration is retired only when **every** one of these holds. Any missing
+or contradictory evidence preserves the row.
+
+| Requirement | Preserve reason code when it fails |
+| --- | --- |
+| `status` is `active` | `already_terminal_registration` |
+| Every field the conjunction reads is present (`REQUIRED_IDENTITY_FIELDS`) | `incomplete_registry_identity` |
+| `last_heartbeat_at` parses as a UTC stamp | `unparsable_heartbeat` |
+| Worker is not live | `worker_live` |
+| PID probe returns a definite answer | `pid_liveness_unknown` |
+| PID probe says the process is gone | `pid_alive` |
+| Heartbeat has expired under the canonical TTL | `heartbeat_not_expired` |
+| `ownership_state` is exactly `stale` | `ambiguous_ownership_state` |
+| Repository binding present and canonical | `repository_binding_ambiguous` |
+| No identity evidence shared with a live or unprobeable worker | `conflicting_identity_evidence` |
+| Not an active workflow-lease owner | `protected_active_workflow_owner` |
+
+Eligible rows carry `eligible_stale_orphan`.
+
+Two properties are worth stating explicitly:
+
+* **`pid_alive` can only withdraw liveness, never grant it**
+  (`WorkerRegistry.is_live`, #948 AC7). A heartbeat-lapsed but still-running
+  process therefore classifies as `stale` in the snapshot, yet #980's added
+  `pid_alive is False` requirement preserves it. An unprobeable PID (`None`)
+  also fails closed.
+* **Trusted launcher identity is not required.** #980 places trusted
+  `client_instance_id` propagation out of scope and lists backfilling trusted
+  identity for legacy workers as a non-goal. Requiring `inst-…` provenance here
+  would preserve every legacy row forever and make the feature inert. What is
+  required is that the registry *fields* the conjunction reads are present.
+
+Multiple processes belonging to one legitimate worker cohort are not treated as
+multiple independent workers: the fleet model from #948/#978 is preserved
+unchanged, and sharing a role or profile is never a duplicate.
+
+## Tools
+
+### `gitea_plan_stale_worker_retirement`
+
+Read-only. Controller and reconciler only.
+
+| Parameter | Meaning |
+| --- | --- |
+| `remote` | `dadeschools` or `prgs` |
+| `host`, `org`, `repo` | Optional overrides (audit context) |
+| `canonical_repository` | Expected repository binding; defaults to the process root |
+
+Returns `registry_fingerprint`, `candidate_fingerprint`,
+`candidate_worker_identities`, per-worker `candidates` and `preserved` entries
+(each with `reason_code`, `detail`, and structured `evidence`),
+`preserved_reason_counts`, `assessed_count`, `candidate_count`,
+`preserved_count`, and `protected_active_workflow_owners`.
+`mutation_performed` is always `false` and `read_only` is always `true`.
+
+Planning is deterministic: the same authoritative registry contents produce the
+same plan and the same tokens regardless of when they are observed.
+
+### `gitea_apply_stale_worker_retirement`
+
+Mutating. Controller and reconciler only.
+
+| Parameter | Meaning |
+| --- | --- |
+| `registry_fingerprint` | The exact stable token the plan returned |
+| `candidate_fingerprint` | The exact candidate-set token the plan returned |
+| `worker_identities` | The exact candidate identities (list, or JSON / comma-separated string) |
+| `remote`, `host`, `org`, `repo` | As above |
+| `canonical_repository` | Must match the value the plan used |
+
+Before touching the registry, apply fails closed on: profile permission, role
+kind, master parity (`mutation_safe`), stable-runtime mode, capability
+resolution refreshed immediately before mutation, worker-registry availability,
+workflow-lease enumeration failure, and daemon-cohort uniqueness
+(`classify_cohort`).
+
+A matching token is necessary but never sufficient. Inside one
+`BEGIN IMMEDIATE` transaction (`WorkerRegistry.retire_stale_workers`) the
+server:
+
+1. re-reads the authoritative rows;
+2. recomputes `registry_fingerprint` from *those* rows and compares — a mismatch
+   returns `registry_revision_moved` with `retired_count: 0` and no write;
+3. recomputes the eligibility plan from *those* rows — re-reading the active
+   workflow leases rather than reusing the set captured before the transaction
+   opened, so a lease acquired after planning still preserves its worker — and
+   compares `candidate_fingerprint`. A mismatch returns `candidate_set_moved`
+   with `retired_count: 0` and no write; a lease-enumeration failure raises and
+   rolls the transaction back;
+4. revalidates every requested identity against that fresh plan;
+5. retires each survivor with a guarded `UPDATE` that additionally asserts
+   `status`, `last_heartbeat_at`, `generation_id`, `session_id`,
+   `fencing_epoch`, and `pid` are unchanged. A guard that matches no row
+   preserves the worker with `row_changed_since_plan`.
+
+There is no window between a safety check and its matching write, so a worker
+that comes back to life, changes ownership, or is retired concurrently cannot be
+removed on the strength of a stale observation. Any exception — including a
+commit failure — rolls the whole transaction back and returns
+`transaction_failed` with `success: false`, `retired_count: 0`, and
+`mutation_performed: false`; a partial write can never be reported as success.
+
+### Outcomes
+
+| `outcome` | Meaning | `mutation_performed` |
+| --- | --- | --- |
+| `planned` | Dry-run result | `false` |
+| `applied` | Transaction ran; see `retired` / `preserved` | `true` only if something was retired |
+| `registry_revision_moved` | Registry changed between plan and apply | `false` |
+| `candidate_set_moved` | Eligibility verdict changed between plan and apply | `false` |
+| `already_retired` | Every requested row is already retired (idempotent replay) | `false` |
+| `nothing_requested` | Empty target list | `false` |
+| `transaction_failed` | Rolled back; nothing retired | `false` |
+
+## What retirement does to the fleet snapshot
+
+A retired row keeps its history: `status` moves to `retired` and `retired_at`,
+`retired_by`, `retirement_reason` are recorded. Nothing is deleted. Because
+`retired` is not `active`, the #978 snapshot counts the row as **historical**,
+not stale, so `stale_worker_count` falls and historical rows never make the live
+fleet unsafe by themselves.
+
+**Retirement does not repair untrusted live identity.** Live workers registered
+under legacy `pid-`/`proc-` instance identities are preserved untouched and
+keep their `legacy_incomplete_identity` blockers. Retiring every stale row can
+therefore legitimately produce:
+
+* `stale_worker_count: 0`
+* residual live `legacy_incomplete_identity` blockers
+* `live_fleet_safe: false`
+
+That is a truthful result, and the `post_apply` block reports the remaining
+blockers rather than claiming the fleet became safe. Trusted
+`client_instance_id` propagation through launchers is a separate enrollment
+problem.
+
+## Permissions
+
+* **Allowed:** `controller`, `reconciler`.
+* **Denied:** author, reviewer, merger — they keep `gitea.read` for diagnosis
+  elsewhere and are refused this surface by role.
+* The Gitea operation gate stays `gitea.read` because the mutation lands in the
+  **local control-plane worker registry**, not in Gitea — the same model the
+  #601 lease lifecycle uses. **No new Gitea write permission is introduced for
+  any profile**, and no author permission is broadened.
+* The fleet snapshot remains observational: nothing here turns it into a gate on
+  ordinary author work.
+
+## Non-goals
+
+* Killing or restarting processes.
+* Editing session files or configuration.
+* Direct database cleanup outside the sanctioned transaction.
+* Retiring live workers.
+* Backfilling trusted identity for legacy workers.
+* Rewriting worker ownership.
+* Cleaning unrelated workflow-lease or issue-claim registries.