Gitea-Tools/docs/stale-worker-retirement.md

Stale worker retirement (#980)

The #978 instance-fleet snapshot made registry accuracy observable but deliberately read-only: a registry full of rows whose owning processes are long gone stays full. This document describes the sanctioned way to retire those rows — a dry-run-first, compare-and-swap-protected workflow available only to controller and reconciler namespaces.

Related: instance-fleet-identity.md (#978), post-restart-reconcile.md (#662).

Why a dedicated registry token

mcp_fleet_snapshot._consistency_token seeds its digest with snapshot_at, formatted at second precision. Its output — surfaced as registry_revision and consistency_token on the snapshot — therefore changes on every call, even when no registry row changed. Any compare-and-swap gated on it can never pass: a dry-run/apply cycle spanning more than one second aborts unconditionally.

That token remains useful as an observation stamp and is unchanged. #980 adds a separate, stable token instead:

Token	Module	Derived from	Stable across time?
registry_revision / consistency_token	mcp_fleet_snapshot	snapshot_at + a subset of row fields	No — moves every second
registry_fingerprint	mcp_fleet_retirement	canonical retirement-relevant row content only	Yes
candidate_fingerprint	mcp_fleet_retirement	canonical content of the selected candidate rows	Yes

registry_fingerprint guarantees:

• identical canonical registry contents always produce the same token, whenever they are observed;
• row iteration order never affects the token (serialized rows are sorted);
• any retirement-relevant change moves it — row creation or deletion, identity change, heartbeat or TTL change, ownership change, registration-state change, PID change, or repository-binding change.

The exact field set is mcp_fleet_retirement.FINGERPRINT_FIELDS. Deliberately excluded: token_fingerprint (credential-adjacent, never a retirement input), the four *_revision columns (revision drift is an independent restart concern and is not part of the eligibility conjunction), and the retired_* bookkeeping columns this feature adds. Numeric values are canonicalised, so a TTL that round-trips through SQLite as 900.0 hashes identically to 900.

Eligibility — the conjunction

A registration is retired only when every one of these holds. Any missing or contradictory evidence preserves the row.

Requirement	Preserve reason code when it fails
status is active	already_terminal_registration
Every field the conjunction reads is present (REQUIRED_IDENTITY_FIELDS)	incomplete_registry_identity
last_heartbeat_at parses as a UTC stamp	unparsable_heartbeat
Worker is not live	worker_live
PID probe returns a definite answer	pid_liveness_unknown
PID probe says the process is gone	pid_alive
Heartbeat has expired under the canonical TTL	heartbeat_not_expired
ownership_state is exactly stale	ambiguous_ownership_state
Repository binding present and canonical	repository_binding_ambiguous
No identity evidence shared with a live or unprobeable worker	conflicting_identity_evidence
Not an active workflow-lease owner	protected_active_workflow_owner

Eligible rows carry eligible_stale_orphan.

Two properties are worth stating explicitly:

• pid_alive can only withdraw liveness, never grant it (WorkerRegistry.is_live, #948 AC7). A heartbeat-lapsed but still-running process therefore classifies as stale in the snapshot, yet #980's added pid_alive is False requirement preserves it. An unprobeable PID (None) also fails closed.
• Trusted launcher identity is not required. #980 places trusted client_instance_id propagation out of scope and lists backfilling trusted identity for legacy workers as a non-goal. Requiring inst-… provenance here would preserve every legacy row forever and make the feature inert. What is required is that the registry fields the conjunction reads are present.

Multiple processes belonging to one legitimate worker cohort are not treated as multiple independent workers: the fleet model from #948/#978 is preserved unchanged, and sharing a role or profile is never a duplicate.

Tools

gitea_plan_stale_worker_retirement

Read-only. Controller and reconciler only.

Parameter	Meaning
remote	dadeschools or prgs
host, org, repo	Optional overrides (audit context)
canonical_repository	Expected repository binding; defaults to the process root

Returns registry_fingerprint, candidate_fingerprint, candidate_worker_identities, per-worker candidates and preserved entries (each with reason_code, detail, and structured evidence), preserved_reason_counts, assessed_count, candidate_count, preserved_count, and protected_active_workflow_owners. mutation_performed is always false and read_only is always true.

Planning is deterministic: the same authoritative registry contents produce the same plan and the same tokens regardless of when they are observed.

gitea_apply_stale_worker_retirement

Mutating. Controller and reconciler only.

Parameter	Meaning
registry_fingerprint	The exact stable token the plan returned
candidate_fingerprint	The exact candidate-set token the plan returned
worker_identities	The exact candidate identities (list, or JSON / comma-separated string)
remote, host, org, repo	As above
canonical_repository	Must match the value the plan used

Before touching the registry, apply fails closed on: profile permission, role kind, master parity (mutation_safe), stable-runtime mode, capability resolution refreshed immediately before mutation, worker-registry availability, workflow-lease enumeration failure, and daemon-cohort uniqueness (classify_cohort).

A matching token is necessary but never sufficient. Inside one BEGIN IMMEDIATE transaction (WorkerRegistry.retire_stale_workers) the server:

1. re-reads the authoritative rows;
2. recomputes registry_fingerprint from those rows and compares — a mismatch returns registry_revision_moved with retired_count: 0 and no write;
3. recomputes the eligibility plan from those rows — re-reading the active workflow leases rather than reusing the set captured before the transaction opened, so a lease acquired after planning still preserves its worker — and compares candidate_fingerprint. A mismatch returns candidate_set_moved with retired_count: 0 and no write; a lease-enumeration failure raises and rolls the transaction back;
4. revalidates every requested identity against that fresh plan;
5. retires each survivor with a guarded UPDATE that additionally asserts status, last_heartbeat_at, generation_id, session_id, fencing_epoch, and pid are unchanged. A guard that matches no row preserves the worker with row_changed_since_plan.

There is no window between a safety check and its matching write, so a worker that comes back to life, changes ownership, or is retired concurrently cannot be removed on the strength of a stale observation. Any exception — including a commit failure — rolls the whole transaction back and returns transaction_failed with success: false, retired_count: 0, and mutation_performed: false; a partial write can never be reported as success.

Outcomes

outcome	Meaning	mutation_performed
planned	Dry-run result	false
applied	Transaction ran; see retired / preserved	true only if something was retired
registry_revision_moved	Registry changed between plan and apply	false
candidate_set_moved	Eligibility verdict changed between plan and apply	false
already_retired	Every requested row is already retired (idempotent replay)	false
nothing_requested	Empty target list	false
transaction_failed	Rolled back; nothing retired	false

What retirement does to the fleet snapshot

A retired row keeps its history: status moves to retired and retired_at, retired_by, retirement_reason are recorded. Nothing is deleted. Because retired is not active, the #978 snapshot counts the row as historical, not stale, so stale_worker_count falls and historical rows never make the live fleet unsafe by themselves.

Retirement does not repair untrusted live identity. Live workers registered under legacy pid-/proc- instance identities are preserved untouched and keep their legacy_incomplete_identity blockers. Retiring every stale row can therefore legitimately produce:

• stale_worker_count: 0
• residual live legacy_incomplete_identity blockers
• live_fleet_safe: false

That is a truthful result, and the post_apply block reports the remaining blockers rather than claiming the fleet became safe. Trusted client_instance_id propagation through launchers is a separate enrollment problem.

Permissions

• Allowed: controller, reconciler.
• Denied: author, reviewer, merger — they keep gitea.read for diagnosis elsewhere and are refused this surface by role.
• The Gitea operation gate stays gitea.read because the mutation lands in the local control-plane worker registry, not in Gitea — the same model the #601 lease lifecycle uses. No new Gitea write permission is introduced for any profile, and no author permission is broadened.
• The fleet snapshot remains observational: nothing here turns it into a gate on ordinary author work.

Non-goals

• Killing or restarting processes.
• Editing session files or configuration.
• Direct database cleanup outside the sanctioned transaction.
• Retiring live workers.
• Backfilling trusted identity for legacy workers.
• Rewriting worker ownership.
• Cleaning unrelated workflow-lease or issue-claim registries.