Gitea-Tools/docs/stale-worker-retirement.md

Stale worker retirement (#980)

The #978 instance-fleet snapshot made registry accuracy observable but deliberately read-only: a registry full of rows whose owning processes are long gone stays full. This document describes the sanctioned way to retire those rows — a dry-run-first, compare-and-swap-protected workflow available only to controller and reconciler namespaces.

Related: instance-fleet-identity.md (#978), post-restart-reconcile.md (#662).

Why a dedicated registry token

mcp_fleet_snapshot._consistency_token seeds its digest with snapshot_at, formatted at second precision. Its output — surfaced as registry_revision and consistency_token on the snapshot — therefore changes on every call, even when no registry row changed. Any compare-and-swap gated on it can never pass: a dry-run/apply cycle spanning more than one second aborts unconditionally.

That token remains useful as an observation stamp and is unchanged. #980 adds a separate, stable token instead:

Token	Module	Derived from	Stable across time?
registry_revision / consistency_token	mcp_fleet_snapshot	snapshot_at + a subset of row fields	No — moves every second
registry_fingerprint	mcp_fleet_retirement	canonical retirement-relevant row content only	Yes
candidate_fingerprint	mcp_fleet_retirement	canonical content of the selected candidate rows	Yes

registry_fingerprint guarantees:

• identical canonical registry contents always produce the same token, whenever they are observed;
• row iteration order never affects the token (serialized rows are sorted);
• any retirement-relevant change moves it — row creation or deletion, identity change, heartbeat or TTL change, ownership change, registration-state change, PID change, or repository-binding change.

The exact field set is mcp_fleet_retirement.FINGERPRINT_FIELDS. Deliberately excluded: token_fingerprint (credential-adjacent, never a retirement input), the four *_revision columns (revision drift is an independent restart concern and is not part of the eligibility conjunction), and the retired_* bookkeeping columns this feature adds. Numeric values are canonicalised, so a TTL that round-trips through SQLite as 900.0 hashes identically to 900.

Eligibility — the conjunction

A registration is retired only when every one of these holds. Any missing or contradictory evidence preserves the row.

Requirement	Preserve reason code when it fails
status is active	already_terminal_registration
Every field the conjunction reads is present (REQUIRED_IDENTITY_FIELDS)	incomplete_registry_identity
last_heartbeat_at parses as a UTC stamp	unparsable_heartbeat
Worker is not live	worker_live
PID probe returns a definite answer	pid_liveness_unknown
PID probe says the process is gone	pid_alive
Heartbeat has expired under the canonical TTL	heartbeat_not_expired
ownership_state is exactly stale	ambiguous_ownership_state
Repository binding present and canonical	repository_binding_ambiguous
No identity evidence shared with a live or unprobeable worker	conflicting_identity_evidence
No other active row claims the same (instance, namespace) while one may be live	client_instance_conflict
Not an active workflow-lease owner	protected_active_workflow_owner
Instance identity is launcher-minted (inst-…)	untrusted_identity_provenance
Row's host_id matches the host running retirement	host_binding_unproven
Boot identity is known on both sides	boot_identity_unknown
The pid number is not occupied by a different incarnation	pid_reuse_detected

Eligible rows carry eligible_stale_orphan.

Three properties are worth stating explicitly:

• pid_alive can only withdraw liveness, never grant it (WorkerRegistry.is_live, #948 AC7). A heartbeat-lapsed but still-running process therefore classifies as stale in the snapshot, yet #980's added pid_alive is False requirement preserves it. An unprobeable PID (None) also fails closed.

• Affirmative identity proof is required (review 657 B2). An earlier revision required only that the pre-existing registry columns were non-null — which a legacy legacy-pid-… row satisfies trivially, so a row that proved nothing about which process it described was retireable. Retirement now needs both halves of a positive proof:

  • Attribution — a launcher-minted inst-… client_instance_id, so the row is known to belong to one specific application launch rather than having been inferred from pid proximity.
  • Fencing — host_id, boot_id, and process_start_time, which turn a bare pid into a statement about one process: which machine it ran on, which boot of that machine, and which incarnation of that pid number.

  Consequence, stated plainly: registrations written before these columns existed, and any row on a legacy instance identity, are preserved permanently. They are retired only after their worker re-registers under a trusted identity — never on weaker evidence. That the alternative would leave legacy rows outstanding indefinitely is not a reason to relax the proof.

• A live pid is an absolute block. Even across a boot boundary, where the number provably cannot belong to the registered process, an occupied pid preserves the row rather than being argued away by the fencing proof.

Multiple processes belonging to one legitimate worker cohort are not treated as multiple independent workers: the fleet model from #948/#978 is preserved unchanged, and sharing a role or profile is never a duplicate.

Tools

gitea_plan_stale_worker_retirement

Read-only. Controller and reconciler only.

Parameter	Meaning
remote	dadeschools or prgs
host, org, repo	Optional overrides (audit context)
canonical_repository	Expected repository binding; defaults to the process root

Returns registry_fingerprint, candidate_fingerprint, candidate_worker_identities, per-worker candidates and preserved entries (each with reason_code, detail, and structured evidence), preserved_reason_counts, assessed_count, candidate_count, preserved_count, and protected_active_workflow_owners. mutation_performed is always false and read_only is always true.

Planning is deterministic: the same authoritative registry contents produce the same plan and the same tokens regardless of when they are observed.

gitea_apply_stale_worker_retirement

Mutating. Controller and reconciler only.

Parameter	Meaning
registry_fingerprint	The exact stable token the plan returned
candidate_fingerprint	The exact candidate-set token the plan returned
worker_identities	The exact candidate identities (list, or JSON / comma-separated string)
remote, host, org, repo	As above
canonical_repository	Must match the value the plan used

Before touching the registry, apply fails closed on: profile permission, role kind, master parity (mutation_safe), stable-runtime mode, capability resolution refreshed immediately before mutation, worker-registry availability, workflow-lease enumeration failure, and daemon-cohort uniqueness (classify_cohort).

A matching token is necessary but never sufficient. Inside one BEGIN IMMEDIATE transaction (WorkerRegistry.retire_stale_workers) the server:

1. re-reads the authoritative rows;
2. recomputes registry_fingerprint from those rows and compares — a mismatch returns registry_revision_moved with retired_count: 0 and no write;
3. recomputes the eligibility plan from those rows — re-reading the active workflow leases rather than reusing the set captured before the transaction opened, so a lease acquired after planning still preserves its worker — and compares candidate_fingerprint. A mismatch returns candidate_set_moved with retired_count: 0 and no write; a lease-enumeration failure raises and rolls the transaction back;
4. revalidates every requested identity against that fresh plan;
5. retires each survivor with a guarded UPDATE that additionally asserts status, last_heartbeat_at, generation_id, session_id, fencing_epoch, and pid are unchanged. A guard that matches no row preserves the worker with row_changed_since_plan.

There is no window between a safety check and its matching write, so a worker that comes back to life, changes ownership, or is retired concurrently cannot be removed on the strength of a stale observation. Any exception — including a commit failure — rolls the whole transaction back and returns transaction_failed with success: false, retired_count: 0, and mutation_performed: false; a partial write can never be reported as success.

Outcomes

outcome	Meaning	mutation_performed
planned	Dry-run result	false
applied	Transaction ran; see retired / preserved	true only if something was retired
registry_revision_moved	Registry changed between plan and apply	false
candidate_set_moved	Eligibility verdict changed between plan and apply	false
already_retired	Every requested row is already retired (idempotent replay)	false
nothing_requested	Empty target list	false
transaction_failed	Rolled back; nothing retired	false

What retirement does to the fleet snapshot

A retired row keeps its history: status moves to retired and retired_at, retired_by, retirement_reason are recorded. Nothing is deleted. Because retired is not active, the #978 snapshot counts the row as historical, not stale, so stale_worker_count falls and historical rows never make the live fleet unsafe by themselves.

Retirement does not repair untrusted live identity. Live workers registered under legacy pid-/proc- instance identities are preserved untouched and keep their legacy_incomplete_identity blockers. Retiring every stale row can therefore legitimately produce:

• stale_worker_count: 0
• residual live legacy_incomplete_identity blockers
• live_fleet_safe: false

That is a truthful result, and the post_apply block reports the remaining blockers rather than claiming the fleet became safe. Trusted client_instance_id propagation through launchers is a separate enrollment problem.

Permissions

Plan and apply are authorized differently, and deliberately so (review 657 B1).

	Plan	Apply
Capability	gitea.read	gitea.worker_registry.retire
Nature	observational; opens no transaction, writes nothing	mutation
Roles	controller, reconciler	controller, reconciler

An earlier revision authorized apply with gitea.read alone, reasoning that the mutation lands in the local control-plane registry rather than in Gitea. That the write is local makes it no less a mutation: sharing an observational permission class with plan meant any profile that could look could also destroy. Apply now requires its own capability.

• Denied: author, reviewer, merger, and every ordinary read-only profile — they lack the capability, so they fail closed on the permission itself rather than on the role check alone. The role restriction remains as defence in depth: a profile mistakenly granted the capability still cannot reach apply from an author, reviewer, or merger role.
• The capability is checked at entry and re-resolved immediately before the registry mutation, so a profile change mid-call cannot be outrun.
• No new Gitea write permission is introduced. gitea.worker_registry.retire authorizes exactly one local control-plane transition (worker_registrations.status -> retired) and grants no branch, issue, PR, review, merge, or restart authority. No author permission is broadened.
• The fleet snapshot remains observational: nothing here turns it into a gate on ordinary author work.

Operator step

No profile holds gitea.worker_registry.retire by default, so apply is inert until an operator adds it to the allowed_operations of the controller or reconciler profile in profiles.json. Removing it again immediately and completely revokes apply, while leaving plan and every other capability untouched. That grant is a configuration change and is outside the scope of the code that implements this feature.

External-state fencing

BEGIN IMMEDIATE locks the worker registry and nothing else, so two inputs the decision depends on sit outside the transaction's isolation domain: the workflow-lease table in a separate control-plane database, and OS process liveness. Re-reading them once during revalidation is not sufficient — the per-target loop runs afterwards, so a lease acquired (or a pid revived) after revalidation but before a given row's UPDATE would go unnoticed, and the registry-column guard cannot catch it because no registry column changed.

Two mechanisms close that window, both applied per target immediately before its own write:

• external_fence_fn — a version token over active leases (external_state_fingerprint), captured inside the transaction before the authoritative read and re-compared before every guarded UPDATE. Any movement raises, rolling back the whole transaction: once the world has changed, every remaining per-row decision was computed against a world that no longer exists. An unreadable lease store raises rather than returning a token, because "unreadable" must not silently compare equal to "unchanged".
• liveness_fn — a re-probe of process liveness and fencing identity that must affirmatively re-establish that this exact process is gone. It compares process_start_time, so a pid number reused since the plan is refused rather than accepted.

A caller that supplies no liveness_fn retires nothing (liveness_reprobe_unavailable) rather than proceeding unfenced. The guarded UPDATE additionally asserts host_id, boot_id, and process_start_time are unchanged, and all three participate in the CAS token, so fencing movement alone is enough to abort.

Non-goals

• Killing or restarting processes.
• Editing session files or configuration.
• Direct database cleanup outside the sanctioned transaction.
• Retiring live workers.
• Backfilling trusted identity for legacy workers.
• Rewriting worker ownership.
• Cleaning unrelated workflow-lease or issue-claim registries.