Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 104 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

feat(fleet): add CAS-protected stale worker retirement capability #980

New Issue
Open
opened 2026-07-30 10:49:39 -05:00 by jcwalker3 · 5 comments
jcwalker3 commented 2026-07-30 10:49:39 -05:00
Owner
Copy Link
Copy Source

Summary

Add a sanctioned controller/reconciler mutation capability that can safely retire conclusively stale MCP worker-registry entries using a dry-run-first, compare-and-swap-protected workflow.

The fleet snapshot introduced through issue #978 and merged PR #979 exposes authoritative worker state, blockers, identity provenance, and fleet revision tokens. However, the control plane currently has no native mutation operation capable of retiring stale worker rows.

Observed live state

At master 108cbfa173de34470d7abc022b929e6b3adc418a, an authoritative fleet assessment found:

  • 90 registered workers.
  • 2 live workers.
  • 88 stale workers with dead PIDs.
  • 88 stale_orphaned_worker findings.
  • 2 live legacy_incomplete_identity blockers.
  • No duplicate instance/profile grouping.
  • No client-instance collision.
  • No foreign-repository finding.
  • No ambiguous stale candidate among the 88 dead-PID rows.
  • Fleet safety remained false.

Capability probes produced:

  • reconcile_instance_fleet → unknown_task, stop_required:true
  • cleanup_stale_workers → unknown_task, stop_required:true
  • gitea_snapshot_instance_fleet → explicitly read-only
  • gitea_reconcile_after_restart → read-only and limited to session/lease reconciliation

Because no CAS-protected apply operation exists, the stale entries cannot be retired through sanctioned tools. Direct database manipulation is prohibited.

The current fleet snapshot is observational and is not proven to gate the ordinary author workflow (including branch, commit, push, or PR operations, PR #906 remediation, issue #650 work, or other unrelated authorized mutations). This issue addresses stale worker-registration accuracy and safe retirement on its own merits, and does not depend on #981, BAA, PR #906, or issue #650.

Required behavior

Implement a canonical native fleet-reconciliation capability available only to authorized controller/reconciler profiles.

The workflow must support:

  1. Authoritative dry run

    • Load a fresh fleet snapshot.
    • Return a dedicated stable registry token (registry_revision or registry_fingerprint).
    • Identify the exact retirement candidate set.
    • Include the reason, worker ID, profile, repository binding, identity provenance, PID status, heartbeat state, and ownership classification for every candidate.
    • Exclude every live, foreign, conflicting, or ambiguous entry.

  2. Strict candidate eligibility

    A worker may be retired only when the control plane conclusively establishes that it is a stale orphan, including:

    • pid_alive:false
    • Expired heartbeat under the canonical policy
    • ownership_state:stale
    • No live-session conflict
    • No repository ambiguity
    • No identity collision
    • No active workflow ownership that requires separate reconciliation

  3. CAS-protected application

    • Require a dedicated stable registry token, called registry_revision or registry_fingerprint, returned by the plan operation.
    • The token must be derived exclusively from canonical retirement-relevant registry content or from an explicitly persisted monotonically increasing registry revision.
    • The token must not incorporate snapshot_at, wall-clock time, request time, report-generation time, or any other volatile observation metadata.
    • Identical canonical registry contents must produce identical tokens across repeated snapshots.
    • Any row creation, update, heartbeat change, identity change, or deletion relevant to retirement safety must change the token.
    • Require the exact token returned by the plan operation, plus an exact candidate-set fingerprint.
    • Immediately before mutation, apply obtains the current token using the same stable algorithm.
    • If the expected and current tokens differ, apply aborts without partial mutation, retiring zero workers, and returns a structured registry_revision_moved or equivalent result.
    • Never silently recompute and apply a different candidate set.
    • Apply must independently revalidate every selected worker immediately before retirement.
    • Active workers and workers with incomplete, ambiguous, or conflicting identity/liveness evidence remain fail-closed and must not be retired.
    • The existing time-seeded fleet_revision must not be used for this CAS unless its semantics are first changed to satisfy this stable contract.
    • Require a fresh identity and capability resolution immediately before application.

  4. Fail-closed safety

    Stop without mutation if:

    • Any candidate becomes live.
    • Ownership becomes ambiguous.
    • A heartbeat or PID state changes.
    • Repository binding differs.
    • Runtime parity is stale.
    • Daemon-cohort uniqueness fails.
    • The candidate set or CAS token changes.
    • The caller lacks the controller/reconciler capability.

  5. Idempotency and auditability

    • Reapplying an already-completed plan must be a safe no-op.
    • Record which entries were retired, which were preserved, the acting identity/profile, the assessed and applied fleet revisions/tokens, and the reason for every decision.
    • Return structured counts for assessed, retired, preserved, blocked, and changed-during-apply entries.

  6. Post-apply verification

    • Return a fresh fleet snapshot and revision.
    • Confirm retired entries no longer count as active or stale workers.
    • Confirm every live or ambiguous entry remains untouched.
    • Report remaining blockers without falsely declaring the fleet safe.

  7. Capability and schema integration

    • Add the task to the canonical capability resolver.
    • Expose dry-run and apply modes through sanctioned native controller/reconciler tools.
    • Document required parameters, revision semantics, result schemas, reason codes, role restrictions, and mutation behavior.

Important separation of concerns

This issue must not solve incomplete live-worker identity by inventing or backfilling trusted IDs.

The two observed live workers used legacy-pid-* identities and must remain untouched by stale-worker retirement. Trusted client_instance_id propagation through launchers is a separate enrollment problem.

Therefore, successfully retiring the 88 stale rows may produce:

  • stale_worker_count: 0
  • Two remaining live legacy_incomplete_identity blockers
  • live_fleet_safe:false

That is an acceptable and truthful result for this issue. The operation must never report complete fleet safety while independent live-identity blockers remain.

Non-goals

  • Killing or restarting processes.
  • Editing session files or configuration.
  • Direct database cleanup.
  • Retiring live workers.
  • Backfilling trusted identity for legacy workers.
  • Rewriting worker ownership.
  • Cleaning unrelated workflow-lease or issue-claim registries.
  • Modifying PR #906 or issue #650.

Acceptance criteria

  • An authorized reconciler can perform a read-only assessment and receive an exact retirement plan with a stable registry CAS token (registry_revision or registry_fingerprint) and candidate fingerprint.
  • An authorized reconciler can apply that unchanged plan through a sanctioned native mutation when the token matches.
  • Only conclusively stale orphan rows are retired.
  • Live, ambiguous, conflicting, and foreign-repository entries are preserved.
  • Identical registry contents observed at different times produce the same CAS token.
  • A retirement-relevant registry change produces a different token.
  • Application fails without mutation when the CAS token or candidate set changes (retiring zero workers).
  • Applying with a matching token still independently revalidates every target immediately before retirement.
  • Active or ambiguously identified workers are preserved fail-closed.
  • A worker becoming live between assessment and application cannot be retired.
  • Unauthorized author, reviewer, or merger profiles cannot invoke the mutation.
  • Reapplication after successful retirement is idempotent.
  • Structured audit evidence identifies every retired and preserved entry.
  • Post-apply output reports remaining blockers accurately.
  • Tests cover revision drift, candidate drift, live-worker races, ambiguous ownership, authorization, idempotency, partial-failure prevention, and post-apply verification.
  • Documentation explains that stale retirement does not repair untrusted live identity.

Related work

  • Issue #978 / merged PR #979: authoritative instance-fleet snapshot and safety assessment.
  • PR #906 / issue #650: unrelated work (#980 does not depend on #906 or #650).
  • Trusted launcher identity propagation should be tracked independently and linked if an issue already exists.
## Summary Add a sanctioned controller/reconciler mutation capability that can safely retire conclusively stale MCP worker-registry entries using a dry-run-first, compare-and-swap-protected workflow. The fleet snapshot introduced through issue #978 and merged PR #979 exposes authoritative worker state, blockers, identity provenance, and fleet revision tokens. However, the control plane currently has no native mutation operation capable of retiring stale worker rows. ## Observed live state At master `108cbfa173de34470d7abc022b929e6b3adc418a`, an authoritative fleet assessment found: * 90 registered workers. * 2 live workers. * 88 stale workers with dead PIDs. * 88 `stale_orphaned_worker` findings. * 2 live `legacy_incomplete_identity` blockers. * No duplicate instance/profile grouping. * No client-instance collision. * No foreign-repository finding. * No ambiguous stale candidate among the 88 dead-PID rows. * Fleet safety remained false. Capability probes produced: * `reconcile_instance_fleet` → `unknown_task`, `stop_required:true` * `cleanup_stale_workers` → `unknown_task`, `stop_required:true` * `gitea_snapshot_instance_fleet` → explicitly read-only * `gitea_reconcile_after_restart` → read-only and limited to session/lease reconciliation Because no CAS-protected apply operation exists, the stale entries cannot be retired through sanctioned tools. Direct database manipulation is prohibited. The current fleet snapshot is observational and is not proven to gate the ordinary author workflow (including branch, commit, push, or PR operations, PR #906 remediation, issue #650 work, or other unrelated authorized mutations). This issue addresses stale worker-registration accuracy and safe retirement on its own merits, and does not depend on #981, BAA, PR #906, or issue #650. ## Required behavior Implement a canonical native fleet-reconciliation capability available only to authorized controller/reconciler profiles. The workflow must support: 1. **Authoritative dry run** * Load a fresh fleet snapshot. * Return a dedicated stable registry token (`registry_revision` or `registry_fingerprint`). * Identify the exact retirement candidate set. * Include the reason, worker ID, profile, repository binding, identity provenance, PID status, heartbeat state, and ownership classification for every candidate. * Exclude every live, foreign, conflicting, or ambiguous entry. 2. **Strict candidate eligibility** A worker may be retired only when the control plane conclusively establishes that it is a stale orphan, including: * `pid_alive:false` * Expired heartbeat under the canonical policy * `ownership_state:stale` * No live-session conflict * No repository ambiguity * No identity collision * No active workflow ownership that requires separate reconciliation 3. **CAS-protected application** * Require a dedicated stable registry token, called `registry_revision` or `registry_fingerprint`, returned by the plan operation. * The token must be derived exclusively from canonical retirement-relevant registry content or from an explicitly persisted monotonically increasing registry revision. * The token must not incorporate `snapshot_at`, wall-clock time, request time, report-generation time, or any other volatile observation metadata. * Identical canonical registry contents must produce identical tokens across repeated snapshots. * Any row creation, update, heartbeat change, identity change, or deletion relevant to retirement safety must change the token. * Require the exact token returned by the plan operation, plus an exact candidate-set fingerprint. * Immediately before mutation, apply obtains the current token using the same stable algorithm. * If the expected and current tokens differ, apply aborts without partial mutation, retiring zero workers, and returns a structured `registry_revision_moved` or equivalent result. * Never silently recompute and apply a different candidate set. * Apply must independently revalidate every selected worker immediately before retirement. * Active workers and workers with incomplete, ambiguous, or conflicting identity/liveness evidence remain fail-closed and must not be retired. * The existing time-seeded `fleet_revision` must not be used for this CAS unless its semantics are first changed to satisfy this stable contract. * Require a fresh identity and capability resolution immediately before application. 4. **Fail-closed safety** Stop without mutation if: * Any candidate becomes live. * Ownership becomes ambiguous. * A heartbeat or PID state changes. * Repository binding differs. * Runtime parity is stale. * Daemon-cohort uniqueness fails. * The candidate set or CAS token changes. * The caller lacks the controller/reconciler capability. 5. **Idempotency and auditability** * Reapplying an already-completed plan must be a safe no-op. * Record which entries were retired, which were preserved, the acting identity/profile, the assessed and applied fleet revisions/tokens, and the reason for every decision. * Return structured counts for assessed, retired, preserved, blocked, and changed-during-apply entries. 6. **Post-apply verification** * Return a fresh fleet snapshot and revision. * Confirm retired entries no longer count as active or stale workers. * Confirm every live or ambiguous entry remains untouched. * Report remaining blockers without falsely declaring the fleet safe. 7. **Capability and schema integration** * Add the task to the canonical capability resolver. * Expose dry-run and apply modes through sanctioned native controller/reconciler tools. * Document required parameters, revision semantics, result schemas, reason codes, role restrictions, and mutation behavior. ## Important separation of concerns This issue must not solve incomplete live-worker identity by inventing or backfilling trusted IDs. The two observed live workers used `legacy-pid-*` identities and must remain untouched by stale-worker retirement. Trusted `client_instance_id` propagation through launchers is a separate enrollment problem. Therefore, successfully retiring the 88 stale rows may produce: * `stale_worker_count: 0` * Two remaining live `legacy_incomplete_identity` blockers * `live_fleet_safe:false` That is an acceptable and truthful result for this issue. The operation must never report complete fleet safety while independent live-identity blockers remain. ## Non-goals * Killing or restarting processes. * Editing session files or configuration. * Direct database cleanup. * Retiring live workers. * Backfilling trusted identity for legacy workers. * Rewriting worker ownership. * Cleaning unrelated workflow-lease or issue-claim registries. * Modifying PR #906 or issue #650. ## Acceptance criteria * An authorized reconciler can perform a read-only assessment and receive an exact retirement plan with a stable registry CAS token (`registry_revision` or `registry_fingerprint`) and candidate fingerprint. * An authorized reconciler can apply that unchanged plan through a sanctioned native mutation when the token matches. * Only conclusively stale orphan rows are retired. * Live, ambiguous, conflicting, and foreign-repository entries are preserved. * Identical registry contents observed at different times produce the same CAS token. * A retirement-relevant registry change produces a different token. * Application fails without mutation when the CAS token or candidate set changes (retiring zero workers). * Applying with a matching token still independently revalidates every target immediately before retirement. * Active or ambiguously identified workers are preserved fail-closed. * A worker becoming live between assessment and application cannot be retired. * Unauthorized author, reviewer, or merger profiles cannot invoke the mutation. * Reapplication after successful retirement is idempotent. * Structured audit evidence identifies every retired and preserved entry. * Post-apply output reports remaining blockers accurately. * Tests cover revision drift, candidate drift, live-worker races, ambiguous ownership, authorization, idempotency, partial-failure prevention, and post-apply verification. * Documentation explains that stale retirement does not repair untrusted live identity. ## Related work * Issue #978 / merged PR #979: authoritative instance-fleet snapshot and safety assessment. * PR #906 / issue #650: unrelated work (#980 does not depend on #906 or #650). * Trusted launcher identity propagation should be tracked independently and linked if an issue already exists.
sysadmin commented 2026-07-30 13:28:58 -05:00
Owner
Copy Link
Copy Source

Canonical Issue State

STATE: needs-revision — R3-C; no external dependency remains, two body/AC defects block a clean author cycle
WHO_IS_NEXT: author
NEXT_ACTION: prgs-author applies the smallest revision below to this issue body, then claims and implements #980 through the ordinary issue-author workflow. No dependency on #981, on a BAA/first-install mechanism, on PR #906 or #650, or on trusted launcher identity remains. Labels unchanged: the live catalog has neither status:needs-revision nor status:needs-triage.
NEXT_PROMPT:

As prgs-author on prgs for Scaled-Tech-Consulting/Gitea-Tools at master 108cbfa173de34470d7abc022b929e6b3adc418a: apply the R3-C minimal revision to issue #980 before implementing it. (1) Requirement 3 and its matching acceptance criterion must not compare-and-swap on the fleet revision token as currently derived: mcp_fleet_snapshot._consistency_token seeds its digest with snapshot_at at second precision, so registry_revision changes on every call even when no registry row changes, and an apply gated on "the exact fleet revision produced by the dry run" can never pass. Either specify a content-only stable registry revision (the already-declared, currently unused registry_revision parameter of snapshot_instance_fleet is the existing extension point) or CAS solely on the candidate-set fingerprint plus per-row compare-and-swap. (2) Correct or drop the sentence "This blocks unrelated authorized work, including author remediation of PR #906 for issue #650" and the framing that fleet safety gates author mutations; that premise was corrected by issue #981 comment 18781 (disposition D2-A) and independently re-verified here. Keep the separation of concerns, the non-goals, and the fail-closed eligibility conjunction unchanged. Then claim and implement #980 through the ordinary author workflow, preserving active and ambiguously identified workers fail-closed.

WHAT_HAPPENED:

Disposition R3-C — goal valid, acceptance criteria require a small correction

Master SHA (parity-verified twice): 108cbfa173de34470d7abc022b929e6b3adc418a

Controller preflight (this session): namespace gitea-controller; identity sysadmin; profile prgs-controller; declared role controller, inferred role_kind reconciler in runtime context; binding prgs to Scaled-Tech-Consulting/Gitea-Tools; startup, runtime, local and live remote head all 108cbfa173de; in_parity:true; live_stale:false; parity-layer mutation_safe:true; control checkout clean on master including untracked files (git status --porcelain --untracked-files=all empty); gitea.read and gitea.issue.comment allowed, label mutation not exercised.

Recorded dependencies — none. The #980 body names #978/PR #979 (merged prerequisite, already landed), PR #906/#650 (a downstream consumer, not a prerequisite), and trusted client_instance_id propagation, which it explicitly places out of scope. It does not name #981 or any BAA/first-install mechanism. gitea_list_dependency_edges returns zero durable edges with #980 as either source or target. #980 carries no comments prior to this one.

Effect of #981 comment 18781 (D2-A). The pause on #980 was inherited entirely from the superseded fleet-deadlock premise of comment 18775. That premise was independently re-verified as unsupported at this revision: live_fleet_safe is produced at mcp_fleet_snapshot.py:804 and has zero non-test readers; mcp_fleet_snapshot is imported by exactly one server surface, gitea_snapshot_instance_fleet (gitea_mcp_server.py:19461), which is controller/reconciler-restricted, mapped to gitea.read in task_capability_map.py:170,174, and returns mutation_performed:false with an explicit read-only permission scope. No ordinary author mutation path consumes a fleet-derived gate. The #981-derived pause on #980 is therefore released.

Author-workflow capability map at this revision. Every operation #980 needs is configured and routes to prgs-author: work_issue -> gitea.pr.create; lock_issue -> gitea.issue.comment; create_branch -> gitea.branch.create; commit_files -> gitea.repo.commit; push_branch -> gitea.branch.push; create_pr -> gitea.pr.create; comment_issue and mark_issue -> gitea.issue.comment. Claim release runs through gitea_mark_issue/lease release rather than a distinct release_issue_claim task key. No author mutation was attempted as a test in this session.

Fleet-layer versus parity-layer gates are distinct and were not conflated. Live snapshot at 108cbfa173de, registry_revision fleetrev-3d8c6d1c4b8a3ab8: 95 registered workers, 1 live, 94 stale, 0 historical; all 94 stale rows carry liveness.pid_alive:false with none unprobeable or alive; blockers are 94 stale_orphaned_worker warnings plus 1 legacy_incomplete_identity; no duplicate namespace worker, no identity collision, no foreign repository, no ambiguous candidate; live_fleet_safe:false and fleet-layer mutation_safe:false while parity-layer mutation_safe:true. The fleet-layer value is observational and gates nothing on the author path.

Prior implementation — primitives only, no duplicate. WorkerRegistry.release (mcp_worker_identity.py:1192) idempotently marks a registration released and never deletes history, but it has zero non-test callers, no MCP tool, no CAS, no eligibility gate, no role restriction and no audit event. Classification inputs already exist: ownership_state values live/stale/historical/live_untrusted_identity/orphaned (mcp_fleet_snapshot.py:331-337), the stale flag (line 377), and WorkerRegistry.is_live (mcp_worker_identity.py:673-717), in which pid_alive can only withdraw liveness and never grant it. No open or merged PR implements worker-registration retirement.

Adjacent open PR #971 (#969) is not a duplicate. Its branch head e8bae606 touches worker_registrations in zero lines; it retires rows in the control-plane sessions table via a new session_lifecycle module and gitea_retire_stale_workflow_sessions. That is the registry #980's own non-goals exclude. It is a useful precedent for CAS retirement and PID-reuse handling, and it is a textual conflict risk only: it also edits control_plane_db.py, gitea_mcp_server.py and task_capability_map.py, and it currently reports mergeable:false against master. This is ordinary merge-conflict risk, not a dependency.

Defect 1, blocking as written. Requirement 3 demands "Require the exact fleet revision produced by the dry run" and "Abort without partial mutation if the fleet revision or candidate set changes", and the acceptance criteria demand both a successful apply of the unchanged plan and a failed apply on revision change. At this revision those cannot both hold. mcp_fleet_snapshot._consistency_token (lines 271-294) seeds its digest with snapshot_at, formatted at second precision by _ts (lines 104-105), and gitea_snapshot_instance_fleet never passes the declared registry_revision override (gitea_mcp_server.py:19598-19603), so the emitted token is time-seeded. Direct evaluation of the pure function with byte-identical rows one second apart yields fleetrev-fd8d1ca7c90315c3 versus fleetrev-cc5da6c5dcff053b. Any dry-run-then-apply cycle spanning more than one second therefore aborts unconditionally. The issue's own "Observed live state" records exactly this drift as evidence of moving fleet state, which would mislead an implementer into treating a derivation artifact as real churn.

Defect 2, obsolete premise in the body. "This blocks unrelated authorized work, including author remediation of PR #906 for issue #650" asserts the fleet-deadlock premise corrected by D2-A. Registry accuracy remains independently valuable, so the goal survives, but the stated motivation is recorded as fact and is false at this revision.

Everything else in #980 verified sound. The separation of concerns is correct: retirement needs no trusted launcher identity, and the issue's declared acceptable outcome of stale_worker_count:0 with residual live legacy_incomplete_identity blockers and live_fleet_safe:false matches how the snapshot computes safety. The eligibility conjunction is strictly narrower than the snapshot's own stale flag: because pid_alive cannot grant liveness, a heartbeat-lapsed but running process classifies as stale, yet #980's added pid_alive:false requirement preserves it, and an unprobeable pid_alive:null also fails closed. The issue distinguishes registered stale rows, live workers, duplicate processes within one cohort, and independent legitimate workers, and excludes live, foreign, conflicting and ambiguous entries from retirement. PID reuse is unaddressed but errs toward over-preservation.

Do not do: implement #980 before the revision lands, activate or design the BAA/first-install lane, retire any worker row, touch #981, PR #906, #650, review 624, branch feat/issue-650-providers-insights, or worktree branches/conflict-fix-pr-906-983e8ac.

WHY: #980 has no remaining external dependency — the #981/BAA pause is released and no fleet-derived gate touches the author path — but its CAS-on-fleet-revision requirement cannot be satisfied against the time-seeded revision token at 108cbfa173de, and its blocking rationale repeats a premise already corrected by #981 comment 18781, so a small body revision is required before an author cycle can close cleanly.
RELATED_PRS: PR #979 (#978) merged and supplies the snapshot this issue builds on. PR #971 (#969) is open, retires control-plane session rows only, touches no worker_registrations line, and shares three files with #980's likely surface. PR #906 (#650) is untouched and is a consumer, not a prerequisite. No PR exists or was created for #980.
BLOCKERS: Acceptance-criteria defect — compare-and-swap on the fleet revision token cannot pass because mcp_fleet_snapshot._consistency_token is time-seeded and gitea_snapshot_instance_fleet never supplies the registry_revision override. Obsolete premise — the PR #906 blocking rationale was superseded by #981 comment 18781 (D2-A). Both are internal to #980 and owned by prgs-author; neither is an external dependency. Labels could not move to status:needs-revision or status:needs-triage because neither label exists in the live catalog. Unblock after the author revises requirement 3, its matching acceptance criterion, and the obsolete rationale sentence.
VALIDATION: Controller-only read plus one issue comment, on a clean control checkout at 108cbfa173de; gitea_whoami then gitea_resolve_task_capability(comment_issue, issue_number=980) immediately before this post, with allowed_in_current_session:true, stop_required:false, mutation_performed:false. No author mutation was performed or tested; no code, branch, worktree, commit, push, PR, review, merge, claim, lease, grant, fleet-registry, database, configuration, session, credential or process change. Source claims were verified by read-only inspection of the control checkout at that SHA; the revision-token instability was reproduced by evaluating the pure _consistency_token function with synthetic rows and no database access. Issue #981, PR #906, issue #650, review 624, branch feat/issue-650-providers-insights and worktree branches/conflict-fix-pr-906-983e8ac were not modified. Labels on #980 were left exactly as found.
LAST_UPDATED_BY: sysadmin / prgs-controller (gitea-controller)

WHO_IS_NEXT: prgs-author to apply the R3-C minimal revision to #980, then claim and implement it through the ordinary issue-author workflow

## Canonical Issue State STATE: needs-revision — R3-C; no external dependency remains, two body/AC defects block a clean author cycle WHO_IS_NEXT: author NEXT_ACTION: prgs-author applies the smallest revision below to this issue body, then claims and implements #980 through the ordinary issue-author workflow. No dependency on #981, on a BAA/first-install mechanism, on PR #906 or #650, or on trusted launcher identity remains. Labels unchanged: the live catalog has neither `status:needs-revision` nor `status:needs-triage`. NEXT_PROMPT: ```text As prgs-author on prgs for Scaled-Tech-Consulting/Gitea-Tools at master 108cbfa173de34470d7abc022b929e6b3adc418a: apply the R3-C minimal revision to issue #980 before implementing it. (1) Requirement 3 and its matching acceptance criterion must not compare-and-swap on the fleet revision token as currently derived: mcp_fleet_snapshot._consistency_token seeds its digest with snapshot_at at second precision, so registry_revision changes on every call even when no registry row changes, and an apply gated on "the exact fleet revision produced by the dry run" can never pass. Either specify a content-only stable registry revision (the already-declared, currently unused registry_revision parameter of snapshot_instance_fleet is the existing extension point) or CAS solely on the candidate-set fingerprint plus per-row compare-and-swap. (2) Correct or drop the sentence "This blocks unrelated authorized work, including author remediation of PR #906 for issue #650" and the framing that fleet safety gates author mutations; that premise was corrected by issue #981 comment 18781 (disposition D2-A) and independently re-verified here. Keep the separation of concerns, the non-goals, and the fail-closed eligibility conjunction unchanged. Then claim and implement #980 through the ordinary author workflow, preserving active and ambiguously identified workers fail-closed. ``` WHAT_HAPPENED: ### Disposition R3-C — goal valid, acceptance criteria require a small correction **Master SHA (parity-verified twice):** `108cbfa173de34470d7abc022b929e6b3adc418a` **Controller preflight (this session):** namespace `gitea-controller`; identity `sysadmin`; profile `prgs-controller`; declared role `controller`, inferred `role_kind` `reconciler` in runtime context; binding `prgs` to `Scaled-Tech-Consulting/Gitea-Tools`; startup, runtime, local and live remote head all `108cbfa173de`; `in_parity:true`; `live_stale:false`; parity-layer `mutation_safe:true`; control checkout clean on `master` including untracked files (`git status --porcelain --untracked-files=all` empty); `gitea.read` and `gitea.issue.comment` allowed, label mutation not exercised. **Recorded dependencies — none.** The #980 body names #978/PR #979 (merged prerequisite, already landed), PR #906/#650 (a downstream consumer, not a prerequisite), and trusted `client_instance_id` propagation, which it explicitly places out of scope. It does not name #981 or any BAA/first-install mechanism. `gitea_list_dependency_edges` returns zero durable edges with #980 as either source or target. #980 carries no comments prior to this one. **Effect of #981 comment 18781 (D2-A).** The pause on #980 was inherited entirely from the superseded fleet-deadlock premise of comment 18775. That premise was independently re-verified as unsupported at this revision: `live_fleet_safe` is produced at `mcp_fleet_snapshot.py:804` and has zero non-test readers; `mcp_fleet_snapshot` is imported by exactly one server surface, `gitea_snapshot_instance_fleet` (`gitea_mcp_server.py:19461`), which is controller/reconciler-restricted, mapped to `gitea.read` in `task_capability_map.py:170,174`, and returns `mutation_performed:false` with an explicit read-only permission scope. No ordinary author mutation path consumes a fleet-derived gate. The #981-derived pause on #980 is therefore released. **Author-workflow capability map at this revision.** Every operation #980 needs is configured and routes to `prgs-author`: `work_issue` -> `gitea.pr.create`; `lock_issue` -> `gitea.issue.comment`; `create_branch` -> `gitea.branch.create`; `commit_files` -> `gitea.repo.commit`; `push_branch` -> `gitea.branch.push`; `create_pr` -> `gitea.pr.create`; `comment_issue` and `mark_issue` -> `gitea.issue.comment`. Claim release runs through `gitea_mark_issue`/lease release rather than a distinct `release_issue_claim` task key. No author mutation was attempted as a test in this session. **Fleet-layer versus parity-layer gates are distinct and were not conflated.** Live snapshot at `108cbfa173de`, `registry_revision` `fleetrev-3d8c6d1c4b8a3ab8`: 95 registered workers, 1 live, 94 stale, 0 historical; all 94 stale rows carry `liveness.pid_alive:false` with none unprobeable or alive; blockers are 94 `stale_orphaned_worker` warnings plus 1 `legacy_incomplete_identity`; no duplicate namespace worker, no identity collision, no foreign repository, no ambiguous candidate; `live_fleet_safe:false` and fleet-layer `mutation_safe:false` while parity-layer `mutation_safe:true`. The fleet-layer value is observational and gates nothing on the author path. **Prior implementation — primitives only, no duplicate.** `WorkerRegistry.release` (`mcp_worker_identity.py:1192`) idempotently marks a registration `released` and never deletes history, but it has zero non-test callers, no MCP tool, no CAS, no eligibility gate, no role restriction and no audit event. Classification inputs already exist: `ownership_state` values `live`/`stale`/`historical`/`live_untrusted_identity`/`orphaned` (`mcp_fleet_snapshot.py:331-337`), the `stale` flag (line 377), and `WorkerRegistry.is_live` (`mcp_worker_identity.py:673-717`), in which `pid_alive` can only withdraw liveness and never grant it. No open or merged PR implements worker-registration retirement. **Adjacent open PR #971 (#969) is not a duplicate.** Its branch head `e8bae606` touches `worker_registrations` in zero lines; it retires rows in the control-plane `sessions` table via a new `session_lifecycle` module and `gitea_retire_stale_workflow_sessions`. That is the registry #980's own non-goals exclude. It is a useful precedent for CAS retirement and PID-reuse handling, and it is a textual conflict risk only: it also edits `control_plane_db.py`, `gitea_mcp_server.py` and `task_capability_map.py`, and it currently reports `mergeable:false` against master. This is ordinary merge-conflict risk, not a dependency. **Defect 1, blocking as written.** Requirement 3 demands "Require the exact fleet revision produced by the dry run" and "Abort without partial mutation if the fleet revision or candidate set changes", and the acceptance criteria demand both a successful apply of the unchanged plan and a failed apply on revision change. At this revision those cannot both hold. `mcp_fleet_snapshot._consistency_token` (lines 271-294) seeds its digest with `snapshot_at`, formatted at second precision by `_ts` (lines 104-105), and `gitea_snapshot_instance_fleet` never passes the declared `registry_revision` override (`gitea_mcp_server.py:19598-19603`), so the emitted token is time-seeded. Direct evaluation of the pure function with byte-identical rows one second apart yields `fleetrev-fd8d1ca7c90315c3` versus `fleetrev-cc5da6c5dcff053b`. Any dry-run-then-apply cycle spanning more than one second therefore aborts unconditionally. The issue's own "Observed live state" records exactly this drift as evidence of moving fleet state, which would mislead an implementer into treating a derivation artifact as real churn. **Defect 2, obsolete premise in the body.** "This blocks unrelated authorized work, including author remediation of PR #906 for issue #650" asserts the fleet-deadlock premise corrected by D2-A. Registry accuracy remains independently valuable, so the goal survives, but the stated motivation is recorded as fact and is false at this revision. **Everything else in #980 verified sound.** The separation of concerns is correct: retirement needs no trusted launcher identity, and the issue's declared acceptable outcome of `stale_worker_count:0` with residual live `legacy_incomplete_identity` blockers and `live_fleet_safe:false` matches how the snapshot computes safety. The eligibility conjunction is strictly narrower than the snapshot's own `stale` flag: because `pid_alive` cannot grant liveness, a heartbeat-lapsed but running process classifies as `stale`, yet #980's added `pid_alive:false` requirement preserves it, and an unprobeable `pid_alive:null` also fails closed. The issue distinguishes registered stale rows, live workers, duplicate processes within one cohort, and independent legitimate workers, and excludes live, foreign, conflicting and ambiguous entries from retirement. PID reuse is unaddressed but errs toward over-preservation. Do not do: implement #980 before the revision lands, activate or design the BAA/first-install lane, retire any worker row, touch #981, PR #906, #650, review 624, branch `feat/issue-650-providers-insights`, or worktree `branches/conflict-fix-pr-906-983e8ac`. WHY: #980 has no remaining external dependency — the #981/BAA pause is released and no fleet-derived gate touches the author path — but its CAS-on-fleet-revision requirement cannot be satisfied against the time-seeded revision token at `108cbfa173de`, and its blocking rationale repeats a premise already corrected by #981 comment 18781, so a small body revision is required before an author cycle can close cleanly. RELATED_PRS: PR #979 (#978) merged and supplies the snapshot this issue builds on. PR #971 (#969) is open, retires control-plane session rows only, touches no `worker_registrations` line, and shares three files with #980's likely surface. PR #906 (#650) is untouched and is a consumer, not a prerequisite. No PR exists or was created for #980. BLOCKERS: Acceptance-criteria defect — compare-and-swap on the fleet revision token cannot pass because `mcp_fleet_snapshot._consistency_token` is time-seeded and `gitea_snapshot_instance_fleet` never supplies the `registry_revision` override. Obsolete premise — the PR #906 blocking rationale was superseded by #981 comment 18781 (D2-A). Both are internal to #980 and owned by prgs-author; neither is an external dependency. Labels could not move to `status:needs-revision` or `status:needs-triage` because neither label exists in the live catalog. Unblock after the author revises requirement 3, its matching acceptance criterion, and the obsolete rationale sentence. VALIDATION: Controller-only read plus one issue comment, on a clean control checkout at `108cbfa173de`; `gitea_whoami` then `gitea_resolve_task_capability(comment_issue, issue_number=980)` immediately before this post, with `allowed_in_current_session:true`, `stop_required:false`, `mutation_performed:false`. No author mutation was performed or tested; no code, branch, worktree, commit, push, PR, review, merge, claim, lease, grant, fleet-registry, database, configuration, session, credential or process change. Source claims were verified by read-only inspection of the control checkout at that SHA; the revision-token instability was reproduced by evaluating the pure `_consistency_token` function with synthetic rows and no database access. Issue #981, PR #906, issue #650, review 624, branch `feat/issue-650-providers-insights` and worktree `branches/conflict-fix-pr-906-983e8ac` were not modified. Labels on #980 were left exactly as found. LAST_UPDATED_BY: sysadmin / prgs-controller (gitea-controller) `WHO_IS_NEXT: prgs-author to apply the R3-C minimal revision to #980, then claim and implement it through the ordinary issue-author workflow`
jcwalker3 commented 2026-07-30 13:40:18 -05:00
Author
Owner
Copy Link
Copy Source

Canonical Issue State

STATE: ready — R3-C specification revision complete; issue #980 body updated with stable registry CAS contract and corrected obsolete rationale
WHO_IS_NEXT: author
NEXT_ACTION: Fresh prgs-author session claims and implements #980 through the ordinary issue-author workflow.
NEXT_PROMPT:

As prgs-author on prgs for Scaled-Tech-Consulting/Gitea-Tools at master 108cbfa173de34470d7abc022b929e6b3adc418a: claim and implement #980 through the ordinary issue-author workflow, using the stable registry CAS (registry_revision or registry_fingerprint) and preserving active or ambiguously identified workers fail-closed.

WHAT_HAPPENED:
Issue #980 body was revised under R3-C:

  1. The CAS requirement now requires a stable content-derived or explicitly persisted registry token (registry_revision or registry_fingerprint) that excludes observation time (snapshot_at, wall-clock time, request time, or report-generation time). Acceptance criteria were updated accordingly.
  2. The obsolete PR #906/#650 blocking rationale was removed. Established that the fleet snapshot is observational and does not gate ordinary author operations.
  3. Confirmed #980 has no dependency on #981, BAA, PR #906, or issue #650. BAA remains paused.
  4. No implementation, claim, branch creation, worktree modification, commit, push, or PR occurred in this session.
  5. All labels (status:ready, control-plane, mcp-health, safety, type:feature, workflow-hardening) remain intact.

WHY:
R3-C review identified that snapshot_at in _consistency_token caused identical registry states observed at different times to produce different tokens, making CAS dry-run/apply impossible. Additionally, PR #906/#650 blocking premise was superseded by D2-A disposition of #981.

RELATED_PRS:
PR #979 (#978) merged prerequisite. PR #971 (#969) open adjacent PR. No PR exists for #980.

BLOCKERS:
None. Issue #980 is ready for ordinary author implementation.

VALIDATION:
Author-only read-after-write verification of issue #980 body edit at master 108cbfa173. Body verified preserved with both corrections present. State (open) and labels (status:ready) preserved intact. Clean control checkout at 108cbfa173de; zero code, branch, worktree, commit, push, PR, claim, lease, or configuration changes performed.

LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author)

WHO_IS_NEXT: a fresh prgs-author session to claim and implement #980 through the ordinary issue-author workflow, using the stable registry CAS and preserving active or ambiguously identified workers fail-closed.

## Canonical Issue State STATE: ready — R3-C specification revision complete; issue #980 body updated with stable registry CAS contract and corrected obsolete rationale WHO_IS_NEXT: author NEXT_ACTION: Fresh prgs-author session claims and implements #980 through the ordinary issue-author workflow. NEXT_PROMPT: ```text As prgs-author on prgs for Scaled-Tech-Consulting/Gitea-Tools at master 108cbfa173de34470d7abc022b929e6b3adc418a: claim and implement #980 through the ordinary issue-author workflow, using the stable registry CAS (registry_revision or registry_fingerprint) and preserving active or ambiguously identified workers fail-closed. ``` WHAT_HAPPENED: Issue #980 body was revised under R3-C: 1. The CAS requirement now requires a stable content-derived or explicitly persisted registry token (registry_revision or registry_fingerprint) that excludes observation time (snapshot_at, wall-clock time, request time, or report-generation time). Acceptance criteria were updated accordingly. 2. The obsolete PR #906/#650 blocking rationale was removed. Established that the fleet snapshot is observational and does not gate ordinary author operations. 3. Confirmed #980 has no dependency on #981, BAA, PR #906, or issue #650. BAA remains paused. 4. No implementation, claim, branch creation, worktree modification, commit, push, or PR occurred in this session. 5. All labels (status:ready, control-plane, mcp-health, safety, type:feature, workflow-hardening) remain intact. WHY: R3-C review identified that snapshot_at in _consistency_token caused identical registry states observed at different times to produce different tokens, making CAS dry-run/apply impossible. Additionally, PR #906/#650 blocking premise was superseded by D2-A disposition of #981. RELATED_PRS: PR #979 (#978) merged prerequisite. PR #971 (#969) open adjacent PR. No PR exists for #980. BLOCKERS: None. Issue #980 is ready for ordinary author implementation. VALIDATION: Author-only read-after-write verification of issue #980 body edit at master 108cbfa173de34470d7abc022b929e6b3adc418a. Body verified preserved with both corrections present. State (open) and labels (status:ready) preserved intact. Clean control checkout at 108cbfa173de; zero code, branch, worktree, commit, push, PR, claim, lease, or configuration changes performed. LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author) `WHO_IS_NEXT: a fresh prgs-author session to claim and implement #980 through the ordinary issue-author workflow, using the stable registry CAS and preserving active or ambiguously identified workers fail-closed.`
jcwalker3 added status:in-progress and removed status:ready labels 2026-07-30 13:49:17 -05:00
jcwalker3 commented 2026-07-30 13:49:18 -05:00
Author
Owner
Copy Link
Copy Source

Issue claim heartbeat

<!-- gitea-issue-claim-heartbeat:v1 --> **Issue claim heartbeat** - kind: claim - issue: #980 - branch: feat/issue-980-stale-worker-retirement - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation
jcwalker3 added status:pr-open and removed status:in-progress labels 2026-07-30 14:25:52 -05:00
jcwalker3 commented 2026-07-30 14:29:55 -05:00
Author
Owner
Copy Link
Copy Source

[THREAD STATE LEDGER]

Canonical Issue State

STATE: implementation complete — PR #982 is open at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da, awaiting independent review
WHO_IS_NEXT: reviewer
NEXT_ACTION: prgs-reviewer independently reviews PR #982 at head c0c6d14b73add36ac157d501c603fd9ea51de1da against the corrected R3-C contract in this issue body.
NEXT_PROMPT:

As prgs-reviewer on prgs for Scaled-Tech-Consulting/Gitea-Tools: independently review PR #982 (issue #980) at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da, base master 108cbfa173de34470d7abc022b929e6b3adc418a. Verify the stable registry CAS token excludes every volatile observation input and moves on every retirement-relevant change; verify the transaction boundary and per-row guarded UPDATE in WorkerRegistry.retire_stale_workers leave no check/delete window; verify active, ambiguous, unprobeable, and incomplete-identity workers are preserved fail-closed; verify no new Gitea write permission or author-permission broadening was introduced. Do not merge without separate merger authorization.

WHAT_HAPPENED: Implemented #980 through the ordinary author workflow at master 108cbfa173de34470d7abc022b929e6b3adc418a. New mcp_fleet_retirement.registry_fingerprint derives the compare-and-swap token exclusively from canonical retirement-relevant registry content, with no snapshot_at, wall-clock, request, report-generation, or serialization time, and with serialized rows sorted so iteration order cannot perturb it; a second candidate_fingerprint pins the exact candidate set the plan returned. The existing time-seeded fleet_revision / registry_revision is left exactly as it is — a separate token was the smaller and safer change, and the #978 snapshot shape and semantics are unchanged. WorkerRegistry.retire_stale_workers performs the whole decision inside one BEGIN IMMEDIATE transaction: re-read rows, recompute the fingerprint from those rows, recompute the eligibility plan from those rows while re-reading active workflow leases, compare the candidate fingerprint, revalidate every target, then retire each survivor with a guarded UPDATE asserting its status, heartbeat, generation, session, fencing epoch, and pid are unchanged. Drift retires zero workers and reports registry_revision_moved or candidate_set_moved; any exception rolls back and reports transaction_failed, so a partial write is never reported as success. Retirement requires the full conjunction — active status, complete registry fields, parsable heartbeat, not live, pid_alive is False, expired heartbeat, ownership_state == "stale", canonical repository binding, no identity evidence shared with a live or unprobeable worker, and no active workflow-lease ownership — and everything else is preserved with a structured reason code. Retired rows become historical rather than stale and keep their history; nothing is deleted. Retirement does not repair untrusted live identity: live workers on legacy instance identities are preserved and keep their legacy_incomplete_identity findings, so the result never claims the fleet became safe. The capability is exposed as gitea_plan_stale_worker_retirement (read-only) and gitea_apply_stale_worker_retirement, both restricted to controller/reconciler role kinds; the Gitea operation gate stays gitea.read because the mutation lands in the local control-plane registry, matching the #601 lease-lifecycle model, so no new Gitea write permission was introduced and no author permission was broadened.

What is true now

  • server-side decision state: no review decision exists on PR #982; merge not performed; issue #980 carries status:pr-open after the create-PR transition.
  • local verdict/state: author implementation finished and verified locally; branch feat/issue-980-stale-worker-retirement, worktree branches/issue-980-stale-worker-retirement, base SHA 108cbfa173de34470d7abc022b929e6b3adc418a, single commit c0c6d14b73add36ac157d501c603fd9ea51de1da, worktree clean after commit.
  • PR #982 read back natively: state open, base master, head feat/issue-980-stale-worker-retirement, mergeable: true, Closes #980 present, 8 changed files, exactly one commit above master, remote head equal to local head.

What changed

  • Server-side: issue lock acquired on #980; label transitions status:in-progress then status:pr-open; one heartbeat comment; branch feat/issue-980-stale-worker-retirement non-force pushed; PR #982 created; this comment.
  • Local-only: eight files under the single commit — mcp_fleet_retirement.py (new), mcp_worker_identity.py, gitea_mcp_server.py, task_capability_map.py, docs/stale-worker-retirement.md (new), docs/instance-fleet-identity.md, docs/mcp-tool-inventory.md, tests/test_issue_980_stale_worker_retirement.py (new).
  • No worker-registry row was retired outside throwaway test databases; no profile, configuration, credential, session, or process was modified.

What is blocked

  • blocker classification: no blocker
  • Nothing obstructs the author phase. PR #982 cannot proceed to merge until an independent reviewer records a verdict and a merger acts; that is ordinary sequencing, not a failing gate.

Who/what acts next

  • next actor: prgs-reviewer
  • required action: independently review PR #982 at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da, with particular attention to token stability, transaction boundaries, race handling, and fail-closed preservation of active or ambiguously identified workers.

Do not do: merge PR #982, self-review it, record a verdict from the author namespace, activate the BAA or any recovery grant, retire any real worker-registry row, or touch #981, PR #906, #650, review 624, branch feat/issue-650-providers-insights, or worktree branches/conflict-fix-pr-906-983e8ac.

WHY: #980 carried the R3-C corrected stable-CAS contract with no external dependency, so the ordinary author cycle implemented that contract; an independent reviewer is required because an author may not review or merge their own PR.

RELATED_PRS: PR #982 implements this issue. PR #979 (#978) is the prerequisite that landed on master and supplies the fleet snapshot. PR #971 (#969) is an adjacent open PR that retires control-plane session rows in a different registry and shares three files as ordinary textual conflict risk, not a dependency. PR #906 (#650) is untouched.

BLOCKERS: None for the author phase; blocker classification: no blocker.

VALIDATION: prgs-author on the gitea-author namespace; identity jcwalker3, profile prgs-author, role author, remote prgs, repository Scaled-Tech-Consulting/Gitea-Tools. Startup, runtime, local and live remote head all 108cbfa173de34470d7abc022b929e6b3adc418a; in_parity:true, live_stale:false, parity-layer mutation_safe:true; control checkout clean on master including untracked files. New suite tests/test_issue_980_stale_worker_retirement.py: 40 passed, 23 subtests passed, including the regression test proving the old snapshot_at derivation moved the token one second apart while the new registry CAS token does not. Adjacent suites (#980, #978, #975, #948, task-capability role invariants): 192 passed, 124 subtests passed. Full suite from the branch worktree: 28 failed, 6206 passed, 6 skipped, 1129 subtests passed, against a master baseline worktree at 108cbfa173de of 28 failed, 6166 passed, 6 skipped, 1106 subtests passed — the failing set is identical, so zero regressions were introduced and no pre-existing failure was masked.

LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author)

WHO_IS_NEXT: prgs-reviewer to independently review PR #982 at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da

[THREAD STATE LEDGER] ## Canonical Issue State STATE: implementation complete — PR #982 is open at exact head `c0c6d14b73add36ac157d501c603fd9ea51de1da`, awaiting independent review WHO_IS_NEXT: reviewer NEXT_ACTION: prgs-reviewer independently reviews PR #982 at head `c0c6d14b73add36ac157d501c603fd9ea51de1da` against the corrected R3-C contract in this issue body. NEXT_PROMPT: ```text As prgs-reviewer on prgs for Scaled-Tech-Consulting/Gitea-Tools: independently review PR #982 (issue #980) at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da, base master 108cbfa173de34470d7abc022b929e6b3adc418a. Verify the stable registry CAS token excludes every volatile observation input and moves on every retirement-relevant change; verify the transaction boundary and per-row guarded UPDATE in WorkerRegistry.retire_stale_workers leave no check/delete window; verify active, ambiguous, unprobeable, and incomplete-identity workers are preserved fail-closed; verify no new Gitea write permission or author-permission broadening was introduced. Do not merge without separate merger authorization. ``` WHAT_HAPPENED: Implemented #980 through the ordinary author workflow at master `108cbfa173de34470d7abc022b929e6b3adc418a`. New `mcp_fleet_retirement.registry_fingerprint` derives the compare-and-swap token exclusively from canonical retirement-relevant registry content, with no `snapshot_at`, wall-clock, request, report-generation, or serialization time, and with serialized rows sorted so iteration order cannot perturb it; a second `candidate_fingerprint` pins the exact candidate set the plan returned. The existing time-seeded `fleet_revision` / `registry_revision` is left exactly as it is — a separate token was the smaller and safer change, and the #978 snapshot shape and semantics are unchanged. `WorkerRegistry.retire_stale_workers` performs the whole decision inside one `BEGIN IMMEDIATE` transaction: re-read rows, recompute the fingerprint from those rows, recompute the eligibility plan from those rows while re-reading active workflow leases, compare the candidate fingerprint, revalidate every target, then retire each survivor with a guarded `UPDATE` asserting its status, heartbeat, generation, session, fencing epoch, and pid are unchanged. Drift retires zero workers and reports `registry_revision_moved` or `candidate_set_moved`; any exception rolls back and reports `transaction_failed`, so a partial write is never reported as success. Retirement requires the full conjunction — active status, complete registry fields, parsable heartbeat, not live, `pid_alive is False`, expired heartbeat, `ownership_state == "stale"`, canonical repository binding, no identity evidence shared with a live or unprobeable worker, and no active workflow-lease ownership — and everything else is preserved with a structured reason code. Retired rows become historical rather than stale and keep their history; nothing is deleted. Retirement does not repair untrusted live identity: live workers on legacy instance identities are preserved and keep their `legacy_incomplete_identity` findings, so the result never claims the fleet became safe. The capability is exposed as `gitea_plan_stale_worker_retirement` (read-only) and `gitea_apply_stale_worker_retirement`, both restricted to controller/reconciler role kinds; the Gitea operation gate stays `gitea.read` because the mutation lands in the local control-plane registry, matching the #601 lease-lifecycle model, so no new Gitea write permission was introduced and no author permission was broadened. ### What is true now - server-side decision state: no review decision exists on PR #982; merge not performed; issue #980 carries `status:pr-open` after the create-PR transition. - local verdict/state: author implementation finished and verified locally; branch `feat/issue-980-stale-worker-retirement`, worktree `branches/issue-980-stale-worker-retirement`, base SHA `108cbfa173de34470d7abc022b929e6b3adc418a`, single commit `c0c6d14b73add36ac157d501c603fd9ea51de1da`, worktree clean after commit. - PR #982 read back natively: state open, base `master`, head `feat/issue-980-stale-worker-retirement`, `mergeable: true`, `Closes #980` present, 8 changed files, exactly one commit above master, remote head equal to local head. ### What changed - Server-side: issue lock acquired on #980; label transitions `status:in-progress` then `status:pr-open`; one heartbeat comment; branch `feat/issue-980-stale-worker-retirement` non-force pushed; PR #982 created; this comment. - Local-only: eight files under the single commit — `mcp_fleet_retirement.py` (new), `mcp_worker_identity.py`, `gitea_mcp_server.py`, `task_capability_map.py`, `docs/stale-worker-retirement.md` (new), `docs/instance-fleet-identity.md`, `docs/mcp-tool-inventory.md`, `tests/test_issue_980_stale_worker_retirement.py` (new). - No worker-registry row was retired outside throwaway test databases; no profile, configuration, credential, session, or process was modified. ### What is blocked - blocker classification: no blocker - Nothing obstructs the author phase. PR #982 cannot proceed to merge until an independent reviewer records a verdict and a merger acts; that is ordinary sequencing, not a failing gate. ### Who/what acts next - next actor: prgs-reviewer - required action: independently review PR #982 at exact head `c0c6d14b73add36ac157d501c603fd9ea51de1da`, with particular attention to token stability, transaction boundaries, race handling, and fail-closed preservation of active or ambiguously identified workers. Do not do: merge PR #982, self-review it, record a verdict from the author namespace, activate the BAA or any recovery grant, retire any real worker-registry row, or touch #981, PR #906, #650, review `624`, branch `feat/issue-650-providers-insights`, or worktree `branches/conflict-fix-pr-906-983e8ac`. WHY: #980 carried the R3-C corrected stable-CAS contract with no external dependency, so the ordinary author cycle implemented that contract; an independent reviewer is required because an author may not review or merge their own PR. RELATED_PRS: PR #982 implements this issue. PR #979 (#978) is the prerequisite that landed on master and supplies the fleet snapshot. PR #971 (#969) is an adjacent open PR that retires control-plane session rows in a different registry and shares three files as ordinary textual conflict risk, not a dependency. PR #906 (#650) is untouched. BLOCKERS: None for the author phase; blocker classification: no blocker. VALIDATION: prgs-author on the `gitea-author` namespace; identity `jcwalker3`, profile `prgs-author`, role `author`, remote `prgs`, repository `Scaled-Tech-Consulting/Gitea-Tools`. Startup, runtime, local and live remote head all `108cbfa173de34470d7abc022b929e6b3adc418a`; `in_parity:true`, `live_stale:false`, parity-layer `mutation_safe:true`; control checkout clean on `master` including untracked files. New suite `tests/test_issue_980_stale_worker_retirement.py`: 40 passed, 23 subtests passed, including the regression test proving the old `snapshot_at` derivation moved the token one second apart while the new registry CAS token does not. Adjacent suites (#980, #978, #975, #948, task-capability role invariants): 192 passed, 124 subtests passed. Full suite from the branch worktree: 28 failed, 6206 passed, 6 skipped, 1129 subtests passed, against a master baseline worktree at `108cbfa173de` of 28 failed, 6166 passed, 6 skipped, 1106 subtests passed — the failing set is identical, so zero regressions were introduced and no pre-existing failure was masked. LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author) `WHO_IS_NEXT: prgs-reviewer to independently review PR #982 at exact head c0c6d14b73add36ac157d501c603fd9ea51de1da`
jcwalker3 commented 2026-07-30 17:05:32 -05:00
Author
Owner
Copy Link
Copy Source

[THREAD STATE LEDGER]

Canonical Issue State

STATE: review 657 remediated — PR #982 advanced to exact head e344e68a1345cb21db06776eef839f863b81f3e5, awaiting a fresh independent review
WHO_IS_NEXT: reviewer
NEXT_ACTION: prgs-reviewer independently reviews PR #982 at exact head e344e68a1345cb21db06776eef839f863b81f3e5 and decides a new formal verdict; review 657 is stale and approves nothing.
NEXT_PROMPT:

As prgs-reviewer on prgs for Scaled-Tech-Consulting/Gitea-Tools: independently review PR #982 (issue #980) at exact head e344e68a1345cb21db06776eef839f863b81f3e5, base master 108cbfa173de34470d7abc022b929e6b3adc418a. Re-prove all three review-657 blockers are closed: (B1) gitea_apply_stale_worker_retirement requires gitea.worker_registry.retire at every enforcement layer and a gitea.read-only profile cannot resolve or invoke it; (B2) retirement requires affirmative launcher-minted inst- attribution plus host/boot/process-start fencing, and every incomplete, legacy, ambiguous, or unproven row is preserved with a structured reason; (B3) lease and OS-liveness evidence is re-fenced inside the transaction immediately before each guarded UPDATE and any movement rolls back with mutation_performed false. Verify the stable registry CAS work from the prior head did not regress. Do not merge without separate merger authorization.

WHAT_HAPPENED: Addressed only the three blocking findings of formal review 657 (REQUEST_CHANGES by sysadmin at head c0c6d14b73add36ac157d501c603fd9ea51de1da) and nothing else. One new commit e344e68a1345cb21db06776eef839f863b81f3e5 on branch feat/issue-980-stale-worker-retirement, seven files, non-force fast-forward push c0c6d14..e344e68.

B1 — dedicated mutation authorization. Apply was gated by _profile_operation_gate("gitea.read") and its immediate re-check resolved the same observational permission, so plan and apply shared one permission class and any profile that could look could also destroy. Apply now requires a dedicated capability, gitea.worker_registry.retire, declared once as RETIREMENT_MUTATION_PERMISSION in gitea_mcp_server.py and mapped in task_capability_map.py for both apply_stale_worker_retirement and gitea_apply_stale_worker_retirement. It is enforced at tool entry and re-resolved through task_capability_map.required_permission immediately before the registry mutation, so the two cannot drift. Plan keeps gitea.read and remains observational. The authorized role stays controller/reconciler, now as defence in depth behind the capability rather than the sole authority: an author, reviewer, or merger is refused even if mistakenly granted the permission. No profile holds the new permission by default, so author, reviewer, merger, and ordinary read-only profiles fail closed on the permission itself. No new Gitea write permission is introduced, no author permission is broadened, and retired_by and the acting identity still come from server-resolved context, never from caller input.

B2 — affirmative identity and liveness proof. "Incomplete identity" previously meant only that pre-existing columns were non-null, which a legacy-pid-… row satisfies trivially, so a row that proved nothing about which process it described was retireable. Retirement now requires a positive two-part proof in assess_retirement_identity_proof. Attribution: the client_instance_id must be launcher-minted inst-…, decided through the existing mcp_fleet_snapshot.assess_instance_identity rather than a re-implemented rule. Fencing: new host_id, boot_id, and process_start_time columns, added additively and probed by the new mcp_process_fencing module, which turn a bare pid into a statement about one process incarnation — which machine, which boot, which occupant of that pid number. Every probe returns None rather than guessing, and None always preserves. Rows written before these columns existed, and any row on a legacy instance identity, are preserved permanently and are retired only after their worker re-registers under a trusted identity. That legacy rows would otherwise remain outstanding indefinitely is explicitly not treated as grounds for a weaker proof, per the review's instruction. Also preserved fail-closed with structured reasons: unproven or mismatched host, unknown boot identity on either side, pid reuse (reported distinctly from a live worker), a second active row claiming the same instance and namespace while one may be live, conflicting session or generation evidence, live or unresolved lease ownership, and any active worker. A live pid remains an absolute block even across a boot boundary. Multiple processes of one legitimate cohort — one instance across distinct namespaces — remain distinguished from independent workers, which is why the conflict key is the (instance, namespace) pair rather than the instance alone.

B3 — external-state fencing and transaction safety. BEGIN IMMEDIATE locks the worker registry and nothing else, and the per-target loop runs after revalidation, so a lease acquired or a pid revived in that window went unnoticed — the registry-column guard could not catch it because no registry column changed. Two mechanisms now close it, both applied per target immediately before its own write. external_fence_fn is a version token over active leases from external_state_fingerprint, captured inside the transaction before the authoritative read and re-compared before every guarded UPDATE; movement raises and rolls the whole transaction back, because a changed world invalidates every remaining decision, and an unreadable lease store raises rather than silently comparing equal. liveness_fn re-probes process liveness and fencing identity and must affirmatively re-establish that this exact process is gone, comparing process_start_time so a pid reused since the plan is refused. A caller supplying no re-probe retires nothing rather than proceeding unfenced. The guarded UPDATE additionally asserts host_id, boot_id, and process_start_time are unchanged, and all three participate in the CAS token, so fencing movement alone aborts. A rollback reports external_state_moved or transaction_failed with success: false, retired_count: 0, and mutation_performed: false.

What is true now

  • server-side decision state: review 657 is recorded against the superseded head c0c6d14b73add36ac157d501c603fd9ea51de1da and now reads stale: true, review_feedback_stale: true, author_pushed_after_request_changes: true; approval_visible: false and approval_at_current_head: false. No verdict exists at the new head, and no merge operation was performed by this session.
  • local verdict/state: implementation complete and verified in worktree branches/issue-980-stale-worker-retirement, clean after commit including untracked files.
  • PR #982 read back natively after the push: state open, base master, head branch feat/issue-980-stale-worker-retirement, mergeable: true, author jcwalker3, Closes #980 present, live head equal to the pushed commit e344e68a1345cb21db06776eef839f863b81f3e5.
  • the PR description still describes the superseded head and its now-corrected claims that apply is gated by gitea.read and that trusted inst-… provenance is not required. An author cannot edit a PR description through the sanctioned tools (gitea_edit_pr has no worktree_path parameter, the #618 wall), so this comment is the authoritative account of the new head.

What changed

  • Server-side: branch feat/issue-980-stale-worker-retirement non-force pushed c0c6d14..e344e68; this comment. No label change, no PR creation, no review verdict, no merge operation.
  • Local-only: seven files in one commit — mcp_process_fencing.py (new), mcp_fleet_retirement.py, mcp_worker_identity.py, gitea_mcp_server.py, task_capability_map.py, docs/stale-worker-retirement.md, tests/test_issue_980_stale_worker_retirement.py.
  • Preserved from the accepted prior work and re-verified: registry_fingerprint still excludes observation time, is order-stable and numeric-typing stable; plan still performs no mutation; registry or candidate movement still retires zero; active and uncertain workers remain fail-closed; retired rows remain historical and idempotent; ordinary author operations are not fleet-gated.

Tests

  • Focused tests/test_issue_980_stale_worker_retirement.py: 87 passed, 46 subtests passed, up from 40 passed / 23 subtests at the reviewed head. New classes cover each blocker's required regressions — DedicatedMutationCapabilityTests (read-only insufficient, author/reviewer/merger denied, granted-but-wrong-role denied, revoking the capability fails closed, granted controller reaches the isolated apply path on a temp database, plan stays read-only), AffirmativeIdentityProofTests (trusted stale row eligible, missing instance identity, legacy row without trusted provenance, each missing fencing column, pid reuse, same pid on another host, unprovable local host, unknown boot, prior boot, conflicting session, conflicting client instance, conflicting generation, live lease, unknown lease state, active worker, one cohort of five, independent workers, mixed eligible and uncertain, structured reason on every uncertain row), and ExternalStateFencingTests plus ExternalStateFingerprintTests (lease acquired between plan and apply, fence moving mid-loop, unreadable lease store, worker becoming active, pid reuse before the write, host or boot movement before the write, omitted re-probe, heartbeat movement, generation and fencing movement, fencing columns in the CAS token, concurrent applies, later-candidate failure and rollback, database failure, counts matching committed state, fence-before-read ordering).
  • Adjacent suites (#980, #978, #975, #948, task-capability role invariants): 242 passed, 156 subtests passed, zero failures.
  • Full suite from the branch worktree at e344e68: 28 failed, 6253 passed, 6 skipped, 1152 subtests passed.
  • Current-master baseline from branches/baseline-980-108cbfa detached at 108cbfa173de34470d7abc022b929e6b3adc418a, clean: 28 failed, 6166 passed, 6 skipped, 1106 subtests passed.
  • Baseline comparison by exact test identifier, not totals: the sorted FAILED lines were captured from each worktree and compared with comm; both directions are empty, so zero regressions were introduced and zero pre-existing failures were masked. The +87 passing delta is this branch's tests. Nothing was deselected; 6 skips are identical on both sides; no test timed out, and no flaky or environment-limited result was observed across the runs. The 28 pre-existing failures are unchanged and out of scope.

What is blocked

  • blocker classification: no blocker
  • Nothing obstructs the author phase. PR #982 stays open until an independent reviewer records a verdict at the new head and a merger acts; that is ordinary sequencing, not a failing gate.
  • One operator follow-up is required before the capability can ever be exercised, and it is deliberately outside this PR: no profile holds gitea.worker_registry.retire, so apply is inert until an operator adds it to the allowed_operations of the controller or reconciler profile in profiles.json. That is a configuration change, and configuration was not modified by this work.

Who/what acts next

  • next actor: prgs-reviewer
  • required action: independently review PR #982 at exact head e344e68a1345cb21db06776eef839f863b81f3e5, re-proving dedicated mutation authorization, affirmative identity and liveness safety, external-state fencing, transaction atomicity, and the regression tests before deciding a new formal verdict.

Do not do: merge PR #982, self-review it, record a verdict from the author namespace, treat review 657 as approving the new head, grant gitea.worker_registry.retire to any profile as part of this review, activate the BAA or any recovery grant, retire any real worker-registry row, or touch #981, PR #906, #650, review 624, branch feat/issue-650-providers-insights, or worktree branches/conflict-fix-pr-906-983e8ac.

WHY: Review 657 recorded three hard-rule blockers at the prior head — apply authorized solely by gitea.read, complete legacy rows retireable without affirmative identity proof, and lease plus OS-liveness evidence unfenced at write time. Each is corrected at the new head with the required regression coverage, and because the head advanced the prior verdict is stale by construction, so an independent reviewer must decide afresh.

RELATED_PRS: PR #982 implements this issue and now sits at head e344e68a1345cb21db06776eef839f863b81f3e5. PR #979 (#978) is the prerequisite that already landed on master at 108cbfa173de34470d7abc022b929e6b3adc418a and supplies the fleet snapshot this work builds on. PR #971 (#969) is an adjacent open PR retiring control-plane session rows in a different registry; it remains a textual conflict risk, not a dependency, and its start-time approach informed the fencing design here. PR #906 (#650) is untouched.

BLOCKERS: None for the author phase; blocker classification: no blocker.

VALIDATION: prgs-author on the gitea-author namespace; identity jcwalker3 verified by gitea_whoami with identity_match: true, profile prgs-author, role author, remote prgs, repository Scaled-Tech-Consulting/Gitea-Tools. Startup, runtime, local and live remote master head all 108cbfa173de34470d7abc022b929e6b3adc418a; in_parity: true, stale: false, live_stale: false, parity-layer mutation_safe: true. Control checkout clean on master including untracked files and equal to live master; all implementation, testing, commit, and push happened in the bound author worktree, never in the control checkout. Capabilities resolved immediately before each mutation in whoami-then-resolve order: push_branch and comment_issue, both allowed_in_current_session: true with stop_required: false and mutation_performed: false. No live worker-registry row was retired: every retirement test constructs a throwaway SQLite database through tempfile.mkstemp, no apply ran against the live registry, and the new capability is unheld by every profile so it could not have run. Issue #981, PR #906, issue #650, review 624, the BAA and recovery-grant proposals, unrelated branches and worktrees, profiles, configuration, credentials, sessions, and running MCP processes were not modified. Labels on #980 were left exactly as found.

LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author)

WHO_IS_NEXT: prgs-reviewer to independently review the new exact PR #982 head e344e68a1345cb21db06776eef839f863b81f3e5

[THREAD STATE LEDGER] ## Canonical Issue State STATE: review 657 remediated — PR #982 advanced to exact head `e344e68a1345cb21db06776eef839f863b81f3e5`, awaiting a fresh independent review WHO_IS_NEXT: reviewer NEXT_ACTION: prgs-reviewer independently reviews PR #982 at exact head `e344e68a1345cb21db06776eef839f863b81f3e5` and decides a new formal verdict; review 657 is stale and approves nothing. NEXT_PROMPT: ```text As prgs-reviewer on prgs for Scaled-Tech-Consulting/Gitea-Tools: independently review PR #982 (issue #980) at exact head e344e68a1345cb21db06776eef839f863b81f3e5, base master 108cbfa173de34470d7abc022b929e6b3adc418a. Re-prove all three review-657 blockers are closed: (B1) gitea_apply_stale_worker_retirement requires gitea.worker_registry.retire at every enforcement layer and a gitea.read-only profile cannot resolve or invoke it; (B2) retirement requires affirmative launcher-minted inst- attribution plus host/boot/process-start fencing, and every incomplete, legacy, ambiguous, or unproven row is preserved with a structured reason; (B3) lease and OS-liveness evidence is re-fenced inside the transaction immediately before each guarded UPDATE and any movement rolls back with mutation_performed false. Verify the stable registry CAS work from the prior head did not regress. Do not merge without separate merger authorization. ``` WHAT_HAPPENED: Addressed only the three blocking findings of formal review 657 (REQUEST_CHANGES by sysadmin at head `c0c6d14b73add36ac157d501c603fd9ea51de1da`) and nothing else. One new commit `e344e68a1345cb21db06776eef839f863b81f3e5` on branch `feat/issue-980-stale-worker-retirement`, seven files, non-force fast-forward push `c0c6d14..e344e68`. B1 — dedicated mutation authorization. Apply was gated by `_profile_operation_gate("gitea.read")` and its immediate re-check resolved the same observational permission, so plan and apply shared one permission class and any profile that could look could also destroy. Apply now requires a dedicated capability, `gitea.worker_registry.retire`, declared once as `RETIREMENT_MUTATION_PERMISSION` in `gitea_mcp_server.py` and mapped in `task_capability_map.py` for both `apply_stale_worker_retirement` and `gitea_apply_stale_worker_retirement`. It is enforced at tool entry and re-resolved through `task_capability_map.required_permission` immediately before the registry mutation, so the two cannot drift. Plan keeps `gitea.read` and remains observational. The authorized role stays controller/reconciler, now as defence in depth behind the capability rather than the sole authority: an author, reviewer, or merger is refused even if mistakenly granted the permission. No profile holds the new permission by default, so author, reviewer, merger, and ordinary read-only profiles fail closed on the permission itself. No new Gitea write permission is introduced, no author permission is broadened, and `retired_by` and the acting identity still come from server-resolved context, never from caller input. B2 — affirmative identity and liveness proof. "Incomplete identity" previously meant only that pre-existing columns were non-null, which a `legacy-pid-…` row satisfies trivially, so a row that proved nothing about which process it described was retireable. Retirement now requires a positive two-part proof in `assess_retirement_identity_proof`. Attribution: the `client_instance_id` must be launcher-minted `inst-…`, decided through the existing `mcp_fleet_snapshot.assess_instance_identity` rather than a re-implemented rule. Fencing: new `host_id`, `boot_id`, and `process_start_time` columns, added additively and probed by the new `mcp_process_fencing` module, which turn a bare pid into a statement about one process incarnation — which machine, which boot, which occupant of that pid number. Every probe returns None rather than guessing, and None always preserves. Rows written before these columns existed, and any row on a legacy instance identity, are preserved permanently and are retired only after their worker re-registers under a trusted identity. That legacy rows would otherwise remain outstanding indefinitely is explicitly not treated as grounds for a weaker proof, per the review's instruction. Also preserved fail-closed with structured reasons: unproven or mismatched host, unknown boot identity on either side, pid reuse (reported distinctly from a live worker), a second active row claiming the same instance and namespace while one may be live, conflicting session or generation evidence, live or unresolved lease ownership, and any active worker. A live pid remains an absolute block even across a boot boundary. Multiple processes of one legitimate cohort — one instance across distinct namespaces — remain distinguished from independent workers, which is why the conflict key is the (instance, namespace) pair rather than the instance alone. B3 — external-state fencing and transaction safety. `BEGIN IMMEDIATE` locks the worker registry and nothing else, and the per-target loop runs after revalidation, so a lease acquired or a pid revived in that window went unnoticed — the registry-column guard could not catch it because no registry column changed. Two mechanisms now close it, both applied per target immediately before its own write. `external_fence_fn` is a version token over active leases from `external_state_fingerprint`, captured inside the transaction before the authoritative read and re-compared before every guarded UPDATE; movement raises and rolls the whole transaction back, because a changed world invalidates every remaining decision, and an unreadable lease store raises rather than silently comparing equal. `liveness_fn` re-probes process liveness and fencing identity and must affirmatively re-establish that this exact process is gone, comparing `process_start_time` so a pid reused since the plan is refused. A caller supplying no re-probe retires nothing rather than proceeding unfenced. The guarded UPDATE additionally asserts `host_id`, `boot_id`, and `process_start_time` are unchanged, and all three participate in the CAS token, so fencing movement alone aborts. A rollback reports `external_state_moved` or `transaction_failed` with `success: false`, `retired_count: 0`, and `mutation_performed: false`. ### What is true now - server-side decision state: review 657 is recorded against the superseded head `c0c6d14b73add36ac157d501c603fd9ea51de1da` and now reads `stale: true`, `review_feedback_stale: true`, `author_pushed_after_request_changes: true`; `approval_visible: false` and `approval_at_current_head: false`. No verdict exists at the new head, and no merge operation was performed by this session. - local verdict/state: implementation complete and verified in worktree `branches/issue-980-stale-worker-retirement`, clean after commit including untracked files. - PR #982 read back natively after the push: state open, base `master`, head branch `feat/issue-980-stale-worker-retirement`, `mergeable: true`, author `jcwalker3`, `Closes #980` present, live head equal to the pushed commit `e344e68a1345cb21db06776eef839f863b81f3e5`. - the PR description still describes the superseded head and its now-corrected claims that apply is gated by `gitea.read` and that trusted `inst-…` provenance is not required. An author cannot edit a PR description through the sanctioned tools (`gitea_edit_pr` has no `worktree_path` parameter, the #618 wall), so this comment is the authoritative account of the new head. ### What changed - Server-side: branch `feat/issue-980-stale-worker-retirement` non-force pushed `c0c6d14..e344e68`; this comment. No label change, no PR creation, no review verdict, no merge operation. - Local-only: seven files in one commit — `mcp_process_fencing.py` (new), `mcp_fleet_retirement.py`, `mcp_worker_identity.py`, `gitea_mcp_server.py`, `task_capability_map.py`, `docs/stale-worker-retirement.md`, `tests/test_issue_980_stale_worker_retirement.py`. - Preserved from the accepted prior work and re-verified: `registry_fingerprint` still excludes observation time, is order-stable and numeric-typing stable; plan still performs no mutation; registry or candidate movement still retires zero; active and uncertain workers remain fail-closed; retired rows remain historical and idempotent; ordinary author operations are not fleet-gated. ### Tests - Focused `tests/test_issue_980_stale_worker_retirement.py`: 87 passed, 46 subtests passed, up from 40 passed / 23 subtests at the reviewed head. New classes cover each blocker's required regressions — `DedicatedMutationCapabilityTests` (read-only insufficient, author/reviewer/merger denied, granted-but-wrong-role denied, revoking the capability fails closed, granted controller reaches the isolated apply path on a temp database, plan stays read-only), `AffirmativeIdentityProofTests` (trusted stale row eligible, missing instance identity, legacy row without trusted provenance, each missing fencing column, pid reuse, same pid on another host, unprovable local host, unknown boot, prior boot, conflicting session, conflicting client instance, conflicting generation, live lease, unknown lease state, active worker, one cohort of five, independent workers, mixed eligible and uncertain, structured reason on every uncertain row), and `ExternalStateFencingTests` plus `ExternalStateFingerprintTests` (lease acquired between plan and apply, fence moving mid-loop, unreadable lease store, worker becoming active, pid reuse before the write, host or boot movement before the write, omitted re-probe, heartbeat movement, generation and fencing movement, fencing columns in the CAS token, concurrent applies, later-candidate failure and rollback, database failure, counts matching committed state, fence-before-read ordering). - Adjacent suites (#980, #978, #975, #948, task-capability role invariants): 242 passed, 156 subtests passed, zero failures. - Full suite from the branch worktree at `e344e68`: 28 failed, 6253 passed, 6 skipped, 1152 subtests passed. - Current-master baseline from `branches/baseline-980-108cbfa` detached at `108cbfa173de34470d7abc022b929e6b3adc418a`, clean: 28 failed, 6166 passed, 6 skipped, 1106 subtests passed. - Baseline comparison by exact test identifier, not totals: the sorted `FAILED` lines were captured from each worktree and compared with `comm`; both directions are empty, so zero regressions were introduced and zero pre-existing failures were masked. The +87 passing delta is this branch's tests. Nothing was deselected; 6 skips are identical on both sides; no test timed out, and no flaky or environment-limited result was observed across the runs. The 28 pre-existing failures are unchanged and out of scope. ### What is blocked - blocker classification: no blocker - Nothing obstructs the author phase. PR #982 stays open until an independent reviewer records a verdict at the new head and a merger acts; that is ordinary sequencing, not a failing gate. - One operator follow-up is required before the capability can ever be exercised, and it is deliberately outside this PR: no profile holds `gitea.worker_registry.retire`, so apply is inert until an operator adds it to the `allowed_operations` of the controller or reconciler profile in `profiles.json`. That is a configuration change, and configuration was not modified by this work. ### Who/what acts next - next actor: prgs-reviewer - required action: independently review PR #982 at exact head `e344e68a1345cb21db06776eef839f863b81f3e5`, re-proving dedicated mutation authorization, affirmative identity and liveness safety, external-state fencing, transaction atomicity, and the regression tests before deciding a new formal verdict. Do not do: merge PR #982, self-review it, record a verdict from the author namespace, treat review 657 as approving the new head, grant `gitea.worker_registry.retire` to any profile as part of this review, activate the BAA or any recovery grant, retire any real worker-registry row, or touch #981, PR #906, #650, review `624`, branch `feat/issue-650-providers-insights`, or worktree `branches/conflict-fix-pr-906-983e8ac`. WHY: Review 657 recorded three hard-rule blockers at the prior head — apply authorized solely by `gitea.read`, complete legacy rows retireable without affirmative identity proof, and lease plus OS-liveness evidence unfenced at write time. Each is corrected at the new head with the required regression coverage, and because the head advanced the prior verdict is stale by construction, so an independent reviewer must decide afresh. RELATED_PRS: PR #982 implements this issue and now sits at head `e344e68a1345cb21db06776eef839f863b81f3e5`. PR #979 (#978) is the prerequisite that already landed on master at `108cbfa173de34470d7abc022b929e6b3adc418a` and supplies the fleet snapshot this work builds on. PR #971 (#969) is an adjacent open PR retiring control-plane session rows in a different registry; it remains a textual conflict risk, not a dependency, and its start-time approach informed the fencing design here. PR #906 (#650) is untouched. BLOCKERS: None for the author phase; blocker classification: no blocker. VALIDATION: prgs-author on the `gitea-author` namespace; identity `jcwalker3` verified by `gitea_whoami` with `identity_match: true`, profile `prgs-author`, role `author`, remote `prgs`, repository `Scaled-Tech-Consulting/Gitea-Tools`. Startup, runtime, local and live remote master head all `108cbfa173de34470d7abc022b929e6b3adc418a`; `in_parity: true`, `stale: false`, `live_stale: false`, parity-layer `mutation_safe: true`. Control checkout clean on `master` including untracked files and equal to live master; all implementation, testing, commit, and push happened in the bound author worktree, never in the control checkout. Capabilities resolved immediately before each mutation in whoami-then-resolve order: `push_branch` and `comment_issue`, both `allowed_in_current_session: true` with `stop_required: false` and `mutation_performed: false`. No live worker-registry row was retired: every retirement test constructs a throwaway SQLite database through `tempfile.mkstemp`, no apply ran against the live registry, and the new capability is unheld by every profile so it could not have run. Issue #981, PR #906, issue #650, review `624`, the BAA and recovery-grant proposals, unrelated branches and worktrees, profiles, configuration, credentials, sessions, and running MCP processes were not modified. Labels on #980 were left exactly as found. LAST_UPDATED_BY: jcwalker3 / prgs-author (gitea-author) `WHO_IS_NEXT: prgs-reviewer to independently review the new exact PR #982 head e344e68a1345cb21db06776eef839f863b81f3e5`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels control-plane mcp-health safety status:pr-open type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#980