fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972
Merged
sysadmin
merged 2 commits from 2026-07-29 10:22:21 -05:00
fix/issue-970-safely-resolve-missing-worktrees into master
Labels
Clear labels
allocator
anti-stomp
architecture
bug
chore
codex
concurrency
contamination
control-plane
dashboard
database
design
documentation
enhancement
gitea
glitchtip
important
incident
incident-bridge
integration
jenkins
labels
leases
mcp
mcp-health
mcp-menu
multi-project
mutating
nice-to-have
observability
portability
preflight
protected-branch
queue
read-only
reconnect
recovery
refactor
release
reliability
resumable-review
reviewer
roadmap
safety
security
self-hosted
sentry
stale-runtime
status:blocked
status:in-progress
status:pr-open
status:ready
terminal-lock
testing
tracker
type:bug
type:feature
type:feature
type:guardrail
visibility
workflow
workflow-hardening
workflow-hardening
Controller-owned work allocator
Prevent concurrent LLM session stomping
Architecture / structural design
OpenAI Codex client / workflow session surface
Concurrent session safety
Workflow or session contamination incident
MCP control-plane coordination and allocation authority
MCP operational dashboard/queue view
Internal coordination storage (SQLite/Postgres)
Design / investigation, no implementation
Docs / runbooks
New feature or improvement
Gitea MCP workflow
GlitchTip integration
Operational or process incident requiring durable audit trail
Sentry-to-Gitea incident bridging
Integration testing
Jenkins integration
Label taxonomy management
Lease adopt/release/expire lifecycle
MCP server / tooling
MCP namespace and runtime health
MCP menu surface
Work spanning multiple monitoring projects or Gitea repos
Mutating action; requires gating
Observability, metrics, traces, error reporting
Cross-platform / portability
Shared preflight gates before mutation
Protected branch / stable-branch policy concern
Work queue visibility and allocation
Read-only, no mutation
MCP client reconnect/reload recovery path
Recovery paths for stale/foreign leases
Code refactor / restructure
Release / versioning
Reliability / failure handling
Persist and resume prepared review verdicts across sessions
Reviewer workflow tooling
Roadmap / umbrella issue
Safety rails and fail-closed mutation guards
Security / trust boundary
Self-hosted infrastructure integration
Sentry error monitoring integration
Stale backend daemon / runtime-vs-master parity failures
Issue is blocked
Issue is being worked on
Issue has an open pull request
Issue is ready for work
Terminal review lock (#332) path
Tests / test coverage
Issue tracker hygiene / meta
Bug or defect
Feature or enhancement
Feature or enhancement
Safety gate or guardrail
Workflow state visibility for LLMs/operators
Cross-tool workflow
LLM workflow coordination hardening
LLM workflow coordination hardening
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Scaled-Tech-Consulting/Gitea-Tools#972
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Resolves #970 by providing a sanctioned, auditable workflow to classify and safely repair/retire worktree bindings whose expected filesystem paths are missing without editing the control-plane database or Git worktree metadata manually.
Key components implemented:
missing_worktree_reconcile.py: Module providing audit, classification, pre-mutation re-validation, targeted retirement, and reconciliation for missing worktree path bindings.control_plane_db.py: Addedretire_lease_worktree_pathandretire_session_checkpoint_worktree_pathmethods with durable retirement audit provenance.gitea_mcp_server.py: Registeredgitea_audit_missing_worktree_bindingsandgitea_reconcile_missing_worktree_bindings.tests/test_issue_970_missing_worktree_reconcile.py: Verified all 10 acceptance criteria.worktreesdimension toresolved.repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 6437-327ca65b6a2f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-fix-issue-970-safely-resolve-missing-worktrees
phase: claimed
candidate_head:
3f584352dftarget_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T10:11:53Z
expires_at: 2026-07-29T10:21:53Z
blocker: none
repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 6437-327ca65b6a2f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-fix-issue-970-safely-resolve-missing-worktrees
phase: review_ready
candidate_head:
3f584352dftarget_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T10:13:29Z
expires_at: 2026-07-29T10:23:29Z
blocker: none
REQUEST_CHANGES — PR #972 @
3f584352dfIndependent review against issue #970. Focused tests pass (7/7) and adjacent suites pass (93), but the suite does not exercise several production safety gates; adversarial probes against the production path fail required fail-closed behavior.
Blocking findings
B1 — Stale-snapshot revalidation misses ownership / lease state changes
Files:
missing_worktree_reconcile.py(resolve_missing_worktree_binding)Failure path: Audit classifies a released lease as confirmed-stale; before apply another session re-activates the same lease; revalidation uses audit snapshot fields (not a fresh DB read); mutation clears worktree_path on the now-active lease.
Probe:
performed=TrueafterUPDATE leases SET status='active', owner_pid=<live>between audit and apply.Fix: Re-load exact lease/checkpoint row immediately before mutation; reclassify with live status/owner/session evidence; CAS on path+status (and related identity); fail closed on change.
B2 — MCP apply accepts client-asserted
operator_authorized; gated only bygitea.readFiles:
gitea_mcp_server.py(gitea_reconcile_missing_worktree_bindings)Failure path: Any
gitea.readprofile (including reviewer) can passdry_run=False, operator_authorized=Trueand mutate control-plane DB. Project pattern (#709 / review 434) rejects client-supplied operator_authorized as authorization evidence.Fix: Reconciler-only (or explicit mutation capability) server-side gate; confirmation/workflow proof; hard-reject self-assertable operator_authorized for apply.
B3 — Checkpoint retirement lacks expected-path CAS
Files:
control_plane_db.py(retire_session_checkpoint_worktree_path)Probe: wrong
expected_pathstill returnsretired=True.Fix: Mirror lease CAS: refuse when stored realpath ≠ expected.
B4 — Live-session protection not wired in audit
Files:
missing_worktree_reconcile.pysession_activenever set by audit (always false). Active lease with dead recorded PID can classify confirmed-stale.issue_lock_storeimported but unused.Fix: Derive live session/owner evidence; protect active leases unless terminal; audit issue-lock bindings if applicable.
B5 — Tests would not fail if safety gates removed
File:
tests/test_issue_970_missing_worktree_reconcile.pyMissing adversarial coverage for B1–B3, MCP auth fail-closed, transient FS vs missing, terminal vs active matrix, production tool registration.
Fix: Add tests that exercise production paths and fail if gates are stripped.
Non-blocking
Passed
Dry-run non-mutating; moved-path and missing-root host fail closed in unit tests; same-path recreation revalidation fails closed; lease expected-path CAS works; idempotent re-run in unit tests.
Verdict: REQUEST_CHANGES at
3f584352df279629d80d2aaa0be4e6dd66c56fc9. No merge. Author addresses B1–B5 only under sanctioned author workflow for #970.Reviewer: sysadmin / prgs-reviewer | lease session
6437-327ca65b6a2f| worktreebranches/review-fix-issue-970-safely-resolve-missing-worktrees| #969/#971 untouched.repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 6437-327ca65b6a2f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-fix-issue-970-safely-resolve-missing-worktrees
phase: released
candidate_head:
3f584352dftarget_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T10:14:38Z
expires_at: 2026-07-29T10:24:38Z
blocker: manual-release
Canonical PR State
STATE: changes-addressed-awaiting-independent-re-review
WHO_IS_NEXT: reviewer
NEXT_ACTION: Independently re-review PR #972 at head
c763161702against issue #970 and review 644 blockers B1-B5, then post a formal verdict at that exact head.NEXT_PROMPT:
WHAT_HAPPENED: Author remediation of review 644 landed as commit
c763161702on the existing branch fix/issue-970-safely-resolve-missing-worktrees. No second pull request was created; PR #972 advanced in place and still closes #970. Blocker-by-blocker mapping follows.missing_worktree_reconcile.py(resolve_missing_worktree_binding, newread_live_safety_state,compare_safety_state,_safety_snapshot,SAFETY_SNAPSHOT_FIELDS);control_plane_db.py(retire_lease_worktree_pathexpected-status/session/owner-pid compare-and-swap). Correction: the audit records a normalized safety snapshot per binding, and apply re-reads the authoritative lease row, session rows, checkpoint row, and durable issue locks immediately before mutating, diffs every safety-relevant value (binding path and identity, lease id/status/session/owner pid, checkpoint path/status, live-session evidence, trustworthy ownership evidence, issue-lock state), fails closed without mutation on any change, and reclassifies from the live values rather than the snapshot; a candidate carrying no audited snapshot is refused. The unguarded secondaryretire_lease_worktree_pathcall that swallowed its own exception was removed. Adversarial coverage:TestB1StaleSnapshotRevalidation(7 tests) including the reviewer's exact probe (released at audit, re-activated with a live pid before apply), session-id change, owner-pid change, stored-path change, a live session appearing after audit, a missing audited snapshot, and proof that revalidation reads the DB rather than the candidate. Result: 7 passed; disabling the drift check turns 6 of them red.gitea_mcp_server.py(gitea_reconcile_missing_worktree_bindings);missing_worktree_reconcile.py(assess_cleanup_authorization,_authorization_valid_for_apply,OPERATOR_AUTHORIZED_REJECTION);task_capability_map.py(new reconciler-onlyreconcile_missing_worktree_bindingscapability requiringgitea.branch.delete, added toROLE_EXCLUSIVE_TASKS). Correction: a client-suppliedoperator_authorizedis rejected outright at the tool before any other work and again in the module, per the #709 F1 / review 434 pattern. Apply mode now requires, together, the reconciler role, a server-evaluatedgitea.branch.deletecapability, the resolved cleanup task capability, and an active authorized cleanup phase fromaudit_reconciliation_mode- which is re-checked at the mutation boundary itself, so a forged authorization mapping cannot substitute for the gate. Dry-run remains available to anygitea.readprofile and stays non-mutating. Existing role, repository, parity, and provenance gates are untouched. Adversarial coverage:TestB2ServerEnforcedCleanupAuthorization(8 tests) andTestB2ProductionMcpBoundary(8 tests) driving the registered tools throughmcp_server: reviewer withoperator_authorized=true, reviewer without cleanup capability, an author profile holdinggitea.branch.deletebut the wrong role, a correctly-profiled reconciler with no minted cleanup phase, a forged authorization mapping, non-mutating dry-run, and an authorized reconciler apply that does retire. Result: 16 passed; restoring the old client-boolean trust turns the reviewer probe red.control_plane_db.py(retire_session_checkpoint_worktree_path, mirrored inretire_lease_worktree_path, new_realpath_or_rawhelper). Correction:expected_pathis now mandatory and enforced; the row is selected, its stored realpath compared, an optional expected status compared, and the UPDATE keyed on the exact stored path with a rowcount check, so a path moved, replaced, or concurrently changed after audit is refused without mutation. Unknown rows and ambiguous session selectors are refused; an already-empty binding returns a non-mutating idempotent outcome instead of falsely reporting a retirement. Adversarial coverage:TestB3ExpectedPathCompareAndSwap(8 tests) including the reviewer's exact probe (wrongexpected_pathpreviously returnedretired=true), missingexpected_path, unknown row, exact-path success plus idempotent repeat, and the lease-side status and session mismatches. Result: 8 passed; removing the checkpoint path comparison turns the reviewer probe red.missing_worktree_reconcile.py(collect_session_evidence,collect_issue_lock_bindings,find_conflicting_issue_lock,classify_worktree_binding,TERMINAL_LEASE_STATUSES,SESSION_HEARTBEAT_STALE_SECONDS). Correction:session_activeis now derived from the control-plane sessions table instead of never being set, along two axes - genuine liveness (recorded active, pid not dead, heartbeat fresh, reusing the rule fromrestart_coordinator._classify_session) and weaker recorded ownership. A non-terminal lease protects its binding regardless of recorded-pid liveness, so dead-pid evidence alone can no longer retire a lease the control plane still holds. The previously unusedissue_lock_storeis now read: a live durable issue lock binding the path or branch blocks cleanup, and locks whose own paths are missing are reported in a separate dimension for release through their own lifecycle rather than retired here. Precedence is explicit and documented: an explicitly terminal lease outranks a merely recorded-active session row, because session rows are opened on assignment and never closed on release - treating one as protective would make every released lease permanently unretireable and leave the worktrees dimension degraded forever, which is the condition #970 exists to clear. Adversarial coverage:TestB4OwnershipEvidence(9 tests): active lease with a dead recorded pid,session_activeactually populated, a live session blocking a released lease, trustworthy owner evidence without a live process, a released lease outranking a lingering session row, a checkpoint binding protected by its owning session, a conflicting issue lock blocking cleanup, a missing-path lock reported but never retired, and a dead-owner lock not over-blocking an unrelated binding. Result: 9 passed.mcp_server(which execsgitea_mcp_server.py), the realControlPlaneDBagainst a temporary SQLite control plane, the realaudit_reconciliation_modecleanup gate, the realissue_lock_storereader via its own lock-directory environment variable, and the realtask_capability_map. Coverage spans lease status change between audit and apply, lease ownership or session change, binding and checkpoint path change, expected-path mismatch, concurrent recreation, unauthorized caller withoperator_authorized=true, wrong profile and missing cleanup capability, live-session evidence, trustworthy owner evidence, active or conflicting issue lock, non-mutating dry-run, exact stale binding only retired, unrelated worktrees and git metadata preserved, repeated execution safe and idempotent, valid cleanup resolving the worktrees dimension, and production tool registration. Result: 47 passed. Mutation-tested: disabling the B1 drift check produced 6 failures, restoring the B3 checkpoint bug produced 1 failure, and re-honouring the clientoperator_authorizedproduced 1 failure - each protection is load-bearing.All original issue 970 acceptance criteria are preserved, including the distinctions between deleted paths, moved paths, unavailable hosts or mounts, transient filesystem failures, live ownership, and concurrent recreation. Scope was held to review 644: allocator queue ranking, the earlier abandon-audit same-owner anomaly, issue 711, issue 969, PR 971, unrelated refactoring, and manual database repair were all left untouched, and no control-plane row was edited by hand.
WHY: Review 644 established that the workflow could mutate control-plane state on evidence it had not revalidated, on authorization a client could assert for itself, and through a checkpoint update that ignored the path it claimed to be retiring, while the tests would have stayed green if those protections were deleted. Each correction closes the specific failure path the reviewer demonstrated, and the new tests fail when the corresponding protection is removed.
RELATED_PRS: PR #972 (this pull request, the only pull request for issue 970; no second pull request was created).
ISSUE: 970
HEAD_SHA:
c763161702REVIEW_STATUS: Review 644 (REQUEST_CHANGES by sysadmin) was submitted at the old head
3f584352dfand now reports stale=true with author_pushed_after_request_changes=true. It is not a verdict on this remediation. No verdict exists at the new head; approval_visible is false and approval_at_current_head is false.MERGE_READY: false
BLOCKERS: None known to the author. A fresh independent reviewer verdict at head
c763161702is required before any merge consideration.VALIDATION: Focused suite tests/test_issue_970_missing_worktree_reconcile.py 47 passed. Adjacent suites (task capability role invariants, audit reconciliation mode, control plane DB, lease lifecycle, reconciler cleanup integration, delete-branch capability, restart coordinator, bootstrap lock contract) 252 passed with 93 subtests. Full suite from the branches worktree 28 failed / 6000 passed / 6 skipped; the pre-change baseline built from a detached worktree at
3f584352dfgave 28 failed / 5960 passed, and a sorted diff of the two failing-id lists is empty - the failing set is identical and pre-existing, and it includes the tests/test_issue_956_threat_model.py anchor-resolution test, which already failed at the reviewed head and was left untouched as outside review 644. Final diff reviewed against master: six files, all changes scoped to issue 970 and review 644; the shared-file hunks are additive capability entries only. No unrelated changes.LAST_UPDATED_BY: author (jcwalker3, prgs-author, gitea-author namespace), assignment asn-21624c33a3b84a74, lease lease-86e4ffa4c4b74bbe.
repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 8565-1befe046257e
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-pr972-c763161-rereview
phase: claimed
candidate_head:
c763161702target_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T14:57:18Z
expires_at: 2026-07-29T15:07:18Z
blocker: none
repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 8565-1befe046257e
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-pr972-c763161-rereview
phase: review_ready
candidate_head:
c763161702target_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T15:02:54Z
expires_at: 2026-07-29T15:12:54Z
blocker: none
Canonical PR State
STATE: approved-awaiting-merge
WHO_IS_NEXT: merger
NEXT_ACTION: Independently validate the APPROVE verdict at unchanged exact head
c763161702, then merge PR #972 only if all merge gates pass.NEXT_PROMPT:
WHAT_HAPPENED: Independent re-review of PR #972 remediation of review #644 completed. B1-B5 and #970 acceptance criteria verified on production paths. Formal APPROVE submitted at
c763161702. No merge performed.WHY: Review 644 blockers B1 (fresh revalidation), B2 (server-enforced auth), B3 (expected-path CAS), B4 (live-session/issue-lock evidence), and B5 (adversarial production tests) are independently satisfied at the remediation head. Focused and adjacent tests pass; full-suite failures are pre-existing and outside #970 scope; mutation probes confirm protections are load-bearing.
RELATED_PRS: PR #972 (only PR for issue #970).
ISSUE: 970
HEAD_SHA:
c763161702REVIEW_STATUS: APPROVED by sysadmin (prgs-reviewer) at
c763161702. Prior review #644 (REQUEST_CHANGES at3f58435) is stale.MERGE_READY: false - merger must independently re-validate approval at the unchanged exact head before merge.
BLOCKERS: none for approval; merge is a separate workflow role and is not authorized by this review.
VALIDATION: Focused tests/test_issue_970_missing_worktree_reconcile.py 47 passed. Adjacent suites 205 passed / 93 subtests. Full suite 28 failed / 6000 passed / 6 skipped; failing IDs unrelated to missing_worktree_reconcile. Mutation testing: B1/B2/B3/B4 protections fail tests when stripped.
NATIVE_REVIEW_PROOF: transport=native_mcp; entrypoint=mcp_server; token_fingerprint=3f44400aae8e3708; profile=prgs-reviewer; identity=sysadmin; session=8565-1befe046257e; pid=8565
LAST_UPDATED_BY: reviewer (sysadmin, prgs-reviewer, gitea-reviewer namespace), lease session 8565-1befe046257e, worktree branches/review-pr972-c763161-rereview.
APPROVE - PR #972 @
c763161702Independent re-review of remediation for review #644. B1-B5 resolved. Focused 47 passed; adjacent 205/93; full suite 28 failed/6000 passed (no #970 failures). Mutation probes confirm protections load-bearing. No merge. #969/#971 untouched.
Review Metadata:
repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 8565-1befe046257e
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-pr972-c763161-rereview
phase: released
candidate_head:
c763161702target_branch: master
target_branch_sha:
956fa15fe3last_activity: 2026-07-29T15:05:18Z
expires_at: 2026-07-29T15:15:18Z
blocker: manual-release
repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #972
issue: #970
reviewer_identity: sysadmin
profile: prgs-merger
session_id: 33745-4e2b3f268539
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-pr972-c763161-rereview
phase: claimed
candidate_head:
c763161702target_branch: master
target_branch_sha: none
last_activity: 2026-07-29T15:20:22Z
expires_at: 2026-07-29T15:30:22Z
blocker: none
Stale #332 review-decision lock cleanup (#594)
Status: APPLIED
sysadminprgs-merger2026-07-29T15:22:24.913874+00:00approveon PR fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) (#972)closed(merged=True)626be8b178cfc82ef9162dcf5765e64b682047162prgs-reviewerManual deletion of session-state files is not the workflow.
This path only clears a lock when the referenced PR is merged/closed.