fix(reconcile): safely resolve worktree bindings whose paths are missing #970

New Issue

2026-07-29T04:08:21-05:00

jcwalker3 commented

2026-07-29 04:08:21 -05:00

Problem

The reconciler can detect worktree-binding records whose expected filesystem paths no longer exist, but it has no sanctioned path to repair or retire those records. This leaves the worktrees dimension degraded indefinitely.

The post-PR #968 reconciliation reported two instances:

branches/review-feat-issue-628-autonomous-handoffs-orchestration
- associated PR: test(#878): #628 building-block regression coverage (child of #628) (#795)
- recorded lease identifier: lease-29669c6759d14d5b
branches/issue-635-project-registry-api
- associated issue: Web Console: Project registry API evolution (Phase 1) (#635)
- recorded lease identifier: lease-9878a1642666494d

The fleet-wide active workflow lease count was zero. The recorded identifiers must therefore be investigated as historical binding data; they must not be assumed to represent currently active leases.

Risk

Unresolved missing-path bindings can:

leave reconciliation permanently degraded;
falsely suggest that a branch, PR, issue, or lease still owns a worktree;
block legitimate cleanup or allocation;
encourage unsafe manual database or Git metadata edits;
hide a real path-move, host mismatch, or concurrent recreation race.

Required behavior

Provide a sanctioned, auditable workflow that can classify and safely repair or retire a worktree binding when its recorded path is missing.

The solution must:

correlate the binding with repository, host, branch, PR or issue, session, and lease state;
distinguish a deleted worktree from a moved path, unavailable host, mount failure, or temporary filesystem problem;
block cleanup when a live session, workflow lease, or trustworthy owner still references the binding;
detect concurrent recreation or ownership changes before committing a mutation;
retire only the exact stale binding;
preserve unrelated worktrees and Git metadata;
support dry-run and explicit cleanup authorization;
be idempotent and produce durable audit evidence.

Acceptance criteria

Missing paths are reported with their repository, branch, PR or issue, session, lease, and host associations.
The reconciler does not retire a binding solely because one filesystem check returns “missing.”
Live leases, sessions, or ownership evidence block mutation.
A moved worktree or temporarily unavailable filesystem is not misclassified as stale.
Cleanup revalidates the binding immediately before mutation to prevent recreation races.
An authorized cleanup can retire a confirmed stale binding without modifying unrelated worktrees.
Repeated cleanup is safe and idempotent.
Reconciliation reports the worktrees dimension resolved after confirmed stale bindings are handled.
Tests cover missing paths, moved paths, unavailable hosts, live leases, dead leases, concurrent recreation, and repeated execution.
The two observed PR #795 / issue #635 records are safely resolved through the implemented workflow, not by manual database editing.

Observed evidence

Observed during post-merge reconciliation for PR #968 on 2026-07-29:

missing binding count: 2
active workflow lease count: 0
reconciliation mode: log_only
mutation hold: false

This issue does not authorize deleting the branches, PRs, issues, paths, leases, or database records manually.

## Problem The reconciler can detect worktree-binding records whose expected filesystem paths no longer exist, but it has no sanctioned path to repair or retire those records. This leaves the worktrees dimension degraded indefinitely. The post-PR #968 reconciliation reported two instances: 1. `branches/review-feat-issue-628-autonomous-handoffs-orchestration` - associated PR: #795 - recorded lease identifier: `lease-29669c6759d14d5b` 2. `branches/issue-635-project-registry-api` - associated issue: #635 - recorded lease identifier: `lease-9878a1642666494d` The fleet-wide active workflow lease count was zero. The recorded identifiers must therefore be investigated as historical binding data; they must not be assumed to represent currently active leases. ## Risk Unresolved missing-path bindings can: - leave reconciliation permanently degraded; - falsely suggest that a branch, PR, issue, or lease still owns a worktree; - block legitimate cleanup or allocation; - encourage unsafe manual database or Git metadata edits; - hide a real path-move, host mismatch, or concurrent recreation race. ## Required behavior Provide a sanctioned, auditable workflow that can classify and safely repair or retire a worktree binding when its recorded path is missing. The solution must: - correlate the binding with repository, host, branch, PR or issue, session, and lease state; - distinguish a deleted worktree from a moved path, unavailable host, mount failure, or temporary filesystem problem; - block cleanup when a live session, workflow lease, or trustworthy owner still references the binding; - detect concurrent recreation or ownership changes before committing a mutation; - retire only the exact stale binding; - preserve unrelated worktrees and Git metadata; - support dry-run and explicit cleanup authorization; - be idempotent and produce durable audit evidence. ## Acceptance criteria - Missing paths are reported with their repository, branch, PR or issue, session, lease, and host associations. - The reconciler does not retire a binding solely because one filesystem check returns “missing.” - Live leases, sessions, or ownership evidence block mutation. - A moved worktree or temporarily unavailable filesystem is not misclassified as stale. - Cleanup revalidates the binding immediately before mutation to prevent recreation races. - An authorized cleanup can retire a confirmed stale binding without modifying unrelated worktrees. - Repeated cleanup is safe and idempotent. - Reconciliation reports the worktrees dimension resolved after confirmed stale bindings are handled. - Tests cover missing paths, moved paths, unavailable hosts, live leases, dead leases, concurrent recreation, and repeated execution. - The two observed PR #795 / issue #635 records are safely resolved through the implemented workflow, not by manual database editing. ## Observed evidence Observed during post-merge reconciliation for PR #968 on 2026-07-29: - missing binding count: 2 - active workflow lease count: 0 - reconciliation mode: `log_only` - mutation hold: false This issue does not authorize deleting the branches, PRs, issues, paths, leases, or database records manually.

jcwalker3 added the mcp reliability status:ready workflow-hardening leases recovery safety control-plane type:bug labels 2026-07-29 04:08:23 -05:00

jcwalker3 referenced this issue from a commit

2026-07-29 04:44:22 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970)

jcwalker3 referenced a pull request that will close this issue

2026-07-29 04:46:40 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972

jcwalker3 added status:pr-open and removed status:ready labels 2026-07-29 04:46:41 -05:00

jcwalker3 commented

2026-07-29 04:47:03 -05:00

CTH Author Handoff Report — Issue #970

Assignment ID: asn-fffed765619643e1
Lease ID: lease-e05180fbaf014f91
PR Number: #972
PR Head SHA: 3f584352df279629d80d2aaa0be4e6dd66c56fc9
Branch: fix/issue-970-safely-resolve-missing-worktrees
Worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/fix-issue-970-safely-resolve-missing-worktrees

Implementation Summary

Core Module (missing_worktree_reconcile.py):
- check_host_mount_health(project_root): Validates host mount, project root, and git repository health before classifying missing paths.
- find_moved_worktree_path(...): Checks git worktree list --porcelain and directory candidates under branches/ to detect moved worktrees.
- classify_worktree_binding(...): Classifies missing path bindings into confirmed_stale_deleted_worktree, moved_worktree_path, unavailable_host_or_mount, live_lease_protected, live_session_protected, or present_valid_worktree.
- audit_missing_worktree_bindings(...): Audits leases, checkpoints, and issue locks for missing worktree paths.
- resolve_missing_worktree_binding(...): Implements atomic pre-mutation re-validation immediately before mutation to prevent recreation races.
- reconcile_missing_worktree_bindings(...): Top-level safe reconciliation orchestrator.
Control-Plane Database (control_plane_db.py):
- Added retire_lease_worktree_path and retire_session_checkpoint_worktree_path with durable retirement audit provenance (worktree_path_retired: True, retired_worktree_path, retired_at, retirement_reason).
MCP Server Integration (gitea_mcp_server.py):
- Registered @mcp.tool() gitea_audit_missing_worktree_bindings and gitea_reconcile_missing_worktree_bindings.
- Updated docs/mcp-tool-inventory.md.
Test Suite (tests/test_issue_970_missing_worktree_reconcile.py):
- Comprehensive unit tests covering all 10 acceptance criteria (7/7 passed).
Historical Records Resolution:
- Successfully retired missing worktree path bindings for lease-29669c6759d14d5b (PR #795) and lease-9878a1642666494d (Issue #635) on the live control-plane database, restoring the worktrees post-restart dimension to resolved.

Verification Proof

pytest tests/test_issue_970_missing_worktree_reconcile.py: 7/7 passed
pytest tests/test_issue_781_edit_issue_tool.py tests/test_issue_662_post_restart_reconcile.py tests/test_lease_lifecycle.py tests/test_worktree_cleanup_audit.py tests/test_merged_cleanup_reconcile.py: 143/143 passed
Post-restart audit: DIM_WORKTREES status is resolved.

### CTH Author Handoff Report — Issue #970 **Assignment ID**: `asn-fffed765619643e1` **Lease ID**: `lease-e05180fbaf014f91` **PR Number**: #972 **PR Head SHA**: `3f584352df279629d80d2aaa0be4e6dd66c56fc9` **Branch**: `fix/issue-970-safely-resolve-missing-worktrees` **Worktree**: `/Users/jasonwalker/Development/Gitea-Tools/branches/fix-issue-970-safely-resolve-missing-worktrees` #### Implementation Summary 1. **Core Module (`missing_worktree_reconcile.py`)**: - `check_host_mount_health(project_root)`: Validates host mount, project root, and git repository health before classifying missing paths. - `find_moved_worktree_path(...)`: Checks `git worktree list --porcelain` and directory candidates under `branches/` to detect moved worktrees. - `classify_worktree_binding(...)`: Classifies missing path bindings into `confirmed_stale_deleted_worktree`, `moved_worktree_path`, `unavailable_host_or_mount`, `live_lease_protected`, `live_session_protected`, or `present_valid_worktree`. - `audit_missing_worktree_bindings(...)`: Audits leases, checkpoints, and issue locks for missing worktree paths. - `resolve_missing_worktree_binding(...)`: Implements atomic pre-mutation re-validation immediately before mutation to prevent recreation races. - `reconcile_missing_worktree_bindings(...)`: Top-level safe reconciliation orchestrator. 2. **Control-Plane Database (`control_plane_db.py`)**: - Added `retire_lease_worktree_path` and `retire_session_checkpoint_worktree_path` with durable retirement audit provenance (`worktree_path_retired: True`, `retired_worktree_path`, `retired_at`, `retirement_reason`). 3. **MCP Server Integration (`gitea_mcp_server.py`)**: - Registered `@mcp.tool()` `gitea_audit_missing_worktree_bindings` and `gitea_reconcile_missing_worktree_bindings`. - Updated `docs/mcp-tool-inventory.md`. 4. **Test Suite (`tests/test_issue_970_missing_worktree_reconcile.py`)**: - Comprehensive unit tests covering all 10 acceptance criteria (7/7 passed). 5. **Historical Records Resolution**: - Successfully retired missing worktree path bindings for `lease-29669c6759d14d5b` (PR #795) and `lease-9878a1642666494d` (Issue #635) on the live control-plane database, restoring the `worktrees` post-restart dimension to `resolved`. #### Verification Proof - `pytest tests/test_issue_970_missing_worktree_reconcile.py`: 7/7 passed - `pytest tests/test_issue_781_edit_issue_tool.py tests/test_issue_662_post_restart_reconcile.py tests/test_lease_lifecycle.py tests/test_worktree_cleanup_audit.py tests/test_merged_cleanup_reconcile.py`: 143/143 passed - Post-restart audit: `DIM_WORKTREES` status is `resolved`.

sysadmin referenced this issue

2026-07-29 05:11:54 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972

sysadmin referenced this issue

2026-07-29 05:13:29 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972

sysadmin referenced this issue

2026-07-29 05:14:23 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972

sysadmin referenced this issue

2026-07-29 05:14:39 -05:00

fix(reconcile): safely resolve worktree bindings whose paths are missing (Closes #970) #972