Unify task leases with sliding heartbeats and safe terminal dead-owner recovery #790

New Issue

2026-07-21T22:33:14-05:00

jcwalker3 commented

2026-07-21 22:33:14 -05:00

Problem

Gitea-Tools currently uses inconsistent ownership and lease lifecycles across task types.

Reviewer and merger PR leases use short-lived ownership with activity-based renewal. Author issue-work leases can receive a fixed four-hour lease that is not shortened by heartbeat loss, process death, issue closure, or completed PR integration.

This can unnecessarily block post-PR reconciliation for hours even when the protected artifacts are independently proven safe.

Observed incident

Issue #787 and PR #789 exposed the defect:

The author work lease was created at 2026-07-22T01:37:57Z.
It expires at 2026-07-22T05:37:57Z.
Effective TTL: four hours.
The current and previous owner PIDs were dead.
Issue #787 was closed.
PR #789 had completed integration into master.
The worktree was clean.
The relevant trees were identical.
The PR head was contained in master.
The branch-cleanup assessment returned safe_to_delete: true.
The reconciler still classified the worktree as active_issue_work with preserve: true.
Cleanup was blocked solely by the unexpired wall-clock lease.
Dead-session recovery appears capable of minting a new full-duration lease, restarting the wait.

This is safe fail-closed behavior, but it creates avoidable multi-hour stalls for terminal work.

Desired design

Use one shared lease lifecycle for every actively owned task:

Acquire a short initial lease.
Record a heartbeat while the owning session is active.
Extend expiry only when a valid owner heartbeat is received.
Stop renewal when the owner exits or loses ownership.
Mark ownership reclaimable after a configurable number of missed heartbeats.
Support a reconciler-only terminal reclaim path when terminality and artifact safety are proven.
Maintain complete ownership and cleanup audit records.

Task classes may have different configurable TTL and heartbeat intervals, but they must share the same acquire, renew, expire, reclaim, release, and audit semantics.

Required lease record

The shared record should include at least:

repository
task type and task identifier
role and profile
identity
session ID
process ID
branch
worktree
acquired timestamp
last heartbeat timestamp
expiry timestamp
lease generation or fencing token
lifecycle state
terminal/reclaim reason
audit provenance

Protection rule

Active work remains protected only while all applicable conditions hold:

the task is nonterminal;
the owner is valid;
the heartbeat is fresh;
the lease generation matches;
ownership has not been canonically released or superseded.

A stale heartbeat alone must not permit unsafe deletion. Reclaim must still use the existing cleanliness, ancestry, publication, worktree-binding, and lifecycle gates.

Terminal recovery

Add a sanctioned reconciler-only operation for terminal work.

It may reclaim an author issue-work lease before its original maximum expiry only when all required proofs succeed:

issue is closed or otherwise canonically terminal;
associated PR completed integration into the target branch;
expected PR head is contained in the current target branch;
owner process is dead or its heartbeat exceeded the allowed grace period;
no newer lease generation or active successor session exists;
worktree is clean;
no unpublished commits or changes exist;
branch contents are represented in the target branch;
no conflicting worktree or task ownership exists;
reconciliation capability and identity are valid.

The operation must fail closed when any proof is missing or ambiguous.

Dead-session recovery

Dead-session recovery must not silently restart a complete long-duration author lease.

It should either:

preserve the original expiry;
issue only a short recovery grace lease; or
create a new lease generation only after an explicit sanctioned ownership adoption.

The selected behavior must be documented and covered by tests.

Configuration

Centralize lease policy instead of duplicating hardcoded TTL values.

Configuration should define, per task class:

initial TTL
heartbeat interval
missed-heartbeat grace period
maximum continuous lease duration, if applicable
recovery grace duration
terminal reclaim eligibility

Existing active leases must have explicit compatibility behavior when configuration changes.

Concurrency safety

Use lease generations or fencing tokens so a delayed heartbeat from an older session cannot revive or overwrite ownership after another session has reclaimed the task.

Heartbeat, adoption, release, and reclaim operations must be atomic and idempotent.

Acceptance criteria

Author, reviewer, merger, and reconciler ownership use the shared heartbeat lifecycle.
Active author work remains protected during long-running sessions through heartbeat renewal.
A dead owner stops extending the lease.
Missing heartbeats make ownership reclaimable after the configured grace period.
Terminal author work can be reclaimed early only through the sanctioned reconciler path and only after all safety gates pass.
Dead-session recovery does not restart a full multi-hour wait without explicit adoption.
Late heartbeats from superseded sessions cannot restore ownership.
TTLs and heartbeat intervals have one documented configuration source.
Existing lease compatibility and migration behavior are documented.
Every acquire, heartbeat, renewal, adoption, expiration, reclaim, and release has auditable provenance.
Issue #787/PR #789 is represented as a regression test.
Existing author, reviewer, merger, reconciliation, anti-stomp, and branch-safety tests remain passing.

Required tests

Include tests for:

Active author session renews a short lease.
Long-running author work remains protected while heartbeats are fresh.
Dead owner stops renewal.
Missed-heartbeat grace expires and permits sanctioned reclamation.
Closed issue plus completed PR integration plus dead owner plus clean/contained artifacts permits terminal reclamation.
Dirty worktree blocks reclamation.
Unpublished commits block reclamation.
Nonterminal issue blocks terminal reclamation.
Live successor session blocks reclamation.
Stale generation heartbeat is rejected.
Duplicate heartbeat is idempotent.
Recovery does not mint an unintended four-hour hold.
Runtime/master parity failure blocks mutation.
Read-after-write verifies the resulting lease and audit state.
Existing lease records receive deterministic compatibility treatment.

Non-goals

Do not weaken worktree cleanliness or ancestry checks.
Do not permit arbitrary lock deletion.
Do not make PID liveness the sole ownership authority.
Do not allow manual lock-file edits as recovery.
Do not use one identical TTL for every task merely because the lifecycle is shared.
Do not delete Issue #787 artifacts as part of implementing this issue.

Relationship

Discovered during post-PR reconciliation for Issue #787 / PR #789.

## Problem Gitea-Tools currently uses inconsistent ownership and lease lifecycles across task types. Reviewer and merger PR leases use short-lived ownership with activity-based renewal. Author issue-work leases can receive a fixed four-hour lease that is not shortened by heartbeat loss, process death, issue closure, or completed PR integration. This can unnecessarily block post-PR reconciliation for hours even when the protected artifacts are independently proven safe. ## Observed incident Issue #787 and PR #789 exposed the defect: - The author work lease was created at `2026-07-22T01:37:57Z`. - It expires at `2026-07-22T05:37:57Z`. - Effective TTL: four hours. - The current and previous owner PIDs were dead. - Issue #787 was closed. - PR #789 had completed integration into master. - The worktree was clean. - The relevant trees were identical. - The PR head was contained in master. - The branch-cleanup assessment returned `safe_to_delete: true`. - The reconciler still classified the worktree as `active_issue_work` with `preserve: true`. - Cleanup was blocked solely by the unexpired wall-clock lease. - Dead-session recovery appears capable of minting a new full-duration lease, restarting the wait. This is safe fail-closed behavior, but it creates avoidable multi-hour stalls for terminal work. ## Desired design Use one shared lease lifecycle for every actively owned task: 1. Acquire a short initial lease. 2. Record a heartbeat while the owning session is active. 3. Extend expiry only when a valid owner heartbeat is received. 4. Stop renewal when the owner exits or loses ownership. 5. Mark ownership reclaimable after a configurable number of missed heartbeats. 6. Support a reconciler-only terminal reclaim path when terminality and artifact safety are proven. 7. Maintain complete ownership and cleanup audit records. Task classes may have different configurable TTL and heartbeat intervals, but they must share the same acquire, renew, expire, reclaim, release, and audit semantics. ## Required lease record The shared record should include at least: - repository - task type and task identifier - role and profile - identity - session ID - process ID - branch - worktree - acquired timestamp - last heartbeat timestamp - expiry timestamp - lease generation or fencing token - lifecycle state - terminal/reclaim reason - audit provenance ## Protection rule Active work remains protected only while all applicable conditions hold: - the task is nonterminal; - the owner is valid; - the heartbeat is fresh; - the lease generation matches; - ownership has not been canonically released or superseded. A stale heartbeat alone must not permit unsafe deletion. Reclaim must still use the existing cleanliness, ancestry, publication, worktree-binding, and lifecycle gates. ## Terminal recovery Add a sanctioned reconciler-only operation for terminal work. It may reclaim an author issue-work lease before its original maximum expiry only when all required proofs succeed: - issue is closed or otherwise canonically terminal; - associated PR completed integration into the target branch; - expected PR head is contained in the current target branch; - owner process is dead or its heartbeat exceeded the allowed grace period; - no newer lease generation or active successor session exists; - worktree is clean; - no unpublished commits or changes exist; - branch contents are represented in the target branch; - no conflicting worktree or task ownership exists; - reconciliation capability and identity are valid. The operation must fail closed when any proof is missing or ambiguous. ## Dead-session recovery Dead-session recovery must not silently restart a complete long-duration author lease. It should either: - preserve the original expiry; - issue only a short recovery grace lease; or - create a new lease generation only after an explicit sanctioned ownership adoption. The selected behavior must be documented and covered by tests. ## Configuration Centralize lease policy instead of duplicating hardcoded TTL values. Configuration should define, per task class: - initial TTL - heartbeat interval - missed-heartbeat grace period - maximum continuous lease duration, if applicable - recovery grace duration - terminal reclaim eligibility Existing active leases must have explicit compatibility behavior when configuration changes. ## Concurrency safety Use lease generations or fencing tokens so a delayed heartbeat from an older session cannot revive or overwrite ownership after another session has reclaimed the task. Heartbeat, adoption, release, and reclaim operations must be atomic and idempotent. ## Acceptance criteria - Author, reviewer, merger, and reconciler ownership use the shared heartbeat lifecycle. - Active author work remains protected during long-running sessions through heartbeat renewal. - A dead owner stops extending the lease. - Missing heartbeats make ownership reclaimable after the configured grace period. - Terminal author work can be reclaimed early only through the sanctioned reconciler path and only after all safety gates pass. - Dead-session recovery does not restart a full multi-hour wait without explicit adoption. - Late heartbeats from superseded sessions cannot restore ownership. - TTLs and heartbeat intervals have one documented configuration source. - Existing lease compatibility and migration behavior are documented. - Every acquire, heartbeat, renewal, adoption, expiration, reclaim, and release has auditable provenance. - Issue #787/PR #789 is represented as a regression test. - Existing author, reviewer, merger, reconciliation, anti-stomp, and branch-safety tests remain passing. ## Required tests Include tests for: 1. Active author session renews a short lease. 2. Long-running author work remains protected while heartbeats are fresh. 3. Dead owner stops renewal. 4. Missed-heartbeat grace expires and permits sanctioned reclamation. 5. Closed issue plus completed PR integration plus dead owner plus clean/contained artifacts permits terminal reclamation. 6. Dirty worktree blocks reclamation. 7. Unpublished commits block reclamation. 8. Nonterminal issue blocks terminal reclamation. 9. Live successor session blocks reclamation. 10. Stale generation heartbeat is rejected. 11. Duplicate heartbeat is idempotent. 12. Recovery does not mint an unintended four-hour hold. 13. Runtime/master parity failure blocks mutation. 14. Read-after-write verifies the resulting lease and audit state. 15. Existing lease records receive deterministic compatibility treatment. ## Non-goals - Do not weaken worktree cleanliness or ancestry checks. - Do not permit arbitrary lock deletion. - Do not make PID liveness the sole ownership authority. - Do not allow manual lock-file edits as recovery. - Do not use one identical TTL for every task merely because the lifecycle is shared. - Do not delete Issue #787 artifacts as part of implementing this issue. ## Relationship Discovered during post-PR reconciliation for Issue #787 / PR #789.

jcwalker3 added the bug type:guardrail workflow-hardening mcp-health labels 2026-07-21 22:33:15 -05:00

jcwalker3 added the status:ready label 2026-07-21 22:37:12 -05:00

jcwalker3 commented

2026-07-21 22:47:14 -05:00

Implementation sequencing with Issue #760

Issue #790 must not be implemented concurrently with Issue #760 without an explicit integration plan. Both may modify the same ownership and lease surfaces:

issue_lock_store.assess_same_issue_lease_conflict
- #760 changes gate ordering so an expired lease can be renewed by the exact existing owner.
- #790 broadens reclaim decisions from is_lease_expired(...) toward shared liveness and heartbeat semantics.
- Independent implementations could conflict both textually and semantically.
Issue-lock expiry and freshness behavior
- #760 retains absolute wall-clock expiry and adds exact-owner renewal.
- #790 introduces sliding heartbeat renewal, lease generations, fencing tokens, and unified cross-role lifecycle behavior.
- #790 may subsume or replace portions of the renewal path introduced by #760.

Before allocating implementation, choose one sequence:

Land #760 first, then rebase and design #790 against its renewal behavior; or
Incorporate #760’s acceptance criteria into #790 and close #760 as superseded after confirming complete coverage.

Do not allocate #760 and #790 concurrently to separate authors unless their code boundaries and final integration order are explicitly coordinated.

Canonical Issue State

STATE: awaiting-sequencing-decision
WHO_IS_NEXT: controller
NEXT_ACTION: Choose and record the #760/#790 implementation order before allocating either issue.
NEXT_PROMPT:

Decide whether to land #760 first and rebase #790 onto it, or incorporate #760's acceptance criteria into #790 and close #760 as superseded after confirming complete coverage. Record the decision on both issues before allocation.

WHY: Both issues modify overlapping issue-lock conflict assessment and expiry/freshness behavior. Concurrent implementation without an integration order risks textual and semantic conflicts.
BLOCKERS: Allocation is blocked only until the controller records the sequencing decision.
VALIDATION: Issues #760 and #790 are open; #790 has status:ready; overlapping implementation surfaces were verified; no implementation has been allocated.

## Implementation sequencing with Issue #760 Issue #790 must not be implemented concurrently with Issue #760 without an explicit integration plan. Both may modify the same ownership and lease surfaces: - `issue_lock_store.assess_same_issue_lease_conflict` - #760 changes gate ordering so an expired lease can be renewed by the exact existing owner. - #790 broadens reclaim decisions from `is_lease_expired(...)` toward shared liveness and heartbeat semantics. - Independent implementations could conflict both textually and semantically. - Issue-lock expiry and freshness behavior - #760 retains absolute wall-clock expiry and adds exact-owner renewal. - #790 introduces sliding heartbeat renewal, lease generations, fencing tokens, and unified cross-role lifecycle behavior. - #790 may subsume or replace portions of the renewal path introduced by #760. Before allocating implementation, choose one sequence: 1. Land #760 first, then rebase and design #790 against its renewal behavior; or 2. Incorporate #760’s acceptance criteria into #790 and close #760 as superseded after confirming complete coverage. Do not allocate #760 and #790 concurrently to separate authors unless their code boundaries and final integration order are explicitly coordinated. ## Canonical Issue State STATE: awaiting-sequencing-decision WHO_IS_NEXT: controller NEXT_ACTION: Choose and record the #760/#790 implementation order before allocating either issue. NEXT_PROMPT: ```text Decide whether to land #760 first and rebase #790 onto it, or incorporate #760's acceptance criteria into #790 and close #760 as superseded after confirming complete coverage. Record the decision on both issues before allocation. ``` WHY: Both issues modify overlapping issue-lock conflict assessment and expiry/freshness behavior. Concurrent implementation without an integration order risks textual and semantic conflicts. BLOCKERS: Allocation is blocked only until the controller records the sequencing decision. VALIDATION: Issues #760 and #790 are open; #790 has status:ready; overlapping implementation surfaces were verified; no implementation has been allocated.

jcwalker3 referenced this issue

2026-07-21 22:53:04 -05:00

Allow exact-owner renewal of expired author issue locks #760

jcwalker3 commented

2026-07-21 23:02:03 -05:00

Ledger correction — supersedes comment 13869

Corrects the VALIDATION field only. The sequencing analysis and every other field in comment 13869 stand unchanged; this comment is the authoritative ledger for Issue #790.

Prior wording asserted status:ready for #790 alone. Both issues carry it, verified live at the time of this comment.

Canonical Issue State

STATE: awaiting-sequencing-decision
WHO_IS_NEXT: controller
NEXT_ACTION: Choose and record the #760/#790 implementation order before allocating either issue.
NEXT_PROMPT:

Decide whether to land #760 first and rebase #790 onto it, or incorporate #760's acceptance criteria into #790 and close #760 as superseded after confirming complete coverage. Record the decision on both issues before allocation.

WHY: Both issues modify overlapping issue-lock conflict assessment and expiry/freshness behavior. Concurrent implementation without an integration order risks textual and semantic conflicts.
BLOCKERS: Allocation is blocked only until the controller records the sequencing decision.
VALIDATION: Issues #760 and #790 are open; both #760 and #790 have status:ready; overlapping implementation surfaces were verified; no implementation has been allocated.

## Ledger correction — supersedes comment 13869 Corrects the `VALIDATION` field only. The sequencing analysis and every other field in comment 13869 stand unchanged; this comment is the authoritative ledger for Issue #790. Prior wording asserted `status:ready` for #790 alone. Both issues carry it, verified live at the time of this comment. ## Canonical Issue State STATE: awaiting-sequencing-decision WHO_IS_NEXT: controller NEXT_ACTION: Choose and record the #760/#790 implementation order before allocating either issue. NEXT_PROMPT: ```text Decide whether to land #760 first and rebase #790 onto it, or incorporate #760's acceptance criteria into #790 and close #760 as superseded after confirming complete coverage. Record the decision on both issues before allocation. ``` WHY: Both issues modify overlapping issue-lock conflict assessment and expiry/freshness behavior. Concurrent implementation without an integration order risks textual and semantic conflicts. BLOCKERS: Allocation is blocked only until the controller records the sequencing decision. VALIDATION: Issues #760 and #790 are open; both #760 and #790 have status:ready; overlapping implementation surfaces were verified; no implementation has been allocated.

jcwalker3 referenced this issue

2026-07-21 23:02:16 -05:00

Allow exact-owner renewal of expired author issue locks #760

jcwalker3 referenced this issue

2026-07-21 23:05:12 -05:00

Allow exact-owner renewal of expired author issue locks #760

jcwalker3 commented

2026-07-21 23:05:36 -05:00

Sequencing decision: Issue #760 lands first

The controller has selected the implementation order:

Implement, review, and land Issue #760 first.
Do not implement Issue #790 concurrently.
After #760 lands, reassess #790 against the delivered renewal behavior.
Rebase or redesign #790 as necessary so its shared heartbeat lifecycle incorporates rather than duplicates or conflicts with #760.

Issue #790 remains open and status:ready, but its implementation is intentionally sequenced behind #760. This resolves the sequencing decision previously recorded in comments 13869 and 13873.

Canonical Issue State

STATE: waiting-on-issue-760
WHO_IS_NEXT: controller
NEXT_ACTION: Hold Issue #790 until Issue #760 lands, then reassess its design and allocation readiness.
NEXT_PROMPT:

After Issue #760 lands, re-read its implementation, tests, and final merged head. Reassess Issue #790 against that delivered behavior, identify which #790 requirements remain, update its implementation plan to avoid duplication or conflict, and only then allocate #790.

WHY: Issue #790 changes the same conflict-assessment and expiry/freshness surfaces as #760 and must build on the behavior that #760 lands first.
BLOCKERS: Issue #790 implementation is blocked until Issue #760 lands and the controller completes the post-merge reassessment.
VALIDATION: Issues #760 and #790 are open and both have status:ready; the controller selected #760 to land first; #790 is sequenced behind #760; no implementation was allocated by this comment.

## Sequencing decision: Issue #760 lands first The controller has selected the implementation order: 1. Implement, review, and land Issue #760 first. 2. Do not implement Issue #790 concurrently. 3. After #760 lands, reassess #790 against the delivered renewal behavior. 4. Rebase or redesign #790 as necessary so its shared heartbeat lifecycle incorporates rather than duplicates or conflicts with #760. Issue #790 remains open and status:ready, but its implementation is intentionally sequenced behind #760. This resolves the sequencing decision previously recorded in comments 13869 and 13873. ## Canonical Issue State STATE: waiting-on-issue-760 WHO_IS_NEXT: controller NEXT_ACTION: Hold Issue #790 until Issue #760 lands, then reassess its design and allocation readiness. NEXT_PROMPT: ```text After Issue #760 lands, re-read its implementation, tests, and final merged head. Reassess Issue #790 against that delivered behavior, identify which #790 requirements remain, update its implementation plan to avoid duplication or conflict, and only then allocate #790. ``` WHY: Issue #790 changes the same conflict-assessment and expiry/freshness surfaces as #760 and must build on the behavior that #760 lands first. BLOCKERS: Issue #790 implementation is blocked until Issue #760 lands and the controller completes the post-merge reassessment. VALIDATION: Issues #760 and #790 are open and both have status:ready; the controller selected #760 to land first; #790 is sequenced behind #760; no implementation was allocated by this comment.

jcwalker3 referenced this issue

2026-07-21 23:19:51 -05:00

Allow exact-owner renewal of expired author issue locks #760

jcwalker3 referenced this issue from a commit

2026-07-21 23:40:03 -05:00

feat(lock): allow exact-owner renewal of expired author issue locks (Closes #760)

jcwalker3 referenced this issue

2026-07-21 23:40:51 -05:00

fix(lock): allow exact-owner renewal of expired author issue locks (Closes #760) #791

jcwalker3 referenced this issue

2026-07-21 23:42:00 -05:00

Allow exact-owner renewal of expired author issue locks #760