Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases Wiki Activity

Use a 10-minute sliding TTL for reviewer and merger PR leases #747

New Issue
Closed
opened 2026-07-18 12:42:00 -05:00 by jcwalker3 · 1 comment
jcwalker3 commented 2026-07-18 12:42:00 -05:00
Owner
Copy Link
Copy Source

Problem

Reviewer and merger PR leases currently use the long fixed thresholds established by #407 AC6/AC7: heartbeat expected at least every 30 minutes, lease considered stale at 30 minutes, reclaimable at 60 minutes, manual cleanup at 120 minutes.

These durations are wrong in both directions for how sessions actually behave:

  1. Too long on abandonment. When a reviewer or merger session dies (daemon crash, transport flap, client restart), the orphaned lease keeps the PR blocked for up to an hour before any other session may reclaim it, and up to two hours before manual cleanup is sanctioned. #718 is the recorded failure mode: a merge deadlocked behind an expired-but-not-reclaimable reviewer lease with no sanctioned acquisition path.
  2. Too coarse on liveness. A 30-minute heartbeat interval means a lease can be held by a process that has been dead for 29 minutes and still look fresh. Liveness signal resolution is poor relative to the length of a real review.

The underlying design flaw is that the lease uses a long fixed expiry as a proxy for "the owner is probably still alive" rather than requiring the owner to continuously prove liveness.

Proposed change

Replace the fixed long expiry with a 10-minute sliding (heartbeat-renewed) TTL for reviewer and merger PR leases:

  • A newly acquired lease expires 10 minutes after acquisition.
  • Every successful heartbeat from the lease owner slides expires_at forward to now + 10 minutes. There is no fixed maximum lifetime — an actively heartbeating session holds the lease as long as it keeps working.
  • Once expires_at passes with no heartbeat, the lease is expired and immediately reclaimable by another session through the existing sanctioned acquire/reclaim path. No separate 60-minute reclaim tier and no 120-minute manual-cleanup tier.

Net effect: a live session is never evicted, and a dead session releases its hold in at most 10 minutes instead of 60–120.

Acceptance criteria

  • AC1 — Reviewer PR lease acquisition sets expires_at = acquired_at + 10 minutes.
  • AC2 — Merger PR lease acquisition sets expires_at = acquired_at + 10 minutes.
  • AC3 — A successful heartbeat by the lease owner slides expires_at to now + 10 minutes and is durably persisted. A heartbeat from a non-owner session does not slide the lease and fails closed.
  • AC4 — A heartbeat against an already-expired lease does not resurrect it; it fails closed with a distinguishable, non-internal_error blocker that names the sanctioned reacquire path.
  • AC5 — An expired lease is reclaimable by another session via the existing sanctioned path with no additional waiting tier. The 60-minute reclaim threshold and the 120-minute manual-cleanup threshold are removed for reviewer and merger PR leases.
  • AC6 — The TTL is defined as a single named constant per lease kind, not duplicated as literals across acquisition, heartbeat, expiry-assessment, and diagnostic call sites.
  • AC7 — Lease-inspection and diagnostic tools report the sliding TTL, the current expires_at, and seconds remaining, so an operator can distinguish "held and live" from "held and dying".
  • AC8 — Regression tests cover: fresh acquire TTL, heartbeat slide, non-owner heartbeat rejection, expiry-then-reclaim by a second session, and heartbeat-after-expiry fail-closed.
  • AC9 — Existing lease rows written under the old thresholds are handled without crashing: on next read they are evaluated against the new TTL rather than the old tiers.

Scope

In scope: reviewer PR leases and merger PR leases, and the lease-inspection/diagnostic surfaces that report their expiry.

Out of scope: issue-claim leases (#268), the review-decision-ledger TTL (#717), and branches/ worktree TTL (#401). Those keep their current durations and are addressed separately.

Relationship to prior issues

  • Supersedes the AC6/AC7 threshold values from #407 for reviewer PR leases, and extends the same policy to merger PR leases, which #407 did not cover.
  • #718 is the motivating failure mode: the deadlock it records is a direct consequence of a long non-reclaimable expiry window.
  • #601 made lease lifecycle first-class and exposed heartbeat freshness; this issue changes the durations that lifecycle enforces, not the lifecycle itself.
## Problem Reviewer and merger PR leases currently use the long fixed thresholds established by #407 AC6/AC7: heartbeat expected at least every 30 minutes, lease considered stale at 30 minutes, reclaimable at 60 minutes, manual cleanup at 120 minutes. These durations are wrong in both directions for how sessions actually behave: 1. **Too long on abandonment.** When a reviewer or merger session dies (daemon crash, transport flap, client restart), the orphaned lease keeps the PR blocked for up to an hour before any other session may reclaim it, and up to two hours before manual cleanup is sanctioned. #718 is the recorded failure mode: a merge deadlocked behind an expired-but-not-reclaimable reviewer lease with no sanctioned acquisition path. 2. **Too coarse on liveness.** A 30-minute heartbeat interval means a lease can be held by a process that has been dead for 29 minutes and still look fresh. Liveness signal resolution is poor relative to the length of a real review. The underlying design flaw is that the lease uses a long fixed expiry as a proxy for "the owner is probably still alive" rather than requiring the owner to continuously prove liveness. ## Proposed change Replace the fixed long expiry with a **10-minute sliding (heartbeat-renewed) TTL** for reviewer and merger PR leases: - A newly acquired lease expires 10 minutes after acquisition. - Every successful heartbeat from the lease owner slides `expires_at` forward to `now + 10 minutes`. There is no fixed maximum lifetime — an actively heartbeating session holds the lease as long as it keeps working. - Once `expires_at` passes with no heartbeat, the lease is expired and immediately reclaimable by another session through the existing sanctioned acquire/reclaim path. No separate 60-minute reclaim tier and no 120-minute manual-cleanup tier. Net effect: a live session is never evicted, and a dead session releases its hold in at most 10 minutes instead of 60–120. ## Acceptance criteria - **AC1** — Reviewer PR lease acquisition sets `expires_at = acquired_at + 10 minutes`. - **AC2** — Merger PR lease acquisition sets `expires_at = acquired_at + 10 minutes`. - **AC3** — A successful heartbeat by the lease owner slides `expires_at` to `now + 10 minutes` and is durably persisted. A heartbeat from a non-owner session does not slide the lease and fails closed. - **AC4** — A heartbeat against an already-expired lease does not resurrect it; it fails closed with a distinguishable, non-`internal_error` blocker that names the sanctioned reacquire path. - **AC5** — An expired lease is reclaimable by another session via the existing sanctioned path with no additional waiting tier. The 60-minute reclaim threshold and the 120-minute manual-cleanup threshold are removed for reviewer and merger PR leases. - **AC6** — The TTL is defined as a single named constant per lease kind, not duplicated as literals across acquisition, heartbeat, expiry-assessment, and diagnostic call sites. - **AC7** — Lease-inspection and diagnostic tools report the sliding TTL, the current `expires_at`, and seconds remaining, so an operator can distinguish "held and live" from "held and dying". - **AC8** — Regression tests cover: fresh acquire TTL, heartbeat slide, non-owner heartbeat rejection, expiry-then-reclaim by a second session, and heartbeat-after-expiry fail-closed. - **AC9** — Existing lease rows written under the old thresholds are handled without crashing: on next read they are evaluated against the new TTL rather than the old tiers. ## Scope In scope: reviewer PR leases and merger PR leases, and the lease-inspection/diagnostic surfaces that report their expiry. Out of scope: issue-claim leases (#268), the review-decision-ledger TTL (#717), and `branches/` worktree TTL (#401). Those keep their current durations and are addressed separately. ## Relationship to prior issues - Supersedes the AC6/AC7 threshold values from #407 for reviewer PR leases, and extends the same policy to merger PR leases, which #407 did not cover. - #718 is the motivating failure mode: the deadlock it records is a direct consequence of a long non-reclaimable expiry window. - #601 made lease lifecycle first-class and exposed heartbeat freshness; this issue changes the durations that lifecycle enforces, not the lifecycle itself.
jcwalker3 added the type:featurestatus:readyworkflow-hardeninganti-stomp labels 2026-07-18 12:42:00 -05:00
jcwalker3 added status:in-progress and removed status:ready labels 2026-07-18 12:43:16 -05:00
jcwalker3 commented 2026-07-18 12:43:16 -05:00
Author
Owner
Copy Link
Copy Source

Issue claim heartbeat

<!-- gitea-issue-claim-heartbeat:v1 --> **Issue claim heartbeat** - kind: claim - issue: #747 - branch: feat/issue-747-sliding-lease-ttl - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation
jcwalker3 added status:pr-open and removed status:in-progress labels 2026-07-18 12:57:11 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels anti-stomp status:pr-open type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#747