Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 80 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

Prevent TTL expiry from silently overwriting unresolved terminal review-decision ledgers #717

New Issue
Open
opened 2026-07-15 21:59:03 -05:00 by jcwalker3 · 1 comment
jcwalker3 commented 2026-07-15 21:59:03 -05:00
Owner
Copy Link
Copy Source

Problem

A confirmed restart path silently overwrote an unresolved prgs-reviewer terminal decision ledger after its generic state TTL expired.

The affected ledger recorded the valid approval and sanctioned merge sequence for PR #710:

The native merger completed PR #710 successfully but reported:

review_decision_lock_cleared_after_merge: false

The reviewer terminal ledger therefore still required post-merge reconciliation or archival.

After an MCP restart, reviewer initialization replaced the existing terminal ledger with an empty ledger without archive, recovery record, quarantine record, or cleanup audit.

Confirmed causal path

  1. The pre-restart reviewer lock contained:
    • ready_pr_number: 710
    • ready_action: approve
    • ready_expected_head_sha: 573e721...
    • terminal live_mutations
    • review ID 439
  2. PR #710 merged while the running MCP process still used the pre-#709 startup code.
  3. Cross-profile archive-and-clear did not complete.
  4. Approximately 7.85 hours elapsed after the ledger’s updated_at.
  5. _load_review_decision_lock called generic session-state loading.
  6. The decision-lock kind was not classified as recovery-critical.
  7. The generic four-hour TTL caused loading to return None.
  8. Because loading returned None, initialization never inspected the existing on-disk terminal envelope.
  9. assess_init_overwrite therefore did not see the terminal mutation.
  10. init_review_decision_lock atomically replaced the profile ledger with a new empty ledger.
  11. No archive, journal, backup, quarantine record, recovery record, or cleanup audit was created.

Current evidence

Lost local ledger

Before restart:

After restart:

  • New writer PID and token fingerprint
  • live_mutations: []
  • all ready_* fields null
  • final_review_decision_ready: false
  • cleanup tool reports “nothing to clear”
  • no cleanup audit ID

Durable Gitea evidence remains

  • PR #710 is merged.
  • Review 439 remains visible, APPROVED, and not dismissed.
  • Review head matches the merged source head.
  • Lease claim comment 11701 remains.
  • Merger adoption comment 11703 remains.
  • Merge commit is 1ab384adc9a083e41556ee04575f21f4b7a2f83a.
  • Issue #709 closed through valid linkage.

The Gitea audit remains intact, but the local terminal ledger and its consumption/archival proof are missing.

Security and integrity impact

This creates an audit-integrity and state-machine defect:

  • unresolved terminal mutations can disappear merely because time passes;
  • restart can convert an unresolved ledger into an apparently clean run;
  • initialization may permit a new terminal decision without proving disposition of the prior one;
  • cleanup reports “nothing to clear” even though required reconciliation never occurred;
  • the absence of live state can be mistaken for successful consumption;
  • AC2 overwrite protection is bypassed because it only examines state after TTL-filtered loading;
  • proceeding normally can conceal or normalize the audit loss.

A terminal review decision must never be treated like disposable cache state.

Required behavior

1. Recovery-critical classification

Decision ledgers containing any terminal mutation or ready decision must be recovery-critical.

They must not expire through a generic TTL while unresolved.

2. Raw-envelope inspection before initialization

Before creating or replacing a decision ledger:

  • inspect any existing on-disk envelope regardless of TTL;
  • validate its schema and profile ownership;
  • identify terminal mutations and ready state;
  • refuse empty initialization when unresolved state exists;
  • never rely only on TTL-filtered loading for overwrite safety.

3. Archive-before-clear

A terminal ledger may become empty only after:

  • sanctioned merge consumption;
  • sanctioned cleanup;
  • sanctioned quarantine;
  • or explicit operator recovery.

Before clearing:

  • write a durable archive/recovery record;
  • verify the archive;
  • record the reason and actor;
  • retain PR, review, action and head;
  • create a durable audit record;
  • only then replace or remove live state.

If archival fails, leave the original ledger unchanged.

4. Post-merge restart reconciliation

At startup, scan unresolved terminal ledgers.

When the associated PR is already merged:

  • verify the exact reviewed and merged head;
  • verify review validity;
  • verify merger adoption and merge commit;
  • archive the ledger as consumed;
  • create a reconciliation audit;
  • clear live state only after successful archival.

Do not silently initialize over it.

5. Expired non-terminal state

Expired ledgers with no terminal mutation may be reinitialized only after raw-envelope validation proves they contain no protected state.

The distinction must be based on content, not age alone.

6. Malformed or unreadable state

Malformed, unreadable, version-unknown or identity-mismatched decision state must fail closed.

It must be quarantined or escalated through a sanctioned operator path, not overwritten.

7. Correct cleanup semantics

Cleanup assessment must distinguish:

  • no ledger ever existed;
  • empty current ledger;
  • properly consumed and archived ledger;
  • expired unresolved ledger;
  • lost or overwritten ledger;
  • merged PR awaiting reconciliation.

is_moot, cleanup_allowed, and reasons must describe the actual state.

8. Incident-specific recovery

Provide a sanctioned recovery procedure for the PR #710 incident.

It must:

  • use surviving Gitea evidence;
  • record that the original local terminal ledger was lost;
  • preserve the known pre-restart metadata;
  • avoid pretending that a normal cleanup occurred;
  • create an irrecoverable_decision_provenance or equivalent record;
  • link PR #710, review 439, head, lease comments and merge commit;
  • create a durable audit comment;
  • explicitly authorize or deny subsequent reviewer-terminal creation.

Do not manually recreate the original ledger or fabricate missing fields.

9. Concurrency and atomicity

Initialization, recovery, archival and cleanup must be mutually exclusive per profile.

Concurrent startups must not:

  • both archive;
  • overwrite each other;
  • create two empty ledgers;
  • permit a new decision during recovery;
  • lose the original envelope.

10. Runtime-version transition

Support the case where:

  • a merge occurs under an older MCP startup head;
  • master advances with new recovery behavior;
  • the MCP process restarts into the newer version;
  • unresolved legacy state must be reconciled safely.

Acceptance criteria

  1. A TTL-expired ledger containing a terminal decision is never overwritten.
  2. A TTL-expired ready decision is never overwritten.
  3. Raw-envelope safety checks execute before empty initialization.
  4. Terminal decision state is recovery-critical regardless of age.
  5. An expired empty ledger may be safely reinitialized.
  6. Malformed expired state fails closed.
  7. Identity-mismatched expired state fails closed.
  8. Unknown-version state fails closed.
  9. Archive failure preserves the original live ledger.
  10. Successful merge reconciliation archives before clearing.
  11. Archive records retain profile, PR, review, action, head and timestamps.
  12. Post-merge restart produces a durable reconciliation audit.
  13. Cleanup reports correctly distinguish empty, consumed, lost and unresolved states.
  14. A new terminal decision is blocked while recovery is incomplete.
  15. Concurrent initialization cannot bypass recovery.
  16. Pre-#709 runtime merge followed by post-#709 restart is covered.
  17. PR #710’s surviving Gitea evidence can produce an explicit loss/recovery record.
  18. Incident recovery does not fabricate the missing original ledger.
  19. Manual state-file deletion or reconstruction remains forbidden.
  20. Full-suite regression coverage remains green.

Required tests

Include:

  • terminal ledger older than TTL;
  • ready decision older than TTL;
  • empty ledger older than TTL;
  • malformed expired envelope;
  • identity mismatch;
  • unknown schema version;
  • archive write failure;
  • interrupted archive;
  • concurrent startup;
  • concurrent cleanup and initialization;
  • merged PR awaiting restart reconciliation;
  • older-runtime merge followed by newer-runtime restart;
  • empty-ledger false-success detection;
  • PR #710 incident recovery from Gitea evidence;
  • audit readback;
  • new-decision blocking until recovery completes.

Scope boundaries

This issue must not:

  • invalidate the legitimate merge of PR #710;
  • dismiss review 439;
  • fabricate the lost terminal ledger;
  • manually modify session-state files;
  • weaken the one-terminal-decision rule;
  • treat Gitea history as a substitute for required local reconciliation;
  • modify PR #715;
  • modify eAgenda or PR #194.

Related work

  • Issue #332: terminal review-decision enforcement
  • Issue #709 / PR #710: post-merge decision reconciliation
  • Issue #714 / PR #715: session-context immutability remediation awaiting review
  • Issue #716: session-state placement and forensic recovery
  • PR #710, review 439, comments 11701 and 11703

Operational blocker

PR #715 review must not proceed under a claim that PR #710 cleanup succeeded. A controller must explicitly decide whether the surviving Gitea audit plus an incident record is sufficient to authorize a new review before this defect is implemented.

## Problem A confirmed restart path silently overwrote an unresolved `prgs-reviewer` terminal decision ledger after its generic state TTL expired. The affected ledger recorded the valid approval and sanctioned merge sequence for PR #710: - PR: #710 - Review: 439 - Approved head: `573e721437ee31f5689472534715e907de2e4085` - Terminal action: approve - Merge commit: `1ab384adc9a083e41556ee04575f21f4b7a2f83a` The native merger completed PR #710 successfully but reported: `review_decision_lock_cleared_after_merge: false` The reviewer terminal ledger therefore still required post-merge reconciliation or archival. After an MCP restart, reviewer initialization replaced the existing terminal ledger with an empty ledger without archive, recovery record, quarantine record, or cleanup audit. ## Confirmed causal path 1. The pre-restart reviewer lock contained: - `ready_pr_number: 710` - `ready_action: approve` - `ready_expected_head_sha: 573e721...` - terminal `live_mutations` - review ID 439 2. PR #710 merged while the running MCP process still used the pre-#709 startup code. 3. Cross-profile archive-and-clear did not complete. 4. Approximately 7.85 hours elapsed after the ledger’s `updated_at`. 5. `_load_review_decision_lock` called generic session-state loading. 6. The decision-lock kind was not classified as recovery-critical. 7. The generic four-hour TTL caused loading to return `None`. 8. Because loading returned `None`, initialization never inspected the existing on-disk terminal envelope. 9. `assess_init_overwrite` therefore did not see the terminal mutation. 10. `init_review_decision_lock` atomically replaced the profile ledger with a new empty ledger. 11. No archive, journal, backup, quarantine record, recovery record, or cleanup audit was created. ## Current evidence ### Lost local ledger Before restart: - Profile: prgs-reviewer - PR: #710 - Review: 439 - Head: `573e721437ee31f5689472534715e907de2e4085` - Terminal mutation: approve - Merge cleanup flag: false After restart: - New writer PID and token fingerprint - `live_mutations: []` - all `ready_*` fields null - `final_review_decision_ready: false` - cleanup tool reports “nothing to clear” - no cleanup audit ID ### Durable Gitea evidence remains - PR #710 is merged. - Review 439 remains visible, APPROVED, and not dismissed. - Review head matches the merged source head. - Lease claim comment 11701 remains. - Merger adoption comment 11703 remains. - Merge commit is `1ab384adc9a083e41556ee04575f21f4b7a2f83a`. - Issue #709 closed through valid linkage. The Gitea audit remains intact, but the local terminal ledger and its consumption/archival proof are missing. ## Security and integrity impact This creates an audit-integrity and state-machine defect: - unresolved terminal mutations can disappear merely because time passes; - restart can convert an unresolved ledger into an apparently clean run; - initialization may permit a new terminal decision without proving disposition of the prior one; - cleanup reports “nothing to clear” even though required reconciliation never occurred; - the absence of live state can be mistaken for successful consumption; - AC2 overwrite protection is bypassed because it only examines state after TTL-filtered loading; - proceeding normally can conceal or normalize the audit loss. A terminal review decision must never be treated like disposable cache state. ## Required behavior ### 1. Recovery-critical classification Decision ledgers containing any terminal mutation or ready decision must be recovery-critical. They must not expire through a generic TTL while unresolved. ### 2. Raw-envelope inspection before initialization Before creating or replacing a decision ledger: - inspect any existing on-disk envelope regardless of TTL; - validate its schema and profile ownership; - identify terminal mutations and ready state; - refuse empty initialization when unresolved state exists; - never rely only on TTL-filtered loading for overwrite safety. ### 3. Archive-before-clear A terminal ledger may become empty only after: - sanctioned merge consumption; - sanctioned cleanup; - sanctioned quarantine; - or explicit operator recovery. Before clearing: - write a durable archive/recovery record; - verify the archive; - record the reason and actor; - retain PR, review, action and head; - create a durable audit record; - only then replace or remove live state. If archival fails, leave the original ledger unchanged. ### 4. Post-merge restart reconciliation At startup, scan unresolved terminal ledgers. When the associated PR is already merged: - verify the exact reviewed and merged head; - verify review validity; - verify merger adoption and merge commit; - archive the ledger as consumed; - create a reconciliation audit; - clear live state only after successful archival. Do not silently initialize over it. ### 5. Expired non-terminal state Expired ledgers with no terminal mutation may be reinitialized only after raw-envelope validation proves they contain no protected state. The distinction must be based on content, not age alone. ### 6. Malformed or unreadable state Malformed, unreadable, version-unknown or identity-mismatched decision state must fail closed. It must be quarantined or escalated through a sanctioned operator path, not overwritten. ### 7. Correct cleanup semantics Cleanup assessment must distinguish: - no ledger ever existed; - empty current ledger; - properly consumed and archived ledger; - expired unresolved ledger; - lost or overwritten ledger; - merged PR awaiting reconciliation. `is_moot`, `cleanup_allowed`, and reasons must describe the actual state. ### 8. Incident-specific recovery Provide a sanctioned recovery procedure for the PR #710 incident. It must: - use surviving Gitea evidence; - record that the original local terminal ledger was lost; - preserve the known pre-restart metadata; - avoid pretending that a normal cleanup occurred; - create an `irrecoverable_decision_provenance` or equivalent record; - link PR #710, review 439, head, lease comments and merge commit; - create a durable audit comment; - explicitly authorize or deny subsequent reviewer-terminal creation. Do not manually recreate the original ledger or fabricate missing fields. ### 9. Concurrency and atomicity Initialization, recovery, archival and cleanup must be mutually exclusive per profile. Concurrent startups must not: - both archive; - overwrite each other; - create two empty ledgers; - permit a new decision during recovery; - lose the original envelope. ### 10. Runtime-version transition Support the case where: - a merge occurs under an older MCP startup head; - master advances with new recovery behavior; - the MCP process restarts into the newer version; - unresolved legacy state must be reconciled safely. ## Acceptance criteria 1. A TTL-expired ledger containing a terminal decision is never overwritten. 2. A TTL-expired ready decision is never overwritten. 3. Raw-envelope safety checks execute before empty initialization. 4. Terminal decision state is recovery-critical regardless of age. 5. An expired empty ledger may be safely reinitialized. 6. Malformed expired state fails closed. 7. Identity-mismatched expired state fails closed. 8. Unknown-version state fails closed. 9. Archive failure preserves the original live ledger. 10. Successful merge reconciliation archives before clearing. 11. Archive records retain profile, PR, review, action, head and timestamps. 12. Post-merge restart produces a durable reconciliation audit. 13. Cleanup reports correctly distinguish empty, consumed, lost and unresolved states. 14. A new terminal decision is blocked while recovery is incomplete. 15. Concurrent initialization cannot bypass recovery. 16. Pre-#709 runtime merge followed by post-#709 restart is covered. 17. PR #710’s surviving Gitea evidence can produce an explicit loss/recovery record. 18. Incident recovery does not fabricate the missing original ledger. 19. Manual state-file deletion or reconstruction remains forbidden. 20. Full-suite regression coverage remains green. ## Required tests Include: - terminal ledger older than TTL; - ready decision older than TTL; - empty ledger older than TTL; - malformed expired envelope; - identity mismatch; - unknown schema version; - archive write failure; - interrupted archive; - concurrent startup; - concurrent cleanup and initialization; - merged PR awaiting restart reconciliation; - older-runtime merge followed by newer-runtime restart; - empty-ledger false-success detection; - PR #710 incident recovery from Gitea evidence; - audit readback; - new-decision blocking until recovery completes. ## Scope boundaries This issue must not: - invalidate the legitimate merge of PR #710; - dismiss review 439; - fabricate the lost terminal ledger; - manually modify session-state files; - weaken the one-terminal-decision rule; - treat Gitea history as a substitute for required local reconciliation; - modify PR #715; - modify eAgenda or PR #194. ## Related work - Issue #332: terminal review-decision enforcement - Issue #709 / PR #710: post-merge decision reconciliation - Issue #714 / PR #715: session-context immutability remediation awaiting review - Issue #716: session-state placement and forensic recovery - PR #710, review 439, comments 11701 and 11703 ## Operational blocker PR #715 review must not proceed under a claim that PR #710 cleanup succeeded. A controller must explicitly decide whether the surviving Gitea audit plus an incident record is sufficient to authorize a new review before this defect is implemented.
sysadmin commented 2026-07-15 22:32:38 -05:00
Owner
Copy Link
Copy Source

Controller-Authorized Incident Exception

  • This is the one-time controller-authorized exception.
  • PR #710’s ledger loss remains unresolved.
  • Normal cleanup did not succeed.
  • Surviving Gitea audit evidence remains intact.
  • The exception applies only to PR #715 at dc899d23c85f879dfa5b2e03c376112f1ea8382a.
  • This run may produce at most one terminal verdict.
  • Issue #717 remains open.
## Controller-Authorized Incident Exception - This is the one-time controller-authorized exception. - PR #710’s ledger loss remains unresolved. - Normal cleanup did not succeed. - Surviving Gitea audit evidence remains intact. - The exception applies only to PR #715 at `dc899d23c85f879dfa5b2e03c376112f1ea8382a`. - This run may produce at most one terminal verdict. - Issue #717 remains open.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels gitea incident mcp recovery reliability safety security status:ready terminal-lock type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#717