fix(reconcile): retire workflow session rows whose owners are no longer live #969

New Issue

2026-07-29T04:07:35-05:00

jcwalker3 commented

2026-07-29 04:07:35 -05:00

Problem

Post-restart reconciliation can remain degraded because session rows continue to be classified as active after their recorded owner processes are no longer live.

During reconciliation following PR #968, 197 of 200 session rows reported dead owner PIDs. The affected records included historical author, reviewer, merger, controller, and Grok sessions. Fleet-wide active workflow leases were zero, but the sessions dimension remained unresolved in log_only mode.

A PID check alone must not be treated as sufficient proof because operating systems can reuse PIDs. Cleanup must consider the recorded process identity, start time, client/session provenance, and any current lease or ownership evidence.

Risk

Stale session rows can:

keep reconciliation permanently degraded;
create false ownership or concurrency signals;
obscure genuinely live sessions;
complicate safe restart and cleanup decisions;
cause future workflow mutations to fail closed unnecessarily.

Required behavior

Implement a safe, auditable lifecycle for session rows whose recorded owner is no longer valid.

The solution must:

distinguish live, disconnected, stale, and terminal session records;
validate more than PID existence, including protection against PID reuse;
preserve live client-managed sessions;
refuse cleanup when a live lease or trustworthy ownership signal remains;
terminalize or retire confirmed stale rows through a sanctioned reconciliation path;
retain sufficient audit history for diagnosis;
expose accurate live and stale counts;
be idempotent and safe across repeated reconciliation runs.

Acceptance criteria

A session whose owner PID is absent and whose other ownership evidence is stale is no longer counted as active.
PID reuse cannot make an unrelated process appear to own the historical session.
Live client-managed sessions are never retired by this cleanup.
A live workflow lease or verified session owner blocks retirement.
Repeated execution produces no duplicate terminalization or inconsistent state.
Reconciliation reports the sessions dimension resolved after all confirmed stale rows are handled.
Tests cover dead owners, PID reuse, live owners, live leases, mixed fleets, concurrent reconciliation, and repeated cleanup.
The cleanup action and reason are recorded in durable audit output.
Existing workflow identity, parity, and mutation-safety gates remain enforced.

Observed evidence

Observed during post-merge reconciliation for PR #968 on 2026-07-29:

session rows inspected: 200
rows with dead owner PIDs: 197
active workflow leases: 0
reconciliation mode: log_only
mutation hold: false

This evidence is the reproduction trigger, not authorization to delete rows manually.

## Problem Post-restart reconciliation can remain degraded because session rows continue to be classified as active after their recorded owner processes are no longer live. During reconciliation following PR #968, 197 of 200 session rows reported dead owner PIDs. The affected records included historical author, reviewer, merger, controller, and Grok sessions. Fleet-wide active workflow leases were zero, but the sessions dimension remained unresolved in `log_only` mode. A PID check alone must not be treated as sufficient proof because operating systems can reuse PIDs. Cleanup must consider the recorded process identity, start time, client/session provenance, and any current lease or ownership evidence. ## Risk Stale session rows can: - keep reconciliation permanently degraded; - create false ownership or concurrency signals; - obscure genuinely live sessions; - complicate safe restart and cleanup decisions; - cause future workflow mutations to fail closed unnecessarily. ## Required behavior Implement a safe, auditable lifecycle for session rows whose recorded owner is no longer valid. The solution must: - distinguish live, disconnected, stale, and terminal session records; - validate more than PID existence, including protection against PID reuse; - preserve live client-managed sessions; - refuse cleanup when a live lease or trustworthy ownership signal remains; - terminalize or retire confirmed stale rows through a sanctioned reconciliation path; - retain sufficient audit history for diagnosis; - expose accurate live and stale counts; - be idempotent and safe across repeated reconciliation runs. ## Acceptance criteria - A session whose owner PID is absent and whose other ownership evidence is stale is no longer counted as active. - PID reuse cannot make an unrelated process appear to own the historical session. - Live client-managed sessions are never retired by this cleanup. - A live workflow lease or verified session owner blocks retirement. - Repeated execution produces no duplicate terminalization or inconsistent state. - Reconciliation reports the sessions dimension resolved after all confirmed stale rows are handled. - Tests cover dead owners, PID reuse, live owners, live leases, mixed fleets, concurrent reconciliation, and repeated cleanup. - The cleanup action and reason are recorded in durable audit output. - Existing workflow identity, parity, and mutation-safety gates remain enforced. ## Observed evidence Observed during post-merge reconciliation for PR #968 on 2026-07-29: - session rows inspected: 200 - rows with dead owner PIDs: 197 - active workflow leases: 0 - reconciliation mode: `log_only` - mutation hold: false This evidence is the reproduction trigger, not authorization to delete rows manually.