Add precise validation-status vocabulary for raw PR-head failures resolved by merge simulation #406

New Issue

2026-07-07T10:11:07-05:00

jcwalker3 commented

2026-07-07 10:11:07 -05:00

Evidence

The PR #386 review report had raw PR-head validation fail with one test failure while the merge-simulation full suite against current master passed cleanly. The report labeled this baseline-equivalent failure accepted — but no baseline worktree comparison was performed in that run. The label claimed a proof path that was never executed.

Problem

Reviewer reports lack precise validation-status vocabulary for the common stale-branch case:

raw PR-head validation fails,
the failure is caused by a stale branch base or a fix that already landed on master,
merge simulation against current target passes cleanly.

Calling that baseline-equivalent failure accepted is misleading unless a true baseline worktree proved the same failure on the target branch. The two proof paths are different claims: baseline-equivalence proves the failure exists on the target; merge-simulation resolution proves the failure disappears once the branch absorbs the target. Reports must say which one actually happened.

Required behavior

Validation status must accurately describe the proof path used.

Acceptance criteria

Add explicit validation statuses:
- passed
- failed
- baseline-equivalent failure accepted
- raw-head failure resolved by merge simulation
- passed after transient failure investigation
baseline-equivalent failure accepted is only allowed when a baseline worktree/command proves the same failure signature on the target branch (per the #323/#325 baseline rules: baseline worktree path, target SHA, exact command, matching failure).
raw-head failure resolved by merge simulation is used when raw PR-head tests fail but merge simulation into current target passes cleanly; the merge simulation itself must be reported under worktree/index mutations per #317.
Final report must include:
- raw PR-head validation result,
- merge simulation result,
- whether a baseline worktree was used (path + SHA, or none),
- exact reason the PR is safe.
Final-report validator rejects misleading status labels — specifically baseline-equivalent failure accepted without baseline worktree proof, and bare passed when any earlier validation in the session failed.
Tests cover:
- raw PR-head pass,
- raw PR-head failure + baseline match,
- raw PR-head failure + merge simulation pass,
- raw PR-head failure + merge simulation failure (must stay failed),
- transient failure then pass,
- wrong status label.
Documentation updates review-merge reporting rules with the status taxonomy.

Scope relationships

#396 — validation failure history and the passed after transient failure investigation status; this issue adds the status taxonomy and proof-path binding, #396 owns per-failure history reporting. Implement compatibly, not duplicatively.
#323 / #325 (closed) — baseline-proof and branches-only baseline rules that criterion 2 reuses.
#316 / #317 (closed) — PR-head vs diagnostic separation and merge-simulation mutation classification that criterion 3 builds on.
#391 / #395 — final-report schema and proof-backed claims validators that should consume these statuses.

Non-goals

Do not allow failing merge-simulation validation to pass.
Do not let a later passing command erase earlier failure history (that history rule stays with #396).
Do not call validation simply passed when any earlier validation failed.

## Evidence The PR #386 review report had raw PR-head validation fail with one test failure while the merge-simulation full suite against current master passed cleanly. The report labeled this `baseline-equivalent failure accepted` — but no baseline worktree comparison was performed in that run. The label claimed a proof path that was never executed. ## Problem Reviewer reports lack precise validation-status vocabulary for the common stale-branch case: * raw PR-head validation fails, * the failure is caused by a stale branch base or a fix that already landed on master, * merge simulation against current target passes cleanly. Calling that `baseline-equivalent failure accepted` is misleading unless a true baseline worktree proved the same failure on the target branch. The two proof paths are different claims: baseline-equivalence proves the failure exists on the target; merge-simulation resolution proves the failure disappears once the branch absorbs the target. Reports must say which one actually happened. ## Required behavior Validation status must accurately describe the proof path used. ## Acceptance criteria 1. Add explicit validation statuses: * `passed` * `failed` * `baseline-equivalent failure accepted` * `raw-head failure resolved by merge simulation` * `passed after transient failure investigation` 2. `baseline-equivalent failure accepted` is only allowed when a baseline worktree/command proves the same failure signature on the target branch (per the #323/#325 baseline rules: baseline worktree path, target SHA, exact command, matching failure). 3. `raw-head failure resolved by merge simulation` is used when raw PR-head tests fail but merge simulation into current target passes cleanly; the merge simulation itself must be reported under worktree/index mutations per #317. 4. Final report must include: * raw PR-head validation result, * merge simulation result, * whether a baseline worktree was used (path + SHA, or `none`), * exact reason the PR is safe. 5. Final-report validator rejects misleading status labels — specifically `baseline-equivalent failure accepted` without baseline worktree proof, and bare `passed` when any earlier validation in the session failed. 6. Tests cover: * raw PR-head pass, * raw PR-head failure + baseline match, * raw PR-head failure + merge simulation pass, * raw PR-head failure + merge simulation failure (must stay `failed`), * transient failure then pass, * wrong status label. 7. Documentation updates review-merge reporting rules with the status taxonomy. ## Scope relationships * **#396** — validation failure history and the `passed after transient failure investigation` status; this issue adds the status taxonomy and proof-path binding, #396 owns per-failure history reporting. Implement compatibly, not duplicatively. * **#323 / #325** (closed) — baseline-proof and branches-only baseline rules that criterion 2 reuses. * **#316 / #317** (closed) — PR-head vs diagnostic separation and merge-simulation mutation classification that criterion 3 builds on. * **#391 / #395** — final-report schema and proof-backed claims validators that should consume these statuses. ## Non-goals * Do not allow failing merge-simulation validation to pass. * Do not let a later passing command erase earlier failure history (that history rule stays with #396). * Do not call validation simply `passed` when any earlier validation failed.

jcwalker3 commented

2026-07-07 10:14:22 -05:00

Second evidence case — PR #387

Operator-reported recurrence of the exact defect this issue tracks: the PR #387 review report labeled a raw PR-head targeted-test failure as baseline-equivalent failure accepted even though no baseline worktree was used in that run; the merge-simulation full suite against current master passed cleanly. Under this issue's taxonomy the correct status is raw-head failure resolved by merge simulation.

Two independent occurrences in one day (PR #386, PR #387) — the mislabel is systematic, not a one-off. A duplicate issue for the #387 case was requested and stopped at the duplicate gate; this comment is the durable capture of that evidence.

## Second evidence case — PR #387 Operator-reported recurrence of the exact defect this issue tracks: the PR #387 review report labeled a raw PR-head targeted-test failure as `baseline-equivalent failure accepted` even though no baseline worktree was used in that run; the merge-simulation full suite against current master passed cleanly. Under this issue's taxonomy the correct status is `raw-head failure resolved by merge simulation`. Two independent occurrences in one day (PR #386, PR #387) — the mislabel is systematic, not a one-off. A duplicate issue for the #387 case was requested and stopped at the duplicate gate; this comment is the durable capture of that evidence.

jcwalker3 commented

2026-07-07 10:30:36 -05:00

Supporting evidence — PR #394

PR #394 is another instance of the validation-status label defect tracked by this issue. The review report labeled the status baseline-equivalent failure accepted, but no baseline worktree was used. The actual proof path was raw PR-head failure plus clean merge-simulation pass.

Evidence:

PR: feat: add reviewer final-report schema MCP validator (Closes #391) (#394)
Reviewed head: a164700ab9e4bd27c3eecb945b0fb1b90ea43e89
Incorrect label: baseline-equivalent failure accepted
Correct label: raw-head failure resolved by merge simulation
Baseline worktree used: false
Merge simulation: passed cleanly

This repeats the same pattern already observed for PRs #386 and #387.

## Supporting evidence — PR #394 PR #394 is another instance of the validation-status label defect tracked by this issue. The review report labeled the status `baseline-equivalent failure accepted`, but no baseline worktree was used. The actual proof path was raw PR-head failure plus clean merge-simulation pass. Evidence: * PR: #394 * Reviewed head: `a164700ab9e4bd27c3eecb945b0fb1b90ea43e89` * Incorrect label: `baseline-equivalent failure accepted` * Correct label: `raw-head failure resolved by merge simulation` * Baseline worktree used: false * Merge simulation: passed cleanly This repeats the same pattern already observed for PRs #386 and #387.

jcwalker3 referenced a pull request that will close this issue

2026-07-07 10:32:35 -05:00

feat: add precise validation status vocabulary for reviewer reports (Closes #406) #412

jcwalker3 referenced this issue from a commit

2026-07-07 10:32:38 -05:00

feat: add precise validation status vocabulary for reviewer reports (Closes #406)

jcwalker3 added the status:in-progress label 2026-07-07 11:53:31 -05:00

jcwalker3 commented

2026-07-07 11:53:31 -05:00

Issue claim heartbeat

kind: claim
issue: Add precise validation-status vocabulary for raw PR-head failures resolved by merge simulation (#406)
branch: feat/issue-406-validation-status-vocabulary
phase: claimed
profile: prgs-author
pr: none
blocker: none
next_action: create worktree and begin implementation

**Issue claim heartbeat** - kind: claim - issue: #406 - branch: feat/issue-406-validation-status-vocabulary - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation

jcwalker3 referenced this issue from a commit

2026-07-07 11:54:25 -05:00

feat: add precise validation status vocabulary for reviewer reports (Closes #406)

jcwalker3 referenced this issue

2026-07-07 12:21:01 -05:00

Web UI: Report audit paste tool #431

sysadmin referenced this issue

2026-07-08 00:47:21 -05:00

feat: add precise validation status vocabulary for reviewer reports (Closes #406) #412

sysadmin closed this issue

2026-07-08 00:48:04 -05:00

sysadmin referenced this issue from a commit