Require PR review reports to account for transient validation failures #396

New Issue

2026-07-07T09:25:09-05:00

jcwalker3 commented

2026-07-07 09:25:09 -05:00

Problem:
Reviewer runs can observe a validation failure, investigate or rerun tests, and later report only the final passing result. This hides important context from the controller and makes it hard to distinguish a real PR regression from an environmental/transient failure.

Observed example:
During PR #381 review, an initial full-suite run exposed a failure around tests/test_worktrees.py::TestWorktreeStart::test_rejects_untraceable_branches. The reviewer investigated /tmp/gitea_issue_lock.json and ran targeted combinations. Later the final report stated 1497 passed, 6 skipped, but did not clearly explain the initial failure, its cause, or why it was safe to proceed.

Required behavior:
Every validation failure observed during a review run must be included in the final report, even if later reruns pass.

Acceptance criteria:

Final report lists every failed validation command observed during the session.
For each failure, report:
- command,
- failing test or error,
- suspected cause,
- whether it reproduced,
- whether it exists on baseline master,
- evidence for why it is or is not PR-caused.
If a later rerun passes, the report must say:
- what changed between runs,
- whether any environmental state was cleaned,
- why the pass is trustworthy.
If the cause is unknown, reviewer must not simply call validation passed; it must use a status such as passed after transient failure investigation.
Tests cover:
- failing validation followed by passing rerun,
- baseline-equivalent failure,
- environmental /tmp state contamination,
- unexplained transient failure,
- final report omitting an earlier failure.
Review handoff template gets a Validation failure history section.
gitea_dry_run_pr_review or final-report validator rejects reports that omit known earlier validation failures.

Non-goals:

Do not require blocking a PR for every transient failure if it is proven non-PR-caused.
Do not weaken exact review/merge capability checks.
Do not allow a later pass to erase the need to report an earlier failure.

Problem: Reviewer runs can observe a validation failure, investigate or rerun tests, and later report only the final passing result. This hides important context from the controller and makes it hard to distinguish a real PR regression from an environmental/transient failure. Observed example: During PR #381 review, an initial full-suite run exposed a failure around `tests/test_worktrees.py::TestWorktreeStart::test_rejects_untraceable_branches`. The reviewer investigated `/tmp/gitea_issue_lock.json` and ran targeted combinations. Later the final report stated `1497 passed, 6 skipped`, but did not clearly explain the initial failure, its cause, or why it was safe to proceed. Required behavior: Every validation failure observed during a review run must be included in the final report, even if later reruns pass. Acceptance criteria: 1. Final report lists every failed validation command observed during the session. 2. For each failure, report: * command, * failing test or error, * suspected cause, * whether it reproduced, * whether it exists on baseline master, * evidence for why it is or is not PR-caused. 3. If a later rerun passes, the report must say: * what changed between runs, * whether any environmental state was cleaned, * why the pass is trustworthy. 4. If the cause is unknown, reviewer must not simply call validation `passed`; it must use a status such as `passed after transient failure investigation`. 5. Tests cover: * failing validation followed by passing rerun, * baseline-equivalent failure, * environmental `/tmp` state contamination, * unexplained transient failure, * final report omitting an earlier failure. 6. Review handoff template gets a `Validation failure history` section. 7. `gitea_dry_run_pr_review` or final-report validator rejects reports that omit known earlier validation failures. Non-goals: * Do not require blocking a PR for every transient failure if it is proven non-PR-caused. * Do not weaken exact review/merge capability checks. * Do not allow a later pass to erase the need to report an earlier failure.

jcwalker3 referenced this issue

2026-07-07 10:11:07 -05:00

Add precise validation-status vocabulary for raw PR-head failures resolved by merge simulation #406

jcwalker3 added the status:in-progress label 2026-07-07 10:19:50 -05:00

jcwalker3 commented

2026-07-07 10:20:30 -05:00

Issue claim heartbeat

kind: claim
issue: Require PR review reports to account for transient validation failures (#396)
branch: feat/issue-396-transient-validation-failure-history
phase: claimed
profile: prgs-author
pr: none
blocker: none
next_action: create worktree and begin implementation

**Issue claim heartbeat** - kind: claim - issue: #396 - branch: feat/issue-396-transient-validation-failure-history - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation

jcwalker3 referenced this issue from a commit

2026-07-07 10:21:33 -05:00

feat: require transient validation failure history in reviewer reports (Closes #396)

jcwalker3 referenced this issue from a commit

2026-07-07 10:22:02 -05:00

feat: require transient validation failure history in reviewer reports (Closes #396)

jcwalker3 referenced a pull request that will close this issue

2026-07-07 10:23:26 -05:00

feat: require transient validation failure history in reviewer reports (Closes #396) #409

jcwalker3 referenced this issue from a commit

2026-07-07 10:26:45 -05:00

feat: require validation failure history in review reports (Closes #396)