feat: require validation failure history in review reports (Closes #396)
Reviewer runs that observe a validation failure and later rerun to green must report the failure and its investigation; a later pass never erases the earlier failure: - New reviewer_validation_failure_history.py verifier: reports must carry a Validation failure history section naming every observed failure with command, failing test or error, suspected cause, reproduced status, baseline master status, and PR-caused evidence; rerun-pass claims must explain what changed between runs, whether environmental state (such as stale /tmp files) was cleaned, and why the pass is trustworthy; an unexplained transient failure cannot be reported as a bare pass and must use the status 'passed after transient failure investigation' (exported as TRANSIENT_INVESTIGATION_STATUS). - Re-export from review_proofs and wire into build_final_report as a downgrade check. - Review-merge final-report schema gains the Validation failure history template section; contract test locks the markers and an export test locks the re-export. - 12 tests covering the issue's scenarios: failing-then-passing rerun, baseline-equivalent failure, /tmp state contamination, unexplained transient failure, and a report omitting an earlier failure. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
@@ -91,4 +91,27 @@ Verifier: `review_proofs.assess_queue_status_report()`.
|
||||
|
||||
Narrative final report and controller handoff must agree on eligibility class,
|
||||
candidate/reviewed head SHA, mutation state, worktree usage, review decision,
|
||||
terminal review mutation, merge result, and linked issue status.
|
||||
terminal review mutation, merge result, and linked issue status.
|
||||
|
||||
### Validation failure history (#396)
|
||||
|
||||
Every validation failure observed during the run must appear here, even if a
|
||||
later rerun passed. A later pass never erases the earlier failure.
|
||||
|
||||
For each observed failure report:
|
||||
|
||||
- Command:
|
||||
- Failing test or error:
|
||||
- Suspected cause:
|
||||
- Reproduced:
|
||||
- Baseline master:
|
||||
- PR-caused evidence:
|
||||
|
||||
If a later rerun passed, also report what changed between runs, whether any
|
||||
environmental state (for example stale `/tmp` files) was cleaned, and why the
|
||||
pass is trustworthy.
|
||||
|
||||
If the cause is unknown, the validation status must be
|
||||
`passed after transient failure investigation` — never a bare `passed`.
|
||||
|
||||
Verifier: `review_proofs.assess_validation_failure_history_report()`.
|
||||
Reference in New Issue
Block a user