feat: require validation failure history in review reports (Closes #396)

Reviewer runs that observe a validation failure and later rerun to green
must report the failure and its investigation; a later pass never erases
the earlier failure:

- New reviewer_validation_failure_history.py verifier: reports must carry
  a Validation failure history section naming every observed failure with
  command, failing test or error, suspected cause, reproduced status,
  baseline master status, and PR-caused evidence; rerun-pass claims must
  explain what changed between runs, whether environmental state (such as
  stale /tmp files) was cleaned, and why the pass is trustworthy; an
  unexplained transient failure cannot be reported as a bare pass and
  must use the status 'passed after transient failure investigation'
  (exported as TRANSIENT_INVESTIGATION_STATUS).
- Re-export from review_proofs and wire into build_final_report as a
  downgrade check.
- Review-merge final-report schema gains the Validation failure history
  template section; contract test locks the markers and an export test
  locks the re-export.
- 12 tests covering the issue's scenarios: failing-then-passing rerun,
  baseline-equivalent failure, /tmp state contamination, unexplained
  transient failure, and a report omitting an earlier failure.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
2026-07-07 11:26:40 -04:00
co-authored by Claude Opus 4.8
parent fcb9944378
commit f20e0fc338
5 changed files with 367 additions and 1 deletions
@@ -91,4 +91,27 @@ Verifier: `review_proofs.assess_queue_status_report()`.
Narrative final report and controller handoff must agree on eligibility class,
candidate/reviewed head SHA, mutation state, worktree usage, review decision,
terminal review mutation, merge result, and linked issue status.
terminal review mutation, merge result, and linked issue status.
### Validation failure history (#396)
Every validation failure observed during the run must appear here, even if a
later rerun passed. A later pass never erases the earlier failure.
For each observed failure report:
- Command:
- Failing test or error:
- Suspected cause:
- Reproduced:
- Baseline master:
- PR-caused evidence:
If a later rerun passed, also report what changed between runs, whether any
environmental state (for example stale `/tmp` files) was cleaned, and why the
pass is trustworthy.
If the cause is unknown, the validation status must be
`passed after transient failure investigation` — never a bare `passed`.
Verifier: `review_proofs.assess_validation_failure_history_report()`.