feat: require validation failure history in review reports (Closes #396)
Reviewer runs that observe a validation failure and later rerun to green must report the failure and its investigation; a later pass never erases the earlier failure: - New reviewer_validation_failure_history.py verifier: reports must carry a Validation failure history section naming every observed failure with command, failing test or error, suspected cause, reproduced status, baseline master status, and PR-caused evidence; rerun-pass claims must explain what changed between runs, whether environmental state (such as stale /tmp files) was cleaned, and why the pass is trustworthy; an unexplained transient failure cannot be reported as a bare pass and must use the status 'passed after transient failure investigation' (exported as TRANSIENT_INVESTIGATION_STATUS). - Re-export from review_proofs and wire into build_final_report as a downgrade check. - Review-merge final-report schema gains the Validation failure history template section; contract test locks the markers and an export test locks the re-export. - 12 tests covering the issue's scenarios: failing-then-passing rerun, baseline-equivalent failure, /tmp state contamination, unexplained transient failure, and a report omitting an earlier failure. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
@@ -74,6 +74,14 @@ def test_review_merge_final_report_schema_exists():
|
||||
assert "Candidate head SHA:" in text
|
||||
assert "Terminal review mutation:" in text
|
||||
assert "Workspace mutations" in text
|
||||
assert "### Validation failure history (#396)" in text
|
||||
assert "passed after transient failure investigation" in text
|
||||
|
||||
|
||||
def test_validation_failure_history_verifier_exported():
|
||||
from review_proofs import assess_validation_failure_history_report
|
||||
|
||||
assert callable(assess_validation_failure_history_report)
|
||||
|
||||
|
||||
def test_reconcile_landed_workflow_contract():
|
||||
|
||||
Reference in New Issue
Block a user