Require true pre-merge baseline proof for non-zero validation exits #533

New Issue

2026-07-08T04:10:10-05:00

jcwalker3 commented

2026-07-08 04:10:10 -05:00

Problem:
A controller audit for PR #527 reproduced a failing test on master after the PR had already merged, then described it as proof of a known baseline failure. That proves the failure exists on current master, but it does not prove the failure pre-existed the PR under review.

Risk:
A PR can introduce or preserve a regression, then once merged, the failure appears on master and may be incorrectly labeled “baseline.” This weakens validation gates and controller closure confidence.

Required behavior:

If a validation command exits non-zero, reports must not call it a clean pass.
If the failure is claimed to be pre-existing, the proof must come from:
- the PR’s pre-merge base commit, or
- a documented known-failure record that predates the PR.
Running the same failure on post-merge master is only “current-master reproduction,” not “pre-existing baseline proof.”
Canonical comments must use distinct labels:
- clean pass
- current-master failure reproduced
- pre-merge baseline-proven failure
- unresolved regression risk
Controller closure should be blocked when “baseline failure” is claimed without valid pre-merge proof.

Acceptance criteria:

Add tests rejecting post-merge master-only proof as sufficient baseline proof.
Add tests accepting pre-merge base proof.
Add tests accepting documented known-failure references that predate the PR.
Update reviewer/controller templates to require base commit, tested commit, command, exit status, and failure signature.

Problem: A controller audit for PR #527 reproduced a failing test on `master` after the PR had already merged, then described it as proof of a known baseline failure. That proves the failure exists on current master, but it does not prove the failure pre-existed the PR under review. Risk: A PR can introduce or preserve a regression, then once merged, the failure appears on master and may be incorrectly labeled “baseline.” This weakens validation gates and controller closure confidence. Required behavior: 1. If a validation command exits non-zero, reports must not call it a clean pass. 2. If the failure is claimed to be pre-existing, the proof must come from: - the PR’s pre-merge base commit, or - a documented known-failure record that predates the PR. 3. Running the same failure on post-merge master is only “current-master reproduction,” not “pre-existing baseline proof.” 4. Canonical comments must use distinct labels: - clean pass - current-master failure reproduced - pre-merge baseline-proven failure - unresolved regression risk 5. Controller closure should be blocked when “baseline failure” is claimed without valid pre-merge proof. Acceptance criteria: - Add tests rejecting post-merge master-only proof as sufficient baseline proof. - Add tests accepting pre-merge base proof. - Add tests accepting documented known-failure references that predate the PR. - Update reviewer/controller templates to require base commit, tested commit, command, exit status, and failure signature.

jcwalker3 referenced this issue from a commit

2026-07-08 22:04:46 -05:00

feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533)

jcwalker3 referenced a pull request that will close this issue

2026-07-08 22:10:17 -05:00

feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533) #574

sysadmin referenced this issue

2026-07-08 22:14:57 -05:00

feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533) #574

sysadmin referenced this issue

2026-07-08 22:20:30 -05:00

feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533) #574