feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533) #574

2026-07-08T22:10:17-05:00

jcwalker3 commented

2026-07-08 22:10:17 -05:00

Summary

Closes #533.

A non-zero validation exit reproduced on current (post-merge) master proves the failure exists on master now — it does NOT prove the failure pre-existed the PR under review. Reports could mislabel such a failure "baseline" and weaken validation gates.

Changes

premerge_baseline_proof.py — assess_premerge_baseline_proof classifies four outcomes: clean pass, current-master failure reproduced, pre-merge baseline-proven failure, unresolved regression risk. A failure may be called baseline/pre-existing only with pre-merge proof: reproduced on the PR pre-merge base commit, or a documented known-failure record predating the PR. Current-master reproduction alone is rejected.
final_report_validator.py — reviewer.premerge_baseline_proof rule on review_pr and reconcile_already_landed tasks.
skills/llm-project-workflow/templates/review-pr.md — baseline-proof section requiring base commit, tested commit, command, exit status, failure signature.
tests/test_premerge_baseline_proof.py — rejects current-master-only proof; accepts pre-merge base proof and documented known-failure records; validator integration.

Validation

Worktree only: branches/issue-533-premerge-baseline-proof @ 7fb8134.

venv/bin/python -m pytest tests/test_premerge_baseline_proof.py tests/test_final_report_validator.py tests/test_llm_workflow_split.py -q → 88 passed
git diff --check → clean
Non-webui suite → 2011 passed. Five failures baseline-proven pre-existing (dogfooding this rule): re-run on a clean base worktree at prgs/master without this diff reproduced all five identically. They are the #475 root-checkout-guard tests (gitea_mcp_server.py RuntimeError) tripping on a dirty shared root checkout — environmental, and this diff touches none of those files.
webui test collection errors also pre-existing on base (webui/lease_loader import of ISSUE_LOCK_FILE), since fixed upstream by #570; not caused by this diff.

Self-review

Not allowed (author = jcwalker3).

## Summary Closes #533. A non-zero validation exit reproduced on current (post-merge) master proves the failure exists on master now — it does NOT prove the failure pre-existed the PR under review. Reports could mislabel such a failure "baseline" and weaken validation gates. ## Changes - `premerge_baseline_proof.py` — `assess_premerge_baseline_proof` classifies four outcomes: `clean pass`, `current-master failure reproduced`, `pre-merge baseline-proven failure`, `unresolved regression risk`. A failure may be called baseline/pre-existing only with pre-merge proof: reproduced on the PR pre-merge base commit, or a documented known-failure record predating the PR. Current-master reproduction alone is rejected. - `final_report_validator.py` — `reviewer.premerge_baseline_proof` rule on `review_pr` and `reconcile_already_landed` tasks. - `skills/llm-project-workflow/templates/review-pr.md` — baseline-proof section requiring base commit, tested commit, command, exit status, failure signature. - `tests/test_premerge_baseline_proof.py` — rejects current-master-only proof; accepts pre-merge base proof and documented known-failure records; validator integration. ## Validation Worktree only: `branches/issue-533-premerge-baseline-proof` @ `7fb8134`. - `venv/bin/python -m pytest tests/test_premerge_baseline_proof.py tests/test_final_report_validator.py tests/test_llm_workflow_split.py -q` → 88 passed - `git diff --check` → clean - Non-webui suite → 2011 passed. Five failures baseline-proven pre-existing (dogfooding this rule): re-run on a clean base worktree at `prgs/master` without this diff reproduced all five identically. They are the #475 root-checkout-guard tests (`gitea_mcp_server.py` RuntimeError) tripping on a dirty shared root checkout — environmental, and this diff touches none of those files. - webui test collection errors also pre-existing on base (`webui/lease_loader` import of `ISSUE_LOCK_FILE`), since fixed upstream by #570; not caused by this diff. ## Self-review Not allowed (author = jcwalker3).

jcwalker3 added 1 commit 2026-07-08 22:10:17 -05:00

feat: require true pre-merge baseline proof for non-zero validation exits (Closes #533 ) 7fb813446f

Add premerge_baseline_proof.py distinguishing four validation outcomes
(clean pass / current-master failure reproduced / pre-merge baseline-proven
failure / unresolved regression risk). A non-zero exit may only be labeled
"baseline"/"pre-existing" with pre-merge proof: the failure reproduced on the
PR's pre-merge base commit, or a documented known-failure record predating the
PR. Current-master (post-merge) reproduction is explicitly rejected as
sufficient baseline proof.

- premerge_baseline_proof.py: assess_premerge_baseline_proof + labels
- final_report_validator.py: reviewer.premerge_baseline_proof rule on
  review_pr and reconcile_already_landed tasks
- skills/llm-project-workflow/templates/review-pr.md: baseline proof section
  requiring base commit, tested commit, command, exit status, failure signature
- tests/test_premerge_baseline_proof.py: reject current-master-only proof,
  accept pre-merge base proof and documented known-failure records

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

sysadmin commented

2026-07-08 22:14:57 -05:00

repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #574
issue: #533
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 38902-27b717ce5a6f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof
phase: claimed
candidate_head: 7fb813446f
target_branch: master
target_branch_sha: none
last_activity: 2026-07-09T03:14:57Z
expires_at: 2026-07-09T05:14:57Z
blocker: none

repo: Scaled-Tech-Consulting/Gitea-Tools pr: #574 issue: #533 reviewer_identity: sysadmin profile: prgs-reviewer session_id: 38902-27b717ce5a6f worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof phase: claimed candidate_head: 7fb813446fe0389d73ce306199887002d455c1a4 target_branch: master target_branch_sha: none last_activity: 2026-07-09T03:14:57Z expires_at: 2026-07-09T05:14:57Z blocker: none

sysadmin commented

2026-07-08 22:20:30 -05:00

repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #574
issue: #533
reviewer_identity: sysadmin
profile: prgs-reviewer
session_id: 38902-27b717ce5a6f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof
phase: claimed
candidate_head: 7fb813446f
target_branch: master
target_branch_sha: none
last_activity: 2026-07-09T03:20:29Z
expires_at: 2026-07-09T05:20:29Z
blocker: none

repo: Scaled-Tech-Consulting/Gitea-Tools pr: #574 issue: #533 reviewer_identity: sysadmin profile: prgs-reviewer session_id: 38902-27b717ce5a6f worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof phase: claimed candidate_head: 7fb813446fe0389d73ce306199887002d455c1a4 target_branch: master target_branch_sha: none last_activity: 2026-07-09T03:20:29Z expires_at: 2026-07-09T05:20:29Z blocker: none

sysadmin approved these changes 2026-07-08 22:20:38 -05:00

sysadmin left a comment

The review of PR #574 (feat: require true pre-merge baseline proof for non-zero validation exits) has completed successfully.

Verified that premerge_baseline_proof.py implements the four pre-merge/reproduction classifications correctly.
Verified that final_report_validator.py integrates the reviewer.premerge_baseline_proof rule as expected.
Verified that skills/llm-project-workflow/templates/review-pr.md documents the baseline proof fields and labels.
Confirmed that the integration unit tests in tests/test_premerge_baseline_proof.py cover all outcomes and pass cleanly.

All acceptance criteria are fully met. Approved!

The review of PR #574 (feat: require true pre-merge baseline proof for non-zero validation exits) has completed successfully. - Verified that `premerge_baseline_proof.py` implements the four pre-merge/reproduction classifications correctly. - Verified that `final_report_validator.py` integrates the `reviewer.premerge_baseline_proof` rule as expected. - Verified that `skills/llm-project-workflow/templates/review-pr.md` documents the baseline proof fields and labels. - Confirmed that the integration unit tests in `tests/test_premerge_baseline_proof.py` cover all outcomes and pass cleanly. All acceptance criteria are fully met. Approved!

sysadmin commented

2026-07-08 22:21:20 -05:00

adopted_at: 2026-07-09T03:21:20Z
adopted_by_identity: sysadmin
adopted_by_profile: prgs-merger
adopted_from_session_id: 38902-27b717ce5a6f
adopted_from_profile: prgs-reviewer
adopted_from_reviewer_identity: sysadmin
adopted_from_comment_id: 7696
adoption_reason: merger-handoff-approved-head

repo: Scaled-Tech-Consulting/Gitea-Tools
pr: #574
issue: #533
reviewer_identity: sysadmin
profile: prgs-merger
session_id: 38902-27b717ce5a6f
worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof
phase: adopted
candidate_head: 7fb813446f
target_branch: master
target_branch_sha: none
last_activity: 2026-07-09T03:21:20Z
expires_at: 2026-07-09T05:21:20Z
blocker: none

adopted_at: 2026-07-09T03:21:20Z adopted_by_identity: sysadmin adopted_by_profile: prgs-merger adopted_from_session_id: 38902-27b717ce5a6f adopted_from_profile: prgs-reviewer adopted_from_reviewer_identity: sysadmin adopted_from_comment_id: 7696 adoption_reason: merger-handoff-approved-head  repo: Scaled-Tech-Consulting/Gitea-Tools pr: #574 issue: #533 reviewer_identity: sysadmin profile: prgs-merger session_id: 38902-27b717ce5a6f worktree: /Users/jasonwalker/Development/Gitea-Tools/branches/review-feat-issue-533-premerge-baseline-proof phase: adopted candidate_head: 7fb813446fe0389d73ce306199887002d455c1a4 target_branch: master target_branch_sha: none last_activity: 2026-07-09T03:21:20Z expires_at: 2026-07-09T05:21:20Z blocker: none

sysadmin merged commit 066eec8477 into master

2026-07-08 22:21:31 -05:00

sysadmin referenced this pull request

2026-07-08 22:30:57 -05:00

feat: allow reconciler profile to run merged cleanup directly (Closes #523) #567

sysadmin referenced this pull request

2026-07-08 22:38:45 -05:00

feat: add server code parity gate for mutations (Closes #420) #444

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#574