Reviewer validation reports must distinguish PR-head tests from diagnostic edited-worktree tests #316

New Issue

2026-07-07T01:16:40-05:00

jcwalker3 commented

2026-07-07 01:16:40 -05:00

A reviewer workflow for PR #280 ran tests, observed a failure, edited a test file locally, ran tests again, then submitted REQUEST_CHANGES.

Problem:

Once a reviewer edits the worktree, subsequent test results no longer validate the PR head. Reports must not mix official PR-head validation results with diagnostic local-fix experiments.

Required behavior:

Before official validation, record:
- PR head SHA
- worktree path
- clean status
After official validation, verify the worktree remains clean.
If the worktree was modified, mark official validation as contaminated unless the mutation occurred after validation and is reported separately.
Any post-edit test run must be labeled:
Diagnostic local experiment — not PR-head validation
Request-changes comments may mention diagnostic findings, but must clearly state the blocker from the unmodified PR head.
Final reports must include:
- official validation command/result
- official validation worktree dirty state after command
- any diagnostic edit path
- diagnostic command/result
- whether diagnostic results were used only for suggested fix

Acceptance criteria:

Reports cannot treat post-edit tests as official PR validation.
Final-report validator fails if worktree dirty state after validation is omitted.
Diagnostic test output is clearly separated from PR-head validation.
Tests cover failing PR-head validation, post-edit diagnostic pass, post-edit diagnostic fail, and no diagnostic edits.

A reviewer workflow for PR #280 ran tests, observed a failure, edited a test file locally, ran tests again, then submitted `REQUEST_CHANGES`. Problem: Once a reviewer edits the worktree, subsequent test results no longer validate the PR head. Reports must not mix official PR-head validation results with diagnostic local-fix experiments. Required behavior: 1. Before official validation, record: * PR head SHA * worktree path * clean status 2. After official validation, verify the worktree remains clean. 3. If the worktree was modified, mark official validation as contaminated unless the mutation occurred after validation and is reported separately. 4. Any post-edit test run must be labeled: `Diagnostic local experiment — not PR-head validation` 5. Request-changes comments may mention diagnostic findings, but must clearly state the blocker from the unmodified PR head. 6. Final reports must include: * official validation command/result * official validation worktree dirty state after command * any diagnostic edit path * diagnostic command/result * whether diagnostic results were used only for suggested fix Acceptance criteria: * Reports cannot treat post-edit tests as official PR validation. * Final-report validator fails if worktree dirty state after validation is omitted. * Diagnostic test output is clearly separated from PR-head validation. * Tests cover failing PR-head validation, post-edit diagnostic pass, post-edit diagnostic fail, and no diagnostic edits.

jcwalker3 added the status:in-progress label 2026-07-07 04:14:10 -05:00

jcwalker3 referenced this issue from a commit

2026-07-07 04:19:40 -05:00

Add validation integrity verifier for PR-head vs diagnostic tests (#316)

jcwalker3 referenced a pull request that will close this issue

2026-07-07 04:20:15 -05:00

Add validation integrity verifier for PR-head vs diagnostic tests (#316) #358

jcwalker3 commented

2026-07-07 04:20:19 -05:00

PR #358 is ready for review.

Adds assess_validation_integrity_report to separate official PR-head validation from diagnostic edited-worktree experiments. Tests: 166 passed in the validation integrity slice.

PR #358 is ready for review. Adds `assess_validation_integrity_report` to separate official PR-head validation from diagnostic edited-worktree experiments. Tests: 166 passed in the validation integrity slice.

jcwalker3 referenced this issue

2026-07-07 04:20:22 -05:00

Add validation integrity verifier for PR-head vs diagnostic tests (#316) #358

jcwalker3 referenced this issue

2026-07-07 04:22:56 -05:00

Fix TestIssueLockArtifactWarning profile gate setup (full suite failure) #359

sysadmin closed this issue

2026-07-07 08:31:02 -05:00

sysadmin removed the status:in-progress label 2026-07-07 08:31:04 -05:00

sysadmin referenced this issue from a commit

2026-07-07 08:31:05 -05:00

Merge pull request 'Add validation integrity verifier for PR-head vs diagnostic tests (#316)' (#358) from feat/issue-316-validation-integrity into master