Reject reviewer queue-status reports that violate loaded workflow proof gates #339

New Issue

2026-07-07T03:16:44-05:00

jcwalker3 commented

2026-07-07 03:16:44 -05:00

Problem

A reviewer LLM loaded the new router skill and review-merge-pr.md workflow, but then produced a queue-status report that violated the loaded workflow. It claimed inventory completeness using the default Gitea page-size assumption, reused prior diagnostic conflict proof as if it were current proof, skipped PRs without fresh per-PR verification, claimed prior REQUEST_CHANGES state without fetching current review feedback in the visible run, and marked gates as passed even though no PR was selected and those gates were not run.

Observed bad patterns

Claimed inventory was complete because gitea_list_prs returned fewer than the default page size.
Reused “prior diagnostic worktree simulation” as proof for current conflicts.
Classified PRs as blocked without live current-session proof for each skipped PR.
Claimed PR #286 still had unresolved REQUEST_CHANGES without visible current-session review-feedback proof.
Reported Already-landed gate: passed when no PR was selected.
Reported Author-safety result: passed when no selected PR/author gate was actually run.
Reported Merge preflight: passed when no selected PR had merge preflight.
Reported review worktree clean fields as false/passed-style values even though no review worktree was created.
Reported Blockers: none while also saying all open PRs were conflicted or blocked.
Claimed the workflow was loaded but did not obey the workflow proof gates.

Required behavior

Reviewer queue-status reports must be mechanically rejected if they use forbidden proof wording or gate-status claims without matching current-session evidence.

Acceptance criteria

Add verifier coverage that rejects inventory-complete claims based on assumed default Gitea page size.
Add verifier coverage that requires explicit final-page/no-next-page/total-count/page-traversal proof before inventory complete is allowed.
Add verifier coverage that rejects “prior diagnostic” conflict proof as current live proof.
Add verifier coverage that requires skipped non-mergeable PRs to have current-session conflict proof or MCP-provided exact conflict proof.
Add verifier coverage that requires skipped prior-REQUEST_CHANGES PRs to have current-session review-feedback proof and unchanged-head proof.
Add verifier coverage that rejects Already-landed gate: passed when no PR was selected and no candidate head SHA was checked.
Add verifier coverage that rejects Author-safety result: passed when no selected PR author was compared against reviewer identity.
Add verifier coverage that rejects Merge preflight: passed when no selected PR entered merge preflight.
Add verifier coverage that rejects review-worktree clean/dirty claims when no review worktree was created; such fields must be not applicable or none.
Add verifier coverage that rejects Blockers: none when the report says all open PRs are conflicted, blocked, or unverified.
Add verifier coverage that rejects reports claiming the canonical workflow was loaded while violating mandatory proof gates from that workflow.
Add or update final-report schema rules so queue-status-only runs have explicit allowed values for non-applicable gates.
Add tests with a fixture similar to the bad queue-status report to prove the verifier catches these contradictions.

Suggested implementation

Extend the final-report verifier or validation ledger verifier to parse queue-status reports.
Add a “queue status only / no selected PR” schema path.
Require non-applicable fields to use none, not applicable, or not run, not passed.
Treat unsupported proof phrases as hard failures unless matching evidence is present in the run ledger.
Ensure the loaded workflow path is not just reported, but its mandatory proof gates are enforced.

Duplicate search summary

Searched open and closed issues (limit=500 each) for: proof wording, pagination proof, live blocker proof, final report verifier, queue status verification, inventory complete default page size, prior diagnostic conflict proof, already-landed gate passed, merge preflight passed, loaded workflow but violated, reviewer final report contradiction, skipped PR current-session proof.

Partial overlaps only: #330, #327, #293, #322, #318, #299, #298, #331 (closed), #271 (closed). None fully cover queue-status-only verifier gates and contradictory gate-status claims when no PR is selected.

Related issues

#330, #327, #293, #322, #318, #321, #299, #298, #331, #271

## Problem A reviewer LLM loaded the new router skill and `review-merge-pr.md` workflow, but then produced a queue-status report that violated the loaded workflow. It claimed inventory completeness using the default Gitea page-size assumption, reused prior diagnostic conflict proof as if it were current proof, skipped PRs without fresh per-PR verification, claimed prior REQUEST_CHANGES state without fetching current review feedback in the visible run, and marked gates as `passed` even though no PR was selected and those gates were not run. ## Observed bad patterns * Claimed inventory was complete because `gitea_list_prs` returned fewer than the default page size. * Reused “prior diagnostic worktree simulation” as proof for current conflicts. * Classified PRs as blocked without live current-session proof for each skipped PR. * Claimed PR #286 still had unresolved REQUEST_CHANGES without visible current-session review-feedback proof. * Reported `Already-landed gate: passed` when no PR was selected. * Reported `Author-safety result: passed` when no selected PR/author gate was actually run. * Reported `Merge preflight: passed` when no selected PR had merge preflight. * Reported review worktree clean fields as `false`/passed-style values even though no review worktree was created. * Reported `Blockers: none` while also saying all open PRs were conflicted or blocked. * Claimed the workflow was loaded but did not obey the workflow proof gates. ## Required behavior Reviewer queue-status reports must be mechanically rejected if they use forbidden proof wording or gate-status claims without matching current-session evidence. ## Acceptance criteria * Add verifier coverage that rejects inventory-complete claims based on assumed default Gitea page size. * Add verifier coverage that requires explicit final-page/no-next-page/total-count/page-traversal proof before `inventory complete` is allowed. * Add verifier coverage that rejects “prior diagnostic” conflict proof as current live proof. * Add verifier coverage that requires skipped non-mergeable PRs to have current-session conflict proof or MCP-provided exact conflict proof. * Add verifier coverage that requires skipped prior-REQUEST_CHANGES PRs to have current-session review-feedback proof and unchanged-head proof. * Add verifier coverage that rejects `Already-landed gate: passed` when no PR was selected and no candidate head SHA was checked. * Add verifier coverage that rejects `Author-safety result: passed` when no selected PR author was compared against reviewer identity. * Add verifier coverage that rejects `Merge preflight: passed` when no selected PR entered merge preflight. * Add verifier coverage that rejects review-worktree clean/dirty claims when no review worktree was created; such fields must be `not applicable` or `none`. * Add verifier coverage that rejects `Blockers: none` when the report says all open PRs are conflicted, blocked, or unverified. * Add verifier coverage that rejects reports claiming the canonical workflow was loaded while violating mandatory proof gates from that workflow. * Add or update final-report schema rules so queue-status-only runs have explicit allowed values for non-applicable gates. * Add tests with a fixture similar to the bad queue-status report to prove the verifier catches these contradictions. ## Suggested implementation * Extend the final-report verifier or validation ledger verifier to parse queue-status reports. * Add a “queue status only / no selected PR” schema path. * Require non-applicable fields to use `none`, `not applicable`, or `not run`, not `passed`. * Treat unsupported proof phrases as hard failures unless matching evidence is present in the run ledger. * Ensure the loaded workflow path is not just reported, but its mandatory proof gates are enforced. ## Duplicate search summary Searched open and closed issues (limit=500 each) for: proof wording, pagination proof, live blocker proof, final report verifier, queue status verification, inventory complete default page size, prior diagnostic conflict proof, already-landed gate passed, merge preflight passed, loaded workflow but violated, reviewer final report contradiction, skipped PR current-session proof. Partial overlaps only: #330, #327, #293, #322, #318, #299, #298, #331 (closed), #271 (closed). None fully cover queue-status-only verifier gates and contradictory gate-status claims when no PR is selected. ## Related issues #330, #327, #293, #322, #318, #321, #299, #298, #331, #271

jcwalker3 added the status:in-progress label 2026-07-07 03:56:36 -05:00

jcwalker3 referenced this issue from a commit

2026-07-07 03:57:53 -05:00

feat: add queue-status report verifier for reviewer handoffs (#339)

jcwalker3 referenced a pull request that will close this issue

2026-07-07 03:57:56 -05:00

feat: add queue-status report verifier for reviewer handoffs (Closes #339) #347

sysadmin referenced this issue

2026-07-07 04:19:57 -05:00

feat: add queue-status report verifier for reviewer handoffs (Closes #339) #347

sysadmin closed this issue

2026-07-07 04:20:09 -05:00