Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 73 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

Reviewer approval must require baseline proof when full test suite fails #323

New Issue
Closed
opened 2026-07-07 01:49:09 -05:00 by jcwalker3 · 1 comment
jcwalker3 commented 2026-07-07 01:49:09 -05:00
Owner
Copy Link
Copy Source

A reviewer workflow approved and merged PR #280 even though the full test suite failed with 23 failures and 7 errors. The report stated the failure signature was identical to master, but the baseline comparison was run in the main checkout and did not provide full failure-signature proof.

Problem

Approving a PR when the full suite fails is high risk unless the workflow proves the failures are pre-existing and unrelated.

Required behavior

  1. If full suite fails on the PR worktree, default action is REQUEST_CHANGES.
  2. Approval may proceed only if a project policy allows baseline comparison and all proof gates pass.
  3. Baseline proof must be collected in a clean baseline worktree under branches/, not the main checkout.
  4. Report must include:
    • PR full-suite command/result
    • baseline full-suite command/result
    • exact baseline SHA
    • exact PR SHA
    • matching failure list
    • proof new/changed tests passed
    • explanation why failures are unrelated to PR
  5. If failure signatures differ or proof is incomplete, request changes.

Acceptance criteria

  • Full-suite failure blocks approval by default.
  • Approval with full-suite failure requires baseline worktree proof.
  • Main checkout baseline testing is rejected.
  • Final-report validator rejects vague "same as master" claims.
  • Tests cover matching baseline failures, mismatched failures, incomplete proof, and full-suite pass.
A reviewer workflow approved and merged PR #280 even though the full test suite failed with 23 failures and 7 errors. The report stated the failure signature was identical to master, but the baseline comparison was run in the main checkout and did not provide full failure-signature proof. ## Problem Approving a PR when the full suite fails is high risk unless the workflow proves the failures are pre-existing and unrelated. ## Required behavior 1. If full suite fails on the PR worktree, default action is `REQUEST_CHANGES`. 2. Approval may proceed only if a project policy allows baseline comparison and all proof gates pass. 3. Baseline proof must be collected in a clean baseline worktree under `branches/`, not the main checkout. 4. Report must include: - PR full-suite command/result - baseline full-suite command/result - exact baseline SHA - exact PR SHA - matching failure list - proof new/changed tests passed - explanation why failures are unrelated to PR 5. If failure signatures differ or proof is incomplete, request changes. ## Acceptance criteria - [ ] Full-suite failure blocks approval by default. - [ ] Approval with full-suite failure requires baseline worktree proof. - [ ] Main checkout baseline testing is rejected. - [ ] Final-report validator rejects vague "same as master" claims. - [ ] Tests cover matching baseline failures, mismatched failures, incomplete proof, and full-suite pass.
jcwalker3 added the status:in-progress label 2026-07-07 04:19:38 -05:00
jcwalker3 commented 2026-07-07 04:25:36 -05:00
Author
Owner
Copy Link
Copy Source

Implementation PR opened: #361 (branch feat/issue-323-full-suite-approval-gate, head 5fa5760).

New assess_full_suite_failure_approval_gate blocks approval fail-closed when the full suite fails, unless the #325 baseline proof passes plus PR/baseline suite commands, new-test evidence, and a pre-existing-failure explanation are provided. Default action on failure is REQUEST_CHANGES. 12 new tests. Awaiting reviewer.

Implementation PR opened: #361 (branch `feat/issue-323-full-suite-approval-gate`, head 5fa5760). New `assess_full_suite_failure_approval_gate` blocks approval fail-closed when the full suite fails, unless the #325 baseline proof passes plus PR/baseline suite commands, new-test evidence, and a pre-existing-failure explanation are provided. Default action on failure is REQUEST_CHANGES. 12 new tests. Awaiting reviewer.
sysadmin removed the status:in-progress label 2026-07-07 08:09:16 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#323