Extends review_proofs.py with the four #179 proofs, the successor set to the #173 checkout/inventory proofs: - assess_capability_evidence: a capability claim (review_pr, merge_pr, ...) counts only with exact evidence citing gitea_resolve_task_capability output or equivalent runtime context; no claims at all fails closed. - assess_sweep_evidence: secret/provenance sweeps must state the exact command/script/pattern/named method, the scope scanned, and a boolean result; vague summaries are downgraded and a missing sweep fails closed. - assess_live_state_recheck: an explicit pre-mutation recheck must prove the PR is still open, the live head equals the pinned head (full 40-hex), the base branch is unchanged, and blocking review state was checked and absent; not performing it fails closed. - assess_role_boundary: a reviewer run using an author namespace (or vice versa) is clean only with an explicit justification; unreported namespace usage fails closed. build_final_report now takes the four proofs as keyword arguments: any missing or failed proof downgrades the grade, merge_allowed additionally requires the proven live-state recheck, and a merge performed without it is a blocked violation. Existing #173 semantics are unchanged otherwise; gates only get stricter. tests/test_review_proofs.py adds 29 tests covering the issue's harness assertions: capability claims without evidence downgraded, vague sweeps downgraded, missing/stale live-state recheck downgrades and blocks merge (violation when a merge is claimed anyway), unjustified author-namespace use downgraded, and the #173 positive baseline preserved. SKILL.md sections F/G and the review-pr/merge-pr templates now require the capability evidence, exact sweep, pre-verdict and pre-merge live-state rechecks, and reviewer-namespace discipline. Closes #179 Co-Authored-By: Claude Fable 5 <[email protected]>
5.8 KiB
5.8 KiB
Template: review a PR
Copy, fill the <...> fields, and paste as the task prompt.
Task: review PR #<pr> for issue #<n>.
Repo name disambiguation (Gitea-Tools blind review hardening):
- "Gitea-Tools", "gitea tool", "MCP Gitea tool", "gitea MCP tool", "gitea-tools repo"
→ MUST resolve to `Scaled-Tech-Consulting/Gitea-Tools` (never treat as mcp-control-plane).
- "mcp-control-plane", "mcp control plane" → only `Scaled-Tech-Consulting/mcp-control-plane`.
- If user says "open PRs", "the queue", "MCP Gitea tooling" without explicit repo,
or reference is ambiguous: check BOTH configured repos:
`Scaled-Tech-Consulting/Gitea-Tools` and `Scaled-Tech-Consulting/mcp-control-plane`.
- In the final report, always state exactly which repo(s) were checked.
If only one was checked: explicitly say "Only <repo> was checked. Other
configured repos were not checked. This is not a complete queue inventory."
- A single-repo "no open PRs" result MUST NOT be reported as global "no open PRs"
if the other configured repo was not inventoried.
Rules (llm-project-workflow):
- Review in a SEPARATE detached review worktree, never the author's folder.
- You must NOT be the PR author. If the authenticated user == PR author, stop.
A different LLM-Agent-SHA does NOT make you a different actor — only a
different authenticated Gitea user does (docs/llm-agent-sha.md).
- Do not pivot from a reviewer queue task into author implementation unless
the operator explicitly retasks the run. If author namespace was used, the
final report must justify why; author mutations after reviewer queue work
without explicit authorization are a role-boundary violation.
- Do not merge if any check fails.
Steps:
1. Identity Checklist: Before claiming/working on review, verify and state:
- Required identity/profile for this task: reviewer (allowed to review/approve/request_changes)
- Current authenticated identity (from whoami): <username>
- Target task role: reviewer identity (must NOT be the PR author)
*If the current identity does not match the required role (or is the PR author), STOP. Relaunch/switch to the correct profile first.*
2. Verify your authenticated identity (whoami) and the active profile.
Capability evidence (#179): cite the exact gitea_resolve_task_capability
output (or runtime context) for review_pr (and merge_pr if merging later);
a bare "capability checks passed" claim is downgraded. Stay in the
reviewer namespace: any author-namespace call must be justified in the
report (#179).
3. Fetch the PR facts: PR author, head SHA, state (must be open), base branch.
Pin the head SHA in your notes; every later step validates THAT SHA.
4. If authenticated user == PR author → STOP (no self-review).
Session-contamination claims must be evidence-backed (#173): if you
cannot evidence whether this session authored/touched the PR branch,
report contamination as UNKNOWN (not contaminated, not clean) and choose
another PR or stop.
Role-boundary claims must also be evidence-backed (#175): report whether
reviewer namespace, author namespace, author mutations, or review mutations
occurred. Use `review_proofs.assess_role_boundary`; if it is not clean,
downgrade or stop instead of claiming an A-level run.
5. scripts/worktree-review <pr-head-branch> # detached, branches/review-*
cd branches/review-<pr-head-branch-slug>
6. Checkout proof (#173) — prove and state, before any diff review or
validation:
- pinned PR head SHA (from Gitea)
- local checkout SHA (git rev-parse HEAD)
- HEAD == pinned PR head SHA
- diff base == the PR base branch
If HEAD does not match the pinned head → STOP before review/merge.
7. Confirm the worktree is clean. Inspect the FULL diff; confirm scope matches
issue #<n>; flag any unrelated files, secrets, or formatting churn. Check that the PR body correctly uses Gitea-closing keywords (`Closes #N` or `Fixes #N`) instead of non-closing ones (`Implements #N`, `Refs #N`).
Secret/provenance sweep must be exact (#179): state the exact command,
script, grep pattern, or named sweep method AND the scope scanned (e.g.
`git diff prgs/master...HEAD | grep -inE '<pattern>'`); "checked the diff
for secrets" alone is downgraded.
8. Run the test suite; report the exact command and exact results — pass/fail
plus passed/skipped/failed counts, any ignored paths and why they are safe
to ignore, and whether the command differs from the repository's canonical
validation command. Only claim a result after the output has been read.
9. Final live-state recheck (#179), immediately before submitting the review
verdict — re-read the live PR and prove:
- PR still open
- live head SHA still equals the pinned head SHA from step 3
- base branch unchanged
- no undismissed REQUEST_CHANGES / blocking review state left unaccounted
If anything moved → STOP, re-pin, re-validate before any verdict.
10. Post the review verdict: approve only if scope is clean and checks pass;
otherwise request changes with specifics. Never merge from this review step.
Include a "Review Metadata" block (attribution only — docs/llm-agent-sha.md):
Review Metadata:
- LLM-Agent-SHA: llm-<12 lowercase hex, e.g. llm-41d0e7aa9f2c>
- LLM-Role: reviewer
- Authenticated-Gitea-User: <whoami result>
- MCP-Profile: <profile name>
- Eligibility: passed/failed
Handoff: end with a section titled exactly `Controller Handoff` per SKILL.md
§K (compact by default; long form if a merge happened or a gate blocked you),
including the review/merge role fields: Selected PR, Reviewer eligibility,
Pinned reviewed head, Review decision, Merge result, Linked issue status,
Cleanup status. If you could not merge, name the exact gate. Reports missing
the handoff are downgraded (review_proofs.assess_controller_handoff).