Harden LLM role/task alignment and PR review feedback discovery #167

New Issue

2026-07-05T11:33:05-05:00

jcwalker3 commented

2026-07-05 11:33:05 -05:00

Problem

A recent LLM session working on Gitea-Tools PR #165 showed a workflow ambiguity:

The operator says the LLM was asked to review PR #165.
The session authenticated as jcwalker3 / prgs-author.
Instead of stopping as "not reviewer," it performed author-side fixes, committed, pushed, and commented.
The author-side fixes appear useful and likely correct, but the behavior would be wrong if the requested task was review.
The session also attempted to read REQUEST_CHANGES feedback via issue comments, but formal PR review comments were not surfaced through that path.
It proceeded using the known fixes from the prompt instead of an MCP-native review-feedback view.
The report also said the full suite was "attempted" but had shell-invocation quirks, rather than reporting the exact failed command/output and whether full suite validation actually passed.

This shows that the MCP workflow needs stronger task/role alignment and better review feedback discovery.

Goals

Prevent an LLM from performing author mutations when the requested task is review.
Make review/change-request feedback discoverable through MCP tools.
Improve validation reporting when commands fail or are skipped.
Reduce ambiguity between:
- review/merge work
- author fix work
- issue discussion/comment work

Requirements

1. Add or improve task/role guard guidance

If the requested task is review or merge:

active identity must be sysadmin / prgs-reviewer
if active identity is jcwalker3 / prgs-author, the LLM must stop
it must not commit, push, edit files, or comment as author unless the operator explicitly changes the task to "address author fixes"

If the requested task is author-side remediation:

active identity should be jcwalker3 / prgs-author
the LLM may commit/push to the PR branch
it must not submit review verdicts or merge

2. Add a review-feedback discovery tool or improve existing tools

Possible tool names:

gitea_list_pr_reviews
gitea_list_pr_review_comments
gitea_get_pr_review_feedback
or extend gitea_view_pr

The tool should expose safe, redacted review feedback:

review verdicts: APPROVE / REQUEST_CHANGES / COMMENT
reviewer username
review timestamp
review body
unresolved change-request summaries, if available
file/path comments, if available
latest review state by reviewer
whether the PR currently has blocking change requests

Do not require LLMs to infer formal review feedback from issue comments.

3. Improve PR state summary

gitea_view_pr or a dedicated helper should clearly report:

whether PR has change requests
whether approval is visible
latest reviewed head SHA
current head SHA
whether review feedback is stale after new commits
whether author fixes have been pushed after REQUEST_CHANGES

4. Improve validation reporting

If a required validation command is not run successfully:

report the exact command
report the exact failure/output
state whether it was skipped, failed, or replaced by targeted checks
do not describe it vaguely as "shell-invocation quirks"
do not imply full-suite success unless the full-suite command passed

5. Update operator guide / skill registry

Docs should clearly distinguish:

"review PR"
"address PR change requests"
"merge PR"
"comment on issue discussion"
"comment on PR"
"author implementation"

Each workflow must state:

required identity/profile
allowed actions
forbidden actions
when to stop

6. Add tests where applicable

Test coverage should prove:

review task with author profile returns stop guidance
author-fix task with author profile is allowed
review/merge task with reviewer profile is allowed only after eligibility checks
review feedback is discoverable without reading issue comments
raw review/comment content is redacted where needed
change-request state is visible and updates after new commits
validation-report helper distinguishes passed, failed, skipped, and not-run commands

Acceptance criteria

An LLM asked to review cannot accidentally perform author-side commits/pushes from prgs-author.
An LLM asked to address change requests can do so as prgs-author.
Formal PR review feedback can be retrieved through MCP without relying on issue comments.
PR summaries clearly show blocking change requests and approval state.
Validation reports do not hide failed/skipped full-suite checks behind vague language.
No permissions are broadened.
No self-review or self-merge paths are introduced.
No secrets, tokens, keychain IDs, raw service URLs, or private config values are exposed.
No production behavior.
No tags/releases.

Notes from PR #165 incident

Grok authenticated as jcwalker3 / prgs-author.
It fixed the requested blockers and pushed head 88296f0262c50bd5e2d5f2aee16cca20c544749a.
It posted a PR comment using issue-comment tooling.
It did not review or merge, which is good.
But if the original task was "review PR #165," it should have stopped immediately instead of doing author work.
The session could not see formal REQUEST_CHANGES through issue comments, which indicates a tooling gap.

Related: #164 (read-only PR queue inventory under author profiles — phase split shares the same task/role guard), #166 (stale/conflicting PR state reconciliation — the "latest reviewed head SHA vs current head SHA" staleness fields overlap), #139 (role-aware task routing umbrella), #141 (gitea_resolve_task_capability — natural home for review-task stop guidance), #142 (structured permission failure reports).

## Problem A recent LLM session working on Gitea-Tools PR #165 showed a workflow ambiguity: * The operator says the LLM was asked to review PR #165. * The session authenticated as `jcwalker3 / prgs-author`. * Instead of stopping as "not reviewer," it performed author-side fixes, committed, pushed, and commented. * The author-side fixes appear useful and likely correct, but the behavior would be wrong if the requested task was review. * The session also attempted to read REQUEST_CHANGES feedback via issue comments, but formal PR review comments were not surfaced through that path. * It proceeded using the known fixes from the prompt instead of an MCP-native review-feedback view. * The report also said the full suite was "attempted" but had shell-invocation quirks, rather than reporting the exact failed command/output and whether full suite validation actually passed. This shows that the MCP workflow needs stronger task/role alignment and better review feedback discovery. ## Goals 1. Prevent an LLM from performing author mutations when the requested task is review. 2. Make review/change-request feedback discoverable through MCP tools. 3. Improve validation reporting when commands fail or are skipped. 4. Reduce ambiguity between: * review/merge work * author fix work * issue discussion/comment work ## Requirements ### 1. Add or improve task/role guard guidance If the requested task is review or merge: * active identity must be `sysadmin / prgs-reviewer` * if active identity is `jcwalker3 / prgs-author`, the LLM must stop * it must not commit, push, edit files, or comment as author unless the operator explicitly changes the task to "address author fixes" If the requested task is author-side remediation: * active identity should be `jcwalker3 / prgs-author` * the LLM may commit/push to the PR branch * it must not submit review verdicts or merge ### 2. Add a review-feedback discovery tool or improve existing tools Possible tool names: * `gitea_list_pr_reviews` * `gitea_list_pr_review_comments` * `gitea_get_pr_review_feedback` * or extend `gitea_view_pr` The tool should expose safe, redacted review feedback: * review verdicts: APPROVE / REQUEST_CHANGES / COMMENT * reviewer username * review timestamp * review body * unresolved change-request summaries, if available * file/path comments, if available * latest review state by reviewer * whether the PR currently has blocking change requests Do not require LLMs to infer formal review feedback from issue comments. ### 3. Improve PR state summary `gitea_view_pr` or a dedicated helper should clearly report: * whether PR has change requests * whether approval is visible * latest reviewed head SHA * current head SHA * whether review feedback is stale after new commits * whether author fixes have been pushed after REQUEST_CHANGES ### 4. Improve validation reporting If a required validation command is not run successfully: * report the exact command * report the exact failure/output * state whether it was skipped, failed, or replaced by targeted checks * do not describe it vaguely as "shell-invocation quirks" * do not imply full-suite success unless the full-suite command passed ### 5. Update operator guide / skill registry Docs should clearly distinguish: * "review PR" * "address PR change requests" * "merge PR" * "comment on issue discussion" * "comment on PR" * "author implementation" Each workflow must state: * required identity/profile * allowed actions * forbidden actions * when to stop ### 6. Add tests where applicable Test coverage should prove: * review task with author profile returns stop guidance * author-fix task with author profile is allowed * review/merge task with reviewer profile is allowed only after eligibility checks * review feedback is discoverable without reading issue comments * raw review/comment content is redacted where needed * change-request state is visible and updates after new commits * validation-report helper distinguishes passed, failed, skipped, and not-run commands ## Acceptance criteria * An LLM asked to review cannot accidentally perform author-side commits/pushes from `prgs-author`. * An LLM asked to address change requests can do so as `prgs-author`. * Formal PR review feedback can be retrieved through MCP without relying on issue comments. * PR summaries clearly show blocking change requests and approval state. * Validation reports do not hide failed/skipped full-suite checks behind vague language. * No permissions are broadened. * No self-review or self-merge paths are introduced. * No secrets, tokens, keychain IDs, raw service URLs, or private config values are exposed. * No production behavior. * No tags/releases. ## Notes from PR #165 incident * Grok authenticated as `jcwalker3 / prgs-author`. * It fixed the requested blockers and pushed head `88296f0262c50bd5e2d5f2aee16cca20c544749a`. * It posted a PR comment using issue-comment tooling. * It did not review or merge, which is good. * But if the original task was "review PR #165," it should have stopped immediately instead of doing author work. * The session could not see formal REQUEST_CHANGES through issue comments, which indicates a tooling gap. **Related:** #164 (read-only PR queue inventory under author profiles — phase split shares the same task/role guard), #166 (stale/conflicting PR state reconciliation — the "latest reviewed head SHA vs current head SHA" staleness fields overlap), #139 (role-aware task routing umbrella), #141 (`gitea_resolve_task_capability` — natural home for review-task stop guidance), #142 (structured permission failure reports).