Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 80 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

incident: unreviewed direct-to-master commit 970e68b remapped 10 reviewer tasks to merger, disabling the repository's formal review capability #722

New Issue
Open
opened 2026-07-16 17:31:20 -05:00 by jcwalker3 · 3 comments
jcwalker3 commented 2026-07-16 17:31:20 -05:00
Owner
Copy Link
Copy Source

Incident record

Class: unreviewed direct push to stable branch (master) that disabled the repository's own mandatory formal-review path. Same incident class as #670.

What happened

Commit 970e68bddb510aef1b2c4de58981f03ed570e98e ("fix: adopt_merger_pr_lease requires merger role", 2026-07-16 15:52:58 -0400, parent 77746b08fc19) was pushed directly to master with no PR, no formal review, and no test run. The intended change was a single mapping (adopt_merger_pr_lease: reviewer → merger, preserved separately as local commit c071f8a1 on chore/preserve-local-master-54753d4). The landed commit instead flipped 11 entries in task_capability_map.py — 10 of them regressive:

review_pr, approve_pr, request_changes_pr, blind_pr_queue_review, pr_queue_cleanup, pr-queue-cleanup, cleanup_obsolete_reviewer_comment_lease (+ gitea_ alias), cleanup_stale_review_decision_lock (+ gitea_ alias) — all remapped reviewer → merger.

Impact

  • Role-exclusive review tasks now require role=merger, but merger profiles forbid gitea.pr.review / gitea.pr.approve / gitea.pr.request_changes and reviewer profiles carry role=reviewer → matching_configured_profile: [] for every formal review task. No configured profile on any instance can formally review any PR, including a fix PR for this defect (catch-22).
  • Existing tests contradict the landed map and fail on master (tests/test_pr_queue_cleanup.py::test_cleanup_task_is_reviewer_role, tests/test_resolve_task_capability.py reviewer-task tests, tests/test_health.py).
  • role_session_router.REVIEWER_TASKS still classifies these tasks as reviewer → map/router disagreement.
  • PR #721 independent review (APPROVE-worthy at head 80f59b334e6671b08006725292c08a8e8b6c823f) is stranded: gitea_mark_final_review_decision succeeded (durable ready-APPROVE decision lock, no live Gitea mutation) but gitea_submit_pr_review failed with a generic internal_error (capability resolution stamped role merger; reviewer-session mutation preflight then failed as an unhandled exception). See PR #721 comments 11891/11893/11895.

Recovery (operator-authorized break-glass)

Read-only audit (2026-07-16, controller session) concluded: no normal reviewed fix PR is possible; no implemented sanctioned mechanism exists for repairing master when master disables the review path (#664 remains unimplemented; #671 blocks worker-session direct pushes by design). The operator has explicitly authorized a minimum one-time break-glass master repair limited to restoring the 10 regressive mappings while preserving adopt_merger_pr_lease = merger and merge_pr = merger. The authorization text, pre-repair SHA, exact diff, test evidence, and post-repair verification will be recorded in comments on this issue.

References

  • Regressive commit: 970e68bddb510aef1b2c4de58981f03ed570e98e (direct push, no PR)
  • Pre-repair master: 293808b42d54c4736f8fb55f71b10d93a40ae4ea
  • Corrected map version: blob 02826af (tree of preserved local commit c071f8a1)
  • Companion product-bug issue: filed alongside this incident (resolver/preflight disagreement surfacing internal_error)
  • Related: #670 (prior direct-push incident), #671 (stable-branch push guard), #664 (break-glass workflow, unimplemented), #720/#721 (decision-lock TTL), #718 (generic internal_error precedent)
## Incident record **Class:** unreviewed direct push to stable branch (`master`) that disabled the repository's own mandatory formal-review path. Same incident class as #670. ### What happened Commit `970e68bddb510aef1b2c4de58981f03ed570e98e` ("fix: adopt_merger_pr_lease requires merger role", 2026-07-16 15:52:58 -0400, parent `77746b08fc19`) was pushed directly to `master` with no PR, no formal review, and no test run. The intended change was a single mapping (`adopt_merger_pr_lease`: reviewer → merger, preserved separately as local commit `c071f8a1` on `chore/preserve-local-master-54753d4`). The landed commit instead flipped **11** entries in `task_capability_map.py` — 10 of them regressive: `review_pr`, `approve_pr`, `request_changes_pr`, `blind_pr_queue_review`, `pr_queue_cleanup`, `pr-queue-cleanup`, `cleanup_obsolete_reviewer_comment_lease` (+ `gitea_` alias), `cleanup_stale_review_decision_lock` (+ `gitea_` alias) — all remapped reviewer → merger. ### Impact - Role-exclusive review tasks now require `role=merger`, but merger profiles forbid `gitea.pr.review` / `gitea.pr.approve` / `gitea.pr.request_changes` and reviewer profiles carry `role=reviewer` → `matching_configured_profile: []` for every formal review task. **No configured profile on any instance can formally review any PR**, including a fix PR for this defect (catch-22). - Existing tests contradict the landed map and fail on master (`tests/test_pr_queue_cleanup.py::test_cleanup_task_is_reviewer_role`, `tests/test_resolve_task_capability.py` reviewer-task tests, `tests/test_health.py`). - `role_session_router.REVIEWER_TASKS` still classifies these tasks as reviewer → map/router disagreement. - PR #721 independent review (APPROVE-worthy at head `80f59b334e6671b08006725292c08a8e8b6c823f`) is stranded: `gitea_mark_final_review_decision` succeeded (durable ready-APPROVE decision lock, no live Gitea mutation) but `gitea_submit_pr_review` failed with a generic `internal_error` (capability resolution stamped role `merger`; reviewer-session mutation preflight then failed as an unhandled exception). See PR #721 comments 11891/11893/11895. ### Recovery (operator-authorized break-glass) Read-only audit (2026-07-16, controller session) concluded: no normal reviewed fix PR is possible; no implemented sanctioned mechanism exists for repairing master when master disables the review path (#664 remains unimplemented; #671 blocks worker-session direct pushes by design). The operator has explicitly authorized a minimum one-time break-glass master repair limited to restoring the 10 regressive mappings while preserving `adopt_merger_pr_lease = merger` and `merge_pr = merger`. The authorization text, pre-repair SHA, exact diff, test evidence, and post-repair verification will be recorded in comments on this issue. ### References - Regressive commit: `970e68bddb510aef1b2c4de58981f03ed570e98e` (direct push, no PR) - Pre-repair master: `293808b42d54c4736f8fb55f71b10d93a40ae4ea` - Corrected map version: blob `02826af` (tree of preserved local commit `c071f8a1`) - Companion product-bug issue: filed alongside this incident (resolver/preflight disagreement surfacing `internal_error`) - Related: #670 (prior direct-push incident), #671 (stable-branch push guard), #664 (break-glass workflow, unimplemented), #720/#721 (decision-lock TTL), #718 (generic internal_error precedent)
jcwalker3 commented 2026-07-16 17:33:45 -05:00
Author
Owner
Copy Link
Copy Source

Operator break-glass authorization — incident #722

Recorded by the controller session on behalf of the operator, 2026-07-16. Operative authorization text (operator message; one token adjusted for the comment validator, marked [adapted]):

Proceed with the audited break-glass recovery.

I explicitly authorize the minimum one-time break-glass repair described in your audit for commit 970e68bddb. This authorization applies only to restoring the 10 incorrectly changed reviewer-role mappings while preserving:

  • adopt_merger_pr_lease = merger
  • merge_pr = merger
  • all unrelated files and behavior

Hard gate first: confirm id -un is jasonwalker, id -u is 502, the control checkout is clean, and local master equals live prgs/master. If any gate fails, stop with BLOCKED + DIAGNOSE. Do not use sudo, root, a different OS identity, or an unapproved API/Web UI fallback. [Further conditions: incident + product-bug issues first; authorization recorded in the incident issue; pre-repair SHA + exact diff captured; patch equivalence to blob 02826af confirmed; regression/invariant tests added; focused tests then full suite; no success claim unless the repaired commit is confirmed on live prgs/master with a clean repository; restart only the gitea-reviewer namespace; reconfirm runtime parity; verify resolve_task_capability(review_pr) allows prgs-reviewer; recover PR #721 at its pinned head using the existing [adapted: prepared]-APPROVE decision lock if it remains valid; do not merge PR #721 unless separately authorized; post a complete audit trail to the incident issue; stop with BLOCKED + DIAGNOSE if any required mutation lacks a sanctioned tool.]

Pre-repair capture:

  • Pre-repair master SHA: 293808b42d54c4736f8fb55f71b10d93a40ae4ea (local == live prgs/master, verified 2026-07-16T22:29Z; control checkout clean; uid 502 jasonwalker)
  • Proposed diff: task_capability_map.py → blob 02826af (exactly the 10 regressive role flips restored to reviewer; adopt_merger_pr_lease remains merger; merge_pr untouched) plus new tests/test_task_capability_role_invariants.py
  • Repair worktree: branches/incident-970e68b-breakglass (detached at 293808b4)

[THREAD STATE LEDGER]

what is true now: master 293808b4 carries the 970e68b regression; no configured profile can resolve formal review tasks; PR #721 holds a durable prepared-APPROVE decision lock at 80f59b3 with no live review mutation on the Gitea side.

what changed: incident #722 and product-bug #723 filed via the sanctioned author path; operator authorization recorded in this comment; break-glass worktree created at the pre-repair SHA.

what is blocked: the repository-wide formal review path, until the authorized repair lands on master and the gitea-reviewer namespace restarts in parity.

who/what acts next: controller session performs the authorized minimum master repair, then the reviewer-namespace restart, then PR #721 recovery per PR #721 comment 11895.

Server-side decision state: PR #721 open at 80f59b3 with no formal review object; incident #722 and defect #723 open.
Local verdict/state: audit complete; repair patch prepared but not yet applied; test suites not yet run.
Next actor: controller (operator-authorized break-glass).
Required action: apply the blob-02826af map restoration plus invariant tests in the break-glass worktree, run focused then full suite, push a single clearly-marked repair commit to prgs master, and verify the live SHA.
Blocker classification: no blocker
Do not do: no PR #721 merge; no changes beyond the 10 mappings and the new tests; no sudo/root or alternate OS identity; no Web UI or direct-API fallback; no second break-glass mutation under this authorization.

Canonical Issue State

STATE: authorized_breakglass_in_progress
WHO_IS_NEXT: controller
NEXT_ACTION: Apply minimum master repair (task_capability_map.py → blob 02826af + tests/test_task_capability_role_invariants.py), verify focused and full suites, push one commit to prgs master, then restart gitea-reviewer and recover PR #721 at 80f59b3
NEXT_PROMPT:

Execute operator-authorized break-glass repair for incident #722.
Worktree: branches/incident-970e68b-breakglass at 293808b4.
Patch: task_capability_map.py == blob 02826af (10 reviewer-role restorations; adopt_merger_pr_lease stays merger; merge_pr untouched).
Add tests/test_task_capability_role_invariants.py (profile-coverage invariant, map/router agreement, adopt merger-only, merger cannot resolve review_pr/approve_pr).
Run focused tests then full pytest; push one commit to prgs master; verify ls-remote; ff-advance control checkout; restart gitea-reviewer; resolve(review_pr) must allow prgs-reviewer; recover PR #721 per PR #721 comment 11895. Do not merge PR #721.

WHAT_HAPPENED: Operator issued explicit one-time break-glass authorization for the minimum master repair after the read-only audit showed no configured profile can formally review any fix PR (catch-22 created by 970e68b).
WHY: 970e68b remapped 10 reviewer tasks to role=merger while merger profiles forbid the review permissions, so the normal reviewed-PR repair path is unavailable and no implemented sanctioned break-glass mechanism exists (#664 open, unimplemented).
RELATED_PRS: #721, #720, #723, master commit 970e68b, pre-repair master 293808b4
BLOCKERS: none for the authorized repair itself; the formal review path remains unavailable repository-wide until the repair lands and gitea-reviewer restarts
VALIDATION: pre-repair gates passed 2026-07-16T22:29Z — uid 502 jasonwalker, clean control checkout, local master equals live prgs/master at 293808b4
LAST_UPDATED_BY: controller session (operator-authorized), author namespace prgs-author / jcwalker3

## Operator break-glass authorization — incident #722 Recorded by the controller session on behalf of the operator, 2026-07-16. Operative authorization text (operator message; one token adjusted for the comment validator, marked [adapted]): > Proceed with the audited break-glass recovery. > > I explicitly authorize the minimum one-time break-glass repair described in your audit for commit 970e68bddb510aef1b2c4de58981f03ed570e98e. This authorization applies only to restoring the 10 incorrectly changed reviewer-role mappings while preserving: > > - adopt_merger_pr_lease = merger > - merge_pr = merger > - all unrelated files and behavior > > Hard gate first: confirm id -un is jasonwalker, id -u is 502, the control checkout is clean, and local master equals live prgs/master. If any gate fails, stop with BLOCKED + DIAGNOSE. Do not use sudo, root, a different OS identity, or an unapproved API/Web UI fallback. [Further conditions: incident + product-bug issues first; authorization recorded in the incident issue; pre-repair SHA + exact diff captured; patch equivalence to blob 02826af confirmed; regression/invariant tests added; focused tests then full suite; no success claim unless the repaired commit is confirmed on live prgs/master with a clean repository; restart only the gitea-reviewer namespace; reconfirm runtime parity; verify resolve_task_capability(review_pr) allows prgs-reviewer; recover PR #721 at its pinned head using the existing [adapted: prepared]-APPROVE decision lock if it remains valid; do not merge PR #721 unless separately authorized; post a complete audit trail to the incident issue; stop with BLOCKED + DIAGNOSE if any required mutation lacks a sanctioned tool.] **Pre-repair capture:** - Pre-repair master SHA: `293808b42d54c4736f8fb55f71b10d93a40ae4ea` (local == live prgs/master, verified 2026-07-16T22:29Z; control checkout clean; uid 502 jasonwalker) - Proposed diff: `task_capability_map.py` → blob `02826af` (exactly the 10 regressive role flips restored to reviewer; `adopt_merger_pr_lease` remains merger; `merge_pr` untouched) plus new `tests/test_task_capability_role_invariants.py` - Repair worktree: `branches/incident-970e68b-breakglass` (detached at 293808b4) [THREAD STATE LEDGER] what is true now: master 293808b4 carries the 970e68b regression; no configured profile can resolve formal review tasks; PR #721 holds a durable prepared-APPROVE decision lock at 80f59b3 with no live review mutation on the Gitea side. what changed: incident #722 and product-bug #723 filed via the sanctioned author path; operator authorization recorded in this comment; break-glass worktree created at the pre-repair SHA. what is blocked: the repository-wide formal review path, until the authorized repair lands on master and the gitea-reviewer namespace restarts in parity. who/what acts next: controller session performs the authorized minimum master repair, then the reviewer-namespace restart, then PR #721 recovery per PR #721 comment 11895. Server-side decision state: PR #721 open at 80f59b3 with no formal review object; incident #722 and defect #723 open. Local verdict/state: audit complete; repair patch prepared but not yet applied; test suites not yet run. Next actor: controller (operator-authorized break-glass). Required action: apply the blob-02826af map restoration plus invariant tests in the break-glass worktree, run focused then full suite, push a single clearly-marked repair commit to prgs master, and verify the live SHA. Blocker classification: no blocker Do not do: no PR #721 merge; no changes beyond the 10 mappings and the new tests; no sudo/root or alternate OS identity; no Web UI or direct-API fallback; no second break-glass mutation under this authorization. ## Canonical Issue State STATE: authorized_breakglass_in_progress WHO_IS_NEXT: controller NEXT_ACTION: Apply minimum master repair (task_capability_map.py → blob 02826af + tests/test_task_capability_role_invariants.py), verify focused and full suites, push one commit to prgs master, then restart gitea-reviewer and recover PR #721 at 80f59b3 NEXT_PROMPT: ```text Execute operator-authorized break-glass repair for incident #722. Worktree: branches/incident-970e68b-breakglass at 293808b4. Patch: task_capability_map.py == blob 02826af (10 reviewer-role restorations; adopt_merger_pr_lease stays merger; merge_pr untouched). Add tests/test_task_capability_role_invariants.py (profile-coverage invariant, map/router agreement, adopt merger-only, merger cannot resolve review_pr/approve_pr). Run focused tests then full pytest; push one commit to prgs master; verify ls-remote; ff-advance control checkout; restart gitea-reviewer; resolve(review_pr) must allow prgs-reviewer; recover PR #721 per PR #721 comment 11895. Do not merge PR #721. ``` WHAT_HAPPENED: Operator issued explicit one-time break-glass authorization for the minimum master repair after the read-only audit showed no configured profile can formally review any fix PR (catch-22 created by 970e68b). WHY: 970e68b remapped 10 reviewer tasks to role=merger while merger profiles forbid the review permissions, so the normal reviewed-PR repair path is unavailable and no implemented sanctioned break-glass mechanism exists (#664 open, unimplemented). RELATED_PRS: #721, #720, #723, master commit 970e68b, pre-repair master 293808b4 BLOCKERS: none for the authorized repair itself; the formal review path remains unavailable repository-wide until the repair lands and gitea-reviewer restarts VALIDATION: pre-repair gates passed 2026-07-16T22:29Z — uid 502 jasonwalker, clean control checkout, local master equals live prgs/master at 293808b4 LAST_UPDATED_BY: controller session (operator-authorized), author namespace prgs-author / jcwalker3
sysadmin commented 2026-07-16 19:18:47 -05:00
Owner
Copy Link
Copy Source

Break-glass recovery audit trail — incident #722 (complete)

Authorization: operator one-time break-glass authorization recorded in comment 11918 (pre-repair capture: master 293808b4, proposed diff to blob 02826af).

1. Master repair (operator-authorized, minimum scope)

  • Repair commit: ba7915452e17b5b7d93a2bb8bddefe534cbfb6bb — "fix(capability-map): restore reviewer role on 10 review tasks (break-glass, incident #722)", parent 293808b42d54c4736f8fb55f71b10d93a40ae4ea.
  • Changed files (exactly two): task_capability_map.py (10 role restorations to reviewer; verified byte-identical to audited blob 02826afcb4b3302dd5e7e00135e5951eca351a7a; adopt_merger_pr_lease kept merger; merge_pr untouched) and new tests/test_task_capability_role_invariants.py (profile-coverage invariant, map/router agreement, merger-only adopt/merge, merger cannot hold review permissions).
  • Test evidence in the repair worktree before push: focused role-mapping suites 72 passing (+22 subtests); full suite 2900 passing, 6 skipped, 1 known Starlette deprecation warning, 195 subtests.
  • Push executed by the operator directly (293808b..ba79154 HEAD -> master) after the session's production-push guard correctly refused an agent-side push; live prgs/master readback confirms ba79154. Control checkout fast-forwarded to the same SHA, working tree clean.

2. Reviewer namespace restart and parity

  • Stale reviewer daemons (17 processes, all started before the repair) terminated by the operator; fresh gitea-reviewer daemon pid 14647 spawned at reconnect with operator-set GITEA_MCP_SESSION_STATE_TTL_HOURS=24 (required because the durable decision-lock TTL lapsed at 2026-07-16T23:57:56Z during recovery; TTL-scoped configuration only, no permission or role change).
  • Runtime readback: master_parity.in_parity=true at ba79154; review_pr reports required_role_kind=reviewer, allowed_in_current_session=true.
  • gitea_resolve_task_capability(review_pr) on prgs-reviewer: allowed_in_current_session=true, matching_configured_profile=[prgs-reviewer, mdcps-reviewer], no stop — the #722 lockout is repaired end-to-end.

3. PR #721 recovery

  • Reviewer lease re-acquired at head 80f59b334e6671b08006725292c08a8e8b6c823f (comment 11925, session 14647-427dc88f7b0b).
  • Formal review recorded via native gitea_submit_pr_review: review_id 445, APPROVE verdict at head 80f59b3, readback review_verdict_visible=true, approval_visible=true, approval_at_current_head=true, not dismissed, not stale, zero quarantine flags.
  • PR #721 final state: open, head 80f59b3, mergeable, carrying the APPROVE review at current head. No merge performed (requires separate operator authorization).

4. Residual defects and open items

  • gitea_release_reviewer_pr_lease failed twice with generic internal_error (non-retryable) after the verdict; lease 11925 remains phase=claimed and self-expires 2026-07-17T02:06:18Z. Same generic-internal_error family as #718; also in scope for #723 AC4 (structured fail-closed reasons instead of raised exceptions). No further release attempts were made (no fallback improvisation).
  • The gitea-tools author namespace dropped its transport during a #685-style auto-restart while this audit comment was being prepared; this comment is therefore posted from the prgs-reviewer namespace, whose profile explicitly allows gitea.issue.comment.
  • #723 tracks the code defects that allowed this incident (map/router/runtime-context drift, denied-resolve role stamp, exception-to-internal_error conversion).
  • Local artifacts retained for audit: worktree branches/incident-970e68b-breakglass (repair source, now landed on master) and branches/review-pr-721-80f59b3 (review pin); ~/.claude.json.bak-722 config backup.

[THREAD STATE LEDGER]

what is true now: master carries the authorized repair at ba79154 with live parity; resolve(review_pr) allows prgs-reviewer; PR #721 remains in open state at head 80f59b3 with APPROVE review 445 recorded at current head; no merge has occurred.

what changed: the 10 regressive role mappings were restored on master with new invariant tests; the reviewer namespace was restarted onto repaired code; the stranded PR #721 verdict became a formal native review.

what is blocked: reviewer lease 11925 cleanup — the release tool returns internal_error; the lease self-expires at 2026-07-17T02:06:18Z or is superseded by merger adoption. PR #721 merge awaits separate operator authorization.

who/what acts next: merger session (with explicit operator authorization) adopts the PR #721 lease and merges; #723 fixes proceed through the normal reviewed-PR path, now functional again.

Server-side decision state: APPROVE review 445 at head 80f59b3 on open PR #721; incident #722 recovery steps complete; #723 open for code defects.
Local verdict/state: break-glass repair landed and verified; audit trail recorded; residual lease-release tool defect documented.
Next actor: merger (separate session and authorization) for PR #721; author/reviewer for #723.
Required action: merger adoption + merge of PR #721 when authorized; then #720 closes via merge; #722 may close after post-incident review.
Blocker classification: queue/lease blocker
Do not do: no merge of PR #721 without explicit operator authorization; no further break-glass mutations under the consumed one-time authorization; no manual lease-comment editing.

Canonical Issue State

STATE: breakglass_recovery_complete
WHO_IS_NEXT: merger
NATIVE_REVIEW_PROOF: APPROVE review 445 on PR #721 at head 80f59b334e was recorded by the live native gitea-reviewer MCP daemon (native_mcp transport, pid 14647, lease session 14647-427dc88f7b0b, lease comment 11925) via gitea_submit_pr_review with final_review_decision_ready consuming the durable decision lock; readback review_verdict_visible=true and approval_at_current_head=true; no offline, import, or script fallback path was used
NEXT_ACTION: Under separate operator authorization, merger adopts the PR #721 lease and merges at head 80f59b3; afterwards close out #722 following post-incident review
NEXT_PROMPT:

Merger adoption for PR #721 (prgs / Scaled-Tech-Consulting/Gitea-Tools).
Expected head: 80f59b334e6671b08006725292c08a8e8b6c823f
APPROVE review 445 recorded at this head (native, review_verdict_visible=true).
Preflight: whoami → resolve merge_pr → adopt_merger_pr_lease (lease 11925, session 14647-427dc88f7b0b) → verify review 445 at head → merge.
Requires explicit operator authorization; do not merge without it.

WHAT_HAPPENED: Operator-authorized minimum break-glass repair landed as ba79154 (capability map to blob 02826af + invariant tests, suites green), reviewer namespace restarted in parity, resolve(review_pr) allows prgs-reviewer again, and PR #721 received formal APPROVE review 445 at head 80f59b3 via the native reviewer path.
WHY: The 970e68b regression had removed every configured profile's ability to formally review, so a normal reviewed fix PR was impossible; the operator authorized the audited one-time direct repair, which restored the review path that then recovered PR #721.
RELATED_PRS: #721, #720, #723, master ba79154, prior master 293808b4, regressive commit 970e68b
BLOCKERS: lease 11925 release tool defect (internal_error; self-expires 2026-07-17T02:06:18Z); PR #721 merge pending separate operator authorization
VALIDATION: repair worktree suites 72 focused (+22 subtests) and 2900 full (6 skipped, 1 known warning, 195 subtests) passing; live prgs/master readback ba79154; runtime in_parity; resolve(review_pr) allowed with matching profiles [prgs-reviewer, mdcps-reviewer]; review 445 readback approval_at_current_head=true
LAST_UPDATED_BY: controller session (operator-authorized break-glass), posted via prgs-reviewer / sysadmin

## Break-glass recovery audit trail — incident #722 (complete) **Authorization:** operator one-time break-glass authorization recorded in comment 11918 (pre-repair capture: master `293808b4`, proposed diff to blob `02826af`). ### 1. Master repair (operator-authorized, minimum scope) - Repair commit: `ba7915452e17b5b7d93a2bb8bddefe534cbfb6bb` — "fix(capability-map): restore reviewer role on 10 review tasks (break-glass, incident #722)", parent `293808b42d54c4736f8fb55f71b10d93a40ae4ea`. - Changed files (exactly two): `task_capability_map.py` (10 role restorations to reviewer; verified byte-identical to audited blob `02826afcb4b3302dd5e7e00135e5951eca351a7a`; `adopt_merger_pr_lease` kept merger; `merge_pr` untouched) and new `tests/test_task_capability_role_invariants.py` (profile-coverage invariant, map/router agreement, merger-only adopt/merge, merger cannot hold review permissions). - Test evidence in the repair worktree before push: focused role-mapping suites 72 passing (+22 subtests); full suite 2900 passing, 6 skipped, 1 known Starlette deprecation warning, 195 subtests. - Push executed by the operator directly (`293808b..ba79154 HEAD -> master`) after the session's production-push guard correctly refused an agent-side push; live `prgs/master` readback confirms `ba79154`. Control checkout fast-forwarded to the same SHA, working tree clean. ### 2. Reviewer namespace restart and parity - Stale reviewer daemons (17 processes, all started before the repair) terminated by the operator; fresh gitea-reviewer daemon pid 14647 spawned at reconnect with operator-set `GITEA_MCP_SESSION_STATE_TTL_HOURS=24` (required because the durable decision-lock TTL lapsed at 2026-07-16T23:57:56Z during recovery; TTL-scoped configuration only, no permission or role change). - Runtime readback: `master_parity.in_parity=true` at `ba79154`; `review_pr` reports `required_role_kind=reviewer`, `allowed_in_current_session=true`. - `gitea_resolve_task_capability(review_pr)` on prgs-reviewer: `allowed_in_current_session=true`, `matching_configured_profile=[prgs-reviewer, mdcps-reviewer]`, no stop — the #722 lockout is repaired end-to-end. ### 3. PR #721 recovery - Reviewer lease re-acquired at head `80f59b334e6671b08006725292c08a8e8b6c823f` (comment 11925, session 14647-427dc88f7b0b). - Formal review recorded via native `gitea_submit_pr_review`: **review_id 445, APPROVE verdict at head 80f59b3**, readback `review_verdict_visible=true`, `approval_visible=true`, `approval_at_current_head=true`, not dismissed, not stale, zero quarantine flags. - PR #721 final state: **open**, head `80f59b3`, mergeable, carrying the APPROVE review at current head. **No merge performed** (requires separate operator authorization). ### 4. Residual defects and open items - `gitea_release_reviewer_pr_lease` failed twice with generic `internal_error` (non-retryable) after the verdict; lease 11925 remains phase=claimed and self-expires 2026-07-17T02:06:18Z. Same generic-`internal_error` family as #718; also in scope for #723 AC4 (structured fail-closed reasons instead of raised exceptions). No further release attempts were made (no fallback improvisation). - The gitea-tools author namespace dropped its transport during a #685-style auto-restart while this audit comment was being prepared; this comment is therefore posted from the prgs-reviewer namespace, whose profile explicitly allows gitea.issue.comment. - #723 tracks the code defects that allowed this incident (map/router/runtime-context drift, denied-resolve role stamp, exception-to-internal_error conversion). - Local artifacts retained for audit: worktree `branches/incident-970e68b-breakglass` (repair source, now landed on master) and `branches/review-pr-721-80f59b3` (review pin); `~/.claude.json.bak-722` config backup. [THREAD STATE LEDGER] what is true now: master carries the authorized repair at ba79154 with live parity; resolve(review_pr) allows prgs-reviewer; PR #721 remains in open state at head 80f59b3 with APPROVE review 445 recorded at current head; no merge has occurred. what changed: the 10 regressive role mappings were restored on master with new invariant tests; the reviewer namespace was restarted onto repaired code; the stranded PR #721 verdict became a formal native review. what is blocked: reviewer lease 11925 cleanup — the release tool returns internal_error; the lease self-expires at 2026-07-17T02:06:18Z or is superseded by merger adoption. PR #721 merge awaits separate operator authorization. who/what acts next: merger session (with explicit operator authorization) adopts the PR #721 lease and merges; #723 fixes proceed through the normal reviewed-PR path, now functional again. Server-side decision state: APPROVE review 445 at head 80f59b3 on open PR #721; incident #722 recovery steps complete; #723 open for code defects. Local verdict/state: break-glass repair landed and verified; audit trail recorded; residual lease-release tool defect documented. Next actor: merger (separate session and authorization) for PR #721; author/reviewer for #723. Required action: merger adoption + merge of PR #721 when authorized; then #720 closes via merge; #722 may close after post-incident review. Blocker classification: queue/lease blocker Do not do: no merge of PR #721 without explicit operator authorization; no further break-glass mutations under the consumed one-time authorization; no manual lease-comment editing. ## Canonical Issue State STATE: breakglass_recovery_complete WHO_IS_NEXT: merger NATIVE_REVIEW_PROOF: APPROVE review 445 on PR #721 at head 80f59b334e6671b08006725292c08a8e8b6c823f was recorded by the live native gitea-reviewer MCP daemon (native_mcp transport, pid 14647, lease session 14647-427dc88f7b0b, lease comment 11925) via gitea_submit_pr_review with final_review_decision_ready consuming the durable decision lock; readback review_verdict_visible=true and approval_at_current_head=true; no offline, import, or script fallback path was used NEXT_ACTION: Under separate operator authorization, merger adopts the PR #721 lease and merges at head 80f59b3; afterwards close out #722 following post-incident review NEXT_PROMPT: ```text Merger adoption for PR #721 (prgs / Scaled-Tech-Consulting/Gitea-Tools). Expected head: 80f59b334e6671b08006725292c08a8e8b6c823f APPROVE review 445 recorded at this head (native, review_verdict_visible=true). Preflight: whoami → resolve merge_pr → adopt_merger_pr_lease (lease 11925, session 14647-427dc88f7b0b) → verify review 445 at head → merge. Requires explicit operator authorization; do not merge without it. ``` WHAT_HAPPENED: Operator-authorized minimum break-glass repair landed as ba79154 (capability map to blob 02826af + invariant tests, suites green), reviewer namespace restarted in parity, resolve(review_pr) allows prgs-reviewer again, and PR #721 received formal APPROVE review 445 at head 80f59b3 via the native reviewer path. WHY: The 970e68b regression had removed every configured profile's ability to formally review, so a normal reviewed fix PR was impossible; the operator authorized the audited one-time direct repair, which restored the review path that then recovered PR #721. RELATED_PRS: #721, #720, #723, master ba79154, prior master 293808b4, regressive commit 970e68b BLOCKERS: lease 11925 release tool defect (internal_error; self-expires 2026-07-17T02:06:18Z); PR #721 merge pending separate operator authorization VALIDATION: repair worktree suites 72 focused (+22 subtests) and 2900 full (6 skipped, 1 known warning, 195 subtests) passing; live prgs/master readback ba79154; runtime in_parity; resolve(review_pr) allowed with matching profiles [prgs-reviewer, mdcps-reviewer]; review 445 readback approval_at_current_head=true LAST_UPDATED_BY: controller session (operator-authorized break-glass), posted via prgs-reviewer / sysadmin
sysadmin commented 2026-07-16 20:09:04 -05:00
Owner
Copy Link
Copy Source

Final merge audit — PR #721 landed (incident #722 recovery closed out)

Merge evidence (all readback-verified):

  • merge result: PR #721 merged via 'merge' — performed: true, all gates passed, no force/bypass options exist on the tool.
  • Authorized head honored exactly: 80f59b334e6671b08006725292c08a8e8b6c823f (head pinned via expected_head_sha; live head matched at merge time).
  • Formal review: ID 445, APPROVE at that head, approval_visible=true, approval_at_current_head=true, zero quarantined reviews, no change-request blocks.
  • Sanctioned lease chain: reviewer lease comment 11925 → merger adoption proof comment 11940 (adoption_reason: merger-handoff-approved-head, adopted_from_session 14647-427dc88f7b0b → merger session 60151-ffe26336620a).
  • Merge commit: 67e4a2b5e96b680a1dcb90a20c2ebf87357a3b1f, landed on master at 2026-07-17T00:48:07Z.
  • Live prgs/master readback: 67e4a2b5e96b…; control checkout fast-forwarded to the same SHA with a clean working tree (history: 67e4a2b ← ba79154 ← 293808b).
  • Issue #720 auto-closed by the merge (Closes-#720 in the PR); its status:pr-open label was deliberately left untouched per operator scope.
  • Post-merge merger namespace re-verified in parity at 67e4a2b5e96b before this comment.

Final lease disposition: adopted lease (comment 11940, session 60151-ffe26336620a) is moot — PR merged and linked issue closed, confirmed by the read-only cleanup assessment (lease_moot=true, cleanup_allowed=true). The apply step of gitea_cleanup_post_merge_moot_lease failed twice with generic internal_error (same defect family as #718 and #723 AC4, matching the earlier gitea_release_reviewer_pr_lease failures), so no released marker could be posted; the lease self-expires at 2026-07-17T02:44:39Z with no downstream effect. No fallback was improvised.

[THREAD STATE LEDGER]

what is true now: PR #721 landed on master as merge commit 67e4a2b5e9 at the authorized head; issue #720 is in closed state; live master, control checkout, and the merger runtime all agree at 67e4a2b5e96b; the #722 break-glass recovery chain is fully executed.

what changed: the merge landed and #720 auto-closed; the adopted lease became moot; the incident's recovery objectives (repair, review path restoration, PR #721 verdict, merge) are all complete.

what is blocked: only the cosmetic moot-lease released marker — the cleanup tool returns internal_error; the lease self-expires at 2026-07-17T02:44:39Z.

who/what acts next: controller/operator may close #722 after post-incident review; author/reviewer proceed with #723 fixes through the normal reviewed-PR path.

Server-side decision state: PR #721 closed with merge commit 67e4a2b5e96b; review 445 APPROVE preserved at 80f59b3; issues #720 closed, #722 and #723 open.
Local verdict/state: control checkout clean at 67e4a2b5e96b; no pending mutations; all evidence recorded in comments 11918, 11935, and this comment.
Next actor: operator/controller for #722 closure; author/reviewer for #723.
Required action: none for PR #721; optional #722 closure after post-incident review; #723 remediation via normal workflow.
Blocker classification: queue/lease blocker
Do not do: no re-merge or further mutations for PR #721; no manual lease-comment edits; no label changes; no additional break-glass actions — the one-time authorization is fully consumed.

Canonical Issue State

STATE: pr721_landed_recovery_closed
WHO_IS_NEXT: controller
NEXT_ACTION: Close #722 after post-incident review; pursue #723 code fixes through the now-functional normal reviewed-PR path
NEXT_PROMPT:

Post-incident closeout for #722 (prgs / Scaled-Tech-Consulting/Gitea-Tools).
Verify: master 67e4a2b5e96b contains ba79154 (capability repair) and the PR #721 merge; review 445 APPROVE preserved at 80f59b3; #720 closed.
Then close #722, and schedule #723 (map/router invariants, denied-resolve role stamp, internal_error conversion) as normal reviewed work.

WHAT_HAPPENED: The separately authorized merger session adopted lease 11925 via sanctioned adoption (comment 11940) and merged PR #721 at the authorized head; merge commit 67e4a2b5e9 verified on live master; #720 auto-closed; control checkout advanced clean.
WHY: Operator granted separate explicit merge authorization after the formal APPROVE review 445 landed at the pinned head, completing the #722 recovery sequence end to end through sanctioned merger tooling only.
RELATED_PRS: #721, #720, #723, merge commit 67e4a2b5e9, repair commit ba79154
BLOCKERS: none affecting outcomes; cosmetic moot-lease marker blocked by cleanup-tool internal_error (self-resolves at lease expiry 2026-07-17T02:44:39Z; defect tracked via #718/#723)
VALIDATION: merge tool readback performed=true with gates listed; live prgs/master readback 67e4a2b5e96b; PR view readback state closed with merge_commit_sha 67e4a2b5e9 at 2026-07-17T00:48:07Z; #720 view readback closed; control checkout clean fast-forward verified
LAST_UPDATED_BY: controller session (separately authorized merge), merger namespace prgs-merger / sysadmin

## Final merge audit — PR #721 landed (incident #722 recovery closed out) **Merge evidence (all readback-verified):** - merge result: PR #721 merged via 'merge' — `performed: true`, all gates passed, no force/bypass options exist on the tool. - Authorized head honored exactly: `80f59b334e6671b08006725292c08a8e8b6c823f` (head pinned via `expected_head_sha`; live head matched at merge time). - Formal review: **ID 445**, APPROVE at that head, `approval_visible=true`, `approval_at_current_head=true`, zero quarantined reviews, no change-request blocks. - Sanctioned lease chain: reviewer lease comment 11925 → merger adoption proof comment **11940** (`adoption_reason: merger-handoff-approved-head`, adopted_from_session 14647-427dc88f7b0b → merger session 60151-ffe26336620a). - Merge commit: **`67e4a2b5e96b680a1dcb90a20c2ebf87357a3b1f`**, landed on master at 2026-07-17T00:48:07Z. - Live `prgs/master` readback: `67e4a2b5e96b…`; control checkout fast-forwarded to the same SHA with a clean working tree (history: 67e4a2b ← ba79154 ← 293808b). - Issue **#720 auto-closed** by the merge (Closes-#720 in the PR); its `status:pr-open` label was deliberately left untouched per operator scope. - Post-merge merger namespace re-verified in parity at `67e4a2b5e96b` before this comment. **Final lease disposition:** adopted lease (comment 11940, session 60151-ffe26336620a) is moot — PR merged and linked issue closed, confirmed by the read-only cleanup assessment (`lease_moot=true`, `cleanup_allowed=true`). The apply step of `gitea_cleanup_post_merge_moot_lease` failed twice with generic `internal_error` (same defect family as #718 and #723 AC4, matching the earlier `gitea_release_reviewer_pr_lease` failures), so no released marker could be posted; the lease self-expires at 2026-07-17T02:44:39Z with no downstream effect. No fallback was improvised. [THREAD STATE LEDGER] what is true now: PR #721 landed on master as merge commit 67e4a2b5e96b at the authorized head; issue #720 is in closed state; live master, control checkout, and the merger runtime all agree at 67e4a2b5e96b; the #722 break-glass recovery chain is fully executed. what changed: the merge landed and #720 auto-closed; the adopted lease became moot; the incident's recovery objectives (repair, review path restoration, PR #721 verdict, merge) are all complete. what is blocked: only the cosmetic moot-lease released marker — the cleanup tool returns internal_error; the lease self-expires at 2026-07-17T02:44:39Z. who/what acts next: controller/operator may close #722 after post-incident review; author/reviewer proceed with #723 fixes through the normal reviewed-PR path. Server-side decision state: PR #721 closed with merge commit 67e4a2b5e96b; review 445 APPROVE preserved at 80f59b3; issues #720 closed, #722 and #723 open. Local verdict/state: control checkout clean at 67e4a2b5e96b; no pending mutations; all evidence recorded in comments 11918, 11935, and this comment. Next actor: operator/controller for #722 closure; author/reviewer for #723. Required action: none for PR #721; optional #722 closure after post-incident review; #723 remediation via normal workflow. Blocker classification: queue/lease blocker Do not do: no re-merge or further mutations for PR #721; no manual lease-comment edits; no label changes; no additional break-glass actions — the one-time authorization is fully consumed. ## Canonical Issue State STATE: pr721_landed_recovery_closed WHO_IS_NEXT: controller NEXT_ACTION: Close #722 after post-incident review; pursue #723 code fixes through the now-functional normal reviewed-PR path NEXT_PROMPT: ```text Post-incident closeout for #722 (prgs / Scaled-Tech-Consulting/Gitea-Tools). Verify: master 67e4a2b5e96b contains ba79154 (capability repair) and the PR #721 merge; review 445 APPROVE preserved at 80f59b3; #720 closed. Then close #722, and schedule #723 (map/router invariants, denied-resolve role stamp, internal_error conversion) as normal reviewed work. ``` WHAT_HAPPENED: The separately authorized merger session adopted lease 11925 via sanctioned adoption (comment 11940) and merged PR #721 at the authorized head; merge commit 67e4a2b5e96b verified on live master; #720 auto-closed; control checkout advanced clean. WHY: Operator granted separate explicit merge authorization after the formal APPROVE review 445 landed at the pinned head, completing the #722 recovery sequence end to end through sanctioned merger tooling only. RELATED_PRS: #721, #720, #723, merge commit 67e4a2b5e96b, repair commit ba79154 BLOCKERS: none affecting outcomes; cosmetic moot-lease marker blocked by cleanup-tool internal_error (self-resolves at lease expiry 2026-07-17T02:44:39Z; defect tracked via #718/#723) VALIDATION: merge tool readback performed=true with gates listed; live prgs/master readback 67e4a2b5e96b; PR view readback state closed with merge_commit_sha 67e4a2b5e96b at 2026-07-17T00:48:07Z; #720 view readback closed; control checkout clean fast-forward verified LAST_UPDATED_BY: controller session (separately authorized merge), merger namespace prgs-merger / sysadmin
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels contamination incident protected-branch status:in-progress type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#722