EXCEPTIONAL OPERATOR-AUTHORIZED BREAK-GLASS RECOVERY - incident #722.
Commit 970e68b ("fix: adopt_merger_pr_lease requires merger role") was an
unreviewed direct push that flipped 11 task_capability_map.py entries to
role="merger" instead of the intended 1. Merger profiles forbid the review
permissions, so no configured profile could resolve review_pr / approve_pr /
request_changes_pr (matching_configured_profile was empty repository-wide)
and no fix PR could be formally reviewed. The operator resolved the catch-22
with a one-time break-glass authorization recorded on issue #722
(comment 11918); this commit is the authorized minimum and nothing more.
Changes:
- task_capability_map.py restored to blob 02826af (the preserved intended
correction, local commit c071f8a1): the 10 regressive entries return to
role="reviewer"; adopt_merger_pr_lease keeps role="merger"; merge_pr is
untouched.
- tests/test_task_capability_role_invariants.py added (#723 AC1/AC2):
profile-coverage invariant for formal review tasks, map/router agreement,
merger-only adopt_merger_pr_lease and merge_pr, merger profiles cannot
hold review permissions.
Validation: focused role-mapping tests 72 passed (+22 subtests); full suite
2900 passed, 6 skipped, 1 known warning, 195 subtests.
Recovery step 1 for incident #722; code-defect follow-ups tracked in #723.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EnHNSVQvJ8nCk7KZ9kL4Ym
Terminal review-decision ledgers are recovery-critical provenance, not
disposable session cache. A generic four-hour TTL previously made
fresh_review_on_current_head_allowed unreachable after age expiry on open
PRs (PR #616 / review 443 reproduction).
- Classify KIND_DECISION_LOCK as RECOVERY_CRITICAL so load/mark_final work
after >4h without hand-editing session-state files
- Stamp kind + recovery_critical on save for compatibility
- Add inspect_state_envelope so assessment reports on-disk evidence instead
of silent "no lock" when TTL would hide non-critical kinds
- Surface disk_inspect on stale decision-lock cleanup assessment
Preserves same-head #332 hard-stop, #620 head-scoped fresh review, #594
moot cleanup, and #709 irrecoverable authorization (no ordinary-profile
permission grant).
Require branch-scoped post-delete not-found (not generic/repo/host 404),
emit consistent top-level cleanup fields, fail closed on ownership inventory
errors, never auto-reclaim expired control-plane leases, include active
comment-backed reviewer leases, apply the same gates to reconcile_merged_cleanups,
and match ownership with normalized host identity.
Require authoritative not-found readback after merged-PR branch DELETE, and
block cleanup when active author/reviewer/merger/controller/reconciler
session, lease, or worktree-binding ownership still uses the target branch.
Address formal review 438 REQUEST_CHANGES on PR #710 (F6/F7/F8).
F6 — HMAC key-version validation fails closed:
- verify_authorization_artifact validates key_version BEFORE any MAC work, so an
attacker-chosen version can never select the signing key.
- Require exactly one nonempty, well-formed key-version field; missing, empty,
unknown, malformed, duplicated (including identical-valued and nested
aliases), and mismatched versions all fail. No versionless legacy fallback.
- Artifact version must equal the configured active version; production now
requires GITEA_IRRECOVERABLE_AUTH_HMAC_KEY_VERSION explicitly (an implicit
default made rotation ambiguous). Version stays inside the signed material.
F7 — strictly canonical incident evidence:
- Replace substring/first-match parsing with an exact schema: marker on line 1,
every field once, fixed order, no duplicate/unknown/empty/conflicting fields
in or outside the signed block. The parsed body is re-rendered and compared
for exact equality before acceptance.
- content_digest now binds the full recovery scope: repository identity, PR,
decision-lock identity, destroyed subject, recovery action, recorded and
expected head, incident issue, evidence author, minting actor, key version,
nonce and issued_at.
- Actor identity is the immutable user id with login consistency; conflicting
ids/logins and display-name-only identities fail closed. Edited comments are
rejected. The independent-author rule is preserved and enforced by stable id.
- build_canonical_incident_body is the single source of the accepted format and
refuses to emit ambiguous evidence.
F8 — archival is a prerequisite for clearing terminal evidence:
- _clear_decision_lock_for_profile no longer swallows archive failures. It
requires a successful write plus a durable read-back matching the PR/head,
and otherwise returns a structured, retry-safe failure that retains the lock
and records actionable recovery evidence.
- Fixes a latent bug the read-back exposed: the archive payload inherited the
source lock's session_profile_lock, so save_state keyed the archive under the
reviewer profile instead of the archive identity and it never read back.
Adversarial regressions added for every listed case: key-version missing/empty/
unknown/malformed/duplicate/rotation/wrong-key-after-restart, reordered fields,
duplicate identical and conflicting fields, conflicting actor ids/names,
digest-preserving substitution, decision-lock and recovery-action substitution,
cross-PR/repo/org/remote/head replay, archive exception/timeout/false/empty/
partial-readback with proof the lock survives, exactly-one permitted clear, and
retry after archive failure. No existing assertion was weakened.
Validation: focused 150 passed in three module orders; full tests/ 2809 passed,
6 skipped, 1 warning (pre-existing StarletteDeprecationWarning in
tests/test_webui_audit.py:8), 161 subtests passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prevent merger-local empty decision locks from standing in for reviewer
terminal cleanup, refuse silent re-init overwrite of unresolved terminal
evidence, record post-merge recovery-required state when audit fails, and
add a truthful irrecoverable-provenance path that never claims applied=true.
Closes#709
Remove the public allow_test_bootstrap production seam so caller-controlled
flags cannot forge native mutation provenance. Bind provenance to the resolved
canonical entrypoint path plus a live stdio transport bind; basename-only
mcp_server.py stack frames and import-only launch no longer authorize
mutations. Test-mode install_test_native_runtime is pytest-only and cannot
reach production Gitea mutation endpoints. Add AC9 regressions for both
reviewer-found bypasses and related spoof vectors.
Refs: PR #696 REQUEST_CHANGES at 253269c; issue #695 comments 11002/11005.
Bind mutation/credential paths to a process-local native MCP runtime so env
spoofing, direct imports, and offline helpers cannot reconstruct session gates
after native transport failure. Quarantine contaminated formal reviews under
controller authority and honor quarantine in review feedback, merge eligibility,
merge mutation, and canonical handoff validation. Add regression coverage for
the second (PR #694 / review 427) incident class.
Fixes the final-report validator defects from Issue #698 (original lead
plus the independent reproduction recorded during the PR #703 formal
review, comment 11246):
- Legacy fields: the review/merger required-field tables no longer demand
'Pinned reviewed head', 'Scratch worktree used', 'Worktree path',
'Worktree dirty', 'Mutations', 'Next', 'Issue/PR', 'Branch/SHA', or
'Files changed' — names the canonical review-merge-final-report schema
forbids or replaces. The canonical names ('Reviewed head SHA', 'Review
worktree path/dirty', 'Safe next action', mutation categories) are
required instead, with backward-compatible aliases where the schema
permits them.
- Structured proof: workflow-load helper results are recognized in
colon, key=value, and JSON renderings; validation pass proof is
accepted anywhere in the Validation field value (for example
'focused 50 passed; full 2665 passed').
- Mutation inference: review mutations are inferred only from
authoritative evidence (performed=true and not gated); read-only
diagnostics and pre-API rejections no longer count as mutations.
- Lease release vs cleanup: canonical reviewer lease release (release
tool call or terminal phase=released marker) is lease lifecycle, not
post-merge cleanup, and no longer triggers the branch/worktree cleanup
checklist; genuine delete/remove claims still require full proof.
- Blocked reports: a legitimately blocked run that states an explicit
'Reviewed/Candidate head SHA: none' with no verdict, merge, or started
validation owes no head proofs; approval-time and merge-time live-head
proofs are demanded only once the corresponding phase begins, and a
report that states no head at all still fails closed.
- action_log robustness: malformed (non-dict) entries and non-list logs
are reported as clear sanitized findings (position and type only, no
content echo) instead of crashing with AttributeError; a defective
validator rule now fails closed with a sanitized block finding rather
than raising a secondary exception.
27 new regression tests, including canonical fixtures modeled on the
PR #703 formal-review handoff and the blocked preflight report from the
prior #698 reproductions. Two legacy-field test fixtures updated to the
canonical schema. Full suite: 2663 passed, 6 skipped, 161 subtests.
Co-Authored-By: Claude Fable 5 <[email protected]>
Fixes the six defects from the formal REQUEST_CHANGES review at
889931d553:
- F1: run sanctioned stale-binding recovery in
gitea_resolve_task_capability BEFORE the terminal launcher probe, so a
worktree removed mid-session can no longer wedge mutation resolution on
'missing cwd' before recovery runs. The fail-closed probe block now also
carries the binding classification evidence.
- F2: _resolve_preflight_workspace_path resolves with the same
verify_paths existence checks as the canonical mutation context, and
_get_workspace_porcelain returns a synthetic tracked-dirty sentinel when
the workspace is missing or git fails — an uninspectable workspace can
never read as clean/empty porcelain.
- F3: the orphaned_expired_superseded_head cleanup matrix now requires
affirmative safe worktree evidence (worktree_clean=true or
worktree_exists=false), aligned with the sibling orphaned_owner_missing
gate and the diagnosis path; unknown evidence fails closed.
- F4: reviewer-session-lease shadows and stale-binding audit records are
recovery-critical kinds exempt from the 4h session-state TTL; they
persist until a sanctioned clear terminally reconciles them.
- F5: session-lease shadows are keyed by lease session id (collision-safe
identity) with a list_states enumeration API; concurrent same-profile
sessions can no longer overwrite or misattribute each other's crash
evidence. Legacy single-slot records remain readable.
- F6: the durable audit record is persisted (status=pending) BEFORE the
environment binding is cleared; if audit persistence fails the clear
does not happen and the failure is reported explicitly.
30 new regression tests cover deleted-worktree recovery ordering, missing/
deleted/symlinked/mid-evaluation path changes, unknown-evidence cleanup,
TTL boundary (before/at/after), interleaved and reconnected concurrent
sessions, and audit-write failure/retry/idempotence/ordering.
Issue #704 dotenv load-path prevention is intentionally NOT included.
Full suite: 2695 passed, 6 skipped, 161 subtests passed.
Co-Authored-By: Claude Fable 5 <[email protected]>
An unhandled daemon crash skips teardown: the durable reviewer lease
comment survives, the in-memory session lease dies, and the
auto-reconnected daemon inherits a stale GITEA_ACTIVE_WORKTREE from its
parent environment (PR #701 / branches/review-pr-654 incident).
AC2 — safe clear/re-bind of stale bindings:
- new stale_binding_recovery.py: classifies the active env binding
(corroborated / unverified_inherited / provably_stale_missing_path /
superseded_by_session_lease) and produces a fail-closed recovery plan;
only provably stale bindings are clear-eligible
- gitea_resolve_task_capability and daemon boot run the sanctioned
recovery (durable audit via session state); runtime context surfaces
the classification read-only
- namespace_workspace_binding.resolve_namespace_workspace(verify_paths=…)
demotes env-bound worktrees whose path no longer exists; mutation and
runtime-context resolution always verify
AC3 — recoverable lease cleanup after unexpected exit:
- durable session-lease shadow (KIND_REVIEWER_SESSION_LEASE) written on
sanctioned record/heartbeat, removed on sanctioned clear; a crash
leaves provable orphan evidence (owner pid + session id)
- assess_crashed_session_lease_orphan derives tri-state
owner_process_alive: only a dead recorded owner pid proves exit; PID
liveness is never ownership proof
- diagnose/cleanup wrappers consume the shadow instead of passing
owner_process_alive=None
- new classification orphaned_expired_superseded_head: an expired
superseded-head lease with no terminal review stops being a permanent
ambiguous/wait; post-expiry it unlocks fresh acquisition, and guarded
cleanup becomes eligible only with owner-exit evidence + clean/absent
worktree + controller authorization + confirmation. Pre-expiry
behavior is unchanged (fail-closed wait, no steal).
Validation: tests/test_issue_702_stale_binding_lease_recovery.py (29
tests covering lost in-session leases, old-head leases, runtime/worktree
mismatch, managed reconnect, safe expiry); full suite 2665 passed,
6 skipped, 161 subtests.
Co-Authored-By: Claude Fable 5 <[email protected]>
Add fail-closed assessment and MCP tool for expired/superseded-head
comment-backed reviewer leases that lack a control-plane lease_id.
- assess_obsolete_reviewer_comment_lease_cleanup eligibility matrix
- diagnose classifications: foreign_active_current_head,
foreign_expired_current_head, foreign_completed_superseded_head,
foreign_expired_superseded_head, orphaned_owner_missing,
ambiguous_conflicting_evidence
- gitea_cleanup_obsolete_reviewer_comment_lease with confirmation
CLEANUP OBSOLETE REVIEWER LEASE <n> and controller_recovery_authorized
- preserve audit history via append-only phase=released marker
- never repoint, transfer validation, use PID ownership, or steal
active current-head foreign leases
- regression for PR #688 lease comment 10749 after recorded expiry
Merged-PR source-branch cleanup is reconciler work (task_capability_map maps
cleanup_merged_pr_branch -> reconciler / gitea.branch.delete), but the
reconciler profile schema, execution-profile docs, and tests never covered
the permission, so no configured profile could run the guarded
gitea_cleanup_merged_pr_branch path.
- reconciler_profile.py: add gitea.branch.delete to
RECONCILER_RECOMMENDED_OPERATIONS (not required; not forbidden)
- docs/gitea-execution-profiles.md: document merged-branch cleanup
ownership, least-privilege constraints, and the no-alias caveat
- tests/test_reconciler_profile.py: reconciler profile with branch.delete
stays valid and classified reconciler; missing grant is reported as
missing-recommended
- tests/test_branch_cleanup_guard.py: author- and merger-shaped profiles
without gitea.branch.delete fail closed on gitea_cleanup_merged_pr_branch
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prevention hardening for the #670 incident (bare direct-to-master commit
2fa97c26 and the PR #654 merger stable-branch push attempt). Worker sessions
must never publish stable branches directly; stable updates land only through
sanctioned Gitea merge tooling or an authorized reconciler path.
New pure module stable_branch_push_guard.py:
- classify_push_command: detects git push <remote> <stable-ref> equivalents
including refspecs (HEAD:<ref>, +refs/heads/x:refs/heads/<ref>), --force,
--delete/:<ref>, and --dry-run/-n no-op probes (dry-run still proves intent).
Fetch/pull and feature-branch pushes are never flagged; sanctioned
gitea_merge_pr / API merge is not a push.
- assess_root_checkout_local_commit: detects control-checkout commits not on an
issue feature branch (branches/ worktrees exempt).
- redact_command: strips URL userinfo/token assignments before logging.
- build_contamination_record + assess_contamination_gate: durable marker shape
and fail-closed gate (reconciler-exempt; comment/lock stay allowed for handoff).
Server wiring (gitea_mcp_server.py, mcp_session_state.py):
- KIND_STABLE_BRANCH_CONTAMINATION durable session marker (per profile identity).
- _enforce_stable_branch_contamination_gate wired into verify_preflight_purity
so review/merge/close/completion mutations fail closed while contaminated.
- gitea_record_stable_branch_push_attempt: classify + mark on detection.
- gitea_audit_stable_branch_contamination: reconciler-only inspect/clear; a
worker session can never self-clear.
Tests (AC5): no-op dry-run push, real direct push, sanctioned Gitea merge,
fetch-only, root-checkout local commit, feature-branch push allowed, gate
block/reconciler-exempt, redaction. 51 new tests; full suite 2596 passed.
Docs: llm-project-workflow SKILL.md — universal rule, blocker class, and a
dedicated "Stable Branch Push Protection" section (worker sessions must never
push stable branches directly).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add orthogonal role:* (single-active owner) and hazard:* (additive warning)
lifecycle labels plus status:changes-requested, folding the #603 state:*
vocabulary into the canonical status:* set rather than a conflicting parallel
prefix.
- issue_workflow_labels.py: ROLE_/HAZARD_ specs + frozensets; role/hazard
transition maps and helpers (transition_role_labels, add/clear_hazard_label,
canonical_role_label, canonical_hazard_label, is_discussion,
is_implementation_candidate, requires_blocking_reason); state:* -> status:*
synonym transitions; assess_issue_labels surfaces role/hazard dims and flags
multiple active role labels
- docs/label-taxonomy.md: role, hazard, state->canonical migration, and
allocator cross-check sections
- tests: role/hazard/discussion/blocking-reason/state-synonym coverage
- manage_labels.py seeds the new labels automatically via CANONICAL_LABEL_SPECS
Labels remain advisory; control-plane leases (#601) and live PR state stay the
source of truth for mutations (#603 AC2).
Closes#603
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace single-page GET labels?limit=100 with api_get_all-backed
_repo_label_id_map so later-page labels (e.g. type:feature,
workflow-hardening) are not falsely rejected during full-set replacement.
Also add post-mutation verification, fix related MCP/CLI label inventory
call sites, and add multi-page regression tests for the #601 reconciler
failure mode.
Closes#627
- Update MCP expired-lock recovery test for sticky expired ownership
(live pid + present worktree still fail closed under #601 reclaim rules)
- Remove trailing blank line at EOF in control_plane_db.py
Add phase-1 observability bridge that turns provider observations into
normal Gitea issues and control-plane incident_links rows. Dry-run is
default; apply creates/links through sanctioned Gitea issue paths only.
Raw incidents remain non-assignable; the #600 allocator sees bridge work
only as ordinary Gitea issues. Builds on #613 substrate and #600 allocator.
Closes#612
Add gitea_allocate_next_work and allocator_service routing policy on top of
the #613 ControlPlaneDB substrate. Workers get atomic assign+lease results
(or wait/no_safe_work/terminal-path outcomes) without self-selecting work
via file locks or comment-only leases. #612 remains downstream.
Closes#600
Prior REQUEST_CHANGES on head A no longer blocks formal re-review on
head B of the same open PR. Locks store reviewed head SHA, hard-stop and
mark_ready/submit gates allow a fresh decision boundary when the live
head differs, and same-head duplicates remain fail-closed. Open-PR lock
cleanup stays forbidden (#594); assessment reports locked vs current
head and stale_by_head / fresh_review_on_current_head_allowed.
Closes#620
Legacy NULL-scope dedupe only compared Gitea targets, so duplicate rows
with the same provider key and issue but different fingerprint/status/
event_count/etc. collapsed to the lowest link_id and silently dropped
observation data. Migration now compares all meaningful observation
fields and refuses to discard conflicts (#619 RC3 / #613).
Address remaining PR #619 blockers on head 783ea88:
- Deduplicate legacy NULL-scope incident_links before normalizing to ''
- Fail closed when duplicate rows disagree on Gitea target
- Require non-empty expected_head_sha for PR assign and mutation
Address REQUEST_CHANGES on PR #619:
- require_valid_assignment rejects terminal work states and head drift
- incident_links scope keys normalize NULL/blank to '' for UNIQUE
- remove trailing whitespace in control-plane-db-substrate.md
Add SQLite single-writer MVP control plane per the allocator ADR:
sessions, work_items (issue/pr only), atomic assign+lease, mutation
gates, terminal-lock index, and provider-neutral incident_links.
DB coordinates concurrency; Gitea remains assignable work; raw
Sentry/GlitchTip incidents are never work items. #600 and #612 build on
this substrate.
- Distinguish client_namespace vs offline_spawn probe sources; only IDE
client probes prove namespace health and feed mutation gates.
- Record assessments in session; gate gitea_submit_pr_review and
gitea_merge_pr when client-namespace health is unhealthy/non-proven.
- Mark test_mcp_conn.py offline-only; align recovery docs to client
reconnect (no PID-kill/config-touch as canonical recovery).
- Rebase onto current master so #590 ledger isolation keeps
TestMergePR.test_unknown_profile_blocks green.
The namespace health check (05fdcee) classified a false-ready namespace but
only returned an advisory blocks_merge_workflow flag; nothing in the merge
gate consumed it. Wire it in so a broken live namespace hard-blocks merge.
- review_merge_state_machine.assess_workflow_blockers: add live_namespace_broken
blocker (registered-in-FastMCP but not callable-through-namespace).
- assess_state_advancement: forward **blocker_kwargs so can_approve/can_merge/
workflow_status honor the full blocker set (also fixes latent drop of
mcp_reconnect_failed/stale_capability_state through those paths).
- gitea_assess_review_merge_state_machine tool: accept live_namespace_broken and
thread it through all state-machine calls.
- Tests: prove can_merge/workflow_status/tool block on live_namespace_broken even
with every review state + pre-merge gate satisfied; bridge classify verdict.
- Docs: enforcement section wiring blocks_merge_workflow -> live_namespace_broken.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
sysadmin
jcwalker3andsysadmin
