feat(bootstrap): add controller-authorized recovery lane for fleet-safety deadlocks #981
Open
opened 2026-07-30 11:11:50 -05:00 by jcwalker3
·
2 comments
No Branch/Tag Specified
master
fix/issue-983-cross-repo-base-ref
feat/issue-980-stale-worker-retirement
fix/issue-975-client-identity-heartbeat
fix/issue-973-cross-repo-canonical-roots
fix/issue-970-safely-resolve-missing-worktrees
fix/issue-969-native-mcp-bootstrap
feat/issue-664-break-glass-restart
feat/issue-708-mcp-namespace-attachment
feat/issue-665-restart-audit
fix/issue-700-durable-walls
fix/issue-704-prevent-env-workspace-bindings
feat/issue-707-cross-project-boundaries
fix/issue-690-review-profile-switch-guard
fix/issue-953-bootstrap-lock-provenance
feat/issue-949-native-fleet-inventory
fix/issue-943-runtime-context-helpers
fix/issue-945-owning-pr-renewal-evidence
fix/issue-941-scope-guard-bootstrap-wiring
docs/issue-930-remote-mcp-coupling-inventory
fix/issue-892-author-bootstrap-deadlock
fix/issue-686-detect-reject-manual-mcp
fix/issue-672-mcp-config-drift
fix/issue-689-deterministic-mcp-namespace
feat/issue-666-concurrent-mcp-restart-tests
feat/issue-659-maintenance-drain-mode
feat/issue-648-notifications-console
fix/issue-670-direct-master-incident
feat/issue-644-console-recovery
feat/issue-650-providers-insights
feat/issue-669-scoped-component-recovery
docs/issue-668-mcp-ha-rolling-restart
feat/issue-667-console-restart-controls
feat/issue-645-linkage-console
feat/issue-643-request-preview-initiate
fix/issue-897-permission-stale-runtime-classification
feat/issue-641-runtime-session-view
feat/issue-663-restart-classes
feat/issue-661-drain-proof-hard-gate
fix/issue-854-semantic-container-exclusion
issue-640
fix/issue-682-starlette-httpx2
v1.1.0
Labels
Clear labels
allocator
anti-stomp
architecture
bug
chore
codex
concurrency
contamination
control-plane
dashboard
database
design
documentation
enhancement
gitea
glitchtip
important
incident
incident-bridge
integration
jenkins
labels
leases
mcp
mcp-health
mcp-menu
multi-project
mutating
nice-to-have
observability
portability
preflight
protected-branch
queue
read-only
reconnect
recovery
refactor
release
reliability
resumable-review
reviewer
roadmap
safety
security
self-hosted
sentry
stale-runtime
status:blocked
status:in-progress
status:pr-open
status:ready
terminal-lock
testing
tracker
type:bug
type:feature
type:feature
type:guardrail
visibility
workflow
workflow-hardening
workflow-hardening
Controller-owned work allocator
Prevent concurrent LLM session stomping
Architecture / structural design
OpenAI Codex client / workflow session surface
Concurrent session safety
Workflow or session contamination incident
MCP control-plane coordination and allocation authority
MCP operational dashboard/queue view
Internal coordination storage (SQLite/Postgres)
Design / investigation, no implementation
Docs / runbooks
New feature or improvement
Gitea MCP workflow
GlitchTip integration
Operational or process incident requiring durable audit trail
Sentry-to-Gitea incident bridging
Integration testing
Jenkins integration
Label taxonomy management
Lease adopt/release/expire lifecycle
MCP server / tooling
MCP namespace and runtime health
MCP menu surface
Work spanning multiple monitoring projects or Gitea repos
Mutating action; requires gating
Observability, metrics, traces, error reporting
Cross-platform / portability
Shared preflight gates before mutation
Protected branch / stable-branch policy concern
Work queue visibility and allocation
Read-only, no mutation
MCP client reconnect/reload recovery path
Recovery paths for stale/foreign leases
Code refactor / restructure
Release / versioning
Reliability / failure handling
Persist and resume prepared review verdicts across sessions
Reviewer workflow tooling
Roadmap / umbrella issue
Safety rails and fail-closed mutation guards
Security / trust boundary
Self-hosted infrastructure integration
Sentry error monitoring integration
Stale backend daemon / runtime-vs-master parity failures
Issue is blocked
Issue is being worked on
Issue has an open pull request
Issue is ready for work
Terminal review lock (#332) path
Tests / test coverage
Issue tracker hygiene / meta
Bug or defect
Feature or enhancement
Feature or enhancement
Safety gate or guardrail
Workflow state visibility for LLMs/operators
Cross-tool workflow
LLM workflow coordination hardening
LLM workflow coordination hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Scaled-Tech-Consulting/Gitea-Tools#981
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Add a narrowly scoped, controller-authorized bootstrap recovery lane for implementing control-plane repairs when the repair itself is blocked by the safety gate it must fix.
This must be an auditable capability with explicit eligibility, authorization, scope, expiration, and revocation—not a general fleet-safety bypass.
Observed deadlock
Issue #980 requires implementing a sanctioned, CAS-protected operation for retiring conclusively stale worker registrations.
Current fleet assessment reported:
stale_orphaned_workerregistrations.legacy_incomplete_identity.live_fleet_safe:false.mutation_safe:false.The ordinary author resolver reports
work_issueas allowed, but the fleet gate still prohibits branch creation, commits, pushes, and PR creation.This creates a circular dependency:
PR #906 remains blocked behind this condition.
Security principle
The recovery lane must not set
live_fleet_safe:true, suppress fleet findings, weaken ordinary fleet enforcement, or allow an author to authorize their own exception.Issue labels such as
bootstrap,control-plane, orstatus:readymust not independently grant recovery access because an author may be able to create or label issues.Authorization must originate from an independently authenticated controller or other canonical operator-controlled authority.
Required behavior
1. Dedicated bootstrap capability
Add a canonical task and capability for infrastructure recovery, such as:
bootstrap_control_plane_repairIt must be distinct from ordinary
work_issueauthorization.The resolver must return structured results including:
2. Controller-issued recovery grant
Only an authorized controller or equivalent operator-controlled role may issue a recovery grant.
The grant must be:
The author must not be able to issue, broaden, renew, or approve their own grant.
3. Minimal operation scope
A valid grant may permit only the operations required to implement and hand off the repair:
It must not authorize:
4. Remaining safety gates
Bootstrap authorization must not disable unrelated protections.
The workflow must still require:
If another safety failure is unrelated to the fleet deadlock being repaired, the operation must stop.
5. Trusted gate consumption
Do not grant authors unrestricted access to the complete controller/reconciler fleet snapshot merely to use the recovery lane.
The author workflow should consume a compact, trusted controller-produced verdict containing:
Detailed fleet inventory must remain restricted to its canonical controller/reconciler roles unless a separately justified permission change is approved.
6. Bootstrap installation path
The design must explain how the first recovery capability can be established when ordinary author implementation is already blocked.
Use an existing sanctioned operator-controlled installation, deployment, or control-plane update path if one exists.
If no such path exists, define a narrowly scoped, auditable first-install procedure that:
The implementation must not merely create another issue that is itself impossible to implement.
7. Audit and revocation
Every grant and attempted use must record:
Expired, revoked, mismatched, or already-consumed grants must fail closed.
8. Independent review
Code produced through the recovery lane must follow the normal independent-review process.
The recovery grant must not authorize approval or merge, and it must not make author-generated test evidence authoritative.
Acceptance criteria
Non-goals
Related work
client_instance_idpropagation remains a separate problem.Canonical handoff
Canonical Issue State
STATE: blocked — Decision C: no sanctioned first-install path exists
WHO_IS_NEXT: controller
NEXT_ACTION: Authenticated controller/operator authority must define and approve a narrowly scoped first-install mechanism for the controller-authorized recovery lane. Do not implement #981 or #980 and do not resume PR #906 until that mechanism exists.
NEXT_PROMPT:
WHAT_HAPPENED:
Decision C — no sanctioned first-install path exists
A controller/reconciler assessment found no existing native recovery-grant capability and no documented operator-controlled first-install path capable of authorizing implementation of #981 while fleet safety remains false.
Assessment evidence:
sysadmin/prgs-controllersysadmin/prgs-reconcilerScaled-Tech-Consulting/Gitea-Tools108cbfa173de34470d7abc022b929e6b3adc418afleetrev-6e7790e8faf9fed6legacy_incomplete_identityblockers: 2live_fleet_safe:falsemutation_safe:falseThe two live workers were separate controller and reconciler sessions. Sharing the same client type did not constitute a duplicate cohort.
The following proposed recovery capabilities were absent or returned
unknown_task:bootstrap_control_plane_repairauthorize_bootstrap_repairbootstrap_repairissue_recovery_grantauthorize_recovery_grantrecovery_authorizationinstall_control_plane_capabilitycontrol_plane_updatereconcile_instance_fleetretire_stale_workersThe existing
issue_irrecoverable_provenance_authorizationcapability is limited to review-decision-lock provenance recovery. It does not authorize author branches, commits, pushes, pull requests, or implementation of #981.The resulting circular dependency is:
Retiring the 91 stale registrations would not independently make the fleet safe. The two live
legacy_incomplete_identityblockers would remain. Trusted launcher identity propagation remains a separate operator-owned problem.Issue #981 therefore remains the durable owner of the missing first-install mechanism. Do not create another issue for the same requirement.
No mutation occurred during the assessment. This establishes only that the assessment session performed no repository mutation; it does not claim that uninspected external resources were independently verified unchanged.
The approving operator authority must be authenticated and validated separately. Do not infer operator authority merely from the
jcwalker3author identity.WHY: Fleet safety is false and no native recovery-grant or documented operator-controlled first-install path can authorize implementing #981 while that condition holds, producing a circular dependency that leaves #981 blocked.
RELATED_PRS: PR #906 remains paused; issue #980 remains paused; no new PR created for #981.
BLOCKERS: live_fleet_safe:false; fleet-layer mutation_safe:false; 2 live legacy_incomplete_identity blockers; 91 stale workers (retirement alone insufficient); missing native first-install/recovery-grant capability; no sanctioned path to authorize #981 implementation. Unblock after authenticated controller/operator authority defines and approves the narrowly scoped first-install mechanism for the controller-authorized recovery lane.
VALIDATION: Assessment-only session — no repository mutation, no branch/commit/push/PR, no lease/grant issuance, no changes to #980, PR #906, #650, or review 624. Live preflight for this recording: identity sysadmin / prgs-controller, repo Scaled-Tech-Consulting/Gitea-Tools, issue #981 open with status:ready before status update, gitea.issue.comment allowed on controller profile.
LAST_UPDATED_BY: sysadmin / prgs-controller (gitea-controller)
WHO_IS_NEXT: authenticated controller/operator authority to define and approve the narrowly scoped first-install mechanism. Issue #980 and PR #906 remain paused.Canonical Issue State
STATE: premise-corrected — D2-A; fleet-deadlock premise unsupported at current master
WHO_IS_NEXT: controller
NEXT_ACTION: Correct or dispose #981 under D2-A (unsupported fleet-deadlock premise). Do not implement #981 as framed. Do not activate BAA/first-install. Separately revalidate whether #980 has any remaining real dependency. Do not resume #980 or PR #906 in this session.
NEXT_PROMPT:
WHAT_HAPPENED:
Disposition D2-A — #981 premise requires correction
Master SHA (parity-verified):
108cbfa173de34470d7abc022b929e6b3adc418aStage −1 native author evidence (submitted; not re-impersonated):
jcwalker3/ profileprgs-author/ roleauthorprgs/Scaled-Tech-Consulting/Gitea-Tools108cbfa173de34470d7abc022b929e6b3adc418a;in_parity:true;live_stale:false; parity-layermutation_safe:truegitea_resolve_task_capability(task="work_issue", issue_number=981, remote="prgs")→required_operation_permission=gitea.pr.create,allowed_in_current_session=true,available_in_session=true,restart_required=false,stop_required=false,mutation_performed=false,exact_safe_next_action="None; ready for operations."gitea.branch.create,gitea.branch.push,gitea.repo.commit,gitea.pr.create, plus issue/PR comment opsController preflight (this session):
gitea-controller; identitysysadmin; profileprgs-controller; role_kindcontrollerprgs→Scaled-Tech-Consulting/Gitea-Toolsin_parity:true;live_stale:false; parity-layermutation_safe:truemasterstatus:blockedbefore this commentcomment_issue/gitea.issue.commentallowed; labels catalog has neitherstatus:needs-revisionnorstatus:needs-triageSource-backed verification at
108cbfa:live_fleet_safeproduced only bymcp_fleet_snapshot.snapshot_instance_fleet; zero non-test Python readers for authorization;gitea_mcp_server.pyhas zerolive_fleet_safeoccurrences; author modules do not import fleet snapshot.gitea_snapshot_instance_fleetis read-only controller/reconciler; capability mapgitea.read; docs state no new mutation permissions.work_issue→gitea.pr.create/ author intask_capability_map.py._profile_operation_gateonly (master parity, runtime mode, profile allow/forbid)._profile_permission_block/_namespace_mutation_blockhave no fleet terms.gitea_lock_issue,gitea_commit_files,gitea_create_prhave no fleet-derived denial path. Task map: lock_issue→gitea.issue.comment, create_branch→gitea.branch.create, commit_files→gitea.repo.commit, push_branch→gitea.branch.push, create_pr→gitea.pr.create.Decision C comment
18775— remains valid in part: assessment context; fleet observational findings as observations; absence of native recovery-grant/bootstrap tasks; irrecoverable-provenance grant not a general author path; BAA paused; #980/PR #906 not resumed by that decision.Superseded from Decision C: conclusion that fleet enforcement blocks ordinary author mutations and creates a circular dependency requiring a recovery grant to implement #981; premise that #981 stays blocked pending first-install because authors cannot mutate while
live_fleet_safe:false; AC framing that ordinary author work is fleet-gated bylive_fleet_safe.Conclusions: fleet-deadlock premise unsupported; Decision C recovery-grant search historically accurate but need not established; BAA remains paused; #981 must not be implemented as framed; no proof all downstream mutations succeed (no mutation test); multi-LLM focus should be trusted identity, registry accuracy, collisions, ownership isolation, safe concurrency.
Labels: live catalog lacks
status:needs-revisionandstatus:needs-triage→ labels left unchanged (includingstatus:blocked); nostatus:ready; issue not closed; body not rewritten.BAA status: paused / not activated.
WHY: Independent controller verification found no authorization consumer of
live_fleet_safeor fleet-layermutation_safeon ordinary author mutation paths; Stage −1work_issueresolved ready; therefore #981's fleet-deadlock premise and Decision C's circular-dependency need for a recovery grant are unsupported at master108cbfa.RELATED_PRS: PR #906 remains paused; issue #980 remains paused pending separate dependency revalidation; no new PR for #981.
BLOCKERS: #981 premise unsupported (not an author fleet block). Remaining real multi-LLM gaps (trusted unique worker identity, registry accuracy, collision detection, ownership isolation, concurrency) are outside this disposition and do not re-establish the #981 fleet-deadlock premise. Labels could not move to needs-revision/needs-triage because those labels do not exist in the live catalog.
VALIDATION: Controller-only disposition on clean master
108cbfa; whoami + resolve_task_capability(comment_issue) immediately before this post; no author mutation test; no code/branch/worktree/commit/push/PR/claim/lease/grant/fleet/config/process changes; #980, PR #906, #650, review 624 untouched.LAST_UPDATED_BY: sysadmin / prgs-controller (gitea-controller)
WHO_IS_NEXT: controller