Scaled-Tech-Consulting/Gitea-Tools
4
0
Fork 0
Code Issues 110 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

Replace full-server MCP resets with scoped component recovery #669

New Issue
Closed
opened 2026-07-10 15:33:41 -05:00 by jcwalker3 · 1 comment
jcwalker3 commented 2026-07-10 15:33:41 -05:00
Owner
Copy Link
Copy Source

Problem statement

Operational recovery still defaults to full MCP server reset. Narrower recoveries (reconnect, capability refresh, namespace rebind, transport recreation, config reload, worker/role/connector restart, lease/lock recovery, quarantine, reconciliation) are incomplete or undocumented as a preferred ladder.

Operational impact

Unrelated concurrent work dies when only one component was unhealthy; contradicts last-resort principle (#655).

Scope

  • Implement or wire preferred recovery actions from #655 ladder with attempt logging.
  • Decision tree: map symptoms → narrowest class (#663).
  • Only escalate to full restart after recorded insufficient narrower attempts.
  • Integrate #584 auto-reconnect, capability refresh, #610 stale runtime, lease recovery, #630 contamination paths.
  • Metrics: fraction of recoveries that avoid full restart.

Non-goals

  • HA multi-instance (#668 design).
  • Normalizing pkill (#630).

Required implementation investigation

#655 #663 #657 #658; transport EOF recovery docs; capability resolver; namespace health.

Proposed implementation direction

RecoveryPlaybook engine returning ordered actions; tools for each narrow recovery; coordinator refuses full restart without attempt log unless break-glass.

Security / workflow-safety

Narrow recoveries still capability-gated; no silent mutation.

Failure behavior

Narrow recovery fail → record reason → allow escalate.

Acceptance criteria

  1. Ladder documented and enforced in coordinator.
  2. At least client reconnect, capability refresh, config reload, namespace rebind, lease recovery paths implemented or explicitly delegated with issue links.
  3. Full restart requires attempt log (except break-glass).
  4. Tests for escalation rules.
  5. Links #652 #653 #655.

Required tests

Playbook unit tests; escalate-only-after-fail cases.

Observability and audit

Each narrow attempt audited (#665).

Dependencies

Rollout

Enable playbook recommendations before hard-require attempt logs.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement scoped recovery ladder enforcement
NEXT_PROMPT: Author scoped recovery under #655; PR; stop

Required proof

Playbook + enforcement + tests.

Required final response and handoff

Brief PR → reviewer.

## Problem statement Operational recovery still defaults to full MCP server reset. Narrower recoveries (reconnect, capability refresh, namespace rebind, transport recreation, config reload, worker/role/connector restart, lease/lock recovery, quarantine, reconciliation) are incomplete or undocumented as a preferred ladder. ## Operational impact Unrelated concurrent work dies when only one component was unhealthy; contradicts last-resort principle (#655). ## Scope * Implement or wire preferred recovery actions from #655 ladder with attempt logging. * Decision tree: map symptoms → narrowest class (#663). * Only escalate to full restart after recorded insufficient narrower attempts. * Integrate #584 auto-reconnect, capability refresh, #610 stale runtime, lease recovery, #630 contamination paths. * Metrics: fraction of recoveries that avoid full restart. ## Non-goals * HA multi-instance (#668 design). * Normalizing pkill (#630). ## Required implementation investigation #655 #663 #657 #658; transport EOF recovery docs; capability resolver; namespace health. ## Proposed implementation direction `RecoveryPlaybook` engine returning ordered actions; tools for each narrow recovery; coordinator refuses full restart without attempt log unless break-glass. ## Security / workflow-safety Narrow recoveries still capability-gated; no silent mutation. ## Failure behavior Narrow recovery fail → record reason → allow escalate. ## Acceptance criteria 1. Ladder documented and enforced in coordinator. 2. At least client reconnect, capability refresh, config reload, namespace rebind, lease recovery paths implemented or explicitly delegated with issue links. 3. Full restart requires attempt log (except break-glass). 4. Tests for escalation rules. 5. Links #652 #653 #655. ## Required tests Playbook unit tests; escalate-only-after-fail cases. ## Observability and audit Each narrow attempt audited (#665). ## Dependencies * Parent **#655** · Depends **#663** **#658** · Related #584 #610 #630 · Vision **#652** · Roadmap **#653** ## Rollout Enable playbook recommendations before hard-require attempt logs. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement scoped recovery ladder enforcement NEXT_PROMPT: Author scoped recovery under #655; PR; stop ``` ## Required proof Playbook + enforcement + tests. ## Required final response and handoff Brief PR → reviewer.
jcwalker3 added status:pr-open and removed status:ready labels 2026-07-25 16:41:46 -05:00
jcwalker3 commented 2026-07-25 16:42:07 -05:00
Author
Owner
Copy Link
Copy Source

Canonical Issue State

STATE: ready-for-review
WHO_IS_NEXT: reviewer
NEXT_ACTION: Review PR #914 for Issue #669
NEXT_PROMPT:

Review PR #914 for Issue #669 in Scaled-Tech-Consulting/Gitea-Tools on remote prgs.

WHAT_HAPPENED: Created branch feat/issue-669-scoped-component-recovery, committed implementation for Issue #669, pushed to remote prgs, and opened PR #914.
WHY: Implement scoped recovery playbook and enforce attempt-log gate before full MCP restarts.
RELATED_PRS: #914
BLOCKERS: none
VALIDATION: pytest tests/test_recovery_playbook.py tests/test_issue_886_apply_authorization_conjunction.py -q (35 passed, 45 subtests passed)
LAST_UPDATED_BY: jcwalker3 (prgs-author)

Canonical Handoff

REPOSITORY: Scaled-Tech-Consulting/Gitea-Tools
ISSUE: 669
PR: 914
WORKFLOW_STATE: needs-review
HEAD_SHA: 461e1dac78a54c7d30851d05ae8e5983fdcc659b
BASE_BRANCH: master
BASE_OR_MERGE_SHA: 211890f361f90a4b2ab95e2f5e4d162e7758dd75
ACTING_ROLE: author
ACTING_IDENTITY: jcwalker3 (prgs-author)
COMPLETED_ACTIONS: Created PR #914 for Issue #669 implementing scoped recovery playbook and attempt-log enforcement before full MCP restart.
VALIDATION_EVIDENCE: pytest tests/test_recovery_playbook.py tests/test_issue_886_apply_authorization_conjunction.py -q (35 passed, 45 subtests passed)
MUTATION_LEDGER: Created branch feat/issue-669-scoped-component-recovery, committed 461e1dac78a54c7d30851d05ae8e5983fdcc659b, pushed to prgs, opened PR #914.
BLOCKERS: none
NEXT_ACTOR: reviewer
NEXT_ACTION: Review PR #914 for Issue #669
PROHIBITED_ACTIONS: Author must not review or merge own PR #914.
NEXT_PROMPT: Review PR #914 for Issue #669 in Scaled-Tech-Consulting/Gitea-Tools on remote prgs.
WORKFLOW_FAILURE_ISSUES: none
LAST_UPDATED: 2026-07-25T21:42:00Z
## Canonical Issue State STATE: ready-for-review WHO_IS_NEXT: reviewer NEXT_ACTION: Review PR #914 for Issue #669 NEXT_PROMPT: ```text Review PR #914 for Issue #669 in Scaled-Tech-Consulting/Gitea-Tools on remote prgs. ``` WHAT_HAPPENED: Created branch feat/issue-669-scoped-component-recovery, committed implementation for Issue #669, pushed to remote prgs, and opened PR #914. WHY: Implement scoped recovery playbook and enforce attempt-log gate before full MCP restarts. RELATED_PRS: #914 BLOCKERS: none VALIDATION: pytest tests/test_recovery_playbook.py tests/test_issue_886_apply_authorization_conjunction.py -q (35 passed, 45 subtests passed) LAST_UPDATED_BY: jcwalker3 (prgs-author) <!-- sph:v1 --> ## Canonical Handoff ```text REPOSITORY: Scaled-Tech-Consulting/Gitea-Tools ISSUE: 669 PR: 914 WORKFLOW_STATE: needs-review HEAD_SHA: 461e1dac78a54c7d30851d05ae8e5983fdcc659b BASE_BRANCH: master BASE_OR_MERGE_SHA: 211890f361f90a4b2ab95e2f5e4d162e7758dd75 ACTING_ROLE: author ACTING_IDENTITY: jcwalker3 (prgs-author) COMPLETED_ACTIONS: Created PR #914 for Issue #669 implementing scoped recovery playbook and attempt-log enforcement before full MCP restart. VALIDATION_EVIDENCE: pytest tests/test_recovery_playbook.py tests/test_issue_886_apply_authorization_conjunction.py -q (35 passed, 45 subtests passed) MUTATION_LEDGER: Created branch feat/issue-669-scoped-component-recovery, committed 461e1dac78a54c7d30851d05ae8e5983fdcc659b, pushed to prgs, opened PR #914. BLOCKERS: none NEXT_ACTOR: reviewer NEXT_ACTION: Review PR #914 for Issue #669 PROHIBITED_ACTIONS: Author must not review or merge own PR #914. NEXT_PROMPT: Review PR #914 for Issue #669 in Scaled-Tech-Consulting/Gitea-Tools on remote prgs. WORKFLOW_FAILURE_ISSUES: none LAST_UPDATED: 2026-07-25T21:42:00Z ```
sysadmin removed the status:pr-open label 2026-07-25 17:14:23 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels anti-stomp mcp-health stale-runtime type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 jwalker (Regular User Account) superadmin (Admin Account) sysadmin (Regular User Account)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#669