Replace full-server MCP resets with scoped component recovery #669

New Issue

2026-07-10T15:33:41-05:00

jcwalker3 commented

2026-07-10 15:33:41 -05:00

Problem statement

Operational recovery still defaults to full MCP server reset. Narrower recoveries (reconnect, capability refresh, namespace rebind, transport recreation, config reload, worker/role/connector restart, lease/lock recovery, quarantine, reconciliation) are incomplete or undocumented as a preferred ladder.

Operational impact

Unrelated concurrent work dies when only one component was unhealthy; contradicts last-resort principle (#655).

Scope

Implement or wire preferred recovery actions from #655 ladder with attempt logging.
Decision tree: map symptoms → narrowest class (#663).
Only escalate to full restart after recorded insufficient narrower attempts.
Integrate #584 auto-reconnect, capability refresh, #610 stale runtime, lease recovery, #630 contamination paths.
Metrics: fraction of recoveries that avoid full restart.

Non-goals

HA multi-instance (#668 design).
Normalizing pkill (#630).

Required implementation investigation

#655 #663 #657 #658; transport EOF recovery docs; capability resolver; namespace health.

Proposed implementation direction

RecoveryPlaybook engine returning ordered actions; tools for each narrow recovery; coordinator refuses full restart without attempt log unless break-glass.

Security / workflow-safety

Narrow recoveries still capability-gated; no silent mutation.

Failure behavior

Narrow recovery fail → record reason → allow escalate.

Acceptance criteria

Ladder documented and enforced in coordinator.
At least client reconnect, capability refresh, config reload, namespace rebind, lease recovery paths implemented or explicitly delegated with issue links.
Full restart requires attempt log (except break-glass).
Tests for escalation rules.
Links #652 #653 #655.

Required tests

Playbook unit tests; escalate-only-after-fail cases.

Observability and audit

Each narrow attempt audited (#665).

Dependencies

Parent #655 · Depends #663 #658 · Related #584 #610 #630 · Vision #652 · Roadmap #653

Rollout

Enable playbook recommendations before hard-require attempt logs.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement scoped recovery ladder enforcement
NEXT_PROMPT: Author scoped recovery under #655; PR; stop

Required proof

Playbook + enforcement + tests.

Required final response and handoff

Brief PR → reviewer.

## Problem statement Operational recovery still defaults to full MCP server reset. Narrower recoveries (reconnect, capability refresh, namespace rebind, transport recreation, config reload, worker/role/connector restart, lease/lock recovery, quarantine, reconciliation) are incomplete or undocumented as a preferred ladder. ## Operational impact Unrelated concurrent work dies when only one component was unhealthy; contradicts last-resort principle (#655). ## Scope * Implement or wire preferred recovery actions from #655 ladder with attempt logging. * Decision tree: map symptoms → narrowest class (#663). * Only escalate to full restart after recorded insufficient narrower attempts. * Integrate #584 auto-reconnect, capability refresh, #610 stale runtime, lease recovery, #630 contamination paths. * Metrics: fraction of recoveries that avoid full restart. ## Non-goals * HA multi-instance (#668 design). * Normalizing pkill (#630). ## Required implementation investigation #655 #663 #657 #658; transport EOF recovery docs; capability resolver; namespace health. ## Proposed implementation direction `RecoveryPlaybook` engine returning ordered actions; tools for each narrow recovery; coordinator refuses full restart without attempt log unless break-glass. ## Security / workflow-safety Narrow recoveries still capability-gated; no silent mutation. ## Failure behavior Narrow recovery fail → record reason → allow escalate. ## Acceptance criteria 1. Ladder documented and enforced in coordinator. 2. At least client reconnect, capability refresh, config reload, namespace rebind, lease recovery paths implemented or explicitly delegated with issue links. 3. Full restart requires attempt log (except break-glass). 4. Tests for escalation rules. 5. Links #652 #653 #655. ## Required tests Playbook unit tests; escalate-only-after-fail cases. ## Observability and audit Each narrow attempt audited (#665). ## Dependencies * Parent **#655** · Depends **#663** **#658** · Related #584 #610 #630 · Vision **#652** · Roadmap **#653** ## Rollout Enable playbook recommendations before hard-require attempt logs. ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement scoped recovery ladder enforcement NEXT_PROMPT: Author scoped recovery under #655; PR; stop ``` ## Required proof Playbook + enforcement + tests. ## Required final response and handoff Brief PR → reviewer.