Remote-MCP 10: Dual-run cutover, parity validation, and rollback runbook #939

New Issue

2026-07-26T00:34:29-05:00

jcwalker3 commented

2026-07-26 00:34:29 -05:00

Parent epic: #929.
Depends on: #938.

Problem

An endpoint that works in tests is not a migration. The local fleet is what every current workflow, skill, and operator habit is written against, and switching to the remote endpoint changes which authority owns locks and leases. Cutting over without a validated dual-run leaves no way to tell whether the remote endpoint behaves like the fleet it replaces, and no way back if it does not.

The specific hazard is coordination state. During any period when both the local fleet and the remote endpoint can serve mutations, two authorities exist for the same locks and leases. That window must be deliberately designed rather than discovered in production.

Scope

Validate the remote endpoint against the local fleet, then cut over with a proven way back.

Define the dual-run period and state explicitly which authority owns coordination state during it. Two authorities mutating the same locks concurrently is not permitted.
Run a behavioral parity comparison across both surfaces for the read-only tool set, and record every divergence with a verdict of acceptable or a defect.
Run a supervised end-to-end canary against the remote endpoint for the author, reviewer, and merger roles, from two different LLM clients.
Define the cutover sequence, its preconditions, and the observable signals that confirm each step.
Write docs/remote-mcp/rollback.md covering how to restore the local fleet, what to do about coordination state written by the remote endpoint, and how to confirm the restoration.
Execute the rollback once, deliberately, and record the result. An untested rollback is not a rollback.
Update the operator-facing workflow documentation and skills so the connection model and its verification steps match reality.

Acceptance criteria

The dual-run design names one coordination authority at every point in time, with no overlap window in which both may mutate.
A parity report covers the read-only tool set on both surfaces, with every divergence recorded and adjudicated.
Canaries pass for author, reviewer, and merger, from two different clients, with evidence recorded on this issue.
The cutover sequence is documented with preconditions and confirmation signals per step.
docs/remote-mcp/rollback.md exists and has been executed once, with the outcome recorded.
Operator documentation and skills describe the connection model, the fleet verification steps, and the failure modes accurately.
No workflow step still instructs an operator to verify the connection by counting local processes, unless the local fleet is the configured mode.

Verification

The parity report is reproducible from a committed script or a documented procedure.
Canary evidence names the client, the role, the issue or pull request touched, and the commit.
The rollback record shows the local fleet serving mutations again after the exercise.

Non-goals

Decommissioning the local fleet permanently. Keeping it as a supported mode is a separate decision for the operator.
Migrating other MCP servers in the operator's configuration.
Performance tuning of the remote endpoint.

Parent epic: #929. Depends on: #938. ## Problem An endpoint that works in tests is not a migration. The local fleet is what every current workflow, skill, and operator habit is written against, and switching to the remote endpoint changes which authority owns locks and leases. Cutting over without a validated dual-run leaves no way to tell whether the remote endpoint behaves like the fleet it replaces, and no way back if it does not. The specific hazard is coordination state. During any period when both the local fleet and the remote endpoint can serve mutations, two authorities exist for the same locks and leases. That window must be deliberately designed rather than discovered in production. ## Scope Validate the remote endpoint against the local fleet, then cut over with a proven way back. - Define the dual-run period and state explicitly which authority owns coordination state during it. Two authorities mutating the same locks concurrently is not permitted. - Run a behavioral parity comparison across both surfaces for the read-only tool set, and record every divergence with a verdict of acceptable or a defect. - Run a supervised end-to-end canary against the remote endpoint for the author, reviewer, and merger roles, from two different LLM clients. - Define the cutover sequence, its preconditions, and the observable signals that confirm each step. - Write `docs/remote-mcp/rollback.md` covering how to restore the local fleet, what to do about coordination state written by the remote endpoint, and how to confirm the restoration. - Execute the rollback once, deliberately, and record the result. An untested rollback is not a rollback. - Update the operator-facing workflow documentation and skills so the connection model and its verification steps match reality. ## Acceptance criteria - The dual-run design names one coordination authority at every point in time, with no overlap window in which both may mutate. - A parity report covers the read-only tool set on both surfaces, with every divergence recorded and adjudicated. - Canaries pass for author, reviewer, and merger, from two different clients, with evidence recorded on this issue. - The cutover sequence is documented with preconditions and confirmation signals per step. - `docs/remote-mcp/rollback.md` exists and has been executed once, with the outcome recorded. - Operator documentation and skills describe the connection model, the fleet verification steps, and the failure modes accurately. - No workflow step still instructs an operator to verify the connection by counting local processes, unless the local fleet is the configured mode. ## Verification - The parity report is reproducible from a committed script or a documented procedure. - Canary evidence names the client, the role, the issue or pull request touched, and the commit. - The rollback record shows the local fleet serving mutations again after the exercise. ## Non-goals - Decommissioning the local fleet permanently. Keeping it as a supported mode is a separate decision for the operator. - Migrating other MCP servers in the operator's configuration. - Performance tuning of the remote endpoint.