feat(control-plane): persist authoritative fleet synchronization and restart receipts #951

New Issue

2026-07-27T21:23:50-05:00

jcwalker3 commented

2026-07-27 21:23:50 -05:00

Related: #949, #950, #948, #900, #665, #662, #661, #660, #658, #659, #655, #669, #591, #559, #929

Problem

The canonical workflow can inspect current parity after a fleet synchronization or restart, but the completed cross-LLM audit could not retrieve an authoritative historical record proving:

Whether the control checkout was fast-forwarded
Which revision it moved from and to
Whether a fleet drain occurred
Whether every expected server stopped and restarted
Which runtime instance existed before and after the operation
Which client or cohort initiated and owned the operation
Whether reconnect and post-restart reconciliation completed
Whether all fleet members joined the intended revision and cohort
Whether any phase partially failed
Whether the reported current state resulted from the claimed operation

Matching current revisions proves present parity. It does not reconstruct or authenticate a past synchronization or restart.

The missing historical record must not be "repaired" by repeating a restart merely to generate evidence.

Required behavior

Add authoritative, durable, read-only-queryable operation receipts for sanctioned control-plane synchronization and restart workflows.

Each receipt must identify:

Stable operation ID
Operation type
Requested action
Initiating profile, role, and authenticated identity
Initiating client and cohort identity
Repository binding
Target fleet or member set
Start and completion timestamps
Initial control-checkout revision
Target remote revision
Resulting control-checkout revision
Whether a fast-forward was required
Whether a fast-forward occurred
Drain request and drain result
Sessions or leases blocking the operation
Expected fleet members
Members observed before the operation
Members stopped
Members restarted
Members reconnected
Members reconciled
Per-member old and new runtime identity
Per-member old and new process or worker identity
Per-member old and new startup revision
Per-member daemon-start, local, and remote revisions
Phase-by-phase outcome
Partial failures and blocked reasons
Final parity result
Final fleet-inventory result when issue #949 becomes available
Final controller-role validation when issue #950 becomes available
Overall terminal status
Whether the receipt is complete enough for a mutation-sensitive gate

The receipt must distinguish:

Requested
Started
Draining
Synchronized
Restarting
Reconnecting
Reconciling
Completed
Partially completed
Failed
Aborted
Unknown because evidence is incomplete

Authoritative evidence requirements

The receipt system must:

Be produced by the sanctioned control-plane operation itself
Not trust caller-supplied process lists or success claims as authoritative
Persist across client and LLM sessions
Remain queryable after server restart
Be immutable or append-only after terminal completion
Record corrections or superseding evidence without silently rewriting history
Fail closed when required phases cannot be proven
Distinguish current-state parity from historical operation provenance
Distinguish "no operation was necessary" from "an operation occurred successfully"
Distinguish "no receipt exists" from "the operation failed"
Avoid reporting a successful fleet restart when only the parent client or some members restarted
Avoid inferring a single cohort solely from matching Git revisions
Avoid manufacturing provenance by rerunning an otherwise unnecessary operation

Native exposure

Provide sanctioned native read-only capabilities through:

gitea-controller
gitea-reconciler

The native interface must support:

Retrieving a receipt by operation ID
Retrieving the most recent operation for a repository or fleet
Listing recent operations with bounded pagination
Filtering by operation type and terminal status
Returning structured machine-readable evidence
Determining whether a receipt satisfies the canonical workflow's provenance gate

Receipt creation should occur automatically as part of sanctioned synchronization, drain, restart, reconnect, and reconciliation operations. A caller must not be able to fabricate a successful receipt through the read interface.

Suggested structured result

Include fields equivalent to:

operation_id
operation_type
repository
initiator
client_identity
cohort_identity
started_at
completed_at
status
phases
initial_revision
target_revision
resulting_revision
fast_forward_required
fast_forward_performed
configured_members
pre_operation_members
post_operation_members
stopped_members
restarted_members
reconnected_members
reconciled_members
missing_members
duplicate_members
unexpected_members
per_member_evidence
parity_satisfied
fleet_gate_satisfied
provenance_complete
mutation_gate_satisfied
blocked_reason
evidence_source

Exact names may follow repository conventions, but equivalent evidence is required.

Acceptance criteria

A successful control-checkout fast-forward records the exact old and new revisions.
A no-op synchronization records that no fast-forward was necessary and does not claim that one occurred.
A successful restart records every expected fleet member before and after the transition.
Per-member runtime and process identities distinguish old instances from restarted instances.
A missing, duplicate, or unexpected member prevents a complete-success verdict.
A member that fails to stop, restart, reconnect, or reconcile is identified explicitly.
Partial completion cannot be reported as complete success.
A restart blocked by live sessions or leases records the blocker without falsely recording a restart.
Matching final Git revisions alone do not prove restart completion or single-cohort membership.
Current green parity without a historical receipt is reported as current-state success with historical provenance unavailable.
Absence of a receipt does not trigger or recommend an unnecessary restart solely to create evidence.
Receipts persist across client sessions and MCP server restarts.
Terminal receipts are immutable or append-only and retain prior evidence.
Controller and reconciler return consistent results for the same operation ID.
Unknown or incomplete evidence produces a fail-closed provenance result.
Read-only receipt queries perform no restart, reconnect, synchronization, lease, issue, branch, or worktree mutation.
Automated tests cover successful, no-op, blocked, partial, failed, aborted, incomplete-evidence, and post-restart retrieval cases.
Canonical workflow documentation explains the distinction between current parity and historical provenance.
An LLM can prove a completed synchronization and restart using only sanctioned native calls.
Integration with issues #949 and #950 is defined without making this issue responsible for implementing those separate fixes.

Non-goals

Implementing fleet discovery or duplicate-process detection; tracked by issue #949
Repairing controller/reconciler role metadata; tracked by issue #950
Automatically terminating duplicate processes
Automatically clearing workflow leases
Reconstructing historical operations for which no authoritative receipt exists
Repeating a restart solely to generate evidence
Cleaning the stale PR #906 author lease
Resuming issue Remote-MCP 02: Introduce a transport-neutral bind seam accepting a remote transport (#931)
Changing normal review or merge rules

Duplicate verdict

NOT a duplicate. Verified by reading the bodies and acceptance criteria of #665, #662, #661, #660, #658, #659, #655, #949, #950, #948, #900, #669, #591, and #559.

Issue	State	Why it does not cover this
#665	open	Closest overlap. Owns restart audit event emission (`mcp.restart.*`) plus automatic incident-issue creation on failed drain, break-glass, unresolved reconcile, or unguarded restart. It is an event stream with correlation ids, not a per-operation receipt: it defines no control-checkout initial/target/resulting revision, no fast-forward-required vs fast-forward-performed distinction, no per-member old/new runtime and process identity across the transition, no bounded read-only retrieval interface keyed by operation ID on controller and reconciler, and no provenance verdict a mutation-sensitive gate can consume. It also cannot distinguish "no operation was necessary" from "an operation succeeded". This issue consumes #665's audit substrate rather than restating it.
#661	closed	`DrainProof` is a pre-restart artifact with a short TTL, non-forgeable only within the emitting process, and it is consumed by the restart gate then expires. It proves readiness before the operation; it never records what the operation actually did, and by construction it is not durable history.
#662	closed	Post-restart reconciliation produces a boot-time completion proof of resolved/unresolved items. It reports the reconcile phase only — it does not join that phase to the synchronization that preceded it, does not carry the fast-forward revisions, does not enumerate per-member before/after identity, and is not retrievable later by operation ID. In the live runtime it currently reports `mode: log_only` with `overall_status: degraded`, which is exactly the state this issue must make historically legible rather than silent.
#660	closed	`SessionCheckpoint` is per-session workflow state — role, issue/PR, worktree, stage, next action — written on stage transitions and pre-drain. It is keyed by session, not by control-plane operation, and it deliberately carries no fleet-wide revision or member-transition evidence.
#658	closed	Restart coordinator and impact analysis is a forward-looking preview: affected sessions, leases, blast radius, allow/deny/override. Dry-run by default and evaluated before the operation. A preview of what a restart would disrupt is not a record of what a restart did.
#659	closed conceptually / open as #659	Maintenance-drain mode implements the drain protocol and exposes drain state to sessions. Drain is one phase this receipt must record; the drain implementation itself neither persists the phase outcome as queryable history nor links it to the synchronization and restart phases.
#655	open	Umbrella governing restart authorization, blast radius, and the last-resort policy. It enumerates children including audit events and post-restart reconciliation, but owns no implementation. This issue is the missing provenance substrate those children stop short of; it does not re-open umbrella policy.
#949	open	Authoritative fleet inventory and duplicate-process detection — a snapshot of the current population. Its non-goals name "Reconstructing historical restart provenance" explicitly, so it disclaims this scope by construction. This issue records the inventory verdict as of an operation's completion; it does not implement the inventory.
#950	open	Controller role metadata and capability routing. Its non-goals name "Restart or synchronization provenance" explicitly. This issue records the controller-role validation result in a receipt; it does not repair the metadata.
#948	open	Defines client/session/generation ownership identity and lease fencing so concurrent LLM clients are not falsely blocked. It supplies the identity vocabulary a receipt cites; it stores no operation history.
#900	open	Cohort lifecycle supervision: bound the eligible population, drain and reap superseded cohorts. A mutating supervisory transition whose status surface describes currently resident cohorts. It does not persist a record of a completed synchronization or restart, and does not cover control-checkout fast-forward at all.
#669	closed	Recovery-ladder selection — prefer scoped component recovery over full-server resets. Chooses which operation to run; records nothing about the run.
#591	closed	Auto-restart namespaces when master advances. The trigger side of the same lifecycle; it produced no durable receipt, which is part of why the audit had nothing to retrieve.
#559	closed	Persists MCP workflow proof and review-decision state across daemon processes. Same durability technique, different subject: review/workflow decisions, not control-plane synchronization and restart operations.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement durable, append-only synchronization and restart operation receipts with read-only native retrieval on gitea-controller and gitea-reconciler per the acceptance criteria
NEXT_PROMPT:
Role: AUTHOR
Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools
Issue: this issue

1. Native MCP only; gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/.
2. Implement the receipt record, its phase and terminal-status model, and its persistence so it survives client sessions and MCP server restarts; make terminal receipts immutable or append-only.
3. Emit receipts from the sanctioned synchronization, drain, restart, reconnect, and reconciliation paths themselves; never from caller-supplied process lists or success claims.
4. Expose read-only retrieval by operation ID, most-recent-per-fleet, bounded pagination, and type/status filtering on gitea-controller and gitea-reconciler; return a provenance verdict for mutation-sensitive gates; fail closed on incomplete evidence.
5. Add regression tests for successful, no-op, blocked, partial, failed, aborted, incomplete-evidence, and post-restart retrieval cases.
6. Update canonical workflow documentation on current parity vs historical provenance.
7. Open a PR with Closes; stop; hand off to an independent reviewer.

Related: #949, #950, #948, #900, #665, #662, #661, #660, #658, #659, #655, #669, #591, #559, #929 ## Problem The canonical workflow can inspect current parity after a fleet synchronization or restart, but the completed cross-LLM audit could not retrieve an authoritative historical record proving: * Whether the control checkout was fast-forwarded * Which revision it moved from and to * Whether a fleet drain occurred * Whether every expected server stopped and restarted * Which runtime instance existed before and after the operation * Which client or cohort initiated and owned the operation * Whether reconnect and post-restart reconciliation completed * Whether all fleet members joined the intended revision and cohort * Whether any phase partially failed * Whether the reported current state resulted from the claimed operation Matching current revisions proves present parity. It does not reconstruct or authenticate a past synchronization or restart. The missing historical record must not be "repaired" by repeating a restart merely to generate evidence. ## Required behavior Add authoritative, durable, read-only-queryable operation receipts for sanctioned control-plane synchronization and restart workflows. Each receipt must identify: * Stable operation ID * Operation type * Requested action * Initiating profile, role, and authenticated identity * Initiating client and cohort identity * Repository binding * Target fleet or member set * Start and completion timestamps * Initial control-checkout revision * Target remote revision * Resulting control-checkout revision * Whether a fast-forward was required * Whether a fast-forward occurred * Drain request and drain result * Sessions or leases blocking the operation * Expected fleet members * Members observed before the operation * Members stopped * Members restarted * Members reconnected * Members reconciled * Per-member old and new runtime identity * Per-member old and new process or worker identity * Per-member old and new startup revision * Per-member daemon-start, local, and remote revisions * Phase-by-phase outcome * Partial failures and blocked reasons * Final parity result * Final fleet-inventory result when issue #949 becomes available * Final controller-role validation when issue #950 becomes available * Overall terminal status * Whether the receipt is complete enough for a mutation-sensitive gate The receipt must distinguish: * Requested * Started * Draining * Synchronized * Restarting * Reconnecting * Reconciling * Completed * Partially completed * Failed * Aborted * Unknown because evidence is incomplete ## Authoritative evidence requirements The receipt system must: * Be produced by the sanctioned control-plane operation itself * Not trust caller-supplied process lists or success claims as authoritative * Persist across client and LLM sessions * Remain queryable after server restart * Be immutable or append-only after terminal completion * Record corrections or superseding evidence without silently rewriting history * Fail closed when required phases cannot be proven * Distinguish current-state parity from historical operation provenance * Distinguish "no operation was necessary" from "an operation occurred successfully" * Distinguish "no receipt exists" from "the operation failed" * Avoid reporting a successful fleet restart when only the parent client or some members restarted * Avoid inferring a single cohort solely from matching Git revisions * Avoid manufacturing provenance by rerunning an otherwise unnecessary operation ## Native exposure Provide sanctioned native read-only capabilities through: * `gitea-controller` * `gitea-reconciler` The native interface must support: * Retrieving a receipt by operation ID * Retrieving the most recent operation for a repository or fleet * Listing recent operations with bounded pagination * Filtering by operation type and terminal status * Returning structured machine-readable evidence * Determining whether a receipt satisfies the canonical workflow's provenance gate Receipt creation should occur automatically as part of sanctioned synchronization, drain, restart, reconnect, and reconciliation operations. A caller must not be able to fabricate a successful receipt through the read interface. ## Suggested structured result Include fields equivalent to: * `operation_id` * `operation_type` * `repository` * `initiator` * `client_identity` * `cohort_identity` * `started_at` * `completed_at` * `status` * `phases` * `initial_revision` * `target_revision` * `resulting_revision` * `fast_forward_required` * `fast_forward_performed` * `configured_members` * `pre_operation_members` * `post_operation_members` * `stopped_members` * `restarted_members` * `reconnected_members` * `reconciled_members` * `missing_members` * `duplicate_members` * `unexpected_members` * `per_member_evidence` * `parity_satisfied` * `fleet_gate_satisfied` * `provenance_complete` * `mutation_gate_satisfied` * `blocked_reason` * `evidence_source` Exact names may follow repository conventions, but equivalent evidence is required. ## Acceptance criteria 1. A successful control-checkout fast-forward records the exact old and new revisions. 2. A no-op synchronization records that no fast-forward was necessary and does not claim that one occurred. 3. A successful restart records every expected fleet member before and after the transition. 4. Per-member runtime and process identities distinguish old instances from restarted instances. 5. A missing, duplicate, or unexpected member prevents a complete-success verdict. 6. A member that fails to stop, restart, reconnect, or reconcile is identified explicitly. 7. Partial completion cannot be reported as complete success. 8. A restart blocked by live sessions or leases records the blocker without falsely recording a restart. 9. Matching final Git revisions alone do not prove restart completion or single-cohort membership. 10. Current green parity without a historical receipt is reported as current-state success with historical provenance unavailable. 11. Absence of a receipt does not trigger or recommend an unnecessary restart solely to create evidence. 12. Receipts persist across client sessions and MCP server restarts. 13. Terminal receipts are immutable or append-only and retain prior evidence. 14. Controller and reconciler return consistent results for the same operation ID. 15. Unknown or incomplete evidence produces a fail-closed provenance result. 16. Read-only receipt queries perform no restart, reconnect, synchronization, lease, issue, branch, or worktree mutation. 17. Automated tests cover successful, no-op, blocked, partial, failed, aborted, incomplete-evidence, and post-restart retrieval cases. 18. Canonical workflow documentation explains the distinction between current parity and historical provenance. 19. An LLM can prove a completed synchronization and restart using only sanctioned native calls. 20. Integration with issues #949 and #950 is defined without making this issue responsible for implementing those separate fixes. ## Non-goals * Implementing fleet discovery or duplicate-process detection; tracked by issue #949 * Repairing controller/reconciler role metadata; tracked by issue #950 * Automatically terminating duplicate processes * Automatically clearing workflow leases * Reconstructing historical operations for which no authoritative receipt exists * Repeating a restart solely to generate evidence * Cleaning the stale PR #906 author lease * Resuming issue #931 * Changing normal review or merge rules ## Duplicate verdict NOT a duplicate. Verified by reading the bodies and acceptance criteria of #665, #662, #661, #660, #658, #659, #655, #949, #950, #948, #900, #669, #591, and #559. | Issue | State | Why it does not cover this | | --- | --- | --- | | #665 | open | Closest overlap. Owns restart *audit event emission* (`mcp.restart.*`) plus automatic incident-issue creation on failed drain, break-glass, unresolved reconcile, or unguarded restart. It is an event stream with correlation ids, not a per-operation receipt: it defines no control-checkout initial/target/resulting revision, no fast-forward-required vs fast-forward-performed distinction, no per-member old/new runtime and process identity across the transition, no bounded read-only retrieval interface keyed by operation ID on controller and reconciler, and no provenance verdict a mutation-sensitive gate can consume. It also cannot distinguish "no operation was necessary" from "an operation succeeded". This issue consumes #665's audit substrate rather than restating it. | | #661 | closed | `DrainProof` is a *pre-restart* artifact with a short TTL, non-forgeable only within the emitting process, and it is consumed by the restart gate then expires. It proves readiness before the operation; it never records what the operation actually did, and by construction it is not durable history. | | #662 | closed | Post-restart reconciliation produces a boot-time completion proof of resolved/unresolved items. It reports the reconcile phase only — it does not join that phase to the synchronization that preceded it, does not carry the fast-forward revisions, does not enumerate per-member before/after identity, and is not retrievable later by operation ID. In the live runtime it currently reports `mode: log_only` with `overall_status: degraded`, which is exactly the state this issue must make historically legible rather than silent. | | #660 | closed | `SessionCheckpoint` is per-session workflow state — role, issue/PR, worktree, stage, next action — written on stage transitions and pre-drain. It is keyed by session, not by control-plane operation, and it deliberately carries no fleet-wide revision or member-transition evidence. | | #658 | closed | Restart coordinator and impact analysis is a *forward-looking* preview: affected sessions, leases, blast radius, allow/deny/override. Dry-run by default and evaluated before the operation. A preview of what a restart would disrupt is not a record of what a restart did. | | #659 | closed conceptually / open as #659 | Maintenance-drain mode implements the drain protocol and exposes drain state to sessions. Drain is one phase this receipt must record; the drain implementation itself neither persists the phase outcome as queryable history nor links it to the synchronization and restart phases. | | #655 | open | Umbrella governing restart authorization, blast radius, and the last-resort policy. It enumerates children including audit events and post-restart reconciliation, but owns no implementation. This issue is the missing provenance substrate those children stop short of; it does not re-open umbrella policy. | | #949 | open | Authoritative fleet inventory and duplicate-process detection — a snapshot of the *current* population. Its non-goals name "Reconstructing historical restart provenance" explicitly, so it disclaims this scope by construction. This issue records the inventory verdict as of an operation's completion; it does not implement the inventory. | | #950 | open | Controller role metadata and capability routing. Its non-goals name "Restart or synchronization provenance" explicitly. This issue records the controller-role validation result in a receipt; it does not repair the metadata. | | #948 | open | Defines client/session/generation ownership identity and lease fencing so concurrent LLM clients are not falsely blocked. It supplies the identity vocabulary a receipt cites; it stores no operation history. | | #900 | open | Cohort lifecycle supervision: bound the eligible population, drain and reap superseded cohorts. A mutating supervisory transition whose status surface describes *currently resident* cohorts. It does not persist a record of a completed synchronization or restart, and does not cover control-checkout fast-forward at all. | | #669 | closed | Recovery-ladder selection — prefer scoped component recovery over full-server resets. Chooses which operation to run; records nothing about the run. | | #591 | closed | Auto-restart namespaces when master advances. The trigger side of the same lifecycle; it produced no durable receipt, which is part of why the audit had nothing to retrieve. | | #559 | closed | Persists MCP workflow proof and review-decision state across daemon processes. Same durability technique, different subject: review/workflow decisions, not control-plane synchronization and restart operations. | ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement durable, append-only synchronization and restart operation receipts with read-only native retrieval on gitea-controller and gitea-reconciler per the acceptance criteria NEXT_PROMPT: Role: AUTHOR Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools Issue: this issue 1. Native MCP only; gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/. 2. Implement the receipt record, its phase and terminal-status model, and its persistence so it survives client sessions and MCP server restarts; make terminal receipts immutable or append-only. 3. Emit receipts from the sanctioned synchronization, drain, restart, reconnect, and reconciliation paths themselves; never from caller-supplied process lists or success claims. 4. Expose read-only retrieval by operation ID, most-recent-per-fleet, bounded pagination, and type/status filtering on gitea-controller and gitea-reconciler; return a provenance verdict for mutation-sensitive gates; fail closed on incomplete evidence. 5. Add regression tests for successful, no-op, blocked, partial, failed, aborted, incomplete-evidence, and post-restart retrieval cases. 6. Update canonical workflow documentation on current parity vs historical provenance. 7. Open a PR with Closes; stop; hand off to an independent reviewer. ```