feat(controller): expose native instance-level fleet identity and health snapshots #978

New Issue

2026-07-30T01:57:00-05:00

jcwalker3 commented

2026-07-30 01:57:00 -05:00

Related: #963, #949, #951, #948, #900, #975

Problem

Issue #963 cannot complete its simultaneous multi-instance fleet-readiness gate because the sanctioned controller/reconciler interface cannot enumerate the complete active fleet or prove which namespace workers belong to each running LLM application instance.

The current duplicate-process model groups workers primarily by profile or generalized client type. This incorrectly treats legitimate concurrent clients as duplicate MCP processes.

A client type such as codex, claude_code, gemini, grok, or kimi_k is not a unique client identity. Multiple concurrent launches of the same application type must also be supported.

For example, two independent Codex launches must be recognized as two valid instances — not collapsed into one Codex participant or rejected merely because they share client_type: codex.

Process inspection, PID proximity, timestamps, configuration files, and direct registry or SQLite access are not authoritative substitutes for native control-plane evidence.

Related issues include #949, #951, #948, #900, and #975, but none supplies the complete instance-level native observability required by #963.

Required identity model

Expose these separate identities:

client_type: application family, such as codex
client_instance_id: unique and stable identity for one running application instance
worker_id: unique identity for one namespace worker
namespace: author, reviewer, merger, controller, or reconciler
fleet_run_id or equivalent: operator-approved enrollment identity for a rollout or canary

Requirements:

Multiple active instances may share the same client_type.
Every application launch receives a distinct client_instance_id.
The trusted launcher or runtime must establish the instance ID.
Instance identity must not be inferred from PID proximity, timestamps, or client type.
All five namespace workers belonging to one application launch report the same client_instance_id.
Each namespace worker has a distinct worker_id, process identity, generation identity, and PID.
Restart and reconnect semantics define whether an instance ID is preserved or replaced.
Collision or reuse of a live client_instance_id fails closed.
Two instances sharing only a client type are not duplicates.

Native fleet snapshot

Add a sanctioned, read-only controller/reconciler capability that returns an authoritative point-in-time fleet snapshot.

Each active worker row must include:

snapshot timestamp and registry revision or consistency token;
client type;
client instance ID;
fleet-run or enrollment identity;
namespace;
profile;
declared role;
authenticated account;
worker ID;
session identity;
generation identity;
process identity and PID;
repository binding;
startup, loaded, parity, and live revisions;
trusted runtime provenance;
heartbeat registration, supervision, age, and freshness;
fencing epoch;
ownership state;
conflicting live sessions;
stale state;
restart requirement;
mutation-safety state.

The native result must classify:

expected and enrolled instances;
missing expected instances;
unmanifested instances;
duplicate namespace workers within one instance;
cross-instance identity, ownership, session, or fencing collisions;
orphaned or unowned workers;
unknown clients;
foreign-repository workers;
old-revision workers;
expired historical rows, reported separately from active blockers.

Acceptance criteria

Two simultaneous Codex instances with different client_instance_id values are accepted independently.
Their ten namespace workers are attributed to the correct two instances.
Sharing only client_type: codex is not classified as duplication.
Two active launches claiming the same client_instance_id fail closed.
More than one active worker for the same namespace within one instance fails closed.
Reused worker, session, generation, process, PID, ownership, or fencing identity is detected.
An extra, unenrolled application instance is classified as unmanifested.
Historical dead rows are distinguished from live conflicts.
Two native snapshots can prove heartbeat continuity and stable ownership.
The fleet gate requires no shell, process-table, SQLite, source, or configuration inspection.
Controller and reconciler roles can access the read-only snapshot without gaining unrelated mutation permissions.
Tests cover:
- multiple instances of the same client type;
- mixed client types;
- duplicate namespace workers;
- instance-ID collision;
- stale and orphaned workers;
- reconnects and restarts;
- unknown clients;
- foreign repositories;
- old revisions;
- historical dead rows.
Documentation defines the identity lifecycle and explains how an operator obtains an exact approved instance manifest.
Issue #963 remains blocked until this capability is merged and its multi-instance fleet gate and canary pass natively.

Scope boundaries

This issue must not:

start the #963 canary;
change issue #963's labels or state;
purge historical registry records;
prohibit multiple instances of the same LLM type;
preserve "exactly one process per profile" as the permanent operating model;
rely on direct database or process inspection as production evidence.

Duplicate verdict

NOT a duplicate. Verified against live open and closed issue state before creation.

Issue	State	Why it does not cover this
#949	open	Its required behavior is explicitly built on `exactly_one_per_profile`, and its acceptance criterion 2 mandates that two processes sharing a profile be reported as a duplicate that fails the mutation gate. That is precisely the model this issue must replace. #949 also has no `client_instance_id` axis, so it cannot attribute five namespace workers to a specific application launch or distinguish two concurrent launches of one client type.
#951	open	Persists synchronization and restart receipts — provenance of past events. It does not enumerate the live fleet or classify instance-level identity collisions.
#948	closed	Defines the client/session/generation ownership model consumed per serving process. It supplies identity primitives; it does not provide a fleet-wide, multi-instance observation surface.
#900	open	Owns cohort lifecycle supervision — draining and reaping superseded cohorts, a mutating transition, framed per profile-cohort rather than per application instance.
#975	closed	Allowlisted client-identity environment keys and refreshed worker registrations so a single runtime reports its own identity. Per-process recognition, not fleet enumeration.
#977	open	Static configuration discovery, audit, and repair across six client configuration files. It inspects configuration at rest, not the live runtime fleet.

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement the read-only native instance-level fleet identity and health snapshot capability on gitea-controller and gitea-reconciler per the acceptance criteria
NEXT_PROMPT:
Role: AUTHOR
Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools
Issue: this issue

1. Native MCP only; gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/.
2. Implement the separate identity model (client_type, client_instance_id, worker_id, namespace, fleet_run_id) so multiple instances of one client type are first-class, and instance identity is established by the trusted launcher/runtime rather than inferred.
3. Implement the read-only fleet snapshot with the full per-worker row and the full classification set; fail closed on instance-ID collision, duplicate namespace worker within an instance, and incomplete evidence.
4. Expose it on gitea-controller and gitea-reconciler as strictly read-only, granting no unrelated mutation permission.
5. Add regression tests for multiple instances of one client type, mixed client types, duplicate namespace workers, instance-ID collision, stale/orphaned workers, reconnects and restarts, unknown clients, foreign repositories, old revisions, and historical dead rows.
6. Document the identity lifecycle and how an operator obtains an exact approved instance manifest.
7. Open a PR with Closes; stop; hand off to an independent reviewer.

Do not do: start the #963 canary; modify #963 or #949 labels or state; purge historical registry records; forbid multiple instances of one LLM type; keep "exactly one process per profile" as the permanent model; use shell, process-table, SQLite, source, or configuration inspection as production evidence.

Related: #963, #949, #951, #948, #900, #975 ## Problem Issue #963 cannot complete its simultaneous multi-instance fleet-readiness gate because the sanctioned controller/reconciler interface cannot enumerate the complete active fleet or prove which namespace workers belong to each running LLM application instance. The current duplicate-process model groups workers primarily by profile or generalized client type. This incorrectly treats legitimate concurrent clients as duplicate MCP processes. A client type such as `codex`, `claude_code`, `gemini`, `grok`, or `kimi_k` is not a unique client identity. Multiple concurrent launches of the same application type must also be supported. For example, two independent Codex launches must be recognized as two valid instances — not collapsed into one Codex participant or rejected merely because they share `client_type: codex`. Process inspection, PID proximity, timestamps, configuration files, and direct registry or SQLite access are not authoritative substitutes for native control-plane evidence. Related issues include #949, #951, #948, #900, and #975, but none supplies the complete instance-level native observability required by #963. ## Required identity model Expose these separate identities: * `client_type`: application family, such as `codex` * `client_instance_id`: unique and stable identity for one running application instance * `worker_id`: unique identity for one namespace worker * `namespace`: author, reviewer, merger, controller, or reconciler * `fleet_run_id` or equivalent: operator-approved enrollment identity for a rollout or canary Requirements: 1. Multiple active instances may share the same `client_type`. 2. Every application launch receives a distinct `client_instance_id`. 3. The trusted launcher or runtime must establish the instance ID. 4. Instance identity must not be inferred from PID proximity, timestamps, or client type. 5. All five namespace workers belonging to one application launch report the same `client_instance_id`. 6. Each namespace worker has a distinct `worker_id`, process identity, generation identity, and PID. 7. Restart and reconnect semantics define whether an instance ID is preserved or replaced. 8. Collision or reuse of a live `client_instance_id` fails closed. 9. Two instances sharing only a client type are not duplicates. ## Native fleet snapshot Add a sanctioned, read-only controller/reconciler capability that returns an authoritative point-in-time fleet snapshot. Each active worker row must include: * snapshot timestamp and registry revision or consistency token; * client type; * client instance ID; * fleet-run or enrollment identity; * namespace; * profile; * declared role; * authenticated account; * worker ID; * session identity; * generation identity; * process identity and PID; * repository binding; * startup, loaded, parity, and live revisions; * trusted runtime provenance; * heartbeat registration, supervision, age, and freshness; * fencing epoch; * ownership state; * conflicting live sessions; * stale state; * restart requirement; * mutation-safety state. The native result must classify: * expected and enrolled instances; * missing expected instances; * unmanifested instances; * duplicate namespace workers within one instance; * cross-instance identity, ownership, session, or fencing collisions; * orphaned or unowned workers; * unknown clients; * foreign-repository workers; * old-revision workers; * expired historical rows, reported separately from active blockers. ## Acceptance criteria 1. Two simultaneous Codex instances with different `client_instance_id` values are accepted independently. 2. Their ten namespace workers are attributed to the correct two instances. 3. Sharing only `client_type: codex` is not classified as duplication. 4. Two active launches claiming the same `client_instance_id` fail closed. 5. More than one active worker for the same namespace within one instance fails closed. 6. Reused worker, session, generation, process, PID, ownership, or fencing identity is detected. 7. An extra, unenrolled application instance is classified as unmanifested. 8. Historical dead rows are distinguished from live conflicts. 9. Two native snapshots can prove heartbeat continuity and stable ownership. 10. The fleet gate requires no shell, process-table, SQLite, source, or configuration inspection. 11. Controller and reconciler roles can access the read-only snapshot without gaining unrelated mutation permissions. 12. Tests cover: * multiple instances of the same client type; * mixed client types; * duplicate namespace workers; * instance-ID collision; * stale and orphaned workers; * reconnects and restarts; * unknown clients; * foreign repositories; * old revisions; * historical dead rows. 13. Documentation defines the identity lifecycle and explains how an operator obtains an exact approved instance manifest. 14. Issue #963 remains blocked until this capability is merged and its multi-instance fleet gate and canary pass natively. ## Scope boundaries This issue must not: * start the #963 canary; * change issue #963's labels or state; * purge historical registry records; * prohibit multiple instances of the same LLM type; * preserve "exactly one process per profile" as the permanent operating model; * rely on direct database or process inspection as production evidence. ## Duplicate verdict NOT a duplicate. Verified against live open and closed issue state before creation. | Issue | State | Why it does not cover this | | --- | --- | --- | | #949 | open | Its required behavior is explicitly built on `exactly_one_per_profile`, and its acceptance criterion 2 mandates that two processes sharing a profile be reported as a duplicate that fails the mutation gate. That is precisely the model this issue must replace. #949 also has no `client_instance_id` axis, so it cannot attribute five namespace workers to a specific application launch or distinguish two concurrent launches of one client type. | | #951 | open | Persists synchronization and restart receipts — provenance of past events. It does not enumerate the live fleet or classify instance-level identity collisions. | | #948 | closed | Defines the client/session/generation ownership model consumed per serving process. It supplies identity primitives; it does not provide a fleet-wide, multi-instance observation surface. | | #900 | open | Owns cohort lifecycle supervision — draining and reaping superseded cohorts, a mutating transition, framed per profile-cohort rather than per application instance. | | #975 | closed | Allowlisted client-identity environment keys and refreshed worker registrations so a single runtime reports its own identity. Per-process recognition, not fleet enumeration. | | #977 | open | Static configuration discovery, audit, and repair across six client configuration files. It inspects configuration at rest, not the live runtime fleet. | ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement the read-only native instance-level fleet identity and health snapshot capability on gitea-controller and gitea-reconciler per the acceptance criteria NEXT_PROMPT: Role: AUTHOR Repo: prgs / Scaled-Tech-Consulting / Gitea-Tools Issue: this issue 1. Native MCP only; gitea_whoami + gitea_resolve_task_capability(work_issue) on prgs-author; bind an issue-backed worktree under branches/. 2. Implement the separate identity model (client_type, client_instance_id, worker_id, namespace, fleet_run_id) so multiple instances of one client type are first-class, and instance identity is established by the trusted launcher/runtime rather than inferred. 3. Implement the read-only fleet snapshot with the full per-worker row and the full classification set; fail closed on instance-ID collision, duplicate namespace worker within an instance, and incomplete evidence. 4. Expose it on gitea-controller and gitea-reconciler as strictly read-only, granting no unrelated mutation permission. 5. Add regression tests for multiple instances of one client type, mixed client types, duplicate namespace workers, instance-ID collision, stale/orphaned workers, reconnects and restarts, unknown clients, foreign repositories, old revisions, and historical dead rows. 6. Document the identity lifecycle and how an operator obtains an exact approved instance manifest. 7. Open a PR with Closes; stop; hand off to an independent reviewer. Do not do: start the #963 canary; modify #963 or #949 labels or state; purge historical registry records; forbid multiple instances of one LLM type; keep "exactly one process per profile" as the permanent model; use shell, process-table, SQLite, source, or configuration inspection as production evidence. ```

jcwalker3 added the mcp testing type:feature status:ready workflow-hardening concurrency safety observability mcp-health control-plane labels 2026-07-30 01:57:01 -05:00

jcwalker3 added status:in-progress and removed status:ready labels 2026-07-30 02:03:00 -05:00

jcwalker3 commented

2026-07-30 02:03:01 -05:00

Issue claim heartbeat

kind: claim
issue: feat(controller): expose native instance-level fleet identity and health snapshots (#978)
branch: feat/issue-978-instance-fleet-snapshot
phase: claimed
profile: prgs-author
pr: none
blocker: none
next_action: create worktree and begin implementation

**Issue claim heartbeat** - kind: claim - issue: #978 - branch: feat/issue-978-instance-fleet-snapshot - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation

jcwalker3 referenced this issue from a commit

2026-07-30 02:14:39 -05:00

feat(controller): expose instance-level fleet identity and health snapshots

jcwalker3 referenced a pull request that will close this issue

2026-07-30 02:16:21 -05:00

feat(controller): expose instance-level fleet identity and health snapshots (Closes #978) #979

jcwalker3 added status:pr-open and removed status:in-progress labels 2026-07-30 02:16:22 -05:00

sysadmin referenced this issue

2026-07-30 04:37:53 -05:00

feat(controller): expose instance-level fleet identity and health snapshots (Closes #978) #979

sysadmin referenced this issue

2026-07-30 06:22:04 -05:00