feat(controller): expose instance-level fleet identity and health snapshots
Add a pure fleet snapshot assessor and a read-only controller/reconciler MCP tool so multi-instance fleets can be enumerated by client_instance_id without treating shared client_type as duplication. Trusted launcher instance IDs, classification (missing/unmanifested/collisions/historical), registry schema extensions, docs, and regression tests for #978. Closes #978 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,171 @@
|
||||
# Instance-level fleet identity and health snapshots
|
||||
|
||||
Issue **#978**. Companion primitives: **#948** (worker ownership), **#975**
|
||||
(heartbeat lifecycle), **#951** (restart receipts).
|
||||
|
||||
## Why this exists
|
||||
|
||||
The multi-instance fleet gate (#963) needs a *native* answer to:
|
||||
|
||||
* which application launches are live;
|
||||
* which of the five namespace workers belong to each launch;
|
||||
* whether two Codex (or Claude, Grok, …) launches are distinct;
|
||||
* whether any live collision is real rather than a shared client type.
|
||||
|
||||
Process tables, PID proximity, configuration files, and direct SQLite access
|
||||
are **not** production evidence. The sanctioned surface is the read-only MCP
|
||||
tool `gitea_snapshot_instance_fleet` on **controller** and **reconciler**
|
||||
namespaces.
|
||||
|
||||
## Identity hierarchy
|
||||
|
||||
| Identity | Scope | Who generates it | Lifetime |
|
||||
| --- | --- | --- | --- |
|
||||
| `client_type` | Application family (`codex`, `claude_code`, `gemini`, `grok`, …) | Launcher sets `GITEA_MCP_CLIENT` | Stable for the product |
|
||||
| `client_instance_id` | One running application launch | **Trusted launcher**, once per launch, as `GITEA_MCP_CLIENT_INSTANCE` | Fresh launch → new ID; reconnect of same launch → same ID; full app restart → new ID |
|
||||
| `fleet_run_id` | Operator-approved enrollment / canary cohort | Operator / controller sets `GITEA_MCP_FLEET_RUN_ID` | Duration of the approved rollout |
|
||||
| `namespace` | `author` \| `reviewer` \| `merger` \| `controller` \| `reconciler` | Profile / MCP server binding | Process lifetime |
|
||||
| `worker_id` / `worker_identity` | One namespace worker process | Runtime registry at first registration | New on worker restart; not reused |
|
||||
| `session_id` | Client session ownership | Launcher `GITEA_MCP_CLIENT_SESSION` or runtime | Session lifetime |
|
||||
| `generation_id` | One daemon launch | Runtime at process boot | New on process restart |
|
||||
| `process_identity` / PID | OS process | Runtime | Process lifetime |
|
||||
|
||||
### Rules
|
||||
|
||||
1. Multiple active instances **may** share the same `client_type`.
|
||||
2. Every application launch receives a **distinct** `client_instance_id`.
|
||||
3. All five namespace workers of one launch report the **same**
|
||||
`client_instance_id`.
|
||||
4. Each namespace worker has a **distinct** `worker_identity`, process
|
||||
identity, generation, and PID.
|
||||
5. Instance identity is **never** inferred from PID proximity, timestamps, or
|
||||
client type alone.
|
||||
6. Live reuse of one `client_instance_id` with conflicting workers fails closed.
|
||||
7. Sharing only a profile or `client_type` is **not** a duplicate.
|
||||
|
||||
This deliberately **replaces** any permanent `exactly_one_per_profile` fleet
|
||||
model (#949 assumption) as the operating rule for multi-instance fleets.
|
||||
|
||||
## How five workers join one instance
|
||||
|
||||
1. The host starts one application instance (for example one Codex session).
|
||||
2. The trusted launcher mints `client_instance_id` (see
|
||||
`mcp_fleet_snapshot.generate_client_instance_id`) and injects:
|
||||
|
||||
```text
|
||||
GITEA_MCP_CLIENT=<client_type>
|
||||
GITEA_MCP_CLIENT_INSTANCE=<client_instance_id>
|
||||
GITEA_MCP_CLIENT_SESSION=<session_id> # optional but recommended
|
||||
GITEA_MCP_FLEET_RUN_ID=<enrollment id> # when on an approved canary
|
||||
```
|
||||
|
||||
3. The launcher starts the five MCP namespace processes (or attaches five
|
||||
role profiles) with **that same environment**.
|
||||
4. Each worker registers once into the worker registry with its own
|
||||
`worker_identity`, `namespace`, `generation_id`, and PID, but the shared
|
||||
`client_instance_id`.
|
||||
|
||||
Worker reconnect (same process, same registration) keeps the instance ID.
|
||||
Worker restart (new process) mints a new worker identity and generation but
|
||||
must still receive the same `GITEA_MCP_CLIENT_INSTANCE` from the parent
|
||||
application if it is the same launch. Full application restart mints a new
|
||||
`client_instance_id`.
|
||||
|
||||
## Approved fleet manifest
|
||||
|
||||
An operator (or controller enrollment step) obtains an approved manifest as a
|
||||
list of expected instances, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"instances": [
|
||||
{
|
||||
"client_type": "codex",
|
||||
"client_instance_id": "inst-codex-20260730T120000Z-abc123def456",
|
||||
"fleet_run_id": "canary-963-2026-07-30",
|
||||
"namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
|
||||
},
|
||||
{
|
||||
"client_type": "codex",
|
||||
"client_instance_id": "inst-codex-20260730T120100Z-fed654cba321",
|
||||
"fleet_run_id": "canary-963-2026-07-30",
|
||||
"namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Pass that list as `expected_manifest` to `gitea_snapshot_instance_fleet`.
|
||||
When a manifest is supplied:
|
||||
|
||||
* instances on the manifest but not live → `missing_expected`;
|
||||
* live instances not on the manifest → `unmanifested`;
|
||||
* both are **active blockers** for fleet-gate safety.
|
||||
|
||||
Without a manifest, the snapshot still enumerates the live fleet and classifies
|
||||
identity collisions; it does not invent enrollment policy.
|
||||
|
||||
## Snapshot consistency
|
||||
|
||||
Each snapshot includes:
|
||||
|
||||
* `snapshot_at` — UTC timestamp;
|
||||
* `consistency_token` / `registry_revision` — content digest over worker
|
||||
identity, instance id, status, heartbeat, fencing, and generation.
|
||||
|
||||
Two successive snapshots can prove heartbeat continuity via
|
||||
`mcp_fleet_snapshot.compare_snapshot_heartbeats`.
|
||||
|
||||
## Classification (active vs historical)
|
||||
|
||||
**Active blockers** (make `live_fleet_safe=false`):
|
||||
|
||||
* missing expected instance;
|
||||
* unmanifested extra instance;
|
||||
* duplicate namespace worker within one instance;
|
||||
* live `client_instance_id` collision;
|
||||
* reused worker / session / generation / process / PID / fencing identity;
|
||||
* orphaned or unowned workers;
|
||||
* unknown client;
|
||||
* foreign-repository workers;
|
||||
* old-revision workers;
|
||||
* stale workers still marked active;
|
||||
* legacy incomplete instance identity.
|
||||
|
||||
**Historical dead rows** (`status` released/superseded) are reported as
|
||||
`historical_dead` findings with `active_blocker=false`. They never
|
||||
automatically make the live fleet unsafe.
|
||||
|
||||
## Fail-closed behaviour and recovery
|
||||
|
||||
| Condition | Mutation safety | Diagnostic reads | Recovery |
|
||||
| --- | --- | --- | --- |
|
||||
| Two instances, same `client_type`, distinct IDs | Safe (if otherwise healthy) | Available | None needed |
|
||||
| Live reuse of one `client_instance_id` | Unsafe | Available | Stop the colliding launch or re-issue a distinct ID |
|
||||
| Two workers, same namespace, one instance | Unsafe | Available | Stop the extra worker |
|
||||
| Legacy registration without trusted instance ID | Unsafe for fleet mutation | Available | Relaunch with launcher-issued `GITEA_MCP_CLIENT_INSTANCE` |
|
||||
| Historical dead row only | Does not block alone | Available | No action required for fleet safety |
|
||||
| Registry unavailable | Snapshot fails closed | N/A | Repair registry path / reconnect namespaces |
|
||||
|
||||
Incomplete or untrusted instance identity **cannot** authorize unsafe
|
||||
mutation. Diagnostic reads remain available where `gitea.read` allows.
|
||||
|
||||
## Permissions
|
||||
|
||||
* **Allowed:** `controller`, `reconciler` with `gitea.read`.
|
||||
* **Denied:** author, reviewer, merger (even with `gitea.read` for other tools).
|
||||
* **No new mutation permissions** are granted to any role.
|
||||
|
||||
## Related surfaces
|
||||
|
||||
* `mcp_worker_identity` — registry, worker identity, heartbeats (#948, #975).
|
||||
* `mcp_fleet_snapshot` — pure snapshot + classification (#978).
|
||||
* `gitea_snapshot_instance_fleet` — sanctioned MCP tool (#978).
|
||||
* `gitea_get_runtime_context` — single-process view (not fleet-wide).
|
||||
|
||||
## Non-goals
|
||||
|
||||
* Starting the #963 canary.
|
||||
* Purging historical registry rows.
|
||||
* Preserving “exactly one process per profile” as the permanent model.
|
||||
* Using shell / process-table / SQLite inspection as production fleet evidence.
|
||||
Reference in New Issue
Block a user