# Instance-level fleet identity and health snapshots

Issue **#978**. Companion primitives: **#948** (worker ownership), **#975**
(heartbeat lifecycle), **#951** (restart receipts).

## Why this exists

The multi-instance fleet gate (#963) needs a *native* answer to:

* which application launches are live;
* which of the five namespace workers belong to each launch;
* whether two Codex (or Claude, Grok, …) launches are distinct;
* whether any live collision is real rather than a shared client type.

Process tables, PID proximity, configuration files, and direct SQLite access
are **not** production evidence. The sanctioned surface is the read-only MCP
tool `gitea_snapshot_instance_fleet` on **controller** and **reconciler**
namespaces.

## Identity hierarchy

| Identity | Scope | Who generates it | Lifetime |
| --- | --- | --- | --- |
| `client_type` | Application family (`codex`, `claude_code`, `gemini`, `grok`, …) | Launcher sets `GITEA_MCP_CLIENT` | Stable for the product |
| `client_instance_id` | One running application launch | **Trusted launcher**, once per launch, as `GITEA_MCP_CLIENT_INSTANCE` | Fresh launch → new ID; reconnect of same launch → same ID; full app restart → new ID |
| `fleet_run_id` | Operator-approved enrollment / canary cohort | Operator / controller sets `GITEA_MCP_FLEET_RUN_ID` | Duration of the approved rollout |
| `namespace` | `author` \| `reviewer` \| `merger` \| `controller` \| `reconciler` | Profile / MCP server binding | Process lifetime |
| `worker_id` / `worker_identity` | One namespace worker process | Runtime registry at first registration | New on worker restart; not reused |
| `session_id` | Client session ownership | Launcher `GITEA_MCP_CLIENT_SESSION` or runtime | Session lifetime |
| `generation_id` | One daemon launch | Runtime at process boot | New on process restart |
| `process_identity` / PID | OS process | Runtime | Process lifetime |

### Rules

1. Multiple active instances **may** share the same `client_type`.
2. Every application launch receives a **distinct** `client_instance_id`.
3. All namespace workers of one launch report the **same**
   `client_instance_id`. For a whole-fleet launch that is five workers; for a
   project-scoped launch it is exactly the namespaces that launch started
   (#985).
4. Each namespace worker has a **distinct** `worker_identity`, process
   identity, generation, and PID.
5. Instance identity is **never** inferred from PID proximity, timestamps, or
   client type alone.
6. Live reuse of one `client_instance_id` with conflicting workers fails closed.
7. Sharing only a profile or `client_type` is **not** a duplicate.

This deliberately **replaces** any permanent `exactly_one_per_profile` fleet
model (#949 assumption) as the operating rule for multi-instance fleets.

## Project-scoped launches and the runnable entry point (#985)

Not every project exposes all five namespaces. A project-scoped configuration
such as Weekly Briefings intentionally has **author, reviewer, and merger
only**, and has no controller or reconciler profile to supply. Requiring all
five would force operators to invent profiles that must not exist, so the
launcher accepts an explicitly validated subset.

### Running a launch

```bash
python3 -m mcp_application_launcher \
    --client claude_code \
    --namespaces author,reviewer,merger \
    --profile author=prgs-author \
    --profile reviewer=prgs-reviewer \
    --profile merger=prgs-merger
```

That mints one trusted `client_instance_id`, writes a per-launch `mcpServers`
config, and executes:

```text
claude --mcp-config <per-launch.json> --strict-mcp-config
```

Add `--dry-run` to print the plan (namespaces, excluded namespaces, minted ID,
argv, config path) as JSON and start nothing. Omit `--namespaces` to launch all
five exactly as before. Arguments after the flags are forwarded to the client.

Other supported clients register one entry in
`mcp_application_launcher.CLIENT_LAUNCH_SPECS`; because the argv builder only
ever receives a config this module wrote, every client necessarily goes through
the same mint-once/propagate-to-all mechanism.

### Why the config is per-launch and never `.mcp.json`

Persisting a trusted ID into a shared, reused `.mcp.json` would give two
concurrent sessions **the same** trusted identity — precisely the reuse case
the duplicate gate must reject. The launcher therefore writes a fresh
owner-readable-only (`0600`) config per launch. Do not commit one, and do not
copy a minted `GITEA_MCP_CLIENT_INSTANCE` into any checked-in configuration.

### Provenance sealing

The `inst-…` format is public and reproducible, so format alone cannot
establish trust: anyone able to set one environment variable could otherwise
hand-write a valid-looking ID and be believed. Trust therefore requires **both**
the format and `GITEA_MCP_INSTANCE_PROVENANCE=trusted_launcher`, which only the
launcher writes. A well-formed but unsealed identity is classified
`unsealed_launcher` and **fails closed** — it is still reported for diagnosis,
but never authorizes trusted attribution. Manually asserted trust is not
possible.

### Migration and coordinated relaunch

Existing configurations that predate this work set no instance key at all, so
their workers register under `legacy-pid-*` (`legacy_incomplete`) and remain
mutually indistinguishable when two launches share a profile.

Migrating is a **coordinated relaunch**, not an in-place edit — a running
worker cannot acquire an identity it was not started with:

1. Stop every LLM client currently running Gitea MCP workers. A single
   surviving legacy cohort keeps the fleet ambiguous.
2. Relaunch each application through the command above.
3. Repeat per application. Concurrent launches are expected and safe: each
   receives its own trusted ID.

### Post-launch verification

* `gitea_get_runtime_context` → `provenance_assessment.attachment` should show
  an `inst-…` `client_instance_id` with
  `instance_id_provenance: trusted_launcher`, not `legacy-pid-*` /
  `legacy_incomplete`.
* Every namespace of the same launch must report that **same** ID; two
  different launches must report different IDs.
* `gitea_resolve_task_capability` should return
  `exact_safe_next_action: "None; ready for operations."` with no
  `blocker_kind`. A `runtime_reconnect_required` naming duplicate PIDs per
  profile means at least one cohort is still on legacy identity, or a second
  cohort is genuinely running.
* `mcp_application_launcher.collect_instance_ids_from_mcp_servers(servers,
  namespaces=[...])` returns `shared_single_trusted_id` for a built config, and
  reports `missing_servers` when an expected namespace entry is absent.

## How five workers join one instance

1. The host starts one application instance (for example one Codex session).
2. The **production application launcher**
   (`mcp_application_launcher.build_application_mcp_servers` /
   `gitea_config.multi_namespace_launcher_entries`) mints exactly one
   `client_instance_id` via `mcp_fleet_snapshot.generate_client_instance_id`
   and injects the same env into every namespace worker:

   ```text
   GITEA_MCP_CLIENT=<client_type>
   GITEA_MCP_CLIENT_INSTANCE=<client_instance_id>
   GITEA_MCP_INSTANCE_PROVENANCE=trusted_launcher
   GITEA_MCP_CLIENT_SESSION=<session_id>          # optional but recommended
   GITEA_MCP_FLEET_RUN_ID=<enrollment id>         # when on an approved canary
   GITEA_CLIENT_MANAGED=1
   GITEA_MCP_PROFILE=<role-profile>
   ```

3. The MCP client starts the five namespace processes (`gitea-author`,
   `gitea-reviewer`, `gitea-merger`, `gitea-controller`, `gitea-reconciler`)
   from that generated `mcpServers` map — each entry carries the **same**
   instance ID.
4. Each worker registers once into the worker registry with its own
   `worker_identity`, `namespace`, `generation_id`, and PID, but the shared
   `client_instance_id`. Workers never mint a trusted instance ID themselves.

Only IDs matching the launcher format `inst-<client>-<timestamp>-<digest>`
are trusted. Missing, `legacy-pid-*` / `pid-*` placeholders, and ordinary
user-supplied strings fail closed as untrusted and cannot authorize
multi-instance fleet mutation safety.

Worker reconnect (same process, same registration) keeps the instance ID.
Worker restart (new process) mints a new worker identity and generation but
must still receive the same `GITEA_MCP_CLIENT_INSTANCE` from the parent
application if it is the same launch. Full application restart mints a new
`client_instance_id` (call the launcher without a prior ID).

### Mutation gate (instance-aware)

The capability / runtime diagnostic gate
(`_check_mcp_runtimes_diagnostics`) is **instance-aware**:

* Two legitimate application instances that share a profile (same
  `GITEA_MCP_PROFILE`) are allowed when each has a distinct trusted
  `client_instance_id`.
* More than one live worker for the same
  `(client_instance_id, profile/namespace)` fails closed as a duplicate
  namespace worker.
* Multiple processes sharing a profile **without** trusted instance
  evidence still fail closed (indistinguishable from a duplicate).

## Approved fleet manifest

An operator (or controller enrollment step) obtains an approved manifest as a
list of expected instances, for example:

```json
{
  "instances": [
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120000Z-abc123def456",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    },
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120100Z-fed654cba321",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    }
  ]
}
```

Pass that list as `expected_manifest` to `gitea_snapshot_instance_fleet`.
When a manifest is supplied:

* instances on the manifest but not live → `missing_expected`;
* live instances not on the manifest → `unmanifested`;
* both are **active blockers** for fleet-gate safety.

Without a manifest, the snapshot still enumerates the live fleet and classifies
identity collisions; it does not invent enrollment policy.

## Snapshot consistency

Each snapshot includes:

* `snapshot_at` — UTC timestamp;
* `consistency_token` / `registry_revision` — content digest over worker
  identity, instance id, status, heartbeat, fencing, and generation.

Two successive snapshots can prove heartbeat continuity via
`mcp_fleet_snapshot.compare_snapshot_heartbeats`.

## Classification (active vs historical)

**Active blockers** (make `live_fleet_safe=false`):

* missing expected instance;
* unmanifested extra instance;
* duplicate namespace worker within one instance;
* live `client_instance_id` collision;
* reused worker / session / generation / process / PID / fencing identity;
* orphaned or unowned workers;
* unknown client;
* foreign-repository workers;
* old-revision workers;
* stale workers still marked active;
* legacy incomplete instance identity.

**Historical dead rows** (`status` released/superseded) are reported as
`historical_dead` findings with `active_blocker=false`. They never
automatically make the live fleet unsafe.

## Fail-closed behaviour and recovery

| Condition | Mutation safety | Diagnostic reads | Recovery |
| --- | --- | --- | --- |
| Two instances, same `client_type`, distinct IDs | Safe (if otherwise healthy) | Available | None needed |
| Live reuse of one `client_instance_id` | Unsafe | Available | Stop the colliding launch or re-issue a distinct ID |
| Two workers, same namespace, one instance | Unsafe | Available | Stop the extra worker |
| Legacy registration without trusted instance ID | Unsafe for fleet mutation | Available | Relaunch with launcher-issued `GITEA_MCP_CLIENT_INSTANCE` |
| Historical dead row only | Does not block alone | Available | No action required for fleet safety |
| Registry unavailable | Snapshot fails closed | N/A | Repair registry path / reconnect namespaces |

Incomplete or untrusted instance identity **cannot** authorize unsafe
mutation. Diagnostic reads remain available where `gitea.read` allows.

## Permissions

* **Allowed:** `controller`, `reconciler` with `gitea.read`.
* **Denied:** author, reviewer, merger (even with `gitea.read` for other tools).
* **No new mutation permissions** are granted to any role.

## Related surfaces

* `mcp_worker_identity` — registry, worker identity, heartbeats (#948, #975).
* `mcp_fleet_snapshot` — pure snapshot + classification (#978).
* `gitea_snapshot_instance_fleet` — sanctioned MCP tool (#978).
* `gitea_get_runtime_context` — single-process view (not fleet-wide).

## Non-goals

* Starting the #963 canary.
* Purging historical registry rows.
* Preserving “exactly one process per profile” as the permanent model.
* Using shell / process-table / SQLite inspection as production fleet evidence.