Gitea-Tools/docs/instance-fleet-identity.md

# Instance-level fleet identity and health snapshots

Issue **#978**. Companion primitives: **#948** (worker ownership), **#975**
(heartbeat lifecycle), **#951** (restart receipts).

## Why this exists

The multi-instance fleet gate (#963) needs a *native* answer to:

* which application launches are live;
* which of the five namespace workers belong to each launch;
* whether two Codex (or Claude, Grok, …) launches are distinct;
* whether any live collision is real rather than a shared client type.

Process tables, PID proximity, configuration files, and direct SQLite access
are **not** production evidence. The sanctioned surface is the read-only MCP
tool `gitea_snapshot_instance_fleet` on **controller** and **reconciler**
namespaces.

## Identity hierarchy

| Identity | Scope | Who generates it | Lifetime |
| --- | --- | --- | --- |
| `client_type` | Application family (`codex`, `claude_code`, `gemini`, `grok`, …) | Launcher sets `GITEA_MCP_CLIENT` | Stable for the product |
| `client_instance_id` | One running application launch | **Trusted launcher**, once per launch, as `GITEA_MCP_CLIENT_INSTANCE` | Fresh launch → new ID; reconnect of same launch → same ID; full app restart → new ID |
| `fleet_run_id` | Operator-approved enrollment / canary cohort | Operator / controller sets `GITEA_MCP_FLEET_RUN_ID` | Duration of the approved rollout |
| `namespace` | `author` \| `reviewer` \| `merger` \| `controller` \| `reconciler` | Profile / MCP server binding | Process lifetime |
| `worker_id` / `worker_identity` | One namespace worker process | Runtime registry at first registration | New on worker restart; not reused |
| `session_id` | Client session ownership | Launcher `GITEA_MCP_CLIENT_SESSION` or runtime | Session lifetime |
| `generation_id` | One daemon launch | Runtime at process boot | New on process restart |
| `process_identity` / PID | OS process | Runtime | Process lifetime |

### Rules

1. Multiple active instances **may** share the same `client_type`.
2. Every application launch receives a **distinct** `client_instance_id`.
3. All five namespace workers of one launch report the **same**
   `client_instance_id`.
4. Each namespace worker has a **distinct** `worker_identity`, process
   identity, generation, and PID.
5. Instance identity is **never** inferred from PID proximity, timestamps, or
   client type alone.
6. Live reuse of one `client_instance_id` with conflicting workers fails closed.
7. Sharing only a profile or `client_type` is **not** a duplicate.

This deliberately **replaces** any permanent `exactly_one_per_profile` fleet
model (#949 assumption) as the operating rule for multi-instance fleets.

## How five workers join one instance

1. The host starts one application instance (for example one Codex session).
2. The **production application launcher**
   (`mcp_application_launcher.build_application_mcp_servers` /
   `gitea_config.multi_namespace_launcher_entries`) mints exactly one
   `client_instance_id` via `mcp_fleet_snapshot.generate_client_instance_id`
   and injects the same env into every namespace worker:

   ```text
   GITEA_MCP_CLIENT=<client_type>
   GITEA_MCP_CLIENT_INSTANCE=<client_instance_id>
   GITEA_MCP_INSTANCE_PROVENANCE=trusted_launcher
   GITEA_MCP_CLIENT_SESSION=<session_id>          # optional but recommended
   GITEA_MCP_FLEET_RUN_ID=<enrollment id>         # when on an approved canary
   GITEA_CLIENT_MANAGED=1
   GITEA_MCP_PROFILE=<role-profile>
   ```

3. The MCP client starts the five namespace processes (`gitea-author`,
   `gitea-reviewer`, `gitea-merger`, `gitea-controller`, `gitea-reconciler`)
   from that generated `mcpServers` map — each entry carries the **same**
   instance ID.
4. Each worker registers once into the worker registry with its own
   `worker_identity`, `namespace`, `generation_id`, and PID, but the shared
   `client_instance_id`. Workers never mint a trusted instance ID themselves.

Only IDs matching the launcher format `inst-<client>-<timestamp>-<digest>`
are trusted. Missing, `legacy-pid-*` / `pid-*` placeholders, and ordinary
user-supplied strings fail closed as untrusted and cannot authorize
multi-instance fleet mutation safety.

Worker reconnect (same process, same registration) keeps the instance ID.
Worker restart (new process) mints a new worker identity and generation but
must still receive the same `GITEA_MCP_CLIENT_INSTANCE` from the parent
application if it is the same launch. Full application restart mints a new
`client_instance_id` (call the launcher without a prior ID).

### Mutation gate (instance-aware)

The capability / runtime diagnostic gate
(`_check_mcp_runtimes_diagnostics`) is **instance-aware**:

* Two legitimate application instances that share a profile (same
  `GITEA_MCP_PROFILE`) are allowed when each has a distinct trusted
  `client_instance_id`.
* More than one live worker for the same
  `(client_instance_id, profile/namespace)` fails closed as a duplicate
  namespace worker.
* Multiple processes sharing a profile **without** trusted instance
  evidence still fail closed (indistinguishable from a duplicate).

## Approved fleet manifest

An operator (or controller enrollment step) obtains an approved manifest as a
list of expected instances, for example:

```json
{
  "instances": [
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120000Z-abc123def456",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    },
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120100Z-fed654cba321",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    }
  ]
}
```

Pass that list as `expected_manifest` to `gitea_snapshot_instance_fleet`.
When a manifest is supplied:

* instances on the manifest but not live → `missing_expected`;
* live instances not on the manifest → `unmanifested`;
* both are **active blockers** for fleet-gate safety.

Without a manifest, the snapshot still enumerates the live fleet and classifies
identity collisions; it does not invent enrollment policy.

## Snapshot consistency

Each snapshot includes:

* `snapshot_at` — UTC timestamp;
* `consistency_token` / `registry_revision` — content digest over worker
  identity, instance id, status, heartbeat, fencing, and generation.

Two successive snapshots can prove heartbeat continuity via
`mcp_fleet_snapshot.compare_snapshot_heartbeats`.

## Classification (active vs historical)

**Active blockers** (make `live_fleet_safe=false`):

* missing expected instance;
* unmanifested extra instance;
* duplicate namespace worker within one instance;
* live `client_instance_id` collision;
* reused worker / session / generation / process / PID / fencing identity;
* orphaned or unowned workers;
* unknown client;
* foreign-repository workers;
* old-revision workers;
* stale workers still marked active;
* legacy incomplete instance identity.

**Historical dead rows** (`status` released/superseded) are reported as
`historical_dead` findings with `active_blocker=false`. They never
automatically make the live fleet unsafe.

## Fail-closed behaviour and recovery

| Condition | Mutation safety | Diagnostic reads | Recovery |
| --- | --- | --- | --- |
| Two instances, same `client_type`, distinct IDs | Safe (if otherwise healthy) | Available | None needed |
| Live reuse of one `client_instance_id` | Unsafe | Available | Stop the colliding launch or re-issue a distinct ID |
| Two workers, same namespace, one instance | Unsafe | Available | Stop the extra worker |
| Legacy registration without trusted instance ID | Unsafe for fleet mutation | Available | Relaunch with launcher-issued `GITEA_MCP_CLIENT_INSTANCE` |
| Historical dead row only | Does not block alone | Available | No action required for fleet safety |
| Registry unavailable | Snapshot fails closed | N/A | Repair registry path / reconnect namespaces |

Incomplete or untrusted instance identity **cannot** authorize unsafe
mutation. Diagnostic reads remain available where `gitea.read` allows.

## Permissions

* **Allowed:** `controller`, `reconciler` with `gitea.read`.
* **Denied:** author, reviewer, merger (even with `gitea.read` for other tools).
* **No new mutation permissions** are granted to any role.

## Related surfaces

* `mcp_worker_identity` — registry, worker identity, heartbeats (#948, #975).
* `mcp_fleet_snapshot` — pure snapshot + classification (#978).
* `gitea_snapshot_instance_fleet` — sanctioned MCP tool (#978).
* `gitea_get_runtime_context` — single-process view (not fleet-wide).

## Non-goals

* Starting the #963 canary.
* Purging historical registry rows.
* Preserving “exactly one process per profile” as the permanent model.
* Using shell / process-table / SQLite inspection as production fleet evidence.