Gitea-Tools/docs/instance-fleet-identity.md

Instance-level fleet identity and health snapshots

Issue #978. Companion primitives: #948 (worker ownership), #975 (heartbeat lifecycle), #951 (restart receipts).

Why this exists

The multi-instance fleet gate (#963) needs a native answer to:

• which application launches are live;
• which of the five namespace workers belong to each launch;
• whether two Codex (or Claude, Grok, …) launches are distinct;
• whether any live collision is real rather than a shared client type.

Process tables, PID proximity, configuration files, and direct SQLite access are not production evidence. The sanctioned surface is the read-only MCP tool gitea_snapshot_instance_fleet on controller and reconciler namespaces.

Identity hierarchy

Identity	Scope	Who generates it	Lifetime
client_type	Application family (codex, claude_code, gemini, grok, …)	Launcher sets GITEA_MCP_CLIENT	Stable for the product
client_instance_id	One running application launch	Trusted launcher, once per launch, as GITEA_MCP_CLIENT_INSTANCE	Fresh launch → new ID; reconnect of same launch → same ID; full app restart → new ID
fleet_run_id	Operator-approved enrollment / canary cohort	Operator / controller sets GITEA_MCP_FLEET_RUN_ID	Duration of the approved rollout
namespace	author | reviewer | merger | controller | reconciler	Profile / MCP server binding	Process lifetime
worker_id / worker_identity	One namespace worker process	Runtime registry at first registration	New on worker restart; not reused
session_id	Client session ownership	Launcher GITEA_MCP_CLIENT_SESSION or runtime	Session lifetime
generation_id	One daemon launch	Runtime at process boot	New on process restart
process_identity / PID	OS process	Runtime	Process lifetime

Rules

1. Multiple active instances may share the same client_type.
2. Every application launch receives a distinct client_instance_id.
3. All five namespace workers of one launch report the same client_instance_id.
4. Each namespace worker has a distinct worker_identity, process identity, generation, and PID.
5. Instance identity is never inferred from PID proximity, timestamps, or client type alone.
6. Live reuse of one client_instance_id with conflicting workers fails closed.
7. Sharing only a profile or client_type is not a duplicate.

This deliberately replaces any permanent exactly_one_per_profile fleet model (#949 assumption) as the operating rule for multi-instance fleets.

How five workers join one instance

1. The host starts one application instance (for example one Codex session).

2. The production application launcher (mcp_application_launcher.build_application_mcp_servers / gitea_config.multi_namespace_launcher_entries) mints exactly one client_instance_id via mcp_fleet_snapshot.generate_client_instance_id and injects the same env into every namespace worker:

   GITEA_MCP_CLIENT=<client_type>
   GITEA_MCP_CLIENT_INSTANCE=<client_instance_id>
   GITEA_MCP_INSTANCE_PROVENANCE=trusted_launcher
   GITEA_MCP_CLIENT_SESSION=<session_id>          # optional but recommended
   GITEA_MCP_FLEET_RUN_ID=<enrollment id>         # when on an approved canary
   GITEA_CLIENT_MANAGED=1
   GITEA_MCP_PROFILE=<role-profile>
   

3. The MCP client starts the five namespace processes (gitea-author, gitea-reviewer, gitea-merger, gitea-controller, gitea-reconciler) from that generated mcpServers map — each entry carries the same instance ID.

4. Each worker registers once into the worker registry with its own worker_identity, namespace, generation_id, and PID, but the shared client_instance_id. Workers never mint a trusted instance ID themselves.

Only IDs matching the launcher format inst-<client>-<timestamp>-<digest> are trusted. Missing, legacy-pid-* / pid-* placeholders, and ordinary user-supplied strings fail closed as untrusted and cannot authorize multi-instance fleet mutation safety.

Worker reconnect (same process, same registration) keeps the instance ID. Worker restart (new process) mints a new worker identity and generation but must still receive the same GITEA_MCP_CLIENT_INSTANCE from the parent application if it is the same launch. Full application restart mints a new client_instance_id (call the launcher without a prior ID).

Mutation gate (instance-aware)

The capability / runtime diagnostic gate (_check_mcp_runtimes_diagnostics) is instance-aware:

• Two legitimate application instances that share a profile (same GITEA_MCP_PROFILE) are allowed when each has a distinct trusted client_instance_id.
• More than one live worker for the same (client_instance_id, profile/namespace) fails closed as a duplicate namespace worker.
• Multiple processes sharing a profile without trusted instance evidence still fail closed (indistinguishable from a duplicate).

Approved fleet manifest

An operator (or controller enrollment step) obtains an approved manifest as a list of expected instances, for example:

{
  "instances": [
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120000Z-abc123def456",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    },
    {
      "client_type": "codex",
      "client_instance_id": "inst-codex-20260730T120100Z-fed654cba321",
      "fleet_run_id": "canary-963-2026-07-30",
      "namespaces": ["author", "reviewer", "merger", "controller", "reconciler"]
    }
  ]
}

Pass that list as expected_manifest to gitea_snapshot_instance_fleet. When a manifest is supplied:

• instances on the manifest but not live → missing_expected;
• live instances not on the manifest → unmanifested;
• both are active blockers for fleet-gate safety.

Without a manifest, the snapshot still enumerates the live fleet and classifies identity collisions; it does not invent enrollment policy.

Snapshot consistency

Each snapshot includes:

• snapshot_at — UTC timestamp;
• consistency_token / registry_revision — content digest over worker identity, instance id, status, heartbeat, fencing, and generation.

Two successive snapshots can prove heartbeat continuity via mcp_fleet_snapshot.compare_snapshot_heartbeats.

Classification (active vs historical)

Active blockers (make live_fleet_safe=false):

• missing expected instance;
• unmanifested extra instance;
• duplicate namespace worker within one instance;
• live client_instance_id collision;
• reused worker / session / generation / process / PID / fencing identity;
• orphaned or unowned workers;
• unknown client;
• foreign-repository workers;
• old-revision workers;
• stale workers still marked active;
• legacy incomplete instance identity.

Historical dead rows (status released/superseded) are reported as historical_dead findings with active_blocker=false. They never automatically make the live fleet unsafe.

Fail-closed behaviour and recovery

Condition	Mutation safety	Diagnostic reads	Recovery
Two instances, same client_type, distinct IDs	Safe (if otherwise healthy)	Available	None needed
Live reuse of one client_instance_id	Unsafe	Available	Stop the colliding launch or re-issue a distinct ID
Two workers, same namespace, one instance	Unsafe	Available	Stop the extra worker
Legacy registration without trusted instance ID	Unsafe for fleet mutation	Available	Relaunch with launcher-issued GITEA_MCP_CLIENT_INSTANCE
Historical dead row only	Does not block alone	Available	No action required for fleet safety
Registry unavailable	Snapshot fails closed	N/A	Repair registry path / reconnect namespaces

Incomplete or untrusted instance identity cannot authorize unsafe mutation. Diagnostic reads remain available where gitea.read allows.

Permissions

• Allowed: controller, reconciler with gitea.read.
• Denied: author, reviewer, merger (even with gitea.read for other tools).
• No new mutation permissions are granted to any role.

Related surfaces

• mcp_worker_identity — registry, worker identity, heartbeats (#948, #975).
• mcp_fleet_snapshot — pure snapshot + classification (#978).
• gitea_snapshot_instance_fleet — sanctioned MCP tool (#978).
• gitea_get_runtime_context — single-process view (not fleet-wide).
• stale-worker-retirement.md — CAS-protected retirement of conclusively stale registrations (#980). Note that the registry_revision this snapshot returns is time-seeded and unsuitable for compare-and-swap; retirement derives its own stable registry_fingerprint from registry content alone.

Non-goals

• Starting the #963 canary.
• Purging historical registry rows.
• Preserving “exactly one process per profile” as the permanent model.
• Using shell / process-table / SQLite inspection as production fleet evidence.