Files
Gitea-Tools/mcp_worker_identity.py
sysadminandClaude Opus 5 1dd30ecb15 feat(mcp): client/session-aware runtime ownership and provenance (#948)
Two surfaces reported different provenance for one process.
`gitea_get_runtime_context` read the live environment and reported
`client_managed`; `mcp_namespace_health.classify_namespace_probe` derived
provenance from `_safe_env_summary()`, whose `SAFE_ENV_KEYS` allowlist never
contained `GITEA_CLIENT_MANAGED`, `GITEA_MCP_CLIENT_MANAGED`, or
`GITEA_SERVER_PROVENANCE`. That lookup could only ever miss, so the health
surface was structurally incapable of returning anything but `manual_launch`.

Neither model could name which client or which session owned a runtime, so a
healthy daemon serving a second client was indistinguishable from a duplicate,
and the profile-wide duplicate scan walled the whole fleet.

Introduce `mcp_worker_identity` as the one authority, splitting two claims the
old code ran together:

* launch provenance — was this hand-launched from a terminal? Answered from the
  environment, which is legitimate because the launcher sets it. Preserves the
  #686 wall unchanged.
* session ownership — which live client session owns this runtime now? Answered
  only from a live attachment record; no environment flag can establish it.

The module provides collision-resistant worker identities
(`<llm-name>-<UTC-timestamp>-<short-sha>`), an atomic SQLite registry with
fencing epochs, heartbeat-based liveness, generation takeover that supersedes
only a non-live claimant, cohort classification, and failure scoping.

Behaviour changes:

* Registering an existing worker identity fails closed; it is never replaced,
  adopted, or merged with. The caller mints a different identity instead.
* A generation held by a live session cannot be claimed by a second one. A
  generation whose claimant is not live is taken over with a higher fencing
  epoch, so stale ownership cannot permanently strand a healthy daemon.
* A superseded session presenting an old epoch is refused and performs no write.
* Liveness comes from heartbeat freshness; a live PID cannot resurrect an
  expired record, and a dead PID withdraws liveness.
* Workers sharing a role or profile no longer trigger a profile-wide duplicate
  block, provided each carries a distinct identity. Processes with no identity
  evidence remain classified as duplicates, so the #686 wall still holds.
* Runtime failures are scoped to a worker identity or generation, never to a
  profile or the fleet.
* Reconnect guidance no longer defaults to Codex. An unidentified client gets
  host-agnostic steps; `gitea_request_mcp_reconnect(client=...)` defaults to
  resolving the client from the live attachment record.
* `resolve_bound_remote` keeps a bound namespace on its remote instead of
  falling through to the `dadeschools` library default.

Absence of proof is now reported as `unproven` rather than asserted as
`manual_launch`. Both still fail closed — `is_client_managed` is unchanged, so
nothing previously refused is now permitted — but remediation names the proof
that is actually missing instead of describing a terminal launch it cannot
evidence. The #686 test is updated for that vocabulary and keeps every
wall-preserving assertion.

Threat-model anchors and their citations in docs/remote-mcp/threat-model.md are
restamped for the line movement in gitea_mcp_server.py.

Tests: tests/test_issue_948_client_session_provenance.py adds 43 cases covering
Codex/Gemini/Antigravity/Claude attachment, same-client new session, cross-client
takeover after a session ends, two live conflicting sessions, stale records,
missing attachment proof, environment flags without attachment, mixed
generations, duplicate cohorts, the hardcoded-client regression, explicit PRGS
selection, default-remote host drift, cross-surface agreement, and fail-closed
handling without false reconnect loops. Synthetic identifiers throughout.

Full suite from a branches/ worktree: 28F/5953P/6S at head vs 28F/5910P/6S at
merge base 8eada1fb, identical failing ID sets.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01F6Vomtndpq2gSBa88Tfcwy
2026-07-29 02:52:18 -04:00

1433 lines
53 KiB
Python

"""Client/session-aware runtime ownership and provenance (#948).
Before this module the control plane had two disagreeing provenance models.
``gitea_get_runtime_context`` read the *live* process environment, while
``mcp_namespace_health.classify_namespace_probe`` read a filtered env summary
whose allowlist never carried the provenance keys — so one surface reported
``client_managed`` while the other reported ``manual_launch`` for the very same
process. Neither model could name *which* client or *which* session owned the
runtime, so a healthy daemon serving a second client looked like a duplicate.
This module is the single authority both surfaces now consult. The model it
implements:
* A **worker identity** is unique and is *not* a role or a profile. Roles and
profiles are reusable capability definitions that many live workers may share
concurrently; exclusivity belongs to an active lease target, never to a role.
* A **generation** is one daemon launch. Exactly one live session may claim a
generation at a time; a takeover mints a higher fencing epoch so the prior
session is fenced if it resumes.
* **Trusted evidence** is a live attachment record binding
``(client_instance_id, session_id, generation_id)`` to the running process.
A process environment flag is corroborating evidence at best: it is set by
whoever launched the process and says nothing about which session owns it
*now*, so it can never establish ownership on its own.
* Missing or self-contradictory evidence fails closed, and the refusal is
scoped to the affected worker identity rather than to every worker sharing
its profile.
"""
from __future__ import annotations
import hashlib
import os
import re
import secrets
import sqlite3
import threading
import time
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import Any, Iterator
# --- Provenance verdicts -------------------------------------------------
#: A live attachment record binds this runtime to a client session.
PROVENANCE_CLIENT_SESSION = "client_session_attached"
#: Positive evidence that the process was launched outside any client session.
PROVENANCE_MANUAL = "manual_launch"
#: No trusted evidence either way — fails closed, distinct from "manual".
PROVENANCE_UNPROVEN = "unproven"
VALID_PROVENANCE = frozenset(
{PROVENANCE_CLIENT_SESSION, PROVENANCE_MANUAL, PROVENANCE_UNPROVEN}
)
# --- Evidence kinds ------------------------------------------------------
EVIDENCE_ATTACHMENT_RECORD = "live_attachment_record"
EVIDENCE_NATIVE_TRANSPORT = "bound_native_transport"
EVIDENCE_PROCESS_LIVENESS = "process_liveness"
#: Never sufficient alone. Named so a report can say *why* it was not enough.
EVIDENCE_ENV_FLAG = "process_env_flag"
#: Evidence kinds that, on their own, establish current session ownership.
SUFFICIENT_EVIDENCE = frozenset({EVIDENCE_ATTACHMENT_RECORD})
# --- Blocker kinds -------------------------------------------------------
BLOCKER_NONE = "none"
BLOCKER_NO_ATTACHMENT = "session_attachment_missing"
BLOCKER_CONFLICTING_SESSIONS = "conflicting_live_sessions"
BLOCKER_FENCED = "superseded_generation_fenced"
BLOCKER_IDENTITY_COLLISION = "worker_identity_collision"
BLOCKER_CONTRADICTORY = "contradictory_provenance_evidence"
# --- Client naming -------------------------------------------------------
#: Returned when no client name is supplied or the name is unusable. There is
#: deliberately no default vendor here: defaulting an unknown client to a
#: specific product emits reconnect steps for a UI the operator is not looking
#: at, which is the #948 hardcoded-``codex`` defect.
UNKNOWN_CLIENT = "unknown_client"
#: Canonical spellings for clients whose aliases we recognise. Absence from
#: this map is not an error — an unrecognised name is slugified and kept, so a
#: new client is named accurately rather than relabelled as a known one.
CLIENT_ALIASES: dict[str, str] = {
"codex": "codex",
"openai": "codex",
"openai_codex": "codex",
"claude": "claude_code",
"claude_code": "claude_code",
"claude_desktop": "claude_code",
"anthropic": "claude_code",
"gemini": "gemini",
"google": "gemini",
"antigravity": "antigravity",
"gemini_antigravity": "antigravity",
"grok": "grok",
"xai": "grok",
}
_SLUG_RE = re.compile(r"[^a-z0-9]+")
#: ``<llm-name>-<UTC-timestamp>-<short-sha>`` per the #948 scope clarification.
WORKER_IDENTITY_RE = re.compile(
r"^(?P<client>[a-z0-9][a-z0-9_]*)-(?P<ts>\d{8}T\d{6}Z)-(?P<digest>[0-9a-f]{12})$"
)
IDENTITY_TIMESTAMP_FORMAT = "%Y%m%dT%H%M%SZ"
IDENTITY_DIGEST_LENGTH = 12
# --- Registry defaults ---------------------------------------------------
REGISTRY_PATH_ENV = "GITEA_WORKER_REGISTRY_DB"
DEFAULT_REGISTRY_PATH = os.path.expanduser(
"~/.cache/gitea-tools/control-plane/worker_registry.sqlite3"
)
#: How long a registration stays live without a heartbeat. Chosen to outlast a
#: full-suite run inside one tool call while still releasing an abandoned
#: worker within a single operator coffee break.
DEFAULT_HEARTBEAT_TTL_SECONDS = 900.0
STATUS_ACTIVE = "active"
STATUS_SUPERSEDED = "superseded"
STATUS_RELEASED = "released"
_TRUE_VALUES = frozenset({"1", "true", "yes", "client_managed"})
_FALSE_VALUES = frozenset({"0", "false", "no", "manual", "manual_launch"})
#: Environment keys that historically stood in for provenance. They are read
#: only as corroboration; see :func:`env_provenance_signal`.
PROVENANCE_ENV_KEYS = (
"GITEA_CLIENT_MANAGED",
"GITEA_MCP_CLIENT_MANAGED",
"GITEA_SERVER_PROVENANCE",
"GITEA_MCP_SANCTIONED_DAEMON",
)
#: Explicit launch-provenance declarations, highest precedence first. Mirrors
#: the historical ``_is_client_managed_process`` precedence exactly, so
#: unifying the two models changes *who decides*, not *what is decided*.
LAUNCH_DECLARATION_ENV_KEYS = (
"GITEA_CLIENT_MANAGED",
"GITEA_MCP_CLIENT_MANAGED",
"GITEA_SERVER_PROVENANCE",
"GITEA_FORCE_CLIENT_MANAGED",
)
#: Env keys a client launcher sets that imply a non-terminal launch.
LAUNCH_CONFIG_ENV_KEYS = (
"GITEA_MCP_CONFIG",
"GITEA_MCP_PROFILE",
"GITEA_PROFILE_NAME",
)
# --- Launch provenance verdicts ------------------------------------------
#: Launch-dimension values. These keep the literals the pre-#948 surfaces
#: already published, so a consumer reading ``server_provenance`` sees the same
#: strings it always did.
LAUNCH_CLIENT_MANAGED = "client_managed"
LAUNCH_MANUAL = PROVENANCE_MANUAL
LAUNCH_UNPROVEN = PROVENANCE_UNPROVEN
# --- Session ownership verdicts ------------------------------------------
#: A live attachment record binds this runtime to exactly one client session.
OWNERSHIP_OWNED = "owned"
#: No live attachment record. Fails closed for ownership-scoped decisions.
OWNERSHIP_UNOWNED = "unowned"
#: More than one live session claims the generation. Fails closed.
OWNERSHIP_CONTESTED = "contested"
_SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS worker_registrations (
worker_identity TEXT PRIMARY KEY,
client_name TEXT NOT NULL,
client_instance_id TEXT NOT NULL,
session_id TEXT NOT NULL,
generation_id TEXT NOT NULL,
role TEXT,
profile TEXT,
namespace TEXT,
remote TEXT,
repository_binding TEXT,
pid INTEGER,
transport TEXT,
token_fingerprint TEXT,
started_at TEXT NOT NULL,
last_heartbeat_at TEXT NOT NULL,
heartbeat_ttl_seconds REAL NOT NULL,
fencing_epoch INTEGER NOT NULL,
status TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_worker_generation
ON worker_registrations(generation_id, status);
CREATE INDEX IF NOT EXISTS idx_worker_session
ON worker_registrations(session_id, status);
CREATE INDEX IF NOT EXISTS idx_worker_profile
ON worker_registrations(profile, status);
"""
class WorkerRegistryError(RuntimeError):
"""Raised for registry misuse that is a programming error, not a refusal."""
def _utc_now() -> datetime:
return datetime.now(timezone.utc)
def _ts(value: datetime) -> str:
return value.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _parse_ts(value: str | None) -> datetime | None:
if not value:
return None
try:
return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(
tzinfo=timezone.utc
)
except (TypeError, ValueError):
return None
def default_registry_path() -> str:
raw = (os.environ.get(REGISTRY_PATH_ENV) or DEFAULT_REGISTRY_PATH).strip()
return raw or DEFAULT_REGISTRY_PATH
# --- Identity ------------------------------------------------------------
def normalize_client_name(raw: str | None) -> str:
"""Canonicalise a client name without inventing one.
An empty or unusable name yields :data:`UNKNOWN_CLIENT` rather than a
specific product, so downstream reconnect guidance describes a UI the
operator is actually looking at (or says it does not know) instead of
always naming one vendor.
"""
text = _SLUG_RE.sub("_", (raw or "").strip().lower()).strip("_")
if not text:
return UNKNOWN_CLIENT
return CLIENT_ALIASES.get(text, text)
def worker_identity_digest(
client_name: str,
session_id: str,
timestamp_ns: int,
nonce: str,
) -> str:
"""Collision-resistant digest over client, session, high-resolution time, nonce."""
payload = f"{client_name}\x1f{session_id}\x1f{timestamp_ns}\x1f{nonce}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:IDENTITY_DIGEST_LENGTH]
def generate_worker_identity(
client_name: str | None,
session_id: str | None,
*,
timestamp_ns: int | None = None,
nonce: str | None = None,
now: datetime | None = None,
) -> str:
"""Mint ``<llm-name>-<UTC-timestamp>-<short-sha>`` (#948 AC27-29).
``timestamp_ns`` and ``nonce`` are injectable so a test can force a
collision deliberately; production callers omit both and get high-resolution
time plus a fresh random nonce.
"""
client = normalize_client_name(client_name)
session = (session_id or "").strip() or "no_session"
ts_ns = int(timestamp_ns if timestamp_ns is not None else time.time_ns())
seed = nonce if nonce is not None else secrets.token_hex(16)
stamp = (now or _utc_now()).astimezone(timezone.utc).strftime(
IDENTITY_TIMESTAMP_FORMAT
)
digest = worker_identity_digest(client, session, ts_ns, seed)
return f"{client}-{stamp}-{digest}"
def parse_worker_identity(identity: str | None) -> dict[str, Any]:
"""Structurally validate a worker identity."""
text = (identity or "").strip()
match = WORKER_IDENTITY_RE.match(text)
if not match:
return {
"valid": False,
"worker_identity": text or None,
"client_name": None,
"minted_at": None,
"digest": None,
"reasons": [
"worker identity must match <llm-name>-<UTC-timestamp>-<short-sha>, "
"e.g. gemini-20260729T061500Z-1a2b3c4d5e6f"
],
}
return {
"valid": True,
"worker_identity": text,
"client_name": match.group("client"),
"minted_at": match.group("ts"),
"digest": match.group("digest"),
"reasons": [],
}
def new_generation_id(pid: int | None = None) -> str:
"""Mint a boot epoch for one daemon launch."""
return f"gen-{int(pid if pid is not None else os.getpid())}-{secrets.token_hex(8)}"
# --- Environment signal --------------------------------------------------
def env_provenance_signal(env: dict[str, str] | None = None) -> dict[str, Any]:
"""Read the legacy env provenance flags as *corroborating* signal only.
Returns the raw reading plus an explicit statement that it does not
establish ownership. Callers must not branch on ``value`` alone; #948
requires a live attachment record for that.
"""
source = os.environ if env is None else env
present: dict[str, str] = {}
for key in PROVENANCE_ENV_KEYS:
raw = source.get(key)
if raw not in (None, ""):
present[key] = str(raw)
value: bool | None = None
for key in PROVENANCE_ENV_KEYS:
raw = (present.get(key) or "").strip().lower()
if not raw:
continue
if raw in _FALSE_VALUES:
value = False
break
if raw in _TRUE_VALUES:
value = True
break
return {
"evidence_kind": EVIDENCE_ENV_FLAG,
"keys_present": sorted(present),
"value": value,
"proves_session_ownership": False,
"reason": (
"process environment flags are set by whoever launched the process and "
"carry no session binding; they corroborate a live attachment record but "
"never establish current ownership on their own (#948)"
),
}
def assess_declared_launch_provenance(env: dict[str, str] | None = None) -> dict[str, Any]:
"""Launch provenance for a process we are observing from the outside.
Deliberately stricter than :func:`assess_launch_provenance`. For a *peer*
process there is no stdin to inspect, and launcher-configuration env is not
evidence of anything: a shell that exports ``GITEA_MCP_PROFILE`` passes it
to a hand-launched server too. So only an explicit declaration counts, and
its absence is reported as ``unproven`` rather than asserted as a manual
launch — naming a proof we do not have was the misdiagnosis #948 reports.
"""
source = os.environ if env is None else env
for key in LAUNCH_DECLARATION_ENV_KEYS:
raw = (source.get(key) or "").strip().lower()
if not raw:
continue
if raw in _FALSE_VALUES:
return {
"launch_provenance": LAUNCH_MANUAL,
"client_managed": False,
"decided_by": f"env:{key}",
"reasons": [f"{key}={raw!r} explicitly declares a manual launch"],
}
if raw in _TRUE_VALUES:
return {
"launch_provenance": LAUNCH_CLIENT_MANAGED,
"client_managed": True,
"decided_by": f"env:{key}",
"reasons": [
f"{key}={raw!r} explicitly declares a client-managed launch"
],
}
return {
"launch_provenance": LAUNCH_UNPROVEN,
"client_managed": False,
"decided_by": "no_declaration",
"reasons": [
"no explicit client-managed declaration on the observed process; "
"launcher-configuration env is not proof for a peer process because a "
"hand-launched server inherits it from the shell"
],
}
def assess_launch_provenance(
env: dict[str, str] | None = None,
*,
stdin_is_tty: bool | None = None,
declared_only: bool = False,
) -> dict[str, Any]:
"""Decide *how the process was launched* — a different question from ownership.
#948 separates two claims the old code ran together:
* **Launch provenance** — was this started by a client, or hand-launched from
a terminal? This is the #686 concern and it is what the environment can
legitimately answer, because the launcher is exactly what sets it.
* **Session ownership** — which live client session owns this runtime *now*?
The environment cannot answer that; only a live attachment record can.
Conflating them is what let one surface call a process ``client_managed``
while another called the same process ``manual_launch``. This function is now
the only implementation of the first claim, so every surface gets one answer.
``declared_only`` selects the strategy for an *observed peer* process, where
stdin cannot be inspected and launcher-config env proves nothing. Callers
assessing their own process leave it false.
"""
if declared_only:
return assess_declared_launch_provenance(env)
source = os.environ if env is None else env
reasons: list[str] = []
for key in LAUNCH_DECLARATION_ENV_KEYS:
raw = (source.get(key) or "").strip().lower()
if not raw:
continue
if raw in _FALSE_VALUES:
return {
"launch_provenance": LAUNCH_MANUAL,
"client_managed": False,
"decided_by": f"env:{key}",
"reasons": [f"{key}={raw!r} explicitly declares a manual launch"],
}
if raw in _TRUE_VALUES:
return {
"launch_provenance": LAUNCH_CLIENT_MANAGED,
"client_managed": True,
"decided_by": f"env:{key}",
"reasons": [f"{key}={raw!r} explicitly declares a client-managed launch"],
}
if stdin_is_tty:
return {
"launch_provenance": LAUNCH_MANUAL,
"client_managed": False,
"decided_by": "stdin_tty",
"reasons": [
"stdin is an interactive terminal, so this process cannot be "
"receiving client stdio"
],
}
present = [k for k in LAUNCH_CONFIG_ENV_KEYS if (source.get(k) or "").strip()]
if present:
return {
"launch_provenance": LAUNCH_CLIENT_MANAGED,
"client_managed": True,
"decided_by": "env:launcher_config",
"reasons": [
f"launcher-supplied configuration env present ({', '.join(present)}) "
"with no terminal on stdin"
],
}
reasons.append(
"no launch declaration, no launcher configuration env, and no terminal "
"evidence; launch provenance is unproven rather than assumed manual"
)
return {
"launch_provenance": LAUNCH_UNPROVEN,
"client_managed": False,
"decided_by": "no_evidence",
"reasons": reasons,
}
# --- Registry ------------------------------------------------------------
class WorkerRegistry:
"""Atomic, fail-closed registry of live client/session runtime ownership.
Backed by SQLite with ``BEGIN IMMEDIATE`` so two clients racing to register
the same identity, or to claim the same generation, resolve to exactly one
winner across processes.
"""
def __init__(self, db_path: str | None = None) -> None:
self.db_path = (db_path or default_registry_path()).strip()
parent = os.path.dirname(self.db_path)
if parent:
os.makedirs(parent, mode=0o700, exist_ok=True)
self._lock = threading.RLock()
self._init_schema()
# -- plumbing --
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self.db_path, timeout=30)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode = WAL")
return conn
@contextmanager
def _tx(self) -> Iterator[sqlite3.Connection]:
with self._lock:
conn = self._connect()
try:
conn.execute("BEGIN IMMEDIATE")
yield conn
conn.commit()
except Exception:
try:
conn.rollback()
except sqlite3.Error:
pass
raise
finally:
conn.close()
def _init_schema(self) -> None:
with self._lock:
conn = self._connect()
try:
conn.executescript(_SCHEMA_SQL)
conn.commit()
finally:
conn.close()
@staticmethod
def _row_to_record(row: sqlite3.Row) -> dict[str, Any]:
return {key: row[key] for key in row.keys()}
# -- liveness --
@staticmethod
def is_live(
record: dict[str, Any] | None,
*,
now: datetime | None = None,
pid_alive: bool | None = None,
) -> dict[str, Any]:
"""Liveness from heartbeat freshness, corroborated by process evidence.
#948 AC7: abandoned generations are detected through authoritative
liveness evidence, not PID comparison alone. A fresh heartbeat is the
authority; ``pid_alive`` can only *withdraw* liveness, never grant it,
because a long-lived daemon PID outlives every session it serves.
"""
if not record:
return {"live": False, "reasons": ["no registration record"]}
if record.get("status") != STATUS_ACTIVE:
return {
"live": False,
"reasons": [f"registration status is {record.get('status')!r}"],
}
beat = _parse_ts(record.get("last_heartbeat_at"))
if beat is None:
return {"live": False, "reasons": ["registration has no parsable heartbeat"]}
ttl = float(record.get("heartbeat_ttl_seconds") or DEFAULT_HEARTBEAT_TTL_SECONDS)
age = ((now or _utc_now()) - beat).total_seconds()
fresh = age <= ttl
reasons: list[str] = []
if not fresh:
reasons.append(
f"heartbeat is {age:.0f}s old, past the {ttl:.0f}s liveness window"
)
if pid_alive is False:
reasons.append(f"recorded pid {record.get('pid')} is no longer running")
return {
"live": bool(fresh and pid_alive is not False),
"heartbeat_age_seconds": age,
"heartbeat_ttl_seconds": ttl,
"heartbeat_fresh": fresh,
"pid_alive": pid_alive,
"reasons": reasons,
}
# -- reads --
def get(self, worker_identity: str) -> dict[str, Any] | None:
conn = self._connect()
try:
row = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
return self._row_to_record(row) if row else None
finally:
conn.close()
def list_workers(
self,
*,
status: str | None = STATUS_ACTIVE,
generation_id: str | None = None,
profile: str | None = None,
role: str | None = None,
) -> list[dict[str, Any]]:
clauses: list[str] = []
params: list[Any] = []
if status:
clauses.append("status = ?")
params.append(status)
if generation_id:
clauses.append("generation_id = ?")
params.append(generation_id)
if profile:
clauses.append("profile = ?")
params.append(profile)
if role:
clauses.append("role = ?")
params.append(role)
where = f" WHERE {' AND '.join(clauses)}" if clauses else ""
conn = self._connect()
try:
rows = conn.execute(
f"SELECT * FROM worker_registrations{where} ORDER BY started_at",
params,
).fetchall()
return [self._row_to_record(r) for r in rows]
finally:
conn.close()
# -- writes --
def register(
self,
*,
worker_identity: str,
client_name: str | None,
client_instance_id: str,
session_id: str,
generation_id: str,
role: str | None = None,
profile: str | None = None,
namespace: str | None = None,
remote: str | None = None,
repository_binding: str | None = None,
pid: int | None = None,
transport: str | None = None,
token_fingerprint: str | None = None,
heartbeat_ttl_seconds: float = DEFAULT_HEARTBEAT_TTL_SECONDS,
now: datetime | None = None,
pid_alive_probe=None,
) -> dict[str, Any]:
"""Atomically register one worker identity.
Fails closed on collision (#948 AC31): an identity already present is
never replaced, adopted, merged with, or corrupted — the caller is told
to mint a different identity and register that instead (AC32). The
existing registration is returned untouched so the caller can see what
it collided with.
"""
parsed = parse_worker_identity(worker_identity)
if not parsed["valid"]:
return {
"success": False,
"registered": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_IDENTITY_COLLISION,
"collision": False,
"reasons": parsed["reasons"],
"exact_next_action": (
"Mint a worker identity with generate_worker_identity() and "
"register that."
),
}
stamp = _ts(now or _utc_now())
with self._tx() as conn:
existing = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
if existing is not None:
record = self._row_to_record(existing)
pid_alive = (
pid_alive_probe(record.get("pid"))
if pid_alive_probe is not None
else None
)
liveness = self.is_live(record, now=now, pid_alive=pid_alive)
return {
"success": False,
"registered": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_IDENTITY_COLLISION,
"collision": True,
"collision_kind": (
"active_worker" if liveness["live"] else "stale_registration"
),
"existing_registration": _public_record(record),
"existing_liveness": liveness,
"reasons": [
f"worker identity {worker_identity!r} is already registered "
f"(status={record.get('status')}, live={liveness['live']}); "
"registration is refused rather than replacing, adopting, or "
"merging with the existing worker (#948 AC31)"
],
"exact_next_action": (
"Generate a different worker identity and atomically register "
"that replacement; do not reuse or take over this one."
),
}
epoch = self._next_epoch(conn, generation_id)
conn.execute(
"""
INSERT INTO worker_registrations (
worker_identity, client_name, client_instance_id, session_id,
generation_id, role, profile, namespace, remote,
repository_binding, pid, transport, token_fingerprint,
started_at, last_heartbeat_at, heartbeat_ttl_seconds,
fencing_epoch, status
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
""",
(
worker_identity,
normalize_client_name(client_name or parsed["client_name"]),
client_instance_id,
session_id,
generation_id,
role,
profile,
namespace,
remote,
repository_binding,
int(pid) if pid is not None else None,
transport,
token_fingerprint,
stamp,
stamp,
float(heartbeat_ttl_seconds),
epoch,
STATUS_ACTIVE,
),
)
row = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
record = self._row_to_record(row)
return {
"success": True,
"registered": True,
"mutation_performed": True,
"blocker_kind": BLOCKER_NONE,
"collision": False,
"registration": _public_record(record),
"fencing_epoch": record["fencing_epoch"],
"reasons": [],
}
@staticmethod
def _next_epoch(conn: sqlite3.Connection, generation_id: str) -> int:
row = conn.execute(
"SELECT MAX(fencing_epoch) AS hi FROM worker_registrations "
"WHERE generation_id = ?",
(generation_id,),
).fetchone()
current = row["hi"] if row and row["hi"] is not None else 0
return int(current) + 1
def heartbeat(
self,
*,
worker_identity: str,
fencing_epoch: int,
now: datetime | None = None,
) -> dict[str, Any]:
"""Renew only the owning registration (#948 AC11).
A stale epoch is refused rather than silently renewed, so a superseded
session that resumes cannot heartbeat its way back into ownership.
"""
stamp = _ts(now or _utc_now())
with self._tx() as conn:
row = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
if row is None:
return {
"success": False,
"renewed": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_NO_ATTACHMENT,
"reasons": [f"no registration for {worker_identity!r}"],
}
record = self._row_to_record(row)
if record["status"] != STATUS_ACTIVE:
return {
"success": False,
"renewed": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_FENCED,
"reasons": [
f"registration status is {record['status']!r}; a superseded "
"registration cannot be renewed (#948 AC15)"
],
}
if int(record["fencing_epoch"]) != int(fencing_epoch):
return {
"success": False,
"renewed": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_FENCED,
"recorded_epoch": int(record["fencing_epoch"]),
"presented_epoch": int(fencing_epoch),
"reasons": [
f"fencing epoch {fencing_epoch} does not match the recorded "
f"epoch {record['fencing_epoch']}; this session has been "
"superseded (#948 AC15/AC16)"
],
}
conn.execute(
"UPDATE worker_registrations SET last_heartbeat_at = ? "
"WHERE worker_identity = ?",
(stamp, worker_identity),
)
return {
"success": True,
"renewed": True,
"mutation_performed": True,
"blocker_kind": BLOCKER_NONE,
"last_heartbeat_at": stamp,
"fencing_epoch": int(fencing_epoch),
"reasons": [],
}
def claim_generation(
self,
*,
worker_identity: str,
generation_id: str,
now: datetime | None = None,
pid_alive_probe=None,
) -> dict[str, Any]:
"""Bind a generation to this worker, superseding only a dead claimant.
A generation held by another *live* session is refused: one daemon must
never be simultaneously claimed by conflicting live sessions. A
generation whose claimant is no longer live is taken over with a higher
fencing epoch, so stale ownership cannot permanently strand a healthy
daemon (#948 AC8/AC14).
"""
with self._tx() as conn:
mine = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
if mine is None:
return {
"success": False,
"claimed": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_NO_ATTACHMENT,
"reasons": [
f"{worker_identity!r} is not registered; register before "
"claiming a generation"
],
}
holders = [
self._row_to_record(r)
for r in conn.execute(
"SELECT * FROM worker_registrations "
"WHERE generation_id = ? AND status = ? AND worker_identity != ?",
(generation_id, STATUS_ACTIVE, worker_identity),
).fetchall()
]
live_holders = []
for holder in holders:
pid_alive = (
pid_alive_probe(holder.get("pid"))
if pid_alive_probe is not None
else None
)
if self.is_live(holder, now=now, pid_alive=pid_alive)["live"]:
live_holders.append(holder)
if live_holders:
return {
"success": False,
"claimed": False,
"mutation_performed": False,
"blocker_kind": BLOCKER_CONFLICTING_SESSIONS,
"conflicting_owners": [_public_record(h) for h in live_holders],
"reasons": [
f"generation {generation_id!r} is already claimed by "
f"{len(live_holders)} live session(s): "
+ ", ".join(
f"{h['worker_identity']} (session {h['session_id']})"
for h in live_holders
)
+ "; a daemon must not be simultaneously claimed by "
"conflicting live sessions (#948)"
],
"exact_next_action": (
"Wait for the live owner's lease to lapse, or attach this "
"worker to its own generation. Do not kill the other process."
),
}
superseded = [h["worker_identity"] for h in holders]
for identity in superseded:
conn.execute(
"UPDATE worker_registrations SET status = ? "
"WHERE worker_identity = ?",
(STATUS_SUPERSEDED, identity),
)
epoch = self._next_epoch(conn, generation_id)
conn.execute(
"UPDATE worker_registrations SET generation_id = ?, fencing_epoch = ?, "
"status = ?, last_heartbeat_at = ? WHERE worker_identity = ?",
(
generation_id,
epoch,
STATUS_ACTIVE,
_ts(now or _utc_now()),
worker_identity,
),
)
row = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
record = self._row_to_record(row)
return {
"success": True,
"claimed": True,
"mutation_performed": True,
"blocker_kind": BLOCKER_NONE,
"registration": _public_record(record),
"fencing_epoch": record["fencing_epoch"],
"superseded_workers": superseded,
"reasons": (
[
f"took over generation {generation_id!r} from "
f"{len(superseded)} non-live claimant(s) with fencing epoch "
f"{record['fencing_epoch']}"
]
if superseded
else []
),
}
def release(
self, *, worker_identity: str, now: datetime | None = None
) -> dict[str, Any]:
"""Mark a registration released. Idempotent; never deletes history."""
with self._tx() as conn:
row = conn.execute(
"SELECT * FROM worker_registrations WHERE worker_identity = ?",
(worker_identity,),
).fetchone()
if row is None:
return {
"success": True,
"released": False,
"mutation_performed": False,
"reasons": [f"no registration for {worker_identity!r}"],
}
conn.execute(
"UPDATE worker_registrations SET status = ?, last_heartbeat_at = ? "
"WHERE worker_identity = ?",
(STATUS_RELEASED, _ts(now or _utc_now()), worker_identity),
)
return {
"success": True,
"released": True,
"mutation_performed": True,
"reasons": [],
}
def _public_record(record: dict[str, Any]) -> dict[str, Any]:
"""Registry row minus anything that should not travel to an LLM surface."""
return {
"worker_identity": record.get("worker_identity"),
"client_name": record.get("client_name"),
"client_instance_id": record.get("client_instance_id"),
"session_id": record.get("session_id"),
"generation_id": record.get("generation_id"),
"role": record.get("role"),
"profile": record.get("profile"),
"namespace": record.get("namespace"),
"remote": record.get("remote"),
"repository_binding": record.get("repository_binding"),
"pid": record.get("pid"),
"transport": record.get("transport"),
"started_at": record.get("started_at"),
"last_heartbeat_at": record.get("last_heartbeat_at"),
"fencing_epoch": record.get("fencing_epoch"),
"status": record.get("status"),
}
# --- The single authoritative verdict ------------------------------------
def assess_provenance(
*,
registry: WorkerRegistry | None = None,
worker_identity: str | None = None,
generation_id: str | None = None,
env: dict[str, str] | None = None,
native_transport_bound: bool | None = None,
namespace: str | None = None,
profile: str | None = None,
role: str | None = None,
now: datetime | None = None,
pid_alive_probe=None,
stdin_is_tty: bool | None = None,
declared_only: bool = False,
) -> dict[str, Any]:
"""Resolve client/session provenance once, for every surface to reuse.
Runtime context, namespace health, namespace attachment, fleet inventory,
capability resolution, and typed reconnect assessment all call this, so the
contradiction that motivated #948 — one surface reporting ``client_managed``
while another reported ``manual_launch`` for the same process — cannot recur
by construction.
Two independent dimensions are reported:
``launch_provenance``
How the process was started. Answered from the environment, which is
legitimate because the launcher is what sets it. Preserves the #686
manual-launch wall unchanged.
``session_ownership``
Which live client session owns this runtime now. Answered *only* from a
live attachment record; no environment flag can establish it.
Missing evidence yields :data:`PROVENANCE_UNPROVEN`, which is deliberately
distinct from :data:`PROVENANCE_MANUAL`: "we cannot prove who owns this" is
not the same claim as "a human launched this from a terminal", and only the
latter justifies telling an operator to stop hand-launching servers.
"""
env_signal = env_provenance_signal(env)
launch = assess_launch_provenance(
env, stdin_is_tty=stdin_is_tty, declared_only=declared_only
)
evidence: list[str] = [EVIDENCE_ENV_FLAG] if env_signal["keys_present"] else []
reasons: list[str] = []
record: dict[str, Any] | None = None
liveness: dict[str, Any] = {"live": False, "reasons": ["no attachment record"]}
if registry is not None and worker_identity:
record = registry.get(worker_identity)
if record is not None:
pid_alive = (
pid_alive_probe(record.get("pid"))
if pid_alive_probe is not None
else None
)
liveness = registry.is_live(record, now=now, pid_alive=pid_alive)
if liveness["live"]:
evidence.append(EVIDENCE_ATTACHMENT_RECORD)
if pid_alive is not None:
evidence.append(EVIDENCE_PROCESS_LIVENESS)
if native_transport_bound:
evidence.append(EVIDENCE_NATIVE_TRANSPORT)
# Conflicting live claims on one generation are contradictory evidence: two
# sessions cannot both currently own the same daemon, so neither claim is
# trustworthy and the assessment fails closed rather than picking a winner.
conflicting: list[dict[str, Any]] = []
effective_generation = generation_id or (record or {}).get("generation_id")
if registry is not None and effective_generation:
for holder in registry.list_workers(generation_id=effective_generation):
if (
record is not None
and holder["worker_identity"] == record["worker_identity"]
):
continue
pid_alive = (
pid_alive_probe(holder.get("pid"))
if pid_alive_probe is not None
else None
)
if registry.is_live(holder, now=now, pid_alive=pid_alive)["live"]:
conflicting.append(_public_record(holder))
trusted = bool(set(evidence) & SUFFICIENT_EVIDENCE)
blocker = BLOCKER_NONE
if conflicting and trusted:
trusted = False
blocker = BLOCKER_CONTRADICTORY
reasons.append(
f"generation {effective_generation!r} is claimed by "
f"{len(conflicting) + 1} live sessions at once; contradictory ownership "
"evidence fails closed (#948)"
)
elif not trusted:
blocker = BLOCKER_NO_ATTACHMENT
if record is None:
reasons.append(
"no live client/session attachment record binds this runtime to a "
"session; a process environment flag alone does not prove current "
"session ownership (#948)"
)
else:
reasons.extend(
f"attachment record is not live: {r}" for r in liveness["reasons"]
)
if conflicting:
session_ownership = OWNERSHIP_CONTESTED
elif trusted:
session_ownership = OWNERSHIP_OWNED
else:
session_ownership = OWNERSHIP_UNOWNED
# The headline verdict. Session ownership decides it when proven; otherwise
# the launch dimension supplies the more specific of "hand-launched" versus
# "cannot tell", so remediation names the proof that is actually missing.
if session_ownership == OWNERSHIP_OWNED:
provenance = PROVENANCE_CLIENT_SESSION
elif launch["launch_provenance"] == PROVENANCE_MANUAL:
provenance = PROVENANCE_MANUAL
reasons.extend(launch["reasons"])
else:
provenance = PROVENANCE_UNPROVEN
client_name = normalize_client_name(
(record or {}).get("client_name")
or parse_worker_identity(worker_identity)["client_name"]
)
return {
"provenance": provenance,
# #948: the *launch* dimension, which is what this field has always
# meant to the #686 manual-launch wall. Keeping it bound to launch
# provenance means unifying the two models changes which code decides,
# not what gets decided, so no mutation that used to be permitted is
# newly refused.
"is_client_managed": bool(launch["client_managed"]),
"launch_provenance": launch["launch_provenance"],
"launch_decided_by": launch["decided_by"],
"launch_reasons": launch["reasons"],
# The *ownership* dimension, which no environment flag can establish.
"session_ownership": session_ownership,
"session_owned": session_ownership == OWNERSHIP_OWNED,
"trusted_evidence": trusted,
"evidence": sorted(set(evidence)),
"sufficient_evidence_kinds": sorted(SUFFICIENT_EVIDENCE),
"env_signal": env_signal,
"env_flag_only": bool(evidence) and not trusted and not conflicting,
"attachment": _public_record(record) if record else None,
"attachment_liveness": liveness,
"client_name": client_name,
"client_known": client_name != UNKNOWN_CLIENT,
"session_id": (record or {}).get("session_id"),
"client_instance_id": (record or {}).get("client_instance_id"),
"generation_id": effective_generation,
"worker_identity": (record or {}).get("worker_identity") or worker_identity,
"fencing_epoch": (record or {}).get("fencing_epoch"),
"conflicting_live_sessions": conflicting,
"blocker_kind": blocker,
"fail_closed": provenance != PROVENANCE_CLIENT_SESSION,
# #948 AC17/AC41: a refusal names the worker it applies to, never the
# profile, so unrelated healthy workers sharing that profile keep working.
"scope": {
"scope_kind": "worker_identity" if worker_identity else "generation",
"worker_identity": worker_identity,
"generation_id": effective_generation,
"namespace": namespace,
"profile": profile,
"role": role,
"profile_wide": False,
"note": (
"role and profile are reusable capability definitions; this verdict "
"binds only the named worker identity/generation"
),
},
"reasons": reasons,
}
# --- Cohort scoping ------------------------------------------------------
def classify_cohort(
records: list[dict[str, Any]],
*,
now: datetime | None = None,
pid_alive_probe=None,
) -> dict[str, Any]:
"""Separate genuine duplicates from valid concurrent workers (#948 AC43).
The pre-#948 scan flagged every additional process sharing a profile as a
duplicate, which is exactly the fleet-wide wall #948 exists to remove.
Sharing a role or profile is legitimate; reusing a *worker identity*, or two
live sessions claiming one generation, is not.
"""
live: list[dict[str, Any]] = []
for record in records:
pid_alive = (
pid_alive_probe(record.get("pid")) if pid_alive_probe is not None else None
)
if WorkerRegistry.is_live(record, now=now, pid_alive=pid_alive)["live"]:
live.append(record)
by_identity: dict[str, list[dict[str, Any]]] = {}
by_generation: dict[str, list[dict[str, Any]]] = {}
for record in live:
by_identity.setdefault(str(record.get("worker_identity")), []).append(record)
by_generation.setdefault(str(record.get("generation_id")), []).append(record)
duplicate_identities = sorted(k for k, v in by_identity.items() if len(v) > 1)
contested_generations = sorted(k for k, v in by_generation.items() if len(v) > 1)
shared_profiles = sorted(
{
str(r.get("profile"))
for r in live
if r.get("profile")
and sum(1 for o in live if o.get("profile") == r.get("profile")) > 1
}
)
blocked = bool(duplicate_identities or contested_generations)
reasons: list[str] = []
if duplicate_identities:
reasons.append(
f"worker identity reused by more than one live worker: "
f"{duplicate_identities}"
)
if contested_generations:
reasons.append(
f"generation claimed by more than one live session: {contested_generations}"
)
if shared_profiles and not blocked:
reasons.append(
f"profile(s) {shared_profiles} are shared by multiple live workers with "
"distinct identities, which is permitted and does not block mutations (#948)"
)
return {
"live_worker_count": len(live),
"live_workers": [_public_record(r) for r in live],
"duplicate_identities": duplicate_identities,
"contested_generations": contested_generations,
"shared_profiles": shared_profiles,
"profile_sharing_permitted": True,
"blocked": blocked,
"blocker_kind": (
BLOCKER_IDENTITY_COLLISION
if duplicate_identities
else BLOCKER_CONFLICTING_SESSIONS
if contested_generations
else BLOCKER_NONE
),
# Never profile-wide: the block names the offending identities only.
"blocked_worker_identities": sorted(
set(duplicate_identities)
| {
str(r.get("worker_identity"))
for gen in contested_generations
for r in by_generation[gen]
}
),
"reasons": reasons,
}
def scope_runtime_failure(
*,
failure_kind: str,
worker_identity: str | None = None,
generation_id: str | None = None,
profile: str | None = None,
role: str | None = None,
all_live_workers: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
"""Scope a stale-runtime/reconnect failure to one worker (#948 AC41).
Returns the set of workers actually affected and, separately, the healthy
workers that must keep operating. A stale generation affects only its own
ownership scope.
"""
workers = list(all_live_workers or [])
affected = [
_public_record(w)
for w in workers
if (worker_identity and w.get("worker_identity") == worker_identity)
or (
not worker_identity
and generation_id
and w.get("generation_id") == generation_id
)
]
affected_ids = {a["worker_identity"] for a in affected}
unaffected = [
_public_record(w)
for w in workers
if w.get("worker_identity") not in affected_ids
]
return {
"failure_kind": failure_kind,
"scope_kind": "worker_identity" if worker_identity else "generation",
"worker_identity": worker_identity,
"generation_id": generation_id,
"profile": profile,
"role": role,
"profile_wide": False,
"fleet_wide": False,
"affected_workers": affected,
"unaffected_workers": unaffected,
"unaffected_worker_count": len(unaffected),
"reasons": [
f"{failure_kind} is scoped to "
+ (
f"worker identity {worker_identity!r}"
if worker_identity
else f"generation {generation_id!r}"
)
+ f"; {len(unaffected)} other live worker(s) sharing this role/profile "
"are unaffected and remain able to mutate (#948 AC41)"
],
}
# --- Client-specific remediation ----------------------------------------
def reconnect_client_for(
provenance: dict[str, Any] | None = None,
*,
fallback: str | None = None,
) -> str:
"""Name the client to address reconnect guidance to.
Derived from the live attachment record, never a hardcoded vendor. When
nothing identifies the client, :data:`UNKNOWN_CLIENT` is returned so the
caller emits generic steps instead of steps for the wrong product.
"""
name = normalize_client_name((provenance or {}).get("client_name"))
if name != UNKNOWN_CLIENT:
return name
return normalize_client_name(fallback)
# --- Remote binding ------------------------------------------------------
def resolve_bound_remote(
*,
requested_remote: str | None,
bound_remote: str | None,
default_remote: str | None = None,
) -> dict[str, Any]:
"""Keep a namespace on the remote it is bound to (#948).
A tool default is not a decision. When a namespace is bound to a remote,
an omitted argument resolves to that binding rather than silently falling
through to the library default and hitting the wrong host; an argument that
contradicts the binding is refused rather than honoured.
"""
requested = (requested_remote or "").strip() or None
bound = (bound_remote or "").strip() or None
default = (default_remote or "").strip() or None
if bound and requested and requested != bound:
return {
"remote": bound,
"resolved_from": "session_binding",
"drifted": True,
"honoured_request": False,
"reasons": [
f"requested remote {requested!r} contradicts the session binding "
f"{bound!r}; the binding wins and the request is refused so a "
"PRGS-bound namespace never mutates another host (#948)"
],
}
if bound and not requested:
return {
"remote": bound,
"resolved_from": "session_binding",
"drifted": False,
"honoured_request": False,
"reasons": [
f"remote omitted; resolved to the session binding {bound!r} rather "
f"than the library default {default!r} (#948)"
],
}
if requested:
return {
"remote": requested,
"resolved_from": "explicit_argument",
"drifted": False,
"honoured_request": True,
"reasons": [],
}
return {
"remote": default,
"resolved_from": "library_default",
"drifted": False,
"honoured_request": False,
"reasons": [
"no remote requested and no session binding recorded; falling back to "
f"the library default {default!r}. Bind the session or pass remote "
"explicitly to avoid host drift (#948)."
],
}