Every pre-existing runtime surface is per-process. gitea_get_runtime_context and gitea_assess_master_parity describe only the server answering the call, and gitea_assess_mcp_namespace_health accepts process, probe_result and registered_tools from the caller, so it cannot constrain the caller. The control-plane sessions table records allocator task sessions, not server processes. Five self-reports of one revision therefore never proved that five processes exist, that no sixth exists, or that all five share one client cohort. Add gitea_assess_fleet_inventory, a strictly read-only capability that takes no evidence parameters. It combines two sources that must agree: - a control-plane runtime registry row each server writes about itself, from the official entrypoint, immediately after the native transport bind; - a process observation the answering server performs, never the caller. A member is running only when both agree and the observed process predates its registration, so configuration alone never counts and a recycled PID cannot impersonate an exited server. Classification is a pure function of the snapshot, so gitea-controller and gitea-reconciler return the same verdict. Missing, duplicate, unexpected, stale and unregistered members are reported in separate fields rather than collapsed into one error, because each needs a different operator action. single_cohort is derived only from recorded cohort identity and stays null when unknown, so matching revisions never establish a cohort. Any unreadable registry, unavailable listing, unregistered process, unknown cohort or unknown revision sets inventory_complete and mutation_gate_satisfied false with a specific blocked_reason. The capability performs no restart, reconnect, drain, lease mutation, issue mutation or process termination, never terminates a duplicate, and never manufactures restart evidence. The only signal sent is signal 0. Also resolves the fleet namespace from the expected roster: role_namespace_gate.infer_mcp_namespace recognises only author and reviewer and echoes the profile name otherwise, which would have mislabelled the controller, merger and reconciler members. Widening that shared helper is controller role-metadata work owned by #950 and is deliberately not done here. Schema v5 to v6 is additive and idempotent: one new table, no existing table, tool signature or result field changed. Two control-plane tests that pinned the schema version to the literal 5 now assert against SCHEMA_VERSION. #950 (controller role metadata), #951 (restart receipts) and #952 (stale-lease consistency) remain separate and untouched. Tests: 118 new across tests/test_mcp_fleet_inventory.py, tests/test_control_plane_db_server_runtimes.py and tests/test_fleet_inventory_tool.py, covering every acceptance criterion including the multi-LLM duplicate-server regression that motivated the issue. Closes #949 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NBaQKcRRKeKbZuhk72MhEf
786 lines
30 KiB
Python
786 lines
30 KiB
Python
"""Authoritative, read-only PRGS MCP fleet inventory (#949).
|
|
|
|
Every pre-existing runtime surface is *per process*. ``gitea_get_runtime_context``
|
|
and ``gitea_assess_master_parity`` describe only the server answering the call.
|
|
``gitea_assess_mcp_namespace_health`` accepts ``process``, ``probe_result`` and
|
|
``registered_tools`` **from the caller**, so it cannot constrain the caller. The
|
|
control-plane ``sessions`` table records allocator *task* sessions, not server
|
|
processes. Five independent self-reports of the same revision therefore never
|
|
proved that exactly five processes exist, that no sixth exists, or that all five
|
|
belong to one client cohort.
|
|
|
|
Evidence model
|
|
--------------
|
|
Two independent sources must agree before a fleet member counts as running:
|
|
|
|
``control-plane runtime registry``
|
|
A row each server writes **about itself** at native transport bind
|
|
(:func:`build_process_runtime_record`). No caller can supply it. It is
|
|
authoritative for identity: namespace, profile, role, repository binding,
|
|
cohort, and the revision the process started at.
|
|
|
|
``server-side process observation``
|
|
A process listing performed by the server answering the inventory call
|
|
(:func:`scan_mcp_server_processes`), never by the caller. It is
|
|
authoritative for existence and liveness, and it is the only source that
|
|
can show a process the registry does not know about.
|
|
|
|
A member is ``live`` only when a registry row has a matching, still-running
|
|
process whose start time precedes the registration (so a recycled PID cannot
|
|
impersonate a dead server). Anything the two sources cannot jointly establish
|
|
is reported as unknown and fails the mutation gate closed — configuration alone
|
|
never counts as a running member, and matching Git revisions never establish a
|
|
single cohort.
|
|
|
|
This module performs no restart, reconnect, lease mutation, issue mutation, or
|
|
process termination. The only signal it ever sends is ``signal 0`` liveness
|
|
probing, which delivers nothing to the target process.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import secrets
|
|
import subprocess
|
|
from datetime import datetime, timezone
|
|
from typing import Any, Iterable, Mapping, Sequence
|
|
|
|
# ── expected fleet ────────────────────────────────────────────────────────────
|
|
|
|
# The configured PRGS fleet. Each entry is one expected member; the roster is
|
|
# the definition of "expected" for missing/unexpected classification.
|
|
EXPECTED_PRGS_FLEET: tuple[dict[str, str], ...] = (
|
|
{"namespace": "gitea-author", "profile": "prgs-author", "role": "author"},
|
|
{
|
|
"namespace": "gitea-controller",
|
|
"profile": "prgs-controller",
|
|
"role": "controller",
|
|
},
|
|
{"namespace": "gitea-reviewer", "profile": "prgs-reviewer", "role": "reviewer"},
|
|
{"namespace": "gitea-merger", "profile": "prgs-merger", "role": "merger"},
|
|
{
|
|
"namespace": "gitea-reconciler",
|
|
"profile": "prgs-reconciler",
|
|
"role": "reconciler",
|
|
},
|
|
)
|
|
|
|
# Cohort identity supplied by a client that manages the whole fleet.
|
|
COHORT_ID_ENV = "GITEA_MCP_CLIENT_COHORT_ID"
|
|
|
|
# Registry rows older than this are pruned at *registration* time (a startup
|
|
# write), never on the read path. Dead rows inside the window are still reported
|
|
# as stale evidence rather than silently dropped.
|
|
RUNTIME_RETENTION_SECONDS = 7 * 24 * 3600
|
|
|
|
# Liveness classifications for a registry row.
|
|
LIVENESS_LIVE = "live"
|
|
LIVENESS_DEAD = "dead"
|
|
LIVENESS_PID_RECYCLED = "pid_recycled"
|
|
LIVENESS_UNOBSERVED = "unobserved"
|
|
LIVENESS_UNKNOWN = "unknown"
|
|
|
|
# Per-member health classifications.
|
|
HEALTH_RUNNING = "running"
|
|
HEALTH_MISSING = "missing"
|
|
HEALTH_DUPLICATE = "duplicate"
|
|
HEALTH_UNEXPECTED = "unexpected"
|
|
HEALTH_STALE = "stale"
|
|
HEALTH_UNKNOWN = "unknown"
|
|
|
|
EVIDENCE_AUTHORITY = "control_plane_runtime_registry+server_process_observation"
|
|
|
|
_MCP_PROCESS_MARKER = "mcp_server.py"
|
|
_LSTART_FORMAT = "%a %b %d %H:%M:%S %Y"
|
|
_ISO_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
|
|
|
|
|
|
# ── time helpers ──────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _utcnow() -> datetime:
|
|
return datetime.now(timezone.utc)
|
|
|
|
|
|
def iso_now() -> str:
|
|
return _utcnow().strftime(_ISO_FORMAT)
|
|
|
|
|
|
def _parse_iso(value: Any) -> datetime | None:
|
|
text = (str(value) if value is not None else "").strip()
|
|
if not text:
|
|
return None
|
|
if text.endswith("Z"):
|
|
text = text[:-1] + "+00:00"
|
|
try:
|
|
parsed = datetime.fromisoformat(text)
|
|
except ValueError:
|
|
return None
|
|
if parsed.tzinfo is None:
|
|
parsed = parsed.replace(tzinfo=timezone.utc)
|
|
return parsed.astimezone(timezone.utc)
|
|
|
|
|
|
def _int_or_none(value: Any) -> int | None:
|
|
try:
|
|
return int(value)
|
|
except (TypeError, ValueError):
|
|
return None
|
|
|
|
|
|
def _clean(value: Any) -> str | None:
|
|
text = (str(value) if value is not None else "").strip()
|
|
return text or None
|
|
|
|
|
|
# ── process-level evidence (server side only) ─────────────────────────────────
|
|
|
|
|
|
def probe_pid_alive(pid: int | None) -> bool | None:
|
|
"""Return whether *pid* exists. ``None`` when it cannot be determined.
|
|
|
|
Uses ``signal 0``, which performs a permission/existence check and delivers
|
|
nothing to the target. This module never sends a terminating signal.
|
|
"""
|
|
resolved = _int_or_none(pid)
|
|
if resolved is None or resolved <= 0:
|
|
return None
|
|
try:
|
|
os.kill(resolved, 0)
|
|
except ProcessLookupError:
|
|
return False
|
|
except PermissionError:
|
|
# The process exists but belongs to another user.
|
|
return True
|
|
except OSError:
|
|
return None
|
|
return True
|
|
|
|
|
|
def scan_mcp_server_processes(*, runner=subprocess.run) -> dict[str, Any]:
|
|
"""Observe running Gitea MCP server processes from this server process.
|
|
|
|
This is deliberately performed by the answering server, never by the caller:
|
|
a caller-supplied process list is exactly the input a fleet gate must not
|
|
trust. When the listing cannot be obtained the result reports
|
|
``available=False`` so the inventory fails closed instead of assuming that
|
|
no unregistered process exists.
|
|
"""
|
|
try:
|
|
proc = runner(
|
|
["ps", "-o", "pid,lstart,command", "-ax"],
|
|
capture_output=True,
|
|
text=True,
|
|
check=True,
|
|
)
|
|
except Exception as exc: # noqa: BLE001 - any failure means unknown evidence
|
|
return {
|
|
"available": False,
|
|
"processes": [],
|
|
"reason": f"process listing unavailable: {exc}",
|
|
}
|
|
|
|
processes: list[dict[str, Any]] = []
|
|
for raw_line in (proc.stdout or "").splitlines()[1:]:
|
|
line = raw_line.strip()
|
|
if not line or _MCP_PROCESS_MARKER not in line:
|
|
continue
|
|
parts = line.split(None, 6)
|
|
if len(parts) < 7:
|
|
continue
|
|
pid = _int_or_none(parts[0])
|
|
if pid is None:
|
|
continue
|
|
try:
|
|
naive = datetime.strptime(" ".join(parts[1:6]), _LSTART_FORMAT)
|
|
started_at = naive.astimezone(timezone.utc)
|
|
except (ValueError, OSError):
|
|
started_at = None
|
|
processes.append(
|
|
{
|
|
"pid": pid,
|
|
"started_at": started_at.strftime(_ISO_FORMAT) if started_at else None,
|
|
"command": parts[6],
|
|
}
|
|
)
|
|
|
|
processes.sort(key=lambda item: item["pid"])
|
|
return {"available": True, "processes": processes, "reason": None}
|
|
|
|
|
|
# ── cohort / registration record ──────────────────────────────────────────────
|
|
|
|
|
|
def namespace_for_profile(profile: str | None, *, default: str | None = None) -> str | None:
|
|
"""Map a configured profile to its fleet namespace.
|
|
|
|
Resolved from the expected roster rather than from name-shape heuristics.
|
|
``role_namespace_gate.infer_mcp_namespace`` only recognises author and
|
|
reviewer, so it returns the *profile* name for controller, merger, and
|
|
reconciler — which would label three of the five members with a namespace
|
|
that does not exist. Widening that helper is controller role-metadata work
|
|
and belongs to #950; the roster already carries the mapping this inventory
|
|
needs, so it is read from there.
|
|
|
|
A profile outside the roster falls back to *default* (typically the
|
|
caller's existing inference), so an unexpected member is still described
|
|
rather than dropped.
|
|
"""
|
|
cleaned = _clean(profile)
|
|
for entry in EXPECTED_PRGS_FLEET:
|
|
if entry["profile"] == cleaned:
|
|
return entry["namespace"]
|
|
return default if default is not None else cleaned
|
|
|
|
|
|
def derive_cohort_identity(env: Mapping[str, str] | None = None) -> dict[str, Any]:
|
|
"""Derive the client/cohort identity of this server process.
|
|
|
|
A cohort is the set of servers a single client launched together. The
|
|
parent process is the durable expression of that: an IDE/CLI client spawns
|
|
every namespace as its own child. A process whose parent has gone away
|
|
(reparented to init) cannot prove which cohort it belongs to, and says so
|
|
rather than guessing.
|
|
|
|
Revisions are deliberately not consulted here. Two servers built from the
|
|
same commit are not thereby one cohort (#949 AC7).
|
|
"""
|
|
source_env = os.environ if env is None else env
|
|
explicit = _clean(source_env.get(COHORT_ID_ENV))
|
|
if explicit:
|
|
return {"cohort_id": explicit, "cohort_source": "explicit_env"}
|
|
try:
|
|
ppid = os.getppid()
|
|
except OSError:
|
|
ppid = 0
|
|
if ppid and ppid > 1:
|
|
return {"cohort_id": f"ppid:{ppid}", "cohort_source": "parent_process"}
|
|
return {
|
|
"cohort_id": None,
|
|
"cohort_source": "unknown",
|
|
"cohort_reason": (
|
|
"parent process is unavailable or reparented to init; this server "
|
|
"cannot prove which client cohort launched it"
|
|
),
|
|
}
|
|
|
|
|
|
def build_process_runtime_record(
|
|
*,
|
|
namespace: str,
|
|
profile: str | None,
|
|
role: str | None,
|
|
remote: str | None = None,
|
|
org: str | None = None,
|
|
repo: str | None = None,
|
|
repository_root: str | None = None,
|
|
pid: int | None = None,
|
|
startup_head: str | None = None,
|
|
daemon_start_head: str | None = None,
|
|
transport: str | None = None,
|
|
client_provenance: str | None = None,
|
|
env: Mapping[str, str] | None = None,
|
|
boot_id: str | None = None,
|
|
registered_at: str | None = None,
|
|
) -> dict[str, Any]:
|
|
"""Build the row a server writes about itself at native transport bind.
|
|
|
|
Every field describes the *calling* process. Nothing here is caller-supplied
|
|
in the MCP sense: the only code that reaches this function is the official
|
|
entrypoint of the process being described.
|
|
"""
|
|
cohort = derive_cohort_identity(env)
|
|
resolved_pid = _int_or_none(pid)
|
|
if resolved_pid is None:
|
|
resolved_pid = os.getpid()
|
|
token = boot_id or secrets.token_hex(8)
|
|
return {
|
|
"runtime_id": f"{namespace}:{resolved_pid}:{token}",
|
|
"namespace": namespace,
|
|
"profile": _clean(profile),
|
|
"role": _clean(role),
|
|
"remote": _clean(remote),
|
|
"org": _clean(org),
|
|
"repo": _clean(repo),
|
|
"repository_root": _clean(repository_root),
|
|
"pid": resolved_pid,
|
|
"cohort_id": cohort["cohort_id"],
|
|
"cohort_source": cohort["cohort_source"],
|
|
"client_provenance": _clean(client_provenance) or "unknown",
|
|
"boot_id": token,
|
|
"startup_head": _clean(startup_head),
|
|
"daemon_start_head": _clean(daemon_start_head),
|
|
"transport": _clean(transport),
|
|
"registered_at": registered_at or iso_now(),
|
|
"status": "running",
|
|
}
|
|
|
|
|
|
# ── classification ────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _expected_index(
|
|
expected_fleet: Sequence[Mapping[str, str]],
|
|
) -> dict[str, dict[str, str]]:
|
|
index: dict[str, dict[str, str]] = {}
|
|
for entry in expected_fleet:
|
|
profile = _clean(entry.get("profile"))
|
|
if profile:
|
|
index[profile] = dict(entry)
|
|
return index
|
|
|
|
|
|
def _sort_key(member: Mapping[str, Any]) -> tuple:
|
|
return (
|
|
str(member.get("namespace") or ""),
|
|
str(member.get("profile") or ""),
|
|
_int_or_none(member.get("pid")) or 0,
|
|
str(member.get("runtime_id") or ""),
|
|
)
|
|
|
|
|
|
def _normalize_row(
|
|
row: Mapping[str, Any],
|
|
*,
|
|
observed_by_pid: Mapping[int, Mapping[str, Any]],
|
|
process_scan_available: bool,
|
|
now: datetime,
|
|
) -> dict[str, Any]:
|
|
pid = _int_or_none(row.get("pid"))
|
|
member: dict[str, Any] = {
|
|
"runtime_id": _clean(row.get("runtime_id")),
|
|
"namespace": _clean(row.get("namespace")),
|
|
"profile": _clean(row.get("profile")),
|
|
"role": _clean(row.get("role")),
|
|
"remote": _clean(row.get("remote")),
|
|
"org": _clean(row.get("org")),
|
|
"repo": _clean(row.get("repo")),
|
|
"repository_root": _clean(row.get("repository_root")),
|
|
"pid": pid,
|
|
"cohort_id": _clean(row.get("cohort_id")),
|
|
"cohort_source": _clean(row.get("cohort_source")) or "unknown",
|
|
"client_provenance": _clean(row.get("client_provenance")) or "unknown",
|
|
"boot_id": _clean(row.get("boot_id")),
|
|
"startup_head": _clean(row.get("startup_head")),
|
|
"daemon_start_head": _clean(row.get("daemon_start_head")),
|
|
"transport": _clean(row.get("transport")),
|
|
"registered_at": _clean(row.get("registered_at")),
|
|
"last_heartbeat_at": _clean(row.get("last_heartbeat_at")),
|
|
"recorded_status": _clean(row.get("status")) or "unknown",
|
|
}
|
|
|
|
pid_alive = probe_pid_alive(pid)
|
|
member["pid_alive"] = pid_alive
|
|
|
|
observed = observed_by_pid.get(pid) if pid is not None else None
|
|
member["process_observed"] = bool(observed) if process_scan_available else None
|
|
|
|
if not process_scan_available:
|
|
# Existence cannot be corroborated; never upgrade to live on the
|
|
# registry's word alone.
|
|
member["liveness"] = LIVENESS_UNKNOWN
|
|
member["liveness_reason"] = (
|
|
"process observation unavailable; registry rows cannot be corroborated"
|
|
)
|
|
elif pid_alive is False:
|
|
member["liveness"] = LIVENESS_DEAD
|
|
member["liveness_reason"] = "recorded PID is not running"
|
|
elif pid_alive is None:
|
|
member["liveness"] = LIVENESS_UNKNOWN
|
|
member["liveness_reason"] = "PID liveness could not be determined"
|
|
elif observed is None:
|
|
member["liveness"] = LIVENESS_UNOBSERVED
|
|
member["liveness_reason"] = (
|
|
"recorded PID is not a running Gitea MCP server process"
|
|
)
|
|
else:
|
|
started_at = _parse_iso(observed.get("started_at"))
|
|
registered_at = _parse_iso(member["registered_at"])
|
|
if started_at and registered_at and started_at > registered_at:
|
|
member["liveness"] = LIVENESS_PID_RECYCLED
|
|
member["liveness_reason"] = (
|
|
"the process now holding this PID started after the registry row "
|
|
"was written; the registered server is gone"
|
|
)
|
|
else:
|
|
member["liveness"] = LIVENESS_LIVE
|
|
member["liveness_reason"] = None
|
|
|
|
heartbeat = _parse_iso(member["last_heartbeat_at"])
|
|
member["heartbeat_age_seconds"] = (
|
|
int((now - heartbeat).total_seconds()) if heartbeat else None
|
|
)
|
|
return member
|
|
|
|
|
|
def _binding_matches(
|
|
member: Mapping[str, Any], expected_binding: Mapping[str, Any] | None
|
|
) -> bool | None:
|
|
if not expected_binding:
|
|
return None
|
|
for field in ("remote", "org", "repo"):
|
|
expected = _clean(expected_binding.get(field))
|
|
if expected is None:
|
|
continue
|
|
actual = _clean(member.get(field))
|
|
if actual is None:
|
|
return None
|
|
if actual != expected:
|
|
return False
|
|
return True
|
|
|
|
|
|
def classify_fleet_inventory(
|
|
*,
|
|
runtime_rows: Iterable[Mapping[str, Any]],
|
|
process_scan: Mapping[str, Any] | None = None,
|
|
expected_fleet: Sequence[Mapping[str, str]] = EXPECTED_PRGS_FLEET,
|
|
expected_binding: Mapping[str, Any] | None = None,
|
|
registry_available: bool = True,
|
|
registry_error: str | None = None,
|
|
now: datetime | None = None,
|
|
answering_namespace: str | None = None,
|
|
) -> dict[str, Any]:
|
|
"""Classify a fleet snapshot. Pure: identical input yields identical output.
|
|
|
|
The verdict never depends on which namespace asked, so controller and
|
|
reconciler agree by construction; ``answering_namespace`` is reported as
|
|
metadata only.
|
|
"""
|
|
moment = now or _utcnow()
|
|
scan = dict(process_scan or {"available": False, "processes": [], "reason": None})
|
|
scan_available = bool(scan.get("available"))
|
|
observed_processes = list(scan.get("processes") or [])
|
|
observed_by_pid: dict[int, Mapping[str, Any]] = {}
|
|
for proc in observed_processes:
|
|
observed_pid = _int_or_none(proc.get("pid"))
|
|
if observed_pid is not None:
|
|
observed_by_pid[observed_pid] = proc
|
|
|
|
expected_index = _expected_index(expected_fleet)
|
|
|
|
members = [
|
|
_normalize_row(
|
|
row,
|
|
observed_by_pid=observed_by_pid,
|
|
process_scan_available=scan_available,
|
|
now=moment,
|
|
)
|
|
for row in (runtime_rows or [])
|
|
]
|
|
|
|
live = [m for m in members if m["liveness"] == LIVENESS_LIVE]
|
|
not_live = [m for m in members if m["liveness"] != LIVENESS_LIVE]
|
|
|
|
live_by_profile: dict[str, list[dict[str, Any]]] = {}
|
|
for member in live:
|
|
live_by_profile.setdefault(member["profile"] or "", []).append(member)
|
|
|
|
running_members: list[dict[str, Any]] = []
|
|
missing_members: list[dict[str, Any]] = []
|
|
duplicate_members: list[dict[str, Any]] = []
|
|
unexpected_members: list[dict[str, Any]] = []
|
|
binding_mismatches: list[dict[str, Any]] = []
|
|
role_mismatches: list[dict[str, Any]] = []
|
|
configured_members: list[dict[str, Any]] = []
|
|
|
|
for entry in expected_fleet:
|
|
profile = _clean(entry.get("profile")) or ""
|
|
instances = sorted(live_by_profile.get(profile, []), key=_sort_key)
|
|
configured_members.append(
|
|
{
|
|
"namespace": _clean(entry.get("namespace")),
|
|
"profile": profile,
|
|
"role": _clean(entry.get("role")),
|
|
"instance_count": len(instances),
|
|
"health": (
|
|
HEALTH_MISSING
|
|
if not instances
|
|
else HEALTH_RUNNING
|
|
if len(instances) == 1
|
|
else HEALTH_DUPLICATE
|
|
),
|
|
}
|
|
)
|
|
if not instances:
|
|
missing_members.append(
|
|
{
|
|
"namespace": _clean(entry.get("namespace")),
|
|
"profile": profile,
|
|
"role": _clean(entry.get("role")),
|
|
"health": HEALTH_MISSING,
|
|
"reason": (
|
|
"no live registry row corroborated by a running server "
|
|
"process"
|
|
),
|
|
}
|
|
)
|
|
continue
|
|
for instance in instances:
|
|
instance["health"] = (
|
|
HEALTH_RUNNING if len(instances) == 1 else HEALTH_DUPLICATE
|
|
)
|
|
instance["expected"] = True
|
|
running_members.append(instance)
|
|
if len(instances) > 1:
|
|
duplicate_members.append(
|
|
{
|
|
"namespace": _clean(entry.get("namespace")),
|
|
"profile": profile,
|
|
"role": _clean(entry.get("role")),
|
|
"health": HEALTH_DUPLICATE,
|
|
"instance_count": len(instances),
|
|
"pids": sorted(
|
|
i["pid"] for i in instances if i["pid"] is not None
|
|
),
|
|
"instances": instances,
|
|
"reason": (
|
|
"more than one live server is registered for this profile"
|
|
),
|
|
}
|
|
)
|
|
|
|
for member in sorted(live, key=_sort_key):
|
|
profile = member["profile"] or ""
|
|
expected_entry = expected_index.get(profile)
|
|
if expected_entry is not None:
|
|
expected_role = _clean(expected_entry.get("role"))
|
|
actual_role = member["role"]
|
|
if expected_role and actual_role and actual_role != expected_role:
|
|
role_mismatches.append(
|
|
{
|
|
"namespace": member["namespace"],
|
|
"profile": profile,
|
|
"pid": member["pid"],
|
|
"expected_role": expected_role,
|
|
"declared_role": actual_role,
|
|
"reason": "declared role does not match the configured profile",
|
|
}
|
|
)
|
|
expected_namespace = _clean(expected_entry.get("namespace"))
|
|
if (
|
|
expected_namespace
|
|
and member["namespace"]
|
|
and member["namespace"] != expected_namespace
|
|
):
|
|
role_mismatches.append(
|
|
{
|
|
"namespace": member["namespace"],
|
|
"profile": profile,
|
|
"pid": member["pid"],
|
|
"expected_namespace": expected_namespace,
|
|
"declared_role": member["role"],
|
|
"reason": (
|
|
"profile is served from a namespace it is not "
|
|
"configured for"
|
|
),
|
|
}
|
|
)
|
|
else:
|
|
member["health"] = HEALTH_UNEXPECTED
|
|
member["expected"] = False
|
|
unexpected_members.append(member)
|
|
|
|
match = _binding_matches(member, expected_binding)
|
|
member["repository_binding_matches"] = match
|
|
if match is False:
|
|
binding_mismatches.append(
|
|
{
|
|
"namespace": member["namespace"],
|
|
"profile": profile,
|
|
"pid": member["pid"],
|
|
"remote": member["remote"],
|
|
"org": member["org"],
|
|
"repo": member["repo"],
|
|
"expected": dict(expected_binding or {}),
|
|
"reason": "member is bound to a different repository",
|
|
}
|
|
)
|
|
elif match is None and expected_binding:
|
|
binding_mismatches.append(
|
|
{
|
|
"namespace": member["namespace"],
|
|
"profile": profile,
|
|
"pid": member["pid"],
|
|
"remote": member["remote"],
|
|
"org": member["org"],
|
|
"repo": member["repo"],
|
|
"expected": dict(expected_binding or {}),
|
|
"reason": "member did not record a complete repository binding",
|
|
}
|
|
)
|
|
|
|
stale_members: list[dict[str, Any]] = []
|
|
for member in sorted(not_live, key=_sort_key):
|
|
member["health"] = (
|
|
HEALTH_UNKNOWN if member["liveness"] == LIVENESS_UNKNOWN else HEALTH_STALE
|
|
)
|
|
stale_members.append(member)
|
|
|
|
registered_pids = {m["pid"] for m in live if m["pid"] is not None}
|
|
unregistered_processes: list[dict[str, Any]] = []
|
|
if scan_available:
|
|
for proc in observed_processes:
|
|
observed_pid = _int_or_none(proc.get("pid"))
|
|
if observed_pid is None or observed_pid in registered_pids:
|
|
continue
|
|
unregistered_processes.append(
|
|
{"pid": observed_pid, "started_at": proc.get("started_at")}
|
|
)
|
|
unregistered_processes.sort(key=lambda item: item["pid"])
|
|
|
|
# Cohort. Derived only from recorded cohort identity — never from revisions.
|
|
cohort_ids = {m["cohort_id"] for m in live}
|
|
cohort_unknown = any(cohort_id is None for cohort_id in cohort_ids)
|
|
known_cohorts = sorted(c for c in cohort_ids if c is not None)
|
|
if not live or cohort_unknown:
|
|
single_cohort: bool | None = None
|
|
mixed_cohort: bool | None = None
|
|
else:
|
|
single_cohort = len(known_cohorts) == 1
|
|
mixed_cohort = len(known_cohorts) > 1
|
|
|
|
# Revision spread. Reported independently of cohort; never used to infer it.
|
|
heads = {m["startup_head"] for m in live}
|
|
head_unknown = any(head is None for head in heads)
|
|
known_heads = sorted(h for h in heads if h is not None)
|
|
mixed_revision = (len(known_heads) > 1) if known_heads else None
|
|
|
|
exactly_one_per_profile = not missing_members and not duplicate_members
|
|
no_unexpected_members = not unexpected_members
|
|
|
|
incomplete_reasons: list[str] = []
|
|
if not registry_available:
|
|
incomplete_reasons.append(
|
|
registry_error or "the control-plane runtime registry could not be read"
|
|
)
|
|
if not scan_available:
|
|
incomplete_reasons.append(
|
|
str(scan.get("reason") or "server-side process observation unavailable")
|
|
)
|
|
if unregistered_processes:
|
|
pids = ", ".join(str(p["pid"]) for p in unregistered_processes)
|
|
incomplete_reasons.append(
|
|
f"running Gitea MCP server process(es) with no runtime registry row "
|
|
f"(PIDs: {pids}); the fleet contains members this inventory cannot "
|
|
f"describe"
|
|
)
|
|
if live and cohort_unknown:
|
|
incomplete_reasons.append(
|
|
"one or more live members did not record a client cohort identity; "
|
|
"matching revisions do not establish a single cohort"
|
|
)
|
|
if live and head_unknown:
|
|
incomplete_reasons.append(
|
|
"one or more live members did not record a startup revision"
|
|
)
|
|
if any(m["liveness"] == LIVENESS_UNKNOWN for m in members):
|
|
incomplete_reasons.append(
|
|
"liveness of one or more registry rows could not be determined"
|
|
)
|
|
|
|
inventory_complete = not incomplete_reasons
|
|
|
|
blocked_reasons: list[str] = list(incomplete_reasons)
|
|
if missing_members:
|
|
names = ", ".join(sorted(m["profile"] for m in missing_members))
|
|
blocked_reasons.append(f"expected fleet member(s) not running: {names}")
|
|
if duplicate_members:
|
|
names = ", ".join(sorted(d["profile"] for d in duplicate_members))
|
|
blocked_reasons.append(
|
|
f"duplicate server(s) registered for profile(s): {names}"
|
|
)
|
|
if unexpected_members:
|
|
names = ", ".join(
|
|
sorted(
|
|
str(m["profile"] or m["namespace"] or "?") for m in unexpected_members
|
|
)
|
|
)
|
|
blocked_reasons.append(f"unexpected fleet member(s) running: {names}")
|
|
if mixed_cohort:
|
|
blocked_reasons.append(
|
|
"live members span more than one client cohort: "
|
|
+ ", ".join(known_cohorts)
|
|
)
|
|
if mixed_revision:
|
|
blocked_reasons.append(
|
|
"live members started at different revisions: " + ", ".join(known_heads)
|
|
)
|
|
if binding_mismatches:
|
|
blocked_reasons.append(
|
|
"one or more live members are not bound to the expected repository"
|
|
)
|
|
if role_mismatches:
|
|
blocked_reasons.append(
|
|
"one or more live members declare a role or namespace that does not "
|
|
"match the configured profile"
|
|
)
|
|
|
|
mutation_gate_satisfied = bool(
|
|
inventory_complete
|
|
and exactly_one_per_profile
|
|
and no_unexpected_members
|
|
and single_cohort is True
|
|
and mixed_revision is False
|
|
and not binding_mismatches
|
|
and not role_mismatches
|
|
)
|
|
if not mutation_gate_satisfied and not blocked_reasons:
|
|
blocked_reasons.append(
|
|
"the fleet snapshot did not establish the exact-one-instance-per-"
|
|
"profile, single-cohort invariant"
|
|
)
|
|
|
|
return {
|
|
"success": True,
|
|
"read_only": True,
|
|
"evidence_authority": EVIDENCE_AUTHORITY,
|
|
"answering_namespace": _clean(answering_namespace),
|
|
"generated_at": moment.strftime(_ISO_FORMAT),
|
|
"inventory_complete": inventory_complete,
|
|
"incomplete_reasons": incomplete_reasons,
|
|
"configured_members": configured_members,
|
|
"running_members": sorted(running_members, key=_sort_key),
|
|
"missing_members": sorted(missing_members, key=_sort_key),
|
|
"duplicate_members": sorted(duplicate_members, key=_sort_key),
|
|
"unexpected_members": sorted(unexpected_members, key=_sort_key),
|
|
"stale_members": stale_members,
|
|
"unregistered_processes": unregistered_processes,
|
|
"repository_binding_mismatches": sorted(binding_mismatches, key=_sort_key),
|
|
"role_mismatches": sorted(role_mismatches, key=_sort_key),
|
|
"cohort_ids": known_cohorts,
|
|
"single_cohort": single_cohort,
|
|
"mixed_cohort": mixed_cohort,
|
|
"startup_revisions": known_heads,
|
|
"mixed_revision": mixed_revision,
|
|
"exactly_one_per_profile": exactly_one_per_profile,
|
|
"no_unexpected_members": no_unexpected_members,
|
|
"expected_member_count": len(expected_fleet),
|
|
"running_member_count": len(running_members),
|
|
"mutation_gate_satisfied": mutation_gate_satisfied,
|
|
"blocked_reason": blocked_reasons[0] if blocked_reasons else None,
|
|
"blocked_reasons": blocked_reasons,
|
|
"process_observation": {
|
|
"available": scan_available,
|
|
"observed_process_count": len(observed_processes),
|
|
"reason": scan.get("reason"),
|
|
},
|
|
"registry": {
|
|
"available": registry_available,
|
|
"row_count": len(members),
|
|
"error": registry_error,
|
|
},
|
|
"mutations_performed": [],
|
|
}
|
|
|
|
|
|
def summarize(result: Mapping[str, Any]) -> str:
|
|
"""One-line human summary of a classification result."""
|
|
if result.get("mutation_gate_satisfied"):
|
|
return (
|
|
f"fleet healthy: {result.get('running_member_count')} of "
|
|
f"{result.get('expected_member_count')} members running in a single "
|
|
f"cohort at one revision"
|
|
)
|
|
return f"fleet not provable: {result.get('blocked_reason')}"
|