jcwalker3andClaude Opus 4.8
1a38ef95e3
Two defects left behind by #948 made sanctioned multi-client operation
impossible. They are inseparable: fixing either alone still leaves the
multi-client canary unable to run.
1. Client-identity environment keys were not recognized.
gitea_mcp_server reads GITEA_MCP_CLIENT, GITEA_MCP_CLIENT_INSTANCE and
GITEA_MCP_CLIENT_SESSION as the authoritative client-identity inputs for
worker registration, but none of the three appeared in
RECOGNIZED_GITEA_ENV_KEYS or matched a recognized prefix. The runtime
diagnostic scans every peer mcp_server.py process environment and classifies
any unlisted GITEA_* key as an unsupported override, which is raised as a
runtime blocker, so gitea_resolve_task_capability returned
blocker_kind=runtime_reconnect_required with stop_required=true. Because that
resolver is the mandatory preflight for every author, reviewer and merger
mutation, setting the very variable #948 requires closed the mutation gate
for the whole fleet, and reconnecting could not clear it: the variable is
re-exported from the client's server definition on every launch.
The three keys are now named individually in the recognized-key set. No
prefix is added, so an unrecognized GITEA_* override is still refused
exactly as before.
A related inconsistency in the same path is also fixed. The diagnostic
reasons are raised as one RuntimeError, but the preflight re-raise
recognized only "stale-runtime:", so an "unsupported-env:" reason was
silently swallowed there while still failing the resolver. Both reason
families now live in RUNTIME_DIAGNOSTIC_HARD_PREFIXES beside the function
that produces them, and both propagate identically. This only widens what is
refused, never what is permitted.
2. WorkerRegistry.heartbeat() had no production caller.
#948 delivered heartbeat() but only tests called it. The single production
writer registers once per process behind an attempted-once flag, and
register() stamps the same timestamp into both started_at and
last_heartbeat_at. Nothing advanced it afterwards: no lifespan hook, no
background task, no atexit handler in a process that blocks in mcp.run().
Since liveness is age against heartbeat_ttl_seconds, that TTL was not a
liveness window at all but a hard cap on how long any client could stay
attached; at 900 seconds a healthy, connected, client-managed process became
session_ownership=unowned with blocker_kind=session_attachment_missing.
WorkerHeartbeatSupervisor in mcp_worker_identity is the missing caller,
started from _active_worker_identity() at the moment register() succeeds,
because that is the only point where identity and fencing_epoch are both
known. It is a daemon thread rather than an asyncio task or a request-driven
refresh because renewal must survive an idle session, and because the
registry performs blocking BEGIN IMMEDIATE sqlite writes that must not run on
the server's event loop. daemon=True is deliberate: a hard kill takes the
thread with it, so a dead worker still goes stale on the normal TTL.
heartbeat_interval_for() returns one third of the TTL, hard-capped at one
half, so two consecutive beats can be lost without the row expiring and no
override can produce an interval that outlives the registration it renews.
heartbeat() gains optional keyword-only expectations (session, generation,
client name, pid); each supplied one must match the recorded row or the
renewal is refused with the existing BLOCKER_FENCED literal rather than a new
blocker_kind, since consumers switch on that value. Omitting them preserves
the pre-existing behavior exactly. Client names are compared normalized, so
several namespaces of one application stay one client while separate
applications stay distinct.
A terminal refusal stops the supervisor permanently and records why, so a
fenced session can never beat its way back into ownership. A transient
failure is counted and beating continues. An atexit hook stops it on orderly
shutdown. status() is surfaced read-only as worker_heartbeat on
gitea_get_runtime_context so a stopped heartbeat is diagnosable before the
TTL turns it into session_attachment_missing; it grants nothing.
claim_generation() still has no production caller. It bumps fencing_epoch,
which would fence the supervisor's cached epoch, and the strict refusal is
left in place deliberately: auto-re-adopting a bumped epoch would defeat
fencing.
No lock or lease TTL is changed, including the author issue-lock TTL, and no
mutation refused today becomes permitted.
Tests: tests/test_issue_975_client_identity_heartbeat.py adds 40 focused tests
covering all 13 acceptance criteria. Every TTL assertion uses an injected
clock; no test waits for a real TTL. The thread-loop tests use a
millisecond-scale interval with bounded polling.
Focused: 40 passed, 15 subtests passed.
Full suite from inside the branches worktree: 28 failed, 6105 passed, 6 skipped,
1105 subtests — an identical failure set to the 324a0c8a baseline measured in a
sibling branches worktree (28 failed, 6065 passed, 1090 subtests). Zero new
failures; the delta is exactly the added tests.
Closes #975
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
760 lines
28 KiB
Python
760 lines
28 KiB
Python
"""Client-identity env recognition and the production heartbeat lifecycle (#975).
|
|
|
|
Two defects left behind by #948, covered here against the issue's acceptance
|
|
criteria:
|
|
|
|
1. ``GITEA_MCP_CLIENT`` / ``GITEA_MCP_CLIENT_INSTANCE`` /
|
|
``GITEA_MCP_CLIENT_SESSION`` were consumed by production while absent from
|
|
the recognized-key allowlist, so the peer-env scan classified them as
|
|
unsupported overrides and the capability resolver refused every mutation
|
|
fleet-wide.
|
|
2. ``WorkerRegistry.heartbeat()`` had no production caller, so
|
|
``last_heartbeat_at`` never left ``started_at`` and ``heartbeat_ttl_seconds``
|
|
became a hard cap on attachment rather than a liveness window.
|
|
|
|
Every TTL assertion here uses an injected clock (AC13). No test waits for a
|
|
real TTL. The one test that exercises the real thread loop uses a
|
|
millisecond-scale interval and polls with a bounded timeout, never a TTL.
|
|
|
|
All client, session, and generation identifiers are synthetic.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import tempfile
|
|
import threading
|
|
import time
|
|
import unittest
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
import gitea_config
|
|
import mcp_worker_identity as mwi
|
|
|
|
|
|
NOW = datetime(2026, 7, 30, 0, 40, 9, tzinfo=timezone.utc)
|
|
|
|
TTL = 900.0
|
|
|
|
#: The three keys production reads as client identity.
|
|
CLIENT_IDENTITY_ENV_KEYS = (
|
|
"GITEA_MCP_CLIENT",
|
|
"GITEA_MCP_CLIENT_INSTANCE",
|
|
"GITEA_MCP_CLIENT_SESSION",
|
|
)
|
|
|
|
|
|
def _registry() -> mwi.WorkerRegistry:
|
|
"""A registry on a throwaway path; never the operator's real one."""
|
|
handle, path = tempfile.mkstemp(suffix=".sqlite3")
|
|
os.close(handle)
|
|
os.unlink(path)
|
|
return mwi.WorkerRegistry(path)
|
|
|
|
|
|
def _attach(
|
|
registry: mwi.WorkerRegistry,
|
|
*,
|
|
client: str = "claude_code",
|
|
session: str = "sess-0001",
|
|
generation: str = "gen-0001",
|
|
profile: str = "prgs-author",
|
|
role: str = "author",
|
|
pid: int = 4242,
|
|
now: datetime = NOW,
|
|
ttl: float = TTL,
|
|
) -> dict:
|
|
"""Register one synthetic worker and return the outcome plus its identity."""
|
|
identity = mwi.generate_worker_identity(client, session, now=now)
|
|
outcome = registry.register(
|
|
worker_identity=identity,
|
|
client_name=client,
|
|
client_instance_id=f"inst-{session}",
|
|
session_id=session,
|
|
generation_id=generation,
|
|
role=role,
|
|
profile=profile,
|
|
pid=pid,
|
|
heartbeat_ttl_seconds=ttl,
|
|
now=now,
|
|
)
|
|
outcome["identity"] = identity
|
|
return outcome
|
|
|
|
|
|
def _supervisor(
|
|
registry: mwi.WorkerRegistry,
|
|
outcome: dict,
|
|
*,
|
|
client: str = "claude_code",
|
|
session: str = "sess-0001",
|
|
generation: str = "gen-0001",
|
|
pid: int = 4242,
|
|
ttl: float = TTL,
|
|
interval_seconds: float | None = None,
|
|
clock=None,
|
|
) -> mwi.WorkerHeartbeatSupervisor:
|
|
"""A supervisor bound to *outcome*'s registration. Never started here."""
|
|
return mwi.WorkerHeartbeatSupervisor(
|
|
registry,
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
session_id=session,
|
|
generation_id=generation,
|
|
client_name=client,
|
|
pid=pid,
|
|
ttl_seconds=ttl,
|
|
interval_seconds=interval_seconds,
|
|
clock=clock,
|
|
)
|
|
|
|
|
|
class _ManualClock:
|
|
"""Controlled time. Nothing in this suite sleeps to reach a TTL."""
|
|
|
|
def __init__(self, start: datetime = NOW) -> None:
|
|
self.now = start
|
|
|
|
def __call__(self) -> datetime:
|
|
return self.now
|
|
|
|
def advance(self, seconds: float) -> datetime:
|
|
self.now = self.now + timedelta(seconds=seconds)
|
|
return self.now
|
|
|
|
|
|
# --- AC1-AC3: environment recognition ------------------------------------
|
|
|
|
|
|
class ClientIdentityEnvRecognitionTests(unittest.TestCase):
|
|
"""AC1-AC3: the keys production consumes are the keys the gate accepts."""
|
|
|
|
def test_each_client_identity_key_is_recognized(self):
|
|
# AC1: individually asserted, so a partial fix cannot pass.
|
|
for key in CLIENT_IDENTITY_ENV_KEYS:
|
|
with self.subTest(key=key):
|
|
self.assertIn(
|
|
key,
|
|
gitea_config.RECOGNIZED_GITEA_ENV_KEYS,
|
|
f"{key} is read by the server but not recognized by the gate",
|
|
)
|
|
|
|
def test_recognized_client_identity_keys_are_not_unconsumed(self):
|
|
# AC3 at the classification layer the resolver reads: with only these
|
|
# keys set there is no unsupported override to raise a blocker from.
|
|
env = {
|
|
"GITEA_MCP_CLIENT": "claude_code",
|
|
"GITEA_MCP_CLIENT_INSTANCE": "inst-0001",
|
|
"GITEA_MCP_CLIENT_SESSION": "sess-0001",
|
|
}
|
|
self.assertEqual(gitea_config.get_unconsumed_gitea_env_overrides(env), {})
|
|
|
|
def test_unknown_gitea_key_is_still_unsupported(self):
|
|
# AC2: the gate is not broadened. An unrecognised override is refused
|
|
# exactly as before, and the recognised siblings do not shield it.
|
|
env = {
|
|
"GITEA_MCP_CLIENT": "claude_code",
|
|
"GITEA_TOTALLY_INVENTED_OVERRIDE": "1",
|
|
}
|
|
unconsumed = gitea_config.get_unconsumed_gitea_env_overrides(env)
|
|
self.assertIn("GITEA_TOTALLY_INVENTED_OVERRIDE", unconsumed)
|
|
self.assertNotIn("GITEA_MCP_CLIENT", unconsumed)
|
|
|
|
def test_no_new_prefix_admits_arbitrary_client_keys(self):
|
|
# A prefix would have been the lazy fix; it would admit anything
|
|
# beginning GITEA_MCP_CLIENT, including typos and future unknowns.
|
|
env = {"GITEA_MCP_CLIENT_SOMETHING_ELSE": "x"}
|
|
self.assertIn(
|
|
"GITEA_MCP_CLIENT_SOMETHING_ELSE",
|
|
gitea_config.get_unconsumed_gitea_env_overrides(env),
|
|
)
|
|
|
|
def test_server_consumes_exactly_the_recognized_key_names(self):
|
|
# Guards the actual defect class: the constants production reads and the
|
|
# allowlist drifting apart again.
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
for name in (
|
|
mcp_server.CLIENT_NAME_ENV,
|
|
mcp_server.CLIENT_INSTANCE_ENV,
|
|
mcp_server.CLIENT_SESSION_ENV,
|
|
):
|
|
with self.subTest(env_key=name):
|
|
self.assertIn(name, gitea_config.RECOGNIZED_GITEA_ENV_KEYS)
|
|
|
|
|
|
class RuntimeDiagnosticReasonPropagationTests(unittest.TestCase):
|
|
"""`unsupported-env:` must not be swallowed where `stale-runtime:` is not."""
|
|
|
|
def test_both_reason_families_are_hard_prefixes(self):
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
self.assertIn("stale-runtime:", mcp_server.RUNTIME_DIAGNOSTIC_HARD_PREFIXES)
|
|
self.assertIn("unsupported-env:", mcp_server.RUNTIME_DIAGNOSTIC_HARD_PREFIXES)
|
|
|
|
def test_unsupported_env_message_is_recognized_as_hard(self):
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
message = (
|
|
"unsupported-env: Unsupported GITEA_* environment variable override(s) "
|
|
"detected: GITEA_INVENTED=1. Unknown env overrides are unsupported."
|
|
)
|
|
self.assertTrue(
|
|
any(p in message for p in mcp_server.RUNTIME_DIAGNOSTIC_HARD_PREFIXES)
|
|
)
|
|
|
|
|
|
# --- Interval policy ------------------------------------------------------
|
|
|
|
|
|
class HeartbeatIntervalPolicyTests(unittest.TestCase):
|
|
"""The interval must always leave room for lost beats inside the TTL."""
|
|
|
|
def test_default_interval_is_one_third_of_ttl(self):
|
|
self.assertAlmostEqual(mwi.heartbeat_interval_for(900.0), 300.0)
|
|
|
|
def test_interval_is_capped_at_half_the_ttl(self):
|
|
# An override may not exceed the ceiling, or it would expire the very
|
|
# registration it exists to renew.
|
|
self.assertAlmostEqual(mwi.heartbeat_interval_for(900.0, 5000), 450.0)
|
|
|
|
def test_override_below_the_ceiling_is_honoured(self):
|
|
self.assertAlmostEqual(mwi.heartbeat_interval_for(900.0, 120), 120.0)
|
|
|
|
def test_unparsable_or_nonpositive_override_falls_back_to_policy(self):
|
|
for bad in ("", "abc", "0", "-5", None):
|
|
with self.subTest(override=bad):
|
|
self.assertAlmostEqual(mwi.heartbeat_interval_for(900.0, bad), 300.0)
|
|
|
|
def test_interval_is_always_strictly_below_the_ttl(self):
|
|
for ttl in (1.0, 30.0, 900.0, 14400.0):
|
|
with self.subTest(ttl=ttl):
|
|
self.assertLess(mwi.heartbeat_interval_for(ttl), ttl)
|
|
|
|
|
|
# --- AC4-AC6: a healthy worker stays owned --------------------------------
|
|
|
|
|
|
class HealthyWorkerRemainsOwnedTests(unittest.TestCase):
|
|
"""AC4-AC6: the TTL becomes a liveness window instead of an attachment cap."""
|
|
|
|
def test_unsupervised_worker_goes_unowned_at_the_ttl(self):
|
|
# The regression itself, asserted so the fix cannot be mistaken for a
|
|
# TTL change: without a beater the row still expires exactly as before.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
record = registry.get(outcome["identity"])
|
|
self.assertEqual(record["started_at"], record["last_heartbeat_at"])
|
|
|
|
after_ttl = NOW + timedelta(seconds=TTL + 1)
|
|
liveness = registry.is_live(record, now=after_ttl, pid_alive=True)
|
|
self.assertFalse(liveness["live"])
|
|
|
|
assessment = mwi.assess_provenance(
|
|
registry=registry,
|
|
worker_identity=outcome["identity"],
|
|
generation_id="gen-0001",
|
|
env={"GITEA_CLIENT_MANAGED": "1"},
|
|
native_transport_bound=True,
|
|
now=after_ttl,
|
|
pid_alive_probe=lambda _pid: True,
|
|
)
|
|
self.assertEqual(assessment["session_ownership"], "unowned")
|
|
self.assertEqual(assessment["blocker_kind"], mwi.BLOCKER_NO_ATTACHMENT)
|
|
|
|
def test_supervised_worker_stays_owned_past_two_full_ttls(self):
|
|
# AC4: beyond two complete TTL periods of simulated time.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
interval = supervisor.interval_seconds
|
|
self.assertLess(interval, TTL)
|
|
|
|
elapsed = 0.0
|
|
target = TTL * 2 + interval
|
|
while elapsed < target:
|
|
clock.advance(interval)
|
|
elapsed += interval
|
|
result = supervisor.beat_once()
|
|
self.assertTrue(result["renewed"], result.get("reasons"))
|
|
|
|
assessment = mwi.assess_provenance(
|
|
registry=registry,
|
|
worker_identity=outcome["identity"],
|
|
generation_id="gen-0001",
|
|
env={"GITEA_CLIENT_MANAGED": "1"},
|
|
native_transport_bound=True,
|
|
now=clock.now,
|
|
pid_alive_probe=lambda _pid: True,
|
|
)
|
|
self.assertEqual(
|
|
assessment["session_ownership"],
|
|
"owned",
|
|
f"ownership lost after {elapsed:.0f}s of simulated time",
|
|
)
|
|
|
|
self.assertGreater(elapsed, TTL * 2)
|
|
|
|
def test_last_heartbeat_advances_and_diverges_from_started_at(self):
|
|
# AC5.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
before = registry.get(outcome["identity"])
|
|
clock.advance(300)
|
|
supervisor.beat_once()
|
|
after = registry.get(outcome["identity"])
|
|
|
|
self.assertEqual(before["started_at"], after["started_at"])
|
|
self.assertNotEqual(after["started_at"], after["last_heartbeat_at"])
|
|
self.assertGreater(after["last_heartbeat_at"], before["last_heartbeat_at"])
|
|
|
|
def test_repeated_heartbeats_create_no_duplicate_rows(self):
|
|
# AC6: heartbeating is not a second registration path.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
for _ in range(5):
|
|
clock.advance(300)
|
|
self.assertTrue(supervisor.beat_once()["renewed"])
|
|
|
|
workers = registry.list_workers()
|
|
self.assertEqual(len(workers), 1)
|
|
self.assertEqual(workers[0]["worker_identity"], outcome["identity"])
|
|
self.assertEqual(supervisor.status()["beats_renewed"], 5)
|
|
|
|
|
|
# --- AC7, AC8, AC12: only the owner may renew -----------------------------
|
|
|
|
|
|
class HeartbeatOwnershipFencingTests(unittest.TestCase):
|
|
"""AC7/AC8/AC12: one worker can never refresh another worker's row."""
|
|
|
|
def test_two_client_names_stay_distinct_rows_and_identities(self):
|
|
# AC7.
|
|
registry = _registry()
|
|
first = _attach(registry, client="claude_code", session="sess-a")
|
|
second = _attach(registry, client="gemini", session="sess-b")
|
|
|
|
self.assertNotEqual(first["identity"], second["identity"])
|
|
rows = {w["worker_identity"]: w for w in registry.list_workers()}
|
|
self.assertEqual(len(rows), 2)
|
|
self.assertEqual(rows[first["identity"]]["client_name"], "claude_code")
|
|
self.assertEqual(rows[second["identity"]]["client_name"], "gemini")
|
|
|
|
def test_wrong_session_cannot_refresh_the_row(self):
|
|
# AC8, session dimension.
|
|
registry = _registry()
|
|
outcome = _attach(registry, session="sess-real")
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_session_id="sess-impostor",
|
|
)
|
|
self.assertFalse(result["renewed"])
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_FENCED)
|
|
self.assertIn(
|
|
"session_id", [field for field, _r, _p in result["expectation_drift"]]
|
|
)
|
|
|
|
def test_wrong_generation_cannot_refresh_the_row(self):
|
|
# AC8, generation dimension.
|
|
registry = _registry()
|
|
outcome = _attach(registry, generation="gen-real")
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_generation_id="gen-other",
|
|
)
|
|
self.assertFalse(result["renewed"])
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_FENCED)
|
|
|
|
def test_wrong_client_name_cannot_refresh_the_row(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry, client="claude_code")
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_client_name="gemini",
|
|
)
|
|
self.assertFalse(result["renewed"])
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_FENCED)
|
|
|
|
def test_wrong_pid_cannot_refresh_the_row(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry, pid=4242)
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_pid=9999,
|
|
)
|
|
self.assertFalse(result["renewed"])
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_FENCED)
|
|
|
|
def test_namespaces_of_one_application_share_a_client_name(self):
|
|
# Several namespaces of one application must stay one client, while
|
|
# separate applications stay distinct. Aliases normalise, and the
|
|
# expectation check compares normalised values, so an author and a
|
|
# reviewer namespace of the same product are not treated as impostors.
|
|
registry = _registry()
|
|
outcome = _attach(registry, client="claude")
|
|
self.assertEqual(
|
|
registry.get(outcome["identity"])["client_name"], "claude_code"
|
|
)
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_client_name="claude_desktop",
|
|
)
|
|
self.assertTrue(result["renewed"], result.get("reasons"))
|
|
|
|
def test_matching_expectations_renew_normally(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
expected_session_id="sess-0001",
|
|
expected_generation_id="gen-0001",
|
|
expected_client_name="claude_code",
|
|
expected_pid=4242,
|
|
)
|
|
self.assertTrue(result["renewed"], result.get("reasons"))
|
|
|
|
def test_omitted_expectations_preserve_pre_fix_behaviour(self):
|
|
# AC12: the #948 contract is unchanged for callers presenting nothing.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
self.assertTrue(
|
|
registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=outcome["fencing_epoch"],
|
|
)["renewed"]
|
|
)
|
|
|
|
def test_stale_fencing_epoch_is_still_refused(self):
|
|
# AC12: the pre-existing epoch fence is untouched.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
result = registry.heartbeat(
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=int(outcome["fencing_epoch"]) + 1,
|
|
)
|
|
self.assertFalse(result["renewed"])
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_FENCED)
|
|
|
|
def test_fenced_supervisor_stops_permanently(self):
|
|
# A fenced session must never beat its way back into ownership.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
supervisor = mwi.WorkerHeartbeatSupervisor(
|
|
registry,
|
|
worker_identity=outcome["identity"],
|
|
fencing_epoch=int(outcome["fencing_epoch"]) + 1,
|
|
session_id="sess-0001",
|
|
generation_id="gen-0001",
|
|
client_name="claude_code",
|
|
pid=4242,
|
|
ttl_seconds=TTL,
|
|
clock=_ManualClock(),
|
|
)
|
|
first = supervisor.beat_once()
|
|
self.assertFalse(first["renewed"])
|
|
self.assertIsNotNone(supervisor.stopped_reason)
|
|
|
|
# A second attempt does not even reach the registry.
|
|
second = supervisor.beat_once()
|
|
self.assertFalse(second["renewed"])
|
|
self.assertFalse(second["beat_attempted"])
|
|
self.assertEqual(supervisor.status()["beats_attempted"], 1)
|
|
|
|
def test_missing_registration_stops_the_supervisor(self):
|
|
registry = _registry()
|
|
supervisor = mwi.WorkerHeartbeatSupervisor(
|
|
registry,
|
|
worker_identity=mwi.generate_worker_identity("grok", "sess-gone", now=NOW),
|
|
fencing_epoch=1,
|
|
ttl_seconds=TTL,
|
|
clock=_ManualClock(),
|
|
)
|
|
result = supervisor.beat_once()
|
|
self.assertEqual(result["blocker_kind"], mwi.BLOCKER_NO_ATTACHMENT)
|
|
self.assertIsNotNone(supervisor.stopped_reason)
|
|
|
|
|
|
# --- AC9, AC10, AC11: stopping and expiry ---------------------------------
|
|
|
|
|
|
class ShutdownAndExpiryTests(unittest.TestCase):
|
|
"""AC9-AC11: orderly shutdown stops beating; dead workers still expire."""
|
|
|
|
def test_stop_ends_the_heartbeat(self):
|
|
# AC9.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
clock.advance(300)
|
|
self.assertTrue(supervisor.beat_once()["renewed"])
|
|
supervisor.stop(reason="orderly shutdown")
|
|
|
|
self.assertFalse(supervisor.status()["running"])
|
|
self.assertEqual(supervisor.stopped_reason, "orderly shutdown")
|
|
|
|
clock.advance(300)
|
|
after_stop = supervisor.beat_once()
|
|
self.assertFalse(after_stop["renewed"])
|
|
self.assertFalse(after_stop["beat_attempted"])
|
|
|
|
def test_stop_is_idempotent_and_keeps_the_first_reason(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
supervisor = _supervisor(registry, outcome, clock=_ManualClock())
|
|
supervisor.stop(reason="orderly shutdown")
|
|
supervisor.stop(reason="second call")
|
|
self.assertEqual(supervisor.stopped_reason, "orderly shutdown")
|
|
|
|
def test_stopped_worker_becomes_unowned_once_the_ttl_elapses(self):
|
|
# AC10: stopping beating restores normal expiration; the fix does not
|
|
# make a worker immortal.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
clock.advance(300)
|
|
supervisor.beat_once()
|
|
supervisor.stop(reason="orderly shutdown")
|
|
|
|
last_beat = clock.now
|
|
clock.advance(TTL + 1)
|
|
record = registry.get(outcome["identity"])
|
|
liveness = registry.is_live(record, now=clock.now, pid_alive=True)
|
|
self.assertFalse(liveness["live"])
|
|
self.assertGreater(liveness["heartbeat_age_seconds"], TTL)
|
|
|
|
assessment = mwi.assess_provenance(
|
|
registry=registry,
|
|
worker_identity=outcome["identity"],
|
|
generation_id="gen-0001",
|
|
env={"GITEA_CLIENT_MANAGED": "1"},
|
|
native_transport_bound=True,
|
|
now=clock.now,
|
|
pid_alive_probe=lambda _pid: True,
|
|
)
|
|
self.assertEqual(assessment["session_ownership"], "unowned")
|
|
self.assertLess(last_beat, clock.now)
|
|
|
|
def test_dead_worker_expires_even_with_a_live_pid(self):
|
|
# A recycled or long-lived pid must not keep a silent worker owned.
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
record = registry.get(outcome["identity"])
|
|
liveness = registry.is_live(
|
|
record, now=NOW + timedelta(seconds=TTL + 1), pid_alive=True
|
|
)
|
|
self.assertFalse(liveness["live"])
|
|
self.assertFalse(liveness["heartbeat_fresh"])
|
|
|
|
def test_one_worker_expiring_does_not_stale_healthy_workers(self):
|
|
# AC11.
|
|
registry = _registry()
|
|
healthy = _attach(registry, client="claude_code", session="sess-live")
|
|
abandoned = _attach(registry, client="gemini", session="sess-dead")
|
|
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(
|
|
registry,
|
|
healthy,
|
|
client="claude_code",
|
|
session="sess-live",
|
|
clock=clock,
|
|
)
|
|
|
|
elapsed = 0.0
|
|
while elapsed < TTL + 300:
|
|
clock.advance(supervisor.interval_seconds)
|
|
elapsed += supervisor.interval_seconds
|
|
self.assertTrue(supervisor.beat_once()["renewed"])
|
|
|
|
healthy_live = registry.is_live(
|
|
registry.get(healthy["identity"]), now=clock.now, pid_alive=True
|
|
)
|
|
abandoned_live = registry.is_live(
|
|
registry.get(abandoned["identity"]), now=clock.now, pid_alive=True
|
|
)
|
|
self.assertTrue(healthy_live["live"])
|
|
self.assertFalse(abandoned_live["live"])
|
|
|
|
|
|
# --- Failure handling and observability -----------------------------------
|
|
|
|
|
|
class HeartbeatFailureHandlingTests(unittest.TestCase):
|
|
"""A heartbeat failure never grants ownership and never crashes a caller."""
|
|
|
|
def test_transient_exception_is_counted_and_beating_continues(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
clock = _ManualClock()
|
|
supervisor = _supervisor(registry, outcome, clock=clock)
|
|
|
|
calls = {"n": 0}
|
|
real = registry.heartbeat
|
|
|
|
def flaky(**kwargs):
|
|
calls["n"] += 1
|
|
if calls["n"] == 1:
|
|
raise RuntimeError("database is locked")
|
|
return real(**kwargs)
|
|
|
|
registry.heartbeat = flaky # type: ignore[method-assign]
|
|
try:
|
|
clock.advance(300)
|
|
first = supervisor.beat_once()
|
|
self.assertFalse(first["renewed"])
|
|
self.assertTrue(first["transient"])
|
|
self.assertIsNone(supervisor.stopped_reason)
|
|
|
|
clock.advance(300)
|
|
second = supervisor.beat_once()
|
|
self.assertTrue(second["renewed"], second.get("reasons"))
|
|
finally:
|
|
registry.heartbeat = real # type: ignore[method-assign]
|
|
|
|
status = supervisor.status()
|
|
self.assertEqual(status["transient_failures"], 1)
|
|
self.assertEqual(status["beats_renewed"], 1)
|
|
self.assertTrue(status["supervised"])
|
|
|
|
def test_status_reports_an_unstarted_supervisor_honestly(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
status = _supervisor(registry, outcome, clock=_ManualClock()).status()
|
|
self.assertTrue(status["supervised"])
|
|
self.assertFalse(status["started"])
|
|
self.assertFalse(status["running"])
|
|
self.assertEqual(status["beats_renewed"], 0)
|
|
self.assertIsNone(status["stopped_reason"])
|
|
|
|
def test_runtime_context_reports_unsupervised_when_none_attached(self):
|
|
# The observability surface must distinguish "no supervisor" from
|
|
# "supervisor with no beats yet", or the regression is invisible again.
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
saved = mcp_server._WORKER_HEARTBEAT_SUPERVISOR
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = None
|
|
try:
|
|
status = mcp_server._worker_heartbeat_status()
|
|
finally:
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = saved
|
|
self.assertFalse(status["supervised"])
|
|
self.assertFalse(status["running"])
|
|
self.assertTrue(status["reasons"])
|
|
|
|
def test_runtime_context_surfaces_an_attached_supervisor(self):
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
supervisor = _supervisor(registry, outcome, clock=_ManualClock())
|
|
|
|
saved = mcp_server._WORKER_HEARTBEAT_SUPERVISOR
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = supervisor
|
|
try:
|
|
status = mcp_server._worker_heartbeat_status()
|
|
finally:
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = saved
|
|
|
|
self.assertTrue(status["supervised"])
|
|
self.assertEqual(status["worker_identity"], outcome["identity"])
|
|
self.assertEqual(status["heartbeat_ttl_seconds"], TTL)
|
|
self.assertLess(status["heartbeat_interval_seconds"], TTL)
|
|
|
|
def test_status_failure_degrades_instead_of_raising(self):
|
|
import gitea_mcp_server as mcp_server
|
|
|
|
class Exploding:
|
|
def status(self):
|
|
raise RuntimeError("boom")
|
|
|
|
saved = mcp_server._WORKER_HEARTBEAT_SUPERVISOR
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = Exploding()
|
|
try:
|
|
status = mcp_server._worker_heartbeat_status()
|
|
finally:
|
|
mcp_server._WORKER_HEARTBEAT_SUPERVISOR = saved
|
|
|
|
self.assertTrue(status["supervised"])
|
|
self.assertFalse(status["running"])
|
|
self.assertTrue(status["reasons"])
|
|
|
|
|
|
class SupervisorThreadLoopTests(unittest.TestCase):
|
|
"""The real thread, on a millisecond interval. Never waits for a TTL."""
|
|
|
|
def test_thread_beats_repeatedly_then_stops_on_request(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
beats = threading.Event()
|
|
seen = {"n": 0}
|
|
real = registry.heartbeat
|
|
|
|
def counting(**kwargs):
|
|
result = real(**kwargs)
|
|
seen["n"] += 1
|
|
if seen["n"] >= 3:
|
|
beats.set()
|
|
return result
|
|
|
|
registry.heartbeat = counting # type: ignore[method-assign]
|
|
supervisor = _supervisor(registry, outcome, ttl=1.0, interval_seconds=0.01)
|
|
try:
|
|
started = supervisor.start()
|
|
self.assertTrue(started["started"])
|
|
self.assertTrue(
|
|
beats.wait(timeout=10.0), "heartbeat thread produced no beats"
|
|
)
|
|
self.assertTrue(supervisor.status()["running"])
|
|
finally:
|
|
supervisor.stop(reason="test teardown")
|
|
registry.heartbeat = real # type: ignore[method-assign]
|
|
|
|
self.assertGreaterEqual(supervisor.status()["beats_renewed"], 3)
|
|
self.assertFalse(supervisor.status()["running"])
|
|
|
|
# Stopping is prompt: the loop waits on an Event, not a sleep.
|
|
deadline = time.monotonic() + 5.0
|
|
while supervisor._thread is not None and supervisor._thread.is_alive():
|
|
if time.monotonic() > deadline:
|
|
self.fail("heartbeat thread did not exit after stop()")
|
|
time.sleep(0.01)
|
|
|
|
def test_start_is_idempotent(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
supervisor = _supervisor(registry, outcome, ttl=1.0, interval_seconds=0.05)
|
|
try:
|
|
self.assertFalse(supervisor.start()["already_running"])
|
|
self.assertTrue(supervisor.start()["already_running"])
|
|
finally:
|
|
supervisor.stop(reason="test teardown")
|
|
|
|
def test_start_refuses_after_a_terminal_stop(self):
|
|
registry = _registry()
|
|
outcome = _attach(registry)
|
|
supervisor = _supervisor(registry, outcome, clock=_ManualClock())
|
|
supervisor.stop(reason="orderly shutdown")
|
|
self.assertFalse(supervisor.start()["started"])
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|