Compare commits

..
Author SHA1 Message Date
sysadmin 1be4600261 fix: address #625 REQUEST_CHANGES for lease reclaim test and EOF blank line
- Update MCP expired-lock recovery test for sticky expired ownership
  (live pid + present worktree still fail closed under #601 reclaim rules)
- Remove trailing blank line at EOF in control_plane_db.py
2026-07-10 11:54:05 -04:00
sysadmin 780ef0b1be feat: first-class control-plane lease lifecycle (Closes #601)
Make active leases queryable workflow state with canonical adopt, release,
expire/reclaim, and abandon operations on the #613 control-plane DB substrate.

- lease_lifecycle.py: policy, proof gates, safe_next_action vocabulary
- control_plane_db: schema v3 provenance columns + list/inspect/adopt/abandon
- MCP tools: gitea_*_workflow_lease(s) for list/inspect/adopt/release/expire/abandon/reclaim
- issue_lock_store: sanctioned reclaim of expired author locks (dead pid / missing wt)
- Tests cover lifecycle, foreign steal refusal, allocator compatibility, merger handoff
2026-07-10 11:30:53 -04:00
sysadmin a654cda058 fix: import Any from typing to resolve IDE/compiler diagnostic warnings 2026-07-10 10:27:21 -04:00
sysadmin ab1c2d7973 Merge pull request 'feat: Sentry/GlitchTip incident bridge via incident_links (Closes #612)' (#623) from feat/issue-612-incident-bridge into master 2026-07-10 08:36:47 -05:00
sysadmin 6a179aeb85 feat: Sentry/GlitchTip incident bridge via incident_links (Closes #612)
Add phase-1 observability bridge that turns provider observations into
normal Gitea issues and control-plane incident_links rows. Dry-run is
default; apply creates/links through sanctioned Gitea issue paths only.
Raw incidents remain non-assignable; the #600 allocator sees bridge work
only as ordinary Gitea issues. Builds on #613 substrate and #600 allocator.

Closes #612
2026-07-10 03:18:54 -04:00
sysadmin f20c8436aa Merge pull request 'feat: controller-owned allocator API on control-plane DB (Closes #600)' (#622) from feat/issue-600-controller-allocator-api into master 2026-07-10 02:11:59 -05:00
sysadmin db5d14184f feat: controller-owned allocator API on control-plane DB (Closes #600)
Add gitea_allocate_next_work and allocator_service routing policy on top of
the #613 ControlPlaneDB substrate. Workers get atomic assign+lease results
(or wait/no_safe_work/terminal-path outcomes) without self-selecting work
via file locks or comment-only leases. #612 remains downstream.

Closes #600
2026-07-10 02:54:20 -04:00
sysadmin 10228e1c06 Merge pull request 'feat: control-plane DB substrate for atomic assign/lease (Closes #613)' (#619) from feat/issue-613-allocator-db-substrate into master 2026-07-10 01:44:19 -05:00
sysadmin d8b2b8f1a7 Merge pull request 'fix: head-scope durable #332 review-decision locks (Closes #620)' (#621) from fix/issue-620-head-scoped-review-locks into master
Reviewed-on: #621
2026-07-10 01:12:57 -05:00
sysadmin 2429d9d2e8 fix: fail closed on conflicting incident_links observation metadata
Legacy NULL-scope dedupe only compared Gitea targets, so duplicate rows
with the same provider key and issue but different fingerprint/status/
event_count/etc. collapsed to the lowest link_id and silently dropped
observation data. Migration now compares all meaningful observation
fields and refuses to discard conflicts (#619 RC3 / #613).
2026-07-09 23:04:17 -04:00
sysadmin 16cd871fbd fix: safe incident_links migration order; require PR head pin
Address remaining PR #619 blockers on head 783ea88:

- Deduplicate legacy NULL-scope incident_links before normalizing to ''
- Fail closed when duplicate rows disagree on Gitea target
- Require non-empty expected_head_sha for PR assign and mutation
2026-07-09 22:51:27 -04:00
sysadmin 783ea88a01 fix: fail closed on stale/terminal assignments; canonicalize incident_links
Address REQUEST_CHANGES on PR #619:

- require_valid_assignment rejects terminal work states and head drift
- incident_links scope keys normalize NULL/blank to '' for UNIQUE
- remove trailing whitespace in control-plane-db-substrate.md
2026-07-09 22:41:25 -04:00
sysadmin 036b78e31e feat: control-plane DB substrate for atomic assign/lease (Closes #613)
Add SQLite single-writer MVP control plane per the allocator ADR:
sessions, work_items (issue/pr only), atomic assign+lease, mutation
gates, terminal-lock index, and provider-neutral incident_links.

DB coordinates concurrency; Gitea remains assignable work; raw
Sentry/GlitchTip incidents are never work items. #600 and #612 build on
this substrate.
2026-07-09 22:27:51 -04:00
14 changed files with 6933 additions and 10 deletions
+610
View File
@@ -0,0 +1,610 @@
"""Controller-owned work allocator policy (#600).
Builds on the #613 control-plane DB substrate (``ControlPlaneDB.assign_and_lease``).
Workers must not self-select exclusive work under the standard multi-LLM
workflow. They call ``gitea_allocate_next_work`` which:
1. Inspects candidate Gitea issues/PRs (never raw monitoring incidents).
2. Applies ADR routing policy (role, terminal path, leases, blocked, deps).
3. Atomically assigns + leases the selected item in one DB transaction.
This module is pure selection + substrate orchestration. Gitea I/O for live
inventory lives in the MCP tool wrapper so tests can inject candidates.
#612 remains downstream: bridge-created Gitea issues become candidates only
after they exist as normal issues; this module never assigns incidents.
"""
from __future__ import annotations
import os
import uuid
from dataclasses import dataclass, field
from typing import Any, Sequence
from control_plane_db import (
ControlPlaneDB,
ControlPlaneError,
InvalidWorkKindError,
LeaseRequiredError,
WORK_KINDS,
)
# Outcomes required by #600 / ADR §5.
OUTCOME_ASSIGNED = "assigned_work"
OUTCOME_WAIT = "wait"
OUTCOME_BLOCKED_TERMINAL = "blocked_by_terminal_path"
OUTCOME_BLOCKED_LEASE = "blocked_by_active_lease"
OUTCOME_NEEDS_CONTROLLER = "needs_controller"
OUTCOME_NO_SAFE = "no_safe_work"
OUTCOME_ROLE_INELIGIBLE = "role_ineligible"
OUTCOME_PREVIEW = "preview" # dry-run only (apply=false)
ROLE_AUTHOR = "author"
ROLE_REVIEWER = "reviewer"
ROLE_MERGER = "merger"
ROLE_RECONCILER = "reconciler"
ROLE_CONTROLLER = "controller"
VALID_ROLES = frozenset(
{ROLE_AUTHOR, ROLE_REVIEWER, ROLE_MERGER, ROLE_RECONCILER, ROLE_CONTROLLER}
)
# Default action matrices by role (mutation gate will re-check).
ROLE_ACTIONS: dict[str, tuple[tuple[str, ...], tuple[str, ...]]] = {
ROLE_AUTHOR: (
("implement", "comment", "push", "create_pr"),
("approve", "merge", "request_changes", "self_select_without_assignment"),
),
ROLE_REVIEWER: (
("review", "comment", "approve", "request_changes"),
("merge", "push", "create_pr", "self_select_without_assignment"),
),
ROLE_MERGER: (
("merge", "comment"),
("approve", "request_changes", "push", "create_pr", "self_select_without_assignment"),
),
ROLE_RECONCILER: (
("comment", "diagnose", "cleanup"),
("approve", "merge", "push", "create_pr", "self_select_without_assignment"),
),
ROLE_CONTROLLER: (
("comment", "diagnose", "allocate"),
("approve", "merge", "push", "create_pr"),
),
}
@dataclass
class WorkCandidate:
"""One assignable Gitea issue or PR presented to the allocator."""
kind: str # issue | pr
number: int
state: str = "open"
labels: tuple[str, ...] = ()
title: str = ""
priority: int = 0
head_sha: str | None = None
# Routing signals (callers derive from Gitea / review feedback).
request_changes_current_head: bool = False
approval_on_current_head: bool = False
approval_stale: bool = False
approval_contaminated: bool = False
mergeable: bool = False
blocked: bool = False
dependency_unmet: bool = False
dependency_reason: str | None = None
already_claimed_elsewhere: bool = False
def __post_init__(self) -> None:
self.kind = (self.kind or "").strip().lower()
self.state = (self.state or "open").strip().lower()
self.labels = tuple(
str(x).strip().lower() for x in (self.labels or ()) if str(x).strip()
)
if self.kind not in WORK_KINDS:
raise InvalidWorkKindError(
f"candidate kind '{self.kind}' is not assignable; only "
f"{sorted(WORK_KINDS)} (never raw incidents)"
)
def as_dict(self) -> dict[str, Any]:
return {
"kind": self.kind,
"number": self.number,
"state": self.state,
"labels": list(self.labels),
"title": self.title,
"priority": self.priority,
"head_sha": self.head_sha,
"request_changes_current_head": self.request_changes_current_head,
"approval_on_current_head": self.approval_on_current_head,
"approval_stale": self.approval_stale,
"approval_contaminated": self.approval_contaminated,
"mergeable": self.mergeable,
"blocked": self.blocked,
"dependency_unmet": self.dependency_unmet,
"dependency_reason": self.dependency_reason,
}
@dataclass
class SkipRecord:
kind: str
number: int
reason: str
def as_dict(self) -> dict[str, Any]:
return {"kind": self.kind, "number": self.number, "reason": self.reason}
def normalize_role(role: str | None, *, profile_name: str | None = None) -> str:
"""Map profile/role strings to a canonical allocator role."""
raw = (role or "").strip().lower()
if raw in VALID_ROLES:
return raw
prof = (profile_name or "").strip().lower()
for token in VALID_ROLES:
if token in prof or prof.endswith(f"-{token}"):
return token
if "author" in raw:
return ROLE_AUTHOR
if "review" in raw:
return ROLE_REVIEWER
if "merg" in raw:
return ROLE_MERGER
if "reconcil" in raw:
return ROLE_RECONCILER
if "control" in raw:
return ROLE_CONTROLLER
raise ControlPlaneError(
f"unknown allocator role '{role}' (profile={profile_name!r}); "
f"expected one of {sorted(VALID_ROLES)}"
)
def expected_role_for_candidate(c: WorkCandidate) -> str:
"""ADR §5.3 routing: which role should take this work next."""
if c.kind == "pr":
if c.approval_contaminated:
return ROLE_RECONCILER
if c.request_changes_current_head:
return ROLE_AUTHOR
if c.approval_stale:
return ROLE_REVIEWER
if c.approval_on_current_head and c.mergeable:
return ROLE_MERGER
# Open PR without terminal verdict → reviewer
return ROLE_REVIEWER
# Issues: ready work → author by default; blocked stays controller/none
labels = set(c.labels)
if "status:blocked" in labels or c.blocked:
return ROLE_CONTROLLER
return ROLE_AUTHOR
def role_actions(role: str) -> tuple[tuple[str, ...], tuple[str, ...]]:
return ROLE_ACTIONS.get(role, ROLE_ACTIONS[ROLE_AUTHOR])
def classify_skip(
c: WorkCandidate,
*,
role: str,
terminal_pr: int | None,
) -> str | None:
"""Return skip reason, or None if candidate is selectable for *role*."""
if c.state in ("merged", "closed"):
return f"{c.kind}#{c.number} is {c.state}; never assign"
if c.blocked or "status:blocked" in c.labels:
return f"{c.kind}#{c.number} is blocked"
if c.dependency_unmet:
return (
c.dependency_reason
or f"{c.kind}#{c.number} has unmet dependencies"
)
if c.already_claimed_elsewhere:
return f"{c.kind}#{c.number} already claimed elsewhere"
if c.kind == "pr" and not (c.head_sha or "").strip():
return f"pr#{c.number} missing head_sha pin"
# Terminal path first: when an active terminal PR exists, only that PR
# (or controller diagnosis) is assignable for review-path roles.
if terminal_pr is not None and c.kind == "pr" and c.number != terminal_pr:
if role in (ROLE_REVIEWER, ROLE_MERGER):
return (
f"pr#{c.number} skipped: active terminal-review lock on "
f"PR #{terminal_pr} must be resolved first"
)
expected = expected_role_for_candidate(c)
if role == ROLE_CONTROLLER:
# Controller may inspect anything but only assigns diagnosis targets
# when contaminated / blocked.
if expected == ROLE_RECONCILER or c.blocked:
return None
return f"{c.kind}#{c.number} does not require controller (expected {expected})"
if role != expected:
return (
f"{c.kind}#{c.number} expects role '{expected}', active role is '{role}'"
)
# Ready-gate for issues: prefer status:ready when labels present.
if c.kind == "issue" and c.labels:
if "status:ready" not in c.labels and "status:in-progress" not in c.labels:
# Allow unlabeled open issues; only skip explicit non-ready states.
if any(l.startswith("status:") for l in c.labels):
return f"issue#{c.number} not status:ready ({','.join(c.labels)})"
return None
def sort_candidates(candidates: Sequence[WorkCandidate]) -> list[WorkCandidate]:
"""Higher priority first; then lower number (older issues) for stability."""
return sorted(
candidates,
key=lambda c: (-int(c.priority), c.kind != "pr", int(c.number)),
)
def allocate_next_work(
db: ControlPlaneDB,
*,
session_id: str,
role: str,
remote: str,
org: str,
repo: str,
candidates: Sequence[WorkCandidate],
apply: bool = False,
profile_name: str | None = None,
username: str | None = None,
lease_ttl_seconds: int | None = None,
) -> dict[str, Any]:
"""Select and optionally reserve the next work unit via control-plane DB.
*apply=False* (default): dry-run selection only — no lease/assignment.
*apply=True*: atomic ``assign_and_lease`` for the selected candidate.
Never uses file locks or comment-only leases as the assignment source.
"""
if db is None:
return {
"success": False,
"outcome": OUTCOME_NO_SAFE,
"reasons": [
"control-plane DB substrate unavailable (fail closed, #600/#613)"
],
"skipped": [],
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
try:
role_norm = normalize_role(role, profile_name=profile_name)
except ControlPlaneError as exc:
return {
"success": False,
"outcome": OUTCOME_ROLE_INELIGIBLE,
"reasons": [str(exc)],
"skipped": [],
"assignment": None,
"substrate": "control_plane_db",
}
session_id = (session_id or "").strip() or f"alloc-{uuid.uuid4().hex[:12]}"
try:
db.upsert_session(
session_id=session_id,
role=role_norm,
profile=profile_name,
pid=os.getpid(),
)
except Exception as exc: # noqa: BLE001 — surface structured
return {
"success": False,
"outcome": OUTCOME_NO_SAFE,
"reasons": [
f"failed to register session in control-plane DB: {exc} "
"(fail closed, #613)"
],
"skipped": [],
"assignment": None,
"substrate": "control_plane_db",
}
# Expire stale leases globally before selection.
try:
db.expire_stale_leases()
except Exception as exc: # noqa: BLE001
return {
"success": False,
"outcome": OUTCOME_NO_SAFE,
"reasons": [f"lease expiry failed: {exc} (fail closed)"],
"skipped": [],
"assignment": None,
"substrate": "control_plane_db",
}
terminal = None
try:
terminal = db.get_active_terminal_lock(remote=remote, org=org, repo=repo)
except Exception as exc: # noqa: BLE001
return {
"success": False,
"outcome": OUTCOME_NO_SAFE,
"reasons": [f"terminal lock lookup failed: {exc} (fail closed)"],
"skipped": [],
"assignment": None,
"substrate": "control_plane_db",
}
terminal_pr = int(terminal["terminal_pr"]) if terminal else None
skipped: list[SkipRecord] = []
ordered = sort_candidates(list(candidates))
selected: WorkCandidate | None = None
for c in ordered:
reason = classify_skip(c, role=role_norm, terminal_pr=terminal_pr)
if reason:
skipped.append(SkipRecord(c.kind, c.number, reason))
continue
selected = c
break
if selected is None:
# If terminal lock blocks all review work, surface that explicitly.
if terminal_pr is not None and role_norm in (ROLE_REVIEWER, ROLE_MERGER):
outcome = OUTCOME_BLOCKED_TERMINAL
reasons = [
f"no safe work for role '{role_norm}': active terminal-review "
f"lock on PR #{terminal_pr} (resolve terminal path first, #332/#600)"
]
else:
outcome = OUTCOME_NO_SAFE
reasons = [
f"no safe assignable work for role '{role_norm}' "
f"among {len(ordered)} candidates"
]
return {
"success": True,
"outcome": outcome,
"apply": bool(apply),
"role": role_norm,
"profile_name": profile_name,
"username": username,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": None,
"expected_role_next": None,
"reasons": reasons,
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
"downstream_note": (
"#612 incident bridge remains downstream of #600; "
"allocator never assigns raw monitoring incidents"
),
}
expected_role = expected_role_for_candidate(selected)
allowed, forbidden = role_actions(role_norm)
selection = {
"kind": selected.kind,
"number": selected.number,
"title": selected.title,
"labels": list(selected.labels),
"head_sha": selected.head_sha,
"priority": selected.priority,
"expected_role_next": expected_role,
"reason_selected": (
f"highest-priority candidate for role '{role_norm}' "
f"(expected_role={expected_role})"
),
}
if not apply:
return {
"success": True,
"outcome": OUTCOME_PREVIEW,
"apply": False,
"role": role_norm,
"profile_name": profile_name,
"username": username,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": selection,
"expected_role_next": expected_role,
"reasons": [
"dry-run only (apply=false); no assignment/lease created — "
"call again with apply=true to reserve via control-plane DB"
],
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
"downstream_note": (
"#612 incident bridge remains downstream of #600; "
"allocator never assigns raw monitoring incidents"
),
}
# Atomic reserve via #613 substrate.
ttl = lease_ttl_seconds if lease_ttl_seconds is not None else None
try:
kwargs: dict[str, Any] = {
"session_id": session_id,
"role": role_norm,
"remote": remote,
"org": org,
"repo": repo,
"kind": selected.kind,
"number": selected.number,
"expected_head_sha": selected.head_sha,
"allowed_actions": allowed,
"forbidden_actions": forbidden,
"phase": "allocated",
}
if ttl is not None:
kwargs["lease_ttl_seconds"] = int(ttl)
result = db.assign_and_lease(**kwargs)
except (InvalidWorkKindError, LeaseRequiredError, ControlPlaneError) as exc:
return {
"success": False,
"outcome": OUTCOME_NO_SAFE,
"apply": True,
"role": role_norm,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": selection,
"expected_role_next": expected_role,
"reasons": [f"atomic assign+lease failed: {exc} (fail closed, #613)"],
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
if result.outcome == "wait":
return {
"success": True,
"outcome": OUTCOME_WAIT,
"apply": True,
"role": role_norm,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": selection,
"expected_role_next": expected_role,
"reasons": [result.reason or "foreign active lease"],
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": result.as_dict(),
"owner_session_id": result.owner_session_id,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
if result.outcome == "no_safe_work":
return {
"success": True,
"outcome": OUTCOME_NO_SAFE,
"apply": True,
"role": role_norm,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": selection,
"expected_role_next": expected_role,
"reasons": [result.reason or "no_safe_work"],
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": result.as_dict(),
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
# assigned
return {
"success": True,
"outcome": OUTCOME_ASSIGNED,
"apply": True,
"role": role_norm,
"profile_name": profile_name,
"username": username,
"session_id": session_id,
"remote": remote,
"org": org,
"repo": repo,
"selected": selection,
"expected_role_next": expected_role,
"reasons": [
selection["reason_selected"],
result.reason or "atomic assign+lease created",
],
"skipped": [s.as_dict() for s in skipped],
"terminal_pr": terminal_pr,
"assignment": result.as_dict(),
"lease_proof": {
"assignment_id": result.assignment_id,
"lease_id": result.lease_id,
"expires_at": result.expires_at,
"expected_head_sha": result.expected_head_sha,
"allowed_actions": list(result.allowed_actions),
"forbidden_actions": list(result.forbidden_actions),
"source": "control_plane_db.assign_and_lease",
},
"next_valid_command": _next_command(role_norm, selected),
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
"downstream_note": (
"#612 incident bridge remains downstream of #600; "
"allocator never assigns raw monitoring incidents"
),
}
def _next_command(role: str, c: WorkCandidate) -> str:
if role == ROLE_AUTHOR and c.kind == "issue":
return f"implement issue #{c.number} under a branches/ worktree; open PR when ready"
if role == ROLE_AUTHOR and c.kind == "pr":
return (
f"address REQUEST_CHANGES on PR #{c.number} at head "
f"{(c.head_sha or '')[:12]} and push fixes"
)
if role == ROLE_REVIEWER:
return (
f"review PR #{c.number} pinned at head {(c.head_sha or '')[:12]} "
"via full reviewer workflow"
)
if role == ROLE_MERGER:
return (
f"merge PR #{c.number} only with explicit operator MERGE "
f"authorization at head {(c.head_sha or '')[:12]}"
)
if role == ROLE_RECONCILER:
return f"diagnose contested state for {c.kind}#{c.number}"
return f"proceed on {c.kind}#{c.number} under role {role}"
def candidate_from_dict(data: dict[str, Any]) -> WorkCandidate:
"""Build a WorkCandidate from a plain dict (tests / MCP inventory)."""
return WorkCandidate(
kind=str(data.get("kind") or "issue"),
number=int(data["number"]),
state=str(data.get("state") or "open"),
labels=tuple(data.get("labels") or ()),
title=str(data.get("title") or ""),
priority=int(data.get("priority") or 0),
head_sha=data.get("head_sha"),
request_changes_current_head=bool(data.get("request_changes_current_head")),
approval_on_current_head=bool(data.get("approval_on_current_head")),
approval_stale=bool(data.get("approval_stale")),
approval_contaminated=bool(data.get("approval_contaminated")),
mergeable=bool(data.get("mergeable")),
blocked=bool(data.get("blocked")),
dependency_unmet=bool(data.get("dependency_unmet")),
dependency_reason=data.get("dependency_reason"),
already_claimed_elsewhere=bool(data.get("already_claimed_elsewhere")),
)
+1751
View File
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,105 @@
# Control-plane DB substrate (#613)
**Status:** Implemented (SQLite single-writer MVP)
**ADR:** [`mcp-allocator-control-plane-observability-adr.md`](mcp-allocator-control-plane-observability-adr.md)
**Module:** `control_plane_db.py`
## Architecture statement
> **DB coordinates, Gitea records, Sentry/GlitchTip observe; the bridge is the only path that turns observations into Gitea work.**
## What this ships
| Capability | Notes |
|------------|--------|
| Schema | `sessions`, `work_items`, `leases`, `assignments`, `terminal_locks`, `events`, `incident_links` |
| Atomic assign+lease | `ControlPlaneDB.assign_and_lease` — one `BEGIN IMMEDIATE` transaction |
| Mutation gate | `require_valid_assignment` — live lease + allowed action + non-terminal work + non-stale head |
| Heartbeat / release / expire | Lease lifecycle helpers |
| Terminal-lock index | Routing signal for #600 (terminal path first) |
| `incident_links` | Provider-neutral link model for #612**not** assignable work; scope keys NULL-safe |
## Hard rules (enforced in code)
1. Assignable `work_items.kind` ∈ {`issue`, `pr`} only — **never** raw Sentry/GlitchTip incidents.
2. Two concurrent sessions cannot both receive an active assignment on the same open work item (second gets `wait`).
3. Merged/closed work items return `no_safe_work` at assign time, and `require_valid_assignment` fails closed if the work item becomes terminal later.
4. Assignments pin `expected_head_sha`; mutations fail closed if the work item head drifts.
5. SQLite path is the **single-writer MVP** (`GITEA_CONTROL_PLANE_DB`, default under `~/.cache/gitea-tools/control-plane/`). Multi-session multi-host production requires **Postgres** or a **single allocator daemon** (ADR §6).
6. `incident_links` optional scope fields are stored as empty strings (never NULL) so UNIQUE is canonical across minimal upserts.
7. Legacy `incident_links` migration collapses NULL-scope duplicates **only** when Gitea targets **and** all meaningful observation metadata agree (fingerprint, status, event_count, permalink, timestamps, linked PRs, etc.). Conflicting metadata fails closed — no silent discard.
## Dependency chain
```text
#613 control-plane DB (this) → #600 allocator API → #612 incident bridge
```
- **#600** must call this substrate (not file locks / comment-only leases alone) for completion.
- **#612** must write `incident_links` here and create **Gitea issues**; the allocator assigns those issues, not raw incidents.
## Allocator API (#600)
Module: `allocator_service.py` · MCP tool: `gitea_allocate_next_work`
- Workers call `gitea_allocate_next_work(apply=false|true)` instead of self-selecting work.
- `apply=false` returns a dry-run selection (`outcome=preview`) with skip reasons.
- `apply=true` reserves via `ControlPlaneDB.assign_and_lease` (atomic assignment+lease).
- Coordination source is **always** the control-plane DB — not file locks or comment-only leases.
- Routing follows ADR §5.3 (REQUEST_CHANGES → author, terminal path first, foreign lease → wait).
- Raw monitoring incidents are never candidates.
## Lease lifecycle API (#601)
Module: `lease_lifecycle.py` · MCP tools: `gitea_*_workflow_lease(s)`
Active control-plane leases are **first-class workflow state**:
| Tool | Purpose |
|------|---------|
| `gitea_list_workflow_leases` | List active (or all) leases for a repo |
| `gitea_inspect_workflow_lease` | Freshness + `safe_next_action` for one lease id |
| `gitea_adopt_workflow_lease` | Owner-resume or sanctioned reclaim with provenance |
| `gitea_release_workflow_lease` | Explicit owner release (audited) |
| `gitea_expire_workflow_leases` | Deterministic expire of past-`expires_at` leases |
| `gitea_abandon_workflow_lease` | Abandon with required proof |
| `gitea_reclaim_expired_workflow_lease` | Expire + re-assign with provenance |
### Hard rules
1. Control-plane DB is the coordination authority — file locks / comment-only leases are not authoritative alone.
2. Active foreign leases cannot be stolen; inspect returns `wait_foreign_active`.
3. Abandon requires `(dead_process OR missing_worktree)` and `no_live_mutation_risk`; foreign abandon also needs `operator_authorized` or full dead+missing+no_open_pr proof.
4. Adopt provenance always records `adopted_from_session_id`, `adopted_by_session_id`, work identity, optional head SHA, and worktree path.
5. Reviewer→merger comment handoff (`gitea_adopt_merger_pr_lease`) is unchanged and remains the Gitea-thread durability path.
```bash
python3 -m pytest tests/test_lease_lifecycle.py tests/test_control_plane_db.py tests/test_allocator_service.py tests/test_merger_lease_adoption.py -q
```
## Incident bridge (#612)
Module: `incident_bridge.py` · MCP tools: `gitea_observability_*`
- Bridge converts Sentry/GlitchTip observations → **normal Gitea issues** + `incident_links`.
- Phase-1: `gitea_observability_reconcile_incident(apply=false|true)` with JSON observation.
- Dry-run performs **no** Gitea mutation and **no** DB write.
- Apply reuses existing links or creates one Gitea issue; never invents `work_items.kind=incident`.
- Config: `GITEA_OBSERVABILITY_PROJECTS_JSON` or `GITEA_OBSERVABILITY_PROJECTS_FILE`.
- Provider tokens never appear in issue bodies, links, or tool results.
- Allocator sees bridge work only after a Gitea issue exists.
## Non-goals (intentionally deferred)
- Full unsupervised watchdog auto-filing (prefer explicit reconcile first)
- Assuming GlitchTip writeback API equals Sentry without verification
- Gitea comment/label mirror writers for every assignment (optional later)
## Tests
```bash
python3 -m pytest tests/test_control_plane_db.py -q
```
+823
View File
@@ -21,7 +21,10 @@ import json
import functools import functools
import contextlib import contextlib
import subprocess import subprocess
import uuid
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from typing import Any
# Mutation-authority record (#199, refs #194). Deliberately in-process, NOT a # Mutation-authority record (#199, refs #194). Deliberately in-process, NOT a
@@ -832,6 +835,10 @@ import review_workflow_boundary # noqa: E402
import review_workflow_load # noqa: E402 import review_workflow_load # noqa: E402
import mcp_session_state # noqa: E402 import mcp_session_state # noqa: E402
import stale_review_decision_lock # noqa: E402 import stale_review_decision_lock # noqa: E402
import allocator_service # noqa: E402
import control_plane_db # noqa: E402
import lease_lifecycle # noqa: E402
import incident_bridge # noqa: E402
import agent_temp_artifacts import agent_temp_artifacts
import issue_lock_worktree # noqa: E402 import issue_lock_worktree # noqa: E402
import issue_lock_provenance # noqa: E402 import issue_lock_provenance # noqa: E402
@@ -11109,6 +11116,822 @@ def gitea_capability_stop_terminal_report() -> dict:
}) })
def _control_plane_db_or_error() -> tuple[Any | None, list[str]]:
"""Open the #613 control-plane DB substrate; fail closed on errors."""
try:
db = control_plane_db.ControlPlaneDB()
return db, []
except Exception as exc: # noqa: BLE001
return None, [
f"control-plane DB substrate unavailable: {_redact(str(exc))} "
"(fail closed, #613/#600)"
]
def _allocator_candidates_from_gitea(
*,
remote: str,
host: str | None,
org: str,
repo: str,
include_issues: bool = True,
include_prs: bool = True,
limit: int = 50,
) -> tuple[list[Any], list[str]]:
"""Build allocator candidates from live Gitea open issues/PRs."""
reasons: list[str] = []
candidates: list[Any] = []
try:
h, o, r = _resolve(remote, host, org, repo)
auth = _auth(h)
except Exception as exc: # noqa: BLE001
return [], [f"failed to resolve Gitea target: {_redact(str(exc))}"]
if include_prs:
try:
prs = api_get_all(
f"{repo_api_url(h, o, r)}/pulls?state=open", auth
) or []
except Exception as exc: # noqa: BLE001
reasons.append(f"failed to list open PRs: {_redact(str(exc))}")
prs = []
for pr in prs[: max(1, int(limit))]:
if not isinstance(pr, dict):
continue
number = pr.get("number")
if number is None:
continue
head = pr.get("head") if isinstance(pr.get("head"), dict) else {}
head_sha = head.get("sha") or pr.get("head_sha")
labels = []
for lab in pr.get("labels") or []:
if isinstance(lab, dict) and lab.get("name"):
labels.append(str(lab["name"]))
elif isinstance(lab, str):
labels.append(lab)
# Lightweight review signals (best-effort; fail soft into reviewer path).
rc_current = False
approval_current = False
approval_stale = False
try:
feedback = gitea_get_pr_review_feedback(
int(number), remote=remote, org=o, repo=r
)
if feedback.get("success"):
rc_current = bool(
feedback.get("has_blocking_change_requests")
and not feedback.get("review_feedback_stale")
)
# Stale RC means author pushed; reviewer still next.
if feedback.get("has_blocking_change_requests") and feedback.get(
"review_feedback_stale"
):
approval_stale = False
except Exception:
pass
mergeable = bool(pr.get("mergeable"))
try:
candidates.append(
allocator_service.WorkCandidate(
kind="pr",
number=int(number),
state="open",
labels=tuple(labels),
title=str(pr.get("title") or ""),
priority=10 if rc_current else 5,
head_sha=head_sha,
request_changes_current_head=rc_current,
approval_on_current_head=approval_current,
approval_stale=approval_stale,
mergeable=mergeable,
)
)
except Exception as exc: # noqa: BLE001
reasons.append(
f"skipped invalid PR candidate #{number}: {_redact(str(exc))}"
)
if include_issues:
try:
issues = api_get_all(
f"{repo_api_url(h, o, r)}/issues?state=open&type=issues", auth
) or []
except Exception as exc: # noqa: BLE001
# Fallback without type filter
try:
issues = api_get_all(
f"{repo_api_url(h, o, r)}/issues?state=open", auth
) or []
except Exception as exc2: # noqa: BLE001
reasons.append(
f"failed to list open issues: {_redact(str(exc2))}"
)
issues = []
for issue in issues[: max(1, int(limit))]:
if not isinstance(issue, dict):
continue
# Pull requests also appear in /issues on Gitea — skip them.
if issue.get("pull_request") is not None:
continue
number = issue.get("number")
if number is None:
continue
labels = []
for lab in issue.get("labels") or []:
if isinstance(lab, dict) and lab.get("name"):
labels.append(str(lab["name"]).lower())
elif isinstance(lab, str):
labels.append(lab.lower())
body = str(issue.get("body") or "")
title = str(issue.get("title") or "")
blocked = "status:blocked" in labels
# Explicit downstream dependency: #612 waits on #600 allocator.
dep_unmet = False
dep_reason = None
# Generic body markers for blocked-on unfinished deps.
# (Hard-coded #612→#600 block removed after #600 merged — #612.)
lower_body = body.lower()
if "blocked on #" in lower_body or "downstream of #" in lower_body:
# Only treat as unmet when body still marks a live open dependency
# pattern; callers may clear labels when deps complete.
dep_unmet = "blocked on #" in lower_body
dep_reason = (
f"issue#{number} body marks an unmet dependency"
if dep_unmet
else None
)
try:
candidates.append(
allocator_service.WorkCandidate(
kind="issue",
number=int(number),
state="open",
labels=tuple(labels),
title=title,
priority=20 if "status:ready" in labels else 1,
blocked=blocked,
dependency_unmet=dep_unmet,
dependency_reason=dep_reason,
)
)
except Exception as exc: # noqa: BLE001
reasons.append(
f"skipped invalid issue candidate #{number}: {_redact(str(exc))}"
)
return candidates, reasons
@mcp.tool()
def gitea_observability_list_projects(
config_path: str | None = None,
mappings_json: str | None = None,
) -> dict:
"""List configured Sentry/GlitchTip → Gitea project mappings (#612).
Does not load provider tokens. Mappings come from
``GITEA_OBSERVABILITY_PROJECTS_JSON``,
``GITEA_OBSERVABILITY_PROJECTS_FILE``, or explicit arguments.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"projects": [],
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
try:
projects = incident_bridge.load_project_mappings(
config_path=config_path, mappings_json=mappings_json
)
except Exception as exc: # noqa: BLE001
return {
"success": False,
"projects": [],
"reasons": [incident_bridge.redact_text(exc)],
}
return {
"success": True,
"projects": [p.as_dict() for p in projects],
"count": len(projects),
"reasons": [] if projects else ["no project mappings configured"],
"note": (
"Provider API tokens are never returned. "
"Raw incidents are not assignable work (#612)."
),
}
@mcp.tool()
def gitea_observability_reconcile_incident(
observation_json: str,
apply: bool = False,
mappings_json: str | None = None,
config_path: str | None = None,
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
force_gitea_issue_number: int | None = None,
worktree_path: str | None = None,
) -> dict:
"""Reconcile one Sentry/GlitchTip observation into Gitea + incident_links (#612).
Phase-1: pass a sanitized observation as JSON. Dry-run is default
(``apply=false``) no Gitea mutation, no DB write.
When ``apply=true``:
* reuses existing ``incident_links`` row when present
* otherwise creates a **normal Gitea issue** (never a raw work_item incident)
* upserts the canonical ``incident_links`` row on the #613 control-plane DB
Never assigns raw provider incidents to the #600 allocator.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"apply": bool(apply),
"outcome": incident_bridge.OUTCOME_BLOCKED,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
"raw_incident_assignable": False,
}
if apply:
create_block = _profile_permission_block(
task_capability_map.required_permission("create_issue"),
number=None,
)
if create_block:
return {
"success": False,
"apply": True,
"outcome": incident_bridge.OUTCOME_BLOCKED,
"reasons": create_block.get("reasons")
if isinstance(create_block, dict)
else [str(create_block)],
"raw_incident_assignable": False,
}
try:
observation = json.loads(observation_json)
except Exception as exc: # noqa: BLE001
return {
"success": False,
"apply": bool(apply),
"outcome": incident_bridge.OUTCOME_BLOCKED,
"reasons": [
f"invalid observation_json: {incident_bridge.redact_text(exc)} "
"(fail closed)"
],
"raw_incident_assignable": False,
}
try:
mappings = incident_bridge.load_project_mappings(
config_path=config_path, mappings_json=mappings_json
)
except Exception as exc: # noqa: BLE001
return {
"success": False,
"apply": bool(apply),
"outcome": incident_bridge.OUTCOME_BLOCKED,
"reasons": [incident_bridge.redact_text(exc)],
"raw_incident_assignable": False,
}
# Allow org/repo overrides on the observation when not mapped.
if isinstance(observation, dict):
try:
_, o_def, r_def = _resolve(remote, host, org, repo)
except ValueError:
o_def, r_def = org, repo
observation.setdefault("gitea_org", o_def)
observation.setdefault("gitea_repo", r_def)
try:
db = control_plane_db.ControlPlaneDB()
except Exception as exc: # noqa: BLE001
return {
"success": False,
"apply": bool(apply),
"outcome": incident_bridge.OUTCOME_BLOCKED,
"reasons": [
f"control-plane DB unavailable: {incident_bridge.redact_text(exc)} "
"(fail closed, #613)"
],
"raw_incident_assignable": False,
}
create_fn = None
if apply and force_gitea_issue_number is None:
def create_fn(title, body, labels, g_org, g_repo):
return gitea_create_issue(
title=title,
body=body,
remote=remote,
host=host,
org=g_org,
repo=g_repo,
labels=list(labels),
issue_type="bug",
initial_status="ready",
require_workflow_labels=False,
worktree_path=worktree_path,
)
result = incident_bridge.reconcile_incident(
db,
observation=observation if isinstance(observation, dict) else {},
mappings=mappings,
apply=bool(apply),
create_issue_fn=create_fn,
force_gitea_issue_number=force_gitea_issue_number,
)
result["remote"] = remote
return result
@mcp.tool()
def gitea_observability_link_issue(
observation_json: str,
gitea_issue_number: int,
apply: bool = False,
mappings_json: str | None = None,
config_path: str | None = None,
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
) -> dict:
"""Explicitly link a provider observation to an existing Gitea issue (#612).
Dry-run by default. ``apply=true`` upserts ``incident_links`` only does
not create a new issue. Fails closed on mapping/fingerprint conflicts.
"""
return gitea_observability_reconcile_incident(
observation_json=observation_json,
apply=apply,
mappings_json=mappings_json,
config_path=config_path,
remote=remote,
host=host,
org=org,
repo=repo,
force_gitea_issue_number=int(gitea_issue_number),
worktree_path=None,
)
@mcp.tool()
def gitea_allocate_next_work(
apply: bool = False,
role: str | None = None,
session_id: str | None = None,
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
include_issues: bool = True,
include_prs: bool = True,
candidates_json: str | None = None,
limit: int = 50,
) -> dict:
"""Controller-owned next-work allocator using the #613 control-plane DB (#600).
Workers must not self-select exclusive work under the standard multi-LLM
workflow. Call this tool instead.
*apply=false* (default): dry-run selection only no assignment/lease.
*apply=true*: atomically assign + lease the selected Gitea issue/PR via
``ControlPlaneDB.assign_and_lease`` (never file locks or comment-only
leases as the coordination source).
Outcomes include: ``assigned_work``, ``preview``, ``wait``,
``blocked_by_terminal_path``, ``no_safe_work``, ``role_ineligible``.
*candidates_json* may inject a JSON list of candidate dicts (tests /
controller overrides). When omitted, open issues/PRs are loaded from Gitea.
#612 remains downstream: raw monitoring incidents are never candidates.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"outcome": allocator_service.OUTCOME_NO_SAFE,
"apply": bool(apply),
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
try:
h, o, r = _resolve(remote, host, org, repo)
except ValueError as exc:
return {
"success": False,
"outcome": allocator_service.OUTCOME_NO_SAFE,
"reasons": [str(exc)],
"assignment": None,
"substrate": "control_plane_db",
}
profile = get_profile()
profile_name = (profile.get("profile_name") or "").strip() or None
active_role = _profile_role_kind(profile)
role_in = (role or active_role or "").strip() or "author"
try:
username = _authenticated_username(h)
except Exception:
username = None
db, db_errs = _control_plane_db_or_error()
if db is None:
return {
"success": False,
"outcome": allocator_service.OUTCOME_NO_SAFE,
"apply": bool(apply),
"reasons": db_errs,
"assignment": None,
"substrate": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
inv_reasons: list[str] = []
candidates: list[Any] = []
if candidates_json:
try:
raw = json.loads(candidates_json)
if not isinstance(raw, list):
raise ValueError("candidates_json must be a JSON list")
for item in raw:
if not isinstance(item, dict):
continue
candidates.append(allocator_service.candidate_from_dict(item))
except Exception as exc: # noqa: BLE001
return {
"success": False,
"outcome": allocator_service.OUTCOME_NO_SAFE,
"apply": bool(apply),
"reasons": [
f"invalid candidates_json: {_redact(str(exc))} (fail closed)"
],
"assignment": None,
"substrate": "control_plane_db",
}
else:
candidates, inv_reasons = _allocator_candidates_from_gitea(
remote=remote,
host=host,
org=o,
repo=r,
include_issues=include_issues,
include_prs=include_prs,
limit=limit,
)
sid = (session_id or "").strip() or (
f"{profile_name or 'session'}-{os.getpid()}-{uuid.uuid4().hex[:8]}"
)
result = allocator_service.allocate_next_work(
db,
session_id=sid,
role=role_in,
remote=remote if remote in REMOTES else remote,
org=o,
repo=r,
candidates=candidates,
apply=bool(apply),
profile_name=profile_name,
username=username,
)
if inv_reasons:
result.setdefault("inventory_warnings", inv_reasons)
result["candidate_count"] = len(candidates)
result["inventory_source"] = (
"candidates_json" if candidates_json else "gitea_live"
)
return result
# ── Control-plane lease lifecycle (#601) ──────────────────────────────────────
@mcp.tool()
def gitea_list_workflow_leases(
remote: str = "dadeschools",
host: str | None = None,
org: str | None = None,
repo: str | None = None,
role: str | None = None,
include_non_active: bool = False,
limit: int = 100,
) -> dict:
"""List control-plane leases as first-class workflow state (#601).
Read-only. Returns active (default) or all leases from the #613 DB substrate.
File locks and comment-only leases are not listed as authoritative here.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"leases": [],
"permission_report": _permission_block_report("gitea.read"),
}
try:
h, o, r = _resolve(remote, host, org, repo)
except ValueError as exc:
return {"success": False, "reasons": [str(exc)], "leases": []}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs, "leases": []}
result = lease_lifecycle.list_active_leases(
db,
remote=remote if remote in REMOTES else remote,
org=o,
repo=r,
role=role,
include_non_active=bool(include_non_active),
limit=int(limit),
)
result["remote"] = remote
result["org"] = o
result["repo"] = r
return result
@mcp.tool()
def gitea_inspect_workflow_lease(
lease_id: str,
session_id: str | None = None,
worktree_path: str | None = None,
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Inspect one control-plane lease with safe_next_action (#601).
Distinguishes owner-resume, wait-foreign, reclaim-expired, abandon-allowed,
and stale prompt ids. Control-plane DB is authoritative.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
profile = get_profile()
profile_name = (profile.get("profile_name") or "").strip() or "session"
sid = (session_id or "").strip() or f"{profile_name}-{os.getpid()}"
return lease_lifecycle.inspect_lease(
db,
lease_id,
caller_session_id=sid,
caller_worktree=worktree_path,
)
@mcp.tool()
def gitea_adopt_workflow_lease(
lease_id: str,
session_id: str | None = None,
role: str | None = None,
worktree_path: str | None = None,
expected_head_sha: str | None = None,
operator_authorized: bool = False,
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Adopt a control-plane lease through the sanctioned path (#601).
Same-owner resume refreshes provenance. Foreign active leases are refused.
Expired leases may be reclaimed; provenance records adopted_from/by.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
profile = get_profile()
profile_name = (profile.get("profile_name") or "").strip() or "session"
active_role = _profile_role_kind(profile) or "author"
sid = (session_id or "").strip() or (
f"{profile_name}-{os.getpid()}-{uuid.uuid4().hex[:8]}"
)
try:
return lease_lifecycle.adopt_lease(
db,
lease_id=lease_id,
adopter_session_id=sid,
role=(role or active_role).strip() or "author",
worktree_path=worktree_path,
expected_head_sha=expected_head_sha,
owner_pid=os.getpid(),
operator_authorized=bool(operator_authorized),
)
except (lease_lifecycle.LeaseLifecycleError, control_plane_db.ControlPlaneError) as exc:
return {
"success": False,
"outcome": "blocked",
"reasons": [_redact(str(exc))],
"lease_id": lease_id,
"authoritative_source": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
@mcp.tool()
def gitea_release_workflow_lease(
lease_id: str,
session_id: str,
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Explicitly release a control-plane lease owned by *session_id* (#601).
Recorded in the DB event log with release provenance. Foreign release fails closed.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
try:
return lease_lifecycle.release_lease(
db, lease_id=lease_id, session_id=session_id
)
except (lease_lifecycle.LeaseLifecycleError, control_plane_db.ControlPlaneError) as exc:
return {
"success": False,
"outcome": "blocked",
"reasons": [_redact(str(exc))],
"lease_id": lease_id,
"authoritative_source": "control_plane_db",
}
@mcp.tool()
def gitea_expire_workflow_leases(
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Expire stale control-plane leases whose expires_at has passed (#601).
Deterministic reclaim prerequisite. Does not steal active non-expired leases.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
return lease_lifecycle.expire_leases(db)
@mcp.tool()
def gitea_abandon_workflow_lease(
lease_id: str,
session_id: str | None = None,
dead_process: bool = False,
missing_worktree: bool = False,
no_open_pr: bool = False,
no_live_mutation_risk: bool = False,
operator_authorized: bool = False,
worktree_path: str | None = None,
owner_pid: int | None = None,
notes: str = "",
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Abandon a control-plane lease with required proof (#601).
Requires (dead_process or missing_worktree) and no_live_mutation_risk.
Foreign abandon also needs operator_authorized or
(dead_process and missing_worktree and no_open_pr).
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
profile = get_profile()
profile_name = (profile.get("profile_name") or "").strip() or "session"
sid = (session_id or "").strip() or f"{profile_name}-{os.getpid()}"
proof = lease_lifecycle.AbandonProof(
dead_process=bool(dead_process),
missing_worktree=bool(missing_worktree),
no_open_pr=bool(no_open_pr),
no_live_mutation_risk=bool(no_live_mutation_risk),
operator_authorized=bool(operator_authorized),
worktree_path=worktree_path,
owner_pid=owner_pid,
notes=notes or "",
)
try:
return lease_lifecycle.abandon_lease(
db,
lease_id=lease_id,
requester_session_id=sid,
proof=proof,
)
except (lease_lifecycle.LeaseLifecycleError, control_plane_db.ControlPlaneError) as exc:
return {
"success": False,
"outcome": "blocked",
"reasons": [_redact(str(exc))],
"lease_id": lease_id,
"authoritative_source": "control_plane_db",
"file_lock_only": False,
"comment_lease_only": False,
}
@mcp.tool()
def gitea_reclaim_expired_workflow_lease(
lease_id: str,
session_id: str | None = None,
role: str | None = None,
worktree_path: str | None = None,
expected_head_sha: str | None = None,
remote: str = "dadeschools",
host: str | None = None,
) -> dict:
"""Reclaim an expired control-plane lease for a new/owner session (#601).
Deterministic: expire markers applied, then atomic assign+lease with provenance.
Active foreign leases are refused.
"""
read_block = _profile_operation_gate("gitea.read")
if read_block:
return {
"success": False,
"reasons": read_block,
"permission_report": _permission_block_report("gitea.read"),
}
db, errs = _control_plane_db_or_error()
if db is None:
return {"success": False, "reasons": errs}
profile = get_profile()
profile_name = (profile.get("profile_name") or "").strip() or "session"
active_role = _profile_role_kind(profile) or "author"
sid = (session_id or "").strip() or (
f"{profile_name}-{os.getpid()}-{uuid.uuid4().hex[:8]}"
)
try:
return lease_lifecycle.reclaim_expired_lease(
db,
lease_id=lease_id,
session_id=sid,
role=(role or active_role).strip() or "author",
worktree_path=worktree_path,
expected_head_sha=expected_head_sha,
)
except (lease_lifecycle.LeaseLifecycleError, control_plane_db.ControlPlaneError) as exc:
return {
"success": False,
"outcome": "blocked",
"reasons": [_redact(str(exc))],
"lease_id": lease_id,
"authoritative_source": "control_plane_db",
}
# ── Entry point ─────────────────────────────────────────────────────────────── # ── Entry point ───────────────────────────────────────────────────────────────
if __name__ == "__main__": if __name__ == "__main__":
+788
View File
@@ -0,0 +1,788 @@
"""Observability incident bridge (#612).
Converts Sentry/GlitchTip **observations** into durable **Gitea issues** and
``incident_links`` rows on the #613 control-plane DB.
Hard rules (ADR):
* Gitea owns work; providers own incidents; the bridge only links them.
* Raw monitoring incidents are **never** assignable ``work_items``.
* The #600 allocator sees bridge work only as normal Gitea issues.
* Phase-1 prefers dry-run / explicit reconcile; apply creates/links issues.
* No tokens, DSNs, cookies, or other secrets in bodies, DB, or logs.
"""
from __future__ import annotations
import json
import os
import re
from dataclasses import dataclass, field
from typing import Any, Callable, Sequence
from control_plane_db import ControlPlaneDB, ControlPlaneError, WORK_KINDS, _norm_scope
PROVIDERS = frozenset({"sentry", "glitchtip"})
OUTCOME_PREVIEW = "preview"
OUTCOME_LINKED = "linked_existing"
OUTCOME_CREATED = "created_issue"
OUTCOME_UPDATED = "updated_link"
OUTCOME_BLOCKED = "blocked"
OUTCOME_NO_ACTION = "no_safe_action"
# Patterns scrubbed from any bridge-facing text (bodies, titles, tags, logs).
_SECRET_PATTERNS: tuple[re.Pattern[str], ...] = (
re.compile(
r"(?i)\b(api[_-]?token|token|secret|password|passwd)\s*[:=]\s*\S+"
),
re.compile(
r"(?i)\bAuthorization\s*[:=]\s*(?:Bearer\s+)?[A-Za-z0-9._\-+=/]{8,}"
),
re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._\-]{8,}"),
re.compile(r"(?i)\bBasic\s+[A-Za-z0-9+/=]{8,}"),
re.compile(r"(?i)\b(dsn|sentry_dsn|glitchtip_dsn)\s*[:=]\s*\S+"),
re.compile(r"https?://[^/\s]+:[^@/\s]+@"), # user:pass@host
re.compile(r"(?i)\b(keychain[_-]?id|private[_-]?key)\s*[:=]\s*\S+"),
re.compile(r"(?i)cookie\s*[:=]\s*[^\s;]+"),
re.compile(r"(?i)\bsession[_-]?id\s*[:=]\s*\S+"),
)
_SENSITIVE_TAG_KEYS = frozenset(
{
"authorization",
"cookie",
"set-cookie",
"password",
"passwd",
"secret",
"token",
"api_key",
"apikey",
"dsn",
"sentry_dsn",
"access_token",
"refresh_token",
}
)
DEFAULT_LABELS = (
"type:bug",
"observability",
"status:ready",
)
@dataclass(frozen=True)
class ProjectMapping:
"""Maps a monitoring project to a Gitea repository."""
name: str
provider: str
monitor_base_url: str
monitor_org: str
monitor_project: str
gitea_org: str
gitea_repo: str
default_labels: tuple[str, ...] = DEFAULT_LABELS
environment_filters: tuple[str, ...] = ()
severity_threshold: str | None = None
def __post_init__(self) -> None:
prov = (self.provider or "").strip().lower()
if prov not in PROVIDERS:
raise ControlPlaneError(
f"unknown provider '{self.provider}'; expected one of {sorted(PROVIDERS)}"
)
object.__setattr__(self, "provider", prov)
object.__setattr__(self, "monitor_base_url", (self.monitor_base_url or "").rstrip("/"))
object.__setattr__(self, "monitor_org", (self.monitor_org or "").strip())
object.__setattr__(self, "monitor_project", (self.monitor_project or "").strip())
object.__setattr__(self, "gitea_org", (self.gitea_org or "").strip())
object.__setattr__(self, "gitea_repo", (self.gitea_repo or "").strip())
object.__setattr__(
self,
"default_labels",
tuple(str(x).strip() for x in (self.default_labels or DEFAULT_LABELS) if str(x).strip()),
)
def as_dict(self) -> dict[str, Any]:
return {
"name": self.name,
"provider": self.provider,
"monitor_base_url": self.monitor_base_url,
"monitor_org": self.monitor_org,
"monitor_project": self.monitor_project,
"gitea_org": self.gitea_org,
"gitea_repo": self.gitea_repo,
"default_labels": list(self.default_labels),
"environment_filters": list(self.environment_filters),
"severity_threshold": self.severity_threshold,
}
def matches_observation(self, obs: dict[str, Any]) -> bool:
if (obs.get("provider") or "").strip().lower() != self.provider:
return False
base = (obs.get("provider_base_url") or obs.get("monitor_base_url") or "").rstrip("/")
if base and self.monitor_base_url and base != self.monitor_base_url:
return False
org = (obs.get("provider_org") or obs.get("monitor_org") or "").strip()
if org and self.monitor_org and org != self.monitor_org:
return False
project = (obs.get("provider_project") or obs.get("monitor_project") or "").strip()
if project and self.monitor_project and project != self.monitor_project:
return False
return True
@dataclass
class NormalizedIncident:
provider: str
provider_base_url: str
provider_org: str
provider_project: str
provider_issue_id: str
provider_short_id: str | None = None
provider_permalink: str | None = None
fingerprint: str | None = None
first_seen: str | None = None
last_seen: str | None = None
event_count: int | None = None
status: str = "open"
environment: str | None = None
severity: str | None = None
culprit: str | None = None
title: str = ""
summary: str = ""
tags: dict[str, str] = field(default_factory=dict)
gitea_org: str = ""
gitea_repo: str = ""
default_labels: tuple[str, ...] = DEFAULT_LABELS
def provider_key(self) -> tuple[str, str, str, str, str]:
return (
self.provider,
_norm_scope(self.provider_base_url),
_norm_scope(self.provider_org),
_norm_scope(self.provider_project),
str(self.provider_issue_id),
)
def as_dict(self) -> dict[str, Any]:
return {
"provider": self.provider,
"provider_base_url": self.provider_base_url,
"provider_org": self.provider_org,
"provider_project": self.provider_project,
"provider_issue_id": self.provider_issue_id,
"provider_short_id": self.provider_short_id,
"provider_permalink": self.provider_permalink,
"fingerprint": self.fingerprint,
"first_seen": self.first_seen,
"last_seen": self.last_seen,
"event_count": self.event_count,
"status": self.status,
"environment": self.environment,
"severity": self.severity,
"culprit": self.culprit,
"title": self.title,
"summary": self.summary,
"tags": dict(self.tags),
"gitea_org": self.gitea_org,
"gitea_repo": self.gitea_repo,
"default_labels": list(self.default_labels),
}
def redact_text(value: Any) -> str:
"""Redact secret-like material from a string (fail closed to empty)."""
if value is None:
return ""
text = str(value)
for pat in _SECRET_PATTERNS:
text = pat.sub("[REDACTED]", text)
return text
def sanitize_tags(tags: Any) -> dict[str, str]:
if not isinstance(tags, dict):
return {}
out: dict[str, str] = {}
for k, v in tags.items():
key = str(k).strip().lower()
if not key or key in _SENSITIVE_TAG_KEYS:
continue
if any(s in key for s in ("token", "secret", "password", "cookie", "auth", "dsn")):
continue
val = redact_text(v)
if "[REDACTED]" in val:
continue
if len(val) > 200:
val = val[:200] + ""
out[key] = val
return out
def project_mapping_from_dict(data: dict[str, Any]) -> ProjectMapping:
labels = data.get("default_labels") or DEFAULT_LABELS
envs = data.get("environment_filters") or ()
return ProjectMapping(
name=str(data.get("name") or data.get("monitor_project") or "unnamed"),
provider=str(data.get("provider") or ""),
monitor_base_url=str(data.get("monitor_base_url") or data.get("provider_base_url") or ""),
monitor_org=str(data.get("monitor_org") or data.get("provider_org") or ""),
monitor_project=str(data.get("monitor_project") or data.get("provider_project") or ""),
gitea_org=str(data.get("gitea_org") or ""),
gitea_repo=str(data.get("gitea_repo") or ""),
default_labels=tuple(labels),
environment_filters=tuple(envs),
severity_threshold=data.get("severity_threshold"),
)
def load_project_mappings(
*,
config_path: str | None = None,
mappings_json: str | None = None,
) -> list[ProjectMapping]:
"""Load project mappings from JSON env, file, or explicit JSON string.
Env: ``GITEA_OBSERVABILITY_PROJECTS_JSON`` or path
``GITEA_OBSERVABILITY_PROJECTS_FILE``.
"""
raw: Any = None
if mappings_json:
raw = json.loads(mappings_json)
else:
env_json = (os.environ.get("GITEA_OBSERVABILITY_PROJECTS_JSON") or "").strip()
path = (
(config_path or "").strip()
or (os.environ.get("GITEA_OBSERVABILITY_PROJECTS_FILE") or "").strip()
)
if env_json:
raw = json.loads(env_json)
elif path and os.path.isfile(path):
with open(path, encoding="utf-8") as fh:
raw = json.load(fh)
if raw is None:
return []
if isinstance(raw, dict) and "projects" in raw:
raw = raw["projects"]
if not isinstance(raw, list):
raise ControlPlaneError("observability project mappings must be a list")
return [project_mapping_from_dict(item) for item in raw if isinstance(item, dict)]
def resolve_mapping(
observation: dict[str, Any],
mappings: Sequence[ProjectMapping],
*,
explicit: ProjectMapping | None = None,
) -> ProjectMapping | None:
if explicit is not None:
return explicit
matches = [m for m in mappings if m.matches_observation(observation)]
if len(matches) == 1:
return matches[0]
if len(matches) > 1:
raise ControlPlaneError(
"ambiguous project mapping for observation: "
+ ", ".join(m.name for m in matches)
+ " (fail closed, #612)"
)
# Fallback: observation itself may carry gitea targets
g_org = (observation.get("gitea_org") or "").strip()
g_repo = (observation.get("gitea_repo") or "").strip()
provider = (observation.get("provider") or "").strip().lower()
if g_org and g_repo and provider in PROVIDERS:
return ProjectMapping(
name=f"inline-{provider}",
provider=provider,
monitor_base_url=str(
observation.get("provider_base_url")
or observation.get("monitor_base_url")
or ""
),
monitor_org=str(
observation.get("provider_org") or observation.get("monitor_org") or ""
),
monitor_project=str(
observation.get("provider_project")
or observation.get("monitor_project")
or ""
),
gitea_org=g_org,
gitea_repo=g_repo,
default_labels=tuple(observation.get("default_labels") or DEFAULT_LABELS),
)
return None
def normalize_incident(
observation: dict[str, Any],
mapping: ProjectMapping,
) -> NormalizedIncident:
"""Normalize + sanitize a raw provider observation (fail closed)."""
if not isinstance(observation, dict):
raise ControlPlaneError("observation must be a dict (fail closed, #612)")
provider = (observation.get("provider") or mapping.provider or "").strip().lower()
if provider not in PROVIDERS:
raise ControlPlaneError(
f"unknown provider '{provider}'; expected {sorted(PROVIDERS)} (fail closed)"
)
if provider != mapping.provider:
raise ControlPlaneError(
f"observation provider '{provider}' does not match mapping "
f"'{mapping.provider}' (fail closed, #612)"
)
issue_id = observation.get("provider_issue_id") or observation.get("id")
if issue_id is None or str(issue_id).strip() == "":
raise ControlPlaneError(
"observation missing provider_issue_id (fail closed, #612)"
)
base = (
observation.get("provider_base_url")
or observation.get("monitor_base_url")
or mapping.monitor_base_url
or ""
)
org = (
observation.get("provider_org")
or observation.get("monitor_org")
or mapping.monitor_org
or ""
)
project = (
observation.get("provider_project")
or observation.get("monitor_project")
or mapping.monitor_project
or ""
)
title = redact_text(
observation.get("title")
or observation.get("culprit")
or f"{provider} incident {issue_id}"
)
meta = observation.get("metadata")
meta_value = meta.get("value") if isinstance(meta, dict) else None
raw_sum = (
observation.get("summary")
or observation.get("message")
or meta_value
or title
)
summary = redact_text(raw_sum)
permalink = observation.get("provider_permalink") or observation.get("permalink")
if permalink:
permalink = redact_text(permalink)
if "[REDACTED]" in permalink:
permalink = None
tags = sanitize_tags(observation.get("tags") or {})
event_count = observation.get("event_count") or observation.get("count")
try:
event_count_i = int(event_count) if event_count is not None else None
except (TypeError, ValueError):
event_count_i = None
return NormalizedIncident(
provider=provider,
provider_base_url=str(base).rstrip("/"),
provider_org=str(org).strip(),
provider_project=str(project).strip(),
provider_issue_id=str(issue_id).strip(),
provider_short_id=redact_text(observation.get("provider_short_id") or observation.get("shortId") or "")
or None,
provider_permalink=permalink,
fingerprint=redact_text(observation.get("fingerprint") or "") or None,
first_seen=str(observation.get("first_seen") or observation.get("firstSeen") or "")
or None,
last_seen=str(observation.get("last_seen") or observation.get("lastSeen") or "")
or None,
event_count=event_count_i,
status=str(observation.get("status") or "open").strip().lower() or "open",
environment=redact_text(observation.get("environment") or "") or None,
severity=redact_text(observation.get("severity") or observation.get("level") or "")
or None,
culprit=redact_text(observation.get("culprit") or "") or None,
title=title[:200],
summary=summary[:2000],
tags=tags,
gitea_org=mapping.gitea_org,
gitea_repo=mapping.gitea_repo,
default_labels=mapping.default_labels,
)
def build_gitea_issue_title(inc: NormalizedIncident) -> str:
env = f" [{inc.environment}]" if inc.environment else ""
sev = f" ({inc.severity})" if inc.severity else ""
return redact_text(f"[obs:{inc.provider}]{env}{sev} {inc.title}")[:180]
def build_gitea_issue_body(inc: NormalizedIncident) -> str:
"""Sanitized durable issue body (no secrets)."""
lines = [
"## Observability incident (bridge #612)",
"",
"<!-- mcp-incident-bridge:v1 -->",
f"<!-- provider={inc.provider} issue_id={inc.provider_issue_id} -->",
"",
f"- **provider:** `{inc.provider}`",
f"- **provider_issue_id:** `{inc.provider_issue_id}`",
]
if inc.provider_short_id:
lines.append(f"- **provider_short_id:** `{inc.provider_short_id}`")
if inc.provider_permalink:
lines.append(f"- **provider_url:** {inc.provider_permalink}")
lines.extend(
[
f"- **provider_org/project:** `{inc.provider_org}` / `{inc.provider_project}`",
f"- **base_url:** `{inc.provider_base_url}`",
f"- **fingerprint:** `{inc.fingerprint or ''}`",
f"- **first_seen:** `{inc.first_seen or ''}`",
f"- **last_seen:** `{inc.last_seen or ''}`",
f"- **event_count:** `{inc.event_count if inc.event_count is not None else ''}`",
f"- **environment:** `{inc.environment or ''}`",
f"- **severity:** `{inc.severity or ''}`",
f"- **culprit:** `{inc.culprit or ''}`",
f"- **status:** `{inc.status}`",
"",
"### Summary",
"",
redact_text(inc.summary) or "(no summary)",
"",
"### Safe tags",
"",
]
)
if inc.tags:
for k, v in sorted(inc.tags.items()):
lines.append(f"- `{k}`: `{v}`")
else:
lines.append("- (none)")
lines.extend(
[
"",
"### Canonical next action",
"",
"Author: investigate and fix under normal Gitea workflow. "
"Allocator may select this issue as ordinary Gitea work "
"(never as a raw provider incident).",
"",
"### Bridge notes",
"",
"- Raw Sentry/GlitchTip incidents are **not** assignable work items.",
"- Gitea remains the durable work record; DB stores `incident_links` only.",
"- Provider tokens must never appear in this issue.",
]
)
return "\n".join(lines)
def _link_conflict(existing: dict[str, Any], inc: NormalizedIncident) -> str | None:
"""Fail closed if existing link targets a different Gitea issue/repo."""
eg_org = str(existing.get("gitea_org") or "")
eg_repo = str(existing.get("gitea_repo") or "")
eg_num = existing.get("gitea_issue_number")
if eg_org and eg_org != inc.gitea_org:
return (
f"existing link points to org '{eg_org}', mapping wants "
f"'{inc.gitea_org}' (fail closed, #612)"
)
if eg_repo and eg_repo != inc.gitea_repo:
return (
f"existing link points to repo '{eg_repo}', mapping wants "
f"'{inc.gitea_repo}' (fail closed, #612)"
)
# fingerprint conflict when both present and disagree
ef = (existing.get("fingerprint") or "").strip()
nf = (inc.fingerprint or "").strip()
if ef and nf and ef != nf:
# Same provider issue id with different fingerprints is ambiguous.
return (
f"fingerprint conflict for provider issue {inc.provider_issue_id}: "
f"link has '{ef}', observation has '{nf}' (fail closed, #612)"
)
return None
CreateIssueFn = Callable[[str, str, list[str], str, str], dict[str, Any]]
# create_issue_fn(title, body, labels, gitea_org, gitea_repo) -> {"number": int, ...}
def reconcile_incident(
db: ControlPlaneDB | None,
*,
observation: dict[str, Any],
mappings: Sequence[ProjectMapping] | None = None,
mapping: ProjectMapping | None = None,
apply: bool = False,
create_issue_fn: CreateIssueFn | None = None,
force_gitea_issue_number: int | None = None,
) -> dict[str, Any]:
"""Reconcile one observation into incident_links + optional Gitea issue.
*apply=False* (default): preview only — no DB mutation, no Gitea mutation.
*apply=True*: upsert link; create Gitea issue when none linked (requires
``create_issue_fn``) or use ``force_gitea_issue_number`` for explicit link.
Never creates control-plane ``work_items`` for raw incidents.
"""
base: dict[str, Any] = {
"success": False,
"apply": bool(apply),
"outcome": OUTCOME_NO_ACTION,
"performed": False,
"gitea_mutated": False,
"db_mutated": False,
"raw_incident_assignable": False,
"work_item_kind_would_be": None,
"reasons": [],
"skipped": None,
"incident": None,
"existing_link": None,
"gitea_issue": None,
"action": None,
"mapping": None,
"substrate": "control_plane_db.incident_links",
"durable_work_system": "gitea_issues",
}
if db is None:
base["reasons"] = [
"control-plane DB substrate unavailable (fail closed, #613/#612)"
]
base["outcome"] = OUTCOME_BLOCKED
return base
try:
maps = list(mappings or [])
resolved = resolve_mapping(observation, maps, explicit=mapping)
if resolved is None:
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [
"no project mapping for observation (fail closed, #612); "
"configure GITEA_OBSERVABILITY_PROJECTS_JSON or pass mapping"
]
return base
base["mapping"] = resolved.as_dict()
inc = normalize_incident(observation, resolved)
base["incident"] = inc.as_dict()
except (ControlPlaneError, json.JSONDecodeError, TypeError, ValueError) as exc:
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [str(exc)]
return base
# Environment filter (optional)
if resolved.environment_filters and inc.environment:
allowed = {e.lower() for e in resolved.environment_filters}
if inc.environment.lower() not in allowed:
base["outcome"] = OUTCOME_NO_ACTION
base["reasons"] = [
f"environment '{inc.environment}' not in filters "
f"{sorted(allowed)}; skip (policy)"
]
base["success"] = True
base["action"] = "skip_environment_filter"
return base
existing = db.get_incident_link_by_provider(
provider=inc.provider,
provider_issue_id=inc.provider_issue_id,
provider_base_url=inc.provider_base_url,
provider_org=inc.provider_org,
provider_project=inc.provider_project,
)
base["existing_link"] = existing
if existing:
conflict = _link_conflict(existing, inc)
if conflict:
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [conflict]
return base
title = build_gitea_issue_title(inc)
body = build_gitea_issue_body(inc)
labels = list(inc.default_labels)
if inc.provider and f"{inc.provider}" not in labels:
labels.append(inc.provider)
if "observability" not in labels:
labels.append("observability")
preview_issue = {
"title": title,
"body_preview": body[:500] + ("" if len(body) > 500 else ""),
"labels": labels,
"gitea_org": inc.gitea_org,
"gitea_repo": inc.gitea_repo,
"would_create": existing is None and force_gitea_issue_number is None,
"would_reuse_issue": int(existing["gitea_issue_number"])
if existing
else force_gitea_issue_number,
}
base["gitea_issue_preview"] = preview_issue
if not apply:
base["success"] = True
base["outcome"] = OUTCOME_PREVIEW
base["action"] = (
"preview_reuse_link" if existing else "preview_create_issue"
)
base["reasons"] = [
"dry-run only (apply=false); no Gitea mutation and no DB write — "
"call again with apply=true to create/link"
]
if existing:
base["gitea_issue"] = {
"number": existing.get("gitea_issue_number"),
"org": existing.get("gitea_org"),
"repo": existing.get("gitea_repo"),
}
# Prove we never treat this as work_item kind incident
base["raw_incident_assignable"] = False
base["work_item_kind_would_be"] = "issue" # only after Gitea issue exists
return base
# --- apply path ---
issue_number: int | None = None
created = False
if existing:
issue_number = int(existing["gitea_issue_number"])
action = "updated_existing_link"
outcome = OUTCOME_UPDATED
elif force_gitea_issue_number is not None:
issue_number = int(force_gitea_issue_number)
action = "link_explicit_issue"
outcome = OUTCOME_LINKED
else:
if create_issue_fn is None:
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [
"apply=true requires create_issue_fn when no existing link "
"(fail closed, #612)"
]
return base
try:
created_res = create_issue_fn(
title, body, labels, inc.gitea_org, inc.gitea_repo
)
except Exception as exc: # noqa: BLE001
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [
f"Gitea issue creation failed: {redact_text(exc)} (fail closed)"
]
return base
number = created_res.get("number") if isinstance(created_res, dict) else None
if number is None:
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [
"Gitea issue creation returned no issue number (fail closed, #612)"
]
base["create_result"] = created_res
return base
issue_number = int(number)
created = True
action = "created_gitea_issue"
outcome = OUTCOME_CREATED
base["gitea_mutated"] = True
base["create_result"] = {
k: created_res.get(k)
for k in ("number", "success", "performed", "reasons")
if isinstance(created_res, dict)
}
assert issue_number is not None
try:
link = db.upsert_incident_link(
provider=inc.provider,
provider_issue_id=inc.provider_issue_id,
gitea_org=inc.gitea_org,
gitea_repo=inc.gitea_repo,
gitea_issue_number=issue_number,
provider_base_url=inc.provider_base_url,
provider_org=inc.provider_org,
provider_project=inc.provider_project,
provider_short_id=inc.provider_short_id,
provider_permalink=inc.provider_permalink,
fingerprint=inc.fingerprint,
first_seen=inc.first_seen,
last_seen=inc.last_seen,
event_count=inc.event_count,
status=inc.status if not created else "open",
)
except Exception as exc: # noqa: BLE001
base["outcome"] = OUTCOME_BLOCKED
base["reasons"] = [
f"incident_links upsert failed: {redact_text(exc)} (fail closed, #613)"
]
base["gitea_issue"] = {
"number": issue_number,
"org": inc.gitea_org,
"repo": inc.gitea_repo,
"created": created,
}
return base
base["success"] = True
base["performed"] = True
base["db_mutated"] = True
base["outcome"] = outcome
base["action"] = action
base["gitea_issue"] = {
"number": issue_number,
"org": inc.gitea_org,
"repo": inc.gitea_repo,
"created": created,
}
base["link"] = {
k: link.get(k)
for k in (
"link_id",
"provider",
"provider_issue_id",
"gitea_org",
"gitea_repo",
"gitea_issue_number",
"fingerprint",
"event_count",
"status",
"last_sync_at",
)
}
base["reasons"] = [
(
f"reused Gitea issue #{issue_number} for provider "
f"{inc.provider}:{inc.provider_issue_id}"
if not created
else f"created Gitea issue #{issue_number} and stored incident_links row"
),
"raw provider incident is not an assignable work_item "
f"(WORK_KINDS={sorted(WORK_KINDS)})",
]
base["raw_incident_assignable"] = False
base["work_item_kind_would_be"] = "issue"
base["allocator_visible_as"] = {
"kind": "issue",
"number": issue_number,
"org": inc.gitea_org,
"repo": inc.gitea_repo,
"note": "allocator selects normal Gitea issues only (#600/#612)",
}
return base
def assert_not_raw_incident_work_item(kind: str) -> None:
"""Guard used by tests / callers: incidents must not enter WORK_KINDS."""
k = (kind or "").strip().lower()
if k not in WORK_KINDS:
if k in ("sentry_incident", "glitchtip_incident", "incident", "observation"):
raise ControlPlaneError(
f"kind '{k}' is not assignable; bridge must create Gitea issues first (#612)"
)
raise ControlPlaneError(f"kind '{k}' is not assignable")
+63
View File
@@ -375,6 +375,64 @@ def _same_realpath(left: str | None, right: str | None) -> bool:
return left == right return left == right
def assess_expired_lock_reclaim(
existing_lock: dict[str, Any] | None,
*,
now: datetime | None = None,
) -> dict[str, Any]:
"""Decide whether an expired/stale author issue lock may be reclaimed (#601).
Required proof (any reclaim of non-live lock):
* lease not live (expired or dead pid)
* owner process dead OR worktree missing
* no force-delete of live foreign ownership
"""
if not existing_lock:
return {
"reclaim_allowed": True,
"reasons": ["no existing lock"],
"freshness": assess_lock_freshness(None, now=now),
}
freshness = assess_lock_freshness(existing_lock, now=now)
if freshness.get("live"):
return {
"reclaim_allowed": False,
"reasons": ["lock is still live; cannot reclaim (fail closed)"],
"freshness": freshness,
}
pid = existing_lock.get("session_pid")
if pid is None:
pid = existing_lock.get("pid")
dead = not is_process_alive(pid) if pid is not None else True
wt = str(existing_lock.get("worktree_path") or "")
missing_wt = (not wt) or (not os.path.isdir(os.path.realpath(wt)))
if not (dead or missing_wt):
return {
"reclaim_allowed": False,
"reasons": [
"expired/stale lock still has live owner pid and present worktree; "
"recovery review required (fail closed)"
],
"freshness": freshness,
"owner_pid_dead": dead,
"worktree_missing": missing_wt,
}
return {
"reclaim_allowed": True,
"reasons": [
"non-live lock with dead process and/or missing worktree; "
"sanctioned reclaim allowed"
],
"freshness": freshness,
"owner_pid_dead": dead,
"worktree_missing": missing_wt,
"prior_branch": existing_lock.get("branch_name"),
"prior_worktree": existing_lock.get("worktree_path"),
"prior_pid": pid,
}
def assess_same_issue_lease_conflict( def assess_same_issue_lease_conflict(
existing_lock: dict[str, Any] | None, existing_lock: dict[str, Any] | None,
*, *,
@@ -405,6 +463,11 @@ def assess_same_issue_lease_conflict(
and _same_realpath(str(existing_worktree or ""), worktree_path) and _same_realpath(str(existing_worktree or ""), worktree_path)
) )
if is_lease_expired(existing_lock, now=now): if is_lease_expired(existing_lock, now=now):
reclaim = assess_expired_lock_reclaim(existing_lock, now=now)
if reclaim.get("reclaim_allowed"):
# #601: expired + dead pid / missing worktree may be reclaimed
# through the normal lock path (sanctioned overwrite).
return None
return ( return (
f"Issue #{issue_number} has an expired {operation_type} lease on " f"Issue #{issue_number} has an expired {operation_type} lease on "
f"branch '{existing_branch}' from worktree '{existing_worktree}'. " f"branch '{existing_branch}' from worktree '{existing_worktree}'. "
+723
View File
@@ -0,0 +1,723 @@
"""First-class control-plane lease lifecycle (#601).
Active leases are queryable workflow state. Canonical operations:
* list / inspect
* adopt (with provenance)
* release (explicit, recorded)
* expire / reclaim
* abandon (requires proof: dead process and/or missing worktree, etc.)
Authority model:
* Control-plane DB is the coordination source for assignment/lease.
* File locks and comment-only leases are **not** authoritative alone.
* Reviewer/merger comment leases (``reviewer_pr_lease`` / merger adoption)
remain for Gitea-thread durability; they must not bypass DB assignment when
the control-plane path is in use.
Fail closed on ambiguous ownership, active foreign leases, mismatched
worktree, stale head, missing capability, and unsafe cleanup.
"""
from __future__ import annotations
import json
import os
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Callable, Mapping
import control_plane_db as cpd
# Outcomes / safe next actions (stable vocabulary for tools + tests).
SAFE_OWNER_RESUME = "owner_resume"
SAFE_WAIT_FOREIGN = "wait_foreign_active"
SAFE_RECLAIM_EXPIRED = "reclaim_expired"
SAFE_ABANDON_ALLOWED = "abandon_allowed"
SAFE_RELEASE_OWNED = "release_owned"
SAFE_STALE_PROMPT = "stale_prompt_lease"
SAFE_UNKNOWN = "inspect_only"
SAFE_NO_AUTHORITY = "file_or_comment_not_authoritative"
LEASE_STATUS_ACTIVE = "active"
LEASE_STATUS_RELEASED = "released"
LEASE_STATUS_EXPIRED = "expired"
LEASE_STATUS_ABANDONED = "abandoned"
AUTHORITATIVE_SOURCE = "control_plane_db"
class LeaseLifecycleError(RuntimeError):
"""Fail-closed lifecycle policy error."""
@dataclass(frozen=True)
class AbandonProof:
"""Required evidence to abandon a non-owned or expired lease safely."""
dead_process: bool = False
missing_worktree: bool = False
same_owner: bool = False
no_open_pr: bool = False
no_live_mutation_risk: bool = False
operator_authorized: bool = False
worktree_path: str | None = None
owner_pid: int | None = None
notes: str = ""
def as_dict(self) -> dict[str, Any]:
return {
"dead_process": self.dead_process,
"missing_worktree": self.missing_worktree,
"same_owner": self.same_owner,
"no_open_pr": self.no_open_pr,
"no_live_mutation_risk": self.no_live_mutation_risk,
"operator_authorized": self.operator_authorized,
"worktree_path": self.worktree_path,
"owner_pid": self.owner_pid,
"notes": self.notes,
}
def is_sufficient(self) -> bool:
"""Abandon requires process death or missing worktree + no mutation risk.
Foreign active leases additionally need operator_authorized unless the
owner process is dead and the worktree is missing.
"""
if not self.no_live_mutation_risk:
return False
if not (self.dead_process or self.missing_worktree):
return False
if self.same_owner:
return True
# Foreign abandon: operator flag OR (dead + missing worktree + no risk)
if self.operator_authorized:
return True
return bool(self.dead_process and self.missing_worktree and self.no_open_pr)
def is_process_alive(pid: int | None) -> bool:
if pid is None:
return False
try:
pid_i = int(pid)
except (TypeError, ValueError):
return False
if pid_i <= 0:
return False
try:
os.kill(pid_i, 0)
return True
except ProcessLookupError:
return False
except PermissionError:
# Exists but not owned by us — treat as alive (fail closed).
return True
except OSError:
return False
def worktree_exists(path: str | None) -> bool:
if not path:
return False
try:
return os.path.isdir(os.path.realpath(path))
except OSError:
return False
def _utc_now() -> datetime:
return datetime.now(timezone.utc)
def _parse_ts(value: str | None) -> datetime | None:
return cpd._parse_ts(value)
def classify_lease_freshness(
lease: Mapping[str, Any],
*,
now: datetime | None = None,
pid_checker: Callable[[int | None], bool] = is_process_alive,
worktree_checker: Callable[[str | None], bool] = worktree_exists,
) -> dict[str, Any]:
"""Classify a control-plane lease row for workflow tooling."""
moment = now or _utc_now()
status = (lease.get("status") or "").strip().lower()
expires = _parse_ts(lease.get("expires_at"))
owner_pid = lease.get("owner_pid")
if owner_pid is None:
owner_pid = lease.get("session_pid")
wt = lease.get("worktree_path")
pid_alive = pid_checker(owner_pid) if owner_pid is not None else None
wt_present = worktree_checker(wt) if wt else None
expired_by_time = bool(expires and expires <= moment)
if status == LEASE_STATUS_ABANDONED:
freshness = "abandoned"
elif status == LEASE_STATUS_RELEASED:
freshness = "released"
elif status == LEASE_STATUS_EXPIRED or expired_by_time:
freshness = "expired"
elif status != LEASE_STATUS_ACTIVE:
freshness = status or "unknown"
elif pid_alive is False:
freshness = "stale_dead_process"
elif wt is not None and wt_present is False:
freshness = "stale_missing_worktree"
else:
freshness = "active"
return {
"freshness": freshness,
"status": status,
"expired_by_time": expired_by_time,
"owner_pid": owner_pid,
"owner_pid_alive": pid_alive,
"worktree_path": wt,
"worktree_present": wt_present,
"expires_at": lease.get("expires_at"),
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
def decide_safe_next_action(
*,
lease: Mapping[str, Any] | None,
caller_session_id: str,
freshness: Mapping[str, Any] | None = None,
caller_worktree: str | None = None,
) -> dict[str, Any]:
"""Return safe_next_action + reasons for a caller inspecting a lease."""
if not lease:
return {
"safe_next_action": SAFE_STALE_PROMPT,
"reasons": ["lease not found in control-plane DB (stale prompt id?)"],
"block": True,
}
fr = freshness or classify_lease_freshness(lease)
owner = str(lease.get("session_id") or "")
same_owner = owner == str(caller_session_id)
status = fr.get("freshness")
reasons: list[str] = []
# Worktree mismatch for owner resume
lease_wt = lease.get("worktree_path")
if (
same_owner
and lease_wt
and caller_worktree
and os.path.realpath(str(lease_wt)) != os.path.realpath(str(caller_worktree))
):
return {
"safe_next_action": SAFE_UNKNOWN,
"reasons": [
f"worktree mismatch: lease has {lease_wt!r}, caller has "
f"{caller_worktree!r} (fail closed)"
],
"block": True,
"same_owner": True,
}
if status in ("abandoned", "released"):
return {
"safe_next_action": SAFE_STALE_PROMPT,
"reasons": [f"lease status is {status}; do not adopt blindly"],
"block": True,
"same_owner": same_owner,
}
if status == "expired" or fr.get("expired_by_time"):
return {
"safe_next_action": SAFE_RECLAIM_EXPIRED,
"reasons": ["lease expired; reclaim via expire+assign or adopt reclaim path"],
"block": False,
"same_owner": same_owner,
}
if status in ("stale_dead_process", "stale_missing_worktree"):
if same_owner:
return {
"safe_next_action": SAFE_OWNER_RESUME,
"reasons": [
f"caller owns lease with freshness={status}; rebind via "
"adopt/owner-resume or abandon with proof"
],
"block": False,
"same_owner": True,
"also_allowed": [SAFE_ABANDON_ALLOWED, SAFE_RELEASE_OWNED],
}
return {
"safe_next_action": SAFE_ABANDON_ALLOWED,
"reasons": [
f"lease freshness={status}; abandon with required proof then reassign"
],
"block": False,
"same_owner": False,
}
if same_owner and status == "active":
return {
"safe_next_action": SAFE_OWNER_RESUME,
"reasons": [
"caller owns active lease; resume via heartbeat/assign owner-resume "
"or release explicitly"
],
"block": False,
"same_owner": True,
"also_allowed": [SAFE_RELEASE_OWNED],
}
if not same_owner and status == "active":
return {
"safe_next_action": SAFE_WAIT_FOREIGN,
"reasons": [
f"foreign active lease held by session {owner}; "
"do not steal (fail closed)"
],
"block": True,
"same_owner": False,
"owner_session_id": owner,
}
return {
"safe_next_action": SAFE_UNKNOWN,
"reasons": [f"unclassified freshness={status}"],
"block": True,
"same_owner": same_owner,
}
def non_db_lease_authority_report(
*,
file_lock_present: bool = False,
comment_lease_present: bool = False,
db_lease_present: bool = False,
) -> dict[str, Any]:
"""File/comment leases alone are not coordination authority (#601 / #600)."""
authoritative = bool(db_lease_present)
reasons: list[str] = []
if file_lock_present and not db_lease_present:
reasons.append(
"file lock present but control-plane DB lease absent — "
"file lock is not authoritative alone"
)
if comment_lease_present and not db_lease_present:
reasons.append(
"comment-only lease present but control-plane DB lease absent — "
"comment lease is not authoritative alone"
)
if authoritative:
reasons.append("control-plane DB lease is the coordination authority")
return {
"authoritative_source": AUTHORITATIVE_SOURCE if authoritative else None,
"db_lease_present": db_lease_present,
"file_lock_present": file_lock_present,
"comment_lease_present": comment_lease_present,
"safe_next_action": (
SAFE_UNKNOWN if authoritative else SAFE_NO_AUTHORITY
),
"reasons": reasons,
"file_lock_only": bool(file_lock_present and not db_lease_present),
"comment_lease_only": bool(comment_lease_present and not db_lease_present),
}
def build_adopt_provenance(
*,
adopted_from_session_id: str,
adopted_by_session_id: str,
work_kind: str,
work_number: int,
remote: str,
org: str,
repo: str,
worktree_path: str | None,
expected_head_sha: str | None,
prior_lease_id: str | None,
reason: str,
) -> dict[str, Any]:
return {
"adopted_from_session_id": adopted_from_session_id,
"adopted_by_session_id": adopted_by_session_id,
"work_kind": work_kind,
"work_number": int(work_number),
"remote": remote,
"org": org,
"repo": repo,
"worktree_path": worktree_path,
"expected_head_sha": expected_head_sha,
"prior_lease_id": prior_lease_id,
"reason": reason,
"recorded_at": cpd._ts(),
"source": "lease_lifecycle.adopt",
}
def inspect_lease(
db: cpd.ControlPlaneDB,
lease_id: str,
*,
caller_session_id: str,
caller_worktree: str | None = None,
) -> dict[str, Any]:
"""Full first-class lease workflow state for one lease id."""
state = db.get_lease_workflow_state(lease_id)
if not state:
decision = decide_safe_next_action(
lease=None, caller_session_id=caller_session_id
)
return {
"success": True,
"found": False,
"lease_id": lease_id,
"lease": None,
"freshness": None,
**decision,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
lease = state["lease"]
freshness = classify_lease_freshness(lease)
decision = decide_safe_next_action(
lease=lease,
caller_session_id=caller_session_id,
freshness=freshness,
caller_worktree=caller_worktree,
)
return {
"success": True,
"found": True,
"lease_id": lease_id,
"lease": lease,
"assignment": state.get("assignment"),
"work_item": state.get("work_item"),
"session": state.get("session"),
"freshness": freshness,
"provenance": state.get("provenance"),
**decision,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
def list_active_leases(
db: cpd.ControlPlaneDB,
*,
remote: str | None = None,
org: str | None = None,
repo: str | None = None,
role: str | None = None,
include_non_active: bool = False,
limit: int = 100,
) -> dict[str, Any]:
statuses = None if include_non_active else (LEASE_STATUS_ACTIVE,)
rows = db.list_leases(
remote=remote,
org=org,
repo=repo,
role=role,
statuses=statuses,
limit=limit,
)
enriched = []
for row in rows:
fr = classify_lease_freshness(row)
enriched.append({**row, "freshness": fr})
return {
"success": True,
"count": len(enriched),
"leases": enriched,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
"include_non_active": include_non_active,
}
def adopt_lease(
db: cpd.ControlPlaneDB,
*,
lease_id: str,
adopter_session_id: str,
role: str,
worktree_path: str | None = None,
expected_head_sha: str | None = None,
owner_pid: int | None = None,
operator_authorized: bool = False,
) -> dict[str, Any]:
"""Sanctioned adopt path with provenance; never silent foreign steal."""
state = db.get_lease_workflow_state(lease_id)
if not state:
raise LeaseLifecycleError(
f"unknown lease_id {lease_id}; stale prompt id (fail closed)"
)
lease = state["lease"]
work = state["work_item"]
freshness = classify_lease_freshness(lease)
owner = str(lease.get("session_id") or "")
same_owner = owner == str(adopter_session_id)
if freshness["freshness"] == "active" and not same_owner:
raise LeaseLifecycleError(
f"refusing to steal active foreign lease {lease_id} owned by "
f"{owner} (fail closed)"
)
if freshness["freshness"] in ("abandoned", "released"):
raise LeaseLifecycleError(
f"lease {lease_id} is {freshness['freshness']}; cannot adopt "
"(fail closed)"
)
# Expired or stale: require abandon-style safety before ownership transfer
# when not same owner; same owner may reclaim.
if not same_owner and freshness["freshness"] in (
"expired",
"stale_dead_process",
"stale_missing_worktree",
):
if not operator_authorized and freshness["freshness"] == "expired":
# Deterministic reclaim of expired foreign lease is allowed
# without operator flag (sanctioned expire reclaim).
pass
elif freshness["freshness"] != "expired" and not operator_authorized:
# stale active-looking requires abandon proof path
raise LeaseLifecycleError(
f"lease {lease_id} freshness={freshness['freshness']}; "
"use abandon with proof before foreign adopt (fail closed)"
)
provenance = build_adopt_provenance(
adopted_from_session_id=owner,
adopted_by_session_id=adopter_session_id,
work_kind=str(work.get("kind")),
work_number=int(work.get("number")),
remote=str(work.get("remote")),
org=str(work.get("org")),
repo=str(work.get("repo")),
worktree_path=worktree_path,
expected_head_sha=expected_head_sha or lease.get("expected_head_sha"),
prior_lease_id=lease_id,
reason=(
"owner-resume-adopt" if same_owner else "sanctioned-reclaim-adopt"
),
)
result = db.adopt_lease(
lease_id=lease_id,
adopter_session_id=adopter_session_id,
role=role,
worktree_path=worktree_path,
expected_head_sha=expected_head_sha,
owner_pid=owner_pid if owner_pid is not None else os.getpid(),
provenance=provenance,
)
return {
"success": True,
"outcome": result.get("outcome"),
"same_owner": same_owner,
"prior_lease_id": lease_id,
"lease": result.get("lease"),
"assignment": result.get("assignment"),
"provenance": provenance,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
"reasons": result.get("reasons") or [],
}
def release_lease(
db: cpd.ControlPlaneDB,
*,
lease_id: str,
session_id: str,
) -> dict[str, Any]:
proof = db.release_lease_recorded(lease_id=lease_id, session_id=session_id)
return {
"success": True,
"outcome": "released",
"lease_id": lease_id,
"session_id": session_id,
"release_proof": proof,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
def expire_leases(db: cpd.ControlPlaneDB) -> dict[str, Any]:
n = db.expire_stale_leases()
return {
"success": True,
"outcome": "expired",
"expired_count": int(n),
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
def reclaim_expired_lease(
db: cpd.ControlPlaneDB,
*,
lease_id: str,
session_id: str,
role: str,
worktree_path: str | None = None,
expected_head_sha: str | None = None,
) -> dict[str, Any]:
"""Expire-if-needed then assign to *session_id* for the same work item."""
state = db.get_lease_workflow_state(lease_id)
if not state:
raise LeaseLifecycleError(f"unknown lease_id {lease_id}")
lease = state["lease"]
work = state["work_item"]
freshness = classify_lease_freshness(lease)
if freshness["freshness"] == "active":
if lease.get("session_id") != session_id:
raise LeaseLifecycleError(
f"cannot reclaim active foreign lease {lease_id} (fail closed)"
)
# owner resume
return adopt_lease(
db,
lease_id=lease_id,
adopter_session_id=session_id,
role=role,
worktree_path=worktree_path,
expected_head_sha=expected_head_sha,
)
if freshness["freshness"] not in (
"expired",
"stale_dead_process",
"stale_missing_worktree",
"abandoned",
"released",
):
raise LeaseLifecycleError(
f"lease {lease_id} freshness={freshness['freshness']} not reclaimable"
)
# Ensure expired marker is applied
db.expire_stale_leases()
# If still active due to clock skew / non-time stale, force expire this lease
refreshed = db.get_lease_workflow_state(lease_id)
if refreshed and refreshed["lease"].get("status") == "active":
db.force_expire_lease(lease_id, reason="reclaim_expired_lease")
head = expected_head_sha or work.get("current_head_sha")
assigned = db.assign_and_lease(
session_id=session_id,
role=role,
remote=str(work["remote"]),
org=str(work["org"]),
repo=str(work["repo"]),
kind=str(work["kind"]),
number=int(work["number"]),
expected_head_sha=head,
phase="reclaimed",
worktree_path=worktree_path,
owner_pid=os.getpid(),
)
if assigned.outcome != "assigned":
raise LeaseLifecycleError(
f"reclaim assign failed: {assigned.outcome}{assigned.reason}"
)
provenance = build_adopt_provenance(
adopted_from_session_id=str(lease.get("session_id")),
adopted_by_session_id=session_id,
work_kind=str(work["kind"]),
work_number=int(work["number"]),
remote=str(work["remote"]),
org=str(work["org"]),
repo=str(work["repo"]),
worktree_path=worktree_path,
expected_head_sha=head,
prior_lease_id=lease_id,
reason="reclaim_expired",
)
db.attach_lease_provenance(assigned.lease_id, provenance)
return {
"success": True,
"outcome": "reclaimed",
"prior_lease_id": lease_id,
"assignment": assigned.as_dict(),
"provenance": provenance,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
def abandon_lease(
db: cpd.ControlPlaneDB,
*,
lease_id: str,
requester_session_id: str,
proof: AbandonProof,
) -> dict[str, Any]:
state = db.get_lease_workflow_state(lease_id)
if not state:
raise LeaseLifecycleError(f"unknown lease_id {lease_id}")
lease = state["lease"]
owner = str(lease.get("session_id") or "")
same_owner = owner == str(requester_session_id)
# Enrich proof with live checks when not provided
owner_pid = proof.owner_pid
if owner_pid is None:
owner_pid = lease.get("owner_pid")
wt = proof.worktree_path if proof.worktree_path is not None else lease.get(
"worktree_path"
)
dead = proof.dead_process or (
owner_pid is not None and not is_process_alive(owner_pid)
)
missing_wt = proof.missing_worktree or (
bool(wt) and not worktree_exists(str(wt))
)
enriched = AbandonProof(
dead_process=bool(dead),
missing_worktree=bool(missing_wt),
same_owner=same_owner or proof.same_owner,
no_open_pr=proof.no_open_pr,
no_live_mutation_risk=proof.no_live_mutation_risk,
operator_authorized=proof.operator_authorized,
worktree_path=str(wt) if wt else None,
owner_pid=int(owner_pid) if owner_pid is not None else None,
notes=proof.notes,
)
if not enriched.is_sufficient():
raise LeaseLifecycleError(
"abandon proof insufficient: need (dead_process or missing_worktree) "
"and no_live_mutation_risk; foreign abandon also needs "
"operator_authorized or (dead+missing_worktree+no_open_pr). "
f"proof={enriched.as_dict()}"
)
# Active foreign with live process + present worktree: never abandon without
# operator (already covered by is_sufficient).
result = db.abandon_lease(
lease_id=lease_id,
requester_session_id=requester_session_id,
proof=enriched.as_dict(),
)
return {
"success": True,
"outcome": "abandoned",
"lease_id": lease_id,
"requester_session_id": requester_session_id,
"prior_owner_session_id": owner,
"abandon_proof": enriched.as_dict(),
"audit": result,
"authoritative_source": AUTHORITATIVE_SOURCE,
"file_lock_only": False,
"comment_lease_only": False,
}
+97
View File
@@ -139,6 +139,103 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
"permission": "gitea.pr.create", "permission": "gitea.pr.create",
"role": "author", "role": "author",
}, },
# #600: controller-owned allocator — any authenticated profile may call;
# routing enforces role match to selected work. Uses control-plane DB (#613).
"allocate_next_work": {
"permission": "gitea.read",
"role": "author",
},
"gitea_allocate_next_work": {
"permission": "gitea.read",
"role": "author",
},
# #601 first-class lease lifecycle — inspect/list need read; mutations gate on
# ownership in the control-plane DB (not a separate Gitea write permission).
"list_workflow_leases": {
"permission": "gitea.read",
"role": "author",
},
"gitea_list_workflow_leases": {
"permission": "gitea.read",
"role": "author",
},
"inspect_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"gitea_inspect_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"adopt_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"gitea_adopt_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"release_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"gitea_release_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"expire_workflow_leases": {
"permission": "gitea.read",
"role": "author",
},
"gitea_expire_workflow_leases": {
"permission": "gitea.read",
"role": "author",
},
"abandon_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"gitea_abandon_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"reclaim_expired_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
"gitea_reclaim_expired_workflow_lease": {
"permission": "gitea.read",
"role": "author",
},
# #612 incident bridge — reconcile uses create_issue for apply;
# dry-run needs read only. Tools gate apply paths themselves.
"observability_reconcile_incident": {
"permission": "gitea.read",
"role": "author",
},
"gitea_observability_reconcile_incident": {
"permission": "gitea.read",
"role": "author",
},
"observability_list_projects": {
"permission": "gitea.read",
"role": "author",
},
"gitea_observability_list_projects": {
"permission": "gitea.read",
"role": "author",
},
"observability_link_issue": {
"permission": "gitea.read",
"role": "author",
},
"gitea_observability_link_issue": {
"permission": "gitea.read",
"role": "author",
},
"reconcile_landed_pr": { "reconcile_landed_pr": {
"permission": "gitea.read", "permission": "gitea.read",
"role": "author", "role": "author",
+366
View File
@@ -0,0 +1,366 @@
"""Tests for controller-owned allocator (#600) on control-plane DB (#613)."""
from __future__ import annotations
import os
import tempfile
import threading
import unittest
from concurrent.futures import ThreadPoolExecutor, as_completed
from allocator_service import (
OUTCOME_ASSIGNED,
OUTCOME_BLOCKED_TERMINAL,
OUTCOME_NO_SAFE,
OUTCOME_PREVIEW,
OUTCOME_WAIT,
WorkCandidate,
allocate_next_work,
candidate_from_dict,
classify_skip,
expected_role_for_candidate,
)
from control_plane_db import ControlPlaneDB, InvalidWorkKindError
class AllocatorServiceTest(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self._tmp.name, "cp.sqlite3")
self.db = ControlPlaneDB(self.db_path)
def tearDown(self) -> None:
self._tmp.cleanup()
def _alloc(self, **kwargs):
defaults = dict(
db=self.db,
session_id="s-test",
role="author",
remote="prgs",
org="org",
repo="repo",
candidates=[],
apply=False,
profile_name="prgs-author",
username="jcwalker3",
)
defaults.update(kwargs)
return allocate_next_work(**defaults)
def test_selects_ready_issue_for_author(self) -> None:
cands = [
WorkCandidate(
kind="issue",
number=612,
labels=("status:ready",),
title="bridge",
dependency_unmet=True,
dependency_reason="downstream of #600",
),
WorkCandidate(
kind="issue",
number=600,
labels=("status:ready", "type:feature"),
title="allocator",
priority=50,
),
WorkCandidate(
kind="issue",
number=601,
labels=("status:blocked",),
title="blocked",
blocked=True,
),
]
res = self._alloc(candidates=cands, apply=False)
self.assertTrue(res["success"])
self.assertEqual(res["outcome"], OUTCOME_PREVIEW)
self.assertEqual(res["selected"]["number"], 600)
self.assertEqual(res["selected"]["kind"], "issue")
# When 600 is highest priority valid work, lower-priority blocked/dep
# candidates are not visited. Prove they are skipped when they sort first.
res2 = self._alloc(
candidates=[
WorkCandidate(
kind="issue",
number=612,
labels=("status:ready",),
priority=99,
dependency_unmet=True,
dependency_reason="downstream of #600",
),
WorkCandidate(
kind="issue",
number=601,
labels=("status:blocked",),
priority=98,
blocked=True,
),
WorkCandidate(
kind="issue",
number=600,
labels=("status:ready",),
priority=1,
),
],
apply=False,
)
skipped_nums = {s["number"] for s in res2["skipped"]}
self.assertIn(612, skipped_nums)
self.assertIn(601, skipped_nums)
self.assertEqual(res2["selected"]["number"], 600)
self.assertIsNone(res["assignment"])
self.assertFalse(res["file_lock_only"])
self.assertFalse(res["comment_lease_only"])
def test_atomic_assign_and_lease_on_apply(self) -> None:
cands = [
WorkCandidate(
kind="issue",
number=600,
labels=("status:ready",),
priority=10,
)
]
res = self._alloc(candidates=cands, apply=True, session_id="s-a")
self.assertEqual(res["outcome"], OUTCOME_ASSIGNED)
asn = res["assignment"]
self.assertEqual(asn["outcome"], "assigned")
self.assertIsNotNone(asn["assignment_id"])
self.assertIsNotNone(asn["lease_id"])
self.assertEqual(asn["work_number"], 600)
self.assertIn("implement", asn["allowed_actions"])
self.assertIn("merge", asn["forbidden_actions"])
proof = res["lease_proof"]
self.assertEqual(proof["source"], "control_plane_db.assign_and_lease")
def test_concurrent_allocators_no_double_assign(self) -> None:
cand = WorkCandidate(
kind="pr",
number=100,
head_sha="a" * 40,
priority=10,
)
# Seed work item so both race the same target
self.db.upsert_work_item(
remote="prgs",
org="org",
repo="repo",
kind="pr",
number=100,
current_head_sha="a" * 40,
)
results = []
lock = threading.Lock()
def worker(sid: str):
r = allocate_next_work(
self.db,
session_id=sid,
role="reviewer",
remote="prgs",
org="org",
repo="repo",
candidates=[cand],
apply=True,
profile_name="prgs-reviewer",
)
with lock:
results.append(r)
with ThreadPoolExecutor(max_workers=2) as pool:
futs = [pool.submit(worker, f"s-{i}") for i in range(2)]
for f in as_completed(futs):
f.result()
outcomes = [r["outcome"] for r in results]
self.assertEqual(outcomes.count(OUTCOME_ASSIGNED), 1, outcomes)
self.assertEqual(outcomes.count(OUTCOME_WAIT), 1, outcomes)
def test_blocked_and_dependency_skipped(self) -> None:
cands = [
WorkCandidate(kind="issue", number=1, blocked=True, labels=("status:blocked",)),
WorkCandidate(
kind="issue",
number=612,
labels=("status:ready",),
dependency_unmet=True,
dependency_reason="downstream of #600",
),
]
res = self._alloc(candidates=cands, apply=True, role="author")
self.assertEqual(res["outcome"], OUTCOME_NO_SAFE)
self.assertIsNone(res["selected"])
reasons = " ".join(s["reason"] for s in res["skipped"])
self.assertIn("blocked", reasons.lower())
self.assertIn("600", reasons)
def test_already_leased_returns_wait(self) -> None:
self.db.upsert_session(session_id="owner", role="author")
self.db.assign_and_lease(
session_id="owner",
role="author",
remote="prgs",
org="org",
repo="repo",
kind="issue",
number=50,
)
res = self._alloc(
session_id="other",
candidates=[
WorkCandidate(kind="issue", number=50, labels=("status:ready",))
],
apply=True,
)
self.assertEqual(res["outcome"], OUTCOME_WAIT)
self.assertEqual(res["owner_session_id"], "owner")
def test_role_ineligible_skipped(self) -> None:
# PR with current-head REQUEST_CHANGES expects author, not reviewer.
cand = WorkCandidate(
kind="pr",
number=9,
head_sha="b" * 40,
request_changes_current_head=True,
)
self.assertEqual(expected_role_for_candidate(cand), "author")
res = self._alloc(
role="reviewer",
profile_name="prgs-reviewer",
candidates=[cand],
apply=False,
)
self.assertEqual(res["outcome"], OUTCOME_NO_SAFE)
self.assertTrue(any("expects role" in s["reason"] for s in res["skipped"]))
def test_terminal_lock_blocks_downstream_reviewer_prs(self) -> None:
self.db.set_terminal_lock(
remote="prgs",
org="org",
repo="repo",
terminal_pr=10,
decision="request_changes",
status="active",
)
cands = [
WorkCandidate(kind="pr", number=11, head_sha="c" * 40, priority=5),
WorkCandidate(kind="pr", number=10, head_sha="d" * 40, priority=1),
]
res = self._alloc(
role="reviewer",
profile_name="prgs-reviewer",
candidates=cands,
apply=False,
)
# Terminal PR #10 is selectable; #11 skipped for terminal path.
self.assertEqual(res["selected"]["number"], 10)
self.assertTrue(
any(
s["number"] == 11 and "terminal-review lock" in s["reason"]
for s in res["skipped"]
)
)
def test_terminal_lock_blocks_all_when_only_downstream(self) -> None:
self.db.set_terminal_lock(
remote="prgs",
org="org",
repo="repo",
terminal_pr=10,
decision="request_changes",
status="active",
)
res = self._alloc(
role="reviewer",
profile_name="prgs-reviewer",
candidates=[
WorkCandidate(kind="pr", number=99, head_sha="e" * 40),
],
apply=False,
)
self.assertEqual(res["outcome"], OUTCOME_BLOCKED_TERMINAL)
def test_no_work_structured(self) -> None:
res = self._alloc(candidates=[], apply=True)
self.assertEqual(res["outcome"], OUTCOME_NO_SAFE)
self.assertIsNone(res["selected"])
self.assertTrue(res["reasons"])
def test_rejects_incident_kind(self) -> None:
with self.assertRaises(InvalidWorkKindError):
WorkCandidate(kind="sentry_incident", number=1)
def test_612_downstream_marker_in_result(self) -> None:
res = self._alloc(
candidates=[
WorkCandidate(kind="issue", number=600, labels=("status:ready",))
],
apply=True,
)
self.assertIn("612", res.get("downstream_note", ""))
def test_substrate_not_file_or_comment_lease(self) -> None:
res = self._alloc(
candidates=[
WorkCandidate(kind="issue", number=1, labels=("status:ready",))
],
apply=True,
)
self.assertEqual(res["substrate"], "control_plane_db")
self.assertFalse(res["file_lock_only"])
self.assertFalse(res["comment_lease_only"])
self.assertEqual(
res["lease_proof"]["source"], "control_plane_db.assign_and_lease"
)
def test_candidate_from_dict(self) -> None:
c = candidate_from_dict(
{
"kind": "pr",
"number": 7,
"head_sha": "f" * 40,
"approval_on_current_head": True,
"mergeable": True,
}
)
self.assertEqual(expected_role_for_candidate(c), "merger")
def test_merger_gets_clean_approval(self) -> None:
c = WorkCandidate(
kind="pr",
number=3,
head_sha="1" * 40,
approval_on_current_head=True,
mergeable=True,
priority=100,
)
res = self._alloc(
role="merger",
profile_name="prgs-merger",
candidates=[c],
apply=True,
session_id="merger-1",
)
self.assertEqual(res["outcome"], OUTCOME_ASSIGNED)
self.assertEqual(res["assignment"]["work_number"], 3)
self.assertIn("merge", res["assignment"]["allowed_actions"])
def test_db_unavailable_fails_closed(self) -> None:
res = allocate_next_work(
None, # type: ignore[arg-type]
session_id="x",
role="author",
remote="prgs",
org="o",
repo="r",
candidates=[],
apply=True,
)
self.assertFalse(res["success"])
self.assertIn("unavailable", res["reasons"][0].lower())
if __name__ == "__main__":
unittest.main()
+824
View File
@@ -0,0 +1,824 @@
"""Tests for control-plane DB substrate (#613)."""
from __future__ import annotations
import os
import tempfile
import threading
import unittest
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import timedelta
from control_plane_db import (
ControlPlaneDB,
InvalidWorkKindError,
LeaseRequiredError,
WORK_KINDS,
_ts,
_utc_now,
)
class ControlPlaneDBTest(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self._tmp.name, "cp.sqlite3")
self.db = ControlPlaneDB(self.db_path)
def tearDown(self) -> None:
self._tmp.cleanup()
def test_schema_and_architecture_meta(self) -> None:
import sqlite3
conn = sqlite3.connect(self.db_path)
try:
rows = dict(conn.execute("SELECT key, value FROM schema_meta").fetchall())
finally:
conn.close()
self.assertEqual(rows["schema_version"], "3")
self.assertIn("DB coordinates", rows["architecture"])
self.assertIn("bridge", rows["architecture"].lower())
def test_rejects_raw_incident_as_work_kind(self) -> None:
with self.assertRaises(InvalidWorkKindError):
self.db.upsert_work_item(
remote="prgs",
org="org",
repo="repo",
kind="sentry_incident",
number=1,
)
with self.assertRaises(InvalidWorkKindError):
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="org",
repo="repo",
kind="glitchtip_incident",
number=9,
)
self.assertEqual(WORK_KINDS, frozenset({"issue", "pr"}))
def test_atomic_assign_and_lease_fields(self) -> None:
self.db.upsert_session(session_id="s-a", role="author", profile="prgs-author")
result = self.db.assign_and_lease(
session_id="s-a",
role="author",
remote="prgs",
org="Scaled-Tech-Consulting",
repo="Gitea-Tools",
kind="issue",
number=613,
expected_head_sha="abc123",
allowed_actions=("implement", "comment"),
forbidden_actions=("approve", "merge"),
)
self.assertEqual(result.outcome, "assigned")
self.assertIsNotNone(result.assignment_id)
self.assertIsNotNone(result.lease_id)
self.assertEqual(result.role, "author")
self.assertEqual(result.work_kind, "issue")
self.assertEqual(result.work_number, 613)
self.assertEqual(result.expected_head_sha, "abc123")
self.assertIn("implement", result.allowed_actions)
self.assertIn("merge", result.forbidden_actions)
self.assertIsNotNone(result.expires_at)
def test_second_session_waits_on_foreign_lease(self) -> None:
self.db.upsert_session(session_id="s1", role="author")
self.db.upsert_session(session_id="s2", role="author")
first = self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=100,
expected_head_sha="deadbeef",
)
self.assertEqual(first.outcome, "assigned")
second = self.db.assign_and_lease(
session_id="s2",
role="author",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=100,
expected_head_sha="deadbeef",
)
self.assertEqual(second.outcome, "wait")
self.assertEqual(second.owner_session_id, "s1")
def test_owner_resume_refreshes_lease(self) -> None:
self.db.upsert_session(session_id="s1", role="reviewer")
a = self.db.assign_and_lease(
session_id="s1",
role="reviewer",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=50,
expected_head_sha="head-50",
)
b = self.db.assign_and_lease(
session_id="s1",
role="reviewer",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=50,
expected_head_sha="head-50",
)
self.assertEqual(b.outcome, "assigned")
self.assertEqual(b.lease_id, a.lease_id)
self.assertIn("owner-resume", b.reason)
def test_require_valid_assignment_gates_mutations(self) -> None:
self.db.upsert_session(session_id="s1", role="author")
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=7,
allowed_actions=("implement",),
forbidden_actions=("merge",),
)
proof = self.db.require_valid_assignment(
session_id="s1",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=7,
action="implement",
)
self.assertEqual(proof["session_id"], "s1")
with self.assertRaises(LeaseRequiredError):
self.db.require_valid_assignment(
session_id="s1",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=7,
action="merge",
)
with self.assertRaises(LeaseRequiredError):
self.db.require_valid_assignment(
session_id="s-other",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=7,
action="implement",
)
def test_expired_lease_allows_reassign(self) -> None:
self.db.upsert_session(session_id="s1", role="author")
self.db.upsert_session(session_id="s2", role="author")
past = _utc_now() - timedelta(hours=1)
# Create lease already expired by using negative TTL edge via direct assign then expire
assigned = self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=3,
lease_ttl_seconds=1,
)
self.assertEqual(assigned.outcome, "assigned")
# Force expiry in DB
import sqlite3
conn = sqlite3.connect(self.db_path)
try:
conn.execute(
"UPDATE leases SET expires_at = ? WHERE lease_id = ?",
(_ts(past), assigned.lease_id),
)
conn.commit()
finally:
conn.close()
n = self.db.expire_stale_leases()
self.assertGreaterEqual(n, 1)
second = self.db.assign_and_lease(
session_id="s2",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=3,
)
self.assertEqual(second.outcome, "assigned")
self.assertEqual(second.session_id, "s2")
def test_merged_work_never_assigned(self) -> None:
self.db.upsert_session(session_id="s1", role="merger")
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="pr",
number=99,
state="merged",
)
result = self.db.assign_and_lease(
session_id="s1",
role="merger",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=99,
expected_head_sha="merged-head",
)
self.assertEqual(result.outcome, "no_safe_work")
def test_four_concurrent_sessions_unique_assignments(self) -> None:
"""Four concurrent assigners on four different issues — all succeed uniquely.
Also two concurrent assigners on the *same* issue: at most one assigned.
"""
for i in range(4):
self.db.upsert_session(session_id=f"sess-{i}", role="author")
def claim_unique(i: int):
return self.db.assign_and_lease(
session_id=f"sess-{i}",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=1000 + i,
)
with ThreadPoolExecutor(max_workers=4) as pool:
results = [f.result() for f in as_completed([pool.submit(claim_unique, i) for i in range(4)])]
self.assertEqual({r.outcome for r in results}, {"assigned"})
numbers = sorted(r.work_number for r in results)
self.assertEqual(numbers, [1000, 1001, 1002, 1003])
# Contention on one item
self.db.upsert_session(session_id="c1", role="author")
self.db.upsert_session(session_id="c2", role="author")
self.db.upsert_session(session_id="c3", role="author")
self.db.upsert_session(session_id="c4", role="author")
barrier = threading.Barrier(4)
outcomes: list[str] = []
lock = threading.Lock()
def contend(sid: str) -> None:
barrier.wait()
res = self.db.assign_and_lease(
session_id=sid,
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=7777,
)
with lock:
outcomes.append(res.outcome)
threads = [threading.Thread(target=contend, args=(f"c{i}",)) for i in range(1, 5)]
for t in threads:
t.start()
for t in threads:
t.join()
self.assertEqual(outcomes.count("assigned"), 1)
self.assertEqual(outcomes.count("wait"), 3)
def test_terminal_lock_index(self) -> None:
self.db.set_terminal_lock(
remote="prgs",
org="o",
repo="r",
terminal_pr=332,
review_id="rev-1",
decision="approve",
)
row = self.db.get_active_terminal_lock(remote="prgs", org="o", repo="r")
self.assertIsNotNone(row)
assert row is not None
self.assertEqual(row["terminal_pr"], 332)
self.assertEqual(row["status"], "active")
def test_incident_links_not_work_items(self) -> None:
link = self.db.upsert_incident_link(
provider="sentry",
provider_base_url="https://sentry.prgs.cc",
provider_org="prgs",
provider_project="gitea-tools-mcp",
provider_issue_id="12345",
gitea_org="Scaled-Tech-Consulting",
gitea_repo="Gitea-Tools",
gitea_issue_number=9001,
fingerprint="fp-1",
)
self.assertEqual(link["gitea_issue_number"], 9001)
found = self.db.get_incident_link_for_gitea_issue(
gitea_org="Scaled-Tech-Consulting",
gitea_repo="Gitea-Tools",
gitea_issue_number=9001,
)
self.assertIsNotNone(found)
# Linking does not create a work_item of incident kind
with self.assertRaises(InvalidWorkKindError):
self.db.upsert_work_item(
remote="prgs",
org="Scaled-Tech-Consulting",
repo="Gitea-Tools",
kind="sentry",
number=12345,
)
def test_heartbeat_and_release(self) -> None:
self.db.upsert_session(session_id="s1", role="author")
a = self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=1,
)
hb = self.db.heartbeat_lease(a.lease_id, session_id="s1")
self.assertEqual(hb["lease_id"], a.lease_id)
self.db.release_lease(a.lease_id, session_id="s1")
# After release another session can claim
self.db.upsert_session(session_id="s2", role="author")
b = self.db.assign_and_lease(
session_id="s2",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=1,
)
self.assertEqual(b.outcome, "assigned")
self.assertEqual(b.session_id, "s2")
def test_require_valid_assignment_rejects_stale_head(self) -> None:
"""Assignment must not authorize mutations after work-item head drifts."""
self.db.upsert_session(session_id="s1", role="author")
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=42,
expected_head_sha="head-v1",
allowed_actions=("implement",),
)
# Head drifts after assignment
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="pr",
number=42,
current_head_sha="head-v2",
)
with self.assertRaises(LeaseRequiredError) as ctx:
self.db.require_valid_assignment(
session_id="s1",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=42,
action="implement",
)
self.assertIn("stale head", str(ctx.exception).lower())
def test_require_valid_assignment_rejects_terminal_state(self) -> None:
"""Assignment must not authorize mutations after work item is merged/closed."""
self.db.upsert_session(session_id="s1", role="author")
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=55,
expected_head_sha="abc",
allowed_actions=("implement",),
)
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="pr",
number=55,
state="merged",
current_head_sha="abc",
)
with self.assertRaises(LeaseRequiredError) as ctx:
self.db.require_valid_assignment(
session_id="s1",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=55,
action="implement",
)
self.assertIn("terminal", str(ctx.exception).lower())
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="issue",
number=56,
state="open",
)
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=56,
allowed_actions=("implement",),
)
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="issue",
number=56,
state="closed",
)
with self.assertRaises(LeaseRequiredError):
self.db.require_valid_assignment(
session_id="s1",
remote="prgs",
org="o",
repo="r",
kind="issue",
number=56,
action="implement",
)
def test_pr_assignment_requires_expected_head_sha(self) -> None:
"""PR assign and mutation must fail closed without a head pin."""
self.db.upsert_session(session_id="s1", role="author")
with self.assertRaises(LeaseRequiredError) as ctx:
self.db.assign_and_lease(
session_id="s1",
role="author",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=88,
expected_head_sha=None,
allowed_actions=("implement",),
)
self.assertIn("expected_head_sha", str(ctx.exception))
# Legacy path: force an unpinned PR assignment into the DB, then
# populate head and prove mutation is still rejected.
import sqlite3
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="pr",
number=89,
current_head_sha=None,
)
conn = sqlite3.connect(self.db_path)
try:
wid = conn.execute(
"SELECT work_item_id FROM work_items WHERE kind='pr' AND number=89"
).fetchone()[0]
conn.execute(
"""
INSERT INTO sessions(session_id, role, started_at, last_heartbeat_at, status)
VALUES ('legacy', 'author', '2020-01-01T00:00:00Z', '2020-01-01T00:00:00Z', 'active')
"""
)
conn.execute(
"""
INSERT INTO leases(
lease_id, work_item_id, session_id, role, phase,
expires_at, heartbeat_at, status
) VALUES (
'lease-legacy', ?, 'legacy', 'author', 'claimed',
'2099-01-01T00:00:00Z', '2020-01-01T00:00:00Z', 'active'
)
""",
(wid,),
)
conn.execute(
"""
INSERT INTO assignments(
assignment_id, work_item_id, session_id, lease_id,
allowed_actions, forbidden_actions, expected_head_sha,
role, status, created_at
) VALUES (
'asn-legacy', ?, 'legacy', 'lease-legacy',
'["implement"]', '["merge"]', NULL,
'author', 'active', '2020-01-01T00:00:00Z'
)
""",
(wid,),
)
conn.commit()
finally:
conn.close()
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="pr",
number=89,
current_head_sha="populated-head",
)
with self.assertRaises(LeaseRequiredError) as ctx2:
self.db.require_valid_assignment(
session_id="legacy",
remote="prgs",
org="o",
repo="r",
kind="pr",
number=89,
action="implement",
)
self.assertIn("expected_head_sha pin", str(ctx2.exception))
def test_migrate_duplicate_null_scope_incident_links(self) -> None:
"""Legacy NULL-scope duplicates must migrate without UNIQUE crash."""
import sqlite3
from control_plane_db import ControlPlaneDB, ControlPlaneError
# Build a v1-like table with nullable scope columns and insert dups
# that collapse under normalization, then open ControlPlaneDB on it.
path = os.path.join(self._tmp.name, "legacy_dups.sqlite3")
conn = sqlite3.connect(path)
try:
conn.executescript(
"""
CREATE TABLE schema_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE incident_links (
link_id INTEGER PRIMARY KEY AUTOINCREMENT,
provider TEXT NOT NULL,
provider_base_url TEXT,
provider_org TEXT,
provider_project TEXT,
provider_issue_id TEXT NOT NULL,
provider_short_id TEXT,
provider_permalink TEXT,
fingerprint TEXT,
gitea_org TEXT NOT NULL,
gitea_repo TEXT NOT NULL,
gitea_issue_number INTEGER NOT NULL,
linked_pr_numbers TEXT,
first_seen TEXT,
last_seen TEXT,
event_count INTEGER,
status TEXT NOT NULL DEFAULT 'open',
release_resolved_at TEXT,
last_sync_at TEXT,
UNIQUE (
provider, provider_base_url, provider_org,
provider_project, provider_issue_id
)
);
INSERT INTO incident_links(
provider, provider_base_url, provider_org, provider_project,
provider_issue_id, gitea_org, gitea_repo, gitea_issue_number, status
) VALUES
('sentry', NULL, NULL, NULL, 'dup-1', 'org', 'repo', 10, 'open'),
('sentry', NULL, NULL, NULL, 'dup-1', 'org', 'repo', 10, 'open');
"""
)
# SQLite allows two NULL-scope rows with same provider/issue under UNIQUE.
n = conn.execute("SELECT COUNT(*) FROM incident_links").fetchone()[0]
self.assertEqual(n, 2)
conn.commit()
finally:
conn.close()
db = ControlPlaneDB(path)
conn2 = sqlite3.connect(path)
try:
n2 = conn2.execute("SELECT COUNT(*) FROM incident_links").fetchone()[0]
rows = conn2.execute(
"SELECT provider_base_url, provider_org, provider_project, gitea_issue_number "
"FROM incident_links"
).fetchall()
finally:
conn2.close()
self.assertEqual(n2, 1)
self.assertEqual(rows[0][0], "")
self.assertEqual(rows[0][1], "")
self.assertEqual(rows[0][2], "")
self.assertEqual(rows[0][3], 10)
# Touch to silence unused import in type checkers if needed
self.assertTrue(issubclass(ControlPlaneError, Exception))
del db
def test_migrate_conflicting_duplicate_incident_links_fails_closed(self) -> None:
"""Conflicting Gitea targets for the same provider key must fail closed."""
import sqlite3
from control_plane_db import ControlPlaneDB, ControlPlaneError
path = os.path.join(self._tmp.name, "legacy_conflict.sqlite3")
conn = sqlite3.connect(path)
try:
conn.executescript(
"""
CREATE TABLE incident_links (
link_id INTEGER PRIMARY KEY AUTOINCREMENT,
provider TEXT NOT NULL,
provider_base_url TEXT,
provider_org TEXT,
provider_project TEXT,
provider_issue_id TEXT NOT NULL,
gitea_org TEXT NOT NULL,
gitea_repo TEXT NOT NULL,
gitea_issue_number INTEGER NOT NULL,
status TEXT NOT NULL DEFAULT 'open',
UNIQUE (
provider, provider_base_url, provider_org,
provider_project, provider_issue_id
)
);
INSERT INTO incident_links(
provider, provider_base_url, provider_org, provider_project,
provider_issue_id, gitea_org, gitea_repo, gitea_issue_number
) VALUES
('sentry', NULL, NULL, NULL, 'dup-c', 'org', 'repo', 1),
('sentry', NULL, NULL, NULL, 'dup-c', 'org', 'repo', 2);
"""
)
conn.commit()
finally:
conn.close()
with self.assertRaises(ControlPlaneError) as ctx:
ControlPlaneDB(path)
self.assertIn("conflicting", str(ctx.exception).lower())
def test_migrate_conflicting_observation_metadata_fails_closed(self) -> None:
"""Same provider key + same Gitea target but differing obs metadata must fail closed.
Regression for silent data loss: migration used to keep lowest link_id and
delete peers after comparing only Gitea targets (#619 RC3).
"""
import sqlite3
from control_plane_db import ControlPlaneDB, ControlPlaneError
path = os.path.join(self._tmp.name, "legacy_meta_conflict.sqlite3")
conn = sqlite3.connect(path)
try:
conn.executescript(
"""
CREATE TABLE schema_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE incident_links (
link_id INTEGER PRIMARY KEY AUTOINCREMENT,
provider TEXT NOT NULL,
provider_base_url TEXT,
provider_org TEXT,
provider_project TEXT,
provider_issue_id TEXT NOT NULL,
provider_short_id TEXT,
provider_permalink TEXT,
fingerprint TEXT,
gitea_org TEXT NOT NULL,
gitea_repo TEXT NOT NULL,
gitea_issue_number INTEGER NOT NULL,
linked_pr_numbers TEXT,
first_seen TEXT,
last_seen TEXT,
event_count INTEGER,
status TEXT NOT NULL DEFAULT 'open',
release_resolved_at TEXT,
last_sync_at TEXT,
UNIQUE (
provider, provider_base_url, provider_org,
provider_project, provider_issue_id
)
);
INSERT INTO incident_links(
provider, provider_base_url, provider_org, provider_project,
provider_issue_id, provider_permalink, fingerprint,
gitea_org, gitea_repo, gitea_issue_number,
event_count, status, first_seen, last_seen
) VALUES
(
'sentry', NULL, NULL, NULL, 'dup-meta',
'https://sentry.example/issues/1', 'fingerprint-A',
'org', 'repo', 42,
1, 'open', '2026-01-01T00:00:00Z', '2026-01-01T01:00:00Z'
),
(
'sentry', NULL, NULL, NULL, 'dup-meta',
'https://sentry.example/issues/1', 'fingerprint-B',
'org', 'repo', 42,
99, 'resolved', '2026-01-01T00:00:00Z', '2026-01-02T00:00:00Z'
);
"""
)
n = conn.execute("SELECT COUNT(*) FROM incident_links").fetchone()[0]
self.assertEqual(n, 2)
conn.commit()
finally:
conn.close()
with self.assertRaises(ControlPlaneError) as ctx:
ControlPlaneDB(path)
msg = str(ctx.exception).lower()
self.assertIn("conflicting", msg)
self.assertIn("observation metadata", msg)
# Rows must still be present — migration must not delete before failing.
conn2 = sqlite3.connect(path)
try:
remaining = conn2.execute("SELECT COUNT(*) FROM incident_links").fetchone()[0]
fps = {
r[0]
for r in conn2.execute(
"SELECT fingerprint FROM incident_links ORDER BY link_id"
).fetchall()
}
finally:
conn2.close()
self.assertEqual(remaining, 2)
self.assertEqual(fps, {"fingerprint-A", "fingerprint-B"})
def test_incident_links_minimal_upsert_is_canonical(self) -> None:
"""Repeated minimal upserts must update one row (NULL-safe uniqueness)."""
a = self.db.upsert_incident_link(
provider="sentry",
provider_issue_id="inc-1",
gitea_org="org",
gitea_repo="repo",
gitea_issue_number=1,
)
b = self.db.upsert_incident_link(
provider="sentry",
provider_issue_id="inc-1",
gitea_org="org",
gitea_repo="repo",
gitea_issue_number=2,
# omit optional scope fields again
)
c = self.db.upsert_incident_link(
provider="sentry",
provider_issue_id="inc-1",
gitea_org="org",
gitea_repo="repo",
gitea_issue_number=3,
provider_base_url=None,
provider_org="",
provider_project=" ",
)
self.assertEqual(a["link_id"], b["link_id"])
self.assertEqual(b["link_id"], c["link_id"])
self.assertEqual(c["gitea_issue_number"], 3)
self.assertEqual(c["provider_base_url"], "")
self.assertEqual(c["provider_org"], "")
self.assertEqual(c["provider_project"], "")
import sqlite3
conn = sqlite3.connect(self.db_path)
try:
n = conn.execute(
"SELECT COUNT(*) FROM incident_links WHERE provider_issue_id = ?",
("inc-1",),
).fetchone()[0]
finally:
conn.close()
self.assertEqual(n, 1)
if __name__ == "__main__":
unittest.main()
+345
View File
@@ -0,0 +1,345 @@
"""Tests for observability incident bridge (#612) on #613 substrate."""
from __future__ import annotations
import json
import os
import tempfile
import unittest
from control_plane_db import ControlPlaneDB, InvalidWorkKindError, WORK_KINDS
from incident_bridge import (
OUTCOME_BLOCKED,
OUTCOME_CREATED,
OUTCOME_LINKED,
OUTCOME_PREVIEW,
OUTCOME_UPDATED,
ProjectMapping,
assert_not_raw_incident_work_item,
build_gitea_issue_body,
load_project_mappings,
normalize_incident,
project_mapping_from_dict,
reconcile_incident,
redact_text,
sanitize_tags,
)
def _mapping(**kwargs) -> ProjectMapping:
base = dict(
name="gitea-tools-mcp",
provider="sentry",
monitor_base_url="https://sentry.prgs.cc",
monitor_org="prgs",
monitor_project="gitea-tools-mcp",
gitea_org="Scaled-Tech-Consulting",
gitea_repo="Gitea-Tools",
default_labels=("type:bug", "observability", "sentry", "status:ready"),
)
base.update(kwargs)
return ProjectMapping(**base)
def _obs(**kwargs) -> dict:
base = dict(
provider="sentry",
provider_base_url="https://sentry.prgs.cc",
provider_org="prgs",
provider_project="gitea-tools-mcp",
provider_issue_id="ISSUE-100",
provider_short_id="GITEA-TOOLS-1A",
provider_permalink="https://sentry.prgs.cc/organizations/prgs/issues/100/",
fingerprint="fp-abc",
title="TypeError: boom",
summary="TypeError: boom in handle_request",
first_seen="2026-07-01T00:00:00Z",
last_seen="2026-07-10T00:00:00Z",
event_count=3,
environment="production",
severity="error",
culprit="handle_request",
tags={"runtime": "python", "release": "1.2.3"},
)
base.update(kwargs)
return base
class IncidentBridgeTest(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.db = ControlPlaneDB(os.path.join(self._tmp.name, "cp.sqlite3"))
self.mapping = _mapping()
self.created: list[dict] = []
def tearDown(self) -> None:
self._tmp.cleanup()
def _create_issue(self, title, body, labels, g_org, g_repo):
n = 9000 + len(self.created)
rec = {
"number": n,
"success": True,
"performed": True,
"title": title,
"body": body,
"labels": labels,
"org": g_org,
"repo": g_repo,
}
self.created.append(rec)
return rec
def test_redact_secrets(self) -> None:
dirty = "token=supersecret123 Authorization: Bearer abcdefghijklmnop"
clean = redact_text(dirty)
self.assertNotIn("supersecret", clean)
self.assertNotIn("abcdefghijklmnop", clean)
self.assertIn("[REDACTED]", clean)
tags = sanitize_tags(
{"token": "x", "runtime": "py", "password": "nope", "release": "1.0"}
)
self.assertEqual(tags, {"runtime": "py", "release": "1.0"})
def test_body_contains_no_secrets(self) -> None:
inc = normalize_incident(
_obs(
summary="fail password=hunter2 token: abc",
tags={"cookie": "sid=1", "runtime": "py"},
),
self.mapping,
)
body = build_gitea_issue_body(inc)
self.assertNotIn("hunter2", body)
self.assertNotIn("sid=1", body)
self.assertIn("provider_issue_id", body)
self.assertIn("not** assignable", body.lower())
def test_preview_no_mutation(self) -> None:
res = reconcile_incident(
self.db,
observation=_obs(),
mapping=self.mapping,
apply=False,
create_issue_fn=self._create_issue,
)
self.assertTrue(res["success"])
self.assertEqual(res["outcome"], OUTCOME_PREVIEW)
self.assertFalse(res["gitea_mutated"])
self.assertFalse(res["db_mutated"])
self.assertFalse(res["raw_incident_assignable"])
self.assertEqual(len(self.created), 0)
self.assertIsNone(
self.db.get_incident_link_by_provider(
provider="sentry",
provider_issue_id="ISSUE-100",
provider_base_url="https://sentry.prgs.cc",
provider_org="prgs",
provider_project="gitea-tools-mcp",
)
)
def test_apply_creates_issue_and_link(self) -> None:
res = reconcile_incident(
self.db,
observation=_obs(),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertTrue(res["success"])
self.assertEqual(res["outcome"], OUTCOME_CREATED)
self.assertTrue(res["gitea_mutated"])
self.assertTrue(res["db_mutated"])
self.assertEqual(len(self.created), 1)
self.assertEqual(res["gitea_issue"]["number"], self.created[0]["number"])
link = self.db.get_incident_link_by_provider(
provider="sentry",
provider_issue_id="ISSUE-100",
provider_base_url="https://sentry.prgs.cc",
provider_org="prgs",
provider_project="gitea-tools-mcp",
)
self.assertIsNotNone(link)
self.assertEqual(int(link["gitea_issue_number"]), self.created[0]["number"])
self.assertEqual(res["allocator_visible_as"]["kind"], "issue")
def test_duplicate_observation_reuses_issue(self) -> None:
first = reconcile_incident(
self.db,
observation=_obs(),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
second = reconcile_incident(
self.db,
observation=_obs(event_count=9, last_seen="2026-07-11T00:00:00Z"),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertEqual(first["outcome"], OUTCOME_CREATED)
self.assertEqual(second["outcome"], OUTCOME_UPDATED)
self.assertEqual(len(self.created), 1)
self.assertEqual(
second["gitea_issue"]["number"], first["gitea_issue"]["number"]
)
link = self.db.get_incident_link_by_provider(
provider="sentry",
provider_issue_id="ISSUE-100",
provider_base_url="https://sentry.prgs.cc",
provider_org="prgs",
provider_project="gitea-tools-mcp",
)
self.assertEqual(int(link["event_count"]), 9)
def test_explicit_link_existing_issue(self) -> None:
res = reconcile_incident(
self.db,
observation=_obs(provider_issue_id="ISSUE-200"),
mapping=self.mapping,
apply=True,
force_gitea_issue_number=555,
create_issue_fn=self._create_issue,
)
self.assertEqual(res["outcome"], OUTCOME_LINKED)
self.assertEqual(len(self.created), 0)
self.assertEqual(res["gitea_issue"]["number"], 555)
link = self.db.get_incident_link_for_gitea_issue(
gitea_org="Scaled-Tech-Consulting",
gitea_repo="Gitea-Tools",
gitea_issue_number=555,
)
self.assertIsNotNone(link)
def test_fingerprint_conflict_fails_closed(self) -> None:
reconcile_incident(
self.db,
observation=_obs(fingerprint="fp-a"),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
res = reconcile_incident(
self.db,
observation=_obs(fingerprint="fp-b"),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertEqual(res["outcome"], OUTCOME_BLOCKED)
self.assertIn("fingerprint conflict", res["reasons"][0].lower())
self.assertEqual(len(self.created), 1)
def test_ambiguous_mapping_fails_closed(self) -> None:
maps = [
_mapping(name="a", monitor_project="gitea-tools-mcp"),
_mapping(name="b", monitor_project="gitea-tools-mcp"),
]
res = reconcile_incident(
self.db,
observation=_obs(),
mappings=maps,
apply=False,
)
self.assertEqual(res["outcome"], OUTCOME_BLOCKED)
self.assertIn("ambiguous", res["reasons"][0].lower())
def test_missing_provider_issue_id_fails_closed(self) -> None:
res = reconcile_incident(
self.db,
observation=_obs(provider_issue_id=""),
mapping=self.mapping,
apply=False,
)
self.assertEqual(res["outcome"], OUTCOME_BLOCKED)
def test_raw_incident_not_work_kind(self) -> None:
self.assertNotIn("sentry_incident", WORK_KINDS)
with self.assertRaises(InvalidWorkKindError):
self.db.upsert_work_item(
remote="prgs",
org="o",
repo="r",
kind="sentry_incident",
number=1,
)
with self.assertRaises(Exception):
assert_not_raw_incident_work_item("glitchtip_incident")
def test_db_unavailable(self) -> None:
res = reconcile_incident(
None,
observation=_obs(),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertFalse(res["success"])
self.assertEqual(res["outcome"], OUTCOME_BLOCKED)
def test_structured_skip_reason_environment(self) -> None:
m = _mapping(environment_filters=("staging",))
res = reconcile_incident(
self.db,
observation=_obs(environment="production"),
mapping=m,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertTrue(res["success"])
self.assertEqual(res["action"], "skip_environment_filter")
self.assertEqual(len(self.created), 0)
def test_load_mappings_json(self) -> None:
payload = json.dumps(
{
"projects": [
{
"name": "gt",
"provider": "glitchtip",
"monitor_base_url": "https://glitchtip.example",
"monitor_org": "org",
"monitor_project": "proj",
"gitea_org": "Scaled-Tech-Consulting",
"gitea_repo": "Gitea-Tools",
}
]
}
)
maps = load_project_mappings(mappings_json=payload)
self.assertEqual(len(maps), 1)
self.assertEqual(maps[0].provider, "glitchtip")
def test_glitchtip_provider_accepted(self) -> None:
m = _mapping(provider="glitchtip", monitor_base_url="https://glitchtip.example")
res = reconcile_incident(
self.db,
observation=_obs(
provider="glitchtip",
provider_base_url="https://glitchtip.example",
),
mapping=m,
apply=True,
create_issue_fn=self._create_issue,
)
self.assertEqual(res["outcome"], OUTCOME_CREATED)
self.assertEqual(res["incident"]["provider"], "glitchtip")
def test_allocator_only_sees_issue_kind(self) -> None:
res = reconcile_incident(
self.db,
observation=_obs(),
mapping=self.mapping,
apply=True,
create_issue_fn=self._create_issue,
)
vis = res["allocator_visible_as"]
self.assertEqual(vis["kind"], "issue")
self.assertIn(vis["kind"], WORK_KINDS)
self.assertFalse(res["raw_incident_assignable"])
if __name__ == "__main__":
unittest.main()
+27 -7
View File
@@ -103,20 +103,40 @@ class TestIssueLockStore(unittest.TestCase):
self.assertIn("live foreign issue lock", block or "") self.assertIn("live foreign issue lock", block or "")
def test_expired_lease_allows_takeover_with_conflict_check(self): def test_expired_lease_allows_takeover_with_conflict_check(self):
# #601: expired + dead pid / missing worktree → sanctioned reclaim (no block).
existing = _lock_record( existing = _lock_record(
branch_name="feat/issue-420-other", branch_name="feat/issue-420-other",
worktree_path="/tmp/other", worktree_path="/tmp/other-does-not-exist-420",
session_pid=1,
work_lease=_lease("2000-01-01T00:00:00Z"), work_lease=_lease("2000-01-01T00:00:00Z"),
) )
incoming = _lock_record(worktree_path="/tmp/mine") incoming = _lock_record(worktree_path="/tmp/mine")
self.assertIsNone(ils.assess_foreign_lock_overwrite(existing, incoming)) self.assertIsNone(ils.assess_foreign_lock_overwrite(existing, incoming))
block = ils.assess_same_issue_lease_conflict( with mock.patch.object(ils, "is_process_alive", return_value=False):
existing, block = ils.assess_same_issue_lease_conflict(
issue_number=420, existing,
branch_name="feat/issue-420-server-code-parity", issue_number=420,
worktree_path="/tmp/mine", branch_name="feat/issue-420-server-code-parity",
worktree_path="/tmp/mine",
)
self.assertIsNone(block)
# Expired but still-live owner pid AND present worktree → fail closed.
present_wt = self.lock_dir # exists
sticky = _lock_record(
branch_name="feat/issue-420-other",
worktree_path=present_wt,
session_pid=os.getpid(),
work_lease=_lease("2000-01-01T00:00:00Z"),
) )
self.assertIn("Recovery review is required", block or "") with mock.patch.object(ils, "is_process_alive", return_value=True):
blocked = ils.assess_same_issue_lease_conflict(
sticky,
issue_number=420,
branch_name="feat/issue-420-server-code-parity",
worktree_path="/tmp/mine",
)
self.assertIn("Recovery review is required", blocked or "")
def test_same_owner_lease_conflict_allows_refresh(self): def test_same_owner_lease_conflict_allows_refresh(self):
worktree = "/tmp/wt-420" worktree = "/tmp/wt-420"
+392
View File
@@ -0,0 +1,392 @@
"""Tests for first-class control-plane lease lifecycle (#601)."""
from __future__ import annotations
import os
import tempfile
import unittest
from datetime import timedelta
from unittest import mock
from allocator_service import allocate_next_work, WorkCandidate
from control_plane_db import ControlPlaneDB, ForeignLeaseError, _ts, _utc_now
import lease_lifecycle as ll
import merger_lease_adoption as mla
import reviewer_pr_lease as rpl
from datetime import datetime, timezone
class LeaseLifecycleTest(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.db_path = os.path.join(self._tmp.name, "cp.sqlite3")
self.db = ControlPlaneDB(self.db_path)
self.db.upsert_session(session_id="owner", role="author", profile="prgs-author", pid=111)
self.db.upsert_session(session_id="other", role="author", profile="prgs-author", pid=222)
def tearDown(self) -> None:
self._tmp.cleanup()
def _assign(self, session_id="owner", number=601, **kwargs):
return self.db.assign_and_lease(
session_id=session_id,
role="author",
remote="prgs",
org="Scaled-Tech-Consulting",
repo="Gitea-Tools",
kind="issue",
number=number,
allowed_actions=("implement", "comment", "push", "create_pr"),
forbidden_actions=("approve", "merge", "self_select_without_assignment"),
worktree_path=kwargs.pop("worktree_path", self._tmp.name),
owner_pid=kwargs.pop("owner_pid", os.getpid()),
**kwargs,
)
def test_list_active_leases_as_workflow_state(self) -> None:
a = self._assign(number=601)
self._assign(session_id="other", number=602)
listed = ll.list_active_leases(
self.db, remote="prgs", org="Scaled-Tech-Consulting", repo="Gitea-Tools"
)
self.assertTrue(listed["success"])
self.assertEqual(listed["count"], 2)
self.assertEqual(listed["authoritative_source"], "control_plane_db")
self.assertFalse(listed["file_lock_only"])
self.assertFalse(listed["comment_lease_only"])
numbers = sorted(x["work_number"] for x in listed["leases"])
self.assertEqual(numbers, [601, 602])
self.assertIsNotNone(a.lease_id)
def test_adopt_lease_owner_resume_preserves_provenance(self) -> None:
a = self._assign()
res = ll.adopt_lease(
self.db,
lease_id=a.lease_id,
adopter_session_id="owner",
role="author",
worktree_path=self._tmp.name,
)
self.assertTrue(res["success"])
self.assertTrue(res["same_owner"])
prov = res["provenance"]
self.assertEqual(prov["adopted_from_session_id"], "owner")
self.assertEqual(prov["adopted_by_session_id"], "owner")
self.assertEqual(prov["work_number"], 601)
self.assertEqual(prov["worktree_path"], self._tmp.name)
state = self.db.get_lease_workflow_state(a.lease_id)
self.assertIsNotNone(state)
self.assertEqual(state["lease"]["status"], "active")
def test_release_lease_explicit_recorded(self) -> None:
a = self._assign()
res = ll.release_lease(self.db, lease_id=a.lease_id, session_id="owner")
self.assertEqual(res["outcome"], "released")
self.assertIn("release_proof", res)
self.assertEqual(res["release_proof"]["status"], "released")
state = self.db.get_lease_workflow_state(a.lease_id)
self.assertEqual(state["lease"]["status"], "released")
def test_expire_and_reclaim_expired_lease(self) -> None:
a = self._assign(lease_ttl_seconds=1)
past = _utc_now() - timedelta(hours=2)
import sqlite3
conn = sqlite3.connect(self.db_path)
try:
conn.execute(
"UPDATE leases SET expires_at = ? WHERE lease_id = ?",
(_ts(past), a.lease_id),
)
conn.commit()
finally:
conn.close()
exp = ll.expire_leases(self.db)
self.assertGreaterEqual(exp["expired_count"], 1)
reclaimed = ll.reclaim_expired_lease(
self.db,
lease_id=a.lease_id,
session_id="other",
role="author",
worktree_path=self._tmp.name,
)
self.assertEqual(reclaimed["outcome"], "reclaimed")
self.assertEqual(reclaimed["assignment"]["session_id"], "other")
self.assertEqual(
reclaimed["provenance"]["adopted_from_session_id"], "owner"
)
self.assertEqual(
reclaimed["provenance"]["adopted_by_session_id"], "other"
)
def test_abandon_stale_lease_with_required_proof(self) -> None:
a = self._assign(owner_pid=99999999, worktree_path="/nonexistent/path/for-601")
# Force active but dead pid + missing worktree
proof = ll.AbandonProof(
dead_process=True,
missing_worktree=True,
no_open_pr=True,
no_live_mutation_risk=True,
owner_pid=99999999,
worktree_path="/nonexistent/path/for-601",
)
res = ll.abandon_lease(
self.db,
lease_id=a.lease_id,
requester_session_id="other",
proof=proof,
)
self.assertEqual(res["outcome"], "abandoned")
self.assertEqual(res["prior_owner_session_id"], "owner")
self.assertTrue(res["abandon_proof"]["dead_process"])
state = self.db.get_lease_workflow_state(a.lease_id)
self.assertEqual(state["lease"]["status"], "abandoned")
def test_refuse_steal_active_foreign_lease(self) -> None:
a = self._assign()
with self.assertRaises(ll.LeaseLifecycleError) as ctx:
ll.adopt_lease(
self.db,
lease_id=a.lease_id,
adopter_session_id="other",
role="author",
)
self.assertIn("steal", str(ctx.exception).lower())
decision = ll.inspect_lease(
self.db, a.lease_id, caller_session_id="other"
)
self.assertEqual(decision["safe_next_action"], ll.SAFE_WAIT_FOREIGN)
self.assertTrue(decision["block"])
def test_refuse_ambiguous_lease_ownership_stale_id(self) -> None:
decision = ll.inspect_lease(
self.db, "lease-does-not-exist", caller_session_id="owner"
)
self.assertFalse(decision["found"])
self.assertEqual(decision["safe_next_action"], ll.SAFE_STALE_PROMPT)
with self.assertRaises(ll.LeaseLifecycleError):
ll.adopt_lease(
self.db,
lease_id="lease-does-not-exist",
adopter_session_id="owner",
role="author",
)
def test_provenance_on_adopt_release_abandon(self) -> None:
a = self._assign()
adopted = ll.adopt_lease(
self.db,
lease_id=a.lease_id,
adopter_session_id="owner",
role="author",
worktree_path=self._tmp.name,
expected_head_sha="abc",
)
self.assertIn("adopted_from_session_id", adopted["provenance"])
self.assertIn("adopted_by_session_id", adopted["provenance"])
released = ll.release_lease(
self.db, lease_id=a.lease_id, session_id="owner"
)
self.assertEqual(released["release_proof"]["session_id"], "owner")
b = self._assign(number=700, owner_pid=1, worktree_path="/no/such/wt")
abandoned = ll.abandon_lease(
self.db,
lease_id=b.lease_id,
requester_session_id="owner",
proof=ll.AbandonProof(
dead_process=True,
missing_worktree=True,
no_live_mutation_risk=True,
no_open_pr=True,
),
)
self.assertIn("abandon_proof", abandoned)
self.assertEqual(abandoned["abandon_proof"]["dead_process"], True)
def test_allocator_assignment_lease_compatible(self) -> None:
"""Allocator assign+lease still works and is listable/inspectable."""
cands = [
WorkCandidate(
kind="issue",
number=601,
labels=("status:ready",),
title="leases",
priority=20,
)
]
res = allocate_next_work(
self.db,
session_id="alloc-s",
role="author",
remote="prgs",
org="Scaled-Tech-Consulting",
repo="Gitea-Tools",
candidates=cands,
apply=True,
profile_name="prgs-author",
username="jcwalker3",
)
self.assertEqual(res["outcome"], "assigned_work")
lid = res["assignment"]["lease_id"]
listed = ll.list_active_leases(self.db, remote="prgs")
ids = [x["lease_id"] for x in listed["leases"]]
self.assertIn(lid, ids)
insp = ll.inspect_lease(self.db, lid, caller_session_id="alloc-s")
self.assertTrue(insp["found"])
self.assertIn(
insp["safe_next_action"],
(ll.SAFE_OWNER_RESUME, ll.SAFE_ABANDON_ALLOWED),
)
def test_reviewer_merger_lease_handoff_still_works(self) -> None:
"""#536 merger adoption path is independent and must not regress."""
rpl.clear_session_lease()
body = mla.format_adoption_body(
repo="Scaled-Tech-Consulting/Gitea-Tools",
pr_number=999,
issue_number=601,
adopter_identity="sysadmin",
adopter_profile="prgs-merger",
adopter_session_id="merger-1",
worktree="branches/merge-pr999",
candidate_head="a" * 40,
target_branch="master",
target_branch_sha="b" * 40,
adopted_from_session_id="reviewer-1",
adopted_from_profile="prgs-reviewer",
adopted_from_reviewer_identity="sysadmin",
adopted_from_comment_id=42,
)
self.assertIn(mla.ADOPTION_MARKER, body)
self.assertIn("adopted_from_session_id: reviewer-1", body)
self.assertIn("adopted_by_profile: prgs-merger", body)
prov = mla.build_lease_provenance(
source=mla.SOURCE_ADOPT,
comment_id=42,
adopted_from_session_id="reviewer-1",
adoption_reason=mla.DEFAULT_ADOPTION_REASON,
)
rpl.record_session_lease(
{
"pr_number": 999,
"session_id": "merger-1",
"comment_id": 42,
},
lease_provenance=prov,
)
proof = mla.describe_session_lease_proof(rpl.get_session_lease())
self.assertTrue(proof["lease_proof_sanctioned"])
self.assertEqual(proof["lease_proof_source"], mla.SOURCE_ADOPT)
rpl.clear_session_lease()
def test_missing_worktree_and_dead_pid_handled_safely(self) -> None:
a = self._assign(owner_pid=1, worktree_path="/definitely/missing/wt-601")
with mock.patch.object(ll, "is_process_alive", return_value=False):
fr = ll.classify_lease_freshness(
self.db.get_lease_workflow_state(a.lease_id)["lease"]
)
self.assertIn(fr["freshness"], ("stale_dead_process", "stale_missing_worktree"))
# Insufficient abandon proof fails closed
with self.assertRaises(ll.LeaseLifecycleError):
ll.abandon_lease(
self.db,
lease_id=a.lease_id,
requester_session_id="other",
proof=ll.AbandonProof(dead_process=True), # missing no_live_mutation_risk
)
def test_file_lock_or_comment_not_authoritative_alone(self) -> None:
report = ll.non_db_lease_authority_report(
file_lock_present=True,
comment_lease_present=False,
db_lease_present=False,
)
self.assertEqual(report["safe_next_action"], ll.SAFE_NO_AUTHORITY)
self.assertTrue(report["file_lock_only"])
self.assertIsNone(report["authoritative_source"])
report2 = ll.non_db_lease_authority_report(
file_lock_present=True,
comment_lease_present=True,
db_lease_present=True,
)
self.assertEqual(report2["authoritative_source"], "control_plane_db")
self.assertFalse(report2["file_lock_only"])
self.assertFalse(report2["comment_lease_only"])
def test_abandon_proof_is_sufficient_rules(self) -> None:
self.assertFalse(ll.AbandonProof(dead_process=True).is_sufficient())
self.assertTrue(
ll.AbandonProof(
dead_process=True,
missing_worktree=True,
no_live_mutation_risk=True,
no_open_pr=True,
).is_sufficient()
)
self.assertTrue(
ll.AbandonProof(
dead_process=True,
no_live_mutation_risk=True,
same_owner=True,
).is_sufficient()
)
self.assertTrue(
ll.AbandonProof(
missing_worktree=True,
no_live_mutation_risk=True,
operator_authorized=True,
).is_sufficient()
)
class IssueLockExpiredReclaimTest(unittest.TestCase):
def test_expired_lock_reclaim_allowed_when_dead_and_missing_wt(self) -> None:
import issue_lock_store as ils
lock = {
"issue_number": 601,
"branch_name": "feat/issue-601-x",
"worktree_path": "/no/such/worktree-601",
"session_pid": 1,
"work_lease": {
"operation_type": "author_issue_work",
"expires_at": "2000-01-01T00:00:00Z",
},
}
with mock.patch.object(ils, "is_process_alive", return_value=False):
res = ils.assess_expired_lock_reclaim(lock)
self.assertTrue(res["reclaim_allowed"])
# Conflict assessor allows takeover
block = ils.assess_same_issue_lease_conflict(
lock,
issue_number=601,
branch_name="feat/issue-601-first-class-leases",
worktree_path="/tmp/new",
)
self.assertIsNone(block)
def test_live_lock_reclaim_refused(self) -> None:
import issue_lock_store as ils
from datetime import datetime, timedelta, timezone
future = (datetime.now(timezone.utc) + timedelta(hours=2)).strftime(
"%Y-%m-%dT%H:%M:%SZ"
)
lock = {
"issue_number": 601,
"branch_name": "feat/issue-601-x",
"worktree_path": tempfile.gettempdir(),
"session_pid": os.getpid(),
"work_lease": {
"operation_type": "author_issue_work",
"expires_at": future,
"last_heartbeat_at": future,
},
}
res = ils.assess_expired_lock_reclaim(lock)
self.assertFalse(res["reclaim_allowed"])
if __name__ == "__main__":
unittest.main()
+19 -3
View File
@@ -5,6 +5,7 @@ the MCP protocol) with mocked API responses.
""" """
import json import json
import os import os
import shutil
import sys import sys
import tempfile import tempfile
import unittest import unittest
@@ -3811,7 +3812,15 @@ class TestIssueLocking(unittest.TestCase):
@patch("mcp_server.api_get_all", return_value=[]) @patch("mcp_server.api_get_all", return_value=[])
@patch("mcp_server.get_auth_header", return_value=FAKE_AUTH) @patch("mcp_server.get_auth_header", return_value=FAKE_AUTH)
def test_lock_issue_blocks_expired_same_operation_lease_for_recovery(self, _auth, _api, _git_state): def test_lock_issue_blocks_expired_same_operation_lease_for_recovery(self, _auth, _api, _git_state):
"""#601: expired lease still blocks takeover when owner pid is live AND worktree exists.
Dead-pid / missing-worktree reclaim is covered by issue_lock_store unit tests.
This MCP path must remain fail-closed for sticky expired foreign ownership.
"""
prgs_repo = mcp_server.REMOTES["prgs"]["repo"] prgs_repo = mcp_server.REMOTES["prgs"]["repo"]
# Present worktree + live pid => reclaim_allowed=False even though expires_at is past.
sticky_worktree = tempfile.mkdtemp(prefix="gitea-sticky-lease-")
self.addCleanup(lambda: shutil.rmtree(sticky_worktree, ignore_errors=True))
issue_lock_store.save_lock_file( issue_lock_store.save_lock_file(
issue_lock_store.lock_file_path( issue_lock_store.lock_file_path(
remote="prgs", remote="prgs",
@@ -3825,15 +3834,22 @@ class TestIssueLocking(unittest.TestCase):
"remote": "prgs", "remote": "prgs",
"org": "Scaled-Tech-Consulting", "org": "Scaled-Tech-Consulting",
"repo": prgs_repo, "repo": prgs_repo,
"worktree_path": "/tmp/other-worktree", "worktree_path": sticky_worktree,
"session_pid": os.getpid(),
"pid": os.getpid(),
"work_lease": { "work_lease": {
"operation_type": "author_issue_work", "operation_type": "author_issue_work",
"expires_at": "2000-01-01T00:00:00Z", "expires_at": "2000-01-01T00:00:00Z",
}, },
}, },
) )
with self.assertRaises(RuntimeError) as ctx: with patch.object(issue_lock_store, "is_process_alive", return_value=True):
gitea_lock_issue(issue_number=196, branch_name="feat/issue-196-mutations", remote="prgs") with self.assertRaises(RuntimeError) as ctx:
gitea_lock_issue(
issue_number=196,
branch_name="feat/issue-196-mutations",
remote="prgs",
)
self.assertIn("Recovery review is required before takeover", str(ctx.exception)) self.assertIn("Recovery review is required before takeover", str(ctx.exception))
@patch("mcp_server.api_get_all", return_value=[]) @patch("mcp_server.api_get_all", return_value=[])