diff --git a/docs/architecture/mcp-allocator-control-plane-observability-adr.md b/docs/architecture/mcp-allocator-control-plane-observability-adr.md new file mode 100644 index 0000000..1ea008f --- /dev/null +++ b/docs/architecture/mcp-allocator-control-plane-observability-adr.md @@ -0,0 +1,301 @@ +# ADR: MCP allocator, control-plane DB, and Sentry/GlitchTip incident bridge architecture + +- **Status:** Accepted (implementation pending; blocks code for #600 / #612 / #613) +- **Date:** 2026-07-09 +- **Tracking issues:** + - [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) — control-plane DB (first) + - [#600](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/600) — allocator API (second) + - [#612](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/612) — Sentry/GlitchTip incident bridge (third) +- **Related:** existing GlitchTip contracts (`glitchtip-to-gitea-workflow-design.md`, `glitchtip-gitea-deduplication-linking-design.md`), trust boundaries (`tool-boundaries.md`, `safety-model.md`), current leases (`pr_work_lease.py`, `issue_lock_store.py`) + +## 1. Context + +Multiple LLM sessions can independently inspect the Gitea queue and start the same issue or PR. Existing coordination (Gitea comment leases, local issue-lock files, labels) often detects collisions **after** work has started. Parallel issues #600, #612, and #613 each describe part of a fix; without one ADR they risk overlapping stores, wrong dependency order, and trust-boundary violations. + +This ADR is the canonical architecture decision **before** implementing those issues. + +## 2. Decision summary (core) + +| Layer | Owns | Must not | +|-------|------|----------| +| **Control-plane DB** | Sessions, atomic assignment, leases, heartbeats, terminal-lock index, events, incident links | Bypass Gitea workflow gates or replace issue/PR history | +| **Gitea** | Durable work record: issues, PRs, comments, labels, reviews, merges | Be the only concurrency lock under multi-session load | +| **Sentry / GlitchTip** | Incidents, events, provider UI | Assign work, approve/merge/close, or mutate Gitea outside the bridge | +| **Incident bridge** | Provider adapters, reconcile/create Gitea issues, link storage upsert, optional provider writeback | Hand raw provider incidents to the allocator as work items | + +**One-liner:** **DB coordinates. Gitea records. Sentry/GlitchTip observe. The bridge is the only path that turns observations into Gitea work. The allocator assigns only Gitea issues/PRs, never raw monitoring incidents.** + +## 3. Dependency order + +Implementation **must** follow: + +```text +#613 control-plane DB → #600 allocator API → #612 incident bridge + (atomic substrate) (routing policy) (feeds Gitea work) +``` + +| Issue | Role | Depends on | +|-------|------|------------| +| **#613** | Durable coordination DB + lease/assignment transactions | — | +| **#600** | `gitea_allocate_next_work` policy and tool surface | **#613** (hard) | +| **#612** | Multi-project Sentry/GlitchTip → Gitea bridge | **#613** for `incident_links` index; **#600** before treating bridge-created issues as allocator feed in multi-worker prod | + +**Hard rules:** + +1. Do **not** implement #600 on file locks / comment-only leases and call it done. +2. Do **not** ship #612 as a second assignment system for raw incidents. +3. Partial previews (read-only list tools, schema stubs) may land earlier if they do not claim “allocator complete” or “bridge complete.” + +## 4. Authority boundaries + +### 4.1 Gitea (durable source of truth for work) + +Gitea remains authoritative for: + +- Issue and PR identity, titles, bodies, state (open/closed/merged) +- Comments, reviews, approvals, REQUEST_CHANGES +- Labels and workflow status labels (`status:ready`, `status:in-progress`, …) +- Merges, closes, branch refs as recorded by Gitea/git hosting + +Operators and auditors read **history** from Gitea. + +### 4.2 Control-plane DB (coordination / index) + +The control-plane DB is authoritative for **live multi-session coordination**: + +- Which session holds which assignment/lease +- Heartbeat freshness and expiry +- Terminal-lock index for routing +- Event log of allocation/lease transitions +- `incident_links` index (see §8) + +It is **not** a substitute for Gitea history. When DB and Gitea disagree on durable work state (e.g. PR already merged), **Gitea wins**; the DB is reconciled. + +### 4.3 Sentry / GlitchTip (observe) + +Providers own incident lifecycle and raw event data. They: + +- Must **not** bypass Gitea gates +- Must **not** assign LLM work +- May receive **writeback** of resolution status only through the bridge, when configured and supported + +### 4.4 Trust boundaries for credentials + +Aligned with `docs/tool-boundaries.md` and `docs/safety-model.md`: + +- **One MCP server process per trust boundary** remains the default. +- Monitor API tokens (Sentry/GlitchTip) live in the **bridge/orchestrator boundary** (or a dedicated observability write/read profile), **not** blindly inside every Gitea MCP author/reviewer/merger process. +- Gitea tokens stay on Gitea MCP profiles only. +- Orchestrators compose services; they must not become a single credential pool that silently mixes Gitea write + monitor tokens into every worker. + +Tool names such as `gitea_observability_*` may exist as a **namespace façade** only if the runtime still enforces separate credential scopes (e.g. bridge process vs pure Gitea mutation process). + +## 5. Allocator rules (#600 on top of #613) + +### 5.1 Worker contract + +- Workers **do not self-select** work under the standard multi-LLM workflow. +- Workers call **`gitea_allocate_next_work`** (with `apply=true` only when taking work). +- Mutations that claim exclusive work require a **valid assignment and lease** from the control-plane DB. +- Return values include at least: role, issue/PR number, expected head SHA (when applicable), lease ID, expiry, allowed actions, forbidden actions — or a non-assignment outcome (`WAIT`, terminal-path block, no safe work, needs controller, etc.). + +### 5.2 Atomic reserve + +- **Assignment creation and lease creation happen in one DB transaction.** +- Two concurrent sessions **must not** receive the same issue/PR as assigned work. +- On contention: second session gets **WAIT** or **owner-resume**, never a duplicate lease. + +### 5.3 Routing policy + +| Condition | Route | +|-----------|--------| +| Current-head **REQUEST_CHANGES** | **Author** (not reviewer/merger) | +| **Stale** approval (approval not on current head) | **Reviewer** | +| **Clean** approval on current head, mergeable | **Merger** | +| Contested / contaminated approval | **Diagnosis / reconciler** (controller path) | +| Active **foreign** lease | **WAIT** or **owner-resume** | +| **Terminal-review** lock present | **Terminal-path resolution first** before downstream review work | +| Merged / closed PR | **Never assign** | +| Issue locked by sanctioned lease | **Never assign** to another worker unless lease cleanup/adoption is sanctioned | + +### 5.4 Relationship to Gitea mirrors + +After a successful assignment, the allocator (or sanctioned tooling) may update Gitea comments/labels so humans and audits see state. Those mirrors **are not** the sole lock source. + +## 6. Control-plane DB topology (#613) + +### 6.1 Preferred architecture (multi-daemon / Proxmox) + +For true multi-session concurrency (multiple MCP daemons, multiple hosts, or Proxmox VMs): + +**Prefer either:** + +1. **Shared Postgres** used by all Gitea MCP profiles / allocator callers, **or** +2. A **single allocator daemon** (sole writer to the coordination store) that all workers call over MCP/RPC. + +Both satisfy: one transactional authority for “who has this work item.” + +### 6.2 SQLite MVP + +SQLite is acceptable **only** for a **single-writer MVP**: + +- One process owns writes (allocator daemon or single co-located MCP server), **or** +- Documented single-host single-daemon experiments with file locking and no multi-host claims. + +SQLite is **not** sufficient to declare multi-session production readiness when four independent LLM sessions talk to four MCP processes without a shared writer. + +### 6.3 Suggested entities (normative shape) + +Minimum logical entities (names may vary; semantics must not): + +1. **sessions** — session_id, role/profile, namespace, pid, started_at, last_heartbeat_at, status +2. **work_items** — remote/org/repo, kind ∈ {`issue`, `pr`}, number, state, priority, current_head_sha, updated_at +3. **leases** — lease_id, work_item_id, session_id, role, phase, expires_at, heartbeat_at, status +4. **assignments** — assignment_id, work_item_id, session_id, allowed_action(s), expected_head_sha, status +5. **terminal_locks** — repo/org, terminal_pr, review_id, decision, status, cleanup_state +6. **events** — event_id, work_item_id, type, message, created_at +7. **incident_links** — provider-neutral link model (see §8) + +**Explicit non-kind:** do **not** use `work_items.kind = sentry_incident` (or glitchtip_incident) as an assignable work unit. Incidents become work only after the bridge creates/links a **Gitea issue**. + +## 7. Lease migration (avoid split-brain) + +### 7.1 Target state + +- **Primary** for live coordination: **control-plane DB** leases and assignments. +- **Gitea comment leases** (e.g. ``) and **local issue-lock files** become: + - **mirrors** of DB state for human visibility / recovery, and/or + - written **only** through the allocator (or sanctioned lease tools that update DB + mirror in one workflow). + +### 7.2 Migration rules + +1. New exclusive claims go through DB assignment+lease first. +2. Comment lease / file lock writers that skip the DB are **deprecated** once #613+#600 land. +3. Reconciler tooling heals: expired DB lease, orphan Gitea comment, orphan local file — fail closed when ambiguous. +4. **Split-brain is a defect:** “DB free + Gitea comment leased” or “DB leased + Gitea free” must be detectable and reconcilable; production paths must not rely on two independent writers. + +### 7.3 Heartbeats + +- Heartbeats update the **DB**. +- Optional Gitea summaries must avoid comment spam (periodic summary or edit-in-place policy). + +## 8. Observability bridge (#612) + +### 8.1 Role + +The bridge: + +1. Scans configured Sentry and/or GlitchTip projects (multi-project mappings). +2. Deduplicates against existing links / Gitea issues. +3. Creates or updates **normal Gitea issues** (labels, sanitized body, provider URL). +4. Upserts **one** canonical `incident_links` row. +5. Optionally writes resolution status back to the provider when supported. +6. Never approves, merges, closes, or otherwise bypasses Gitea workflow gates. + +### 8.2 Allocator interaction + +- The allocator sees observability work **only after** a Gitea issue exists (typically `status:ready` + observability labels). +- **Do not assign raw Sentry/GlitchTip incidents** as work items. +- Bridge AC “allocator can see linked issues” means: **as ordinary Gitea issues**, not a special incident queue. + +### 8.3 Provider adapters and trust + +- Separate **Sentry** and **GlitchTip** adapters; document API differences; do not assume writeback parity. +- Self-hosted base URLs supported (e.g. `https://sentry.prgs.cc`). +- Tokens from environment / secret store only. +- Prefer phase-1 **explicit reconcile / dry-run** before unsupervised watchdog auto-filing, consistent with existing GlitchTip filing safety contracts. + +### 8.4 Home of implementation + +Filing/orchestration may live in: + +- a control-plane / bridge package or profile, and/or +- Gitea-Tools as operator-facing tools that **delegate** to that boundary, + +but must not violate §4.4 credential isolation. + +## 9. Incident link storage (single source of truth) + +### 9.1 Canonical model: `incident_links` (provider-neutral) + +Prefer a **provider-neutral** model over Sentry-only `sentry_links`: + +| Field (logical) | Purpose | +|-----------------|---------| +| provider | `sentry` \| `glitchtip` \| … | +| provider_base_url | Self-hosted base | +| provider_org / provider_project | Mapping key | +| provider_issue_id / provider_short_id | Provider identity | +| provider_permalink | Human URL | +| fingerprint | Stable dedupe key when available | +| gitea_org / gitea_repo / gitea_issue_number | Linked work | +| linked_pr_numbers | Optional PR association | +| first_seen / last_seen / event_count | Summary metrics (safe) | +| status | link lifecycle | +| release_resolved_at / last_sync_at | Sync bookkeeping | + +### 9.2 Gitea as human mirror + +- Issue body markers / structured comments (e.g. HTML comment metadata) remain the **human- and audit-visible mirror**. +- They must stay consistent with `incident_links` but are **not** a second independent write path for inventing links without the bridge. + +### 9.3 One truth + +- **Do not** maintain two independent systems of record (e.g. full mapping only in #612 storage **and** a separate #613 `sentry_links` with different semantics). +- #612 implementation **must** use the same `incident_links` model introduced under #613 (or a single agreed store both reference). + +## 10. Security and redaction + +Mandatory for bridge payloads, Gitea issue bodies/comments, DB-stored event summaries, and logs: + +- No API tokens, DSNs, passwords, cookies, auth headers +- No keychain IDs, private config contents, raw session-state files, full prompt bodies +- Sanitize stack locals and request data; store only safe tags/context +- Logs must never print provider tokens or DSNs +- Redaction tests are required for #612 + +## 11. Non-goals + +- Replace Gitea as the durable workflow record +- Let Sentry/GlitchTip assign work or mutate Gitea workflow state outside sanctioned tools +- Let the bridge approve, merge, close, release, or bypass Gitea gates +- Implement #600 on file locks / comments alone and call allocator complete +- Treat raw monitoring incidents as `work_items` for the allocator +- Put monitor tokens into every Gitea MCP worker process by default + +## 12. Consequences + +### Positive + +- Clear implementation order and ownership +- Multi-session safety becomes testable at the DB layer +- Observability becomes assignable work without special-casing the allocator +- Trust boundaries stay compatible with existing control-plane docs + +### Costs / risks + +- Migration from comment/file leases requires reconciler work +- Shared Postgres or single allocator daemon is operational cost +- Bridge must be careful not to spam Gitea issues (dedupe, caps, policy modes) + +### Follow-ups (documentation only until implemented) + +1. Update #600, #612, and #613 bodies to **link this ADR** and restate hard dependencies. +2. Implement #613 schema + atomic assign/lease. +3. Implement #600 against the DB. +4. Implement #612 against `incident_links` + Gitea create/update. +5. Deprecate dual writers for leases. + +## 13. Acceptance criteria for *this* ADR + +This document is accepted when: + +1. It is merged into `docs/architecture/` on the default branch. +2. #600 / #612 / #613 (or their PRs) reference this path as the architecture source of truth. +3. No implementation PR for those issues claims completion without conforming to §§2–11. + +## 14. Document history + +| Date | Change | +|------|--------| +| 2026-07-09 | Initial ADR: dependency order, authority model, topology, lease migration, bridge, `incident_links`, security, non-goals |