Gitea-Tools/docs/architecture/mcp-allocator-control-plane-observability-adr.md

# ADR: MCP allocator, control-plane DB, and Sentry/GlitchTip incident bridge architecture

- **Status:** Accepted (implementation pending; blocks code for #600 / #612 / #613)
- **Date:** 2026-07-09
- **Tracking issues:**
  - [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) — control-plane DB (first)
  - [#600](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/600) — allocator API (second)
  - [#612](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/612) — Sentry/GlitchTip incident bridge (third)
- **Related:** existing GlitchTip contracts (`glitchtip-to-gitea-workflow-design.md`, `glitchtip-gitea-deduplication-linking-design.md`), trust boundaries (`tool-boundaries.md`, `safety-model.md`), current leases (`pr_work_lease.py`, `issue_lock_store.py`)

## 1. Context

Multiple LLM sessions can independently inspect the Gitea queue and start the same issue or PR. Existing coordination (Gitea comment leases, local issue-lock files, labels) often detects collisions **after** work has started. Parallel issues #600, #612, and #613 each describe part of a fix; without one ADR they risk overlapping stores, wrong dependency order, and trust-boundary violations.

This ADR is the canonical architecture decision **before** implementing those issues.

## 2. Decision summary (core)

| Layer | Owns | Must not |
|-------|------|----------|
| **Control-plane DB** | Sessions, atomic assignment, leases, heartbeats, terminal-lock index, events, incident links | Bypass Gitea workflow gates or replace issue/PR history |
| **Gitea** | Durable work record: issues, PRs, comments, labels, reviews, merges | Be the only concurrency lock under multi-session load |
| **Sentry / GlitchTip** | Incidents, events, provider UI | Assign work, approve/merge/close, or mutate Gitea outside the bridge |
| **Incident bridge** | Provider adapters, reconcile/create Gitea issues, link storage upsert, optional provider writeback | Hand raw provider incidents to the allocator as work items |

**One-liner:** **DB coordinates. Gitea records. Sentry/GlitchTip observe. The bridge is the only path that turns observations into Gitea work. The allocator assigns only Gitea issues/PRs, never raw monitoring incidents.**

## 3. Dependency order

Implementation **must** follow:

```text
#613 control-plane DB  →  #600 allocator API  →  #612 incident bridge
   (atomic substrate)      (routing policy)       (feeds Gitea work)
```

| Issue | Role | Depends on |
|-------|------|------------|
| **#613** | Durable coordination DB + lease/assignment transactions | — |
| **#600** | `gitea_allocate_next_work` policy and tool surface | **#613** (hard) |
| **#612** | Multi-project Sentry/GlitchTip → Gitea bridge | **#613** for `incident_links` index; **#600** before treating bridge-created issues as allocator feed in multi-worker prod |

**Hard rules:**

1. Do **not** implement #600 on file locks / comment-only leases and call it done.
2. Do **not** ship #612 as a second assignment system for raw incidents.
3. Partial previews (read-only list tools, schema stubs) may land earlier if they do not claim “allocator complete” or “bridge complete.”

## 4. Authority boundaries

### 4.1 Gitea (durable source of truth for work)

Gitea remains authoritative for:

- Issue and PR identity, titles, bodies, state (open/closed/merged)
- Comments, reviews, approvals, REQUEST_CHANGES
- Labels and workflow status labels (`status:ready`, `status:in-progress`, …)
- Merges, closes, branch refs as recorded by Gitea/git hosting

Operators and auditors read **history** from Gitea.

### 4.2 Control-plane DB (coordination / index)

The control-plane DB is authoritative for **live multi-session coordination**:

- Which session holds which assignment/lease
- Heartbeat freshness and expiry
- Terminal-lock index for routing
- Event log of allocation/lease transitions
- `incident_links` index (see §8)

It is **not** a substitute for Gitea history. When DB and Gitea disagree on durable work state (e.g. PR already merged), **Gitea wins**; the DB is reconciled.

### 4.3 Sentry / GlitchTip (observe)

Providers own incident lifecycle and raw event data. They:

- Must **not** bypass Gitea gates
- Must **not** assign LLM work
- May receive **writeback** of resolution status only through the bridge, when configured and supported

### 4.4 Trust boundaries for credentials

Aligned with `docs/tool-boundaries.md` and `docs/safety-model.md`:

- **One MCP server process per trust boundary** remains the default.
- Monitor API tokens (Sentry/GlitchTip) live in the **bridge/orchestrator boundary** (or a dedicated observability write/read profile), **not** blindly inside every Gitea MCP author/reviewer/merger process.
- Gitea tokens stay on Gitea MCP profiles only.
- Orchestrators compose services; they must not become a single credential pool that silently mixes Gitea write + monitor tokens into every worker.

Tool names such as `gitea_observability_*` may exist as a **namespace façade** only if the runtime still enforces separate credential scopes (e.g. bridge process vs pure Gitea mutation process).

## 5. Allocator rules (#600 on top of #613)

### 5.1 Worker contract

- Workers **do not self-select** work under the standard multi-LLM workflow.
- Workers call **`gitea_allocate_next_work`** (with `apply=true` only when taking work).
- Mutations that claim exclusive work require a **valid assignment and lease** from the control-plane DB.
- Return values include at least: role, issue/PR number, expected head SHA (when applicable), lease ID, expiry, allowed actions, forbidden actions — or a non-assignment outcome (`WAIT`, terminal-path block, no safe work, needs controller, etc.).

### 5.2 Atomic reserve

- **Assignment creation and lease creation happen in one DB transaction.**
- Two concurrent sessions **must not** receive the same issue/PR as assigned work.
- On contention: second session gets **WAIT** or **owner-resume**, never a duplicate lease.

### 5.3 Routing policy

| Condition | Route |
|-----------|--------|
| Current-head **REQUEST_CHANGES** | **Author** (not reviewer/merger) |
| **Stale** approval (approval not on current head) | **Reviewer** |
| **Clean** approval on current head, mergeable | **Merger** |
| Contested / contaminated approval | **Diagnosis / reconciler** (controller path) |
| Active **foreign** lease | **WAIT** or **owner-resume** |
| **Terminal-review** lock present | **Terminal-path resolution first** before downstream review work |
| Merged / closed PR | **Never assign** |
| Issue locked by sanctioned lease | **Never assign** to another worker unless lease cleanup/adoption is sanctioned |

### 5.4 Relationship to Gitea mirrors

After a successful assignment, the allocator (or sanctioned tooling) may update Gitea comments/labels so humans and audits see state. Those mirrors **are not** the sole lock source.

## 6. Control-plane DB topology (#613)

### 6.1 Preferred architecture (multi-daemon / Proxmox)

For true multi-session concurrency (multiple MCP daemons, multiple hosts, or Proxmox VMs):

**Prefer either:**

1. **Shared Postgres** used by all Gitea MCP profiles / allocator callers, **or**
2. A **single allocator daemon** (sole writer to the coordination store) that all workers call over MCP/RPC.

Both satisfy: one transactional authority for “who has this work item.”

### 6.2 SQLite MVP

SQLite is acceptable **only** for a **single-writer MVP**:

- One process owns writes (allocator daemon or single co-located MCP server), **or**
- Documented single-host single-daemon experiments with file locking and no multi-host claims.

SQLite is **not** sufficient to declare multi-session production readiness when four independent LLM sessions talk to four MCP processes without a shared writer.

### 6.3 Suggested entities (normative shape)

Minimum logical entities (names may vary; semantics must not):

1. **sessions** — session_id, role/profile, namespace, pid, started_at, last_heartbeat_at, status  
2. **work_items** — remote/org/repo, kind ∈ {`issue`, `pr`}, number, state, priority, current_head_sha, updated_at  
3. **leases** — lease_id, work_item_id, session_id, role, phase, expires_at, heartbeat_at, status  
4. **assignments** — assignment_id, work_item_id, session_id, allowed_action(s), expected_head_sha, status  
5. **terminal_locks** — repo/org, terminal_pr, review_id, decision, status, cleanup_state  
6. **events** — event_id, work_item_id, type, message, created_at  
7. **incident_links** — provider-neutral link model (see §8)

**Explicit non-kind:** do **not** use `work_items.kind = sentry_incident` (or glitchtip_incident) as an assignable work unit. Incidents become work only after the bridge creates/links a **Gitea issue**.

## 7. Lease migration (avoid split-brain)

### 7.1 Target state

- **Primary** for live coordination: **control-plane DB** leases and assignments.
- **Gitea comment leases** (e.g. `<!-- mcp-review-lease:v1 -->`) and **local issue-lock files** become:
  - **mirrors** of DB state for human visibility / recovery, and/or
  - written **only** through the allocator (or sanctioned lease tools that update DB + mirror in one workflow).

### 7.2 Migration rules

1. New exclusive claims go through DB assignment+lease first.
2. Comment lease / file lock writers that skip the DB are **deprecated** once #613+#600 land.
3. Reconciler tooling heals: expired DB lease, orphan Gitea comment, orphan local file — fail closed when ambiguous.
4. **Split-brain is a defect:** “DB free + Gitea comment leased” or “DB leased + Gitea free” must be detectable and reconcilable; production paths must not rely on two independent writers.

### 7.3 Heartbeats

- Heartbeats update the **DB**.
- Optional Gitea summaries must avoid comment spam (periodic summary or edit-in-place policy).

## 8. Observability bridge (#612)

### 8.1 Role

The bridge:

1. Scans configured Sentry and/or GlitchTip projects (multi-project mappings).
2. Deduplicates against existing links / Gitea issues.
3. Creates or updates **normal Gitea issues** (labels, sanitized body, provider URL).
4. Upserts **one** canonical `incident_links` row.
5. Optionally writes resolution status back to the provider when supported.
6. Never approves, merges, closes, or otherwise bypasses Gitea workflow gates.

### 8.2 Allocator interaction

- The allocator sees observability work **only after** a Gitea issue exists (typically `status:ready` + observability labels).
- **Do not assign raw Sentry/GlitchTip incidents** as work items.
- Bridge AC “allocator can see linked issues” means: **as ordinary Gitea issues**, not a special incident queue.

### 8.3 Provider adapters and trust

- Separate **Sentry** and **GlitchTip** adapters; document API differences; do not assume writeback parity.
- Self-hosted base URLs supported (e.g. `https://sentry.prgs.cc`).
- Tokens from environment / secret store only.
- Prefer phase-1 **explicit reconcile / dry-run** before unsupervised watchdog auto-filing, consistent with existing GlitchTip filing safety contracts.

### 8.4 Home of implementation

Filing/orchestration may live in:

- a control-plane / bridge package or profile, and/or  
- Gitea-Tools as operator-facing tools that **delegate** to that boundary,

but must not violate §4.4 credential isolation.

## 9. Incident link storage (single source of truth)

### 9.1 Canonical model: `incident_links` (provider-neutral)

Prefer a **provider-neutral** model over Sentry-only `sentry_links`:

| Field (logical) | Purpose |
|-----------------|---------|
| provider | `sentry` \| `glitchtip` \| … |
| provider_base_url | Self-hosted base |
| provider_org / provider_project | Mapping key |
| provider_issue_id / provider_short_id | Provider identity |
| provider_permalink | Human URL |
| fingerprint | Stable dedupe key when available |
| gitea_org / gitea_repo / gitea_issue_number | Linked work |
| linked_pr_numbers | Optional PR association |
| first_seen / last_seen / event_count | Summary metrics (safe) |
| status | link lifecycle |
| release_resolved_at / last_sync_at | Sync bookkeeping |

### 9.2 Gitea as human mirror

- Issue body markers / structured comments (e.g. HTML comment metadata) remain the **human- and audit-visible mirror**.
- They must stay consistent with `incident_links` but are **not** a second independent write path for inventing links without the bridge.

### 9.3 One truth

- **Do not** maintain two independent systems of record (e.g. full mapping only in #612 storage **and** a separate #613 `sentry_links` with different semantics).
- #612 implementation **must** use the same `incident_links` model introduced under #613 (or a single agreed store both reference).

## 10. Security and redaction

Mandatory for bridge payloads, Gitea issue bodies/comments, DB-stored event summaries, and logs:

- No API tokens, DSNs, passwords, cookies, auth headers
- No keychain IDs, private config contents, raw session-state files, full prompt bodies
- Sanitize stack locals and request data; store only safe tags/context
- Logs must never print provider tokens or DSNs
- Redaction tests are required for #612

## 11. Non-goals

- Replace Gitea as the durable workflow record  
- Let Sentry/GlitchTip assign work or mutate Gitea workflow state outside sanctioned tools  
- Let the bridge approve, merge, close, release, or bypass Gitea gates  
- Implement #600 on file locks / comments alone and call allocator complete  
- Treat raw monitoring incidents as `work_items` for the allocator  
- Put monitor tokens into every Gitea MCP worker process by default  

## 12. Consequences

### Positive

- Clear implementation order and ownership  
- Multi-session safety becomes testable at the DB layer  
- Observability becomes assignable work without special-casing the allocator  
- Trust boundaries stay compatible with existing control-plane docs  

### Costs / risks

- Migration from comment/file leases requires reconciler work  
- Shared Postgres or single allocator daemon is operational cost  
- Bridge must be careful not to spam Gitea issues (dedupe, caps, policy modes)  

### Follow-ups (documentation only until implemented)

1. Update #600, #612, and #613 bodies to **link this ADR** and restate hard dependencies.  
2. Implement #613 schema + atomic assign/lease.  
3. Implement #600 against the DB.  
4. Implement #612 against `incident_links` + Gitea create/update.  
5. Deprecate dual writers for leases.

## 13. Acceptance criteria for *this* ADR

This document is accepted when:

1. It is merged into `docs/architecture/` on the default branch.  
2. #600 / #612 / #613 (or their PRs) reference this path as the architecture source of truth.  
3. No implementation PR for those issues claims completion without conforming to §§2–11.

## 14. Document history

| Date | Change |
|------|--------|
| 2026-07-09 | Initial ADR: dependency order, authority model, topology, lease migration, bridge, `incident_links`, security, non-goals |