sysadminandClaude Opus 4.8
e168978579
Adds the read half of the inbound observability path: pull unresolved issues
and events from the self-hosted Sentry API, normalize them into #612
observations, and reconcile them into durable Gitea issues.
Design: the existing #612 incident_bridge already owns dedupe, linking,
redaction, and issue creation on the #613 incident_links substrate, so this
change adds only what was genuinely missing - a Sentry API read layer,
observation mapping, a policy gate, and a watchdog. No second linking store is
introduced, which is what makes the mapping survive restarts (AC6).
New module sentry_incident_bridge.py:
* list_issues() with Link-header cursor pagination and statsPeriod windowing
* get_issue_events() returning sanitized recent + latest event
* observation_from_issue() mapping onto the #612 observation contract
* should_bridge_issue() policy gate (unresolved + event-count threshold)
* watchdog() scanning and reconciling, dry-run by default
* HTTP access injected as http_fn so the surface is testable without Sentry
New MCP tools: gitea_sentry_list_issues, gitea_sentry_get_issue_events,
gitea_sentry_reconcile_issue, gitea_sentry_link_gitea_issue,
gitea_sentry_watchdog.
Safety:
* SENTRY_AUTH_TOKEN is read from the environment only and never returned,
logged, or stored; config projections cannot carry it by construction
* missing config fails closed as not_configured, and a missing token fails
closed as missing_token before any HTTP call is made
* apply=true requires both MCP_SENTRY_ISSUE_BRIDGE_ENABLED and issue-create
permission; Sentry outages create nothing
* secrets are redacted and absolute local paths reduced to a category token
before any value can reach a Gitea issue body
* raw Sentry incidents are never assignable control-plane work items
Tests: 38 new cases covering create, update/recurrence, dedupe, resolved-issue
non-reopen, redaction, pagination, missing token, unavailable server,
self-hosted base URL, restart persistence, policy gates, and the five tool
wrappers.
Full suite: 3733 passed, 6 skipped. The 2 remaining failures
(test_issue_702_review_findings_f1_f6, test_reconciler_supersession_close) were
verified to fail identically on unmodified master at fcf6981 and are unrelated
to this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011w1WGVV3duWEf45SJRJ1DL
191 lines
9.5 KiB
Markdown
191 lines
9.5 KiB
Markdown
# Self-hosted Sentry observability for the Gitea MCP server (#606)
|
||
|
||
Optional, **off-by-default** instrumentation that reports MCP runtime errors,
|
||
fail-closed workflow blockers, lease / terminal-lock / stale-runtime
|
||
collisions, and recurring watchdog check-ins to a **self-hosted** Sentry at
|
||
`https://sentry.prgs.cc/`.
|
||
|
||
> **Gitea remains the source of truth.** Sentry is observe-only. It never
|
||
> approves, merges, closes, or otherwise mutates Gitea workflow state, and it
|
||
> never bypasses leases, #332, workflow roles, or the MCP gates. Sentry alerts
|
||
> may only feed the *sanctioned* Gitea issue/comment path via the #612 incident
|
||
> bridge — never a direct write.
|
||
|
||
Implemented by [`sentry_observability.py`](../../sentry_observability.py).
|
||
|
||
---
|
||
|
||
## 1. Create the Sentry project
|
||
|
||
1. Sign in to the self-hosted Sentry at **`https://sentry.prgs.cc/`** (this is
|
||
**not** Sentry Cloud — do not use `*.ingest.sentry.io`).
|
||
2. Create a new **Python** project named **`gitea-tools-mcp`**.
|
||
3. Open **Settings → Projects → gitea-tools-mcp → Client Keys (DSN)** and copy
|
||
the DSN. It looks like `https://<publickey>@sentry.prgs.cc/<project-id>`.
|
||
4. **Never commit the DSN.** It is a runtime secret supplied via env var only.
|
||
|
||
## 2. Configure the environment
|
||
|
||
All configuration is env-var driven (see [`.env.example`](../../.env.example)):
|
||
|
||
| Variable | Purpose | Default |
|
||
|----------|---------|---------|
|
||
| `MCP_SENTRY_ENABLED` | Master gate (`1/true/yes/on`). Required. | off |
|
||
| `SENTRY_DSN` | Self-hosted DSN. Required. | *(empty)* |
|
||
| `SENTRY_ENVIRONMENT` | `local` / `dev` / `prod` tag. | `development` |
|
||
| `SENTRY_RELEASE` | Release id (git SHA or version). | *(none)* |
|
||
| `MCP_SENTRY_TRACES_SAMPLE_RATE` | Perf-trace sample rate `0.0–1.0` (clamped). | `0.0` |
|
||
| `MCP_SENTRY_ENABLE_LOGS` | Forward Python logs as structured logs. | off |
|
||
|
||
**The feature stays completely off unless `MCP_SENTRY_ENABLED` is truthy *and*
|
||
`SENTRY_DSN` is non-empty.** With either missing, `init_sentry()` is a no-op,
|
||
the SDK is never initialised, and no events are sent — existing tool behaviour
|
||
and API-call patterns are unchanged.
|
||
|
||
### Per-environment examples
|
||
|
||
```bash
|
||
# local (quiet: capture errors/blockers, no traces)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=local
|
||
|
||
# dev (light tracing + logs)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=dev
|
||
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.2
|
||
export MCP_SENTRY_ENABLE_LOGS=1
|
||
|
||
# prod (errors/blockers + low-rate tracing, release-tagged)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=prod
|
||
export SENTRY_RELEASE="$(git rev-parse --short HEAD)"
|
||
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.05
|
||
```
|
||
|
||
The optional SDK is pinned in [`requirements.txt`](../../requirements.txt)
|
||
(`sentry-sdk==2.20.0`). It is imported lazily: if the package is absent, the
|
||
module still imports and every entry point is a safe no-op.
|
||
|
||
## 3. What is instrumented
|
||
|
||
| Signal | Where | Notes |
|
||
|--------|-------|-------|
|
||
| Startup init | `gitea_mcp_server.py` `__main__`, before `mcp.run` | Prints a redaction-safe status line to stderr. |
|
||
| Failing mutations (exceptions) | `_audited(...)` context manager | `capture_exception` with scrubbed tags. |
|
||
| Fail-closed blockers / failed mutations | `_audit_pr_result(...)` (BLOCKED/FAILED) | Structured `capture_workflow_blocker` event incl. the canonical next action when available (criterion 7). |
|
||
| Allocator watchdog check-ins | `gitea_allocate_next_work` tool | `allocator_health`, `stale_lease_scan`, `terminal_lock_scan`. |
|
||
| Namespace-health check-in | `gitea_assess_mcp_namespace_health` tool | `namespace_health`. |
|
||
|
||
All capture paths are **best-effort / fail open**: a Sentry outage or capture
|
||
error never breaks an MCP tool success path.
|
||
|
||
## 4. Cron / watchdog monitors
|
||
|
||
`sentry_observability.MONITOR_SLUGS` defines stable check-in slugs:
|
||
|
||
| Registry key | Sentry monitor slug | Wired at |
|
||
|--------------|--------------------|----------|
|
||
| `stale_lease_scan` | `gitea-mcp-stale-lease-scan` | allocator run (global lease expiry) |
|
||
| `terminal_lock_scan` | `gitea-mcp-terminal-lock-scan` | allocator run (terminal-lock lookup) |
|
||
| `allocator_health` | `gitea-mcp-allocator-health` | allocator run |
|
||
| `namespace_health` | `gitea-mcp-namespace-health` | namespace-health probe |
|
||
| `dashboard_freshness` | `gitea-mcp-dashboard-freshness` | call `monitor_checkin("dashboard_freshness", ...)` from the dashboard refresh job (#605) |
|
||
| `reconciler_cleanup` | `gitea-mcp-reconciler-cleanup` | call `monitor_checkin("reconciler_cleanup", ...)` from the reconciler cleanup entrypoint |
|
||
|
||
Create matching Cron monitors in Sentry with those slugs. Emit an
|
||
`in_progress` check-in at job start and `ok`/`error` at completion via
|
||
`sentry_observability.monitor_checkin(slug_key, status)`.
|
||
|
||
## 5. Redaction guarantees (fail closed)
|
||
|
||
Redaction fails *closed*: if a field cannot be proven safe it is dropped rather
|
||
than sent. The `before_send` (and `before_send_log`) hook `scrub_event`
|
||
recursively redacts every outgoing event; on any error it drops the event
|
||
entirely. Guarantees, proven by `tests/test_sentry_observability.py`:
|
||
|
||
- **No** tokens, passwords, keychain IDs, DSNs, cookies, or `user:pass@host`.
|
||
- **No** raw session-state or full prompt/comment bodies — `session_id` is only
|
||
ever surfaced as a 12-char `session_id_hash`.
|
||
- **No** private config contents or raw credential headers.
|
||
- **No** full local filesystem paths — a worktree path collapses to a coarse
|
||
`worktree_category` (`author` / `reviewer` / `merger` / `reconciler` /
|
||
`branches` / `root` / `other`).
|
||
- Only the allowlisted tag keys in `ALLOWED_TAG_KEYS` are ever attached.
|
||
|
||
## 6. Coexistence with GlitchTip / the #612 incident bridge
|
||
|
||
This is the **outbound** path (MCP → Sentry SDK). It complements — it does not
|
||
replace — the **inbound** [`incident_bridge.py`](../../incident_bridge.py)
|
||
(#612), which turns Sentry/GlitchTip *observations* into durable Gitea issues
|
||
and `incident_links` rows.
|
||
|
||
- Prefer **one** observability path per environment. Point the MCP server's
|
||
`SENTRY_DSN` at the same self-hosted `gitea-tools-mcp` project that the #612
|
||
bridge reconciles from, so an MCP-reported error and its Gitea issue line up.
|
||
- GlitchTip is Sentry-protocol compatible; if an existing GlitchTip DSN is in
|
||
use, either migrate it to `https://sentry.prgs.cc/` or document the split
|
||
(MCP → Sentry, legacy → GlitchTip) explicitly for operators.
|
||
- The bridge remains the **only** sanctioned route from an alert back into
|
||
Gitea workflow state.
|
||
|
||
## 7. Reading Sentry back into Gitea (#607)
|
||
|
||
[`sentry_incident_bridge.py`](../../sentry_incident_bridge.py) supplies the
|
||
**read** half of the inbound path: it pulls unresolved issues/events from the
|
||
self-hosted Sentry API, normalizes them into #612 observations, and hands them
|
||
to `incident_bridge.reconcile_incident`. It never adds a second linking store —
|
||
`incident_links` on the #613 control-plane DB stays canonical, which is what
|
||
makes the mapping survive restarts.
|
||
|
||
### Configuration
|
||
|
||
| Variable | Purpose | Default |
|
||
| --- | --- | --- |
|
||
| `SENTRY_BASE_URL` | Self-hosted Sentry root | `https://sentry.prgs.cc` |
|
||
| `SENTRY_AUTH_TOKEN` | API token — **env only**, never logged or returned | _(unset)_ |
|
||
| `SENTRY_ORG` | Sentry organization slug | _(unset)_ |
|
||
| `SENTRY_PROJECT` | Sentry project slug | _(unset)_ |
|
||
| `MCP_SENTRY_ISSUE_BRIDGE_ENABLED` | Required for `apply=true` | `false` |
|
||
| `MCP_SENTRY_MIN_EVENTS_FOR_ISSUE` | Recurrence threshold before an issue is worth filing | `2` |
|
||
| `MCP_SENTRY_LOOKBACK` | Scan window (`statsPeriod`, e.g. `24h`) | `24h` |
|
||
|
||
Missing org/project fails closed as `not_configured`; a missing token fails
|
||
closed as `missing_token` **before** any HTTP call is made.
|
||
|
||
### Tools
|
||
|
||
| Tool | Mode | Purpose |
|
||
| --- | --- | --- |
|
||
| `gitea_sentry_list_issues` | read-only | Unresolved issues, `Link`-header pagination |
|
||
| `gitea_sentry_get_issue_events` | read-only | Sanitized recent + latest event for one issue |
|
||
| `gitea_sentry_reconcile_issue` | dry-run default | One Sentry issue → durable Gitea issue |
|
||
| `gitea_sentry_link_gitea_issue` | dry-run default | Link a Sentry issue to an existing Gitea issue |
|
||
| `gitea_sentry_watchdog` | dry-run default | Scan + create/update issues for active incidents |
|
||
|
||
### Policy
|
||
|
||
- **Dedupe:** one Sentry issue maps to exactly one Gitea issue, keyed by
|
||
provider + base URL + org + project + issue id. Recurrence updates the link
|
||
(and its `event_count`) instead of filing a duplicate.
|
||
- **No reopen:** a Sentry issue that is no longer `unresolved` is skipped; the
|
||
bridge never reopens or re-files a closed Gitea issue.
|
||
- **Threshold:** issues below `MCP_SENTRY_MIN_EVENTS_FOR_ISSUE` are skipped, so
|
||
one-off noise does not become durable work.
|
||
- **Apply is explicit:** `apply=true` requires both
|
||
`MCP_SENTRY_ISSUE_BRIDGE_ENABLED` and issue-create permission on the profile.
|
||
- **Outages fail closed:** an unreachable Sentry returns `sentry_unavailable`
|
||
and creates nothing.
|
||
- **Redaction:** secrets are scrubbed and absolute local paths are reduced to a
|
||
category token (`[path:author]`, `[path:root]`, …) before any value reaches a
|
||
Gitea issue body. Sensitive tag keys (`authorization`, `cookie`, …) are
|
||
dropped, and permalinks carrying embedded credentials are discarded entirely.
|
||
|
||
## 8. Non-goals
|
||
|
||
- Sentry must **not** become the workflow source of truth.
|
||
- Sentry must **not** approve, merge, close, or mutate Gitea workflow state.
|
||
- Sentry must **not** bypass leases, #332, workflow roles, or the MCP gates.
|