sysadminandClaude Opus 4.8
78cc37a977
Add an env-var-gated, off-by-default Sentry SDK integration so MCP runtime errors, fail-closed workflow blockers, lease/terminal-lock/stale-runtime collisions, and recurring watchdog check-ins are visible in a self-hosted Sentry at https://sentry.prgs.cc/. Gitea stays the source of truth; Sentry is observe-only. New module `sentry_observability.py` mirrors the `gitea_audit` conventions (env-gated, best-effort, redacting): - Config from MCP_SENTRY_ENABLED / SENTRY_DSN / SENTRY_ENVIRONMENT / SENTRY_RELEASE / MCP_SENTRY_TRACES_SAMPLE_RATE / MCP_SENTRY_ENABLE_LOGS. Active only when enabled AND a DSN is present; otherwise a hard no-op. - Fail OPEN for observability (never blocks a tool success path) and fail CLOSED for redaction (drop a field rather than risk leaking it). - `scrub_event` before_send/before_send_log hook + allowlisted tags: no tokens/passwords/keychain ids/DSNs/cookies, no raw session-state or full prompt bodies (session_id -> 12-char hash), no full filesystem paths (worktree path -> coarse category). Reuses incident_bridge + gitea_audit scrubbers. - capture_exception, capture_workflow_blocker (with canonical next action), and monitor_checkin with six stable cron slugs (stale lease scan, terminal lock scan, allocator health, namespace health, dashboard freshness, reconciler cleanup). - `sentry_sdk` is a lazily-imported optional dependency; the module imports and no-ops cleanly when the package is absent. Wiring in gitea_mcp_server.py (additive, guarded, best-effort): - init_sentry() in __main__ before mcp.run. - capture_exception in the `_audited` failure path; capture_workflow_blocker in `_audit_pr_result` BLOCKED/FAILED path. - allocator + namespace-health watchdog check-ins at their MCP tool sites (domain modules left pure). Also: pin `sentry-sdk==2.20.0` (optional), document the six env vars in `.env.example`, and add `docs/observability/sentry-integration.md` covering project creation in https://sentry.prgs.cc/, DSN handling, local/dev/prod config, redaction guarantees, and coexistence with the #612 incident bridge. Tests: tests/test_sentry_observability.py (36 cases) cover disabled / enabled / missing-DSN / missing-SDK, redaction, exception capture, workflow-blocker capture, and cron check-in behaviour. Full suite: 2632 passed, 6 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
139 lines
6.7 KiB
Markdown
139 lines
6.7 KiB
Markdown
# Self-hosted Sentry observability for the Gitea MCP server (#606)
|
||
|
||
Optional, **off-by-default** instrumentation that reports MCP runtime errors,
|
||
fail-closed workflow blockers, lease / terminal-lock / stale-runtime
|
||
collisions, and recurring watchdog check-ins to a **self-hosted** Sentry at
|
||
`https://sentry.prgs.cc/`.
|
||
|
||
> **Gitea remains the source of truth.** Sentry is observe-only. It never
|
||
> approves, merges, closes, or otherwise mutates Gitea workflow state, and it
|
||
> never bypasses leases, #332, workflow roles, or the MCP gates. Sentry alerts
|
||
> may only feed the *sanctioned* Gitea issue/comment path via the #612 incident
|
||
> bridge — never a direct write.
|
||
|
||
Implemented by [`sentry_observability.py`](../../sentry_observability.py).
|
||
|
||
---
|
||
|
||
## 1. Create the Sentry project
|
||
|
||
1. Sign in to the self-hosted Sentry at **`https://sentry.prgs.cc/`** (this is
|
||
**not** Sentry Cloud — do not use `*.ingest.sentry.io`).
|
||
2. Create a new **Python** project named **`gitea-tools-mcp`**.
|
||
3. Open **Settings → Projects → gitea-tools-mcp → Client Keys (DSN)** and copy
|
||
the DSN. It looks like `https://<publickey>@sentry.prgs.cc/<project-id>`.
|
||
4. **Never commit the DSN.** It is a runtime secret supplied via env var only.
|
||
|
||
## 2. Configure the environment
|
||
|
||
All configuration is env-var driven (see [`.env.example`](../../.env.example)):
|
||
|
||
| Variable | Purpose | Default |
|
||
|----------|---------|---------|
|
||
| `MCP_SENTRY_ENABLED` | Master gate (`1/true/yes/on`). Required. | off |
|
||
| `SENTRY_DSN` | Self-hosted DSN. Required. | *(empty)* |
|
||
| `SENTRY_ENVIRONMENT` | `local` / `dev` / `prod` tag. | `development` |
|
||
| `SENTRY_RELEASE` | Release id (git SHA or version). | *(none)* |
|
||
| `MCP_SENTRY_TRACES_SAMPLE_RATE` | Perf-trace sample rate `0.0–1.0` (clamped). | `0.0` |
|
||
| `MCP_SENTRY_ENABLE_LOGS` | Forward Python logs as structured logs. | off |
|
||
|
||
**The feature stays completely off unless `MCP_SENTRY_ENABLED` is truthy *and*
|
||
`SENTRY_DSN` is non-empty.** With either missing, `init_sentry()` is a no-op,
|
||
the SDK is never initialised, and no events are sent — existing tool behaviour
|
||
and API-call patterns are unchanged.
|
||
|
||
### Per-environment examples
|
||
|
||
```bash
|
||
# local (quiet: capture errors/blockers, no traces)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=local
|
||
|
||
# dev (light tracing + logs)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=dev
|
||
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.2
|
||
export MCP_SENTRY_ENABLE_LOGS=1
|
||
|
||
# prod (errors/blockers + low-rate tracing, release-tagged)
|
||
export MCP_SENTRY_ENABLED=1
|
||
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
|
||
export SENTRY_ENVIRONMENT=prod
|
||
export SENTRY_RELEASE="$(git rev-parse --short HEAD)"
|
||
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.05
|
||
```
|
||
|
||
The optional SDK is pinned in [`requirements.txt`](../../requirements.txt)
|
||
(`sentry-sdk==2.20.0`). It is imported lazily: if the package is absent, the
|
||
module still imports and every entry point is a safe no-op.
|
||
|
||
## 3. What is instrumented
|
||
|
||
| Signal | Where | Notes |
|
||
|--------|-------|-------|
|
||
| Startup init | `gitea_mcp_server.py` `__main__`, before `mcp.run` | Prints a redaction-safe status line to stderr. |
|
||
| Failing mutations (exceptions) | `_audited(...)` context manager | `capture_exception` with scrubbed tags. |
|
||
| Fail-closed blockers / failed mutations | `_audit_pr_result(...)` (BLOCKED/FAILED) | Structured `capture_workflow_blocker` event incl. the canonical next action when available (criterion 7). |
|
||
| Allocator watchdog check-ins | `gitea_allocate_next_work` tool | `allocator_health`, `stale_lease_scan`, `terminal_lock_scan`. |
|
||
| Namespace-health check-in | `gitea_assess_mcp_namespace_health` tool | `namespace_health`. |
|
||
|
||
All capture paths are **best-effort / fail open**: a Sentry outage or capture
|
||
error never breaks an MCP tool success path.
|
||
|
||
## 4. Cron / watchdog monitors
|
||
|
||
`sentry_observability.MONITOR_SLUGS` defines stable check-in slugs:
|
||
|
||
| Registry key | Sentry monitor slug | Wired at |
|
||
|--------------|--------------------|----------|
|
||
| `stale_lease_scan` | `gitea-mcp-stale-lease-scan` | allocator run (global lease expiry) |
|
||
| `terminal_lock_scan` | `gitea-mcp-terminal-lock-scan` | allocator run (terminal-lock lookup) |
|
||
| `allocator_health` | `gitea-mcp-allocator-health` | allocator run |
|
||
| `namespace_health` | `gitea-mcp-namespace-health` | namespace-health probe |
|
||
| `dashboard_freshness` | `gitea-mcp-dashboard-freshness` | call `monitor_checkin("dashboard_freshness", ...)` from the dashboard refresh job (#605) |
|
||
| `reconciler_cleanup` | `gitea-mcp-reconciler-cleanup` | call `monitor_checkin("reconciler_cleanup", ...)` from the reconciler cleanup entrypoint |
|
||
|
||
Create matching Cron monitors in Sentry with those slugs. Emit an
|
||
`in_progress` check-in at job start and `ok`/`error` at completion via
|
||
`sentry_observability.monitor_checkin(slug_key, status)`.
|
||
|
||
## 5. Redaction guarantees (fail closed)
|
||
|
||
Redaction fails *closed*: if a field cannot be proven safe it is dropped rather
|
||
than sent. The `before_send` (and `before_send_log`) hook `scrub_event`
|
||
recursively redacts every outgoing event; on any error it drops the event
|
||
entirely. Guarantees, proven by `tests/test_sentry_observability.py`:
|
||
|
||
- **No** tokens, passwords, keychain IDs, DSNs, cookies, or `user:pass@host`.
|
||
- **No** raw session-state or full prompt/comment bodies — `session_id` is only
|
||
ever surfaced as a 12-char `session_id_hash`.
|
||
- **No** private config contents or raw credential headers.
|
||
- **No** full local filesystem paths — a worktree path collapses to a coarse
|
||
`worktree_category` (`author` / `reviewer` / `merger` / `reconciler` /
|
||
`branches` / `root` / `other`).
|
||
- Only the allowlisted tag keys in `ALLOWED_TAG_KEYS` are ever attached.
|
||
|
||
## 6. Coexistence with GlitchTip / the #612 incident bridge
|
||
|
||
This is the **outbound** path (MCP → Sentry SDK). It complements — it does not
|
||
replace — the **inbound** [`incident_bridge.py`](../../incident_bridge.py)
|
||
(#612), which turns Sentry/GlitchTip *observations* into durable Gitea issues
|
||
and `incident_links` rows.
|
||
|
||
- Prefer **one** observability path per environment. Point the MCP server's
|
||
`SENTRY_DSN` at the same self-hosted `gitea-tools-mcp` project that the #612
|
||
bridge reconciles from, so an MCP-reported error and its Gitea issue line up.
|
||
- GlitchTip is Sentry-protocol compatible; if an existing GlitchTip DSN is in
|
||
use, either migrate it to `https://sentry.prgs.cc/` or document the split
|
||
(MCP → Sentry, legacy → GlitchTip) explicitly for operators.
|
||
- The bridge remains the **only** sanctioned route from an alert back into
|
||
Gitea workflow state.
|
||
|
||
## 7. Non-goals
|
||
|
||
- Sentry must **not** become the workflow source of truth.
|
||
- Sentry must **not** approve, merge, close, or mutate Gitea workflow state.
|
||
- Sentry must **not** bypass leases, #332, workflow roles, or the MCP gates.
|