Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 77 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

feat(observability): optional self-hosted Sentry instrumentation for MCP workflow failures (Closes #606)

Browse Source 
Add an env-var-gated, off-by-default Sentry SDK integration so MCP runtime
errors, fail-closed workflow blockers, lease/terminal-lock/stale-runtime
collisions, and recurring watchdog check-ins are visible in a self-hosted
Sentry at https://sentry.prgs.cc/. Gitea stays the source of truth; Sentry is
observe-only.

New module `sentry_observability.py` mirrors the `gitea_audit` conventions
(env-gated, best-effort, redacting):

- Config from MCP_SENTRY_ENABLED / SENTRY_DSN / SENTRY_ENVIRONMENT /
  SENTRY_RELEASE / MCP_SENTRY_TRACES_SAMPLE_RATE / MCP_SENTRY_ENABLE_LOGS.
  Active only when enabled AND a DSN is present; otherwise a hard no-op.
- Fail OPEN for observability (never blocks a tool success path) and fail
  CLOSED for redaction (drop a field rather than risk leaking it).
- `scrub_event` before_send/before_send_log hook + allowlisted tags: no
  tokens/passwords/keychain ids/DSNs/cookies, no raw session-state or full
  prompt bodies (session_id -> 12-char hash), no full filesystem paths
  (worktree path -> coarse category). Reuses incident_bridge + gitea_audit
  scrubbers.
- capture_exception, capture_workflow_blocker (with canonical next action),
  and monitor_checkin with six stable cron slugs (stale lease scan, terminal
  lock scan, allocator health, namespace health, dashboard freshness,
  reconciler cleanup).
- `sentry_sdk` is a lazily-imported optional dependency; the module imports
  and no-ops cleanly when the package is absent.

Wiring in gitea_mcp_server.py (additive, guarded, best-effort):
- init_sentry() in __main__ before mcp.run.
- capture_exception in the `_audited` failure path; capture_workflow_blocker
  in `_audit_pr_result` BLOCKED/FAILED path.
- allocator + namespace-health watchdog check-ins at their MCP tool sites
  (domain modules left pure).

Also: pin `sentry-sdk==2.20.0` (optional), document the six env vars in
`.env.example`, and add `docs/observability/sentry-integration.md` covering
project creation in https://sentry.prgs.cc/, DSN handling, local/dev/prod
config, redaction guarantees, and coexistence with the #612 incident bridge.

Tests: tests/test_sentry_observability.py (36 cases) cover disabled / enabled /
missing-DSN / missing-SDK, redaction, exception capture, workflow-blocker
capture, and cron check-in behaviour. Full suite: 2632 passed, 6 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
sysadmin
2026-07-12 02:35:47 -04:00
co-authored by Claude Opus 4.8
parent 22698c1b5f
commit 78cc37a977
6 changed files with 1157 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/observability/sentry-integration.md
+138
View File
@@ -0,0 +1,138 @@
# Self-hosted Sentry observability for the Gitea MCP server (#606)
Optional, **off-by-default** instrumentation that reports MCP runtime errors,
fail-closed workflow blockers, lease / terminal-lock / stale-runtime
collisions, and recurring watchdog check-ins to a **self-hosted** Sentry at
`https://sentry.prgs.cc/`.
> **Gitea remains the source of truth.** Sentry is observe-only. It never
> approves, merges, closes, or otherwise mutates Gitea workflow state, and it
> never bypasses leases, #332, workflow roles, or the MCP gates. Sentry alerts
> may only feed the *sanctioned* Gitea issue/comment path via the #612 incident
> bridge — never a direct write.
Implemented by [`sentry_observability.py`](../../sentry_observability.py).
---
## 1. Create the Sentry project
1. Sign in to the self-hosted Sentry at **`https://sentry.prgs.cc/`** (this is
**not** Sentry Cloud — do not use `*.ingest.sentry.io`).
2. Create a new **Python** project named **`gitea-tools-mcp`**.
3. Open **Settings → Projects → gitea-tools-mcp → Client Keys (DSN)** and copy
the DSN. It looks like `https://<publickey>@sentry.prgs.cc/<project-id>`.
4. **Never commit the DSN.** It is a runtime secret supplied via env var only.
## 2. Configure the environment
All configuration is env-var driven (see [`.env.example`](../../.env.example)):
| Variable | Purpose | Default |
|----------|---------|---------|
| `MCP_SENTRY_ENABLED` | Master gate (`1/true/yes/on`). Required. | off |
| `SENTRY_DSN` | Self-hosted DSN. Required. | *(empty)* |
| `SENTRY_ENVIRONMENT` | `local` / `dev` / `prod` tag. | `development` |
| `SENTRY_RELEASE` | Release id (git SHA or version). | *(none)* |
| `MCP_SENTRY_TRACES_SAMPLE_RATE` | Perf-trace sample rate `0.0–1.0` (clamped). | `0.0` |
| `MCP_SENTRY_ENABLE_LOGS` | Forward Python logs as structured logs. | off |
**The feature stays completely off unless `MCP_SENTRY_ENABLED` is truthy *and*
`SENTRY_DSN` is non-empty.** With either missing, `init_sentry()` is a no-op,
the SDK is never initialised, and no events are sent — existing tool behaviour
and API-call patterns are unchanged.
### Per-environment examples
```bash
# local (quiet: capture errors/blockers, no traces)
export MCP_SENTRY_ENABLED=1
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
export SENTRY_ENVIRONMENT=local
# dev (light tracing + logs)
export MCP_SENTRY_ENABLED=1
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
export SENTRY_ENVIRONMENT=dev
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.2
export MCP_SENTRY_ENABLE_LOGS=1
# prod (errors/blockers + low-rate tracing, release-tagged)
export MCP_SENTRY_ENABLED=1
export SENTRY_DSN="https://<key>@sentry.prgs.cc/<id>"
export SENTRY_ENVIRONMENT=prod
export SENTRY_RELEASE="$(git rev-parse --short HEAD)"
export MCP_SENTRY_TRACES_SAMPLE_RATE=0.05
```
The optional SDK is pinned in [`requirements.txt`](../../requirements.txt)
(`sentry-sdk==2.20.0`). It is imported lazily: if the package is absent, the
module still imports and every entry point is a safe no-op.
## 3. What is instrumented
| Signal | Where | Notes |
|--------|-------|-------|
| Startup init | `gitea_mcp_server.py` `__main__`, before `mcp.run` | Prints a redaction-safe status line to stderr. |
| Failing mutations (exceptions) | `_audited(...)` context manager | `capture_exception` with scrubbed tags. |
| Fail-closed blockers / failed mutations | `_audit_pr_result(...)` (BLOCKED/FAILED) | Structured `capture_workflow_blocker` event incl. the canonical next action when available (criterion 7). |
| Allocator watchdog check-ins | `gitea_allocate_next_work` tool | `allocator_health`, `stale_lease_scan`, `terminal_lock_scan`. |
| Namespace-health check-in | `gitea_assess_mcp_namespace_health` tool | `namespace_health`. |
All capture paths are **best-effort / fail open**: a Sentry outage or capture
error never breaks an MCP tool success path.
## 4. Cron / watchdog monitors
`sentry_observability.MONITOR_SLUGS` defines stable check-in slugs:
| Registry key | Sentry monitor slug | Wired at |
|--------------|--------------------|----------|
| `stale_lease_scan` | `gitea-mcp-stale-lease-scan` | allocator run (global lease expiry) |
| `terminal_lock_scan` | `gitea-mcp-terminal-lock-scan` | allocator run (terminal-lock lookup) |
| `allocator_health` | `gitea-mcp-allocator-health` | allocator run |
| `namespace_health` | `gitea-mcp-namespace-health` | namespace-health probe |
| `dashboard_freshness` | `gitea-mcp-dashboard-freshness` | call `monitor_checkin("dashboard_freshness", ...)` from the dashboard refresh job (#605) |
| `reconciler_cleanup` | `gitea-mcp-reconciler-cleanup` | call `monitor_checkin("reconciler_cleanup", ...)` from the reconciler cleanup entrypoint |
Create matching Cron monitors in Sentry with those slugs. Emit an
`in_progress` check-in at job start and `ok`/`error` at completion via
`sentry_observability.monitor_checkin(slug_key, status)`.
## 5. Redaction guarantees (fail closed)
Redaction fails *closed*: if a field cannot be proven safe it is dropped rather
than sent. The `before_send` (and `before_send_log`) hook `scrub_event`
recursively redacts every outgoing event; on any error it drops the event
entirely. Guarantees, proven by `tests/test_sentry_observability.py`:
- **No** tokens, passwords, keychain IDs, DSNs, cookies, or `user:pass@host`.
- **No** raw session-state or full prompt/comment bodies — `session_id` is only
ever surfaced as a 12-char `session_id_hash`.
- **No** private config contents or raw credential headers.
- **No** full local filesystem paths — a worktree path collapses to a coarse
`worktree_category` (`author` / `reviewer` / `merger` / `reconciler` /
`branches` / `root` / `other`).
- Only the allowlisted tag keys in `ALLOWED_TAG_KEYS` are ever attached.
## 6. Coexistence with GlitchTip / the #612 incident bridge
This is the **outbound** path (MCP → Sentry SDK). It complements — it does not
replace — the **inbound** [`incident_bridge.py`](../../incident_bridge.py)
(#612), which turns Sentry/GlitchTip *observations* into durable Gitea issues
and `incident_links` rows.
- Prefer **one** observability path per environment. Point the MCP server's
`SENTRY_DSN` at the same self-hosted `gitea-tools-mcp` project that the #612
bridge reconciles from, so an MCP-reported error and its Gitea issue line up.
- GlitchTip is Sentry-protocol compatible; if an existing GlitchTip DSN is in
use, either migrate it to `https://sentry.prgs.cc/` or document the split
(MCP → Sentry, legacy → GlitchTip) explicitly for operators.
- The bridge remains the **only** sanctioned route from an alert back into
Gitea workflow state.
## 7. Non-goals
- Sentry must **not** become the workflow source of truth.
- Sentry must **not** approve, merge, close, or mutate Gitea workflow state.
- Sentry must **not** bypass leases, #332, workflow roles, or the MCP gates.