feat(observability): optional self-hosted Sentry instrumentation for MCP workflow failures (Closes #606) #679

2026-07-12T01:36:40-05:00

jcwalker3 commented

2026-07-12 01:36:40 -05:00

Summary

Closes #606. Adds optional, off-by-default self-hosted Sentry SDK
instrumentation so MCP runtime errors, fail-closed workflow blockers,
lease/terminal-lock/stale-runtime collisions, and recurring watchdog check-ins
are visible in Sentry at https://sentry.prgs.cc/ (never Sentry Cloud). Gitea
stays the durable source of truth; Sentry is observe-only.

What changed

New sentry_observability.py — mirrors the gitea_audit conventions
(env-gated, best-effort, redacting):
- Config from MCP_SENTRY_ENABLED, SENTRY_DSN, SENTRY_ENVIRONMENT,
  SENTRY_RELEASE, MCP_SENTRY_TRACES_SAMPLE_RATE, MCP_SENTRY_ENABLE_LOGS.
  Active only when enabled AND a DSN is present; otherwise a hard no-op.
- Fail open for observability (never blocks a tool success path); fail
  closed for redaction (drop a field rather than risk a leak).
- scrub_event before_send/before_send_log hook + allowlisted tags: no
  tokens/passwords/keychain-ids/DSNs/cookies, no raw session-state or full
  prompt bodies (session_id → 12-char hash), no full filesystem paths
  (worktree path → coarse category). Reuses incident_bridge + gitea_audit
  scrubbers.
- capture_exception, capture_workflow_blocker (carries the canonical next
  action), and monitor_checkin with six stable cron slugs.
- sentry_sdk is a lazily-imported optional dependency — the module imports
  and no-ops cleanly when the package is absent.
gitea_mcp_server.py (additive, guarded, best-effort): init_sentry()
in __main__ before mcp.run; exception capture in the _audited failure
path; blocker capture in the _audit_pr_result BLOCKED/FAILED path; allocator
- namespace-health watchdog check-ins at their MCP tool sites (domain modules
  left pure).
requirements.txt — pin sentry-sdk==2.20.0 (optional).
.env.example — document the six env vars.
docs/observability/sentry-integration.md — project creation in
https://sentry.prgs.cc/, DSN handling, local/dev/prod config, redaction
guarantees, and coexistence with the #612 incident bridge.

Acceptance criteria mapping

Disabled / missing DSN → no init, no events ✓ (test_init_noop_*)
Enabled + self-hosted DSN → exceptions routed to the DSN's project ✓
(test_init_configures_sdk_with_scrubber, _audited capture)
Workflow blockers as structured events with tags ✓
(build_blocker_event, _audit_pr_result wiring)
Python logs when MCP_SENTRY_ENABLE_LOGS=1 ✓
(test_init_enable_logs_wires_log_scrubber)
Redaction proven for secrets/tokens/keychain-ids/config/session-state ✓
(redaction test block)
Cron check-ins for the six watchdog jobs ✓ (MONITOR_SLUGS + wiring)
Events include a canonical next action when available ✓
Tests cover disabled/enabled/missing-DSN/redaction/exception/blocker/cron ✓
Docs for project creation, DSN, and per-env config ✓
Gitea remains canonical ✓ (Sentry observe-only; non-goals documented)

Non-goals (enforced)

Sentry does not become the workflow authority and cannot approve/merge/close/
mutate Gitea state or bypass leases/#332/roles/MCP gates. Alerts may only reach
Gitea through the sanctioned #612 incident-bridge path.

Validation

New suite tests/test_sentry_observability.py: 36 passing.
Full suite: 2632 passed, 6 skipped (no regressions).

Reviewer handoff

Scope is limited to the six files above; allocator_service.py and other
domain modules are intentionally untouched (observability wiring lives in the
MCP server layer, consistent with _audit).
Please confirm the redaction allowlist/scrubber coverage and the guarded/
fail-open placement of the capture calls. This PR complements #612 and does
not replace it.

🤖 Generated with Claude Code

## Summary Closes #606. Adds **optional, off-by-default** self-hosted Sentry SDK instrumentation so MCP runtime errors, fail-closed workflow blockers, lease/terminal-lock/stale-runtime collisions, and recurring watchdog check-ins are visible in Sentry at `https://sentry.prgs.cc/` (never Sentry Cloud). Gitea stays the durable source of truth; Sentry is observe-only. ## What changed - **New `sentry_observability.py`** — mirrors the `gitea_audit` conventions (env-gated, best-effort, redacting): - Config from `MCP_SENTRY_ENABLED`, `SENTRY_DSN`, `SENTRY_ENVIRONMENT`, `SENTRY_RELEASE`, `MCP_SENTRY_TRACES_SAMPLE_RATE`, `MCP_SENTRY_ENABLE_LOGS`. Active **only** when enabled AND a DSN is present; otherwise a hard no-op. - Fail **open** for observability (never blocks a tool success path); fail **closed** for redaction (drop a field rather than risk a leak). - `scrub_event` `before_send`/`before_send_log` hook + allowlisted tags: no tokens/passwords/keychain-ids/DSNs/cookies, no raw session-state or full prompt bodies (`session_id` → 12-char hash), no full filesystem paths (worktree path → coarse category). Reuses `incident_bridge` + `gitea_audit` scrubbers. - `capture_exception`, `capture_workflow_blocker` (carries the canonical next action), and `monitor_checkin` with six stable cron slugs. - `sentry_sdk` is a lazily-imported optional dependency — the module imports and no-ops cleanly when the package is absent. - **`gitea_mcp_server.py`** (additive, guarded, best-effort): `init_sentry()` in `__main__` before `mcp.run`; exception capture in the `_audited` failure path; blocker capture in the `_audit_pr_result` BLOCKED/FAILED path; allocator + namespace-health watchdog check-ins at their MCP tool sites (domain modules left pure). - **`requirements.txt`** — pin `sentry-sdk==2.20.0` (optional). - **`.env.example`** — document the six env vars. - **`docs/observability/sentry-integration.md`** — project creation in `https://sentry.prgs.cc/`, DSN handling, local/dev/prod config, redaction guarantees, and coexistence with the #612 incident bridge. ## Acceptance criteria mapping 1. Disabled / missing DSN → no init, no events ✓ (`test_init_noop_*`) 2. Enabled + self-hosted DSN → exceptions routed to the DSN's project ✓ (`test_init_configures_sdk_with_scrubber`, `_audited` capture) 3. Workflow blockers as structured events with tags ✓ (`build_blocker_event`, `_audit_pr_result` wiring) 4. Python logs when `MCP_SENTRY_ENABLE_LOGS=1` ✓ (`test_init_enable_logs_wires_log_scrubber`) 5. Redaction proven for secrets/tokens/keychain-ids/config/session-state ✓ (redaction test block) 6. Cron check-ins for the six watchdog jobs ✓ (`MONITOR_SLUGS` + wiring) 7. Events include a canonical next action when available ✓ 8. Tests cover disabled/enabled/missing-DSN/redaction/exception/blocker/cron ✓ 9. Docs for project creation, DSN, and per-env config ✓ 10. Gitea remains canonical ✓ (Sentry observe-only; non-goals documented) ## Non-goals (enforced) Sentry does not become the workflow authority and cannot approve/merge/close/ mutate Gitea state or bypass leases/#332/roles/MCP gates. Alerts may only reach Gitea through the sanctioned #612 incident-bridge path. ## Validation - New suite `tests/test_sentry_observability.py`: 36 passing. - Full suite: **2632 passed, 6 skipped** (no regressions). ## Reviewer handoff - Scope is limited to the six files above; `allocator_service.py` and other domain modules are intentionally untouched (observability wiring lives in the MCP server layer, consistent with `_audit`). - Please confirm the redaction allowlist/scrubber coverage and the guarded/ fail-open placement of the capture calls. This PR complements #612 and does not replace it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jcwalker3 added 1 commit 2026-07-12 01:36:40 -05:00

feat(observability): optional self-hosted Sentry instrumentation for MCP workflow failures (Closes #606 ) 78cc37a977

Add an env-var-gated, off-by-default Sentry SDK integration so MCP runtime
errors, fail-closed workflow blockers, lease/terminal-lock/stale-runtime
collisions, and recurring watchdog check-ins are visible in a self-hosted
Sentry at https://sentry.prgs.cc/. Gitea stays the source of truth; Sentry is
observe-only.

New module `sentry_observability.py` mirrors the `gitea_audit` conventions
(env-gated, best-effort, redacting):

- Config from MCP_SENTRY_ENABLED / SENTRY_DSN / SENTRY_ENVIRONMENT /
  SENTRY_RELEASE / MCP_SENTRY_TRACES_SAMPLE_RATE / MCP_SENTRY_ENABLE_LOGS.
  Active only when enabled AND a DSN is present; otherwise a hard no-op.
- Fail OPEN for observability (never blocks a tool success path) and fail
  CLOSED for redaction (drop a field rather than risk leaking it).
- `scrub_event` before_send/before_send_log hook + allowlisted tags: no
  tokens/passwords/keychain ids/DSNs/cookies, no raw session-state or full
  prompt bodies (session_id -> 12-char hash), no full filesystem paths
  (worktree path -> coarse category). Reuses incident_bridge + gitea_audit
  scrubbers.
- capture_exception, capture_workflow_blocker (with canonical next action),
  and monitor_checkin with six stable cron slugs (stale lease scan, terminal
  lock scan, allocator health, namespace health, dashboard freshness,
  reconciler cleanup).
- `sentry_sdk` is a lazily-imported optional dependency; the module imports
  and no-ops cleanly when the package is absent.

Wiring in gitea_mcp_server.py (additive, guarded, best-effort):
- init_sentry() in __main__ before mcp.run.
- capture_exception in the `_audited` failure path; capture_workflow_blocker
  in `_audit_pr_result` BLOCKED/FAILED path.
- allocator + namespace-health watchdog check-ins at their MCP tool sites
  (domain modules left pure).

Also: pin `sentry-sdk==2.20.0` (optional), document the six env vars in
`.env.example`, and add `docs/observability/sentry-integration.md` covering
project creation in https://sentry.prgs.cc/, DSN handling, local/dev/prod
config, redaction guarantees, and coexistence with the #612 incident bridge.

Tests: tests/test_sentry_observability.py (36 cases) cover disabled / enabled /
missing-DSN / missing-SDK, redaction, exception capture, workflow-blocker
capture, and cron check-in behaviour. Full suite: 2632 passed, 6 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

jcwalker3 referenced this pull request

2026-07-12 01:39:18 -05:00

Integrate self-hosted Sentry observability for MCP workflow collisions and runtime failures #606

jcwalker3 commented

2026-07-12 01:39:51 -05:00

[THREAD STATE LEDGER] PR #679 / Issue #606 — author implementation complete, review requested

What is true now:

PR state: open
PR number: #679 (head feat/issue-606-sentry-observability, base master)
Current head SHA: 78cc37a977
Server-side decision state: no review verdict posted to Gitea yet
Local verdict/state: author implementation complete; PR opened
Latest known validation: new suite 36 passed; full suite 2632 passed, 6 skipped (no regressions)

What changed:

Optional self-hosted Sentry observability added: sentry_observability.py, additive guarded wiring in gitea_mcp_server.py, pinned optional sentry-sdk==2.20.0, six env vars in .env.example, docs/observability/sentry-integration.md, and tests/test_sentry_observability.py

What is blocked:

Blocker classification: no blocker

Who/what acts next:

Next actor: reviewer
Required action: review this PR against #606 acceptance criteria — off-by-default gating, redaction fails closed (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path to category), guarded fail-open capture, six cron monitor slugs wired, Sentry observe-only; then post an APPROVE or REQUEST_CHANGES review verdict to Gitea
Do not do: merge before a posted APPROVE review verdict; treat Sentry as workflow authority; expand scope beyond the six changed files
Resume from: this PR diff and tests/test_sentry_observability.py

Canonical PR State

STATE: implemented; open; awaiting reviewer verdict
WHO_IS_NEXT: reviewer
NEXT_ACTION: review PR #679 against #606 acceptance criteria, then post an APPROVE or REQUEST_CHANGES review verdict to Gitea
NEXT_PROMPT:

Review PR #679 on Scaled-Tech-Consulting/Gitea-Tools (remote prgs; base master, head 78cc37a977aac2d9004d53b9a689ca61b992a9f2) for #606. Verify off-by-default no-op (MCP_SENTRY_ENABLED off or empty SENTRY_DSN => no init/events), fail-closed redaction (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path -> category), guarded fail-open capture in gitea_mcp_server.py, six wired cron monitor slugs, and Sentry observe-only (no Gitea mutation / no lease/#332/role bypass). Run the full pytest suite. Scope: sentry_observability.py, gitea_mcp_server.py, requirements.txt, .env.example, tests/test_sentry_observability.py, docs/observability/sentry-integration.md. Then post an APPROVE or REQUEST_CHANGES verdict.

WHAT_HAPPENED: Implemented optional self-hosted Sentry SDK instrumentation (env-gated, off-by-default, fail-open capture, fail-closed redaction) with cron watchdog check-ins; wiring additive in the MCP server layer; domain modules left pure.
WHY: Workflow collisions and late #332 blockers need structured observability without making Sentry the workflow authority; Gitea remains canonical.
RELATED_PRS: none supersede this. Complements #612 incident bridge; does not replace it.
BLOCKERS: none
VALIDATION: new suite 36 passed; full suite 2632 passed, 6 skipped; scope limited to six files; allocator_service.py untouched.
LAST_UPDATED_BY: jcwalker3 (prgs-author)

[THREAD STATE LEDGER] PR #679 / Issue #606 — author implementation complete, review requested What is true now: - PR state: open - PR number: #679 (head feat/issue-606-sentry-observability, base master) - Current head SHA: 78cc37a977aac2d9004d53b9a689ca61b992a9f2 - Server-side decision state: no review verdict posted to Gitea yet - Local verdict/state: author implementation complete; PR opened - Latest known validation: new suite 36 passed; full suite 2632 passed, 6 skipped (no regressions) What changed: - Optional self-hosted Sentry observability added: `sentry_observability.py`, additive guarded wiring in `gitea_mcp_server.py`, pinned optional `sentry-sdk==2.20.0`, six env vars in `.env.example`, `docs/observability/sentry-integration.md`, and `tests/test_sentry_observability.py` What is blocked: - Blocker classification: no blocker Who/what acts next: - Next actor: reviewer - Required action: review this PR against #606 acceptance criteria — off-by-default gating, redaction fails closed (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path to category), guarded fail-open capture, six cron monitor slugs wired, Sentry observe-only; then post an APPROVE or REQUEST_CHANGES review verdict to Gitea - Do not do: merge before a posted APPROVE review verdict; treat Sentry as workflow authority; expand scope beyond the six changed files - Resume from: this PR diff and `tests/test_sentry_observability.py` ## Canonical PR State STATE: implemented; open; awaiting reviewer verdict WHO_IS_NEXT: reviewer NEXT_ACTION: review PR #679 against #606 acceptance criteria, then post an APPROVE or REQUEST_CHANGES review verdict to Gitea NEXT_PROMPT: ```text Review PR #679 on Scaled-Tech-Consulting/Gitea-Tools (remote prgs; base master, head 78cc37a977aac2d9004d53b9a689ca61b992a9f2) for #606. Verify off-by-default no-op (MCP_SENTRY_ENABLED off or empty SENTRY_DSN => no init/events), fail-closed redaction (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path -> category), guarded fail-open capture in gitea_mcp_server.py, six wired cron monitor slugs, and Sentry observe-only (no Gitea mutation / no lease/#332/role bypass). Run the full pytest suite. Scope: sentry_observability.py, gitea_mcp_server.py, requirements.txt, .env.example, tests/test_sentry_observability.py, docs/observability/sentry-integration.md. Then post an APPROVE or REQUEST_CHANGES verdict. ``` WHAT_HAPPENED: Implemented optional self-hosted Sentry SDK instrumentation (env-gated, off-by-default, fail-open capture, fail-closed redaction) with cron watchdog check-ins; wiring additive in the MCP server layer; domain modules left pure. WHY: Workflow collisions and late #332 blockers need structured observability without making Sentry the workflow authority; Gitea remains canonical. RELATED_PRS: none supersede this. Complements #612 incident bridge; does not replace it. BLOCKERS: none VALIDATION: new suite 36 passed; full suite 2632 passed, 6 skipped; scope limited to six files; `allocator_service.py` untouched. LAST_UPDATED_BY: jcwalker3 (prgs-author)

You are not authorized to merge this pull request.

This pull request can be merged automatically.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/issue-606-sentry-observability:feat/issue-606-sentry-observability

git checkout feat/issue-606-sentry-observability

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#679