feat(observability): optional self-hosted Sentry instrumentation for MCP workflow failures (Closes #606)

Add an env-var-gated, off-by-default Sentry SDK integration so MCP runtime
errors, fail-closed workflow blockers, lease/terminal-lock/stale-runtime
collisions, and recurring watchdog check-ins are visible in a self-hosted
Sentry at https://sentry.prgs.cc/. Gitea stays the source of truth; Sentry is
observe-only.

New module `sentry_observability.py` mirrors the `gitea_audit` conventions
(env-gated, best-effort, redacting):

- Config from MCP_SENTRY_ENABLED / SENTRY_DSN / SENTRY_ENVIRONMENT /
  SENTRY_RELEASE / MCP_SENTRY_TRACES_SAMPLE_RATE / MCP_SENTRY_ENABLE_LOGS.
  Active only when enabled AND a DSN is present; otherwise a hard no-op.
- Fail OPEN for observability (never blocks a tool success path) and fail
  CLOSED for redaction (drop a field rather than risk leaking it).
- `scrub_event` before_send/before_send_log hook + allowlisted tags: no
  tokens/passwords/keychain ids/DSNs/cookies, no raw session-state or full
  prompt bodies (session_id -> 12-char hash), no full filesystem paths
  (worktree path -> coarse category). Reuses incident_bridge + gitea_audit
  scrubbers.
- capture_exception, capture_workflow_blocker (with canonical next action),
  and monitor_checkin with six stable cron slugs (stale lease scan, terminal
  lock scan, allocator health, namespace health, dashboard freshness,
  reconciler cleanup).
- `sentry_sdk` is a lazily-imported optional dependency; the module imports
  and no-ops cleanly when the package is absent.

Wiring in gitea_mcp_server.py (additive, guarded, best-effort):
- init_sentry() in __main__ before mcp.run.
- capture_exception in the `_audited` failure path; capture_workflow_blocker
  in `_audit_pr_result` BLOCKED/FAILED path.
- allocator + namespace-health watchdog check-ins at their MCP tool sites
  (domain modules left pure).

Also: pin `sentry-sdk==2.20.0` (optional), document the six env vars in
`.env.example`, and add `docs/observability/sentry-integration.md` covering
project creation in https://sentry.prgs.cc/, DSN handling, local/dev/prod
config, redaction guarantees, and coexistence with the #612 incident bridge.

Tests: tests/test_sentry_observability.py (36 cases) cover disabled / enabled /
missing-DSN / missing-SDK, redaction, exception capture, workflow-blocker
capture, and cron check-in behaviour. Full suite: 2632 passed, 6 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

gitea_mcp_server.py

@@ -918,6 +918,7 @@ import allocator_service  # noqa: E402
 import control_plane_db  # noqa: E402
 import lease_lifecycle  # noqa: E402
 import incident_bridge  # noqa: E402
+import sentry_observability  # noqa: E402  (#606 optional Sentry observability)
 import agent_temp_artifacts
 import issue_lock_worktree  # noqa: E402
 import issue_lock_provenance  # noqa: E402
@@ -1777,6 +1778,18 @@ def _audited(action: str, *, host, remote, org=None, repo=None,
                result=gitea_audit.FAILED, reason=_redact(str(exc)),
                request_metadata=request_metadata, issue_number=issue_number,
                pr_number=pr_number, target_branch=target_branch)
+        # #606: best-effort Sentry capture of the failing mutation (fail open).
+        sentry_observability.capture_exception(
+            exc,
+            tags={
+                "mutation_tool": action,
+                "remote": remote,
+                "repo": repo,
+                "org": org,
+                "issue_number": issue_number,
+                "pr_number": pr_number,
+            },
+        )
         raise
     _audit(action, host=host, remote=remote, org=org, repo=repo,
            result=gitea_audit.SUCCEEDED, request_metadata=request_metadata,
@@ -1823,6 +1836,20 @@ def _audit_pr_result(action: str):
                             "merge_method": result.get("merge_method"),
                         },
                     )
+                    # #606: surface fail-closed blockers / failed mutations to
+                    # Sentry as structured events (best-effort, fail open).
+                    if status in (gitea_audit.BLOCKED, gitea_audit.FAILED):
+                        sentry_observability.capture_workflow_blocker(
+                            action,
+                            message="; ".join(reasons) or action,
+                            next_action=result.get("safe_next_action"),
+                            level="error" if status == gitea_audit.FAILED else "warning",
+                            tags={
+                                "mutation_tool": action,
+                                "pr_number": result.get("pr_number"),
+                                "current_head_sha": result.get("head_sha"),
+                            },
+                        )
             except Exception:
                 pass  # best-effort; never break the tool
             return result
@@ -9954,6 +9981,11 @@ def gitea_assess_mcp_namespace_health(
         probe_source=probe_source,
     )
     _record_live_namespace_health(result)
+    # #606: namespace-health watchdog check-in (best-effort, fail open).
+    sentry_observability.monitor_checkin(
+        "namespace_health",
+        "ok" if result.get("healthy", result.get("callable", True)) else "error",
+    )
     return result
 
 
@@ -11881,6 +11913,18 @@ def gitea_allocate_next_work(
     result["inventory_source"] = (
         "candidates_json" if candidates_json else "gitea_live"
     )
+    # #606: watchdog check-ins for the recurring jobs this allocator run
+    # performs — global stale-lease expiry, terminal-lock lookup, and the
+    # allocator itself. Best-effort; a failed selection reports "error".
+    _alloc_ok = bool(result.get("success"))
+    sentry_observability.monitor_checkin(
+        "allocator_health", "ok" if _alloc_ok else "error"
+    )
+    if _alloc_ok:
+        # These two scans complete inside allocate_next_work before selection;
+        # a successful result proves both ran.
+        sentry_observability.monitor_checkin("stale_lease_scan", "ok")
+        sentry_observability.monitor_checkin("terminal_lock_scan", "ok")
     return result
 
 
@@ -12205,4 +12249,10 @@ if __name__ == "__main__":
     # processes (e.g. review_pr.py) can detect and refuse profile
     # side-channel overrides (#199).
     _export_session_profile_lock()
+    # #606: optional self-hosted Sentry observability. No-op unless
+    # MCP_SENTRY_ENABLED is truthy and SENTRY_DSN is set; never blocks startup.
+    _sentry_status = sentry_observability.init_sentry()
+    sys.stderr.write(
+        f"--- Sentry observability: {_sentry_status.get('reason')} ---\n"
+    )
     mcp.run(transport="stdio")