Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 73 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

Integrate self-hosted Sentry observability for MCP workflow collisions and runtime failures #606

New Issue
Open
opened 2026-07-09 16:18:05 -05:00 by jcwalker3 · 3 comments
jcwalker3 commented 2026-07-09 16:18:05 -05:00
Owner
Copy Link
Copy Source

Problem

LLM sessions are still stepping on each other by selecting the same issues/PRs, using stale lease prompts, hitting active foreign leases, and discovering #332 terminal-review blockers too late. The project needs observability for these recurring workflow failures.

Sentry server

  • Use self-hosted Sentry at https://sentry.prgs.cc/.
  • Do not assume Sentry Cloud.
  • Sentry should observe failures and workflow events, but Gitea remains the durable source of truth.

Goal

Add optional Sentry SDK instrumentation to the Python MCP server and workflow tools so runtime errors, workflow blockers, logs, traces, stale sessions, lease collisions, and recurring watchdog failures are visible in Sentry and can be routed back into durable Gitea issues/comments.

Configuration

Add env-var controlled setup:

  • MCP_SENTRY_ENABLED
  • SENTRY_DSN
  • SENTRY_ENVIRONMENT
  • SENTRY_RELEASE
  • MCP_SENTRY_TRACES_SAMPLE_RATE
  • MCP_SENTRY_ENABLE_LOGS

Rules:

  • Do not hardcode the DSN.
  • Do not hardcode secrets or auth tokens.
  • Support DSNs pointing at https://sentry.prgs.cc/.

Instrumentation scope

  • MCP tool entry/exit
  • fail-closed blockers
  • active foreign lease
  • stale prompt lease ID
  • #332 terminal lock occupied
  • terminal approval contaminated
  • cleanup refused
  • wrong repo/default repo mismatch
  • stale runtime detected
  • namespace EOF/closed transport
  • root checkout mutation attempt
  • mtime/source-file manipulation detection
  • raw API/direct internals bypass detection
  • review/merge mutation blocked
  • queue allocator duplicate-work prevention
  • worktree binding mismatch

Suggested Sentry tags/context

  • role/profile
  • namespace
  • repo/org
  • issue number
  • PR number
  • blocker type
  • workflow hash
  • session_id hash only
  • PID
  • worktree category, not full sensitive path unless explicitly allowed
  • lease comment ID
  • expected head SHA
  • current head SHA
  • terminal-lock state
  • capability name
  • mutation tool name

Cron monitors

Add Sentry Cron Monitor check-ins for recurring/watchdog jobs:

  • stale lease scan
  • terminal lock scan
  • queue allocator health
  • MCP namespace health
  • dashboard freshness
  • reconciler cleanup job

Redaction requirements

  • no tokens
  • no passwords
  • no keychain IDs
  • no private config contents
  • no raw session-state contents
  • no full prompt bodies by default
  • no full local filesystem paths unless explicitly sanitized/allowed
  • no raw headers containing credentials

Non-goals

  • Sentry must not become the workflow source of truth.
  • Sentry must not directly approve, merge, close, or mutate Gitea workflow state.
  • Sentry must not bypass leases, #332, workflow roles, or Gitea MCP gates.
  • Sentry alerts may trigger a sanctioned Gitea issue/comment path only.

Acceptance criteria

  1. With MCP_SENTRY_ENABLED=0 or missing SENTRY_DSN, behavior is unchanged and no events are sent.
  2. With Sentry enabled and a DSN for https://sentry.prgs.cc/, MCP exceptions appear in the correct Sentry project.
  3. Workflow blockers are captured as structured Sentry events with useful tags.
  4. Python logs can be captured when MCP_SENTRY_ENABLE_LOGS=1.
  5. Redaction tests prove secrets, tokens, keychain IDs, private config contents, and raw session-state contents are not sent.
  6. Cron/check-in monitors exist for stale lease scan, terminal lock scan, allocator health, namespace health, dashboard freshness, and reconciler cleanup.
  7. Sentry events include a canonical next action when available.
  8. Tests cover Sentry disabled, enabled, missing DSN, redaction, exception capture, workflow-blocker capture, and cron check-in behavior.
  9. Documentation explains how to create the gitea-tools-mcp project in https://sentry.prgs.cc/, obtain the DSN, and configure local/dev/prod environments.
  10. Gitea remains the canonical workflow record.

Related coordination work

Complements (does not replace) anti-stomp/workflow hardening issues:

  • #600 work allocator
  • #601 lease lifecycle
  • #602 terminal-lock early check
  • #603 lifecycle labels
  • #604 anti-stomp preflight
  • #605 MCP dashboard

Also note existing glitchtip label/history if any prior error-reporting path exists — prefer a single clear observability path and document coexistence or migration.

Implementation notes

  • Prefer a small sentry_observability (or similar) module with init, capture, redaction, and cron helpers used from MCP entrypoints.
  • Fail open for observability: Sentry outages must not block MCP tool success paths.
  • Fail closed for redaction: if redaction cannot be proven safe, drop the field rather than send it.
## Problem LLM sessions are still stepping on each other by selecting the same issues/PRs, using stale lease prompts, hitting active foreign leases, and discovering #332 terminal-review blockers too late. The project needs observability for these recurring workflow failures. ## Sentry server * Use self-hosted Sentry at `https://sentry.prgs.cc/`. * Do **not** assume Sentry Cloud. * Sentry should observe failures and workflow events, but **Gitea remains the durable source of truth**. ## Goal Add optional Sentry SDK instrumentation to the Python MCP server and workflow tools so runtime errors, workflow blockers, logs, traces, stale sessions, lease collisions, and recurring watchdog failures are visible in Sentry and can be routed back into durable Gitea issues/comments. ## Configuration Add env-var controlled setup: * `MCP_SENTRY_ENABLED` * `SENTRY_DSN` * `SENTRY_ENVIRONMENT` * `SENTRY_RELEASE` * `MCP_SENTRY_TRACES_SAMPLE_RATE` * `MCP_SENTRY_ENABLE_LOGS` Rules: * Do not hardcode the DSN. * Do not hardcode secrets or auth tokens. * Support DSNs pointing at `https://sentry.prgs.cc/`. ## Instrumentation scope * MCP tool entry/exit * fail-closed blockers * active foreign lease * stale prompt lease ID * #332 terminal lock occupied * terminal approval contaminated * cleanup refused * wrong repo/default repo mismatch * stale runtime detected * namespace EOF/closed transport * root checkout mutation attempt * mtime/source-file manipulation detection * raw API/direct internals bypass detection * review/merge mutation blocked * queue allocator duplicate-work prevention * worktree binding mismatch ## Suggested Sentry tags/context * role/profile * namespace * repo/org * issue number * PR number * blocker type * workflow hash * session_id hash only * PID * worktree category, not full sensitive path unless explicitly allowed * lease comment ID * expected head SHA * current head SHA * terminal-lock state * capability name * mutation tool name ## Cron monitors Add Sentry Cron Monitor check-ins for recurring/watchdog jobs: * stale lease scan * terminal lock scan * queue allocator health * MCP namespace health * dashboard freshness * reconciler cleanup job ## Redaction requirements * no tokens * no passwords * no keychain IDs * no private config contents * no raw session-state contents * no full prompt bodies by default * no full local filesystem paths unless explicitly sanitized/allowed * no raw headers containing credentials ## Non-goals * Sentry must **not** become the workflow source of truth. * Sentry must **not** directly approve, merge, close, or mutate Gitea workflow state. * Sentry must **not** bypass leases, #332, workflow roles, or Gitea MCP gates. * Sentry alerts may trigger a **sanctioned** Gitea issue/comment path only. ## Acceptance criteria 1. With `MCP_SENTRY_ENABLED=0` or missing `SENTRY_DSN`, behavior is unchanged and no events are sent. 2. With Sentry enabled and a DSN for `https://sentry.prgs.cc/`, MCP exceptions appear in the correct Sentry project. 3. Workflow blockers are captured as structured Sentry events with useful tags. 4. Python logs can be captured when `MCP_SENTRY_ENABLE_LOGS=1`. 5. Redaction tests prove secrets, tokens, keychain IDs, private config contents, and raw session-state contents are not sent. 6. Cron/check-in monitors exist for stale lease scan, terminal lock scan, allocator health, namespace health, dashboard freshness, and reconciler cleanup. 7. Sentry events include a canonical next action when available. 8. Tests cover Sentry disabled, enabled, missing DSN, redaction, exception capture, workflow-blocker capture, and cron check-in behavior. 9. Documentation explains how to create the `gitea-tools-mcp` project in `https://sentry.prgs.cc/`, obtain the DSN, and configure local/dev/prod environments. 10. Gitea remains the canonical workflow record. ## Related coordination work Complements (does not replace) anti-stomp/workflow hardening issues: * #600 work allocator * #601 lease lifecycle * #602 terminal-lock early check * #603 lifecycle labels * #604 anti-stomp preflight * #605 MCP dashboard Also note existing `glitchtip` label/history if any prior error-reporting path exists — prefer a single clear observability path and document coexistence or migration. ## Implementation notes * Prefer a small `sentry_observability` (or similar) module with init, capture, redaction, and cron helpers used from MCP entrypoints. * Fail open for observability: Sentry outages must not block MCP tool success paths. * Fail closed for redaction: if redaction cannot be proven safe, drop the field rather than send it.
jcwalker3 commented 2026-07-09 16:18:24 -05:00
Author
Owner
Copy Link
Copy Source

Canonical Issue State

STATE: ready for independent implementation
WHO_IS_NEXT: author
NEXT_ACTION: claim this issue and implement optional self-hosted Sentry SDK instrumentation for MCP workflow collisions and runtime failures
NEXT_PROMPT:

Implement #606 on Scaled-Tech-Consulting/Gitea-Tools: optional Sentry observability against https://sentry.prgs.cc/ (not cloud). Env-var controlled (MCP_SENTRY_ENABLED, SENTRY_DSN, SENTRY_ENVIRONMENT, SENTRY_RELEASE, MCP_SENTRY_TRACES_SAMPLE_RATE, MCP_SENTRY_ENABLE_LOGS). Instrument MCP entry/exit, fail-closed blockers, lease/terminal-lock/stale-runtime collisions, cron monitors. Hard redaction (no tokens/keychain/private config/session-state/full prompts/paths by default). Sentry is observe-only; Gitea remains source of truth. Tests for disabled/enabled/missing DSN/redaction/blockers/cron.

WHAT_HAPPENED: Durable issue #606 created for self-hosted Sentry integration. Complements anti-stomp series #600–#605.
WHY: Workflow collisions and late blockers need structured observability without making Sentry the workflow authority.
RELATED_PRS: none yet
BLOCKERS: none
VALIDATION: issue created with type:feature status:ready and labels workflow-hardening,observability,sentry,self-hosted,anti-stomp,mcp-health
LAST_UPDATED_BY: jcwalker3 (prgs-author)

## Canonical Issue State STATE: ready for independent implementation WHO_IS_NEXT: author NEXT_ACTION: claim this issue and implement optional self-hosted Sentry SDK instrumentation for MCP workflow collisions and runtime failures NEXT_PROMPT: ```text Implement #606 on Scaled-Tech-Consulting/Gitea-Tools: optional Sentry observability against https://sentry.prgs.cc/ (not cloud). Env-var controlled (MCP_SENTRY_ENABLED, SENTRY_DSN, SENTRY_ENVIRONMENT, SENTRY_RELEASE, MCP_SENTRY_TRACES_SAMPLE_RATE, MCP_SENTRY_ENABLE_LOGS). Instrument MCP entry/exit, fail-closed blockers, lease/terminal-lock/stale-runtime collisions, cron monitors. Hard redaction (no tokens/keychain/private config/session-state/full prompts/paths by default). Sentry is observe-only; Gitea remains source of truth. Tests for disabled/enabled/missing DSN/redaction/blockers/cron. ``` WHAT_HAPPENED: Durable issue #606 created for self-hosted Sentry integration. Complements anti-stomp series #600–#605. WHY: Workflow collisions and late blockers need structured observability without making Sentry the workflow authority. RELATED_PRS: none yet BLOCKERS: none VALIDATION: issue created with type:feature status:ready and labels workflow-hardening,observability,sentry,self-hosted,anti-stomp,mcp-health LAST_UPDATED_BY: jcwalker3 (prgs-author)
jcwalker3 added status:in-progress and removed status:ready labels 2026-07-12 01:19:07 -05:00
jcwalker3 commented 2026-07-12 01:19:07 -05:00
Author
Owner
Copy Link
Copy Source

Issue claim heartbeat

<!-- gitea-issue-claim-heartbeat:v1 --> **Issue claim heartbeat** - kind: claim - issue: #606 - branch: feat/issue-606-sentry-observability - phase: claimed - profile: prgs-author - pr: none - blocker: none - next_action: create worktree and begin implementation
jcwalker3 added status:pr-open and removed status:in-progress labels 2026-07-12 01:36:41 -05:00
jcwalker3 commented 2026-07-12 01:39:18 -05:00
Author
Owner
Copy Link
Copy Source

[THREAD STATE LEDGER] Issue #606 / PR #679 — implementation complete, review requested

What is true now:

  • PR state: open
  • PR number: #679 (head feat/issue-606-sentry-observability, base master)
  • Current head SHA: 78cc37a977
  • Server-side decision state: no review verdict posted to Gitea yet
  • Local verdict/state: author implementation complete; PR opened
  • Latest known validation: new suite 36 passed; full suite 2632 passed, 6 skipped (no regressions)

What changed:

  • Added optional self-hosted Sentry observability (sentry_observability.py), additive guarded wiring in gitea_mcp_server.py, pinned optional sentry-sdk==2.20.0, documented six env vars in .env.example, added docs/observability/sentry-integration.md, and regression tests tests/test_sentry_observability.py
  • Issue advanced to status:pr-open via PR #679

What is blocked:

  • Blocker classification: no blocker

Who/what acts next:

  • Next actor: reviewer
  • Required action: review PR #679 against #606 acceptance criteria — off-by-default gating, redaction fails closed (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path to category), guarded fail-open capture, six cron monitor slugs wired, Sentry observe-only; then post an APPROVE or REQUEST_CHANGES review verdict to Gitea
  • Do not do: merge before a posted APPROVE review verdict; treat Sentry as workflow authority; expand scope beyond the six changed files
  • Resume from: PR #679 diff and tests/test_sentry_observability.py

Canonical Issue State

STATE: implemented; PR #679 open; awaiting reviewer verdict
WHO_IS_NEXT: reviewer
NEXT_ACTION: review PR #679 against #606 acceptance criteria, then post an APPROVE or REQUEST_CHANGES review verdict to Gitea
NEXT_PROMPT:

Review PR #679 on Scaled-Tech-Consulting/Gitea-Tools (remote prgs; branch feat/issue-606-sentry-observability, base master, head 78cc37a977aac2d9004d53b9a689ca61b992a9f2) implementing #606 optional self-hosted Sentry observability. Verify: (1) with MCP_SENTRY_ENABLED off or SENTRY_DSN empty it is a hard no-op (no init, no events); (2) redaction fails closed — no tokens/passwords/keychain-ids/DSNs/cookies/session-state/full prompt bodies/full filesystem paths; session_id surfaced only as a 12-char hash; worktree path collapses to a category; (3) capture calls in gitea_mcp_server.py are guarded and fail open (never break a tool success path); (4) all six cron monitor slugs exist and are wired; (5) Sentry stays observe-only (no Gitea mutation, no lease/#332/role bypass). Run the full pytest suite. Scope is limited to sentry_observability.py, gitea_mcp_server.py, requirements.txt, .env.example, tests/test_sentry_observability.py, docs/observability/sentry-integration.md. Then post an APPROVE or REQUEST_CHANGES verdict to Gitea.

WHAT_HAPPENED: Implemented optional self-hosted Sentry SDK instrumentation (env-gated, off-by-default, fail-open capture, fail-closed redaction) with cron watchdog check-ins; wiring is additive in the MCP server layer; domain modules left pure.
WHY: Workflow collisions and late #332 blockers need structured observability without making Sentry the workflow authority; Gitea remains canonical.
RELATED_PRS: #679 (this work). Complements #612 incident bridge; does not replace it.
BLOCKERS: none
VALIDATION: new suite 36 passed; full suite 2632 passed, 6 skipped; scope limited to six files; allocator_service.py untouched.
LAST_UPDATED_BY: jcwalker3 (prgs-author)

[THREAD STATE LEDGER] Issue #606 / PR #679 — implementation complete, review requested What is true now: - PR state: open - PR number: #679 (head feat/issue-606-sentry-observability, base master) - Current head SHA: 78cc37a977aac2d9004d53b9a689ca61b992a9f2 - Server-side decision state: no review verdict posted to Gitea yet - Local verdict/state: author implementation complete; PR opened - Latest known validation: new suite 36 passed; full suite 2632 passed, 6 skipped (no regressions) What changed: - Added optional self-hosted Sentry observability (`sentry_observability.py`), additive guarded wiring in `gitea_mcp_server.py`, pinned optional `sentry-sdk==2.20.0`, documented six env vars in `.env.example`, added `docs/observability/sentry-integration.md`, and regression tests `tests/test_sentry_observability.py` - Issue advanced to status:pr-open via PR #679 What is blocked: - Blocker classification: no blocker Who/what acts next: - Next actor: reviewer - Required action: review PR #679 against #606 acceptance criteria — off-by-default gating, redaction fails closed (no tokens/keychain/DSN/session-state/full prompts/full paths; session_id hashed; worktree path to category), guarded fail-open capture, six cron monitor slugs wired, Sentry observe-only; then post an APPROVE or REQUEST_CHANGES review verdict to Gitea - Do not do: merge before a posted APPROVE review verdict; treat Sentry as workflow authority; expand scope beyond the six changed files - Resume from: PR #679 diff and `tests/test_sentry_observability.py` ## Canonical Issue State STATE: implemented; PR #679 open; awaiting reviewer verdict WHO_IS_NEXT: reviewer NEXT_ACTION: review PR #679 against #606 acceptance criteria, then post an APPROVE or REQUEST_CHANGES review verdict to Gitea NEXT_PROMPT: ```text Review PR #679 on Scaled-Tech-Consulting/Gitea-Tools (remote prgs; branch feat/issue-606-sentry-observability, base master, head 78cc37a977aac2d9004d53b9a689ca61b992a9f2) implementing #606 optional self-hosted Sentry observability. Verify: (1) with MCP_SENTRY_ENABLED off or SENTRY_DSN empty it is a hard no-op (no init, no events); (2) redaction fails closed — no tokens/passwords/keychain-ids/DSNs/cookies/session-state/full prompt bodies/full filesystem paths; session_id surfaced only as a 12-char hash; worktree path collapses to a category; (3) capture calls in gitea_mcp_server.py are guarded and fail open (never break a tool success path); (4) all six cron monitor slugs exist and are wired; (5) Sentry stays observe-only (no Gitea mutation, no lease/#332/role bypass). Run the full pytest suite. Scope is limited to sentry_observability.py, gitea_mcp_server.py, requirements.txt, .env.example, tests/test_sentry_observability.py, docs/observability/sentry-integration.md. Then post an APPROVE or REQUEST_CHANGES verdict to Gitea. ``` WHAT_HAPPENED: Implemented optional self-hosted Sentry SDK instrumentation (env-gated, off-by-default, fail-open capture, fail-closed redaction) with cron watchdog check-ins; wiring is additive in the MCP server layer; domain modules left pure. WHY: Workflow collisions and late #332 blockers need structured observability without making Sentry the workflow authority; Gitea remains canonical. RELATED_PRS: #679 (this work). Complements #612 incident bridge; does not replace it. BLOCKERS: none VALIDATION: new suite 36 passed; full suite 2632 passed, 6 skipped; scope limited to six files; `allocator_service.py` untouched. LAST_UPDATED_BY: jcwalker3 (prgs-author)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels anti-stomp mcp-health observability self-hosted sentry status:pr-open type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#606