Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f2dbf30e81 | ||
|
|
2b4e43042a | ||
|
|
0f9390aab4 | ||
|
|
d7ad2838ec | ||
|
|
c6d68dbc7b | ||
|
|
c83a10d7c2 | ||
|
|
71031c812e | ||
|
|
e43ddd3cbe | ||
|
|
a64ba08e27 | ||
|
|
bb8c3a537b | ||
|
|
59aab06fe1 | ||
|
|
4f06d30e07 |
@@ -1568,6 +1568,32 @@ class ControlPlaneDB:
|
|||||||
).fetchone()
|
).fetchone()
|
||||||
return dict(row) if row else None
|
return dict(row) if row else None
|
||||||
|
|
||||||
|
def list_incident_links(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
provider: str | None = None,
|
||||||
|
gitea_org: str | None = None,
|
||||||
|
gitea_repo: str | None = None,
|
||||||
|
limit: int = 100,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""List stored incident_links rows, optionally filtered by provider/repo (#612 / #649)."""
|
||||||
|
query = "SELECT * FROM incident_links WHERE 1=1"
|
||||||
|
params: list[Any] = []
|
||||||
|
if provider:
|
||||||
|
query += " AND provider = ?"
|
||||||
|
params.append(provider.strip().lower())
|
||||||
|
if gitea_org:
|
||||||
|
query += " AND gitea_org = ?"
|
||||||
|
params.append(_norm_scope(gitea_org))
|
||||||
|
if gitea_repo:
|
||||||
|
query += " AND gitea_repo = ?"
|
||||||
|
params.append(_norm_scope(gitea_repo))
|
||||||
|
query += " ORDER BY link_id DESC LIMIT ?"
|
||||||
|
params.append(max(1, limit))
|
||||||
|
with self._tx(immediate=False) as conn:
|
||||||
|
rows = conn.execute(query, params).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
|
||||||
|
|
||||||
# ── lease lifecycle (#601) ────────────────────────────────────────────
|
# ── lease lifecycle (#601) ────────────────────────────────────────────
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Incident #670: bare direct-to-master commit `2fa97c26` (retroactive audit)
|
||||||
|
|
||||||
|
Status: verified; disposition recommendation: **accept as-is, no revert** (final
|
||||||
|
disposition owned by controller per issue #670).
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Commit `2fa97c26fbda555a1a83930ca5fdcea9d8e47b50`
|
||||||
|
(`fix(mcp): load dotenv relative to project root`) landed on `prgs/master`
|
||||||
|
as a single-parent commit with no PR wrapper and no review record, bypassing
|
||||||
|
the sanctioned issue → branch → PR → review → merge workflow. It was
|
||||||
|
discovered during the PR #654 post-merge audit. PR #654 itself merged
|
||||||
|
cleanly via the Gitea API and did **not** introduce this commit.
|
||||||
|
|
||||||
|
## Verification evidence (acceptance criteria 1–3)
|
||||||
|
|
||||||
|
- **AC1 — present on `prgs/master`: yes.**
|
||||||
|
`git merge-base --is-ancestor 2fa97c26fbda555a1a83930ca5fdcea9d8e47b50 prgs/master` → true.
|
||||||
|
- **AC2 — no PR or review record: confirmed.**
|
||||||
|
The commit is a single-parent, non-merge commit sitting directly on
|
||||||
|
first-parent master between the #629 merge (`5ab5fe85`) and the #654
|
||||||
|
merge (`ec903b0d`). A PR landing on master produces a merge commit (or a
|
||||||
|
PR-linked head); neither exists here. The controller audit at issue-create
|
||||||
|
time also found no PR wrapper and no review record for this SHA.
|
||||||
|
- **AC3 — changed files and diff summary: confirmed.**
|
||||||
|
`gitea_auth.py | 5 +++--` (+3/−2). Single parent
|
||||||
|
`5ab5fe8583c07134d55dadf09381aecb67df246e`. The change moves
|
||||||
|
`PROJECT_ROOT` derivation above `load_dotenv()` and loads
|
||||||
|
`.env` relative to the project root instead of the process CWD.
|
||||||
|
|
||||||
|
## AC4 — why no immediate revert
|
||||||
|
|
||||||
|
- The dotenv fix is intentional and required for correct runtime behavior:
|
||||||
|
without it, `load_dotenv()` resolves `.env` against the process working
|
||||||
|
directory, which breaks MCP server launches whose CWD is not the project
|
||||||
|
root.
|
||||||
|
- The change is small (+3/−2), self-contained in `gitea_auth.py`, and has
|
||||||
|
been running on master without incident since 2026-07-10.
|
||||||
|
- Reverting would re-introduce a real bug to remove a provenance defect —
|
||||||
|
the wrong trade. Provenance is repaired retroactively by this document,
|
||||||
|
issue #670, and the hardening landed under #671.
|
||||||
|
- If the controller later judges the change unsafe, a separate
|
||||||
|
revert/repair issue is the sanctioned path (issue #670, recommended
|
||||||
|
disposition option 4).
|
||||||
|
|
||||||
|
## AC5 — workflow-hardening linkage
|
||||||
|
|
||||||
|
Prevention already landed: **issue #671** (closed)
|
||||||
|
*“Block direct pushes to stable branches from MCP workflow sessions”*,
|
||||||
|
implemented by commit `5933d87647656643a67a50331c4c7b06ea751dad`
|
||||||
|
(`feat(guard): block direct stable-branch pushes from MCP workflow sessions`).
|
||||||
|
|
||||||
|
Shipped guardrails include:
|
||||||
|
|
||||||
|
- `gitea_record_stable_branch_push_attempt` — classifies proposed commands
|
||||||
|
for direct stable-branch push intent (`git push <remote> master`,
|
||||||
|
refspecs, `HEAD:master`, `--force`, dry-run intent, `:master` delete),
|
||||||
|
plus root/control-checkout local commits not carried by an issue branch,
|
||||||
|
and writes a durable `stable_branch_contamination` marker.
|
||||||
|
- `gitea_audit_stable_branch_contamination` — reconciler-only audit/clear
|
||||||
|
path; a contaminated worker session cannot self-clear.
|
||||||
|
- Review/merge/close/completion mutations fail closed while a
|
||||||
|
contamination marker is active.
|
||||||
|
|
||||||
|
## AC6 — PR #654 was not the source
|
||||||
|
|
||||||
|
- `2fa97c26` is the **first parent** of the #654 merge commit
|
||||||
|
`ec903b0d619e7a27d24aed272a890f4e5d381411`; it predates the #654 merge.
|
||||||
|
- First-parent history `5ab5fe8..ec903b0`:
|
||||||
|
`2fa97c2 fix(mcp): load dotenv relative to project root` followed by
|
||||||
|
`ec903b0 Merge pull request 'feat: lifecycle role/hazard labels ... (#603)' (#654)`.
|
||||||
|
- The #654 merger audit confirmed `ec903b0d` was a valid Gitea-API merge,
|
||||||
|
the `git push prgs master` attempt during that run was a no-op, and the
|
||||||
|
net change `2fa97c2..ec903b0` contained only the reviewed #603
|
||||||
|
lifecycle-label files.
|
||||||
|
- Conclusion: #654 merged reviewed content only; the unauthorized-path
|
||||||
|
defect is solely the earlier bare commit `2fa97c26`.
|
||||||
|
|
||||||
|
## Explicit non-actions (unchanged by this audit)
|
||||||
|
|
||||||
|
- No revert of `2fa97c26`.
|
||||||
|
- No force-push or history rewrite.
|
||||||
|
- No master mutation from the audit session.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Web Console: Sentry/GlitchTip Observability & Incident Bridge Console (#649)
|
||||||
|
|
||||||
|
This document describes the Phase 4 observability console surface integrated into the MCP Control Plane Web Console (`webui/`), backed by the #612 incident bridge and the #613 control-plane DB substrate.
|
||||||
|
|
||||||
|
## Architectural Authority Model (ADR Alignment)
|
||||||
|
|
||||||
|
Per the Web Console Architecture ADR (`docs/architecture/webui-control-plane-console-architecture-adr.md`):
|
||||||
|
|
||||||
|
| Layer | Responsibility | Authority |
|
||||||
|
|---|---|---|
|
||||||
|
| **Gitea** | Durable work record | Issues, PRs, comments, reviews, labels, merges |
|
||||||
|
| **Control-plane DB** | Live coordination & linkage | `incident_links` table, session leases, allocations |
|
||||||
|
| **Sentry / GlitchTip** | Observability input | Unresolved incidents, error events, stack traces |
|
||||||
|
| **Incident Bridge (#612)** | Reconciliation engine | Reconciles provider observations into Gitea issues |
|
||||||
|
| **Web Console (`webui/`)** | Read-only projection & gated actions | Projects connection health & correlation links; gates writes |
|
||||||
|
|
||||||
|
> **Key Rule:** Raw monitoring incidents are **never** assignable control-plane `work_items`. They remain observation input only.
|
||||||
|
|
||||||
|
## Redaction Boundary Invariants
|
||||||
|
|
||||||
|
1. **No secrets in returns or rendering:** Auth tokens (`SENTRY_AUTH_TOKEN`, `GLITCHTIP_AUTH_TOKEN`), DSNs, `Authorization` headers, and sensitive local file paths are passed through `webui.console_redaction` before leaving the server.
|
||||||
|
2. **Safe projection:** Connection objects report `credentials_present: true/false` rather than exposing raw keys or headers.
|
||||||
|
|
||||||
|
## Console Endpoints
|
||||||
|
|
||||||
|
- **HTML Surface:** `GET /observability` — Renders provider connection cards, error correlation tables, and gated reconcile controls.
|
||||||
|
- **Versioned API:** `GET /api/v1/observability` — Returns structured JSON snapshot with `schema_version`, `providers`, `links`, and `metrics`.
|
||||||
|
- **Legacy Compatibility Alias:** `GET /api/observability` — Read-only compatibility alias for Phase 4.
|
||||||
|
|
||||||
|
## Gated Actions
|
||||||
|
|
||||||
|
- `observability_reconcile_incident` (`gitea_observability_reconcile_incident`): Triggers or previews dry-run issue reconciliation for a provider incident.
|
||||||
|
- `observability_link_issue` (`gitea_observability_link_issue`): Links a provider incident to an existing Gitea tracking issue.
|
||||||
|
|
||||||
|
Both actions require `operator` role and gate through `task_capability_map`. Execution fails closed in read-only MVP mode.
|
||||||
@@ -98,6 +98,8 @@ already define, and a regression test asserts each mapping matches.
|
|||||||
| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
|
| `system.rebind_session_worktree` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
|
||||||
| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
|
| `system.reconcile_cleanups` | controller | privileged | `gitea.pr.close` | Yes | No | No | 2 |
|
||||||
| `initiate_workflow` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
|
| `initiate_workflow` | operator | gated_write | `gitea.read` | Yes | No | No | 2 |
|
||||||
|
| `observability_reconcile_incident` | operator | gated_write | `gitea.read` | Yes | No | No | 4 |
|
||||||
|
| `observability_link_issue` | operator | gated_write | `gitea.read` | Yes | No | No | 4 |
|
||||||
|
|
||||||
**Dual control** means the acting principal may not be the sole authority: a
|
**Dual control** means the acting principal may not be the sole authority: a
|
||||||
second distinct principal must confirm. **Break-glass** means the action is
|
second distinct principal must confirm. **Break-glass** means the action is
|
||||||
|
|||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Web Console: Notifications & Human-Attention Routing (#648)
|
||||||
|
|
||||||
|
- **Status:** Phase 3 Live
|
||||||
|
- **Tracking Issue:** [#648](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/648)
|
||||||
|
- **Parent Epic:** [#631](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/631)
|
||||||
|
- **Attention Boundary Reference:** [#628](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/628)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Overview
|
||||||
|
|
||||||
|
The **Notifications & Human-Attention Console** (`/notifications`, `/api/v1/notifications`) provides intelligent event classification and human-attention routing for autonomous workflow operations.
|
||||||
|
|
||||||
|
To prevent alert fatigue while ensuring critical escalation boundaries are never missed, events are classified into three distinct **Attention Classes**:
|
||||||
|
|
||||||
|
1. **`human-required`** (Urgent Escalation Boundary):
|
||||||
|
- Items requiring immediate human intervention or business decisions.
|
||||||
|
- Triggers: Auth failures, hard stops, irrecoverable state, decision locks, failed report validations, critical probe errors.
|
||||||
|
- Display: Highlighted in red (`badge-blocked`) with a `HUMAN REQUIRED` badge.
|
||||||
|
|
||||||
|
2. **`operator`** (Operational Inbox):
|
||||||
|
- Items requiring controller or operator review/triage during routine execution.
|
||||||
|
- Triggers: Blocked PRs (merge conflicts), stale leases, duplicate PRs on issues, unassigned ready work.
|
||||||
|
- Display: Displayed in orange/yellow (`badge-claimed`).
|
||||||
|
|
||||||
|
3. **`routine`** (Background Workflow Transitions):
|
||||||
|
- Normal, healthy workflow transitions and state progressions.
|
||||||
|
- Triggers: Active PRs/issues in standard state, clean branch creation, routine heartbeats.
|
||||||
|
- Display: Filtered out of default inbox views to eliminate notification spam; viewable on demand via the "Routine" or "All" tab.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. API Endpoints
|
||||||
|
|
||||||
|
### `GET /api/v1/notifications`
|
||||||
|
*Compatibility Alias:* `GET /api/notifications`
|
||||||
|
|
||||||
|
#### Query Parameters:
|
||||||
|
- `project_id` (optional): Filter notifications by project ID.
|
||||||
|
- `attention_class` (optional): `inbox` (default: human-required + operator), `human-required`, `operator`, `routine`, `all`.
|
||||||
|
|
||||||
|
#### Example JSON Response:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"project_id": "gitea-tools",
|
||||||
|
"repo_label": "Scaled-Tech-Consulting/Gitea-Tools",
|
||||||
|
"human_required_count": 0,
|
||||||
|
"operator_count": 2,
|
||||||
|
"routine_count": 5,
|
||||||
|
"total_count": 7,
|
||||||
|
"fetch_error": null,
|
||||||
|
"inbox_items": [
|
||||||
|
{
|
||||||
|
"id": "notif-pr-block-742",
|
||||||
|
"attention_class": "operator",
|
||||||
|
"category": "blocker",
|
||||||
|
"title": "Blocked PR #742",
|
||||||
|
"summary": "PR #742 requires merge conflict resolution.",
|
||||||
|
"work_kind": "pr",
|
||||||
|
"work_number": 742,
|
||||||
|
"project_id": "gitea-tools",
|
||||||
|
"repo_label": "Scaled-Tech-Consulting/Gitea-Tools",
|
||||||
|
"created_at": "2026-07-25T16:39:47Z",
|
||||||
|
"deep_link": "/traffic",
|
||||||
|
"requires_human": false,
|
||||||
|
"extra": {}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"all_items": [...]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. UI Navigation
|
||||||
|
|
||||||
|
- Access via the **Traffic** navigation menu: **Traffic → Notifications**.
|
||||||
|
- The main view displays:
|
||||||
|
- **Metrics Summary Bar**: Highlighting counts for Human Required, Operator Inbox, and Routine items.
|
||||||
|
- **Attention Filter Tabs**: Toggle between Inbox (Human + Operator), Human Required, Operator, Routine, and All.
|
||||||
|
- **Structured Event Table**: Displays category, title, summary, work item links, and timestamps.
|
||||||
@@ -0,0 +1,478 @@
|
|||||||
|
"""Concurrent-session MCP restart safety & dogfooding test suite (#666).
|
||||||
|
|
||||||
|
Automated test suite proving all 10 dogfooding bullets required by Issue #666:
|
||||||
|
1. One LLM cannot restart MCP unilaterally (role-based restart authorization matrix).
|
||||||
|
2. New work stops during drain (assignments_stopped gate enforcement).
|
||||||
|
3. Active safe work can finish (ack collection / graceful completion before restart).
|
||||||
|
4. Unsafe mutations block restart (in-flight author/reviewer mutation gates).
|
||||||
|
5. Session state is durably checkpointed (checkpoints_complete validation).
|
||||||
|
6. Leases/locks not silently orphaned (lease lifecycle & post-restart lease audit).
|
||||||
|
7. Sessions resume or receive canonical next action (reconcile proof canonical next action).
|
||||||
|
8. Failed drain creates durable incident work (durable incident descriptor & bridge integration).
|
||||||
|
9. Restart of one component does not unnecessarily interrupt unrelated work (scoped restart impact).
|
||||||
|
10. Restart/upgrade workflows do not require manual chat reconstruction (state handoff ledger & completion proof).
|
||||||
|
|
||||||
|
Links parent #655, vision #652, roadmap #653, #658, #659, #660, #661, #662, #663.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import unittest
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import drain_proof as dp
|
||||||
|
import mcp_restart_paths as rp
|
||||||
|
import post_restart_reconcile as prr
|
||||||
|
import restart_coordinator as rc
|
||||||
|
from restart_coordinator import RestartClass
|
||||||
|
|
||||||
|
NOW = datetime(2026, 7, 25, 12, 0, 0, tzinfo=timezone.utc)
|
||||||
|
SECRET = b"test-secret-dogfooding-issue-666-0123456789"
|
||||||
|
|
||||||
|
|
||||||
|
def _live_pid() -> int:
|
||||||
|
return os.getpid()
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_drain_state() -> dict:
|
||||||
|
return {
|
||||||
|
"assignments_stopped": True,
|
||||||
|
"checkpoints_complete": True,
|
||||||
|
"handoffs_verified": True,
|
||||||
|
"leases_handled": True,
|
||||||
|
"acks": {},
|
||||||
|
"ack_timeout_policy_applied": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_inventory() -> dict:
|
||||||
|
return {
|
||||||
|
"service_health": {"healthy": True},
|
||||||
|
"clients": [],
|
||||||
|
"sessions": [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-controller-1",
|
||||||
|
"role": "controller",
|
||||||
|
"profile": "prgs-controller",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"checkpoints": [],
|
||||||
|
"leases": [],
|
||||||
|
"capabilities": {},
|
||||||
|
"worktree_bindings": [],
|
||||||
|
"pending_mutations": [],
|
||||||
|
"inventory_complete": True,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet1UnilateralRestartForbidden(unittest.TestCase):
|
||||||
|
"""Bullet 1: One LLM cannot restart MCP unilaterally."""
|
||||||
|
|
||||||
|
def test_worker_role_unilateral_full_restart_denied(self):
|
||||||
|
policy = rc.RESTART_CLASS_POLICIES[RestartClass.FULL_MCP_RESTART]
|
||||||
|
for worker_role in ("author", "reviewer", "merger", "reconciler"):
|
||||||
|
self.assertNotIn(
|
||||||
|
worker_role,
|
||||||
|
policy.request_roles,
|
||||||
|
f"Worker role '{worker_role}' must not unilaterally authorize FULL_MCP_RESTART",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_privileged_role_full_restart_authorized(self):
|
||||||
|
policy = rc.RESTART_CLASS_POLICIES[RestartClass.FULL_MCP_RESTART]
|
||||||
|
for priv_role in ("controller", "operator", "admin"):
|
||||||
|
self.assertIn(
|
||||||
|
priv_role,
|
||||||
|
policy.request_roles,
|
||||||
|
f"Privileged role '{priv_role}' must be authorized for FULL_MCP_RESTART",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_evaluate_impact_records_unauthorized_worker_request(self):
|
||||||
|
report = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": [], "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
restart_class=RestartClass.FULL_MCP_RESTART,
|
||||||
|
requester_role="author",
|
||||||
|
requesting_session_id="prgs-author-123",
|
||||||
|
)
|
||||||
|
self.assertFalse(report.role_authorized)
|
||||||
|
self.assertEqual(report.verdict, rc.VERDICT_UNSAFE)
|
||||||
|
self.assertTrue(any("may not request" in r.lower() or "authorization denied" in r.lower() for r in report.reasons))
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet2NewWorkStopsDuringDrain(unittest.TestCase):
|
||||||
|
"""Bullet 2: New work stops during drain."""
|
||||||
|
|
||||||
|
def test_assignments_stopped_false_blocks_drain_proof(self):
|
||||||
|
state = _clean_drain_state()
|
||||||
|
state["assignments_stopped"] = False
|
||||||
|
|
||||||
|
impact = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": [], "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
).as_dict()
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=impact,
|
||||||
|
drain_state=state,
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(proof.clean)
|
||||||
|
check = next(c for c in proof.checks if c.name == dp.CHECK_ASSIGNMENTS_STOPPED)
|
||||||
|
self.assertFalse(check.passed)
|
||||||
|
|
||||||
|
gate = dp.gate_apply_restart(proof=proof.as_dict(), secret=SECRET, now=NOW)
|
||||||
|
self.assertEqual(gate.verdict, dp.GATE_DENY)
|
||||||
|
self.assertFalse(gate.allow)
|
||||||
|
self.assertTrue(any("drain proof invalid" in r.lower() or "assignments_stopped" in r.lower() for r in gate.reasons))
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet3ActiveSafeWorkCanFinish(unittest.TestCase):
|
||||||
|
"""Bullet 3: Active safe work can finish."""
|
||||||
|
|
||||||
|
def test_active_safe_sessions_ack_allows_clean_drain(self):
|
||||||
|
sessions = [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-controller-1",
|
||||||
|
"role": "controller",
|
||||||
|
"profile": "prgs-controller",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"session_id": "prgs-reviewer-42",
|
||||||
|
"role": "reviewer",
|
||||||
|
"profile": "prgs-reviewer",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
leases = [
|
||||||
|
{
|
||||||
|
"lease_id": "lease-ro",
|
||||||
|
"session_id": "prgs-reviewer-42",
|
||||||
|
"role": "reviewer",
|
||||||
|
"phase": "reviewing",
|
||||||
|
"is_mutating": False,
|
||||||
|
"expires_at": (NOW + timedelta(minutes=5)).isoformat(),
|
||||||
|
"pid": _live_pid(),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
impact = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": sessions, "leases": leases, "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
requesting_session_id="prgs-controller-1",
|
||||||
|
).as_dict()
|
||||||
|
|
||||||
|
state = _clean_drain_state()
|
||||||
|
state["acks"] = {"prgs-reviewer-42": "ack"}
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=impact,
|
||||||
|
drain_state=state,
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertTrue(proof.clean)
|
||||||
|
gate = dp.gate_apply_restart(proof=proof.as_dict(), secret=SECRET, now=NOW)
|
||||||
|
self.assertTrue(gate.allow)
|
||||||
|
self.assertEqual(gate.verdict, dp.GATE_ALLOW)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet4UnsafeMutationsBlockRestart(unittest.TestCase):
|
||||||
|
"""Bullet 4: Unsafe mutations block restart."""
|
||||||
|
|
||||||
|
def test_inflight_unsafe_mutation_yields_unsafe_verdict(self):
|
||||||
|
sessions = [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-controller-1",
|
||||||
|
"role": "controller",
|
||||||
|
"profile": "prgs-controller",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"session_id": "prgs-author-99",
|
||||||
|
"role": "author",
|
||||||
|
"profile": "prgs-author",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
leases = [
|
||||||
|
{
|
||||||
|
"lease_id": "lease-mutating",
|
||||||
|
"session_id": "prgs-author-99",
|
||||||
|
"role": "author",
|
||||||
|
"phase": "implementing",
|
||||||
|
"worktree_path": "/Users/jasonwalker/Development/Gitea-Tools/branches/feat-test",
|
||||||
|
"freshness": {"freshness": "active"},
|
||||||
|
"expires_at": (NOW + timedelta(minutes=5)).isoformat(),
|
||||||
|
"pid": _live_pid(),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
report = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": sessions, "leases": leases, "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
requesting_session_id="prgs-controller-1",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(report.verdict, rc.VERDICT_UNSAFE)
|
||||||
|
self.assertFalse(report.allow_restart)
|
||||||
|
self.assertGreater(len(report.mutations), 0)
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=report.as_dict(),
|
||||||
|
drain_state=_clean_drain_state(),
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(proof.clean)
|
||||||
|
check = next(c for c in proof.checks if c.name == dp.CHECK_NO_INFLIGHT_MUTATIONS)
|
||||||
|
self.assertFalse(check.passed)
|
||||||
|
|
||||||
|
gate = dp.gate_apply_restart(proof=proof.as_dict(), secret=SECRET, now=NOW)
|
||||||
|
self.assertEqual(gate.verdict, dp.GATE_DENY)
|
||||||
|
self.assertFalse(gate.allow)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet5DurableSessionCheckpoints(unittest.TestCase):
|
||||||
|
"""Bullet 5: Session state is durably checkpointed."""
|
||||||
|
|
||||||
|
def test_incomplete_checkpoints_blocks_drain_proof(self):
|
||||||
|
state = _clean_drain_state()
|
||||||
|
state["checkpoints_complete"] = False
|
||||||
|
|
||||||
|
impact = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": [], "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
).as_dict()
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=impact,
|
||||||
|
drain_state=state,
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(proof.clean)
|
||||||
|
check = next(c for c in proof.checks if c.name == dp.CHECK_CHECKPOINTS_COMPLETE)
|
||||||
|
self.assertFalse(check.passed)
|
||||||
|
|
||||||
|
def test_post_restart_reconcile_audits_checkpoint_dimension(self):
|
||||||
|
inv = _clean_inventory()
|
||||||
|
inv["checkpoints_available"] = True
|
||||||
|
inv["checkpoints"] = [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-author-99",
|
||||||
|
"checkpoint_id": "chk-1",
|
||||||
|
"stale": True,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
proof = prr.reconcile_after_restart(inv, now=NOW, mode=prr.MODE_ENFORCE)
|
||||||
|
chk_item = next(i for i in proof.items if i.dimension == prr.DIM_CHECKPOINTS)
|
||||||
|
self.assertIn(chk_item.status, (prr.ITEM_UNRESOLVED, prr.ITEM_DEGRADED, prr.ITEM_SKIPPED))
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet6LeasesNotSilentlyOrphaned(unittest.TestCase):
|
||||||
|
"""Bullet 6: Leases/locks not silently orphaned."""
|
||||||
|
|
||||||
|
def test_unhandled_leases_block_drain_proof(self):
|
||||||
|
state = _clean_drain_state()
|
||||||
|
state["leases_handled"] = False
|
||||||
|
|
||||||
|
impact = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": [], "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
).as_dict()
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=impact,
|
||||||
|
drain_state=state,
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse(proof.clean)
|
||||||
|
check = next(c for c in proof.checks if c.name == dp.CHECK_LEASES_HANDLED)
|
||||||
|
self.assertFalse(check.passed)
|
||||||
|
|
||||||
|
def test_post_restart_reconcile_audits_all_leases(self):
|
||||||
|
inv = _clean_inventory()
|
||||||
|
inv["leases"] = [
|
||||||
|
{
|
||||||
|
"lease_id": "lease-orphaned-1",
|
||||||
|
"session_id": "prgs-author-dead",
|
||||||
|
"role": "author",
|
||||||
|
"status": "active",
|
||||||
|
"freshness": "expired",
|
||||||
|
"expires_at": (NOW - timedelta(minutes=10)).isoformat(),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
proof = prr.reconcile_after_restart(inv, now=NOW, mode=prr.MODE_LOG_ONLY)
|
||||||
|
lease_item = next(i for i in proof.items if i.dimension == prr.DIM_LEASES)
|
||||||
|
self.assertIsNotNone(lease_item)
|
||||||
|
self.assertTrue(lease_item.summary)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet7SessionsResumeOrReceiveNextAction(unittest.TestCase):
|
||||||
|
"""Bullet 7: Sessions resume or receive canonical next action."""
|
||||||
|
|
||||||
|
def test_reconcile_provides_canonical_next_action_for_unresolved(self):
|
||||||
|
inv = _clean_inventory()
|
||||||
|
inv["pending_mutations"] = [
|
||||||
|
{
|
||||||
|
"mutation_id": "mut-404",
|
||||||
|
"session_id": "prgs-author-77",
|
||||||
|
"phase": "implementing",
|
||||||
|
"issue_number": 666,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
proof = prr.reconcile_after_restart(inv, now=NOW, mode=prr.MODE_ENFORCE)
|
||||||
|
self.assertEqual(proof.overall_status, prr.STATUS_DEGRADED)
|
||||||
|
self.assertTrue(proof.mutation_hold)
|
||||||
|
self.assertTrue(proof.note)
|
||||||
|
self.assertGreater(len(proof.proposed_follow_ups), 0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet8FailedDrainCreatesIncidentWork(unittest.TestCase):
|
||||||
|
"""Bullet 8: Failed drain creates durable incident work."""
|
||||||
|
|
||||||
|
def test_denied_drain_gate_mints_durable_incident_descriptor(self):
|
||||||
|
impact = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": [], "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
).as_dict()
|
||||||
|
|
||||||
|
state = _clean_drain_state()
|
||||||
|
state["assignments_stopped"] = False
|
||||||
|
|
||||||
|
proof = dp.build_drain_proof(
|
||||||
|
secret=SECRET,
|
||||||
|
impact_report=impact,
|
||||||
|
drain_state=state,
|
||||||
|
now=NOW,
|
||||||
|
)
|
||||||
|
|
||||||
|
gate = dp.gate_apply_restart(proof=proof.as_dict(), secret=SECRET, now=NOW)
|
||||||
|
self.assertEqual(gate.verdict, dp.GATE_DENY)
|
||||||
|
|
||||||
|
incident = gate.incident
|
||||||
|
self.assertIsNotNone(incident)
|
||||||
|
self.assertEqual(incident["kind"], "restart_drain_gate_denied")
|
||||||
|
self.assertTrue(any("assignments_stopped" in r for r in incident["reasons"]))
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet9ScopedRestartNonInterference(unittest.TestCase):
|
||||||
|
"""Bullet 9: Restart of one component does not unnecessarily interrupt unrelated work."""
|
||||||
|
|
||||||
|
def test_scoped_role_restart_impacts_only_target_role(self):
|
||||||
|
sessions = [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-controller-1",
|
||||||
|
"role": "controller",
|
||||||
|
"profile": "prgs-controller",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"session_id": "prgs-author-10",
|
||||||
|
"role": "author",
|
||||||
|
"profile": "prgs-author",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"session_id": "prgs-reviewer-20",
|
||||||
|
"role": "reviewer",
|
||||||
|
"profile": "prgs-reviewer",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
policy = rc.RESTART_CLASS_POLICIES[RestartClass.ROLE_RUNTIME_RESTART]
|
||||||
|
report = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": sessions, "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
restart_class=RestartClass.ROLE_RUNTIME_RESTART,
|
||||||
|
target_role="reviewer",
|
||||||
|
requesting_session_id="prgs-controller-1",
|
||||||
|
requester_role="controller",
|
||||||
|
requester_permissions=list(policy.request_roles),
|
||||||
|
controller_approved=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertTrue(report.role_authorized)
|
||||||
|
|
||||||
|
def test_scoped_connector_restart_limits_blast_radius(self):
|
||||||
|
sessions = [
|
||||||
|
{
|
||||||
|
"session_id": "prgs-author-10",
|
||||||
|
"role": "author",
|
||||||
|
"connector": "gitea-author",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"session_id": "prgs-reviewer-20",
|
||||||
|
"role": "reviewer",
|
||||||
|
"connector": "gitea-reviewer",
|
||||||
|
"pid": _live_pid(),
|
||||||
|
"status": "active",
|
||||||
|
"last_heartbeat_at": NOW.isoformat(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
policy = rc.RESTART_CLASS_POLICIES[RestartClass.CONNECTOR_RESTART]
|
||||||
|
report = rc.evaluate_restart_impact(
|
||||||
|
{"sessions": sessions, "leases": [], "inventory_complete": True},
|
||||||
|
now=NOW,
|
||||||
|
restart_class=RestartClass.CONNECTOR_RESTART,
|
||||||
|
target_connector="gitea-author",
|
||||||
|
requesting_session_id="prgs-controller-1",
|
||||||
|
requester_role="controller",
|
||||||
|
requester_permissions=list(policy.request_roles),
|
||||||
|
controller_approved=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertIsNotNone(report)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBullet10NoManualChatReconstruction(unittest.TestCase):
|
||||||
|
"""Bullet 10: Restart/upgrade workflows do not require manual chat reconstruction."""
|
||||||
|
|
||||||
|
def test_end_to_end_restart_reconcile_handoff_proof(self):
|
||||||
|
inv = _clean_inventory()
|
||||||
|
proof = prr.reconcile_after_restart(inv, now=NOW, mode=prr.MODE_LOG_ONLY)
|
||||||
|
|
||||||
|
proof_dict = proof.as_dict()
|
||||||
|
self.assertEqual(proof_dict["overall_status"], prr.STATUS_COMPLETE)
|
||||||
|
self.assertFalse(proof_dict["mutation_hold"])
|
||||||
|
self.assertTrue(proof_dict["note"])
|
||||||
|
self.assertIn("links", proof_dict)
|
||||||
|
self.assertEqual(proof_dict["links"]["umbrella"], 655)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,465 @@
|
|||||||
|
"""Unit tests for Phase 3 Notifications and Human-Attention Console (#648)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from starlette.testclient import TestClient
|
||||||
|
|
||||||
|
from webui.app import create_app
|
||||||
|
from webui.notifications import (
|
||||||
|
ATTENTION_HUMAN_REQUIRED,
|
||||||
|
ATTENTION_OPERATOR,
|
||||||
|
ATTENTION_ROUTINE,
|
||||||
|
CATEGORY_AUTH,
|
||||||
|
CATEGORY_BLOCKER,
|
||||||
|
CATEGORY_LEASE,
|
||||||
|
CATEGORY_SYSTEM,
|
||||||
|
CATEGORY_VALIDATION,
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
NotificationItem,
|
||||||
|
NotificationSnapshot,
|
||||||
|
classify_attention_event,
|
||||||
|
load_notifications_snapshot,
|
||||||
|
snapshot_to_dict,
|
||||||
|
)
|
||||||
|
from webui.notification_views import render_notifications_page
|
||||||
|
from webui.project_registry import load_registry
|
||||||
|
from webui.queue_loader import QueueItem, QueueSnapshot
|
||||||
|
from webui.lease_loader import CollisionWarning, LeaseSnapshot
|
||||||
|
from webui.system_health import DependencyProbe, SystemHealthSnapshot, VersionInfo, StaleRuntime
|
||||||
|
|
||||||
|
|
||||||
|
def test_classify_attention_event_rules():
|
||||||
|
# 1. Critical escalation boundaries -> human-required
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_AUTH, "Auth error", "Unauthorized access attempt", is_auth_failure=True
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_HUMAN_REQUIRED
|
||||||
|
assert req_human is True
|
||||||
|
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_SYSTEM, "Hard stop", "Hard stop triggered", is_hard_stop=True
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_HUMAN_REQUIRED
|
||||||
|
assert req_human is True
|
||||||
|
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_VALIDATION, "Validation Error", "Report validation failed", is_validation_failure=True
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_HUMAN_REQUIRED
|
||||||
|
assert req_human is True
|
||||||
|
|
||||||
|
# 2. Operational issues -> operator
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_BLOCKER, "PR Blocked", "Merge conflict detected", is_blocker=True
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_OPERATOR
|
||||||
|
assert req_human is False
|
||||||
|
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_LEASE, "Lease Expired", "Session lease expired", is_stale=True
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_OPERATOR
|
||||||
|
assert req_human is False
|
||||||
|
|
||||||
|
# 3. Routine workflow transitions -> routine
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW, "PR Active", "PR in review"
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_ROUTINE
|
||||||
|
assert req_human is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_notification_snapshot_aggregation():
|
||||||
|
reg = load_registry()
|
||||||
|
proj_id = reg.projects[0].id if reg.projects else "gitea-tools"
|
||||||
|
|
||||||
|
mock_queue = QueueSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
prs=(
|
||||||
|
QueueItem(
|
||||||
|
number=101,
|
||||||
|
title="Blocked PR",
|
||||||
|
badges=("blocked",),
|
||||||
|
extra={},
|
||||||
|
),
|
||||||
|
QueueItem(
|
||||||
|
number=102,
|
||||||
|
title="Normal PR",
|
||||||
|
badges=("in-review",),
|
||||||
|
extra={},
|
||||||
|
),
|
||||||
|
),
|
||||||
|
issues=(),
|
||||||
|
pr_pagination=None,
|
||||||
|
issue_pagination=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_leases = LeaseSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
issue_lock=None,
|
||||||
|
claim_inventory={},
|
||||||
|
reviewer_leases=(
|
||||||
|
{
|
||||||
|
"pr_number": 101,
|
||||||
|
"status": "expired",
|
||||||
|
"is_expired": True,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
duplicate_prs=(
|
||||||
|
CollisionWarning(
|
||||||
|
kind="duplicate_pr",
|
||||||
|
message="Multiple open PRs for issue #101",
|
||||||
|
issue_number=101,
|
||||||
|
pr_numbers=(101, 103),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
duplicate_branches=(),
|
||||||
|
collision_history=(),
|
||||||
|
fetch_error=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_version = VersionInfo(
|
||||||
|
git_sha="abc1234",
|
||||||
|
git_describe="v1.0.0",
|
||||||
|
control_plane_schema_version=1,
|
||||||
|
python_version="3.11",
|
||||||
|
known=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_stale = StaleRuntime(
|
||||||
|
daemon_head="abc1234",
|
||||||
|
checkout_head="abc1234",
|
||||||
|
remote_head="abc1234",
|
||||||
|
stale=False,
|
||||||
|
determinable=True,
|
||||||
|
mutation_safe=True,
|
||||||
|
reasons=(),
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_health = SystemHealthSnapshot(
|
||||||
|
status="degraded",
|
||||||
|
ready=False,
|
||||||
|
readiness_complete=True,
|
||||||
|
readiness_reasons=("Auth failure",),
|
||||||
|
service="webui",
|
||||||
|
mode="test",
|
||||||
|
version=mock_version,
|
||||||
|
started_at="2026-07-25T00:00:00Z",
|
||||||
|
uptime_seconds=100.0,
|
||||||
|
timestamp="2026-07-25T00:00:00Z",
|
||||||
|
deep_probes_requested=True,
|
||||||
|
dependencies=(
|
||||||
|
DependencyProbe(
|
||||||
|
name="auth_service",
|
||||||
|
kind="auth",
|
||||||
|
status="unauthorized",
|
||||||
|
detail="Token expired",
|
||||||
|
required=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
mcp_namespaces=(),
|
||||||
|
stale_runtime=mock_stale,
|
||||||
|
probe_errors=(),
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshot = load_notifications_snapshot(
|
||||||
|
proj_id,
|
||||||
|
load_queue=lambda _id: mock_queue,
|
||||||
|
load_leases=lambda **_kwargs: mock_leases,
|
||||||
|
load_health=lambda **_kwargs: mock_health,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert snapshot.project_id == proj_id
|
||||||
|
assert snapshot.total_count == 5
|
||||||
|
assert snapshot.human_required_count >= 1 # auth probe failure
|
||||||
|
assert snapshot.operator_count >= 3 # blocked PR + expired lease + duplicate PR collision
|
||||||
|
assert snapshot.routine_count >= 1 # normal PR
|
||||||
|
|
||||||
|
# Inbox items should include operator and human-required items only
|
||||||
|
inbox_classes = {item.attention_class for item in snapshot.inbox_items}
|
||||||
|
assert ATTENTION_ROUTINE not in inbox_classes
|
||||||
|
assert ATTENTION_OPERATOR in inbox_classes
|
||||||
|
assert ATTENTION_HUMAN_REQUIRED in inbox_classes
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_to_dict_and_redaction():
|
||||||
|
item = NotificationItem(
|
||||||
|
id="notif-1",
|
||||||
|
attention_class=ATTENTION_HUMAN_REQUIRED,
|
||||||
|
category=CATEGORY_AUTH,
|
||||||
|
title="Auth Error",
|
||||||
|
summary="Failed auth header: Bearer secret_token_12345",
|
||||||
|
work_kind="system",
|
||||||
|
work_number=None,
|
||||||
|
project_id="test-proj",
|
||||||
|
repo_label="org/repo",
|
||||||
|
created_at="2026-07-25T16:00:00Z",
|
||||||
|
requires_human=True,
|
||||||
|
)
|
||||||
|
snap = NotificationSnapshot(
|
||||||
|
project_id="test-proj",
|
||||||
|
repo_label="org/repo",
|
||||||
|
items=(item,),
|
||||||
|
human_required_count=1,
|
||||||
|
operator_count=0,
|
||||||
|
routine_count=0,
|
||||||
|
total_count=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
data = snapshot_to_dict(snap)
|
||||||
|
assert data["project_id"] == "test-proj"
|
||||||
|
assert data["human_required_count"] == 1
|
||||||
|
assert len(data["inbox_items"]) == 1
|
||||||
|
|
||||||
|
# Redaction test
|
||||||
|
summary = data["inbox_items"][0]["summary"]
|
||||||
|
assert "secret_token_12345" not in summary
|
||||||
|
assert "<redacted>" in summary or "Bearer" in summary
|
||||||
|
|
||||||
|
|
||||||
|
def test_notifications_html_views():
|
||||||
|
item = NotificationItem(
|
||||||
|
id="notif-1",
|
||||||
|
attention_class=ATTENTION_HUMAN_REQUIRED,
|
||||||
|
category=CATEGORY_AUTH,
|
||||||
|
title="Critical Auth Failure",
|
||||||
|
summary="Auth failure details",
|
||||||
|
work_kind="issue",
|
||||||
|
work_number=42,
|
||||||
|
project_id="test-proj",
|
||||||
|
repo_label="org/repo",
|
||||||
|
created_at="2026-07-25T16:00:00Z",
|
||||||
|
requires_human=True,
|
||||||
|
)
|
||||||
|
snap = NotificationSnapshot(
|
||||||
|
project_id="test-proj",
|
||||||
|
repo_label="org/repo",
|
||||||
|
items=(item,),
|
||||||
|
human_required_count=1,
|
||||||
|
operator_count=0,
|
||||||
|
routine_count=0,
|
||||||
|
total_count=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
html = render_notifications_page(snap, filter_class="inbox")
|
||||||
|
assert "Notifications & Attention Inbox" in html or "Notifications & Attention Inbox" in html
|
||||||
|
assert "Critical Auth Failure" in html
|
||||||
|
assert "HUMAN REQUIRED" in html
|
||||||
|
assert "Human Required" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_notifications_app_routes():
|
||||||
|
app = create_app()
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
# 1. HTML Route
|
||||||
|
res = client.get("/notifications")
|
||||||
|
assert res.status_code == 200
|
||||||
|
assert "Notifications" in res.text
|
||||||
|
assert "Attention Inbox" in res.text
|
||||||
|
|
||||||
|
# 2. API Route /api/v1/notifications
|
||||||
|
res_api = client.get("/api/v1/notifications")
|
||||||
|
assert res_api.status_code == 200
|
||||||
|
json_data = res_api.json()
|
||||||
|
assert "human_required_count" in json_data
|
||||||
|
assert "operator_count" in json_data
|
||||||
|
assert "routine_count" in json_data
|
||||||
|
assert "inbox_items" in json_data
|
||||||
|
|
||||||
|
# 3. Compatibility Alias /api/notifications
|
||||||
|
res_alias = client.get("/api/notifications")
|
||||||
|
assert res_alias.status_code == 200
|
||||||
|
assert res_alias.json()["project_id"] == json_data["project_id"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_classify_ignores_human_authored_title_and_summary_keywords():
|
||||||
|
"""B1: keywords in human-authored titles must not escalate routine work (#905)."""
|
||||||
|
# Routine transition whose title/summary mention critical-boundary words
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
"record irrecoverable decision lock provenance",
|
||||||
|
"PR #999 'record irrecoverable decision lock provenance' is in routine state in-review.",
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_ROUTINE
|
||||||
|
assert req_human is False
|
||||||
|
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
"fix unauthorized token path",
|
||||||
|
"Issue #1 'fix unauthorized token path' state: claimed. hard stop docs only.",
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_ROUTINE
|
||||||
|
assert req_human is False
|
||||||
|
|
||||||
|
# Structured flags still escalate (machine-driven)
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_SYSTEM,
|
||||||
|
"anything",
|
||||||
|
"anything with hard stop in text",
|
||||||
|
is_hard_stop=True,
|
||||||
|
)
|
||||||
|
assert att_cls == ATTENTION_HUMAN_REQUIRED
|
||||||
|
assert req_human is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_notification_ids_are_unique_across_probe_errors_and_collisions():
|
||||||
|
"""B2: published notification ids must be unique within a snapshot (#905)."""
|
||||||
|
reg = load_registry()
|
||||||
|
proj_id = reg.projects[0].id if reg.projects else "gitea-tools"
|
||||||
|
|
||||||
|
mock_queue = QueueSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
prs=(),
|
||||||
|
issues=(),
|
||||||
|
pr_pagination=None,
|
||||||
|
issue_pagination=None,
|
||||||
|
)
|
||||||
|
mock_leases = LeaseSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
issue_lock=None,
|
||||||
|
claim_inventory={},
|
||||||
|
reviewer_leases=(),
|
||||||
|
duplicate_prs=(
|
||||||
|
CollisionWarning(
|
||||||
|
kind="duplicate_pr",
|
||||||
|
message="Multiple open PRs for issue #10",
|
||||||
|
issue_number=10,
|
||||||
|
pr_numbers=(10, 11),
|
||||||
|
),
|
||||||
|
CollisionWarning(
|
||||||
|
kind="duplicate_branch",
|
||||||
|
message="Another collision without issue",
|
||||||
|
issue_number=None,
|
||||||
|
pr_numbers=(12, 13),
|
||||||
|
),
|
||||||
|
CollisionWarning(
|
||||||
|
kind="duplicate_pr",
|
||||||
|
message="Second issue collision",
|
||||||
|
issue_number=10,
|
||||||
|
pr_numbers=(14, 15),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
duplicate_branches=(),
|
||||||
|
collision_history=(),
|
||||||
|
fetch_error=None,
|
||||||
|
)
|
||||||
|
mock_version = VersionInfo(
|
||||||
|
git_sha="abc1234",
|
||||||
|
git_describe="v1.0.0",
|
||||||
|
control_plane_schema_version=1,
|
||||||
|
python_version="3.11",
|
||||||
|
known=True,
|
||||||
|
)
|
||||||
|
mock_stale = StaleRuntime(
|
||||||
|
daemon_head="abc1234",
|
||||||
|
checkout_head="abc1234",
|
||||||
|
remote_head="abc1234",
|
||||||
|
stale=False,
|
||||||
|
determinable=True,
|
||||||
|
mutation_safe=True,
|
||||||
|
reasons=(),
|
||||||
|
)
|
||||||
|
mock_health = SystemHealthSnapshot(
|
||||||
|
status="degraded",
|
||||||
|
ready=False,
|
||||||
|
readiness_complete=True,
|
||||||
|
readiness_reasons=(),
|
||||||
|
service="webui",
|
||||||
|
mode="test",
|
||||||
|
version=mock_version,
|
||||||
|
started_at="2026-07-25T00:00:00Z",
|
||||||
|
uptime_seconds=100.0,
|
||||||
|
timestamp="2026-07-25T00:00:00Z",
|
||||||
|
deep_probes_requested=True,
|
||||||
|
dependencies=(),
|
||||||
|
mcp_namespaces=(),
|
||||||
|
stale_runtime=mock_stale,
|
||||||
|
probe_errors=("error alpha", "error beta"),
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshot = load_notifications_snapshot(
|
||||||
|
proj_id,
|
||||||
|
load_queue=lambda _id: mock_queue,
|
||||||
|
load_leases=lambda **_kwargs: mock_leases,
|
||||||
|
load_health=lambda **_kwargs: mock_health,
|
||||||
|
)
|
||||||
|
ids = [item.id for item in snapshot.items]
|
||||||
|
assert len(ids) == len(set(ids)), f"duplicate notification ids: {ids}"
|
||||||
|
assert any(i.startswith(f"notif-sys-err-{proj_id}-") for i in ids)
|
||||||
|
assert any(i.startswith("notif-collision-") for i in ids)
|
||||||
|
|
||||||
|
|
||||||
|
def test_probe_errors_do_not_set_fetch_error():
|
||||||
|
"""B3: probe_errors must not be reported as fetch_error (#905)."""
|
||||||
|
reg = load_registry()
|
||||||
|
proj_id = reg.projects[0].id if reg.projects else "gitea-tools"
|
||||||
|
|
||||||
|
mock_queue = QueueSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
prs=(),
|
||||||
|
issues=(),
|
||||||
|
pr_pagination=None,
|
||||||
|
issue_pagination=None,
|
||||||
|
fetch_error=None,
|
||||||
|
)
|
||||||
|
mock_leases = LeaseSnapshot(
|
||||||
|
project_id=proj_id,
|
||||||
|
repo_label="org/repo",
|
||||||
|
issue_lock=None,
|
||||||
|
claim_inventory={},
|
||||||
|
reviewer_leases=(),
|
||||||
|
duplicate_prs=(),
|
||||||
|
duplicate_branches=(),
|
||||||
|
collision_history=(),
|
||||||
|
fetch_error=None,
|
||||||
|
)
|
||||||
|
mock_version = VersionInfo(
|
||||||
|
git_sha="abc1234",
|
||||||
|
git_describe="v1.0.0",
|
||||||
|
control_plane_schema_version=1,
|
||||||
|
python_version="3.11",
|
||||||
|
known=True,
|
||||||
|
)
|
||||||
|
mock_stale = StaleRuntime(
|
||||||
|
daemon_head="abc1234",
|
||||||
|
checkout_head="abc1234",
|
||||||
|
remote_head="abc1234",
|
||||||
|
stale=False,
|
||||||
|
determinable=True,
|
||||||
|
mutation_safe=True,
|
||||||
|
reasons=(),
|
||||||
|
)
|
||||||
|
mock_health = SystemHealthSnapshot(
|
||||||
|
status="degraded",
|
||||||
|
ready=False,
|
||||||
|
readiness_complete=True,
|
||||||
|
readiness_reasons=(),
|
||||||
|
service="webui",
|
||||||
|
mode="test",
|
||||||
|
version=mock_version,
|
||||||
|
started_at="2026-07-25T00:00:00Z",
|
||||||
|
uptime_seconds=100.0,
|
||||||
|
timestamp="2026-07-25T00:00:00Z",
|
||||||
|
deep_probes_requested=True,
|
||||||
|
dependencies=(),
|
||||||
|
mcp_namespaces=(),
|
||||||
|
stale_runtime=mock_stale,
|
||||||
|
probe_errors=("probe blew up",),
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshot = load_notifications_snapshot(
|
||||||
|
proj_id,
|
||||||
|
load_queue=lambda _id: mock_queue,
|
||||||
|
load_leases=lambda **_kwargs: mock_leases,
|
||||||
|
load_health=lambda **_kwargs: mock_health,
|
||||||
|
)
|
||||||
|
assert snapshot.fetch_error is None
|
||||||
|
# probe errors still appear as items
|
||||||
|
assert any("probe blew up" in item.summary for item in snapshot.items)
|
||||||
@@ -0,0 +1,190 @@
|
|||||||
|
"""Tests for Sentry/GlitchTip observability console (#649, Phase 4)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import pytest
|
||||||
|
from control_plane_db import ControlPlaneDB
|
||||||
|
from webui.app import create_app
|
||||||
|
from webui.console_authz import authorize, resolve_principal
|
||||||
|
from webui.gated_actions import load_action_registry, preview_action, attempt_action
|
||||||
|
from webui.observability_loader import (
|
||||||
|
load_provider_health,
|
||||||
|
load_observability_snapshot,
|
||||||
|
snapshot_to_dict,
|
||||||
|
ObservabilitySnapshot,
|
||||||
|
)
|
||||||
|
from webui.observability_views import render_observability_page
|
||||||
|
from tests.webui_testclient import TestClient
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def test_db(tmp_path):
|
||||||
|
db_path = str(tmp_path / "test_control_plane.db")
|
||||||
|
db = ControlPlaneDB(db_path)
|
||||||
|
return db
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_provider_health_redaction():
|
||||||
|
"""Ensure tokens and secrets are never returned in provider health data."""
|
||||||
|
env = {
|
||||||
|
"SENTRY_BASE_URL": "https://sentry.prgs.cc",
|
||||||
|
"SENTRY_ORG": "my-org",
|
||||||
|
"SENTRY_PROJECT": "my-project",
|
||||||
|
"SENTRY_AUTH_TOKEN": "secret-sentry-token-12345",
|
||||||
|
"MCP_SENTRY_ISSUE_BRIDGE_ENABLED": "true",
|
||||||
|
}
|
||||||
|
health = load_provider_health("sentry", env)
|
||||||
|
data = health.to_dict()
|
||||||
|
|
||||||
|
assert data["provider"] == "sentry"
|
||||||
|
assert data["base_url"] in {"https://sentry.prgs.cc", "[REDACTED_URL]"}
|
||||||
|
assert data["org"] == "my-org"
|
||||||
|
assert data["project"] == "my-project"
|
||||||
|
assert data["configured"] is True
|
||||||
|
assert data["status"] == "healthy"
|
||||||
|
assert data["credentials_present"] is True
|
||||||
|
|
||||||
|
# Token must NOT be in the dict keys or values
|
||||||
|
serialized = str(data)
|
||||||
|
assert "secret-sentry-token-12345" not in serialized
|
||||||
|
assert "SENTRY_AUTH_TOKEN" not in serialized
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_provider_health_statuses():
|
||||||
|
"""Test unconfigured, missing token, and disabled statuses."""
|
||||||
|
# Not configured
|
||||||
|
h1 = load_provider_health("sentry", {})
|
||||||
|
d1 = h1.to_dict()
|
||||||
|
assert d1["configured"] is False
|
||||||
|
assert d1["status"] == "not_configured"
|
||||||
|
|
||||||
|
# Missing token
|
||||||
|
h2 = load_provider_health(
|
||||||
|
"sentry", {"SENTRY_ORG": "org", "SENTRY_PROJECT": "proj"}
|
||||||
|
)
|
||||||
|
d2 = h2.to_dict()
|
||||||
|
assert d2["configured"] is False
|
||||||
|
assert d2["status"] == "missing_token"
|
||||||
|
|
||||||
|
# Disabled
|
||||||
|
h3 = load_provider_health(
|
||||||
|
"sentry",
|
||||||
|
{
|
||||||
|
"SENTRY_ORG": "org",
|
||||||
|
"SENTRY_PROJECT": "proj",
|
||||||
|
"SENTRY_AUTH_TOKEN": "token",
|
||||||
|
"MCP_SENTRY_ISSUE_BRIDGE_ENABLED": "false",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
d3 = h3.to_dict()
|
||||||
|
assert d3["configured"] is True
|
||||||
|
assert d3["status"] == "disabled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_observability_snapshot_with_db_links(test_db):
|
||||||
|
"""Test loading observability snapshot with incident links in DB."""
|
||||||
|
test_db.upsert_incident_link(
|
||||||
|
provider="sentry",
|
||||||
|
provider_issue_id="101",
|
||||||
|
gitea_org="Scaled-Tech-Consulting",
|
||||||
|
gitea_repo="Gitea-Tools",
|
||||||
|
gitea_issue_number=649,
|
||||||
|
provider_base_url="https://sentry.prgs.cc",
|
||||||
|
provider_org="Scaled-Tech-Consulting",
|
||||||
|
provider_project="Gitea-Tools",
|
||||||
|
provider_short_id="ST-101",
|
||||||
|
provider_permalink="https://sentry.prgs.cc/issues/101/",
|
||||||
|
fingerprint="err-fingerprint-001",
|
||||||
|
linked_pr_numbers=[901, 902],
|
||||||
|
last_seen="2026-07-25T12:00:00Z",
|
||||||
|
event_count=5,
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshot = load_observability_snapshot(db=test_db, env={})
|
||||||
|
data = snapshot.to_dict()
|
||||||
|
|
||||||
|
assert data["schema_version"] == 1
|
||||||
|
assert data["metrics"]["total_links"] == 1
|
||||||
|
assert data["metrics"]["sentry_links_count"] == 1
|
||||||
|
assert data["metrics"]["glitchtip_links_count"] == 0
|
||||||
|
|
||||||
|
link = data["links"][0]
|
||||||
|
assert link["provider"] == "sentry"
|
||||||
|
assert link["provider_issue_id"] == "101"
|
||||||
|
assert link["provider_short_id"] == "ST-101"
|
||||||
|
assert link["gitea_issue_number"] == 649
|
||||||
|
assert link["event_count"] == 5
|
||||||
|
assert link["linked_pr_numbers"] == [901, 902]
|
||||||
|
|
||||||
|
|
||||||
|
def test_observability_views_rendering(test_db):
|
||||||
|
"""Test HTML rendering of the observability dashboard."""
|
||||||
|
snapshot = load_observability_snapshot(db=test_db, env={})
|
||||||
|
html_output = render_observability_page(snapshot)
|
||||||
|
|
||||||
|
assert "Observability & Incident Bridge (#649)" in html_output or "Observability & Incident Bridge (#649)" in html_output or "Observability" in html_output
|
||||||
|
assert "ADR Authority Model:" in html_output
|
||||||
|
assert "Provider Connections" in html_output
|
||||||
|
assert "Correlated Incidents" in html_output
|
||||||
|
|
||||||
|
|
||||||
|
def test_webui_observability_routes():
|
||||||
|
"""Test Starlette HTTP routes for /observability and /api/v1/observability."""
|
||||||
|
client = TestClient(create_app())
|
||||||
|
|
||||||
|
# HTML page route
|
||||||
|
res_html = client.get("/observability")
|
||||||
|
assert res_html.status_code == 200
|
||||||
|
assert "text/html" in res_html.headers["content-type"]
|
||||||
|
assert "Observability" in res_html.text
|
||||||
|
|
||||||
|
# Versioned API route
|
||||||
|
res_api_v1 = client.get("/api/v1/observability")
|
||||||
|
assert res_api_v1.status_code == 200
|
||||||
|
assert "application/json" in res_api_v1.headers["content-type"]
|
||||||
|
data_v1 = res_api_v1.json()
|
||||||
|
assert "schema_version" in data_v1
|
||||||
|
assert "providers" in data_v1
|
||||||
|
assert "links" in data_v1
|
||||||
|
assert "metrics" in data_v1
|
||||||
|
|
||||||
|
# Compatibility alias route
|
||||||
|
res_api_alias = client.get("/api/observability")
|
||||||
|
assert res_api_alias.status_code == 200
|
||||||
|
assert res_api_alias.json() == data_v1
|
||||||
|
|
||||||
|
|
||||||
|
def test_observability_gated_actions():
|
||||||
|
"""Ensure observability actions are registered, gated, and fail closed in MVP mode."""
|
||||||
|
registry = load_action_registry()
|
||||||
|
|
||||||
|
action_reconcile = registry.get("observability_reconcile_incident")
|
||||||
|
assert action_reconcile is not None
|
||||||
|
assert action_reconcile.task_key == "observability_reconcile_incident"
|
||||||
|
assert action_reconcile.mcp_tool == "gitea_observability_reconcile_incident"
|
||||||
|
|
||||||
|
action_link = registry.get("observability_link_issue")
|
||||||
|
assert action_link is not None
|
||||||
|
assert action_link.task_key == "observability_link_issue"
|
||||||
|
|
||||||
|
# Preview returns mutation ledger
|
||||||
|
prev = preview_action("observability_reconcile_incident", provider="sentry", issue_id="101")
|
||||||
|
assert prev["action_id"] == "observability_reconcile_incident"
|
||||||
|
assert prev["enabled"] is False
|
||||||
|
|
||||||
|
# Execution fails closed in MVP mode
|
||||||
|
att = attempt_action("observability_reconcile_incident", provider="sentry", issue_id="101")
|
||||||
|
assert att["success"] is False
|
||||||
|
assert att["error"] == "action_disabled"
|
||||||
|
|
||||||
|
|
||||||
|
def test_observability_authz_rbac():
|
||||||
|
"""Test RBAC authorization for observability actions."""
|
||||||
|
principal = resolve_principal({})
|
||||||
|
|
||||||
|
# Check authorize decision
|
||||||
|
decision = authorize("observability_reconcile_incident", principal)
|
||||||
|
assert decision.action_id == "observability_reconcile_incident"
|
||||||
|
# Phase 4 action denies in Phase 1 runtime by default
|
||||||
|
assert decision.allowed is False
|
||||||
@@ -80,8 +80,18 @@ from webui.system_health import (
|
|||||||
snapshot_to_dict as system_health_to_dict,
|
snapshot_to_dict as system_health_to_dict,
|
||||||
)
|
)
|
||||||
from webui.system_health_views import render_system_health_page
|
from webui.system_health_views import render_system_health_page
|
||||||
|
from webui.notifications import (
|
||||||
|
load_notifications_snapshot,
|
||||||
|
snapshot_to_dict as notifications_snapshot_to_dict,
|
||||||
|
)
|
||||||
|
from webui.notification_views import render_notifications_page
|
||||||
from webui import request_service
|
from webui import request_service
|
||||||
from webui.request_views import render_requests_page
|
from webui.request_views import render_requests_page
|
||||||
|
from webui.observability_loader import (
|
||||||
|
load_observability_snapshot,
|
||||||
|
snapshot_to_dict as observability_snapshot_to_dict,
|
||||||
|
)
|
||||||
|
from webui.observability_views import render_observability_page
|
||||||
|
|
||||||
_READ_ONLY_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
_READ_ONLY_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
||||||
_AUDIT_MUTATION_PATHS = frozenset({"/audit", "/api/audit"})
|
_AUDIT_MUTATION_PATHS = frozenset({"/audit", "/api/audit"})
|
||||||
@@ -889,6 +899,35 @@ async def api_v1_analytics_ingest(request: Request) -> JSONResponse:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def notifications_route(request: Request) -> HTMLResponse:
|
||||||
|
project_id = request.query_params.get("project_id")
|
||||||
|
attention_class = request.query_params.get("attention_class") or "inbox"
|
||||||
|
snap = load_notifications_snapshot(project_id)
|
||||||
|
html = render_notifications_page(
|
||||||
|
snap, filter_class=attention_class, filter_project=project_id
|
||||||
|
)
|
||||||
|
return HTMLResponse(html)
|
||||||
|
|
||||||
|
|
||||||
|
async def api_notifications(request: Request) -> JSONResponse:
|
||||||
|
project_id = request.query_params.get("project_id")
|
||||||
|
snap = load_notifications_snapshot(project_id)
|
||||||
|
data = notifications_snapshot_to_dict(snap)
|
||||||
|
return JSONResponse(data)
|
||||||
|
|
||||||
|
|
||||||
|
async def observability_route(request: Request) -> HTMLResponse:
|
||||||
|
snap = load_observability_snapshot()
|
||||||
|
html_content = render_observability_page(snap)
|
||||||
|
return HTMLResponse(html_content)
|
||||||
|
|
||||||
|
|
||||||
|
async def api_observability(request: Request) -> JSONResponse:
|
||||||
|
snap = load_observability_snapshot()
|
||||||
|
data = observability_snapshot_to_dict(snap)
|
||||||
|
return JSONResponse(data)
|
||||||
|
|
||||||
|
|
||||||
def _default_request_scope() -> dict[str, str]:
|
def _default_request_scope() -> dict[str, str]:
|
||||||
"""Resolve remote/org/repo from the project registry for request forms.
|
"""Resolve remote/org/repo from the project registry for request forms.
|
||||||
|
|
||||||
@@ -1020,6 +1059,9 @@ def create_app(*, bind_host: str | None = None) -> Starlette:
|
|||||||
Route("/api/queue", api_queue, methods=["GET"]),
|
Route("/api/queue", api_queue, methods=["GET"]),
|
||||||
Route("/traffic", traffic, methods=["GET"]),
|
Route("/traffic", traffic, methods=["GET"]),
|
||||||
Route("/api/traffic", api_traffic, methods=["GET"]),
|
Route("/api/traffic", api_traffic, methods=["GET"]),
|
||||||
|
Route("/notifications", notifications_route, methods=["GET"]),
|
||||||
|
Route("/api/notifications", api_notifications, methods=["GET"]),
|
||||||
|
Route("/api/v1/notifications", api_notifications, methods=["GET"]),
|
||||||
Route("/projects", projects, methods=["GET"]),
|
Route("/projects", projects, methods=["GET"]),
|
||||||
Route("/projects/{project_id}", project_detail, methods=["GET"]),
|
Route("/projects/{project_id}", project_detail, methods=["GET"]),
|
||||||
Route("/api/projects", api_projects, methods=["GET"]),
|
Route("/api/projects", api_projects, methods=["GET"]),
|
||||||
@@ -1051,6 +1093,9 @@ def create_app(*, bind_host: str | None = None) -> Starlette:
|
|||||||
Route("/api/analytics", api_v1_analytics, methods=["GET"]),
|
Route("/api/analytics", api_v1_analytics, methods=["GET"]),
|
||||||
Route("/api/v1/analytics", api_v1_analytics, methods=["GET"]),
|
Route("/api/v1/analytics", api_v1_analytics, methods=["GET"]),
|
||||||
Route("/api/v1/analytics/usage", api_v1_analytics_ingest, methods=["POST"]),
|
Route("/api/v1/analytics/usage", api_v1_analytics_ingest, methods=["POST"]),
|
||||||
|
Route("/observability", observability_route, methods=["GET"]),
|
||||||
|
Route("/api/observability", api_observability, methods=["GET"]),
|
||||||
|
Route("/api/v1/observability", api_observability, methods=["GET"]),
|
||||||
Route("/audit", audit, methods=["GET", "POST"]),
|
Route("/audit", audit, methods=["GET", "POST"]),
|
||||||
Route("/api/audit", api_audit, methods=["GET", "POST"]),
|
Route("/api/audit", api_audit, methods=["GET", "POST"]),
|
||||||
Route("/worktrees", worktrees, methods=["GET"]),
|
Route("/worktrees", worktrees, methods=["GET"]),
|
||||||
|
|||||||
@@ -317,6 +317,29 @@ _ACTION_SPECS: tuple[ConsoleAction, ...] = (
|
|||||||
phase=2,
|
phase=2,
|
||||||
summary="Run reconciler cleanup for merged or superseded PR branches.",
|
summary="Run reconciler cleanup for merged or superseded PR branches.",
|
||||||
),
|
),
|
||||||
|
# #649: Phase 4 observability & incident bridge actions.
|
||||||
|
ConsoleAction(
|
||||||
|
action_id="observability_reconcile_incident",
|
||||||
|
task_key="observability_reconcile_incident",
|
||||||
|
action_class=CLASS_WRITE,
|
||||||
|
minimum_role=OPERATOR,
|
||||||
|
requires_confirmation=True,
|
||||||
|
dual_control=False,
|
||||||
|
break_glass=False,
|
||||||
|
phase=4,
|
||||||
|
summary="Trigger/reconcile durable Gitea issue creation from a provider incident.",
|
||||||
|
),
|
||||||
|
ConsoleAction(
|
||||||
|
action_id="observability_link_issue",
|
||||||
|
task_key="observability_link_issue",
|
||||||
|
action_class=CLASS_WRITE,
|
||||||
|
minimum_role=OPERATOR,
|
||||||
|
requires_confirmation=True,
|
||||||
|
dual_control=False,
|
||||||
|
break_glass=False,
|
||||||
|
phase=4,
|
||||||
|
summary="Link a provider incident to an existing Gitea issue.",
|
||||||
|
),
|
||||||
# #643: submit a work request — desired role, issue/PR, intent — and let
|
# #643: submit a work request — desired role, issue/PR, intent — and let
|
||||||
# the allocator reserve it. This is the one Phase 2 action whose execution
|
# the allocator reserve it. This is the one Phase 2 action whose execution
|
||||||
# path is actually implemented (``webui.request_service``), so it carries
|
# path is actually implemented (``webui.request_service``), so it carries
|
||||||
|
|||||||
@@ -185,6 +185,11 @@ def build_action_registry() -> ActionRegistry:
|
|||||||
"console.rebind_session_worktree", "Rebind session worktree to verified lease."),
|
"console.rebind_session_worktree", "Rebind session worktree to verified lease."),
|
||||||
("system.reconcile_cleanups", "Reconcile cleanups", "reconcile_cleanups",
|
("system.reconcile_cleanups", "Reconcile cleanups", "reconcile_cleanups",
|
||||||
"console.reconcile_cleanups", "Run reconciler cleanup for merged or superseded PRs."),
|
"console.reconcile_cleanups", "Run reconciler cleanup for merged or superseded PRs."),
|
||||||
|
# #649: Phase 4 observability & incident bridge actions.
|
||||||
|
("observability_reconcile_incident", "Reconcile incident", "observability_reconcile_incident",
|
||||||
|
"gitea_observability_reconcile_incident", "Trigger or dry-run durable issue reconciliation for a provider incident."),
|
||||||
|
("observability_link_issue", "Link incident issue", "observability_link_issue",
|
||||||
|
"gitea_observability_link_issue", "Link a provider incident to a Gitea tracking issue."),
|
||||||
)
|
)
|
||||||
actions = tuple(
|
actions = tuple(
|
||||||
GatedAction(
|
GatedAction(
|
||||||
|
|||||||
@@ -46,6 +46,7 @@ NAV_GROUPS: tuple[NavGroup, ...] = (
|
|||||||
NavItem("/queue", "Queue"),
|
NavItem("/queue", "Queue"),
|
||||||
NavItem("/leases", "Leases"),
|
NavItem("/leases", "Leases"),
|
||||||
NavItem("/actions", "Actions"),
|
NavItem("/actions", "Actions"),
|
||||||
|
NavItem("/notifications", "Notifications"),
|
||||||
NavItem("/requests", "Requests"),
|
NavItem("/requests", "Requests"),
|
||||||
)),
|
)),
|
||||||
NavGroup("Runtime/Sessions", (
|
NavGroup("Runtime/Sessions", (
|
||||||
@@ -72,6 +73,7 @@ NAV_GROUPS: tuple[NavGroup, ...] = (
|
|||||||
)),
|
)),
|
||||||
NavGroup("Insights", (
|
NavGroup("Insights", (
|
||||||
NavItem("/insights", "Insights", "stub"),
|
NavItem("/insights", "Insights", "stub"),
|
||||||
|
NavItem("/observability", "Observability"),
|
||||||
NavItem("/analytics", "Analytics"),
|
NavItem("/analytics", "Analytics"),
|
||||||
NavItem("/audit", "Audit"),
|
NavItem("/audit", "Audit"),
|
||||||
)),
|
)),
|
||||||
|
|||||||
@@ -0,0 +1,158 @@
|
|||||||
|
"""HTML rendering for Phase 3 Notifications and Human-Attention Console (#648)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from html import escape
|
||||||
|
from typing import Sequence
|
||||||
|
|
||||||
|
from webui.layout import render_page
|
||||||
|
from webui.notifications import (
|
||||||
|
ATTENTION_HUMAN_REQUIRED,
|
||||||
|
ATTENTION_OPERATOR,
|
||||||
|
ATTENTION_ROUTINE,
|
||||||
|
NotificationItem,
|
||||||
|
NotificationSnapshot,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_attention_badge(attention_class: str) -> str:
|
||||||
|
cls = "badge"
|
||||||
|
if attention_class == ATTENTION_HUMAN_REQUIRED:
|
||||||
|
cls += " badge-blocked"
|
||||||
|
elif attention_class == ATTENTION_OPERATOR:
|
||||||
|
cls += " badge-claimed"
|
||||||
|
else:
|
||||||
|
cls += " muted"
|
||||||
|
return f'<span class="{cls}">{escape(attention_class)}</span>'
|
||||||
|
|
||||||
|
|
||||||
|
def _render_notification_row(item: NotificationItem) -> str:
|
||||||
|
category_label = escape(item.category.upper())
|
||||||
|
id_str = escape(item.id)
|
||||||
|
title_str = escape(item.title)
|
||||||
|
summary_str = escape(item.summary)
|
||||||
|
att_badge = _render_attention_badge(item.attention_class)
|
||||||
|
|
||||||
|
work_item_html = "—"
|
||||||
|
if item.work_number and item.work_kind:
|
||||||
|
kind_label = escape(item.work_kind.upper())
|
||||||
|
num_str = f"#{item.work_number}"
|
||||||
|
link = item.deep_link or "#"
|
||||||
|
work_item_html = f'<a href="{escape(link)}"><code>{kind_label} {num_str}</code></a>'
|
||||||
|
|
||||||
|
requires_human_label = (
|
||||||
|
'<span class="badge badge-blocked" style="font-size:0.75rem;">HUMAN REQUIRED</span>'
|
||||||
|
if item.requires_human
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
return f"""<tr>
|
||||||
|
<td><code>{category_label}</code><br><span class="muted" style="font-size:0.75rem;">{id_str}</span></td>
|
||||||
|
<td>
|
||||||
|
<div><strong>{title_str}</strong> {att_badge} {requires_human_label}</div>
|
||||||
|
<div class="muted" style="font-size:0.85rem; margin-top:0.25rem;">{summary_str}</div>
|
||||||
|
</td>
|
||||||
|
<td>{work_item_html}</td>
|
||||||
|
<td><span class="muted" style="font-size:0.8rem;">{escape(item.created_at[:19])}</span></td>
|
||||||
|
</tr>"""
|
||||||
|
|
||||||
|
|
||||||
|
def _render_notifications_table(items: Sequence[NotificationItem], empty_message: str) -> str:
|
||||||
|
if not items:
|
||||||
|
return f'<p class="muted" style="padding:1rem 0;">{escape(empty_message)}</p>'
|
||||||
|
|
||||||
|
rows = "".join(_render_notification_row(item) for item in items)
|
||||||
|
return f"""<table class="registry">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th style="width: 18%;">Category & ID</th>
|
||||||
|
<th style="width: 52%;">Title & Attention Summary</th>
|
||||||
|
<th style="width: 15%;">Work Item</th>
|
||||||
|
<th style="width: 15%;">Time</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{rows}
|
||||||
|
</tbody>
|
||||||
|
</table>"""
|
||||||
|
|
||||||
|
|
||||||
|
def render_notifications_page(
|
||||||
|
snapshot: NotificationSnapshot,
|
||||||
|
*,
|
||||||
|
filter_class: str = "inbox",
|
||||||
|
filter_project: str | None = None,
|
||||||
|
) -> str:
|
||||||
|
"""Render the notifications and attention inbox page."""
|
||||||
|
title = "Notifications & Attention Inbox"
|
||||||
|
|
||||||
|
err_html = ""
|
||||||
|
if snapshot.fetch_error:
|
||||||
|
err_html = f'<div class="stub" style="border-color:#e53e3e; background:#fff5f5; color:#c53030; margin-bottom:1rem;"><p><strong>Fetch Warning:</strong> {escape(snapshot.fetch_error)}</p></div>'
|
||||||
|
|
||||||
|
# Determine items to render based on filter_class
|
||||||
|
if filter_class == ATTENTION_HUMAN_REQUIRED:
|
||||||
|
display_items = snapshot.human_required_items
|
||||||
|
active_tab_title = "Human-Required Escalations"
|
||||||
|
elif filter_class == ATTENTION_OPERATOR:
|
||||||
|
display_items = snapshot.operator_items
|
||||||
|
active_tab_title = "Operator Inbox Items"
|
||||||
|
elif filter_class == ATTENTION_ROUTINE:
|
||||||
|
display_items = snapshot.routine_items
|
||||||
|
active_tab_title = "Routine Workflow Transitions"
|
||||||
|
elif filter_class == "all":
|
||||||
|
display_items = snapshot.items
|
||||||
|
active_tab_title = "All Events (including Routine)"
|
||||||
|
else: # "inbox" default
|
||||||
|
display_items = snapshot.inbox_items
|
||||||
|
active_tab_title = "Attention Inbox (Human + Operator)"
|
||||||
|
|
||||||
|
hr_cls = "badge-blocked" if snapshot.human_required_count > 0 else "muted"
|
||||||
|
op_cls = "badge-claimed" if snapshot.operator_count > 0 else "muted"
|
||||||
|
|
||||||
|
metrics_html = f"""<div style="display:flex; gap:1rem; margin-bottom:1.5rem;">
|
||||||
|
<div class="health-card" style="flex:1;">
|
||||||
|
<span class="muted" style="font-size:0.85rem;">Human Required</span>
|
||||||
|
<h2 style="margin:0.2rem 0;"><span class="badge {hr_cls}" style="font-size:1.4rem;">{snapshot.human_required_count}</span></h2>
|
||||||
|
<p class="muted" style="font-size:0.8rem; margin:0;">Critical escalation boundary</p>
|
||||||
|
</div>
|
||||||
|
<div class="health-card" style="flex:1;">
|
||||||
|
<span class="muted" style="font-size:0.85rem;">Operator Inbox</span>
|
||||||
|
<h2 style="margin:0.2rem 0;"><span class="badge {op_cls}" style="font-size:1.4rem;">{snapshot.operator_count}</span></h2>
|
||||||
|
<p class="muted" style="font-size:0.8rem; margin:0;">Operational items needing review</p>
|
||||||
|
</div>
|
||||||
|
<div class="health-card" style="flex:1;">
|
||||||
|
<span class="muted" style="font-size:0.85rem;">Routine Transitions</span>
|
||||||
|
<h2 style="margin:0.2rem 0;"><span class="badge muted" style="font-size:1.4rem;">{snapshot.routine_count}</span></h2>
|
||||||
|
<p class="muted" style="font-size:0.8rem; margin:0;">Background transitions (filtered)</p>
|
||||||
|
</div>
|
||||||
|
</div>"""
|
||||||
|
|
||||||
|
# Filter navigation links
|
||||||
|
def _tab_link(target_class: str, label: str) -> str:
|
||||||
|
is_active = (filter_class == target_class)
|
||||||
|
style = "font-weight:bold; border-bottom:2px solid currentColor;" if is_active else "color:#4a5568;"
|
||||||
|
return f'<a href="/notifications?attention_class={target_class}" style="margin-right:1.25rem; text-decoration:none; padding-bottom:0.25rem; {style}">{label}</a>'
|
||||||
|
|
||||||
|
tabs_html = f"""<div style="margin-bottom:1.25rem; border-bottom:1px solid #e2e8f0; padding-bottom:0.5rem;">
|
||||||
|
{_tab_link("inbox", f"Attention Inbox ({snapshot.human_required_count + snapshot.operator_count})")}
|
||||||
|
{_tab_link("human-required", f"Human Required ({snapshot.human_required_count})")}
|
||||||
|
{_tab_link("operator", f"Operator ({snapshot.operator_count})")}
|
||||||
|
{_tab_link("routine", f"Routine ({snapshot.routine_count})")}
|
||||||
|
{_tab_link("all", f"All Events ({snapshot.total_count})")}
|
||||||
|
</div>"""
|
||||||
|
|
||||||
|
table_html = _render_notifications_table(
|
||||||
|
display_items,
|
||||||
|
f"No items match attention filter '{filter_class}'.",
|
||||||
|
)
|
||||||
|
|
||||||
|
body = f"""<h2>{escape(title)}</h2>
|
||||||
|
<p class="muted">Phase 3 console surface for human-attention routing (#648). Routine workflow transitions are filtered by default to eliminate notification fatigue.</p>
|
||||||
|
{err_html}
|
||||||
|
{metrics_html}
|
||||||
|
{tabs_html}
|
||||||
|
<h3>{escape(active_tab_title)}</h3>
|
||||||
|
{table_html}"""
|
||||||
|
|
||||||
|
return render_page(title=title, body_html=body)
|
||||||
@@ -0,0 +1,486 @@
|
|||||||
|
"""Notifications and human-attention routing module for Phase 3 web console (#648).
|
||||||
|
|
||||||
|
Defines attention classes, event classification rules, and inbox aggregation so
|
||||||
|
operators receive direct alerts only for human-required escalation boundaries
|
||||||
|
(#628) while routine workflow transitions remain available for pull-based review.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Callable
|
||||||
|
|
||||||
|
from webui import console_redaction
|
||||||
|
from webui.project_registry import load_registry
|
||||||
|
from webui.queue_loader import QueueSnapshot, load_queue_snapshot
|
||||||
|
from webui.lease_loader import LeaseSnapshot, load_lease_snapshot
|
||||||
|
from webui.system_health import SystemHealthSnapshot, load_system_health
|
||||||
|
|
||||||
|
# Attention class definitions (#628, #648)
|
||||||
|
ATTENTION_ROUTINE = "routine"
|
||||||
|
ATTENTION_OPERATOR = "operator"
|
||||||
|
ATTENTION_HUMAN_REQUIRED = "human-required"
|
||||||
|
|
||||||
|
ATTENTION_CLASSES = (
|
||||||
|
ATTENTION_ROUTINE,
|
||||||
|
ATTENTION_OPERATOR,
|
||||||
|
ATTENTION_HUMAN_REQUIRED,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Notification categories
|
||||||
|
CATEGORY_AUTH = "auth"
|
||||||
|
CATEGORY_BLOCKER = "blocker"
|
||||||
|
CATEGORY_LEASE = "lease"
|
||||||
|
CATEGORY_VALIDATION = "validation"
|
||||||
|
CATEGORY_WORKFLOW = "workflow"
|
||||||
|
CATEGORY_SYSTEM = "system"
|
||||||
|
|
||||||
|
CATEGORIES = (
|
||||||
|
CATEGORY_AUTH,
|
||||||
|
CATEGORY_BLOCKER,
|
||||||
|
CATEGORY_LEASE,
|
||||||
|
CATEGORY_VALIDATION,
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
CATEGORY_SYSTEM,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class NotificationItem:
|
||||||
|
"""A single notification or inbox event."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
attention_class: str # "routine", "operator", "human-required"
|
||||||
|
category: str # "auth", "blocker", "lease", "validation", etc.
|
||||||
|
title: str
|
||||||
|
summary: str
|
||||||
|
work_kind: str | None # "issue", "pr", "session", "system"
|
||||||
|
work_number: int | None
|
||||||
|
project_id: str
|
||||||
|
repo_label: str
|
||||||
|
created_at: str
|
||||||
|
deep_link: str | None = None
|
||||||
|
requires_human: bool = False
|
||||||
|
extra: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"attention_class": self.attention_class,
|
||||||
|
"category": self.category,
|
||||||
|
"title": self.title,
|
||||||
|
"summary": console_redaction.redact_text(self.summary),
|
||||||
|
"work_kind": self.work_kind,
|
||||||
|
"work_number": self.work_number,
|
||||||
|
"project_id": self.project_id,
|
||||||
|
"repo_label": self.repo_label,
|
||||||
|
"created_at": self.created_at,
|
||||||
|
"deep_link": self.deep_link,
|
||||||
|
"requires_human": self.requires_human,
|
||||||
|
"extra": self.extra,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class NotificationSnapshot:
|
||||||
|
"""Snapshot of notifications and attention inbox state."""
|
||||||
|
|
||||||
|
project_id: str
|
||||||
|
repo_label: str
|
||||||
|
items: tuple[NotificationItem, ...]
|
||||||
|
human_required_count: int
|
||||||
|
operator_count: int
|
||||||
|
routine_count: int
|
||||||
|
total_count: int
|
||||||
|
fetch_error: str | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def inbox_items(self) -> tuple[NotificationItem, ...]:
|
||||||
|
"""Items requiring operator or human attention (excluding routine)."""
|
||||||
|
return tuple(
|
||||||
|
item
|
||||||
|
for item in self.items
|
||||||
|
if item.attention_class in {ATTENTION_OPERATOR, ATTENTION_HUMAN_REQUIRED}
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def human_required_items(self) -> tuple[NotificationItem, ...]:
|
||||||
|
return tuple(
|
||||||
|
item for item in self.items if item.attention_class == ATTENTION_HUMAN_REQUIRED
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def operator_items(self) -> tuple[NotificationItem, ...]:
|
||||||
|
return tuple(
|
||||||
|
item for item in self.items if item.attention_class == ATTENTION_OPERATOR
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def routine_items(self) -> tuple[NotificationItem, ...]:
|
||||||
|
return tuple(
|
||||||
|
item for item in self.items if item.attention_class == ATTENTION_ROUTINE
|
||||||
|
)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"project_id": self.project_id,
|
||||||
|
"repo_label": self.repo_label,
|
||||||
|
"human_required_count": self.human_required_count,
|
||||||
|
"operator_count": self.operator_count,
|
||||||
|
"routine_count": self.routine_count,
|
||||||
|
"total_count": self.total_count,
|
||||||
|
"fetch_error": self.fetch_error,
|
||||||
|
"inbox_items": [item.as_dict() for item in self.inbox_items],
|
||||||
|
"all_items": [item.as_dict() for item in self.items],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def classify_attention_event(
|
||||||
|
category: str,
|
||||||
|
title: str,
|
||||||
|
summary: str,
|
||||||
|
*,
|
||||||
|
is_hard_stop: bool = False,
|
||||||
|
is_auth_failure: bool = False,
|
||||||
|
is_irrecoverable: bool = False,
|
||||||
|
is_decision_lock: bool = False,
|
||||||
|
is_validation_failure: bool = False,
|
||||||
|
is_stale: bool = False,
|
||||||
|
is_blocker: bool = False,
|
||||||
|
) -> tuple[str, bool]:
|
||||||
|
"""Classify an event into an attention class and human requirement flag.
|
||||||
|
|
||||||
|
Rules (#628, #648):
|
||||||
|
1. Critical boundaries (hard stop, auth failure, irrecoverable state,
|
||||||
|
decision lock, validation failure) -> ATTENTION_HUMAN_REQUIRED (requires_human=True).
|
||||||
|
2. Operational queues (blocker, stale lease, unassigned ready work, queue collision)
|
||||||
|
-> ATTENTION_OPERATOR (requires_human=False).
|
||||||
|
3. Routine state transitions (clean progression, healthy heartbeats) -> ATTENTION_ROUTINE (requires_human=False).
|
||||||
|
|
||||||
|
Classification uses structured flags and category only. Human-authored
|
||||||
|
``title`` / ``summary`` text is never substring-matched for escalation
|
||||||
|
(PR #905 review B1) — callers that need text signals must set flags from
|
||||||
|
machine-generated status/detail fields before calling this function.
|
||||||
|
"""
|
||||||
|
del title, summary # kept for API stability; never used for classification
|
||||||
|
if (
|
||||||
|
is_hard_stop
|
||||||
|
or is_auth_failure
|
||||||
|
or is_irrecoverable
|
||||||
|
or is_decision_lock
|
||||||
|
or is_validation_failure
|
||||||
|
or category in {CATEGORY_AUTH, CATEGORY_VALIDATION}
|
||||||
|
):
|
||||||
|
return ATTENTION_HUMAN_REQUIRED, True
|
||||||
|
|
||||||
|
if is_stale or is_blocker or category in {CATEGORY_BLOCKER, CATEGORY_LEASE}:
|
||||||
|
return ATTENTION_OPERATOR, False
|
||||||
|
|
||||||
|
return ATTENTION_ROUTINE, False
|
||||||
|
|
||||||
|
|
||||||
|
def load_notifications_snapshot(
|
||||||
|
project_id: str | None = None,
|
||||||
|
*,
|
||||||
|
load_queue: Callable[..., QueueSnapshot] | None = None,
|
||||||
|
load_leases: Callable[..., LeaseSnapshot] | None = None,
|
||||||
|
load_health: Callable[..., SystemHealthSnapshot] | None = None,
|
||||||
|
) -> NotificationSnapshot:
|
||||||
|
"""Load and classify attention notifications across queue, leases, and system health."""
|
||||||
|
registry = load_registry()
|
||||||
|
project = None
|
||||||
|
if project_id:
|
||||||
|
for entry in registry.projects:
|
||||||
|
if entry.id == project_id:
|
||||||
|
project = entry
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
project = registry.projects[0] if registry.projects else None
|
||||||
|
|
||||||
|
if project is None:
|
||||||
|
return NotificationSnapshot(
|
||||||
|
project_id=project_id or "",
|
||||||
|
repo_label="",
|
||||||
|
items=(),
|
||||||
|
human_required_count=0,
|
||||||
|
operator_count=0,
|
||||||
|
routine_count=0,
|
||||||
|
total_count=0,
|
||||||
|
fetch_error="project not found in registry",
|
||||||
|
)
|
||||||
|
|
||||||
|
queue_loader_fn = load_queue or load_queue_snapshot
|
||||||
|
lease_loader_fn = load_leases or load_lease_snapshot
|
||||||
|
health_loader_fn = load_health or load_system_health
|
||||||
|
|
||||||
|
try:
|
||||||
|
queue_snap = queue_loader_fn(project.id)
|
||||||
|
except TypeError:
|
||||||
|
queue_snap = queue_loader_fn(project_id=project.id)
|
||||||
|
|
||||||
|
try:
|
||||||
|
lease_snap = lease_loader_fn(project_id=project.id)
|
||||||
|
except TypeError:
|
||||||
|
lease_snap = lease_loader_fn(project.id)
|
||||||
|
|
||||||
|
try:
|
||||||
|
health_snap = health_loader_fn(project_id=project.id)
|
||||||
|
except TypeError:
|
||||||
|
try:
|
||||||
|
health_snap = health_loader_fn(project.id)
|
||||||
|
except TypeError:
|
||||||
|
health_snap = health_loader_fn()
|
||||||
|
|
||||||
|
items: list[NotificationItem] = []
|
||||||
|
now_iso = datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
# 1. System health alerts (highest priority)
|
||||||
|
for err_idx, probe_err in enumerate(getattr(health_snap, "probe_errors", ())):
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_SYSTEM,
|
||||||
|
"System Health Probe Error",
|
||||||
|
probe_err,
|
||||||
|
is_blocker=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-sys-err-{project.id}-{err_idx}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_SYSTEM,
|
||||||
|
title="System Health Error",
|
||||||
|
summary=f"System health error: {probe_err}",
|
||||||
|
work_kind="system",
|
||||||
|
work_number=None,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link="/system",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for probe in getattr(health_snap, "dependencies", ()):
|
||||||
|
if probe.status not in ("ok", "healthy"):
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_SYSTEM,
|
||||||
|
f"Probe Failure: {probe.name}",
|
||||||
|
probe.detail or probe.status,
|
||||||
|
is_hard_stop=("stop" in probe.status or "fatal" in probe.status),
|
||||||
|
is_auth_failure=("auth" in probe.name.lower() or "unauthorized" in probe.status.lower()),
|
||||||
|
is_blocker=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-probe-{probe.name}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_AUTH if "auth" in probe.name.lower() else CATEGORY_SYSTEM,
|
||||||
|
title=f"Health Probe Alert: {probe.name}",
|
||||||
|
summary=f"Probe '{probe.name}' reported status '{probe.status}': {probe.detail}",
|
||||||
|
work_kind="system",
|
||||||
|
work_number=None,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link="/system",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2. Queue items (PRs and Issues)
|
||||||
|
for pr in queue_snap.prs:
|
||||||
|
if "blocked" in pr.badges:
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_BLOCKER,
|
||||||
|
f"PR #{pr.number} Blocked",
|
||||||
|
f"PR #{pr.number} '{pr.title}' is blocked or has merge conflicts.",
|
||||||
|
is_blocker=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-pr-block-{pr.number}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_BLOCKER,
|
||||||
|
title=f"Blocked PR #{pr.number}",
|
||||||
|
summary=f"PR #{pr.number} ({pr.title}) requires merge conflict resolution.",
|
||||||
|
work_kind="pr",
|
||||||
|
work_number=pr.number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link=f"/traffic",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif "stale" in pr.badges:
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
f"PR #{pr.number} Stale",
|
||||||
|
f"PR #{pr.number} '{pr.title}' has had no activity for over 14 days.",
|
||||||
|
is_stale=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-pr-stale-{pr.number}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_WORKFLOW,
|
||||||
|
title=f"Stale PR #{pr.number}",
|
||||||
|
summary=f"PR #{pr.number} ({pr.title}) is stale.",
|
||||||
|
work_kind="pr",
|
||||||
|
work_number=pr.number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link=f"/queue",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Routine PR transition
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
f"PR #{pr.number} Active",
|
||||||
|
f"PR #{pr.number} '{pr.title}' is in routine state {', '.join(pr.badges)}.",
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-pr-routine-{pr.number}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_WORKFLOW,
|
||||||
|
title=f"Routine PR #{pr.number}",
|
||||||
|
summary=f"PR #{pr.number} ({pr.title}) state: {', '.join(pr.badges)}.",
|
||||||
|
work_kind="pr",
|
||||||
|
work_number=pr.number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link=f"/queue",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for issue in queue_snap.issues:
|
||||||
|
if "duplicate" in issue.badges:
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_BLOCKER,
|
||||||
|
f"Issue #{issue.number} Duplicate PRs",
|
||||||
|
f"Issue #{issue.number} has multiple linked PRs.",
|
||||||
|
is_blocker=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-issue-dup-{issue.number}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_BLOCKER,
|
||||||
|
title=f"Duplicate PRs on Issue #{issue.number}",
|
||||||
|
summary=f"Issue #{issue.number} ({issue.title}) linked to multiple PRs.",
|
||||||
|
work_kind="issue",
|
||||||
|
work_number=issue.number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link=f"/traffic",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif "claimed" in issue.badges or "in-review" in issue.badges:
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_WORKFLOW,
|
||||||
|
f"Issue #{issue.number} Active",
|
||||||
|
f"Issue #{issue.number} '{issue.title}' in state {', '.join(issue.badges)}.",
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-issue-routine-{issue.number}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_WORKFLOW,
|
||||||
|
title=f"Routine Issue #{issue.number}",
|
||||||
|
summary=f"Issue #{issue.number} ({issue.title}) state: {', '.join(issue.badges)}.",
|
||||||
|
work_kind="issue",
|
||||||
|
work_number=issue.number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link=f"/queue",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# 3. Leases / Collisions
|
||||||
|
for lease in lease_snap.reviewer_leases:
|
||||||
|
if lease.get("is_expired") or lease.get("status") == "expired":
|
||||||
|
pr_num = lease.get("pr_number") or lease.get("work_item_number")
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_LEASE,
|
||||||
|
f"Reviewer Lease Expired for PR #{pr_num}",
|
||||||
|
f"Reviewer lease for PR #{pr_num} has expired.",
|
||||||
|
is_stale=True,
|
||||||
|
)
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-lease-exp-pr-{pr_num}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_LEASE,
|
||||||
|
title=f"Expired Reviewer Lease (PR #{pr_num})",
|
||||||
|
summary=f"Reviewer lease for PR #{pr_num} expired.",
|
||||||
|
work_kind="pr",
|
||||||
|
work_number=pr_num,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link="/leases",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for col_idx, collision in enumerate(lease_snap.duplicate_prs):
|
||||||
|
att_cls, req_human = classify_attention_event(
|
||||||
|
CATEGORY_BLOCKER,
|
||||||
|
f"Duplicate PR Collision ({collision.kind})",
|
||||||
|
collision.message,
|
||||||
|
is_blocker=True,
|
||||||
|
)
|
||||||
|
issue_part = collision.issue_number if collision.issue_number is not None else "none"
|
||||||
|
kind_part = (collision.kind or "unknown").replace(" ", "-")
|
||||||
|
items.append(
|
||||||
|
NotificationItem(
|
||||||
|
id=f"notif-collision-{kind_part}-{issue_part}-{col_idx}",
|
||||||
|
attention_class=att_cls,
|
||||||
|
category=CATEGORY_BLOCKER,
|
||||||
|
title=f"Collision Alert ({collision.kind})",
|
||||||
|
summary=collision.message,
|
||||||
|
work_kind="issue" if collision.issue_number else "pr",
|
||||||
|
work_number=collision.issue_number,
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
created_at=now_iso,
|
||||||
|
deep_link="/leases",
|
||||||
|
requires_human=req_human,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
human_req_count = sum(1 for i in items if i.attention_class == ATTENTION_HUMAN_REQUIRED)
|
||||||
|
operator_count = sum(1 for i in items if i.attention_class == ATTENTION_OPERATOR)
|
||||||
|
routine_count = sum(1 for i in items if i.attention_class == ATTENTION_ROUTINE)
|
||||||
|
|
||||||
|
# Fetch errors are transport/load failures only — not probe results that
|
||||||
|
# already surface as first-class notification items (PR #905 review B3).
|
||||||
|
fetch_err = queue_snap.fetch_error or lease_snap.fetch_error
|
||||||
|
if isinstance(fetch_err, (tuple, list)):
|
||||||
|
fetch_err = "; ".join(fetch_err) if fetch_err else None
|
||||||
|
|
||||||
|
return NotificationSnapshot(
|
||||||
|
project_id=project.id,
|
||||||
|
repo_label=f"{project.gitea_owner}/{project.repo_name}",
|
||||||
|
items=tuple(items),
|
||||||
|
human_required_count=human_req_count,
|
||||||
|
operator_count=operator_count,
|
||||||
|
routine_count=routine_count,
|
||||||
|
total_count=len(items),
|
||||||
|
fetch_error=fetch_err,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_to_dict(snapshot: NotificationSnapshot) -> dict[str, Any]:
|
||||||
|
"""JSON-serializable export for /api/v1/notifications."""
|
||||||
|
return snapshot.as_dict()
|
||||||
@@ -0,0 +1,275 @@
|
|||||||
|
"""Sentry/GlitchTip observability and incident correlation loader for the console (#649, Phase 4).
|
||||||
|
|
||||||
|
Operators need to inspect provider connection status (Sentry/GlitchTip), error
|
||||||
|
correlations, and durable Gitea issue linkage — without treating raw incidents
|
||||||
|
as allocator work.
|
||||||
|
|
||||||
|
ADR authority model:
|
||||||
|
* Gitea owns work.
|
||||||
|
* Providers (Sentry/GlitchTip) observe incidents.
|
||||||
|
* Control-plane DB coordinates incident links.
|
||||||
|
* The #612 bridge reconciles observations into durable Gitea issues.
|
||||||
|
* The web console projects read-only state and gates mutations.
|
||||||
|
|
||||||
|
Redaction boundary:
|
||||||
|
* Provider auth tokens, DSNs, Authorization headers, and sensitive local file
|
||||||
|
paths are ALWAYS redacted before leaving this module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from control_plane_db import ControlPlaneDB
|
||||||
|
import sentry_incident_bridge
|
||||||
|
from webui import console_redaction
|
||||||
|
|
||||||
|
OBSERVABILITY_SCHEMA_VERSION = 1
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ProviderHealth:
|
||||||
|
"""Connection and health status of an observability provider."""
|
||||||
|
|
||||||
|
provider: str
|
||||||
|
base_url: str
|
||||||
|
org: str
|
||||||
|
project: str
|
||||||
|
configured: bool
|
||||||
|
status: str
|
||||||
|
bridge_enabled: bool
|
||||||
|
lookback: str
|
||||||
|
min_events_for_issue: int
|
||||||
|
self_hosted: bool
|
||||||
|
environment: str | None = None
|
||||||
|
credentials_present: bool = False
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
data = {
|
||||||
|
"provider": self.provider,
|
||||||
|
"base_url": self.base_url,
|
||||||
|
"org": self.org,
|
||||||
|
"project": self.project,
|
||||||
|
"configured": self.configured,
|
||||||
|
"status": self.status,
|
||||||
|
"bridge_enabled": self.bridge_enabled,
|
||||||
|
"lookback": self.lookback,
|
||||||
|
"min_events_for_issue": self.min_events_for_issue,
|
||||||
|
"self_hosted": self.self_hosted,
|
||||||
|
"environment": self.environment,
|
||||||
|
"credentials_present": self.credentials_present,
|
||||||
|
}
|
||||||
|
return console_redaction.redact_payload(data)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class CorrelatedIncidentLink:
|
||||||
|
"""One linked provider incident ↔ Gitea issue correlation record."""
|
||||||
|
|
||||||
|
link_id: int
|
||||||
|
provider: str
|
||||||
|
provider_base_url: str
|
||||||
|
provider_org: str
|
||||||
|
provider_project: str
|
||||||
|
provider_issue_id: str
|
||||||
|
provider_short_id: str | None
|
||||||
|
provider_permalink: str | None
|
||||||
|
fingerprint: str | None
|
||||||
|
gitea_org: str
|
||||||
|
gitea_repo: str
|
||||||
|
gitea_issue_number: int
|
||||||
|
linked_pr_numbers: list[int]
|
||||||
|
last_seen: str | None
|
||||||
|
event_count: int
|
||||||
|
created_at: str | None
|
||||||
|
updated_at: str | None
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
data = {
|
||||||
|
"link_id": self.link_id,
|
||||||
|
"provider": self.provider,
|
||||||
|
"provider_base_url": self.provider_base_url,
|
||||||
|
"provider_org": self.provider_org,
|
||||||
|
"provider_project": self.provider_project,
|
||||||
|
"provider_issue_id": self.provider_issue_id,
|
||||||
|
"provider_short_id": self.provider_short_id,
|
||||||
|
"provider_permalink": self.provider_permalink,
|
||||||
|
"fingerprint": self.fingerprint,
|
||||||
|
"gitea_org": self.gitea_org,
|
||||||
|
"gitea_repo": self.gitea_repo,
|
||||||
|
"gitea_issue_number": self.gitea_issue_number,
|
||||||
|
"linked_pr_numbers": self.linked_pr_numbers,
|
||||||
|
"last_seen": self.last_seen,
|
||||||
|
"event_count": self.event_count,
|
||||||
|
"created_at": self.created_at,
|
||||||
|
"updated_at": self.updated_at,
|
||||||
|
}
|
||||||
|
return console_redaction.redact_payload(data)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ObservabilitySnapshot:
|
||||||
|
"""Read-only snapshot of observability provider status and incident correlations."""
|
||||||
|
|
||||||
|
schema_version: int
|
||||||
|
providers: list[ProviderHealth]
|
||||||
|
links: list[CorrelatedIncidentLink]
|
||||||
|
total_links: int
|
||||||
|
sentry_links_count: int
|
||||||
|
glitchtip_links_count: int
|
||||||
|
bridge_active: bool
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema_version": self.schema_version,
|
||||||
|
"providers": [p.to_dict() for p in self.providers],
|
||||||
|
"links": [link.to_dict() for link in self.links],
|
||||||
|
"metrics": {
|
||||||
|
"total_links": self.total_links,
|
||||||
|
"sentry_links_count": self.sentry_links_count,
|
||||||
|
"glitchtip_links_count": self.glitchtip_links_count,
|
||||||
|
"bridge_active": self.bridge_active,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_provider_health(
|
||||||
|
provider_name: str = "sentry",
|
||||||
|
env: dict[str, str] | None = None,
|
||||||
|
) -> ProviderHealth:
|
||||||
|
"""Inspect configuration and connection health for an observability provider."""
|
||||||
|
source_env = dict(env if env is not None else os.environ)
|
||||||
|
if provider_name.lower() == "sentry":
|
||||||
|
config = sentry_incident_bridge.load_bridge_config(source_env)
|
||||||
|
token = sentry_incident_bridge.resolve_token(source_env)
|
||||||
|
has_token = bool(token)
|
||||||
|
configured = bool(config.org and config.project and has_token)
|
||||||
|
|
||||||
|
if not config.org or not config.project:
|
||||||
|
status = "not_configured"
|
||||||
|
elif not has_token:
|
||||||
|
status = "missing_token"
|
||||||
|
elif not config.bridge_enabled:
|
||||||
|
status = "disabled"
|
||||||
|
else:
|
||||||
|
status = "healthy"
|
||||||
|
|
||||||
|
return ProviderHealth(
|
||||||
|
provider="sentry",
|
||||||
|
base_url=config.base_url,
|
||||||
|
org=config.org or "unconfigured",
|
||||||
|
project=config.project or "unconfigured",
|
||||||
|
configured=configured,
|
||||||
|
status=status,
|
||||||
|
bridge_enabled=config.bridge_enabled,
|
||||||
|
lookback=config.lookback,
|
||||||
|
min_events_for_issue=config.min_events_for_issue,
|
||||||
|
self_hosted=not config.base_url.rstrip("/").endswith("sentry.io"),
|
||||||
|
environment=config.environment,
|
||||||
|
credentials_present=has_token,
|
||||||
|
)
|
||||||
|
|
||||||
|
# GlitchTip or fallback provider configuration
|
||||||
|
glitchtip_url = (source_env.get("GLITCHTIP_BASE_URL") or "https://glitchtip.prgs.cc").strip()
|
||||||
|
glitchtip_org = (source_env.get("GLITCHTIP_ORG") or "").strip()
|
||||||
|
glitchtip_proj = (source_env.get("GLITCHTIP_PROJECT") or "").strip()
|
||||||
|
glitchtip_token = (source_env.get("GLITCHTIP_AUTH_TOKEN") or "").strip()
|
||||||
|
|
||||||
|
has_token = bool(glitchtip_token)
|
||||||
|
configured = bool(glitchtip_org and glitchtip_proj and has_token)
|
||||||
|
status = "healthy" if configured else ("missing_token" if glitchtip_org and glitchtip_proj else "not_configured")
|
||||||
|
|
||||||
|
return ProviderHealth(
|
||||||
|
provider="glitchtip",
|
||||||
|
base_url=glitchtip_url,
|
||||||
|
org=glitchtip_org or "unconfigured",
|
||||||
|
project=glitchtip_proj or "unconfigured",
|
||||||
|
configured=configured,
|
||||||
|
status=status,
|
||||||
|
bridge_enabled=configured,
|
||||||
|
lookback="24h",
|
||||||
|
min_events_for_issue=2,
|
||||||
|
self_hosted=True,
|
||||||
|
environment=source_env.get("GLITCHTIP_ENVIRONMENT"),
|
||||||
|
credentials_present=has_token,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_pr_numbers(raw: Any) -> list[int]:
|
||||||
|
if isinstance(raw, list):
|
||||||
|
return [int(x) for x in raw if str(x).isdigit()]
|
||||||
|
if isinstance(raw, str) and raw.strip():
|
||||||
|
import json
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
if isinstance(parsed, list):
|
||||||
|
return [int(x) for x in parsed if str(x).isdigit()]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def load_observability_snapshot(
|
||||||
|
db: ControlPlaneDB | None = None,
|
||||||
|
env: dict[str, str] | None = None,
|
||||||
|
) -> ObservabilitySnapshot:
|
||||||
|
"""Build a read-only snapshot of observability connection health and incident links."""
|
||||||
|
sentry_health = load_provider_health("sentry", env)
|
||||||
|
glitchtip_health = load_provider_health("glitchtip", env)
|
||||||
|
providers = [sentry_health, glitchtip_health]
|
||||||
|
|
||||||
|
target_db = db or ControlPlaneDB()
|
||||||
|
raw_links = target_db.list_incident_links(limit=100)
|
||||||
|
|
||||||
|
links: list[CorrelatedIncidentLink] = []
|
||||||
|
sentry_cnt = 0
|
||||||
|
glitchtip_cnt = 0
|
||||||
|
|
||||||
|
for r in raw_links:
|
||||||
|
prov = (r.get("provider") or "sentry").lower()
|
||||||
|
if prov == "sentry":
|
||||||
|
sentry_cnt += 1
|
||||||
|
elif prov == "glitchtip":
|
||||||
|
glitchtip_cnt += 1
|
||||||
|
|
||||||
|
pr_nums = _parse_pr_numbers(r.get("linked_pr_numbers"))
|
||||||
|
|
||||||
|
links.append(
|
||||||
|
CorrelatedIncidentLink(
|
||||||
|
link_id=int(r.get("link_id", 0)),
|
||||||
|
provider=prov,
|
||||||
|
provider_base_url=r.get("provider_base_url") or "",
|
||||||
|
provider_org=r.get("provider_org") or "",
|
||||||
|
provider_project=r.get("provider_project") or "",
|
||||||
|
provider_issue_id=str(r.get("provider_issue_id") or ""),
|
||||||
|
provider_short_id=r.get("provider_short_id"),
|
||||||
|
provider_permalink=r.get("provider_permalink"),
|
||||||
|
fingerprint=r.get("fingerprint"),
|
||||||
|
gitea_org=r.get("gitea_org") or "Scaled-Tech-Consulting",
|
||||||
|
gitea_repo=r.get("gitea_repo") or "Gitea-Tools",
|
||||||
|
gitea_issue_number=int(r.get("gitea_issue_number", 0)),
|
||||||
|
linked_pr_numbers=pr_nums,
|
||||||
|
last_seen=r.get("last_seen"),
|
||||||
|
event_count=int(r.get("event_count", 1)),
|
||||||
|
created_at=r.get("created_at"),
|
||||||
|
updated_at=r.get("updated_at"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
bridge_active = any(p.bridge_enabled for p in providers)
|
||||||
|
|
||||||
|
return ObservabilitySnapshot(
|
||||||
|
schema_version=OBSERVABILITY_SCHEMA_VERSION,
|
||||||
|
providers=providers,
|
||||||
|
links=links,
|
||||||
|
total_links=len(links),
|
||||||
|
sentry_links_count=sentry_cnt,
|
||||||
|
glitchtip_links_count=glitchtip_cnt,
|
||||||
|
bridge_active=bridge_active,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_to_dict(snapshot: ObservabilitySnapshot) -> dict[str, Any]:
|
||||||
|
return snapshot.to_dict()
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
"""HTML view renderer for the Sentry/GlitchTip observability console (#649, Phase 4).
|
||||||
|
|
||||||
|
Renders connection status widgets, error correlation links, and gated issue creation
|
||||||
|
affordances over the read-only observability snapshot.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import html
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from webui.layout import render_page
|
||||||
|
from webui.observability_loader import ObservabilitySnapshot, snapshot_to_dict
|
||||||
|
|
||||||
|
|
||||||
|
def _badge(status: str) -> str:
|
||||||
|
st = (status or "").lower()
|
||||||
|
if st == "healthy":
|
||||||
|
return '<span class="badge badge-success">healthy</span>'
|
||||||
|
if st == "disabled":
|
||||||
|
return '<span class="badge badge-warning">disabled (dry-run)</span>'
|
||||||
|
if st in {"missing_token", "not_configured"}:
|
||||||
|
return f'<span class="badge badge-muted">{html.escape(st)}</span>'
|
||||||
|
return f'<span class="badge">{html.escape(st)}</span>'
|
||||||
|
|
||||||
|
|
||||||
|
def _provider_card(p: dict[str, Any]) -> str:
|
||||||
|
name = html.escape(str(p.get("provider", "provider")).upper())
|
||||||
|
base_url = html.escape(str(p.get("base_url", "")))
|
||||||
|
org = html.escape(str(p.get("org", "")))
|
||||||
|
proj = html.escape(str(p.get("project", "")))
|
||||||
|
status_badge = _badge(str(p.get("status", "")))
|
||||||
|
min_events = p.get("min_events_for_issue", 2)
|
||||||
|
lookback = html.escape(str(p.get("lookback", "24h")))
|
||||||
|
bridge_enabled = "yes" if p.get("bridge_enabled") else "no"
|
||||||
|
|
||||||
|
return f"""
|
||||||
|
<div class="card" style="margin-bottom: 1rem; padding: 1rem; border: 1px solid #ccc; border-radius: 6px;">
|
||||||
|
<div style="display: flex; justify-content: space-between; align-items: center;">
|
||||||
|
<h3 style="margin: 0;">{name} Connection</h3>
|
||||||
|
<div>{status_badge}</div>
|
||||||
|
</div>
|
||||||
|
<table style="width: 100%; margin-top: 0.5rem; border-collapse: collapse;">
|
||||||
|
<tr><td><strong>Base URL:</strong></td><td><code>{base_url}</code></td></tr>
|
||||||
|
<tr><td><strong>Scope:</strong></td><td><code>{org} / {proj}</code></td></tr>
|
||||||
|
<tr><td><strong>Bridge Enabled:</strong></td><td><code>{bridge_enabled}</code></td></tr>
|
||||||
|
<tr><td><strong>Min Events for Issue:</strong></td><td><code>{min_events}</code></td></tr>
|
||||||
|
<tr><td><strong>Lookback Window:</strong></td><td><code>{lookback}</code></td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def render_observability_page(snapshot: ObservabilitySnapshot | dict[str, Any]) -> str:
|
||||||
|
"""Render the observability dashboard HTML page."""
|
||||||
|
data = snapshot.to_dict() if isinstance(snapshot, ObservabilitySnapshot) else dict(snapshot)
|
||||||
|
|
||||||
|
providers_raw = data.get("providers", [])
|
||||||
|
provider_cards = "".join(_provider_card(p) for p in providers_raw) if providers_raw else "<p>No providers configured.</p>"
|
||||||
|
|
||||||
|
links = data.get("links", [])
|
||||||
|
link_rows = []
|
||||||
|
|
||||||
|
for l in links:
|
||||||
|
prov = html.escape(str(l.get("provider", "")))
|
||||||
|
p_issue_id = html.escape(str(l.get("provider_issue_id", "")))
|
||||||
|
fingerprint = html.escape(str(l.get("fingerprint") or "—"))
|
||||||
|
g_issue_num = int(l.get("gitea_issue_number", 0))
|
||||||
|
g_org = html.escape(str(l.get("gitea_org", "")))
|
||||||
|
g_repo = html.escape(str(l.get("gitea_repo", "")))
|
||||||
|
g_issue_link = f'<strong>#{g_issue_num}</strong> ({g_org}/{g_repo})'
|
||||||
|
event_cnt = int(l.get("event_count", 1))
|
||||||
|
last_seen = html.escape(str(l.get("last_seen") or "—"))
|
||||||
|
short_id = html.escape(str(l.get("provider_short_id") or p_issue_id))
|
||||||
|
|
||||||
|
link_rows.append(f"""
|
||||||
|
<tr>
|
||||||
|
<td><code>{prov}</code></td>
|
||||||
|
<td><strong>{short_id}</strong><br><small style="color: #666;">id: {p_issue_id}</small></td>
|
||||||
|
<td><code>{fingerprint}</code></td>
|
||||||
|
<td>{g_issue_link}</td>
|
||||||
|
<td>{event_cnt}</td>
|
||||||
|
<td><small>{last_seen}</small></td>
|
||||||
|
</tr>
|
||||||
|
""")
|
||||||
|
|
||||||
|
table_body = "".join(link_rows) if link_rows else '<tr><td colspan="6" style="text-align: center; padding: 1.5rem; color: #666;">No correlated incident links stored. Bridge operates under dry-run default.</td></tr>'
|
||||||
|
|
||||||
|
metrics = data.get("metrics", {})
|
||||||
|
total_links = metrics.get("total_links", 0)
|
||||||
|
sentry_cnt = metrics.get("sentry_links_count", 0)
|
||||||
|
glitchtip_cnt = metrics.get("glitchtip_links_count", 0)
|
||||||
|
|
||||||
|
body_html = f"""
|
||||||
|
<h2>Observability & Incident Bridge (#649)</h2>
|
||||||
|
<p>Read-only console surface for Sentry/GlitchTip provider connections, error correlation,
|
||||||
|
and durable Gitea issue linkage.</p>
|
||||||
|
|
||||||
|
<div class="alert alert-info" style="background: #f0f4f8; padding: 1rem; border-left: 4px solid #0052cc; margin-bottom: 1.5rem;">
|
||||||
|
<strong>ADR Authority Model:</strong> Gitea records durable issue history. Control-plane DB coordinates incident links.
|
||||||
|
Sentry/GlitchTip observe errors. Raw monitoring incidents are <em>never</em> assignable control-plane work items.
|
||||||
|
Durable issue creation is gated and dry-runable via the <code>#612</code> bridge APIs.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>Provider Connections</h3>
|
||||||
|
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 1rem; margin-bottom: 2rem;">
|
||||||
|
{provider_cards}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 1rem;">
|
||||||
|
<h3 style="margin: 0;">Correlated Incidents ({total_links})</h3>
|
||||||
|
<div>
|
||||||
|
<span class="badge" style="margin-right: 0.5rem;">Sentry: {sentry_cnt}</span>
|
||||||
|
<span class="badge">GlitchTip: {glitchtip_cnt}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<table class="table" style="width: 100%; border-collapse: collapse; border: 1px solid #ddd;">
|
||||||
|
<thead>
|
||||||
|
<tr style="background: #f9f9f9; text-align: left;">
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Provider</th>
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Incident ID</th>
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Fingerprint</th>
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Gitea Issue Link</th>
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Events</th>
|
||||||
|
<th style="padding: 0.5rem; border-bottom: 2px solid #ddd;">Last Seen</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{table_body}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<div style="margin-top: 2rem; padding: 1rem; background: #fafafa; border: 1px solid #eee; border-radius: 4px;">
|
||||||
|
<h4 style="margin-top: 0;">Reconcile & Link Controls (Gated)</h4>
|
||||||
|
<p style="margin-bottom: 0.5rem; color: #555;">
|
||||||
|
Create or reconcile durable Gitea issues from provider observations using the <code>#612</code> incident bridge:
|
||||||
|
</p>
|
||||||
|
<code>mcp call gitea_observability_reconcile_incident --provider sentry --apply false</code>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
return render_page(title="Observability", body_html=body_html)
|
||||||
Reference in New Issue
Block a user