feat(webui): model usage, token cost, latency, and performance analytics (Closes #651)

2026-07-24 07:16:44 -04:00
parent c33c69b3f3
commit 5c5c1fdf77
8 changed files with 1226 additions and 2 deletions
@@ -0,0 +1,125 @@
+# Model Usage, Token Cost, Latency, and Workflow Analytics (Phase 4)
+
+- **Tracking Issue:** [#651](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
+- **Parent Epic:** [#631](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
+- **Console Surface:** `/analytics`, `/api/v1/analytics`, `/api/v1/analytics/usage`
+
+## 1. Overview
+
+The Web Console Analytics module provides durable, aggregate visibility into **model usage, token cost, latency percentiles, and workflow-stage performance** across projects, worker roles, AI models, issues, and PRs.
+
+### Non-Goals
+- No mandatory client-side telemetry that leaks prompts or secret keys.
+- No third-party payment provider or billing integration.
+- No automatic model routing changes without controller policy (#647).
+
+---
+
+## 2. Event Schema (`usage_events`)
+
+Usage metrics are stored in the control-plane database under table `usage_events`.
+
+| Column | Type | Description |
+|---|---|---|
+| `usage_id` | `INTEGER` | Primary key (autoincrement) |
+| `session_id` | `TEXT` | Optional active session identifier |
+| `remote` | `TEXT` | Known Gitea instance (`dadeschools` or `prgs`) |
+| `org` | `TEXT` | Repository owner / organization |
+| `repo` | `TEXT` | Repository name |
+| `project_id` | `TEXT` | Optional project identifier |
+| `role` | `TEXT` | Active worker role (`author`, `reviewer`, `merger`, `reconciler`, `controller`) |
+| `model` | `TEXT` | LLM model identifier (e.g. `gemini-3.6-flash`, `claude-3-5-sonnet`) |
+| `issue_number` | `INTEGER` | Correlated Gitea issue number (optional) |
+| `pr_number` | `INTEGER` | Correlated Gitea PR number (optional) |
+| `stage` | `TEXT` | Workflow stage (`preflight`, `implementation`, `review`, `merge`, `reconciliation`) |
+| `input_tokens` | `INTEGER` | Input token count (optional / nullable) |
+| `output_tokens` | `INTEGER` | Output token count (optional / nullable) |
+| `total_tokens` | `INTEGER` | Total token count (optional / nullable) |
+| `estimated_cost_usd` | `REAL` | Estimated USD cost (optional / nullable) |
+| `latency_ms` | `INTEGER` | Request latency in milliseconds (optional / nullable) |
+| `duration_ms` | `INTEGER` | Stage execution duration in milliseconds (optional / nullable) |
+| `status` | `TEXT` | Outcome status (`success`, `failure`, `timeout`) |
+| `metadata` | `TEXT` | Redacted metadata or summary string |
+| `created_at` | `TEXT` | ISO 8601 UTC timestamp |
+
+---
+
+## 3. Handling of Missing Data ("Unknown" vs. Zero Fabrication)
+
+To ensure operational metrics accurately reflect evidence:
+- **Untracked or missing metrics are displayed as `Unknown`**, never zero-fabricated.
+- If an event omits `estimated_cost_usd`, `latency_ms`, or token counts, the aggregator marks those fields as missing (`None`) rather than defaulting to `0` or `$0.00`.
+- Summary tables and KPI cards explicitly indicate when data is unmeasured or partially reported.
+
+---
+
+## 4. Redaction & Security Rules
+
+Per `#633` security policy:
+- Free-text fields (`metadata`, `prompt_summary`, `session_id`) are run through `console_redaction.redact_text` before persistence and output serialization.
+- Secret tokens, keychain commands, authorization headers, passwords, and JWTs are stripped automatically.
+
+---
+
+## 5. Opt-in Instrumentation Guide
+
+Applications, MCP servers, and background sessions can report usage metrics through either Python API or HTTP ingestion.
+
+### Python Ingestion
+
+```python
+from webui.analytics_loader import record_usage
+
+record_usage(
+    remote="dadeschools",
+    org="Scaled-Tech-Consulting",
+    repo="Gitea-Tools",
+    role="author",
+    model="gemini-3.6-flash",
+    issue_number=651,
+    stage="implementation",
+    input_tokens=1420,
+    output_tokens=380,
+    total_tokens=1800,
+    estimated_cost_usd=0.00045,
+    latency_ms=320,
+    duration_ms=4500,
+    status="success",
+    metadata={"note": "Implementation of analytics module"},
+)
+```
+
+### HTTP Ingestion API
+
+```http
+POST /api/v1/analytics/usage HTTP/1.1
+Content-Type: application/json
+
+{
+  "remote": "dadeschools",
+  "org": "Scaled-Tech-Consulting",
+  "repo": "Gitea-Tools",
+  "role": "author",
+  "model": "gemini-3.6-flash",
+  "issue_number": 651,
+  "stage": "implementation",
+  "input_tokens": 1420,
+  "output_tokens": 380,
+  "total_tokens": 1800,
+  "estimated_cost_usd": 0.00045,
+  "latency_ms": 320,
+  "duration_ms": 4500,
+  "status": "success",
+  "metadata": "Analytics schema landed"
+}
+```
+
+---
+
+## 6. Querying Analytics API
+
+```http
+GET /api/v1/analytics?role=author&stage=implementation HTTP/1.1
+```
+
+Returns `AnalyticsSnapshot` JSON containing aggregations (`by_model`, `by_stage`, `by_role`, `by_work_item`, `by_project`) and latency percentiles (`p50`, `p90`, `p95`, `p99`).