126 lines
4.6 KiB
Markdown
126 lines
4.6 KiB
Markdown
# Model Usage, Token Cost, Latency, and Workflow Analytics (Phase 4)
|
|
|
|
- **Tracking Issue:** [#651](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
|
|
- **Parent Epic:** [#631](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
|
|
- **Console Surface:** `/analytics`, `/api/v1/analytics`, `/api/v1/analytics/usage`
|
|
|
|
## 1. Overview
|
|
|
|
The Web Console Analytics module provides durable, aggregate visibility into **model usage, token cost, latency percentiles, and workflow-stage performance** across projects, worker roles, AI models, issues, and PRs.
|
|
|
|
### Non-Goals
|
|
- No mandatory client-side telemetry that leaks prompts or secret keys.
|
|
- No third-party payment provider or billing integration.
|
|
- No automatic model routing changes without controller policy (#647).
|
|
|
|
---
|
|
|
|
## 2. Event Schema (`usage_events`)
|
|
|
|
Usage metrics are stored in the control-plane database under table `usage_events`.
|
|
|
|
| Column | Type | Description |
|
|
|---|---|---|
|
|
| `usage_id` | `INTEGER` | Primary key (autoincrement) |
|
|
| `session_id` | `TEXT` | Optional active session identifier |
|
|
| `remote` | `TEXT` | Known Gitea instance (`dadeschools` or `prgs`) |
|
|
| `org` | `TEXT` | Repository owner / organization |
|
|
| `repo` | `TEXT` | Repository name |
|
|
| `project_id` | `TEXT` | Optional project identifier |
|
|
| `role` | `TEXT` | Active worker role (`author`, `reviewer`, `merger`, `reconciler`, `controller`) |
|
|
| `model` | `TEXT` | LLM model identifier (e.g. `gemini-3.6-flash`, `claude-3-5-sonnet`) |
|
|
| `issue_number` | `INTEGER` | Correlated Gitea issue number (optional) |
|
|
| `pr_number` | `INTEGER` | Correlated Gitea PR number (optional) |
|
|
| `stage` | `TEXT` | Workflow stage (`preflight`, `implementation`, `review`, `merge`, `reconciliation`) |
|
|
| `input_tokens` | `INTEGER` | Input token count (optional / nullable) |
|
|
| `output_tokens` | `INTEGER` | Output token count (optional / nullable) |
|
|
| `total_tokens` | `INTEGER` | Total token count (optional / nullable) |
|
|
| `estimated_cost_usd` | `REAL` | Estimated USD cost (optional / nullable) |
|
|
| `latency_ms` | `INTEGER` | Request latency in milliseconds (optional / nullable) |
|
|
| `duration_ms` | `INTEGER` | Stage execution duration in milliseconds (optional / nullable) |
|
|
| `status` | `TEXT` | Outcome status (`success`, `failure`, `timeout`) |
|
|
| `metadata` | `TEXT` | Redacted metadata or summary string |
|
|
| `created_at` | `TEXT` | ISO 8601 UTC timestamp |
|
|
|
|
---
|
|
|
|
## 3. Handling of Missing Data ("Unknown" vs. Zero Fabrication)
|
|
|
|
To ensure operational metrics accurately reflect evidence:
|
|
- **Untracked or missing metrics are displayed as `Unknown`**, never zero-fabricated.
|
|
- If an event omits `estimated_cost_usd`, `latency_ms`, or token counts, the aggregator marks those fields as missing (`None`) rather than defaulting to `0` or `$0.00`.
|
|
- Summary tables and KPI cards explicitly indicate when data is unmeasured or partially reported.
|
|
|
|
---
|
|
|
|
## 4. Redaction & Security Rules
|
|
|
|
Per `#633` security policy:
|
|
- Free-text fields (`metadata`, `prompt_summary`, `session_id`) are run through `console_redaction.redact_text` before persistence and output serialization.
|
|
- Secret tokens, keychain commands, authorization headers, passwords, and JWTs are stripped automatically.
|
|
|
|
---
|
|
|
|
## 5. Opt-in Instrumentation Guide
|
|
|
|
Applications, MCP servers, and background sessions can report usage metrics through either Python API or HTTP ingestion.
|
|
|
|
### Python Ingestion
|
|
|
|
```python
|
|
from webui.analytics_loader import record_usage
|
|
|
|
record_usage(
|
|
remote="dadeschools",
|
|
org="Scaled-Tech-Consulting",
|
|
repo="Gitea-Tools",
|
|
role="author",
|
|
model="gemini-3.6-flash",
|
|
issue_number=651,
|
|
stage="implementation",
|
|
input_tokens=1420,
|
|
output_tokens=380,
|
|
total_tokens=1800,
|
|
estimated_cost_usd=0.00045,
|
|
latency_ms=320,
|
|
duration_ms=4500,
|
|
status="success",
|
|
metadata={"note": "Implementation of analytics module"},
|
|
)
|
|
```
|
|
|
|
### HTTP Ingestion API
|
|
|
|
```http
|
|
POST /api/v1/analytics/usage HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"remote": "dadeschools",
|
|
"org": "Scaled-Tech-Consulting",
|
|
"repo": "Gitea-Tools",
|
|
"role": "author",
|
|
"model": "gemini-3.6-flash",
|
|
"issue_number": 651,
|
|
"stage": "implementation",
|
|
"input_tokens": 1420,
|
|
"output_tokens": 380,
|
|
"total_tokens": 1800,
|
|
"estimated_cost_usd": 0.00045,
|
|
"latency_ms": 320,
|
|
"duration_ms": 4500,
|
|
"status": "success",
|
|
"metadata": "Analytics schema landed"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Querying Analytics API
|
|
|
|
```http
|
|
GET /api/v1/analytics?role=author&stage=implementation HTTP/1.1
|
|
```
|
|
|
|
Returns `AnalyticsSnapshot` JSON containing aggregations (`by_model`, `by_stage`, `by_role`, `by_work_item`, `by_project`) and latency percentiles (`p50`, `p90`, `p95`, `p99`).
|