Model Usage, Token Cost, Latency, and Workflow Analytics (Phase 4)

Tracking Issue: #651
Parent Epic: #631
Console Surface: /analytics, /api/v1/analytics, /api/v1/analytics/usage

1. Overview

The Web Console Analytics module provides durable, aggregate visibility into model usage, token cost, latency percentiles, and workflow-stage performance across projects, worker roles, AI models, issues, and PRs.

Non-Goals

No mandatory client-side telemetry that leaks prompts or secret keys.
No third-party payment provider or billing integration.
No automatic model routing changes without controller policy (#647).

2. Event Schema (`usage_events`)

Usage metrics are stored in the control-plane database under table usage_events.

Column	Type	Description
`usage_id`	`INTEGER`	Primary key (autoincrement)
`session_id`	`TEXT`	Optional active session identifier
`remote`	`TEXT`	Known Gitea instance (`dadeschools` or `prgs`)
`org`	`TEXT`	Repository owner / organization
`repo`	`TEXT`	Repository name
`project_id`	`TEXT`	Optional project identifier
`role`	`TEXT`	Active worker role (`author`, `reviewer`, `merger`, `reconciler`, `controller`)
`model`	`TEXT`	LLM model identifier (e.g. `gemini-3.6-flash`, `claude-3-5-sonnet`)
`issue_number`	`INTEGER`	Correlated Gitea issue number (optional)
`pr_number`	`INTEGER`	Correlated Gitea PR number (optional)
`stage`	`TEXT`	Workflow stage (`preflight`, `implementation`, `review`, `merge`, `reconciliation`)
`input_tokens`	`INTEGER`	Input token count (optional / nullable)
`output_tokens`	`INTEGER`	Output token count (optional / nullable)
`total_tokens`	`INTEGER`	Total token count (optional / nullable)
`estimated_cost_usd`	`REAL`	Estimated USD cost (optional / nullable)
`latency_ms`	`INTEGER`	Request latency in milliseconds (optional / nullable)
`duration_ms`	`INTEGER`	Stage execution duration in milliseconds (optional / nullable)
`status`	`TEXT`	Outcome status (`success`, `failure`, `timeout`)
`metadata`	`TEXT`	Redacted metadata or summary string
`created_at`	`TEXT`	ISO 8601 UTC timestamp

3. Handling of Missing Data ("Unknown" vs. Zero Fabrication)

To ensure operational metrics accurately reflect evidence:

Untracked or missing metrics are displayed as Unknown, never zero-fabricated.
If an event omits estimated_cost_usd, latency_ms, or token counts, the aggregator marks those fields as missing (None) rather than defaulting to 0 or $0.00.
Summary tables and KPI cards explicitly indicate when data is unmeasured or partially reported.

4. Redaction & Security Rules

Per #633 security policy:

Free-text fields (metadata, prompt_summary, session_id) are run through console_redaction.redact_text before persistence and output serialization.
Secret tokens, keychain commands, authorization headers, passwords, and JWTs are stripped automatically.

5. Opt-in Instrumentation Guide

Applications, MCP servers, and background sessions can report usage metrics through either Python API or HTTP ingestion.

Python Ingestion

from webui.analytics_loader import record_usage

record_usage(
    remote="dadeschools",
    org="Scaled-Tech-Consulting",
    repo="Gitea-Tools",
    role="author",
    model="gemini-3.6-flash",
    issue_number=651,
    stage="implementation",
    input_tokens=1420,
    output_tokens=380,
    total_tokens=1800,
    estimated_cost_usd=0.00045,
    latency_ms=320,
    duration_ms=4500,
    status="success",
    metadata={"note": "Implementation of analytics module"},
)

HTTP Ingestion API

POST /api/v1/analytics/usage HTTP/1.1
Content-Type: application/json

{
  "remote": "dadeschools",
  "org": "Scaled-Tech-Consulting",
  "repo": "Gitea-Tools",
  "role": "author",
  "model": "gemini-3.6-flash",
  "issue_number": 651,
  "stage": "implementation",
  "input_tokens": 1420,
  "output_tokens": 380,
  "total_tokens": 1800,
  "estimated_cost_usd": 0.00045,
  "latency_ms": 320,
  "duration_ms": 4500,
  "status": "success",
  "metadata": "Analytics schema landed"
}

6. Querying Analytics API

GET /api/v1/analytics?role=author&stage=implementation HTTP/1.1

Returns AnalyticsSnapshot JSON containing aggregations (by_model, by_stage, by_role, by_work_item, by_project) and latency percentiles (p50, p90, p95, p99).

4.6 KiB Raw Blame History