Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 73 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

Web Console: Model usage, token cost, latency, and workflow-performance analytics (Phase 4) #651

New Issue
Open
opened 2026-07-10 14:42:32 -05:00 by jcwalker3 · 0 comments
jcwalker3 commented 2026-07-10 14:42:32 -05:00
Owner
Copy Link
Copy Source

Problem statement

There is no durable console analytics for model usage, token cost, latency, and workflow-stage performance correlated to issues/PRs/roles.

User and operational impact

Cannot optimize cost/latency or identify expensive failing stages; budget and performance conversations lack evidence.

Scope

  • Ingest or accept optional usage metrics from session instrumentation (when available).
  • Aggregate by project, role, model, issue/PR, stage.
  • Dashboards for cost, tokens, latency percentiles, stage duration.
  • Missing data must display as unknown, not zero-fabricated.

Explicit non-goals

  • No mandatory client-side telemetry that leaks prompts with secrets.
  • No billing integration with payment providers in this issue.
  • No automatic model routing changes without controller policy (#647).

Required implementation investigation

Session logs if any, #637 timeline, #650 insights, #631, privacy/redaction rules #633.

Proposed architecture or implementation direction

Metrics store (CP-DB or append-only files) with schema; aggregation API; charts simple server-rendered tables first.

Security and workflow-safety requirements

Redact prompts/secrets from metrics; RBAC; retention limits.

Acceptance criteria

  1. Schema for usage events documented.
  2. Aggregations by project/role/model/stage.
  3. Unknown data not shown as zero without label.
  4. Tests for aggregation and redaction.
  5. Docs for enabling instrumentation opt-in.

Required tests

Aggregation unit tests; privacy redaction tests.

Observability and audit requirements

Metrics pipeline health optional on system health (#634).

Dependencies and linkage

Canonical issue state

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Implement usage/cost/latency analytics schema + views
NEXT_PROMPT: Author analytics only; PR; stop

Required final evidence

PR with schema, API/views, tests, docs.

Required final response and handoff expectations

Brief PR → reviewer.

## Problem statement There is no durable console analytics for **model usage, token cost, latency, and workflow-stage performance** correlated to issues/PRs/roles. ## User and operational impact Cannot optimize cost/latency or identify expensive failing stages; budget and performance conversations lack evidence. ## Scope * Ingest or accept optional usage metrics from session instrumentation (when available). * Aggregate by project, role, model, issue/PR, stage. * Dashboards for cost, tokens, latency percentiles, stage duration. * Missing data must display as unknown, not zero-fabricated. ## Explicit non-goals * No mandatory client-side telemetry that leaks prompts with secrets. * No billing integration with payment providers in this issue. * No automatic model routing changes without controller policy (#647). ## Required implementation investigation Session logs if any, #637 timeline, #650 insights, #631, privacy/redaction rules #633. ## Proposed architecture or implementation direction Metrics store (CP-DB or append-only files) with schema; aggregation API; charts simple server-rendered tables first. ## Security and workflow-safety requirements Redact prompts/secrets from metrics; RBAC; retention limits. ## Acceptance criteria 1. Schema for usage events documented. 2. Aggregations by project/role/model/stage. 3. Unknown data not shown as zero without label. 4. Tests for aggregation and redaction. 5. Docs for enabling instrumentation opt-in. ## Required tests Aggregation unit tests; privacy redaction tests. ## Observability and audit requirements Metrics pipeline health optional on system health (#634). ## Dependencies and linkage * Parent: #631 · Soft-depends: #637, #633, #634 · Related: #650 ## Canonical issue state ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Implement usage/cost/latency analytics schema + views NEXT_PROMPT: Author analytics only; PR; stop ``` ## Required final evidence PR with schema, API/views, tests, docs. ## Required final response and handoff expectations Brief PR → reviewer.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
allocator
Controller-owned work allocator
 anti-stomp
Prevent concurrent LLM session stomping
 architecture
Architecture / structural design
 bug
chore
codex
OpenAI Codex client / workflow session surface
 concurrency
Concurrent session safety
 contamination
Workflow or session contamination incident
 control-plane
MCP control-plane coordination and allocation authority
 dashboard
MCP operational dashboard/queue view
 database
Internal coordination storage (SQLite/Postgres)
 design
Design / investigation, no implementation
 documentation
Docs / runbooks
 enhancement
New feature or improvement
 gitea
Gitea MCP workflow
 glitchtip
GlitchTip integration
 important
incident
Operational or process incident requiring durable audit trail
 incident-bridge
Sentry-to-Gitea incident bridging
 integration
Integration testing
 jenkins
Jenkins integration
 labels
Label taxonomy management
 leases
Lease adopt/release/expire lifecycle
 mcp
MCP server / tooling
 mcp-health
MCP namespace and runtime health
 mcp-menu
MCP menu surface
 multi-project
Work spanning multiple monitoring projects or Gitea repos
 mutating
Mutating action; requires gating
 nice-to-have
observability
Observability, metrics, traces, error reporting
 portability
Cross-platform / portability
 preflight
Shared preflight gates before mutation
 protected-branch
Protected branch / stable-branch policy concern
 queue
Work queue visibility and allocation
 read-only
Read-only, no mutation
 reconnect
MCP client reconnect/reload recovery path
 recovery
Recovery paths for stale/foreign leases
 refactor
Code refactor / restructure
 release
Release / versioning
 reliability
Reliability / failure handling
 resumable-review
Persist and resume prepared review verdicts across sessions
 reviewer
Reviewer workflow tooling
 roadmap
Roadmap / umbrella issue
 safety
Safety rails and fail-closed mutation guards
 security
Security / trust boundary
 self-hosted
Self-hosted infrastructure integration
 sentry
Sentry error monitoring integration
 stale-runtime
Stale backend daemon / runtime-vs-master parity failures
 status:blocked
Issue is blocked
 status:in-progress
Issue is being worked on
 status:pr-open
Issue has an open pull request
 status:ready
Issue is ready for work
 terminal-lock
Terminal review lock (#332) path
 testing
Tests / test coverage
 tracker
Issue tracker hygiene / meta
 type:bug
Bug or defect
 type:feature
Feature or enhancement
 type:feature
Feature or enhancement
 type:guardrail
Safety gate or guardrail
 visibility
Workflow state visibility for LLMs/operators
 workflow
Cross-tool workflow
 workflow-hardening
LLM workflow coordination hardening
 workflow-hardening
LLM workflow coordination hardening
No labels dashboard observability status:ready type:feature workflow-hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcwalker3 sysadmin
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Scaled-Tech-Consulting/Gitea-Tools#651