Merge branch 'master' into feat/issue-636-inventory-api
This commit is contained in:
@@ -101,6 +101,7 @@ that gates each call, not which tools exist.
|
||||
- `gitea_get_profile`
|
||||
- `gitea_get_runtime_context`
|
||||
- `gitea_get_shell_health`
|
||||
- `gitea_heartbeat_issue_lock`
|
||||
- `gitea_heartbeat_reviewer_pr_lease`
|
||||
- `gitea_inspect_workflow_lease`
|
||||
- `gitea_issue_irrecoverable_provenance_authorization`
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Model Usage, Token Cost, Latency, and Workflow Analytics (Phase 4)
|
||||
|
||||
- **Tracking Issue:** [#651](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
|
||||
- **Parent Epic:** [#631](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/651)
|
||||
- **Console Surface:** `/analytics`, `/api/v1/analytics`, `/api/v1/analytics/usage`
|
||||
|
||||
## 1. Overview
|
||||
|
||||
The Web Console Analytics module provides durable, aggregate visibility into **model usage, token cost, latency percentiles, and workflow-stage performance** across projects, worker roles, AI models, issues, and PRs.
|
||||
|
||||
### Non-Goals
|
||||
- No mandatory client-side telemetry that leaks prompts or secret keys.
|
||||
- No third-party payment provider or billing integration.
|
||||
- No automatic model routing changes without controller policy (#647).
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Schema (`usage_events`)
|
||||
|
||||
Usage metrics are stored in the control-plane database under table `usage_events`.
|
||||
|
||||
| Column | Type | Description |
|
||||
|---|---|---|
|
||||
| `usage_id` | `INTEGER` | Primary key (autoincrement) |
|
||||
| `session_id` | `TEXT` | Optional active session identifier |
|
||||
| `remote` | `TEXT` | Known Gitea instance (`dadeschools` or `prgs`) |
|
||||
| `org` | `TEXT` | Repository owner / organization |
|
||||
| `repo` | `TEXT` | Repository name |
|
||||
| `project_id` | `TEXT` | Optional project identifier |
|
||||
| `role` | `TEXT` | Active worker role (`author`, `reviewer`, `merger`, `reconciler`, `controller`) |
|
||||
| `model` | `TEXT` | LLM model identifier (e.g. `gemini-3.6-flash`, `claude-3-5-sonnet`) |
|
||||
| `issue_number` | `INTEGER` | Correlated Gitea issue number (optional) |
|
||||
| `pr_number` | `INTEGER` | Correlated Gitea PR number (optional) |
|
||||
| `stage` | `TEXT` | Workflow stage (`preflight`, `implementation`, `review`, `merge`, `reconciliation`) |
|
||||
| `input_tokens` | `INTEGER` | Input token count (optional / nullable) |
|
||||
| `output_tokens` | `INTEGER` | Output token count (optional / nullable) |
|
||||
| `total_tokens` | `INTEGER` | Total token count (optional / nullable) |
|
||||
| `estimated_cost_usd` | `REAL` | Estimated USD cost (optional / nullable) |
|
||||
| `latency_ms` | `INTEGER` | Request latency in milliseconds (optional / nullable) |
|
||||
| `duration_ms` | `INTEGER` | Stage execution duration in milliseconds (optional / nullable) |
|
||||
| `status` | `TEXT` | Outcome status (`success`, `failure`, `timeout`) |
|
||||
| `metadata` | `TEXT` | Redacted metadata or summary string |
|
||||
| `created_at` | `TEXT` | ISO 8601 UTC timestamp |
|
||||
|
||||
---
|
||||
|
||||
## 3. Handling of Missing Data ("Unknown" vs. Zero Fabrication)
|
||||
|
||||
To ensure operational metrics accurately reflect evidence:
|
||||
- **Untracked or missing metrics are displayed as `Unknown`**, never zero-fabricated.
|
||||
- If an event omits `estimated_cost_usd`, `latency_ms`, or token counts, the aggregator marks those fields as missing (`None`) rather than defaulting to `0` or `$0.00`.
|
||||
- Summary tables and KPI cards explicitly indicate when data is unmeasured or partially reported.
|
||||
|
||||
---
|
||||
|
||||
## 4. Redaction & Security Rules
|
||||
|
||||
Per `#633` security policy:
|
||||
- Free-text fields (`metadata`, `prompt_summary`, `session_id`) are run through `console_redaction.redact_text` before persistence and output serialization.
|
||||
- Secret tokens, keychain commands, authorization headers, passwords, and JWTs are stripped automatically.
|
||||
|
||||
---
|
||||
|
||||
## 5. Opt-in Instrumentation Guide
|
||||
|
||||
Applications, MCP servers, and background sessions can report usage metrics through either Python API or HTTP ingestion.
|
||||
|
||||
### Python Ingestion
|
||||
|
||||
```python
|
||||
from webui.analytics_loader import record_usage
|
||||
|
||||
record_usage(
|
||||
remote="dadeschools",
|
||||
org="Scaled-Tech-Consulting",
|
||||
repo="Gitea-Tools",
|
||||
role="author",
|
||||
model="gemini-3.6-flash",
|
||||
issue_number=651,
|
||||
stage="implementation",
|
||||
input_tokens=1420,
|
||||
output_tokens=380,
|
||||
total_tokens=1800,
|
||||
estimated_cost_usd=0.00045,
|
||||
latency_ms=320,
|
||||
duration_ms=4500,
|
||||
status="success",
|
||||
metadata={"note": "Implementation of analytics module"},
|
||||
)
|
||||
```
|
||||
|
||||
### HTTP Ingestion API (authorized write)
|
||||
|
||||
`POST /api/v1/analytics/usage` is a **gated write**. It runs through
|
||||
`console_authz` action `record_analytics_usage` (operator+, Phase 2 execution).
|
||||
Unauthenticated or phase-inactive requests receive **403** and do not write.
|
||||
Prefer in-process `record_usage` for MCP / session instrumentation.
|
||||
|
||||
```http
|
||||
POST /api/v1/analytics/usage HTTP/1.1
|
||||
Content-Type: application/json
|
||||
# Requires authenticated principal with record_analytics_usage execution enabled
|
||||
|
||||
{
|
||||
"remote": "dadeschools",
|
||||
"org": "Scaled-Tech-Consulting",
|
||||
"repo": "Gitea-Tools",
|
||||
"role": "author",
|
||||
"model": "gemini-3.6-flash",
|
||||
"issue_number": 651,
|
||||
"stage": "implementation",
|
||||
"input_tokens": 1420,
|
||||
"output_tokens": 380,
|
||||
"total_tokens": 1800,
|
||||
"estimated_cost_usd": 0.00045,
|
||||
"latency_ms": 320,
|
||||
"duration_ms": 4500,
|
||||
"status": "success",
|
||||
"metadata": "Analytics schema landed"
|
||||
}
|
||||
```
|
||||
|
||||
### Retention
|
||||
|
||||
`usage_events` is retained with hard caps applied on every write:
|
||||
|
||||
| Limit | Default |
|
||||
|---|---|
|
||||
| Max rows | 10,000 (`ControlPlaneDB.USAGE_EVENTS_MAX_ROWS`) |
|
||||
| Max age | 90 days (`ControlPlaneDB.USAGE_EVENTS_RETENTION_DAYS`) |
|
||||
|
||||
Older rows (by `created_at`) and excess oldest rows (by `usage_id`) are deleted
|
||||
after each insert so unbounded growth / DoS-by-volume cannot fill the DB.
|
||||
|
||||
---
|
||||
|
||||
## 6. Querying Analytics API
|
||||
|
||||
```http
|
||||
GET /api/v1/analytics?role=author&stage=implementation HTTP/1.1
|
||||
```
|
||||
|
||||
Returns `AnalyticsSnapshot` JSON containing aggregations (`by_model`, `by_stage`, `by_role`, `by_work_item`, `by_project`) and latency percentiles (`p50`, `p90`, `p95`, `p99`).
|
||||
@@ -0,0 +1,56 @@
|
||||
# Post-restart MCP reconciliation (#662)
|
||||
|
||||
After an MCP process restart, sessions, leases, capabilities, worktrees, and
|
||||
interrupted mutations must be reconciled before operators claim a clean runtime.
|
||||
This document describes the #662 completion-proof path.
|
||||
|
||||
## Components
|
||||
|
||||
| Piece | Where | Responsibility |
|
||||
|-------|-------|----------------|
|
||||
| `post_restart_reconcile.reconcile_after_restart` | `post_restart_reconcile.py` | Pure classification: inventory → completion proof DTO. No I/O. |
|
||||
| `RestartCompletionProof` | `post_restart_reconcile.py` | Machine-readable proof (`.as_dict()` is JSON-serializable). |
|
||||
| `gitea_reconcile_after_restart` | `gitea_mcp_server.py` | MCP tool: gathers inventory from the #613 control-plane DB + master-parity, classifies, returns the proof. Read-only. |
|
||||
| Boot hook | `gitea_assess_master_parity` | First post-restart parity probe also runs reconcile once (log-only by default). |
|
||||
|
||||
## Dimensions
|
||||
|
||||
The assessor classifies:
|
||||
|
||||
- **service_health** — process healthy / parity mutation-safe
|
||||
- **clients** — connected client descriptors (optional inventory)
|
||||
- **sessions** — active session rows with dead owner pids are unresolved
|
||||
- **checkpoints** — soft-depends on #660; skipped with reason when schema absent
|
||||
- **leases** — live control-plane leases after restart
|
||||
- **capabilities** — master-parity / stale-runtime (#610)
|
||||
- **worktrees** — lease-bound paths missing on disk
|
||||
- **interrupted_mutations** — mutating lease phases or explicit pending inventory; **never auto-resumed**
|
||||
- **duplicates** — multiple live claims on the same work item
|
||||
- **queue** — allocator resume safety
|
||||
|
||||
## Modes
|
||||
|
||||
| Mode | Env / arg | Behavior |
|
||||
|------|-----------|----------|
|
||||
| `log_only` (default) | unset or `GITEA_POST_RESTART_RECONCILE_MODE=log_only` | Proof only; `mutation_hold=false` |
|
||||
| `enforce` | `GITEA_POST_RESTART_RECONCILE_MODE=enforce` or `mode=enforce` | Sets `mutation_hold=true` when overall status is degraded/failed or interrupted mutations remain |
|
||||
|
||||
## Follow-up issues
|
||||
|
||||
Unresolved dimensions produce `proposed_follow_ups` entries suitable for durable
|
||||
Gitea issues. The MCP tool **does not create** those issues in v1 (rollout is
|
||||
log-only first). Controllers may file them from the proof payload.
|
||||
|
||||
## Links
|
||||
|
||||
- Umbrella: #655
|
||||
- Vision: #652 · Roadmap: #653
|
||||
- Checkpoint schema: #660 (soft dependency)
|
||||
- Drain proof: #661 (soft)
|
||||
- This issue: #662
|
||||
|
||||
## Non-goals
|
||||
|
||||
- HA multi-instance failover
|
||||
- Automatic silent mutation replay
|
||||
- Implementing the #660 checkpoint schema itself
|
||||
@@ -91,6 +91,7 @@ already define, and a regression test asserts each mapping matches.
|
||||
| `close_pr` | controller | privileged | `gitea.pr.close` | Yes | No | No | 3 |
|
||||
| `merge_pr` | controller | privileged | `gitea.pr.merge` | Yes | **Yes** | **Yes** | 3 |
|
||||
| `delete_branch` | admin | destructive | `gitea.branch.delete` | Yes | **Yes** | **Yes** | 3 |
|
||||
| `record_analytics_usage` | operator | gated_write | `runtime.record_analytics_usage` | Yes | No | No | 2 |
|
||||
| `system.reload_namespace` | controller | privileged | `runtime.reload_namespace` | Yes | No | No | 2 |
|
||||
| `system.restart_namespace` | admin | destructive | `runtime.restart_namespace` | Yes | **Yes** | **Yes** | 2 |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user