Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c1ecadce8e |
@@ -1,167 +0,0 @@
|
|||||||
# ADR: High-availability and rolling-restart architecture for Gitea MCP control plane
|
|
||||||
|
|
||||||
- **Status:** Proposed (Design ADR under [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668))
|
|
||||||
- **Date:** 2026-07-25
|
|
||||||
- **Tracking Issue:** [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668)
|
|
||||||
- **Policy Version:** `mcp-ha-rolling-restart/v1`
|
|
||||||
- **Related:**
|
|
||||||
- Parent: [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655) — Governed MCP restart coordination and zero-disruption recovery
|
|
||||||
- Governance Policy: [#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656) / `docs/architecture/mcp-restart-governance.md`
|
|
||||||
- Control-Plane DB Substrate: [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) / `docs/architecture/control-plane-db-substrate.md`
|
|
||||||
- Runtime Policy: [#615](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/615) / `docs/architecture/mcp-stable-control-runtime-policy-adr.md`
|
|
||||||
- Product Vision: [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) (Phase 5 Maturity)
|
|
||||||
- Delivery Roadmap: [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Context & Problem Statement
|
|
||||||
|
|
||||||
The Gitea MCP server operates as the authoritative **control plane** for managing issues, Pull Requests, code mutations, formal reviews, and workflow reconciliations. Under single-process governance ([#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656)), process restarts are strictly controlled using pre-flight checks, drain phases, and operator approvals.
|
|
||||||
|
|
||||||
However, a single-instance control plane inherently presents fundamental constraints:
|
|
||||||
|
|
||||||
1. **Downtime during updates:** Even a perfectly executed single-process drain requires a window where incoming client requests must be paused or rejected while the server binary or python environment reloads.
|
|
||||||
2. **Single point of failure:** Infrastructure issues, process crashes, or unhandled host-level terminations immediately disconnect active LLM sessions and leave transient workflows incomplete.
|
|
||||||
3. **Multi-agent concurrency bottlenecks:** High volumes of concurrent multi-LLM tasks put all lock management, lease allocation, and Gitea API interactions through a single process event loop.
|
|
||||||
|
|
||||||
To achieve true zero-disruption operation and seamless rolling deployments without stopping active work, the system requires a high-availability (HA), multi-instance MCP architecture.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Architectural Principles & Non-Goals
|
|
||||||
|
|
||||||
### 2.1 Core Architectural Principles
|
|
||||||
* **Gitea as Canonical Work SoT:** Gitea remains the ultimate System of Record (SoT) for issue states, pull requests, labels, and audit comments. The MCP control plane does not duplicate domain entities.
|
|
||||||
* **Control-Plane DB as Multi-Instance State Substrate:** The control-plane SQLite/durable database ([#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613)) acts as the single source of truth for workflow leases, session tokens, assignment records, and lock fences across all MCP nodes.
|
|
||||||
* **Stateless Worker Nodes:** MCP role server processes (`gitea-author`, `gitea-reviewer`, `gitea-merger`, `gitea-reconciler`, `gitea-controller`) maintain no unique in-memory state; any node can handle any request given a valid session resume token.
|
|
||||||
* **Fail-Closed Split-Brain Defense:** In any network partition or quorum loss scenario, nodes must fail closed rather than risk double-mutations or conflicting Gitea states.
|
|
||||||
|
|
||||||
### 2.2 Non-Goals
|
|
||||||
* **Replacing Gitea:** We do not replace Gitea issue/PR tracking with an independent database.
|
|
||||||
* **Immediate Multi-Node Cluster Execution in v1:** This ADR defines the target architecture and phased roadmap; immediate implementation occurs incrementally post-[#655] v1.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. High-Availability & Rolling-Restart Architecture
|
|
||||||
|
|
||||||
### 3.1 Architecture Overview
|
|
||||||
|
|
||||||
```
|
|
||||||
+----------------------------+
|
|
||||||
| LLM Clients / IDE Sessions |
|
|
||||||
+--------------+-------------+
|
|
||||||
|
|
|
||||||
v
|
|
||||||
+----------------------------+
|
|
||||||
| HA Proxy / Router |
|
|
||||||
| (Health-based & Affinity) |
|
|
||||||
+------+--------------+------+
|
|
||||||
| |
|
|
||||||
+--------------+ +--------------+
|
|
||||||
v v
|
|
||||||
+--------------------+ +--------------------+
|
|
||||||
| MCP Instance Node A| | MCP Instance Node B|
|
|
||||||
| (Version N) | | (Version N+1) |
|
|
||||||
+---------+----------+ +---------+----------+
|
|
||||||
| |
|
|
||||||
+----------------------+----------------------+
|
|
||||||
|
|
|
||||||
v
|
|
||||||
+----------------------------+
|
|
||||||
| Control-Plane DB Substrate|
|
|
||||||
| (Shared Lease & Locks) |
|
|
||||||
+--------------+-------------+
|
|
||||||
|
|
|
||||||
v
|
|
||||||
+----------------------------+
|
|
||||||
| Gitea API |
|
|
||||||
+----------------------------+
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3.2 Key System Components
|
|
||||||
|
|
||||||
#### A. Multiple MCP Instance Cohorts
|
|
||||||
* The control plane runs across $N \ge 2$ redundant process nodes.
|
|
||||||
* Dual-namespace deployment allows running the old version (Node A) alongside a updated version (Node B) during rolling upgrades.
|
|
||||||
|
|
||||||
#### B. Shared Durable Session Storage & Resume Tokens
|
|
||||||
* Session context, preflight verification proofs, and capability resolution states are stored in the shared control-plane database.
|
|
||||||
* Client requests carry an explicit `session_id` and `resume_token`. If an MCP instance restarts or a request routes to a different instance, the target node validates the token against the database without requiring full session re-initialization.
|
|
||||||
|
|
||||||
#### C. Shared Lease Authority & Fencing Counters
|
|
||||||
* Workflow leases (`gitea_allocate_next_work`, `gitea_adopt_workflow_lease`) use monotonic fencing tokens (`lease_generation_id`).
|
|
||||||
* When Node B acquires or renews a lease, it increments the generation counter. Any delayed or out-of-order write attempt from Node A using an older generation token is rejected by database constraints.
|
|
||||||
|
|
||||||
#### D. Leader Election & Coordinated Drain
|
|
||||||
* Node clusters elect a primary coordinator node for administrative background tasks (such as stale lease cleanup or incident Watchdogs).
|
|
||||||
* During a rolling deployment:
|
|
||||||
1. Node B (new version) is launched and registers as healthy.
|
|
||||||
2. Router directs new session creations to Node B.
|
|
||||||
3. Node A enters `MAINTENANCE_DRAIN` status ([#659](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/659)), completing in-flight mutations while refusing new tasks.
|
|
||||||
4. Once all active sessions migrate or complete, Node A shuts down cleanly.
|
|
||||||
|
|
||||||
#### E. Idempotent Mutations & Failover Safety
|
|
||||||
* All state-changing tool executions (PR creation, review submission, merge operations, label changes) carry a deterministic `idempotency_key`.
|
|
||||||
* If a network connection flaps or a node fails mid-mutation, the re-issued request with the same `idempotency_key` is recognized by the control-plane substrate, returning the existing recorded result without repeating side effects on Gitea.
|
|
||||||
|
|
||||||
#### F. Schema Version Compatibility
|
|
||||||
* Database migrations follow non-breaking additive patterns.
|
|
||||||
* During rolling upgrades where Node A (Version $N$) and Node B (Version $N+1$) run concurrently, both versions operate against the shared schema without structural conflicts.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Split-Brain & Failure Behavior
|
|
||||||
|
|
||||||
### 4.1 Split-Brain Risk Scenarios & Mitigation
|
|
||||||
|
|
||||||
| Scenario | Risk | Mitigation Strategy |
|
|
||||||
|---|---|---|
|
|
||||||
| **Network Partition between Nodes** | Both Node A and Node B attempt to process operations for the same issue/PR. | **Generation Fencing:** Lease renewal requires updating the DB generation counter. The node isolated from the DB fails closed immediately. |
|
|
||||||
| **Stale Node Recovery** | Node A recovers after a long pause and executes a queued mutation. | **Lease Expiry & TTL Fencing:** Transactions verify that `expires_at > NOW()` within the atomic SQLite transaction boundaries. |
|
|
||||||
| **Database Connection Loss** | Node loses access to shared control-plane DB substrate. | **Strict Fail-Closed:** The node immediately marks all task capabilities as `blocked` and rejects mutation tools until DB connectivity is re-established. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Phased Implementation Milestones
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart TD
|
|
||||||
M1[Milestone 1: Shared Control-Plane DB Schema & Resume Tokens] --> M2[Milestone 2: Idempotent Mutation Layer]
|
|
||||||
M2 --> M3[Milestone 3: Health Routing & Standby Failover]
|
|
||||||
M3 --> M4[Milestone 4: Active-Active Rolling Deployment & Auto-Drain]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Milestone 1: Shared Control-Plane DB Schema & Resume Tokens (Post-#655)
|
|
||||||
* Extend [#613] Control-Plane DB schema to store multi-instance node heartbeat records and session resume tokens.
|
|
||||||
* Enable session lookup across instances via `session_id`.
|
|
||||||
|
|
||||||
### Milestone 2: Idempotent Mutation Layer & Lease Fencing
|
|
||||||
* Add mandatory `idempotency_key` tracking to all Gitea mutation tools.
|
|
||||||
* Implement monotonic lease fencing counters in `gitea_allocate_next_work` and `gitea_adopt_workflow_lease`.
|
|
||||||
|
|
||||||
### Milestone 3: Health-Based Routing & Active-Passive Standby
|
|
||||||
* Introduce lightweight proxy/router capable of checking node health endpoints.
|
|
||||||
* Implement active-standby failover where standby node automatically assumes work if active node fails health checks.
|
|
||||||
|
|
||||||
### Milestone 4: Active-Active Horizontal Deployment & Rolling Upgrade Automation
|
|
||||||
* Enable true active-active multi-instance execution.
|
|
||||||
* Integrate automated zero-downtime rolling upgrades coordinated with `gitea_request_mcp_restart` maintenance drain.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Observability & Audit Requirements
|
|
||||||
|
|
||||||
High-availability control plane operations must expose clear telemetry and audit trails:
|
|
||||||
|
|
||||||
* **Node Registry Telemetry:** Active nodes, version numbers, uptime, and heartbeat timestamps reported via `gitea_get_runtime_context`.
|
|
||||||
* **Lease Fencing Metrics:** Tracking lease acquire latency, fence rejection counts, and lease handoff durations.
|
|
||||||
* **Failover & Re-route Audit Logs:** Durable logging of session migrations between nodes, drain initiation, and process retirement events.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Tradeoffs & Accepted Risks
|
|
||||||
|
|
||||||
* **Increased Architectural Complexity:** Moving from a single process to a multi-instance control plane requires robust DB locking, proxy routing, and migration governance.
|
|
||||||
* **Database Dependency:** The control-plane database substrate becomes a critical shared dependency for multi-node deployments. High availability for the underlying SQLite file system / DB must be guaranteed.
|
|
||||||
@@ -134,6 +134,19 @@ tool argument expresses caller intent and cannot be self-asserted by a worker
|
|||||||
session. `break_glass_requested` and `break_glass_authorized` are both reported,
|
session. `break_glass_requested` and `break_glass_authorized` are both reported,
|
||||||
so a bypass is never silent.
|
so a bypass is never silent.
|
||||||
|
|
||||||
|
### Break-glass Restart Workflow (`gitea_break_glass_restart`, #664)
|
||||||
|
|
||||||
|
The dedicated MCP tool `gitea_break_glass_restart` provides the privileged emergency break-glass restart workflow when graceful drain cannot complete:
|
||||||
|
|
||||||
|
- **Role Authorization (#664 AC1)**: Ordinary LLM worker roles (`author`, `reviewer`, `merger`, `reconciler`) are denied fail-closed. Privileged `controller` role or explicit `GITEA_BREAKGLASS_RESTART_AUTHORIZATION` is required.
|
||||||
|
- **Required Parameters (#664 AC2)**:
|
||||||
|
- `reason`: Mandatory non-empty string (min 10 characters).
|
||||||
|
- `confirmation`: Must equal exactly `"I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"`.
|
||||||
|
- `impact_ack`: Must be `True`.
|
||||||
|
- **Automatic Incident Creation (#664 AC3)**: Creates a Gitea incident issue (`[INCIDENT] Break-glass MCP restart invoked by ...`) detailing the reason, timestamp, disrupted sessions, and linking `#652 #653 #655 #630 #658 #662 #664`.
|
||||||
|
- **Immutable Audit Entry**: Records an immutable audit log entry under `event="break_glass_mcp_restart"`.
|
||||||
|
- **Mandatory Reconciliation (#664 AC4)**: Sets `reconciliation_required=True` requiring post-restart reconciliation via `gitea_reconcile_after_restart` (#662).
|
||||||
|
|
||||||
### Fail closed on apply
|
### Fail closed on apply
|
||||||
|
|
||||||
A missing, malformed, expired, unclean, tampered, or fingerprint-stale drain
|
A missing, malformed, expired, unclean, tampered, or fingerprint-stale drain
|
||||||
@@ -150,3 +163,4 @@ profiles are operational metadata only.
|
|||||||
|
|
||||||
A representative dry-run report is in
|
A representative dry-run report is in
|
||||||
[`mcp-restart-impact-sample.json`](./mcp-restart-impact-sample.json).
|
[`mcp-restart-impact-sample.json`](./mcp-restart-impact-sample.json).
|
||||||
|
|
||||||
|
|||||||
@@ -22812,6 +22812,205 @@ def gitea_request_mcp_restart(
|
|||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
BREAK_GLASS_CONFIRMATION_PHRASE = "I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"
|
||||||
|
|
||||||
|
|
||||||
|
@mcp.tool()
|
||||||
|
def gitea_break_glass_restart(
|
||||||
|
reason: str,
|
||||||
|
confirmation: str,
|
||||||
|
impact_ack: bool = False,
|
||||||
|
restart_class: str = "full_mcp_restart",
|
||||||
|
create_incident_issue: bool = True,
|
||||||
|
dry_run: bool = False,
|
||||||
|
remote: str = "dadeschools",
|
||||||
|
host: str | None = None,
|
||||||
|
org: str | None = None,
|
||||||
|
repo: str | None = None,
|
||||||
|
worktree_path: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""Privileged emergency break-glass MCP restart workflow (#664).
|
||||||
|
|
||||||
|
Break-glass restart permits emergency recovery when graceful drain cannot
|
||||||
|
complete. It requires:
|
||||||
|
1. Privileged caller authorization (ordinary LLM author/reviewer/merger/reconciler
|
||||||
|
roles are denied fail-closed; controller/admin/sysadmin or GITEA_BREAKGLASS_RESTART_AUTHORIZATION
|
||||||
|
is required).
|
||||||
|
2. Explicit non-empty reason (minimum 10 characters).
|
||||||
|
3. Exact confirmation string matching 'I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION'.
|
||||||
|
4. Mandatory impact acknowledgement (impact_ack=True).
|
||||||
|
5. Immutable audit entry recorded.
|
||||||
|
6. Automatic incident record created on Gitea.
|
||||||
|
7. Mandatory post-restart reconciliation requirement (#662).
|
||||||
|
"""
|
||||||
|
read_block = _profile_operation_gate("gitea.read")
|
||||||
|
if read_block:
|
||||||
|
return {
|
||||||
|
"success": False,
|
||||||
|
"performed": False,
|
||||||
|
"break_glass_executed": False,
|
||||||
|
"reasons": read_block,
|
||||||
|
"permission_report": _permission_block_report("gitea.read"),
|
||||||
|
"blocker_kind": "permission_denied",
|
||||||
|
}
|
||||||
|
|
||||||
|
h, o, r = _resolve(remote, host, org, repo)
|
||||||
|
profile = get_profile()
|
||||||
|
active_role = _profile_role_kind(profile)
|
||||||
|
break_glass_env_auth = bool(
|
||||||
|
(os.environ.get("GITEA_BREAKGLASS_RESTART_AUTHORIZATION") or "").strip()
|
||||||
|
)
|
||||||
|
|
||||||
|
# AC1: Ordinary LLM roles (author, reviewer, merger, reconciler) cannot invoke break-glass
|
||||||
|
# unless explicit environment break-glass authorization is configured.
|
||||||
|
if active_role in ("author", "reviewer", "merger", "reconciler") and not break_glass_env_auth:
|
||||||
|
return {
|
||||||
|
"success": False,
|
||||||
|
"performed": False,
|
||||||
|
"break_glass_executed": False,
|
||||||
|
"active_role": active_role,
|
||||||
|
"reasons": [
|
||||||
|
f"ordinary LLM role '{active_role}' is forbidden from break-glass restarts (#664 AC1); "
|
||||||
|
"privileged controller, operator, or GITEA_BREAKGLASS_RESTART_AUTHORIZATION required"
|
||||||
|
],
|
||||||
|
"blocker_kind": "role_authorization",
|
||||||
|
}
|
||||||
|
|
||||||
|
# AC2: Required fields enforced
|
||||||
|
clean_reason = (reason or "").strip()
|
||||||
|
if not clean_reason or len(clean_reason) < 10:
|
||||||
|
return {
|
||||||
|
"success": False,
|
||||||
|
"performed": False,
|
||||||
|
"break_glass_executed": False,
|
||||||
|
"reasons": [
|
||||||
|
"reason is required and must be at least 10 characters long (#664 AC2)"
|
||||||
|
],
|
||||||
|
"blocker_kind": "missing_required_fields",
|
||||||
|
}
|
||||||
|
|
||||||
|
clean_confirmation = (confirmation or "").strip()
|
||||||
|
if clean_confirmation != BREAK_GLASS_CONFIRMATION_PHRASE:
|
||||||
|
return {
|
||||||
|
"success": False,
|
||||||
|
"performed": False,
|
||||||
|
"break_glass_executed": False,
|
||||||
|
"reasons": [
|
||||||
|
f"confirmation string mismatch; must equal exactly '{BREAK_GLASS_CONFIRMATION_PHRASE}' (#664 AC2)"
|
||||||
|
],
|
||||||
|
"blocker_kind": "confirmation_mismatch",
|
||||||
|
}
|
||||||
|
|
||||||
|
if not impact_ack:
|
||||||
|
return {
|
||||||
|
"success": False,
|
||||||
|
"performed": False,
|
||||||
|
"break_glass_executed": False,
|
||||||
|
"reasons": [
|
||||||
|
"impact_ack must be True to acknowledge disruption of in-flight sessions (#664 AC2)"
|
||||||
|
],
|
||||||
|
"blocker_kind": "impact_ack_required",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Evaluate impact / disrupted sessions
|
||||||
|
impact_result = gitea_request_mcp_restart(
|
||||||
|
remote=remote,
|
||||||
|
host=host,
|
||||||
|
org=org,
|
||||||
|
repo=repo,
|
||||||
|
dry_run=True,
|
||||||
|
restart_class=restart_class,
|
||||||
|
request_break_glass=True,
|
||||||
|
)
|
||||||
|
disrupted_sessions = list(impact_result.get("affected_sessions") or [])
|
||||||
|
disrupted_count = len(disrupted_sessions)
|
||||||
|
|
||||||
|
identity = _authenticated_username(h) or profile.get("username") or "unknown"
|
||||||
|
now_iso = datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
audit_payload = {
|
||||||
|
"event": "break_glass_mcp_restart",
|
||||||
|
"actor": identity,
|
||||||
|
"role": active_role,
|
||||||
|
"timestamp": now_iso,
|
||||||
|
"reason": clean_reason,
|
||||||
|
"confirmation": clean_confirmation,
|
||||||
|
"restart_class": restart_class,
|
||||||
|
"disrupted_sessions_count": disrupted_count,
|
||||||
|
"disrupted_sessions": [s.get("session_id") if isinstance(s, dict) else str(s) for s in disrupted_sessions],
|
||||||
|
"dry_run": dry_run,
|
||||||
|
"remote": remote,
|
||||||
|
"org": o,
|
||||||
|
"repo": r,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Save immutable audit record
|
||||||
|
saved_audit = mcp_session_state.save_state(
|
||||||
|
kind="break_glass_audit",
|
||||||
|
payload=audit_payload,
|
||||||
|
remote=remote,
|
||||||
|
org=o,
|
||||||
|
repo=r,
|
||||||
|
profile_identity=profile.get("profile_name", "unknown"),
|
||||||
|
)
|
||||||
|
|
||||||
|
incident_issue_result = None
|
||||||
|
if create_incident_issue and not dry_run:
|
||||||
|
issue_title = f"[INCIDENT] Break-glass MCP restart invoked by {identity}"
|
||||||
|
issue_body = (
|
||||||
|
f"## Break-glass MCP restart incident report (#664)\n\n"
|
||||||
|
f"- **Invoked by**: `{identity}` (role: `{active_role}`)\n"
|
||||||
|
f"- **Timestamp**: `{now_iso}`\n"
|
||||||
|
f"- **Reason**: {clean_reason}\n"
|
||||||
|
f"- **Confirmation**: `{clean_confirmation}`\n"
|
||||||
|
f"- **Disrupted Sessions Count**: `{disrupted_count}`\n\n"
|
||||||
|
f"### Mandatory Post-Restart Reconciliation (#662)\n"
|
||||||
|
f"Post-restart reconciliation must be executed via `gitea_reconcile_after_restart` "
|
||||||
|
f"to clean up orphaned leases, inspect worktree integrity, and handle disrupted work.\n\n"
|
||||||
|
f"### Cross-references\n"
|
||||||
|
f"Ref #652 #653 #655 #630 #658 #662 #664\n"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
incident_issue_result = api_request(
|
||||||
|
"POST",
|
||||||
|
f"{repo_api_url(h, o, r)}/issues",
|
||||||
|
_auth(h),
|
||||||
|
{
|
||||||
|
"title": issue_title,
|
||||||
|
"body": issue_body,
|
||||||
|
"labels": ["incident", "mcp-health", "break-glass"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
incident_issue_result = {"error": _redact(str(exc))}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"success": True,
|
||||||
|
"performed": not dry_run,
|
||||||
|
"dry_run": dry_run,
|
||||||
|
"break_glass_executed": not dry_run,
|
||||||
|
"would_execute": True,
|
||||||
|
"actor": identity,
|
||||||
|
"role": active_role,
|
||||||
|
"restart_class": restart_class,
|
||||||
|
"reason": clean_reason,
|
||||||
|
"confirmation": clean_confirmation,
|
||||||
|
"disrupted_sessions_count": disrupted_count,
|
||||||
|
"disrupted_sessions": disrupted_sessions,
|
||||||
|
"audit_record": audit_payload,
|
||||||
|
"saved_audit": dict(saved_audit or audit_payload),
|
||||||
|
"incident_issue": incident_issue_result,
|
||||||
|
"reconciliation_required": True,
|
||||||
|
"reconciliation_tool": "gitea_reconcile_after_restart",
|
||||||
|
"follow_up_issue_required": True,
|
||||||
|
"cross_references": ["#652", "#653", "#655", "#630", "#658", "#662", "#664"],
|
||||||
|
"reasons": [
|
||||||
|
"break-glass restart dry-run evaluated successfully" if dry_run
|
||||||
|
else "break-glass restart executed with incident creation and mandatory reconciliation"
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# --- #662 post-restart reconciliation ---------------------------------------
|
# --- #662 post-restart reconciliation ---------------------------------------
|
||||||
|
|
||||||
_POST_RESTART_LAST_PROOF: dict | None = None
|
_POST_RESTART_LAST_PROOF: dict | None = None
|
||||||
|
|||||||
@@ -538,6 +538,15 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
|
|||||||
"permission": "runtime.record_analytics_usage",
|
"permission": "runtime.record_analytics_usage",
|
||||||
"role": "author",
|
"role": "author",
|
||||||
},
|
},
|
||||||
|
# #664: emergency break-glass MCP restart workflow (privileged controller role).
|
||||||
|
"break_glass_restart": {
|
||||||
|
"permission": "gitea.read",
|
||||||
|
"role": "controller",
|
||||||
|
},
|
||||||
|
"gitea_break_glass_restart": {
|
||||||
|
"permission": "gitea.read",
|
||||||
|
"role": "controller",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,190 @@
|
|||||||
|
"""Tests for emergency break-glass MCP restart workflow (#664)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import gitea_mcp_server
|
||||||
|
|
||||||
|
|
||||||
|
class TestBreakGlassRestart(unittest.TestCase):
|
||||||
|
"""Test suite for gitea_break_glass_restart tool and guardrails (#664)."""
|
||||||
|
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.env_patcher = patch.dict(os.environ, {}, clear=False)
|
||||||
|
self.env_patcher.start()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.env_patcher.stop()
|
||||||
|
|
||||||
|
def test_ordinary_role_denied_fail_closed(self) -> None:
|
||||||
|
"""AC1: Ordinary LLM roles (author, reviewer, merger, reconciler) cannot invoke break-glass."""
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-author", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="author"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Emergency restart needed due to deadlocked worker daemon processes",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertFalse(res["success"])
|
||||||
|
self.assertFalse(res["break_glass_executed"])
|
||||||
|
self.assertEqual(res["blocker_kind"], "role_authorization")
|
||||||
|
self.assertIn("ordinary LLM role 'author' is forbidden", res["reasons"][0])
|
||||||
|
|
||||||
|
def test_short_reason_denied(self) -> None:
|
||||||
|
"""AC2: Reason is required and must be at least 10 characters long."""
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Too short",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertFalse(res["success"])
|
||||||
|
self.assertEqual(res["blocker_kind"], "missing_required_fields")
|
||||||
|
self.assertIn("at least 10 characters", res["reasons"][0])
|
||||||
|
|
||||||
|
def test_confirmation_mismatch_denied(self) -> None:
|
||||||
|
"""AC2: Confirmation phrase must match exact required string."""
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Emergency restart needed due to stuck daemon processes",
|
||||||
|
confirmation="wrong_confirmation_phrase",
|
||||||
|
impact_ack=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertFalse(res["success"])
|
||||||
|
self.assertEqual(res["blocker_kind"], "confirmation_mismatch")
|
||||||
|
self.assertIn("confirmation string mismatch", res["reasons"][0])
|
||||||
|
|
||||||
|
def test_impact_ack_required_denied(self) -> None:
|
||||||
|
"""AC2: impact_ack=True is mandatory."""
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Emergency restart needed due to stuck daemon processes",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=False,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertFalse(res["success"])
|
||||||
|
self.assertEqual(res["blocker_kind"], "impact_ack_required")
|
||||||
|
|
||||||
|
def test_dry_run_evaluation(self) -> None:
|
||||||
|
"""AC5: Dry-run evaluation returns preview without live execution or incident creation."""
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": [{"session_id": "s1"}]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Emergency restart needed due to hung worker process cohort",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=True,
|
||||||
|
dry_run=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertTrue(res["success"])
|
||||||
|
self.assertTrue(res["dry_run"])
|
||||||
|
self.assertFalse(res["break_glass_executed"])
|
||||||
|
self.assertTrue(res["would_execute"])
|
||||||
|
self.assertTrue(res["reconciliation_required"])
|
||||||
|
self.assertEqual(res["reconciliation_tool"], "gitea_reconcile_after_restart")
|
||||||
|
self.assertIn("#664", res["cross_references"])
|
||||||
|
|
||||||
|
def test_privileged_execute_creates_incident_and_audit(self) -> None:
|
||||||
|
"""AC3 & AC4: Execution creates incident issue, audit entry, and mandates post-restart reconcile."""
|
||||||
|
mock_api_request = MagicMock(return_value={"number": 999, "title": "[INCIDENT] Break-glass"})
|
||||||
|
mock_save_state = MagicMock(return_value={"saved": True})
|
||||||
|
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": [{"session_id": "s1"}]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "api_request", mock_api_request
|
||||||
|
), patch("mcp_session_state.save_state", mock_save_state):
|
||||||
|
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Emergency break-glass restart due to unrecoverable transport deadlock",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=True,
|
||||||
|
dry_run=False,
|
||||||
|
create_incident_issue=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertTrue(res["success"])
|
||||||
|
self.assertFalse(res["dry_run"])
|
||||||
|
self.assertTrue(res["break_glass_executed"])
|
||||||
|
self.assertTrue(res["reconciliation_required"])
|
||||||
|
self.assertEqual(res["reconciliation_tool"], "gitea_reconcile_after_restart")
|
||||||
|
self.assertEqual(res["incident_issue"]["number"], 999)
|
||||||
|
mock_save_state.assert_called_once()
|
||||||
|
mock_api_request.assert_called_once()
|
||||||
|
|
||||||
|
def test_env_authorization_override_for_worker_role(self) -> None:
|
||||||
|
"""Environment break-glass authorization enables privileged break-glass for configured sessions."""
|
||||||
|
os.environ["GITEA_BREAKGLASS_RESTART_AUTHORIZATION"] = "authorized-token"
|
||||||
|
with patch.object(
|
||||||
|
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-author", "allowed_operations": ["gitea.read"]}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_role_kind", return_value="author"
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": []}
|
||||||
|
), patch.object(
|
||||||
|
gitea_mcp_server, "_authenticated_username", return_value="jcwalker3"
|
||||||
|
):
|
||||||
|
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||||
|
reason="Authorized emergency break-glass restart test",
|
||||||
|
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||||
|
impact_ack=True,
|
||||||
|
dry_run=True,
|
||||||
|
remote="prgs",
|
||||||
|
)
|
||||||
|
self.assertTrue(res["success"])
|
||||||
|
self.assertTrue(res["dry_run"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user