Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sysadmin
|
6010f4295b |
@@ -0,0 +1,167 @@
|
||||
# ADR: High-availability and rolling-restart architecture for Gitea MCP control plane
|
||||
|
||||
- **Status:** Proposed (Design ADR under [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668))
|
||||
- **Date:** 2026-07-25
|
||||
- **Tracking Issue:** [#668](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/668)
|
||||
- **Policy Version:** `mcp-ha-rolling-restart/v1`
|
||||
- **Related:**
|
||||
- Parent: [#655](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/655) — Governed MCP restart coordination and zero-disruption recovery
|
||||
- Governance Policy: [#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656) / `docs/architecture/mcp-restart-governance.md`
|
||||
- Control-Plane DB Substrate: [#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613) / `docs/architecture/control-plane-db-substrate.md`
|
||||
- Runtime Policy: [#615](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/615) / `docs/architecture/mcp-stable-control-runtime-policy-adr.md`
|
||||
- Product Vision: [#652](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/652) (Phase 5 Maturity)
|
||||
- Delivery Roadmap: [#653](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/653)
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Problem Statement
|
||||
|
||||
The Gitea MCP server operates as the authoritative **control plane** for managing issues, Pull Requests, code mutations, formal reviews, and workflow reconciliations. Under single-process governance ([#656](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/656)), process restarts are strictly controlled using pre-flight checks, drain phases, and operator approvals.
|
||||
|
||||
However, a single-instance control plane inherently presents fundamental constraints:
|
||||
|
||||
1. **Downtime during updates:** Even a perfectly executed single-process drain requires a window where incoming client requests must be paused or rejected while the server binary or python environment reloads.
|
||||
2. **Single point of failure:** Infrastructure issues, process crashes, or unhandled host-level terminations immediately disconnect active LLM sessions and leave transient workflows incomplete.
|
||||
3. **Multi-agent concurrency bottlenecks:** High volumes of concurrent multi-LLM tasks put all lock management, lease allocation, and Gitea API interactions through a single process event loop.
|
||||
|
||||
To achieve true zero-disruption operation and seamless rolling deployments without stopping active work, the system requires a high-availability (HA), multi-instance MCP architecture.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural Principles & Non-Goals
|
||||
|
||||
### 2.1 Core Architectural Principles
|
||||
* **Gitea as Canonical Work SoT:** Gitea remains the ultimate System of Record (SoT) for issue states, pull requests, labels, and audit comments. The MCP control plane does not duplicate domain entities.
|
||||
* **Control-Plane DB as Multi-Instance State Substrate:** The control-plane SQLite/durable database ([#613](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/613)) acts as the single source of truth for workflow leases, session tokens, assignment records, and lock fences across all MCP nodes.
|
||||
* **Stateless Worker Nodes:** MCP role server processes (`gitea-author`, `gitea-reviewer`, `gitea-merger`, `gitea-reconciler`, `gitea-controller`) maintain no unique in-memory state; any node can handle any request given a valid session resume token.
|
||||
* **Fail-Closed Split-Brain Defense:** In any network partition or quorum loss scenario, nodes must fail closed rather than risk double-mutations or conflicting Gitea states.
|
||||
|
||||
### 2.2 Non-Goals
|
||||
* **Replacing Gitea:** We do not replace Gitea issue/PR tracking with an independent database.
|
||||
* **Immediate Multi-Node Cluster Execution in v1:** This ADR defines the target architecture and phased roadmap; immediate implementation occurs incrementally post-[#655] v1.
|
||||
|
||||
---
|
||||
|
||||
## 3. High-Availability & Rolling-Restart Architecture
|
||||
|
||||
### 3.1 Architecture Overview
|
||||
|
||||
```
|
||||
+----------------------------+
|
||||
| LLM Clients / IDE Sessions |
|
||||
+--------------+-------------+
|
||||
|
|
||||
v
|
||||
+----------------------------+
|
||||
| HA Proxy / Router |
|
||||
| (Health-based & Affinity) |
|
||||
+------+--------------+------+
|
||||
| |
|
||||
+--------------+ +--------------+
|
||||
v v
|
||||
+--------------------+ +--------------------+
|
||||
| MCP Instance Node A| | MCP Instance Node B|
|
||||
| (Version N) | | (Version N+1) |
|
||||
+---------+----------+ +---------+----------+
|
||||
| |
|
||||
+----------------------+----------------------+
|
||||
|
|
||||
v
|
||||
+----------------------------+
|
||||
| Control-Plane DB Substrate|
|
||||
| (Shared Lease & Locks) |
|
||||
+--------------+-------------+
|
||||
|
|
||||
v
|
||||
+----------------------------+
|
||||
| Gitea API |
|
||||
+----------------------------+
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Key System Components
|
||||
|
||||
#### A. Multiple MCP Instance Cohorts
|
||||
* The control plane runs across $N \ge 2$ redundant process nodes.
|
||||
* Dual-namespace deployment allows running the old version (Node A) alongside a updated version (Node B) during rolling upgrades.
|
||||
|
||||
#### B. Shared Durable Session Storage & Resume Tokens
|
||||
* Session context, preflight verification proofs, and capability resolution states are stored in the shared control-plane database.
|
||||
* Client requests carry an explicit `session_id` and `resume_token`. If an MCP instance restarts or a request routes to a different instance, the target node validates the token against the database without requiring full session re-initialization.
|
||||
|
||||
#### C. Shared Lease Authority & Fencing Counters
|
||||
* Workflow leases (`gitea_allocate_next_work`, `gitea_adopt_workflow_lease`) use monotonic fencing tokens (`lease_generation_id`).
|
||||
* When Node B acquires or renews a lease, it increments the generation counter. Any delayed or out-of-order write attempt from Node A using an older generation token is rejected by database constraints.
|
||||
|
||||
#### D. Leader Election & Coordinated Drain
|
||||
* Node clusters elect a primary coordinator node for administrative background tasks (such as stale lease cleanup or incident Watchdogs).
|
||||
* During a rolling deployment:
|
||||
1. Node B (new version) is launched and registers as healthy.
|
||||
2. Router directs new session creations to Node B.
|
||||
3. Node A enters `MAINTENANCE_DRAIN` status ([#659](https://gitea.prgs.cc/Scaled-Tech-Consulting/Gitea-Tools/issues/659)), completing in-flight mutations while refusing new tasks.
|
||||
4. Once all active sessions migrate or complete, Node A shuts down cleanly.
|
||||
|
||||
#### E. Idempotent Mutations & Failover Safety
|
||||
* All state-changing tool executions (PR creation, review submission, merge operations, label changes) carry a deterministic `idempotency_key`.
|
||||
* If a network connection flaps or a node fails mid-mutation, the re-issued request with the same `idempotency_key` is recognized by the control-plane substrate, returning the existing recorded result without repeating side effects on Gitea.
|
||||
|
||||
#### F. Schema Version Compatibility
|
||||
* Database migrations follow non-breaking additive patterns.
|
||||
* During rolling upgrades where Node A (Version $N$) and Node B (Version $N+1$) run concurrently, both versions operate against the shared schema without structural conflicts.
|
||||
|
||||
---
|
||||
|
||||
## 4. Split-Brain & Failure Behavior
|
||||
|
||||
### 4.1 Split-Brain Risk Scenarios & Mitigation
|
||||
|
||||
| Scenario | Risk | Mitigation Strategy |
|
||||
|---|---|---|
|
||||
| **Network Partition between Nodes** | Both Node A and Node B attempt to process operations for the same issue/PR. | **Generation Fencing:** Lease renewal requires updating the DB generation counter. The node isolated from the DB fails closed immediately. |
|
||||
| **Stale Node Recovery** | Node A recovers after a long pause and executes a queued mutation. | **Lease Expiry & TTL Fencing:** Transactions verify that `expires_at > NOW()` within the atomic SQLite transaction boundaries. |
|
||||
| **Database Connection Loss** | Node loses access to shared control-plane DB substrate. | **Strict Fail-Closed:** The node immediately marks all task capabilities as `blocked` and rejects mutation tools until DB connectivity is re-established. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Phased Implementation Milestones
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
M1[Milestone 1: Shared Control-Plane DB Schema & Resume Tokens] --> M2[Milestone 2: Idempotent Mutation Layer]
|
||||
M2 --> M3[Milestone 3: Health Routing & Standby Failover]
|
||||
M3 --> M4[Milestone 4: Active-Active Rolling Deployment & Auto-Drain]
|
||||
```
|
||||
|
||||
### Milestone 1: Shared Control-Plane DB Schema & Resume Tokens (Post-#655)
|
||||
* Extend [#613] Control-Plane DB schema to store multi-instance node heartbeat records and session resume tokens.
|
||||
* Enable session lookup across instances via `session_id`.
|
||||
|
||||
### Milestone 2: Idempotent Mutation Layer & Lease Fencing
|
||||
* Add mandatory `idempotency_key` tracking to all Gitea mutation tools.
|
||||
* Implement monotonic lease fencing counters in `gitea_allocate_next_work` and `gitea_adopt_workflow_lease`.
|
||||
|
||||
### Milestone 3: Health-Based Routing & Active-Passive Standby
|
||||
* Introduce lightweight proxy/router capable of checking node health endpoints.
|
||||
* Implement active-standby failover where standby node automatically assumes work if active node fails health checks.
|
||||
|
||||
### Milestone 4: Active-Active Horizontal Deployment & Rolling Upgrade Automation
|
||||
* Enable true active-active multi-instance execution.
|
||||
* Integrate automated zero-downtime rolling upgrades coordinated with `gitea_request_mcp_restart` maintenance drain.
|
||||
|
||||
---
|
||||
|
||||
## 6. Observability & Audit Requirements
|
||||
|
||||
High-availability control plane operations must expose clear telemetry and audit trails:
|
||||
|
||||
* **Node Registry Telemetry:** Active nodes, version numbers, uptime, and heartbeat timestamps reported via `gitea_get_runtime_context`.
|
||||
* **Lease Fencing Metrics:** Tracking lease acquire latency, fence rejection counts, and lease handoff durations.
|
||||
* **Failover & Re-route Audit Logs:** Durable logging of session migrations between nodes, drain initiation, and process retirement events.
|
||||
|
||||
---
|
||||
|
||||
## 7. Tradeoffs & Accepted Risks
|
||||
|
||||
* **Increased Architectural Complexity:** Moving from a single process to a multi-instance control plane requires robust DB locking, proxy routing, and migration governance.
|
||||
* **Database Dependency:** The control-plane database substrate becomes a critical shared dependency for multi-node deployments. High availability for the underlying SQLite file system / DB must be guaranteed.
|
||||
@@ -134,19 +134,6 @@ tool argument expresses caller intent and cannot be self-asserted by a worker
|
||||
session. `break_glass_requested` and `break_glass_authorized` are both reported,
|
||||
so a bypass is never silent.
|
||||
|
||||
### Break-glass Restart Workflow (`gitea_break_glass_restart`, #664)
|
||||
|
||||
The dedicated MCP tool `gitea_break_glass_restart` provides the privileged emergency break-glass restart workflow when graceful drain cannot complete:
|
||||
|
||||
- **Role Authorization (#664 AC1)**: Ordinary LLM worker roles (`author`, `reviewer`, `merger`, `reconciler`) are denied fail-closed. Privileged `controller` role or explicit `GITEA_BREAKGLASS_RESTART_AUTHORIZATION` is required.
|
||||
- **Required Parameters (#664 AC2)**:
|
||||
- `reason`: Mandatory non-empty string (min 10 characters).
|
||||
- `confirmation`: Must equal exactly `"I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"`.
|
||||
- `impact_ack`: Must be `True`.
|
||||
- **Automatic Incident Creation (#664 AC3)**: Creates a Gitea incident issue (`[INCIDENT] Break-glass MCP restart invoked by ...`) detailing the reason, timestamp, disrupted sessions, and linking `#652 #653 #655 #630 #658 #662 #664`.
|
||||
- **Immutable Audit Entry**: Records an immutable audit log entry under `event="break_glass_mcp_restart"`.
|
||||
- **Mandatory Reconciliation (#664 AC4)**: Sets `reconciliation_required=True` requiring post-restart reconciliation via `gitea_reconcile_after_restart` (#662).
|
||||
|
||||
### Fail closed on apply
|
||||
|
||||
A missing, malformed, expired, unclean, tampered, or fingerprint-stale drain
|
||||
@@ -163,4 +150,3 @@ profiles are operational metadata only.
|
||||
|
||||
A representative dry-run report is in
|
||||
[`mcp-restart-impact-sample.json`](./mcp-restart-impact-sample.json).
|
||||
|
||||
|
||||
@@ -22812,205 +22812,6 @@ def gitea_request_mcp_restart(
|
||||
return payload
|
||||
|
||||
|
||||
BREAK_GLASS_CONFIRMATION_PHRASE = "I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION"
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def gitea_break_glass_restart(
|
||||
reason: str,
|
||||
confirmation: str,
|
||||
impact_ack: bool = False,
|
||||
restart_class: str = "full_mcp_restart",
|
||||
create_incident_issue: bool = True,
|
||||
dry_run: bool = False,
|
||||
remote: str = "dadeschools",
|
||||
host: str | None = None,
|
||||
org: str | None = None,
|
||||
repo: str | None = None,
|
||||
worktree_path: str | None = None,
|
||||
) -> dict:
|
||||
"""Privileged emergency break-glass MCP restart workflow (#664).
|
||||
|
||||
Break-glass restart permits emergency recovery when graceful drain cannot
|
||||
complete. It requires:
|
||||
1. Privileged caller authorization (ordinary LLM author/reviewer/merger/reconciler
|
||||
roles are denied fail-closed; controller/admin/sysadmin or GITEA_BREAKGLASS_RESTART_AUTHORIZATION
|
||||
is required).
|
||||
2. Explicit non-empty reason (minimum 10 characters).
|
||||
3. Exact confirmation string matching 'I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION'.
|
||||
4. Mandatory impact acknowledgement (impact_ack=True).
|
||||
5. Immutable audit entry recorded.
|
||||
6. Automatic incident record created on Gitea.
|
||||
7. Mandatory post-restart reconciliation requirement (#662).
|
||||
"""
|
||||
read_block = _profile_operation_gate("gitea.read")
|
||||
if read_block:
|
||||
return {
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": read_block,
|
||||
"permission_report": _permission_block_report("gitea.read"),
|
||||
"blocker_kind": "permission_denied",
|
||||
}
|
||||
|
||||
h, o, r = _resolve(remote, host, org, repo)
|
||||
profile = get_profile()
|
||||
active_role = _profile_role_kind(profile)
|
||||
break_glass_env_auth = bool(
|
||||
(os.environ.get("GITEA_BREAKGLASS_RESTART_AUTHORIZATION") or "").strip()
|
||||
)
|
||||
|
||||
# AC1: Ordinary LLM roles (author, reviewer, merger, reconciler) cannot invoke break-glass
|
||||
# unless explicit environment break-glass authorization is configured.
|
||||
if active_role in ("author", "reviewer", "merger", "reconciler") and not break_glass_env_auth:
|
||||
return {
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"active_role": active_role,
|
||||
"reasons": [
|
||||
f"ordinary LLM role '{active_role}' is forbidden from break-glass restarts (#664 AC1); "
|
||||
"privileged controller, operator, or GITEA_BREAKGLASS_RESTART_AUTHORIZATION required"
|
||||
],
|
||||
"blocker_kind": "role_authorization",
|
||||
}
|
||||
|
||||
# AC2: Required fields enforced
|
||||
clean_reason = (reason or "").strip()
|
||||
if not clean_reason or len(clean_reason) < 10:
|
||||
return {
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"reason is required and must be at least 10 characters long (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "missing_required_fields",
|
||||
}
|
||||
|
||||
clean_confirmation = (confirmation or "").strip()
|
||||
if clean_confirmation != BREAK_GLASS_CONFIRMATION_PHRASE:
|
||||
return {
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
f"confirmation string mismatch; must equal exactly '{BREAK_GLASS_CONFIRMATION_PHRASE}' (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "confirmation_mismatch",
|
||||
}
|
||||
|
||||
if not impact_ack:
|
||||
return {
|
||||
"success": False,
|
||||
"performed": False,
|
||||
"break_glass_executed": False,
|
||||
"reasons": [
|
||||
"impact_ack must be True to acknowledge disruption of in-flight sessions (#664 AC2)"
|
||||
],
|
||||
"blocker_kind": "impact_ack_required",
|
||||
}
|
||||
|
||||
# Evaluate impact / disrupted sessions
|
||||
impact_result = gitea_request_mcp_restart(
|
||||
remote=remote,
|
||||
host=host,
|
||||
org=org,
|
||||
repo=repo,
|
||||
dry_run=True,
|
||||
restart_class=restart_class,
|
||||
request_break_glass=True,
|
||||
)
|
||||
disrupted_sessions = list(impact_result.get("affected_sessions") or [])
|
||||
disrupted_count = len(disrupted_sessions)
|
||||
|
||||
identity = _authenticated_username(h) or profile.get("username") or "unknown"
|
||||
now_iso = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
audit_payload = {
|
||||
"event": "break_glass_mcp_restart",
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"timestamp": now_iso,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"restart_class": restart_class,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": [s.get("session_id") if isinstance(s, dict) else str(s) for s in disrupted_sessions],
|
||||
"dry_run": dry_run,
|
||||
"remote": remote,
|
||||
"org": o,
|
||||
"repo": r,
|
||||
}
|
||||
|
||||
# Save immutable audit record
|
||||
saved_audit = mcp_session_state.save_state(
|
||||
kind="break_glass_audit",
|
||||
payload=audit_payload,
|
||||
remote=remote,
|
||||
org=o,
|
||||
repo=r,
|
||||
profile_identity=profile.get("profile_name", "unknown"),
|
||||
)
|
||||
|
||||
incident_issue_result = None
|
||||
if create_incident_issue and not dry_run:
|
||||
issue_title = f"[INCIDENT] Break-glass MCP restart invoked by {identity}"
|
||||
issue_body = (
|
||||
f"## Break-glass MCP restart incident report (#664)\n\n"
|
||||
f"- **Invoked by**: `{identity}` (role: `{active_role}`)\n"
|
||||
f"- **Timestamp**: `{now_iso}`\n"
|
||||
f"- **Reason**: {clean_reason}\n"
|
||||
f"- **Confirmation**: `{clean_confirmation}`\n"
|
||||
f"- **Disrupted Sessions Count**: `{disrupted_count}`\n\n"
|
||||
f"### Mandatory Post-Restart Reconciliation (#662)\n"
|
||||
f"Post-restart reconciliation must be executed via `gitea_reconcile_after_restart` "
|
||||
f"to clean up orphaned leases, inspect worktree integrity, and handle disrupted work.\n\n"
|
||||
f"### Cross-references\n"
|
||||
f"Ref #652 #653 #655 #630 #658 #662 #664\n"
|
||||
)
|
||||
try:
|
||||
incident_issue_result = api_request(
|
||||
"POST",
|
||||
f"{repo_api_url(h, o, r)}/issues",
|
||||
_auth(h),
|
||||
{
|
||||
"title": issue_title,
|
||||
"body": issue_body,
|
||||
"labels": ["incident", "mcp-health", "break-glass"],
|
||||
},
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
incident_issue_result = {"error": _redact(str(exc))}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"performed": not dry_run,
|
||||
"dry_run": dry_run,
|
||||
"break_glass_executed": not dry_run,
|
||||
"would_execute": True,
|
||||
"actor": identity,
|
||||
"role": active_role,
|
||||
"restart_class": restart_class,
|
||||
"reason": clean_reason,
|
||||
"confirmation": clean_confirmation,
|
||||
"disrupted_sessions_count": disrupted_count,
|
||||
"disrupted_sessions": disrupted_sessions,
|
||||
"audit_record": audit_payload,
|
||||
"saved_audit": dict(saved_audit or audit_payload),
|
||||
"incident_issue": incident_issue_result,
|
||||
"reconciliation_required": True,
|
||||
"reconciliation_tool": "gitea_reconcile_after_restart",
|
||||
"follow_up_issue_required": True,
|
||||
"cross_references": ["#652", "#653", "#655", "#630", "#658", "#662", "#664"],
|
||||
"reasons": [
|
||||
"break-glass restart dry-run evaluated successfully" if dry_run
|
||||
else "break-glass restart executed with incident creation and mandatory reconciliation"
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# --- #662 post-restart reconciliation ---------------------------------------
|
||||
|
||||
_POST_RESTART_LAST_PROOF: dict | None = None
|
||||
|
||||
@@ -538,15 +538,6 @@ TASK_CAPABILITY_MAP: dict[str, dict[str, str]] = {
|
||||
"permission": "runtime.record_analytics_usage",
|
||||
"role": "author",
|
||||
},
|
||||
# #664: emergency break-glass MCP restart workflow (privileged controller role).
|
||||
"break_glass_restart": {
|
||||
"permission": "gitea.read",
|
||||
"role": "controller",
|
||||
},
|
||||
"gitea_break_glass_restart": {
|
||||
"permission": "gitea.read",
|
||||
"role": "controller",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,190 +0,0 @@
|
||||
"""Tests for emergency break-glass MCP restart workflow (#664)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import gitea_mcp_server
|
||||
|
||||
|
||||
class TestBreakGlassRestart(unittest.TestCase):
|
||||
"""Test suite for gitea_break_glass_restart tool and guardrails (#664)."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self.env_patcher = patch.dict(os.environ, {}, clear=False)
|
||||
self.env_patcher.start()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.env_patcher.stop()
|
||||
|
||||
def test_ordinary_role_denied_fail_closed(self) -> None:
|
||||
"""AC1: Ordinary LLM roles (author, reviewer, merger, reconciler) cannot invoke break-glass."""
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-author", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="author"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to deadlocked worker daemon processes",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertEqual(res["blocker_kind"], "role_authorization")
|
||||
self.assertIn("ordinary LLM role 'author' is forbidden", res["reasons"][0])
|
||||
|
||||
def test_short_reason_denied(self) -> None:
|
||||
"""AC2: Reason is required and must be at least 10 characters long."""
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Too short",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "missing_required_fields")
|
||||
self.assertIn("at least 10 characters", res["reasons"][0])
|
||||
|
||||
def test_confirmation_mismatch_denied(self) -> None:
|
||||
"""AC2: Confirmation phrase must match exact required string."""
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to stuck daemon processes",
|
||||
confirmation="wrong_confirmation_phrase",
|
||||
impact_ack=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "confirmation_mismatch")
|
||||
self.assertIn("confirmation string mismatch", res["reasons"][0])
|
||||
|
||||
def test_impact_ack_required_denied(self) -> None:
|
||||
"""AC2: impact_ack=True is mandatory."""
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to stuck daemon processes",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=False,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertFalse(res["success"])
|
||||
self.assertEqual(res["blocker_kind"], "impact_ack_required")
|
||||
|
||||
def test_dry_run_evaluation(self) -> None:
|
||||
"""AC5: Dry-run evaluation returns preview without live execution or incident creation."""
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": [{"session_id": "s1"}]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency restart needed due to hung worker process cohort",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"])
|
||||
self.assertTrue(res["dry_run"])
|
||||
self.assertFalse(res["break_glass_executed"])
|
||||
self.assertTrue(res["would_execute"])
|
||||
self.assertTrue(res["reconciliation_required"])
|
||||
self.assertEqual(res["reconciliation_tool"], "gitea_reconcile_after_restart")
|
||||
self.assertIn("#664", res["cross_references"])
|
||||
|
||||
def test_privileged_execute_creates_incident_and_audit(self) -> None:
|
||||
"""AC3 & AC4: Execution creates incident issue, audit entry, and mandates post-restart reconcile."""
|
||||
mock_api_request = MagicMock(return_value={"number": 999, "title": "[INCIDENT] Break-glass"})
|
||||
mock_save_state = MagicMock(return_value={"saved": True})
|
||||
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-controller", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="controller"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_auth", return_value={"Authorization": "token test"}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": [{"session_id": "s1"}]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="sysadmin"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "api_request", mock_api_request
|
||||
), patch("mcp_session_state.save_state", mock_save_state):
|
||||
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Emergency break-glass restart due to unrecoverable transport deadlock",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=False,
|
||||
create_incident_issue=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"])
|
||||
self.assertFalse(res["dry_run"])
|
||||
self.assertTrue(res["break_glass_executed"])
|
||||
self.assertTrue(res["reconciliation_required"])
|
||||
self.assertEqual(res["reconciliation_tool"], "gitea_reconcile_after_restart")
|
||||
self.assertEqual(res["incident_issue"]["number"], 999)
|
||||
mock_save_state.assert_called_once()
|
||||
mock_api_request.assert_called_once()
|
||||
|
||||
def test_env_authorization_override_for_worker_role(self) -> None:
|
||||
"""Environment break-glass authorization enables privileged break-glass for configured sessions."""
|
||||
os.environ["GITEA_BREAKGLASS_RESTART_AUTHORIZATION"] = "authorized-token"
|
||||
with patch.object(
|
||||
gitea_mcp_server, "get_profile", return_value={"profile_name": "prgs-author", "allowed_operations": ["gitea.read"]}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_role_kind", return_value="author"
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_profile_operation_gate", return_value=None
|
||||
), patch.object(
|
||||
gitea_mcp_server, "gitea_request_mcp_restart", return_value={"affected_sessions": []}
|
||||
), patch.object(
|
||||
gitea_mcp_server, "_authenticated_username", return_value="jcwalker3"
|
||||
):
|
||||
res = gitea_mcp_server.gitea_break_glass_restart(
|
||||
reason="Authorized emergency break-glass restart test",
|
||||
confirmation="I_ACKNOWLEDGE_BREAK_GLASS_MCP_RESTART_DISRUPTION",
|
||||
impact_ack=True,
|
||||
dry_run=True,
|
||||
remote="prgs",
|
||||
)
|
||||
self.assertTrue(res["success"])
|
||||
self.assertTrue(res["dry_run"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user