incident: repository-wide regression in master test suite #673

New Issue

2026-07-10T17:05:19-05:00

jcwalker3 commented

2026-07-10 17:05:19 -05:00

Triage and Diagnosis

The master branch currently experiences a total of 71 test failures. These have been triaged and clustered into four distinct causal families:

Family 1: `#558` Sanctioned Runtime / Authentication Regressions

Symptoms: UnsanctionedRuntimeError: Unsanctioned runtime blocked get_auth_header (#558) in test_config.py and test_credentials.py.
Root Cause: The safety guards in mcp_daemon_guard.py check PYTEST_CURRENT_TEST in os.environ to detect if the code is running under pytest. However, several tests clear or isolate the environment using patch.dict(clear=True) or os.environ.clear(), which removes this indicator and triggers false-positive blocks.
Responsible PR/Commit: be592d6 (feat: add root checkout guard for contaminated control checkout) / #475.
Remediation: Fallback to checking "pytest" in sys.modules in mcp_daemon_guard.py.

Family 2: Path Resolution & Host Workspace Leaks / `#475` & `#510`

Symptoms:
- AssertionError: '/Users/jasonwalker/Development/Gitea-Tools' != '/Users/jasonwalker' in guard alignment tests.
- RuntimeError: Root checkout guard (#475): control checkout has tracked local edits... in test_mcp_server.py.
Root Cause:
1. Six test files hardcode parents[3] to resolve the control repository root, which assumes the tests are run from a branches worktree. When run from the root checkout directory, this resolves to the parent home directory.
2. The test test_verify_preflight_blocks_control_checkout_with_test_porcelain fails when the host machine's active profile is merger/reconciler because it does not mock the active profile.
3. Local edits in the root checkout (such as the stderr redirect edit in mcp_server.py) trigger _enforce_root_checkout_guard across dozens of tests in test_mcp_server.py since the tests query the host's actual git status.
Responsible Commits: be592d6 / #475, bd22135 / #560.
Remediation:
- Dynamically resolve parent directories in tests based on whether "branches" is in the path parts.
- Mock get_profile to return an author role in test_author_mutation_worktree.py.
- Globally patch _enforce_root_checkout_guard and _enforce_branches_only_author_mutation in setUpModule for test_mcp_server.py.

Family 3: Tracker Cleanup Mocks Escaping to Gitea API

Symptoms: AssertionError: 'error fetching repo labels: HTTP 401...' != 'released' in TestTrackerHygieneCleanup tests.
Root Cause: Label pagination introduced in #627 calls api_get_all from gitea_auth.py, which delegates to gitea_auth.api_request. Since the tests only patch mcp_server.api_request, the paginated label requests escape the mock, hit the real Gitea API, and return HTTP 401.
Responsible PR/Commit: 5ab5fe8 (PR #629/fix: paginate labels) / #627.
Remediation: Map gitea_auth.api_request to mcp_server.api_request in setUpModule of tests/test_mcp_server.py so that mock side-effects apply to both references.

Family 4: Web UI NameError on `/worktrees`

Symptoms: NameError: name 'REVIEW_WORKTREE_RE' is not defined when loading /worktrees.
Root Cause: In webui/worktree_scanner.py line 150, the code references REVIEW_WORKTREE_RE.search(rel_path) but the regular expression is defined with a leading underscore _REVIEW_WORKTREE_RE on line 18.
Responsible PR/Commit: bd22135 (fix: bind issue comments to explicit worktree path) / #560.
Remediation: Change REVIEW_WORKTREE_RE to _REVIEW_WORKTREE_RE in webui/worktree_scanner.py line 150.

Strategy and Decision

Coordinated Repair: One single coordinated repair PR is safer and cleaner than separate PRs. Because the uncommitted edits on disk (Family 2) affect tests in test_mcp_server.py (which contains Family 3), separating them would cause intermediate states to fail CI verification. A single coordinated branch resolves the path issues, mock escapes, and NameErrors atomically.
Master Freeze: Merges are frozen until this coordinated repair branch is verified green.
Merge Gate: Document a hard merge gate requiring full-suite CI on current head, green baseline, and non-optional branch protection.

Next Actions & Handoff

STATE: ready-for-author
WHO_IS_NEXT: author
NEXT_ACTION: Fix NameErrors, pytest environments, path resolutions, and mock escapes in the isolated repair worktree.
NEXT_PROMPT: Author implements the repair branch, runs the full pytest suite, creates a PR, and verifies the merge gate.

## Triage and Diagnosis The master branch currently experiences a total of 71 test failures. These have been triaged and clustered into four distinct causal families: ### Family 1: `#558` Sanctioned Runtime / Authentication Regressions * **Symptoms**: `UnsanctionedRuntimeError: Unsanctioned runtime blocked get_auth_header (#558)` in `test_config.py` and `test_credentials.py`. * **Root Cause**: The safety guards in `mcp_daemon_guard.py` check `PYTEST_CURRENT_TEST` in `os.environ` to detect if the code is running under pytest. However, several tests clear or isolate the environment using `patch.dict(clear=True)` or `os.environ.clear()`, which removes this indicator and triggers false-positive blocks. * **Responsible PR/Commit**: `be592d6` (feat: add root checkout guard for contaminated control checkout) / `#475`. * **Remediation**: Fallback to checking `"pytest" in sys.modules` in `mcp_daemon_guard.py`. ### Family 2: Path Resolution & Host Workspace Leaks / `#475` & `#510` * **Symptoms**: - `AssertionError: '/Users/jasonwalker/Development/Gitea-Tools' != '/Users/jasonwalker'` in guard alignment tests. - `RuntimeError: Root checkout guard (#475): control checkout has tracked local edits...` in `test_mcp_server.py`. * **Root Cause**: 1. Six test files hardcode `parents[3]` to resolve the control repository root, which assumes the tests are run from a branches worktree. When run from the root checkout directory, this resolves to the parent home directory. 2. The test `test_verify_preflight_blocks_control_checkout_with_test_porcelain` fails when the host machine's active profile is merger/reconciler because it does not mock the active profile. 3. Local edits in the root checkout (such as the stderr redirect edit in `mcp_server.py`) trigger `_enforce_root_checkout_guard` across dozens of tests in `test_mcp_server.py` since the tests query the host's actual git status. * **Responsible Commits**: `be592d6` / `#475`, `bd22135` / `#560`. * **Remediation**: - Dynamically resolve parent directories in tests based on whether `"branches"` is in the path parts. - Mock `get_profile` to return an author role in `test_author_mutation_worktree.py`. - Globally patch `_enforce_root_checkout_guard` and `_enforce_branches_only_author_mutation` in `setUpModule` for `test_mcp_server.py`. ### Family 3: Tracker Cleanup Mocks Escaping to Gitea API * **Symptoms**: `AssertionError: 'error fetching repo labels: HTTP 401...' != 'released'` in `TestTrackerHygieneCleanup` tests. * **Root Cause**: Label pagination introduced in `#627` calls `api_get_all` from `gitea_auth.py`, which delegates to `gitea_auth.api_request`. Since the tests only patch `mcp_server.api_request`, the paginated label requests escape the mock, hit the real Gitea API, and return HTTP 401. * **Responsible PR/Commit**: `5ab5fe8` (PR #629/fix: paginate labels) / `#627`. * **Remediation**: Map `gitea_auth.api_request` to `mcp_server.api_request` in `setUpModule` of `tests/test_mcp_server.py` so that mock side-effects apply to both references. ### Family 4: Web UI NameError on `/worktrees` * **Symptoms**: `NameError: name 'REVIEW_WORKTREE_RE' is not defined` when loading `/worktrees`. * **Root Cause**: In `webui/worktree_scanner.py` line 150, the code references `REVIEW_WORKTREE_RE.search(rel_path)` but the regular expression is defined with a leading underscore `_REVIEW_WORKTREE_RE` on line 18. * **Responsible PR/Commit**: `bd22135` (fix: bind issue comments to explicit worktree path) / `#560`. * **Remediation**: Change `REVIEW_WORKTREE_RE` to `_REVIEW_WORKTREE_RE` in `webui/worktree_scanner.py` line 150. --- ## Strategy and Decision * **Coordinated Repair**: One single coordinated repair PR is safer and cleaner than separate PRs. Because the uncommitted edits on disk (Family 2) affect tests in `test_mcp_server.py` (which contains Family 3), separating them would cause intermediate states to fail CI verification. A single coordinated branch resolves the path issues, mock escapes, and NameErrors atomically. * **Master Freeze**: Merges are frozen until this coordinated repair branch is verified green. * **Merge Gate**: Document a hard merge gate requiring full-suite CI on current head, green baseline, and non-optional branch protection. --- ## Next Actions & Handoff ```text STATE: ready-for-author WHO_IS_NEXT: author NEXT_ACTION: Fix NameErrors, pytest environments, path resolutions, and mock escapes in the isolated repair worktree. NEXT_PROMPT: Author implements the repair branch, runs the full pytest suite, creates a PR, and verifies the merge gate. ```

jcwalker3 commented

2026-07-10 17:09:25 -05:00

Remediation and Verification Complete

The coordinated repair branch fix/issue-673-remediate-regressions has been successfully implemented and validated.

Verification Evidence

Test Suite Status: 100% green.
Results: 2545 passed, 6 skipped (0 failures).
Execution Time: 35.10s.

Branch and Commits

Repair Branch: fix/issue-673-remediate-regressions
Commit: a7aac0d ("fix(test): remediate repository-wide regressions (Closes #673)")

Handoff and Next Actions

The repository is ready for review and merge.

STATE: ready-for-merger
WHO_IS_NEXT: merger
NEXT_ACTION: Review PR and merge the coordinated repair branch `fix/issue-673-remediate-regressions` into master.
NEXT_PROMPT: Merger checks out fix/issue-673-remediate-regressions, verifies the 100% green test suite status, and merges it into master.

## Remediation and Verification Complete The coordinated repair branch `fix/issue-673-remediate-regressions` has been successfully implemented and validated. ### Verification Evidence * **Test Suite Status**: 100% green. * **Results**: `2545 passed`, `6 skipped` (0 failures). * **Execution Time**: 35.10s. ### Branch and Commits * **Repair Branch**: `fix/issue-673-remediate-regressions` * **Commit**: `a7aac0d` ("fix(test): remediate repository-wide regressions (Closes #673)") ### Handoff and Next Actions The repository is ready for review and merge. ```text STATE: ready-for-merger WHO_IS_NEXT: merger NEXT_ACTION: Review PR and merge the coordinated repair branch `fix/issue-673-remediate-regressions` into master. NEXT_PROMPT: Merger checks out fix/issue-673-remediate-regressions, verifies the 100% green test suite status, and merges it into master. ```

jcwalker3 referenced this issue from a commit

2026-07-11 18:10:25 -05:00

fix(test): remediate repository-wide regressions (Closes #673)

jcwalker3 referenced a pull request that will close this issue

2026-07-11 18:13:47 -05:00

fix(test): remediate repository-wide regressions (Closes #673) #674

jcwalker3 added the status:pr-open label 2026-07-11 18:13:48 -05:00

sysadmin referenced this issue

2026-07-11 18:15:21 -05:00

fix(test): remediate repository-wide regressions (Closes #673) #674

sysadmin closed this issue

2026-07-11 18:16:05 -05:00