runtime_recovery_guard segment splitter misses & and subshell forms, so a real daemon kill classifies as clean
#787
Closed
opened 2026-07-21 18:07:57 -05:00 by jcwalker3
·
2 comments
No Branch/Tag Specified
Labels
Clear labels
allocator
anti-stomp
architecture
bug
chore
codex
concurrency
contamination
control-plane
dashboard
database
design
documentation
enhancement
gitea
glitchtip
important
incident
incident-bridge
integration
jenkins
labels
leases
mcp
mcp-health
mcp-menu
multi-project
mutating
nice-to-have
observability
portability
preflight
protected-branch
queue
read-only
reconnect
recovery
refactor
release
reliability
resumable-review
reviewer
roadmap
safety
security
self-hosted
sentry
stale-runtime
status:blocked
status:in-progress
status:pr-open
status:ready
terminal-lock
testing
tracker
type:bug
type:feature
type:feature
type:guardrail
visibility
workflow
workflow-hardening
workflow-hardening
Controller-owned work allocator
Prevent concurrent LLM session stomping
Architecture / structural design
OpenAI Codex client / workflow session surface
Concurrent session safety
Workflow or session contamination incident
MCP control-plane coordination and allocation authority
MCP operational dashboard/queue view
Internal coordination storage (SQLite/Postgres)
Design / investigation, no implementation
Docs / runbooks
New feature or improvement
Gitea MCP workflow
GlitchTip integration
Operational or process incident requiring durable audit trail
Sentry-to-Gitea incident bridging
Integration testing
Jenkins integration
Label taxonomy management
Lease adopt/release/expire lifecycle
MCP server / tooling
MCP namespace and runtime health
MCP menu surface
Work spanning multiple monitoring projects or Gitea repos
Mutating action; requires gating
Observability, metrics, traces, error reporting
Cross-platform / portability
Shared preflight gates before mutation
Protected branch / stable-branch policy concern
Work queue visibility and allocation
Read-only, no mutation
MCP client reconnect/reload recovery path
Recovery paths for stale/foreign leases
Code refactor / restructure
Release / versioning
Reliability / failure handling
Persist and resume prepared review verdicts across sessions
Reviewer workflow tooling
Roadmap / umbrella issue
Safety rails and fail-closed mutation guards
Security / trust boundary
Self-hosted infrastructure integration
Sentry error monitoring integration
Stale backend daemon / runtime-vs-master parity failures
Issue is blocked
Issue is being worked on
Issue has an open pull request
Issue is ready for work
Terminal review lock (#332) path
Tests / test coverage
Issue tracker hygiene / meta
Bug or defect
Feature or enhancement
Feature or enhancement
Safety gate or guardrail
Workflow state visibility for LLMs/operators
Cross-tool workflow
LLM workflow coordination hardening
LLM workflow coordination hardening
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Scaled-Tech-Consulting/Gitea-Tools#787
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Follow-up defect against #630, landed by PR #786 at merge commit
35e94e107c.Problem
runtime_recovery_guard._SEGMENT_SPLIT_REsplits a compound command line so each simple command can be classified independently:The comment above it states that it splits "on shell separators", but the background-job separator
&is not in the alternation, and a subshell wrapper( ... )is never stripped before tokenising._analyse_kill_segmentinspects only the first command token of a segment, so in both forms the kill verb is never reached and the command is reported as containing no process kill at all.Observed behavior
Measured directly against the merged implementation:
All three kill the daemons. The first is an ordinary command line, not an evasion attempt.
Impact
_enforce_runtime_recovery_contamination_gate; when no marker is written, review, merge, close and completion mutations stay reachable after a real kill. The guard silently reports a clean session.gitea_record_daemon_process_kill_attemptreturnscontaminated: falseand writes nothing, so there is also no forensic record for a later reconciler audit.Scope
_SEGMENT_SPLIT_REwhile keeping the two-character logical form matched first, for example(?:\|\||&&|[|;&\n]).(and a trailing)from a segment before tokenising, so a subshell-wrapped kill reaches the classifier.tests/test_issue_630_runtime_recovery_guard.pyproving both forms classify as contamination. The file currently contains no case with a background separator or a subshell.pkill -u mcpuser -f myapp,ps aux | grep mcp_server, a barekillof an unknown pid, and a commit message quoting the kill string.Non-goals
1ec4672fad.Acceptance criteria
sleep 1 & pkill -f mcp_server.pyclassifies as contamination with reason classmanual_daemon_kill.(pkill -f mcp_server.py)classifies as contamination with reason classmanual_daemon_kill.killof a pid not known to be a daemon, keep their current non-contaminating outcomes.gitea_record_daemon_process_kill_attemptwrites a durable marker for both newly detected forms.Provenance
Found during independent review of PR #786 at head
1ec4672fad. The reviewer verdict recording this finding could not be posted: a concurrent session held the reviewer lease (session90916-0215a28b2122, lease comment 13774) and approved and merged the PR before the finding could be submitted. The lease was not stolen. This issue is the sanctioned route for the finding.Related observation, not in this scope
Detection is opt-in: a marker is written only when a session calls
gitea_record_daemon_process_kill_attempt. The incident behind #630 ran a kill and then closed an issue, and such a session has no reason to call the tool.gitea_record_pre_review_commandandgitea_assess_gitea_operation_pathalready receive the same command strings and already classify them, so feeding the classifier from those two points would make detection automatic. That deserves its own issue rather than being folded in here.[THREAD STATE LEDGER] Issue #787 — author implementation published as PR #789 at head
6b58f04What is true now:
status:pr-open.fix/issue-787-kill-segment-separators; published head6b58f04d396b881d5d242eacac45bcfdfefe2d00, parent35e94e107c32cfdc4b9ab9a95cb4e94df94077d4; the author issue lock for #787 is still held by this session.tests/test_issue_630_runtime_recovery_guard.py55 passed; full suite 4181 passed with 11 failures proven pre-existing on unmodified master by a stash-and-rerun at35e94e10.What changed:
runtime_recovery_guard._SEGMENT_SPLIT_REwidened to(?:\|\||&&|[|;&\n]), adding the background-job separator while keeping the two-character logical forms first in the alternation so&&and||are consumed whole.runtime_recovery_guard._strip_subshell(), applied inside_split_segments(), removes a leading(and a trailing)(nested wrappers too) before tokenising.tests/test_issue_630_runtime_recovery_guard.pygrew from 46 to 55 cases: both forms named in the issue, four further background/subshell variants, an invariant test that&&/||still split whole, two record-tool cases proving a durable marker is written for each new form, and the two no-false-positive cases the issue named that previously had no coverage.status:pr-open.What is blocked:
6b58f04d396b881d5d242eacac45bcfdfefe2d00against base master35e94e107c32cfdc4b9ab9a95cb4e94df94077d4, which is the branch parent, so no rebase is outstanding.Who/what acts next:
6b58f04d396b881d5d242eacac45bcfdfefe2d00from the reviewer namespace, and record the verdict through the review API.Canonical Issue State
STATE:
author-implementation-published
WHO_IS_NEXT:
reviewer
NEXT_ACTION:
Fresh native review of PR #789 at head
6b58f04d39, verifying acceptance criteria 1 through 5 against the diff and the test file.NEXT_PROMPT:
WHAT_HAPPENED:
The segment splitter in
runtime_recovery_guardomitted the background-job separator&and never stripped a subshell wrapper, so_analyse_kill_segmentsaw a non-kill token in command position and reported no process kill for two command lines that do kill the daemons. Both fixes are in place, the classifier now reports contamination with reason classmanual_daemon_killfor each, and the record tool writes a durable marker for both. Direct-command behaviour is unchanged.WHY:
The missing marker propagated to
_enforce_runtime_recovery_contamination_gate. With no marker written, review, merge, close and completion mutations stayed reachable after a real daemon kill and the session was reported as clean, which defeats the #630 guard for these inputs and leaves no forensic record for a later reconciler audit.RELATED_PRS:
6b58f04d396b881d5d242eacac45bcfdfefe2d00, base master, awaiting a first review verdict.35e94e10; this issue is the follow-up defect found during its review and is the sanctioned route for that finding.BLOCKERS:
none
VALIDATION:
runtime_recovery_guard.pyat35e94e10:sleep 1 & pkill -f mcp_server.pyand(pkill -f mcp_server.py)both gave process_kill False, contamination False, reason_class None, whilepkill -f mcp_server.pygave process_kill True, contamination True, reason_class manual_daemon_kill. This matches the observed behaviour recorded in the issue.6b58f04: both forms give process_kill True, contamination True, reason_class manual_daemon_kill, ambiguous False.pkill -u mcpuser -f myappstays non-contaminating and unambiguous,ps aux | grep mcp_serverstays a non-kill,kill 31337stays ambiguous and non-contaminating, and a commit message quoting the kill string stays a non-kill.python3.14 -m pytest tests/test_issue_630_runtime_recovery_guard.py -q -sgave 55 passed in 0.82s.35e94e10; re-running the five affected modules gave the identical 11 failures (11 failed, 254 passed) — 6 intests/test_commit_payloads.py, 2 intests/test_issue_702_review_findings_f1_f6.py, 1 intests/test_mcp_server.py::TestPreflightVerification, 1 intests/test_post_merge_moot_lease.py, 1 intests/test_reconciler_supersession_close.py. The stash was then restored and the focused suite re-run at 55 passed.6b58f04carries treeb1522636eef0ed6920195ac33373349c55981283, matching the locally tested commit84526e9exactly, with parent35e94e10.LAST_UPDATED_BY:
prgs-author (jcwalker3), MCP author namespace, daemon pid 84100
Author remediation of review #497 — PR #789 at new head
aa4fe1ccIssue #787 · PR #789 · previous head
6b58f04d396b881d5d242eacac45bcfdfefe2d00· new headaa4fe1cc7bd2d6f07ea35775a5c5292d94a938e0Review #497 and disposition of each finding
Blocking review #497 (REQUEST_CHANGES, reviewer
sysadmin, recorded against head6b58f04) is undismissed but now stale against the live head, andauthor_pushed_after_request_changesreads true.&made quoted text carrying an ampersand plus a kill verb classify as a manual daemon kill. Fixed at the class level with quote-aware segmentation, not by special-casing the three named strings. New coverage asserts contamination False for each of them, including the fullsleep 1 & pkill -f mcp_server.pystring quoted inside a non-executing command, which is the AC5 case the reviewer found unprotected.35e94e1047, rejected head6b58f0455, so the rejected head added 8, not 9. The PR body now carries 47 / 55 / 64 and the corrected arithmetic._strip_subshellnow removes a trailing)only when it closes a leading(the same call stripped, sokill $(pgrep -f myapp)is no longer mangled;_is_redirectionkeeps2>&1,>&2and&>logout of separator position. No unrelated expansion beyond these two.Quote-aware parsing behaviour
_SEGMENT_SPLIT_REis replaced by a character scan._iter_active()yields only the positions where a shell metacharacter is syntactically active — outside single and double quotes, and not backslash-escaped — and_split_segments()separates only at those positions. A backslash does not escape inside single quotes, matching POSIX. An unterminated quote swallows the rest of the line exactly as the shell does, and a shell rejects such a command as a syntax error rather than running its tail, so nothing executable hides behind one.Logical-separator precedence is preserved:
&&and||are consumed whole, and;,|, newline and standalone&keep their prior behaviour._split_segments("a && b || c")is['a','b','c'],_split_segments("a & b")is['a','b'],_split_segments("a; b\nc | d")is['a','b','c','d'].True-positive reproductions at the new head
False-positive reproductions, rejected head versus new head
Unchanged at both heads:
ps aux | grep mcp_servernon-kill;pkill -u mcpuser -f myappkill, non-contaminating, unambiguous;kill 31337ambiguous, non-contaminating;kill $(pgrep -f myapp)ambiguous, non-contaminating;npm run dev & sleep 2andsystemctl restart myapp && echo oknon-kills;a 2>&1non-kill.Marker behaviour is asserted directly: a new record-tool case drives
gitea_record_daemon_process_kill_attemptwith each of the three F1 commands and assertscontaminatedFalse,markedFalse, and that no marker loads afterwards. The two pre-existing record-tool cases for the genuine background-separator and subshell forms still assertcontaminatedandmarkedtrue with a loaded marker.Test evidence
Focused suite at the new head, cwd
/Users/jasonwalker/Development/Gitea-Tools/branches/fix-issue-787-kill-segment-separators, interpreter/opt/homebrew/bin/python3.14:Focused counts re-measured per F2, each in its own detached worktree, both removed afterwards:
35e94e10branches/verify-789-base6b58f04branches/verify-789-rejectedaa4fe1ccbranches/fix-issue-787-kill-segment-separatorsNine cases are added on top of the rejected head.
Broader suite at the new head:
The same five modules at base
35e94e10inbranches/verify-789-base:The failing identifiers match test for test at both commits: 6 in
tests/test_commit_payloads.py, 2 intests/test_issue_702_review_findings_f1_f6.py, 1 intests/test_mcp_server.py::TestPreflightVerification, 1 intests/test_post_merge_moot_lease.py, 1 intests/test_reconciler_supersession_close.py. They predate this branch and do not bear on this remediation.Command-path proof
mcp_server, pid 52938, mode production, token fingerprint001cc43cc20d517e.gitea-tools/prgs-author, identityjcwalker3(user_id 1), role author.startup_head==current_head==35e94e107c32cfdc4b9ab9a95cb4e94df94077d4,stalefalse, mutation gate enforced.gitea_whoami->gitea_resolve_task_capability-> mutate order, with nothing between resolve and mutation:gitea_lock_issue(tasklock_issue),gitea_commit_files(taskcommit_files),gitea_edit_pr(taskedit_pr),gitea_create_issue_comment(taskcomment_issue).gitea_commit_fileswithlocal_pathsources;content_source_proofreportslocal_pathfor both files. No raw-git push, no direct API call, no CLI fallback.fix/issue-787-kill-segment-separators, worktree/Users/jasonwalker/Development/Gitea-Tools/branches/fix-issue-787-kill-segment-separators, ownerprgs-author, pid 52938, freshness live. It was re-acquired through the sanctionedgitea_lock_issuedead-session recovery path: prior pid 84100 was dead,recovery_modepublished_owning_pr,head_relationequal,adoption_decisionADOPT. No lock file was hand-edited, no lease was stolen, no process was killed.aa4fe1cccarries tree9b12bc5f52b57ff15ac12da2009d1a203a3643ab, identical to the tree of the locally tested commit9e89efcf, andgit diff HEAD FETCH_HEADis empty. Its parent is the rejected head6b58f04, so PR #789 was updated in place and no replacement PR exists.[THREAD STATE LEDGER] Issue #787 / PR #789 — author remediation published at head
aa4fe1ccWhat is true now:
aa4fe1cc7bd2d6f07ea35775a5c5292d94a938e0; issue #787 remains in open state; review #497 REQUEST_CHANGES is undismissed but now stale against the live head,has_blocking_change_requeststrue,approval_at_current_headfalse, and no merge event has occurred.What changed:
fix/issue-787-kill-segment-separators:runtime_recovery_guard.pyandtests/test_issue_630_runtime_recovery_guard.py.6b58f04->aa4fe1ccviagitea_commit_files; the PR #789 title and body were edited in place with the corrected F2 counts.gitea_lock_issue; no lock, lease, or reviewer artifact was altered by hand. Two temporary verification worktrees were created and removed; the reviewer'sreview-pr-789andreview-789-baselineworktrees were left untouched.What is blocked:
Who/what acts next:
aa4fe1cc7bd2d6f07ea35775a5c5292d94a938e0, confirming that the three F1 commands classify as non-contaminating, that the required true positives still classify asmanual_daemon_kill, that logical-separator precedence is intact, and that the corrected F2 counts match a fresh measurement.aa4fe1cc; do not remove the author worktree or release the #787 lock while the PR is in open state.Canonical Issue State
STATE:
author-remediation-published-awaiting-review
WHO_IS_NEXT:
reviewer
NEXT_ACTION:
A fresh independent reviewer examines PR #789 at head
aa4fe1cc7bd2d6f07ea35775a5c5292d94a938e0and records a verdict.NEXT_PROMPT:
WHAT_HAPPENED:
Blocking review #497 found that adding
&to a quote-unaware splitter created a false-positive class in which any quoted mention of the canonical kill string classified as a manual daemon kill, and that the AC5 commit-message test omitted the ampersand so it could not catch the regression. The author replaced the regex alternation with a quote-aware character scan that recognises a separator only where it is syntactically active, added nine focused cases covering the three F1 commands plus double-quoted, single-quoted and backslash-escaped ampersands, the POSIX single-quote rule, the pre-existing;and|class, the two F3 forms, and a record-tool case asserting no marker is written for a quoted mention. The F2 counts were re-measured and the PR body corrected. The remediation was published to PR #789 in place at headaa4fe1cc.WHY:
A false contamination marker fails review, merge, close and completion mutations closed and by design only a reconciler may clear it, so a false positive is an operator-visible stall rather than a warning. The regressing string is the exact example this issue is written around, which made a documentation line or commit message about #787 a trigger for it. Fixing the quote-unawareness rather than the three named strings also retires the
;and|instances of the same class that predate this issue.ISSUE:
787
HEAD_SHA:
aa4fe1cc7bRELATED_PRS:
789
REVIEW_STATUS:
request_changes at the prior head
6b58f04, now stale; no verdict exists at headaa4fe1ccMERGE_READY:
no
BLOCKERS:
No blocker classification applies on the author side. PR #789 awaits a fresh independent review verdict at head
aa4fe1cc, which only a reviewer may record.VALIDATION:
Focused suite 64 passed at head
aa4fe1cc, 55 at rejected head6b58f04, 47 at base35e94e10, each measured in its own worktree with/opt/homebrew/bin/python3.14. Broader suite at the new head: 11 failed, 4190 passed, 6 skipped, 493 subtests passed. The five affected modules at base35e94e10: 11 failed, 254 passed, with identifiers matching test for test, so those failures predate the branch. A 28-command classification probe was run against the new head and the rejected head; all required true positives are preserved and all seven quoted-text false positives are eliminated.LAST_UPDATED_BY:
prgs-author (jcwalker3), gitea-tools MCP namespace, native stdio transport, pid 52938
Controller Handoff
35e94e107c(startup_head == current_head, stale false)6b58f04d39aa4fe1cc7b9b12bc5f52, identical to locally tested commit 9e89efcf6b58f04d3935e94e107c6b58f04, undismissed, now stale against the live head6b58f04before editing, discarding only a sibling local commit with an identical tree; two temporary verification worktrees created under branches/ and removedaa4fe1cc7baa4fe1ccWHO_IS_NEXT: reviewer