Scaled-Tech-Consulting/Gitea-Tools
2
0
Fork 0
Code Issues 91 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(webui): compensate stray allocations and keep preview side-effect free

Browse Source 
Addresses both blockers from the PR #902 review (review 589) for issue #643.

B1 — an allocator-created assignment could be orphaned and reported as no
mutation.

apply_request re-previews, then re-runs the allocator with apply=True. The CAS
fingerprint hashes only {kind, number} plus exclusions, so a competing lease
taken on the requested unit inside the window leaves the fingerprint identical:
the pin passes, the selection loop skips the now-claimed unit and commits an
assignment on the *next* one, and _selection_matches then fails on egress. The
old code returned mutation_performed False with that lease still committed and
owned by a synthetic session nothing heartbeats. The window contains a second
full load_queue_snapshot(), so it is seconds wide, and foreign sessions acting
on this repo concurrently are an observed condition.

The egress mismatch now releases the assignment the allocator created before
refusing. When the release succeeds the refusal reports mutation_performed
False and a compensation record; when it fails the response carries
mutation_performed True, an explicit orphaned_assignment, and a
gitea_release_workflow_lease reclaim action, because claiming nothing changed
while a lease is live is the defect rather than a report of it.

The same state was reachable through _run_allocator's bare except Exception:
allocate_next_work only catches InvalidWorkKindError, LeaseRequiredError and
ControlPlaneError, so anything raised after assign_and_lease committed arrived
as "no result" with a durable lease. A None result on the apply path now sweeps
and releases whatever this flow's session owns. That sweep is only possible
because of the B2 fix below — the session id is now stable across the flow, so
the lease is findable.

B2 — the "read-only" preview wrote to the control-plane DB.

allocate_next_work called db.upsert_session and db.expire_stale_leases
unconditionally, before the apply branch was consulted, and default_allocator
minted a fresh webui-request-<hex> per call. Every preview therefore appended a
never-reused session row and mutated global lease state while the payload said
dry_run True / mutation_performed False, driven by an operator refreshing a
form. One apply wrote two rows and bound the lease to the second, which is why
B1's orphan had no reclaimable owner.

allocate_next_work gains a keyword-only side_effect_free flag, default False so
every existing caller is byte-for-byte unchanged. Under the flag both writes are
suppressed and expired leases are instead filtered out of the claim map in
memory, which reaches the same selection the sweep would have produced without
persisting anything; a claim whose expiry cannot be parsed is kept, since an
unreadable expiry is not evidence that work is free. side_effect_free with
apply=True fails closed rather than silently reserving. request_service mints
one session id per request flow and threads it through both the dry-run and the
apply, and the dry-run now routes through the side-effect-free path.

Coverage.

test_allocator_drift_on_apply_is_not_read_as_an_assignment asserted the defect —
it built the orphan state and then required mutation_performed to be False, which
a leak satisfies. It now requires the compensating release. Added: release-failure
surfacing a reclaim action, an assignment with no lease id, the post-commit
exception route, and session-id identity across the flow. The two areas the review
named as having zero coverage now have it: default_allocator past its two
fail-closed early returns (side_effect_free routing, session-id pass-through and
minting, scope and fingerprint propagation) and default_claims_source (scoped
read, and an unreadable substrate denying rather than reading as "nothing
claimed"). allocator_service gains side-effect-free tests against a real
temp-file DB plus the expiry-filter unit tests.

Every new guard was mutation-tested by reverting it one at a time: B1
compensation removed → 3 failures; post-commit sweep removed → 1; session id
re-minted per call → 2; side_effect_free ignored → 3; in-memory expiry filter
removed → 1.

Verification: WEBUI_TEST_OFFLINE=1 python -m pytest tests/ -q from this
branches/ worktree gives 23 failed, 5263 passed, 6 skipped, 899 subtests. The
sorted FAILED set is identical to the reviewer's clean-master baseline at
2f4dec83 (23 failed, 5190 passed) — no new, changed, or disappeared failure —
and 21 tests were added over the reviewed head's 5242. Zero conflict markers;
py_compile passes; git diff --check clean.

Refs #643, PR #902

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01V6xFqovhbArPv61j9KCGkL
This commit is contained in:
sysadmin
2026-07-25 03:32:24 -04:00
co-authored by Claude Opus 5
parent 433f66add8
commit 53ce1b1a5e
4 changed files with 839 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
webui/request_service.py
+264 -18
View File
@@ -35,6 +35,7 @@ the resulting assignment by ``correlation_id``.
from __future__ import annotations
import inspect
import uuid
from dataclasses import dataclass, field
from typing import Any, Callable, Mapping, Sequence
@@ -321,6 +322,17 @@ def _correlation_id() -> str:
return f"req-{uuid.uuid4().hex}"
def _new_session_id() -> str:
"""Mint the control-plane session id for one request flow (#643).
Minted once per flow and reused by the dry-run and the apply, so a lease the
apply creates is owned by an id the caller still holds and can release. When
each call minted its own id, an apply wrote two session rows and bound the
lease to the second — leaving it with no owner able to reclaim it.
"""
return f"webui-request-{uuid.uuid4().hex[:12]}"
def _selection_matches(
selection: Mapping[str, Any] | None, request: WorkRequest
) -> bool:
@@ -561,8 +573,15 @@ def preview_request(
claims_source: ClaimsFn | None = None,
correlation_id: str | None = None,
audit: bool = True,
session_id: str | None = None,
) -> RequestPreview:
"""Build the read-only intent preview for *request*. Never mutates."""
"""Build the read-only intent preview for *request*. Never mutates.
"Never mutates" is now literal on the default path: the allocator runs
``side_effect_free``, so no control-plane session row is written and no lease
sweep runs. *session_id* is threaded from an enclosing apply so both halves
of that flow share one control-plane identity (#643).
"""
who = principal or console_authz.ANONYMOUS
corr = correlation_id or _correlation_id()
decision = console_authz.authorize(ACTION_ID, who, for_execution=False)
@@ -573,7 +592,9 @@ def preview_request(
if decision.allowed:
# An unauthorized principal never reaches the allocator or the
# control-plane DB: a denial must not double as a queue oracle.
allocation = _run_allocator(request, allocator, apply=False)
allocation = _run_allocator(
request, allocator, apply=False, session_id=session_id
)
claims = _load_claims(request, claims_source)
checks.append(_capability_check(request))
checks.append(_lease_check(request, claims))
@@ -650,9 +671,20 @@ def apply_request(
The only path to an assignment is the allocator agreeing, on a dry-run,
that this work unit is what the requested role should take next. Every
refusal returns before any mutation is attempted.
One exception is unavoidable and is compensated rather than prevented: the
allocator can commit an assignment and only then reveal, on egress, that it
selected a different work unit than the one requested. That assignment is
released before refusing, and ``mutation_performed`` reports whether any
durable control-plane state survives this call — ``False`` once the stray
assignment is gone, ``True`` with an explicit ``orphaned_assignment`` and
reclaim action when the release did not succeed (#643).
"""
who = principal or console_authz.ANONYMOUS
corr = correlation_id or _correlation_id()
# One identity for the whole flow, so a lease the apply creates is owned by
# an id this function still holds and can release.
session_id = _new_session_id()
execution_decision = console_authz.authorize(ACTION_ID, who, for_execution=True)
authorization = execution_decision.to_dict()
@@ -664,6 +696,7 @@ def apply_request(
*,
status: int,
extra: dict[str, Any] | None = None,
mutation_performed: bool = False,
) -> dict[str, Any]:
_audit(
request,
@@ -689,7 +722,9 @@ def apply_request(
"authorization": authorization,
"assignment": None,
"correlation_id": corr,
"mutation_performed": False,
# True only when durable control-plane state survives this refusal —
# an assignment the allocator created that could not be released.
"mutation_performed": bool(mutation_performed),
"status_code": status,
}
payload.update(extra or {})
@@ -724,6 +759,7 @@ def apply_request(
claims_source=claims_source,
correlation_id=corr,
audit=False,
session_id=session_id,
)
if not preview.authorized:
outcome = (
@@ -747,21 +783,37 @@ def apply_request(
allocator,
apply=True,
expected_candidate_set_fingerprint=fingerprint,
session_id=session_id,
)
if not allocation:
# "No result" is not proof of "no write": allocate_next_work only catches
# three exception types, so anything raised after assign_and_lease
# committed lands here with a lease already durable. Release whatever
# this flow's session owns before refusing.
sweep = _sweep_session_leases(request, session_id=session_id)
extra: dict[str, Any] = {}
detail = "the allocator returned no result; nothing was assigned."
if sweep.get("released"):
extra["compensation"] = sweep
detail = (
"the allocator returned no result after creating an assignment; "
"the assignment was released and nothing remains assigned."
)
elif sweep.get("error"):
extra["compensation"] = sweep
return _refuse(
OUTCOME_WAIT,
REASON_EVIDENCE_UNAVAILABLE,
"the allocator returned no result; nothing was assigned.",
detail,
status=503,
extra=extra or None,
)
assignment = allocation.get("assignment") or None
outcome = _clean(allocation.get("outcome"))
committed = bool(outcome == allocator_service.OUTCOME_ASSIGNED and assignment)
assigned = bool(
outcome == allocator_service.OUTCOME_ASSIGNED
and assignment
and _selection_matches(allocation.get("selected"), request)
committed and _selection_matches(allocation.get("selected"), request)
)
if not assigned:
blocked = outcome in {
@@ -769,15 +821,45 @@ def apply_request(
allocator_service.OUTCOME_BLOCKED_TERMINAL,
allocator_service.OUTCOME_BLOCKED_EXCLUDED_OWN_LEASE,
}
extra = {"allocator_evidence": _allocator_evidence(allocation)}
detail = (
f"the allocator returned {outcome or 'no outcome'} rather than "
"an assignment for this work unit; nothing was assigned."
)
if committed:
# The allocator assigned a *different* unit than the one requested.
# That assignment is real and durable; releasing it is the whole
# difference between a refusal and a silent leak.
compensation = _release_stray_assignment(
request, allocation, session_id=session_id
)
extra["compensation"] = compensation
if compensation.get("released"):
detail = (
"the allocator assigned a different work unit than the one "
"requested; that assignment was released and nothing "
"remains assigned."
)
else:
extra["orphaned_assignment"] = compensation
return _refuse(
OUTCOME_BLOCKED if blocked else OUTCOME_WAIT,
REASON_ALLOCATOR_OUTCOME,
(
"the allocator assigned a different work unit than the "
"one requested and it could not be released; it must be "
"reclaimed explicitly."
),
status=409,
extra=extra,
mutation_performed=True,
)
return _refuse(
OUTCOME_BLOCKED if blocked else OUTCOME_WAIT,
REASON_ALLOCATOR_OUTCOME,
(
f"the allocator returned {outcome or 'no outcome'} rather than "
"an assignment for this work unit; nothing was assigned."
),
detail,
status=409,
extra={"allocator_evidence": _allocator_evidence(allocation)},
extra=extra,
)
_audit(
@@ -853,19 +935,175 @@ def _run_allocator(
*,
apply: bool,
expected_candidate_set_fingerprint: str | None = None,
session_id: str | None = None,
) -> dict[str, Any] | None:
fn = allocator or default_allocator
kwargs: dict[str, Any] = {
"request": request,
"apply": apply,
"expected_candidate_set_fingerprint": expected_candidate_set_fingerprint,
}
if session_id:
# Injected allocators in tests predate this argument; only pass it to
# callables that accept it so a fake signature is never broken.
if _accepts_session_id(fn):
kwargs["session_id"] = session_id
try:
result = fn(
request=request,
apply=apply,
expected_candidate_set_fingerprint=expected_candidate_set_fingerprint,
)
result = fn(**kwargs)
except Exception: # noqa: BLE001 — an allocator failure denies, never proceeds
return None
return result if isinstance(result, dict) else None
def _accepts_session_id(fn: AllocatorFn) -> bool:
try:
params = inspect.signature(fn).parameters
except (TypeError, ValueError): # builtins / C callables
return False
if "session_id" in params:
return True
return any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values())
def _release_stray_assignment(
request: WorkRequest,
allocation: Mapping[str, Any],
*,
session_id: str | None,
) -> dict[str, Any]:
"""Release an assignment the allocator committed for the wrong work unit.
The apply path can commit an assignment and then discover, on egress, that
the allocator selected a *different* unit than the one requested — the CAS
fingerprint hashes only ``{kind, number}`` plus exclusions, so a lease taken
on the requested unit inside the window leaves the fingerprint identical and
the pin passes while the selection moves on. Without compensation that lease
is orphaned: owned by a session nothing heartbeats, and invisible because the
response says nothing was mutated.
Returns a record describing what was attempted so the caller can report it,
including a reclaim action when the release itself did not succeed.
"""
assignment = allocation.get("assignment") or {}
lease_id = _clean(assignment.get("lease_id")) or None
assignment_id = _clean(assignment.get("assignment_id")) or None
owner = _clean(allocation.get("session_id")) or session_id or None
selected = allocation.get("selected") or {}
record: dict[str, Any] = {
"attempted": False,
"released": False,
"lease_id": lease_id,
"assignment_id": assignment_id,
"owner_session_id": owner,
"selected": {
"kind": _clean(selected.get("kind")).lower() or None,
"number": selected.get("number"),
},
}
if not lease_id:
# Nothing durable to release (or the allocator reported no lease id);
# still surface the assignment id so a leak is never silent.
record["detail"] = (
"the allocator reported an assignment with no lease id; nothing "
"could be released automatically."
)
if assignment_id:
record["reclaim_action"] = _reclaim_action(record)
return record
if not owner:
record["detail"] = (
"the owning session id is unknown; the assignment cannot be "
"released automatically."
)
record["reclaim_action"] = _reclaim_action(record)
return record
record["attempted"] = True
try:
import control_plane_db
control_plane_db.ControlPlaneDB().release_lease(lease_id, session_id=owner)
except Exception as exc: # noqa: BLE001 — a failed release must be reported
record["error"] = str(exc)
record["detail"] = (
"the allocator created an assignment for a different work unit and "
"releasing it failed; it must be reclaimed explicitly."
)
record["reclaim_action"] = _reclaim_action(record)
return record
record["released"] = True
record["detail"] = (
"the allocator created an assignment for a different work unit; it was "
"released, so no assignment persists from this request."
)
return record
def _reclaim_action(record: Mapping[str, Any]) -> dict[str, Any]:
"""The explicit operator action for an assignment that outlived its request."""
lease_id = record.get("lease_id")
return {
"tool": "gitea_release_workflow_lease",
"lease_id": lease_id,
"session_id": record.get("owner_session_id"),
"assignment_id": record.get("assignment_id"),
"instructions": (
"A control-plane assignment was created for a work unit other than "
"the requested one and could not be released automatically. Call "
f"gitea_release_workflow_lease(lease_id={lease_id!r}, "
f"session_id={record.get('owner_session_id')!r}) to reclaim it, or "
"wait for the lease TTL to expire."
),
}
def _sweep_session_leases(
request: WorkRequest, *, session_id: str | None
) -> dict[str, Any]:
"""Release any lease this flow's session owns after an indeterminate apply.
``_run_allocator`` returns ``None`` for *any* exception, and
``allocate_next_work`` only catches ``InvalidWorkKindError``,
``LeaseRequiredError`` and ``ControlPlaneError`` — so an unexpected error
raised *after* ``assign_and_lease`` committed surfaces as "no result" with a
lease already written. Because the session id is now stable across the flow,
that lease is findable: anything this session owns after a failed apply is by
definition unclaimed by anyone, so it is released.
"""
record: dict[str, Any] = {"attempted": False, "released": [], "session_id": session_id}
if not session_id:
return record
record["attempted"] = True
try:
import control_plane_db
db = control_plane_db.ControlPlaneDB()
leases = db.list_leases(
remote=request.remote,
org=request.org,
repo=request.repo,
statuses=("active",),
)
for row in leases:
if _clean(row.get("session_id")) != session_id:
continue
lease_id = _clean(row.get("lease_id")) or None
if not lease_id:
continue
db.release_lease(lease_id, session_id=session_id)
record["released"].append(
{
"lease_id": lease_id,
"work_kind": row.get("work_kind"),
"work_number": row.get("work_number"),
}
)
except Exception as exc: # noqa: BLE001 — best-effort; never mask the refusal
record["error"] = str(exc)
return record
def _load_claims(
request: WorkRequest, claims_source: ClaimsFn | None
) -> Mapping[tuple[str, int], dict[str, Any]] | None:
@@ -894,12 +1132,19 @@ def default_allocator(
request: WorkRequest,
apply: bool,
expected_candidate_set_fingerprint: str | None = None,
session_id: str | None = None,
) -> dict[str, Any] | None:
"""Run the real allocator over the live queue for *request*'s scope.
An incomplete candidate inventory returns ``None`` rather than a ranking
over a partial set (#758): selecting from a short list can pick the wrong
work unit, so the request denies instead.
*session_id* is supplied by the caller so the dry-run and the apply share one
control-plane identity; the lease an apply creates is then owned by an id the
request flow still holds. A dry run additionally goes through the allocator's
``side_effect_free`` path, so a preview writes no session row and sweeps no
leases (#643).
"""
import control_plane_db
@@ -917,7 +1162,7 @@ def default_allocator(
db = control_plane_db.ControlPlaneDB()
result = allocator_service.allocate_next_work(
db,
session_id=f"webui-request-{uuid.uuid4().hex[:12]}",
session_id=session_id or _new_session_id(),
role=request.desired_role,
remote=request.remote,
org=request.org,
@@ -926,6 +1171,7 @@ def default_allocator(
apply=bool(apply),
allocation_mode="role_scoped",
expected_candidate_set_fingerprint=expected_candidate_set_fingerprint,
side_effect_free=not apply,
)
if isinstance(result, dict):
result.setdefault("candidate_count", len(candidates))