Test Reviewer Capacity Before Raising an AI Queue Limit

A larger queue still needs a review reservation

An AI drafting service has two items waiting for review. One reserves two review units and the other reserves one. The current window has a declared capacity of three units. Raising the queue limit to fifty creates room for more item records, while the review budget is already fully reserved. A rule that checks only the item count will admit another draft into that window.

The Python fixture below reproduces that error. Its corrected admission function passes 21 local tests. A controlled copy with the budget check removed fails two tests. Both failures admit work that should remain held. The tests also check snapshot freshness, a changed ledger version and repeated delivery of the same work item. You can copy the complete function and test file into an empty directory and reproduce the assertions.

The example uses fictional review units and a fixed clock. An owner supplies the budget and each item's declared cost. The code makes a decision about those inputs. Estimating actual reviewer availability and evaluating the quality of reviewed work remain separate operating tasks. The tests were run locally by an AI agent in Python 3.14.7, without model calls or a production queue.

An AI agent prepared this tutorial, code and synthetic fixtures and ran the local checks. Human manuscript review has not been performed. The author affiliation is Pharos Production. The reported observations come from the included local fixture.

Decide what the review budget represents

Choose a unit that the review owner can explain. A team might start with fixed units for predefined review classes, with a difficult item reserving more than a routine one. Another team might allocate minutes after measuring its workflow. Either approach needs a documented mapping from work to the reservation it consumes. This fixture uses integers so the examples stay deterministic and easy to inspect.

In Google's SRE Workbook chapter on identifying and recovering from overload, the authors write "The team should work together to decide what measures to use." The chapter discusses team workload and overload. It gives a reason to involve the receiving reviewers when selecting an input to an admission policy. It supplies no conversion from a generated answer's length to a person's available review capacity.

For W-117, assume every reservation belongs to one review scope and one planning window. Item A reserves two units. Item B reserves one. Both remain unresolved. A proposed item C costs two. With a capacity of five, C fits exactly. With a capacity of three, C must wait. The same arithmetic applies when the item-count limit is fifty because that limit constrains a different part of the state.

Keep the total declared window capacity separate from used reservations. Recomputing used units from the current reservation ledger avoids treating a previously calculated free balance as current. The fixture sums all retained reservations. Completion, cancellation and reassignment require their own approved transitions before a reservation disappears. Treating a lost acknowledgement as completion would release capacity while the original work could still need review.

A review unit can be revised as the team learns more. Save the mapping version with the operating configuration and apply changes deliberately. An item that becomes more expensive after an artifact changes needs an explicit revision decision. The admission path should make that change visible instead of silently reducing another item's allocation.

Define the state passed to the function

The ledger contains the review scope, the planning window and a monotonically advancing version. It also carries the window's start and end and the reservations already accepted. The snapshot supplies the declared capacity for that same scope, window and ledger version. Its capture time lets the policy decide how old the observation is when new work arrives.

All times in the fixture are integer ticks in the same synthetic clock domain. W-117 starts at tick 90 and ends at tick 160. A capacity snapshot captured at tick 100 is checked at tick 110. The maximum permitted age is thirty ticks. These values are chosen inputs for the demonstration. A deployed service needs an agreed clock source and a policy for observations delayed by collection or transport.

The function expects a well-formed internal dictionary envelope with the documented keys. It validates the integer values relevant to the rule, including rejection of booleans as integers. A public-facing adapter would need its own schema, payload bounds and authorization checks before building that envelope. Keep that adapter's rejected input separate from a valid request receiving a HOLD decision.

An existing reservation is checked before the freshness requirements for new admission. A matching replay returns ALREADY_ADMITTED and the existing ledger. That path retrieves an earlier allocation. It gives no permission to reserve more work. A cost or revision change under an existing work ID raises a conflict, so the caller must resolve the identity instead of letting duplicate delivery rewrite history.

Implement the reservation decision

Save the following file as admission.py. The return value contains a decision, a reason and a copied ledger. HOLD returns the prior state. ADMITTED adds one reservation and advances the version. ALREADY_ADMITTED returns the existing allocation. All three results can be displayed to an operator without converting a held item into a failed review.

The checks run before the returned state is committed. In a real service, the caller must atomically compare the version and persist the update. This pure function gives the caller a state transition to attempt. The database section later explains the additional boundary needed when two workers compete for the same capacity.

from copy import deepcopy


def integer(value, name, minimum=0):
    if type(value) is not int or value < minimum:
        raise ValueError(f"{name} must be an integer >= {minimum}")
    return value


def admit(ledger, snapshot, work, *, now, queue_limit, max_age):
    """Plan one reservation. The caller must commit it atomically."""
    for name in ("scope", "window"):
        if not isinstance(ledger[name], str) or not ledger[name]:
            raise ValueError(f"ledger {name} is required")
    for name in ("id", "revision"):
        if not isinstance(work[name], str) or not work[name]:
            raise ValueError(f"work {name} is required")
    cost = integer(work["cost"], "cost", 1)
    integer(now, "now")
    integer(queue_limit, "queue_limit")
    integer(max_age, "max_age")
    integer(ledger["version"], "ledger version")
    integer(ledger["start"], "window start")
    integer(ledger["end"], "window end")
    if ledger["end"] <= ledger["start"]:
        raise ValueError("window end must follow start")
    for item in ledger["reservations"].values():
        integer(item["cost"], "reserved cost", 1)
        if not isinstance(item["revision"], str) or not item["revision"]:
            raise ValueError("reservation revision is required")

    def result(decision, reason, state=None):
        return {
            "decision": decision,
            "reason": reason,
            "ledger": deepcopy(ledger if state is None else state),
        }

    existing = ledger["reservations"].get(work["id"])
    reservation = {"cost": cost, "revision": work["revision"]}
    if existing is not None:
        if existing != reservation:
            raise ValueError("work identity conflicts with reservation")
        return result("ALREADY_ADMITTED", "EXISTING_RESERVATION")

    integer(snapshot["captured_at"], "snapshot time")
    integer(snapshot["version"], "snapshot version")
    capacity = integer(snapshot["capacity"], "capacity")
    if (snapshot["scope"], snapshot["window"]) != (
        ledger["scope"], ledger["window"]
    ):
        return result("HOLD", "SCOPE_MISMATCH")
    if not ledger["start"] <= now < ledger["end"]:
        return result("HOLD", "WINDOW_CLOSED")
    if not ledger["start"] <= snapshot["captured_at"] <= now:
        return result("HOLD", "INVALID_SNAPSHOT_TIME")
    if now - snapshot["captured_at"] > max_age:
        return result("HOLD", "STALE_SNAPSHOT")
    if snapshot["version"] != ledger["version"]:
        return result("HOLD", "VERSION_CONFLICT")
    if len(ledger["reservations"]) >= queue_limit:
        return result("HOLD", "QUEUE_FULL")
    used = sum(item["cost"] for item in ledger["reservations"].values())
    if used + cost > capacity:
        return result("HOLD", "REVIEW_BUDGET_EXHAUSTED")
    updated = deepcopy(ledger)
    updated["reservations"][work["id"]] = reservation
    updated["version"] += 1
    return result("ADMITTED", "RESERVED", updated)

The strict comparison in the budget check matters. Used units plus requested units equal to capacity are permitted. An item exceeding the budget by one unit is held. The queue-slot check remains independent, so a request can be held because the queue is full even when unused review units remain. The reason is selected from the first applicable new-work check in the function.

A caller should keep those reasons distinct in its records. A capacity owner can update an old snapshot, whereas a queue owner may need to resolve accepted work. Combining both into a generic busy response makes it harder to determine what action could change the result. Save the request identity and the state version beside the decision so a later reviewer can reproduce the same branch.

Run the complete test file

Save this second file as test_admission.py beside admission.py. The cases use the standard library's unittest framework. Each test starts with the same ledger and modifies only the condition it is testing. That makes the two budget regressions easy to trace back to the capacity rule.

The assert_held helper checks the returned decision and reason together. It also compares the entire returned ledger and the original input against the prior state. A function that reports HOLD while dropping item B would therefore fail the same contract. The success test inspects the actual reservation and the advanced version, then another test confirms that the caller's dictionaries were preserved.

from copy import deepcopy
import unittest

from admission import admit


class AdmissionTests(unittest.TestCase):
    def setUp(self):
        self.ledger = {
            "scope": "ai-draft-review", "window": "W-117", "version": 7,
            "start": 90, "end": 160,
            "reservations": {
                "A": {"cost": 2, "revision": "r1"},
                "B": {"cost": 1, "revision": "r1"},
            },
        }
        self.snapshot = {
            "scope": "ai-draft-review", "window": "W-117", "version": 7,
            "captured_at": 100, "capacity": 5,
        }
        self.work = {"id": "C", "revision": "r1", "cost": 2}
        self.options = {"now": 110, "queue_limit": 50, "max_age": 30}

    def check_admission(self):
        return admit(self.ledger, self.snapshot, self.work, **self.options)

    def assert_held(self, reason):
        before = deepcopy(self.ledger)
        result = self.check_admission()
        self.assertEqual((result["decision"], result["reason"]), ("HOLD", reason))
        self.assertEqual(result["ledger"], before)
        self.assertEqual(self.ledger, before)

    def test_exhausted_budget_despite_larger_queue(self):
        self.snapshot["capacity"] = 3
        self.assert_held("REVIEW_BUDGET_EXHAUSTED")

    def test_over_budget(self):
        self.work["cost"] = 3
        self.assert_held("REVIEW_BUDGET_EXHAUSTED")

    def test_exact_fit_reserves_and_advances_version(self):
        result = self.check_admission()
        self.assertEqual(result["decision"], "ADMITTED")
        self.assertEqual(result["ledger"]["version"], 8)
        self.assertEqual(result["ledger"]["reservations"]["C"], {"cost": 2, "revision": "r1"})

    def test_queue_slots_are_an_independent_limit(self):
        self.options["queue_limit"] = 2
        self.assert_held("QUEUE_FULL")

    def test_stale_snapshot(self):
        self.options["now"] = 131
        self.assert_held("STALE_SNAPSHOT")

    def test_snapshot_at_age_boundary(self):
        self.options["now"] = 130
        self.assertEqual(self.check_admission()["decision"], "ADMITTED")

    def test_future_snapshot(self):
        self.snapshot["captured_at"] = 111
        self.assert_held("INVALID_SNAPSHOT_TIME")

    def test_snapshot_before_window(self):
        self.snapshot["captured_at"] = 89
        self.assert_held("INVALID_SNAPSHOT_TIME")

    def test_closed_window(self):
        self.options["now"] = 160
        self.assert_held("WINDOW_CLOSED")

    def test_window_not_started(self):
        self.options["now"] = 89
        self.assert_held("WINDOW_CLOSED")

    def test_wrong_scope(self):
        self.snapshot["scope"] = "other-queue"
        self.assert_held("SCOPE_MISMATCH")

    def test_wrong_window(self):
        self.snapshot["window"] = "W-118"
        self.assert_held("SCOPE_MISMATCH")

    def test_stale_version(self):
        self.snapshot["version"] = 6
        self.assert_held("VERSION_CONFLICT")

    def test_replay_retains_existing_reservation(self):
        self.work = {"id": "A", "revision": "r1", "cost": 2}
        result = self.check_admission()
        self.assertEqual(result["decision"], "ALREADY_ADMITTED")
        self.assertEqual(result["ledger"], self.ledger)

    def test_replay_with_changed_cost_raises(self):
        self.work = {"id": "A", "revision": "r1", "cost": 1}
        with self.assertRaises(ValueError):
            self.check_admission()

    def test_replay_with_changed_revision_raises(self):
        self.work = {"id": "A", "revision": "r2", "cost": 2}
        with self.assertRaises(ValueError):
            self.check_admission()

    def test_old_snapshot_replay_is_a_read_of_existing_work(self):
        self.work = {"id": "A", "revision": "r1", "cost": 2}
        self.snapshot["captured_at"] = 0
        result = self.check_admission()
        self.assertEqual(result["decision"], "ALREADY_ADMITTED")
        self.assertEqual(result["ledger"], self.ledger)

    def test_success_does_not_mutate_inputs(self):
        before = deepcopy((self.ledger, self.snapshot, self.work))
        self.check_admission()
        self.assertEqual((self.ledger, self.snapshot, self.work), before)

    def test_zero_cost_raises(self):
        self.work["cost"] = 0
        with self.assertRaises(ValueError):
            self.check_admission()

    def test_boolean_capacity_raises(self):
        self.snapshot["capacity"] = True
        with self.assertRaises(ValueError):
            self.check_admission()

    def test_new_work_after_prior_admission_needs_new_snapshot(self):
        self.ledger = self.check_admission()["ledger"]
        self.work = {"id": "D", "revision": "r1", "cost": 1}
        self.assert_held("VERSION_CONFLICT")


if __name__ == "__main__":
    unittest.main(verbosity=2)

Run python3 -m unittest -v test_admission.py from that directory. The supplied corrected fixture ran 21 tests successfully in the local execution. The recorded elapsed duration was 0.001 seconds. That small local duration describes this test process on the machine used for the article. It carries no implication about review latency, model throughput or a deployed service's response time.

For the controlled failure, copy both files to a separate directory and change only the condition if used + cost > capacity to if False. Run the same command there. The local run kept nineteen tests passing and failed test_exhausted_budget_despite_larger_queue and test_over_budget. Both expected a held decision and received ADMITTED with reason RESERVED. Restore the original condition and rerun the corrected file before adopting any part of the example.

Read the exception tests as input-contract checks. Their expected result is ValueError, so a conflicting revision cannot become an ordinary held request that the caller retries indefinitely. The caller needs to correct the identity or invoke an authorized change process. The boolean-capacity test also matters in Python because booleans participate in integer behavior. The explicit type check keeps a supplied True from becoming a one-unit budget.

The test suite uses independent setup for every case. A prior success cannot accidentally consume capacity needed by a later assertion. The test for a stale snapshot after admission deliberately carries the returned ledger into the next call, preserving the old snapshot to expose the version conflict. That case models the change the caller must recognize after another reservation becomes authoritative.

Keep the failure output with the corrected output. A green result alone cannot show whether the tests would notice removal of the decision that protects the budget. Here the deliberate defect demonstrates sensitivity to that particular fault. Other defects require their own relevant assertions, especially once the function is attached to persistence or a queue transport.

Follow the two budget outcomes through the ledger

With the initial capacity of five, the ledger has three units reserved and C requests two. The function returns ADMITTED, adds C at revision r1 and changes the ledger version from seven to eight. The original ledger passed into the call still has only A and B. That input preservation allows a caller to retain the pre-decision state for a comparison or failed commit.

Now use capacity three with the same queue limit of fifty. The function returns HOLD with reason REVIEW_BUDGET_EXHAUSTED. The returned ledger keeps A and B unchanged at version seven. C stays outside this admission ledger. A real intake service should retain C in its own pending-request record and give it an owner or an appropriate response. This small fixture has no transport or pending-request database.

The snapshot-age boundary is tested separately. At tick 130, the snapshot from tick 100 is exactly thirty ticks old and new admission can proceed. At tick 131, it is stale and the function returns HOLD. The window itself closes at tick 160. An expired window blocks new admission even when a snapshot would otherwise be recent enough.

Existing A and B remain visible after every held decision. An operator can still identify the work already promised to reviewers. Raising a limit or pausing intake should preserve those commitments. A dashboard that counts only items admitted after the latest configuration change would hide the backlog that determines how much of the current budget is already occupied.

Refresh capacity with an identified owner

The reservation ledger can be current while the declared budget is wrong. A reviewer may become unavailable after the capacity owner published the snapshot. The version check detects changes to the ledger it names. It cannot discover an absence or a changed operating commitment that nobody recorded. The service needs a defined source for capacity and a way for the owner to reduce it when circumstances change.

A reduced capacity can fall below units already reserved. Leave those commitments visible and hold new admission while the owner examines them. The budget predicate naturally holds a new request in that state because existing cost plus a positive requested cost exceeds the new capacity. Resolving the overcommitment requires a recorded decision about accepted work, such as reassignment or an agreed delay. The admission function has no authority to make that choice.

Freshness should cover the source's actual update process. A polling adapter can attach the current time to a value fetched from an old cache, making the wrapper appear fresh. Keep the original capacity observation time through the adapter and distinguish it from retrieval time. The fixture's captured_at field represents the observation time. The owner needs to confirm that the supplying component preserves that meaning.

Store the capacity-policy revision alongside operating decisions. This example passes maximum age and queue limit directly to the function. A deployment might load them from configuration. When those values change, save the effective revision with the admission record so the same input can be replayed under the rule that originally evaluated it. A decision log should identify both the reservation version and the configuration used to judge it.

Treat work identity and content revision as a contract

Repeated delivery of A with cost two and revision r1 returns ALREADY_ADMITTED. The allocation survives intact and the ledger version stays seven. That result supports a caller recovering after an acknowledgement was lost. The caller can search for the original work identity and present the earlier outcome without inventing a new reservation.

A request for A at cost one raises ValueError. A request for A at revision r2 also raises ValueError. The conflict forces the integration to decide whether the caller made an error or the artifact really changed. Reviewing a new revision can create additional work even when the task keeps the same friendly title. The policy should define whether a revision replaces unfinished work or needs a separately reserved review.

The fixture allows retrieval of a matching existing reservation using an old snapshot. The result describes the recorded reservation and leaves the state unchanged. A new item with that same old snapshot still passes through the freshness checks. Keeping these paths distinct is useful during an outage when the service can read previous decisions but cannot establish a current capacity observation.

Retain an immutable content reference with each real review task. The fixture's r1 label represents a version supplied by the caller. A production implementation should be able to retrieve the exact artifact it names. If a document changes under the same label, matching the label will preserve allocation arithmetic while leaving the reviewer with the wrong material.

Put the check and reservation inside one commit boundary

Two workers can read version seven and each plan a reservation against the same remaining capacity. The pure function can return a valid update to both. Only an atomic persistence rule prevents both updates from becoming independently accepted work. Use the storage system's transaction or conditional-write mechanism to compare the expected ledger version when committing the reservation.

If one worker commits version eight, the other worker's commit against version seven must fail. The losing worker then rereads the authoritative ledger, obtains a valid capacity observation for that state and reevaluates its original work request. The next decision may be a budget hold or a successful admission. Reusing the earlier proposed result after a failed compare would discard the protection provided by the version field.

Keep the queue message aligned with the durable reservation. A transactional outbox is one way to record the accepted work and a publishable message together. The relay can deliver that message more than once, so the reviewer-facing consumer still needs stable task identity. Persisting a reservation and losing the message creates a recovery task. Sending work before the reservation commits creates a different gap because reviewers can receive work the ledger never accepted.

An unknown commit result needs investigation using the original work ID. First read the durable reservation or its decision record. A network timeout after a successful commit can leave the caller uncertain while capacity is already reserved. Creating another work ID during that uncertainty can allocate capacity twice. The replay test checks the function's behavior once the original reservation is visible. It does not execute the database failure or relay recovery.

Add integration tests at those boundaries before enabling automatic retries. Save evidence for one winning reservation under concurrent attempts, recovery after a lost commit response and duplicate message delivery. Verify the consumer's task record and the reservation ledger together. Those observations extend the local policy test into the components that actually promise work to reviewers.

Carry unresolved work into the next planning decision

A new planning window needs an explicit transition. W-117 can close while A remains unresolved. Copying a fresh capacity number into a new window without accounting for A can make previously promised work disappear from the planning view. The owner should decide how unfinished reservations affect the next window and retain the link to the original commitment.

The fixture handles one ledger and supplies no rollover function. At the closed boundary it preserves the existing reservations and holds new work. That behavior gives the operator time to reconcile what remains before publishing a successor window. The receiving owner needs the work identities, artifact revisions and remaining obligations, together with the capacity basis selected for the new period.

Completion also needs a defined evidence rule. A reviewer opening an item does not establish that its review is finished. Rework may require another allocation after findings are returned. Cancellation should name who authorized releasing the reservation and whether a queued copy can still reach a reviewer. Record those transitions separately so an admission log can explain why used capacity changed.

Priority handling is another policy decision. The example admits work in call order and retains a single queue-slot limit. A service with urgent work may reserve some capacity for that class or require a named override. Test the override's accounting and visibility before treating it as an operating path. A manually approved urgent item should remain visible in the same budget explanation.

Use the fixture as a release question

Before increasing an AI queue limit, inspect what the change permits when every declared review unit is already reserved. Run the exhausted-budget case, the freshness boundary and the repeated-work case against the candidate implementation. Inspect the retained unresolved items and the reason returned to the caller. A larger limit should lead to a comprehensible decision under each of those states.

The local result here establishes the behavior of a small function over synthetic inputs. The next engineering decision is whether the surrounding storage and message flow preserve the same reservation contract. Keep that decision attached to the tested implementation and the review owner's current capacity definition so a queue configuration change has an inspectable basis.

More insights to read

1. AI Agent Integration Estimates: Price the Connector Failure Paths
2. Queue Migration Acceptance: Residual Work and Final Ownership
3. Recover an AI Tool Call Without Repeating Its Effect
4. A Tool Timeout Needs a Saved Operation Identity
5. Application Support Acceptance: Requests, Releases and Recovery

About the author

Dmytro Nasyrov. Photo supplied by the author.

Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the founder and CTO of Pharos Production. He works on production software architecture for FinTech, AI, Web3 and blockchain systems.