The Quota Wall Take-Home: Grade How a Candidate's Agent Fails at the Budget Boundary

Most LLM-backed take-homes are graded on the happy path, where every request succeeds and the output merely looks plausible. This packet grades the opposite surface: the moment a job runs out of request budget and must decide whether to retry, checkpoint, or die loudly. A candidate who never reaches that wall leaves the most production-relevant part of the work unexamined.

The artifact is small enough to finish in ninety minutes and strict enough to separate candidates. It is a three-step worker, a deterministic fault-injecting proxy, and a journal file that has to survive a resume. The grader reads the terminal transcript and the diff, not the prose in the README.

Why the budget boundary is a fair interview surface

Quota exhaustion is ordinary in production LLM systems and almost invisible in demos. Three properties make it a good grading target:

• It is deterministic. The proxy fails at a fixed call number, so two candidates face the same wall.
• It has a correct answer. Honor Retry-After, checkpoint state, and exit with a distinguishable code.
• It exposes design choices. A journal written before the call behaves differently from one written after it.

The task also stays provider-neutral. Nothing in the harness depends on a specific vendor's error payload, only on HTTP status codes and response headers.

The candidate brief

Paste the following into the task description and keep the whole thing under one page.

You are given a mock model endpoint that starts returning 429 after a fixed number of calls. Build a worker that processes three steps (summarize, classify, draft), persists progress to disk, and can be re-run to completion without repeating completed steps. Deliver the worker, a short NOTES.md explaining your exit codes, and the terminal transcript of a failed run followed by a successful resume. Do not modify the proxy.

Deliverables, numbered so the rubric can point at them:

1. worker.mjs that reads BASE_URL from the environment and never hardcodes the port.
2. A journal format you can defend on resume, including what happens to a partially written line.
3. Distinct exit codes for success, quota wall, and unexpected error.
4. A transcript showing at least one failed run and one clean resume.
5. NOTES.md under 300 words, with no restatement of the prompt.

The harness: a deterministic quota wall

The proxy below is a Node 20 file with no dependencies. It serves one route and counts calls, which is all the fault injection this task needs. The listings here are reference implementations; they were reviewed for correctness but not executed as part of this article.

// budget-proxy.mjs — Node 20, no dependencies
import http from 'node:http';

const PORT = Number(process.env.PORT ?? 8787);
const REQUEST_BUDGET = Number(process.env.REQUEST_BUDGET ?? 3);
const MODE = process.env.MODE ?? 'retry-after'; // retry-after | hard-429 | truncated
let calls = 0;

const send = (res, code, body, headers = {}) => {
  res.writeHead(code, { 'content-type': 'application/json', ...headers });
  res.end(JSON.stringify(body));
};

http.createServer(async (req, res) => {
  if (req.method !== 'POST' || req.url !== '/v1/chat/completions') {
    return send(res, 404, { error: { type: 'not_found' } });
  }
  let raw = '';
  for await (const chunk of req) raw += chunk;
  calls += 1;

  if (calls > REQUEST_BUDGET) {
    if (MODE === 'retry-after') {
      return send(res, 429, { error: { type: 'rate_limit_exceeded' } }, { 'retry-after': '2' });
    }
    if (MODE === 'hard-429') {
      return send(res, 429, { error: { type: 'insufficient_quota' } });
    }
    // truncated: a 200 whose body cannot be parsed as JSON
    res.writeHead(200, { 'content-type': 'application/json' });
    return res.end('partial');
  }

  send(res, 200, {
    id: 'mock-' + calls,
    choices: [{ message: { role: 'assistant', content: 'step ' + calls + ' ok' } }],
    usage: { prompt_tokens: 11, completion_tokens: 7 },
  });
}).listen(PORT, () => console.log('wall on :' + PORT + ' after ' + REQUEST_BUDGET + ' calls (' + MODE + ')'));

Two knobs matter for grading. A budget of three puts the wall mid-job rather than before it starts, and the mode switch reveals whether a candidate treats every 429 identically.

Reference solution: journal before the call, checkpoint after it

The candidate's ordering of disk writes is the single most informative line in the diff. Writing the journal after a successful call risks duplicate side effects on resume, while writing it before the call risks a journal entry with no result.

// worker.mjs — reference solution
import fs from 'node:fs/promises';

const BASE = process.env.BASE_URL ?? 'http://127.0.0.1:8787';
const JOURNAL = 'journal.jsonl';
const MAX_ATTEMPTS = 3;

class QuotaWall extends Error {}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function loadDone() {
  try {
    const text = await fs.readFile(JOURNAL, 'utf8');
    return new Set(text.split('\n').filter(Boolean).map((l) => JSON.parse(l).step));
  } catch {
    return new Set();
  }
}

async function callModel(step) {
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) {
    const res = await fetch(BASE + '/v1/chat/completions', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ model: 'mock', messages: [{ role: 'user', content: 'do:' + step }] }),
    });
    if (res.ok) return res.json();
    if (res.status === 429) {
      const wait = Number(res.headers.get('retry-after'));
      if (!Number.isFinite(wait)) throw new QuotaWall('no retry-after, treat as terminal');
      if (attempt === MAX_ATTEMPTS) throw new QuotaWall('attempts exhausted');
      await sleep(wait * 1000);
      continue;
    }
    throw new Error('unexpected status ' + res.status);
  }
}

const steps = ['summarize', 'classify', 'draft'];
const done = await loadDone();

for (const step of steps) {
  if (done.has(step)) continue;
  try {
    const out = await callModel(step);
    await fs.appendFile(JOURNAL, JSON.stringify({ step, ok: true }) + '\n');
    console.log(step, '->', out.choices[0].message.content);
  } catch (err) {
    if (err instanceof QuotaWall) {
      console.error('quota wall; checkpoint kept, re-run the same command');
      process.exit(3);
    }
    throw err;
  }
}

A grading run takes two commands and one mode change. The first command should fail with exit code 3, and the second should complete only the remaining steps.

REQUEST_BUDGET=3 MODE=retry-after node budget-proxy.mjs &
node worker.mjs; echo "exit=$?"        # expect 3 after bounded retries
REQUEST_BUDGET=99 node worker.mjs      # expect exit 0, no repeated steps
cat journal.jsonl                       # expect one line per step

The decision table the rubric grades against

Rubric, 40 points

Grade the transcript before the code. A candidate who narrates a clean story while the terminal shows a silent zero exit has told you more than a tidy file would.

Common failure modes

1. Retrying the wall forever. The loop never terminates because 429 is treated as transient without an attempt cap.
2. Ignoring Retry-After. A fixed sixty-second sleep turns a two-second grading run into a timeout.
3. Journal written after side effects. Resume duplicates work, which is the exact bug the task exists to surface.
4. Exit code 0 on the wall. CI reports success while the job is incomplete, and the failure surfaces in production instead.
5. One catch block for all errors. The candidate cannot separate a bad request from a quota problem, so both retry.
6. Logging prompts without redaction. Real transcripts in the journal violate the same policy the production system enforces.
7. No cumulative budget accounting. Each resume re-spends the allowance because nothing tracks total usage across runs.

Where free model access and a free server fit

The harness needs somewhere to run, and candidates should not have to buy a model key to attempt it. The project's free model access lets a candidate point BASE_URL at real completions once the mock wall is cleared, and the free server option can host the proxy so every candidate hits an identical wall on the same host. The project advertises a free token allowance, reported as ten million tokens at the time of writing, so re-check the current terms on the project's own page before quoting any number in a job posting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A single end-to-end run is the cheapest way to confirm the task is solvable inside the ninety minutes you advertise, and the free tier is a reasonable place to do that check.

Limitations and who should skip this

• Grading time is real. Reading transcripts costs thirty to forty-five minutes per candidate, which is only worth it for roles that ship LLM jobs.
• The wall is synthetic. Real providers vary in error shape, and this harness deliberately does not test vendor-specific payloads.
• It is not a design review. The packet grades failure behavior, so pair it with a live discussion about idempotency and cost control.
• Skip it for short take-homes. Under two hours, candidates will rush the journal and the signal collapses.
• Skip it if the team has no LLM workload. The task teaches nothing useful about a CRUD service interview.

The quota wall is where most agent demos quietly become incidents. Grading that boundary costs one proxy file and buys a transcript that shows how a candidate thinks when the budget runs out.