How to Stop Concurrent Jobs From Spending the Same Usage Limit Twice

Let's say you have a customer on a plan of 100,000 tokens a month. Five of their jobs start at the same moment, and each job expects to use 40,000. Each job checks the account before it starts. Each check sees 100,000 left, so each job runs.

The five jobs spend 200,000 tokens but the plan allowed only 100,000. Every check was correct when it ran, and the customer still ended up 100,000 over.

This failure is easy to miss, because it needs no bug in the check. It needs only two jobs that start close together and take a long time to finish. AI agents and long-running jobs fit that description well, and they are the work that costs the most.

So, how do you make a usage limit hold when many expensive jobs start at once, when some of them cost more than expected, and when some of them never finish?

This post answers that question with a small system I built called Earmark. Earmark is a gate that sits in front of expensive work. It holds part of the customer's limit before a job starts, stops the job at what it holds, and charges the real cost when the job ends. The plan limits come from Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to read a limit, and a way to report usage.

Why a limit check is not a limit

A limit is the most a customer may use in one billing cycle. A billing cycle is one month of usage. A check is a read: it looks at what the customer has used so far and compares that number with the limit.

A check is a read, and a limit has to be a promise. The difference shows up when work takes time. Usage is counted when a job ends, so a job that is still running has not spent anything yet. A new job that checks the account cannot see it.

Four things go wrong, and each one needs a different part of the design.

Every job passes the check. This is the case from the start of this post. Five jobs look at the same remaining capacity at the same moment, and none of them can see the others.

A job costs more than expected. The estimate was 10,000 tokens, and the model used 15,000. A check at the start cannot stop this, because the job was allowed in at the start.

A job fails halfway. It used 60% of its estimate and then stopped. If you charge nothing, you give the work away. If you charge the estimate, you bill for work that did not happen.

A worker disappears. The process is killed in the middle of a job. Whatever the system set aside for that job stays set aside, and the customer is locked out of their own limit.

What the provider gives you

A gate needs a number to enforce, and Kinde is where that number lives. In Kinde, a plan has features, and a feature can be metered. A metered feature is billed by consumption and has a usage limit that you set for each plan. Kinde's billing concepts page describes features and limits, and its metered usage page describes how to report consumption.

The Kinde pages I read about plans and billing concepts do not say whether Kinde blocks work when a limit is reached. Kinde's guide on billing for AI and LLM-based APIs covers cost control for this kind of work, but it does not describe holding capacity before a request runs. So I treat Kinde as the source of the limit and let the gate enforce it.

Earmark reads the limit from the customer's entitlements. An entitlement is a feature that a plan grants to a customer. The API reference for the billing entitlements endpoint lists GET /api/v1/billing/entitlements, which takes a customer_id and returns each entitlement with a feature_code and an entitlement_limit_max. The call needs the read:billing_entitlements scope. A null maximum means the plan sets no maximum, and Earmark treats that as no limit.

Settled usage goes the other way, from the gate to the meter. The meter is the provider's running total for the cycle. Earmark sends the settled total as an absolute value, which sets the meter to that number, so a repeated report changes nothing. Proving that the meter matches what happened is a separate problem, and I covered it in my earlier post on reconciling metered usage.

Reserve: check and hold in one step

The first design decision is the most important one. The check and the hold must be one transaction. If a job checks first and holds second, two jobs can still check at the same moment.

A hold is a part of the limit that is promised to a running job. Earmark keeps one budget row for each account and cycle. The row has two numbers: used, which is final usage, and held, which is the sum of the holds of running jobs. The gate counts both.

export function admit(b: Budget, amount: number): Admission {
  const remaining = Math.max(0, b.limit - b.used - b.held);
  if (amount > remaining) {
    return { admitted: false, remaining, reason: `The job needs ${amount} and ${remaining} is left.` };
  }
  return { admitted: true, remaining: remaining - amount };
}

The function is pure, which means the same input always gives the same answer, so a test can check each rule alone. The database call around it reads the budget row, runs admit, and writes the new held value in the same transaction.

That transaction is what makes the five-job race safe. Convex, the database I used, runs mutations as serializable transactions. Its documentation describes optimistic concurrency control: a mutation that read data which another mutation changed is run again. When two mutations touch the same budget row, one of them commits first, and the other is run again and sees the new held value. Any database with serializable transactions, or a conditional update on one row, gives you the same guarantee.

Each job also sends a key. If the same key arrives twice, the gate returns the same job and holds nothing new. This matters because a client that times out will try again, and a second hold for the same job would take capacity that no job uses.

Run: the hold is also the cap

Once a job holds 40,000, the hold works as a limit on the job. The worker charges each step against the hold before it starts the next step.

export function chargeStep(held: number, used: number, tokens: number) {
  const room = Math.max(0, held - used);
  const charged = Math.max(0, Math.min(Math.floor(tokens), room));
  return { charged, capped: charged < tokens };
}

If a step would pass the hold, the gate charges only what is left and marks the job as capped. The job stops. This handles the job that costs more than expected: the customer cannot be charged for more than the hold, and no job can take capacity that another job holds.

A capped job loses its unfinished work, and that is a real cost. I chose it on purpose. The alternative is to let a job pass its hold and hope that capacity is free, which brings back the first problem in a smaller form.

Every charge also renews the hold. Earmark gives each hold a lease, which is a time limit that a sign of life renews. The lease lasts two minutes. A worker that is alive and charging keeps its hold. A worker that stops does not.

Settle: pay for what was used

When a job ends, the gate settles it. It charges what the job used and returns the rest of the hold to the budget.

export function settle(held: number, used: number) {
  const charged = Math.min(used, held);
  return { charged, released: held - charged };
}

A job that held 40,000 and used 31,200 is charged 31,200, and 8,800 goes back. A job that failed halfway is charged for the half it did. Partial completion needs no special case, because every ending is the same step with a different used value.

The idea is older than usage billing. A card authorization holds an amount on a customer's card before the final price is known. Stripe's documentation on placing a hold describes the same pattern: you capture up to the held amount, and the rest of the hold is released. Settling a job is the same move with tokens.

A worker that disappears never calls settle, so something else must. A sweeper runs on a schedule and looks for running jobs whose lease has run out. It settles each one as expired, at the usage it had recorded, and returns the rest of the hold. A worker that comes back late finds its job already settled, and the gate ignores it. Settling a job twice does nothing, so the late call and the sweeper cannot charge the same job twice.

Fail closed when the limit is unknown

A gate is only as good as its number. The limit comes from the provider, and the provider can be slow or down.

Earmark keeps the last limit it read. It uses that limit as it is for 60 seconds, so it asks the provider at most once a minute for each account. If the provider cannot be read, Earmark keeps using the last known limit for up to 10 minutes. After that, it refuses new work until the limit can be read again.

I think refusing is the right default for expensive work. A wrong refusal costs the customer a short delay. A wrong admission costs real money, and it may spend past a limit that changed while the gate could not see it. For cheap work I would choose differently, because the price of a mistake is small. The cost of the work should set the default.

A plan can also change while jobs run. Earmark does not change a running job's hold when the plan changes. The new limit applies to the next admission.

Keep the budget row small

Every admission and every settlement writes the same budget row, so that row is a serial point. This is the price of an exact limit, and it is the right price. The design goal is to keep everything else off that row.

Convex's error documentation names two causes of write conflicts: frequent writes to one document, and mutations that read a whole table or a wide range. Both apply here, so Earmark follows three rules.

First, the rate limit is a counter and not a scan. Earmark allows 120 new jobs a minute for each account. Counting recent job rows would make every running job conflict with every new admission. Earmark keeps a small counter on the account's billing row instead.

Second, the report state lives in its own table. Whether a new total is waiting for the provider is not part of admission, so it does not belong on the budget row.

Third, a conflict is retried only where retrying is safe. Admission is safe to repeat because it is keyed, and settlement is safe because a job settles once. Each retry waits a random time that grows, up to one second, and gives up after eight tries.

What happened when I ran it

I ran the system in two ways. The first is the Concurrency Lab, a page in the app that starts five jobs of 40,000 at the same moment against a plan of 100,000. It runs the same jobs twice: once with a plain check, and once with a hold.

With a plain check, all five jobs started and 200,000 was spent, which is 100,000 over the plan. With a hold, two jobs started, three were refused with the amount that was left, and 80,000 was spent. This is the case from the start of this post.

The second run is a load script. It sends 120 jobs of 1,000 tokens, 50 requests at a time, against an account that has room for exactly 100 of them.

The gate admitted 100 jobs and refused 20. No request failed, and the run spent 100,000 tokens, which is all the plan had left and not one token more. Nothing was still held when the jobs ended. The run took 6.7 seconds, about 18 requests a second. The median time to start a job was 1,316 ms, the 95th percentile was 3,717 ms, and the 99th was 5,099 ms.

Limits

This system has edges, and I want to name them.

• The numbers come from a laptop. I ran the load script from a laptop in Lagos against a Convex cloud development deployment. The latency includes that distance. The run shows that the limit holds under concurrency, not what a production host will do.
• One account is one serial point. The load run used one account, and every admission wrote one budget row. Many accounts run in parallel because each has its own row, but I did not measure that.
• The worker is simulated. The jobs are real database records that hold, charge and settle. The work itself is a loop that charges tokens. I did not run an actual model, and I did not kill a real process. The crash path is tested with a simulated crash.
• I did not run it against a live Kinde account. The limit read and the usage report are tested with fake responses. I took the entitlements endpoint from its API reference, so the scope and field names still need a live check.
• A hard cap costs work. A job that reaches its hold stops, and the customer loses what it had not finished. A softer policy is possible, but it brings the overspend back.
• The gate is only as good as the estimate. An estimate that is too low means more capped jobs. An estimate that is too high locks capacity, and the gate refuses a job even though room exists. Earmark does not learn estimates. You have to supply a good one.
• Plan changes and caching are only touched on. Earmark keeps a limit for a minute and uses the new one at the next admission. Keeping every service in agreement during a plan change is a larger problem than this post covers.

Three questions to ask of your own gate

Before you copy this design, ask these questions of the gate you have now.

Does your check also take a hold? If it only reads usage, it is a check and not a limit.

What happens to the hold when a worker stops? If the answer is "nothing", one crash can lock a customer out until someone notices.

How far apart are your estimates and your real costs? Pull the last month of finished jobs and compare the two. The size of that gap tells you how much capacity your holds will lock up and how often your jobs will be capped.

Run it yourself

The code is in the repository. It runs with no billing account, using a built-in stand-in for the plan limit. In the project folder, run these commands:

npm install
npx convex dev
npm run dev

Open /app and open the Concurrency Lab. To repeat the load run, start with a fresh account and run node scripts/load.mjs --jobs 120 --estimate 1000 --limit 100000 --concurrency 50 --url http://localhost:3000, using the port that npm run dev prints. The 49 tests run with npm test.

FAQ

Why hold the estimate instead of charging it?
A hold is a promise that can be returned. A charge is final. When the job ends early or costs less, the unused hold goes back to the customer's limit, and the customer pays only for what the job used.

Why not lock the whole account while a job runs?
A lock makes every other job wait, even when the plan has plenty of room. A hold takes only the part that the job needs, so many jobs can run at once until the limit is full.

What if the estimate is wrong?
If it is too low, the job is capped at its hold and stops. If it is too high, the extra hold goes back when the job settles, but it blocks other jobs while the job runs. Set the estimate close to what the job costs at the high end of normal.

Why a lease and not a timeout on the job?
A job can run for a long time and still be healthy. A lease asks a different question: has the worker shown a sign of life recently? Every charge renews it, so long jobs keep their hold and dead workers lose it.

Does this need Kinde?
No. The gate needs a way to read a limit and a way to report settled usage. Kinde supplies both in the version I built. The built-in stand-in in the repository shows the same behavior with no account.

Why send settled usage and not holds to the meter?
A hold is a promise and not usage. If the meter showed holds, a job that was refused or that returned most of its hold would still change the customer's invoice.

Sources

• The Kinde billing model, Kinde docs
• About plans, Kinde docs
• Commonly used billing concepts and terms, Kinde docs
• Add metered usage for a customer via API, Kinde docs
• Billing for AI and LLM-based APIs: cost control strategies, Kinde
• Kinde Billing Entitlements API reference, apis.io
• Optimistic concurrency control, Convex docs
• Errors and warnings, Convex docs
• Earmark source code