Metering streaming AI inference without slowing it down

An inference API with usage-based billing has an uncomfortable property. Every request is a small financial transaction. Before a single token streams, the platform must know the caller can pay for it. After the stream ends, it must charge for exactly what was used, exactly once. And it must do all of this in single-digit milliseconds, because billing overhead is pure tax on every call.

On one of the platforms I built, the shape of the problem came down to three constraints that rule out all the comfortable answers.

Why the obvious designs fail

Read-then-write double-spends. The naive check (read the balance, compare, decrement) is a race. Two concurrent requests both read a balance of 10, both approve a cost of 8, and you've sold 16 credits of inference for 10 credits of money. Load makes it worse precisely when it matters.

The ledger is too slow for the request path. A SQL transaction with row locks fixes correctness and costs tens of milliseconds under contention, on every call forever. The financial system of record and the enforcement point want different data stores.

Streams don't know their cost up front. An LLM response might be ten tokens or ten thousand. You can't charge the exact amount at request time because the exact amount doesn't exist yet.

Reserve, stream, settle

This is the shape hotels and fuel pumps landed on decades ago: a two-phase transaction. At request time, reserve a conservative estimate. After the stream ends, settle for actual usage. Release the hold, and deduct the real cost.

The part that makes it fast is where the phases run. Both are single atomic scripts against an in-memory store. Authorization checks balance − reserved ≥ cost and increments the reservation in one round-trip, with no locks, no transactions, and no second network hop. The store executes scripts serially, so the race disappears by construction rather than by coordination.

Settlement is asynchronous. The client's stream ends the moment the model stops, not after bookkeeping. The usage event fans out through a queue, and a worker deducts actual usage, releases the reservation, and writes the durable record. The whole authorize path was budgeted at single-digit milliseconds p50 end to end, itemized per step. That's a design budget, not a production benchmark. Holding it is exactly why nothing heavier than the in-memory store is allowed on the path.

Three speeds of data

The reservation trick only works if you let each store do the one thing it's best at. The architecture runs on three speeds. An in-memory store enforces: balances and reservations, sub-millisecond. A wide-column key-value store counts: usage counters, idempotency records, a warm fallback, single-digit milliseconds. A relational database accounts: the ACID ledger that auditors and refunds live on. The enforcement layer runs minutes ahead of the relational store, and a compaction cron reconciles them.

At-least-once, exactly-once-billed

Queues redeliver. A settlement event processed twice is a customer charged twice, so a third atomic script makes settlement claim-or-replay. The first delivery claims the settlement, and every redelivery replays to the same result. The same discipline repeats at every async boundary (conditional puts keyed on transaction IDs, dedup on webhook events), which is what lets standard at-least-once queues carry the async tier without making queue-level deduplication the billing boundary.

The failure stance

One last decision makes the whole thing coherent. The billing path fails closed. If its stores are unreachable, the API returns 503 rather than serve unbilled inference. The developer console makes the opposite choice. Its caches fall through to the database, because a cache outage should not take down customers. Failure semantics are a product decision per service, and writing them down before the incident is most of the value.