Inference is the meter. Training was the purchase.

You typed something into a chatbot, hit send, and watched the words arrive one at a time. That was inference. The model was not learning from your question, and it was not getting better at anything while you waited. It was finished software, running.

The word carries two senses. In ordinary use it means drawing a conclusion from evidence, the way a detective infers a motive. In computing it means something narrower and more literal: taking a model whose training is already over and running it against input it has never seen. That second sense is the one this article uses from here on.

Training is where the weights are set. A model is built once, the numbers settle, and then they are frozen. Inference is every later run of those frozen weights. A model in production does not change when you use it. It reads, it produces, it forgets, and the next request starts from the same fixed numbers as the last one.

One request is not one calculation, though. It splits into two phases. In prefill, the model reads your entire prompt in a single shot, which saturates the arithmetic units and is limited by raw compute. Then comes decode, where output is produced one token at a time, and each new token requires the model to look back across everything already read and written. That lookback is limited by memory bandwidth, not arithmetic, and it is only affordable because of the KV cache: the running state the model keeps for the conversation so far. Decode depends on that cache, and the cache grows with every token in the exchange.

Avi Chawla draws the line that separates a language model from older machine learning. In classical ML, a request is one forward pass and the GPU is done. A language model turns one request into a long run of sequential passes, with prefill and decode contending for the same chip while the cache keeps growing behind them.

The rest of this article is about that loop, because the loop is what costs money.

The useful difference between the two jobs is not what they produce. It is the shape each one makes over time. The inference engineer dylan, who works on managed inference at Nebius Token Factory, states it directly: "Training is bursty. You spin up a cluster, run for weeks or months, and stop. Inference runs continuously, scales with usage, and the usage curve is exponential."

Read that as two different kinds of thing. Training is a project. It is scoped, budgeted, started, finished. You book the machines, you run them hard, and at some point the run ends and the cluster goes quiet. Inference is a utility. It has no completion date. It runs for as long as anyone is still using the product, and the meter reads higher the more they use it.

The accounting follows the shape. Training is money spent once to own the use of a model, and inference is money spent again on every request that model ever answers. The first is a purchase with a receipt. The second is a bill that arrives every month for the life of the product, charged to whoever is serving the request rather than to whoever built the model.

They are not even the same shape of machine. dylan gives the CPU-to-GPU ratio as roughly 1:8 for training, 1:4 for basic chat inference, and 1:1 for agentic inference, which in his experience sometimes tips CPU-heavy. A training cluster is a wall of accelerators fed by a thin strip of general compute. An agent loop spends far more of its time on the work around the model: tool calls, planning, retries, assembling what comes back.

Which leaves the consequence. Because inference is metered per use, the cost of an AI product rises with its success instead of falling. Ordinary software does the reverse.

dylan opens his account of the year with a line that reads like a contradiction: "Inference got a hundred times cheaper this year. The compute bill went up anyway." Every figure that follows is his, taken from what he sees running managed inference rather than from any published index. Both halves of the line are true at once.

Take the price first. Twelve months ago, he says, a million tokens of frontier-class reasoning cost on the order of $60. Today he puts equivalent quality at roughly $0.50. For reasoning-grade intelligence he calls that about a 128 times drop in a year, and for GPT-4-level output roughly 100 times since the original GPT-4 shipped. On his numbers the unit got radically cheaper.

Then take the consumption. The same year that collapsed the price changed what a single request is. dylan gives three multipliers:

• A reasoning model burns roughly 10 times the output tokens of a non-reasoning model on the same task, because it spends most of its tokens thinking out loud before it answers.
• An agentic workflow chains roughly 20 times the requests of a single-shot completion, because it loops, calls tools, plans, retries and synthesizes.
• One modern deep-research query costs more compute than 10 original GPT-4 queries combined.

Those stack, and dylan multiplies them out in one line: "100x cheaper tokens times 10 000 more tokens equals a 100x larger total bill." The price per unit fell by two orders of magnitude. The number of units rose by four. Nobody sent fewer requests because tokens got cheap. They sent more, and they sent heavier ones.

He names the mechanism himself. This is the Jevons paradox, after the economist who observed in 1865 that making coal-burning engines more efficient raised coal consumption instead of reducing it. Cheaper coal made uses viable that had not been viable at the old price, and the new demand swamped the saving. dylan's version of it is three sentences long: "Cheaper tokens. More tokens. Same coal as 1865."

Which lands hardest on anyone building on a model they do not own. dylan states it as a problem that did not exist twelve months ago: a company running an AI product on top of someone else's closed API watches its gross margins get worse as its customers get more value out of the product, because the more they use it, the more compute the company pays for. Usage and cost move in the same direction, and the company in the middle absorbs the gap.

Almost nobody is shocked by the price of a token. They are shocked by two other things: which meter was running while they worked, and how many tokens the work turned out to need. Those are separate failures, and they get treated as one.

Start with the meter. Marty Kausas, who runs a software company, described a bill about to go from $400K to $1.4M a year. Usage had not exploded. His company was about to cross 150 seats, past which the tier changes: seats stop including any usage, and every token bills at standard API rates. At his run rate he called that 3.5 times overnight. The work stayed the same. The meter changed underneath it.

What changed	What it cost	Described by
Crossed 150 seats into the enterprise tier	$400K to $1.4M a year	Marty Kausas
Signed in to a subscription, billed against API credits	$20 in under an hour	One developer
Retry loop on a single error	$20 and 40 apologies	Param
No view of what the agent was spending on	$200 or more a day	An operator Om Patel described

It happens one person at a time. Kausas also admits he accidentally spent $4,000 in three days inside a coding agent. One developer was signed in to a subscription account while their requests were quietly drawing down API credits, and went through $20 of tokens in under an hour. Another developer puts the gap at its widest: roughly $400 a month on a subscription, against an estimated $20,000 a month for the same work billed through the API. Same person, same tokens, fifty times apart depending on which door the request went through. Kausas concludes that visibility has to come first, because people see their own number and they are shocked.

The second failure is the count. Param spent $20 on tokens, in his words, "just to have it apologize for the same error 40 times". A retry loop is a multiplier on everything the run costs, and nothing in the loop asks whether attempt 12 is going better than attempt 11. Nothing about that spend shows up as a pricing decision. It shows up as a bad afternoon.

Om Patel described an operator spending $200 or more a day on a coding agent with no visibility into what was eating the tokens, and building a dashboard over the session transcripts to find out. The bill is legible. The behaviour behind it is not.

One engineer puts the accounting where it belongs. On one job the token cost came to $1.61. The real cost was 120 seconds of latency assembling the tool graph, three retries to get the data back, and a designer on $150 an hour waiting through it. "The $1.61 is the noise. The $150/hr is the signal."

The obvious reaction to a bill like that is to buy the hardware and stop paying per token. The trade is real, but it is a trade and not an escape. You put down a meter you can read and pick up one you have to run yourself.

Start with the demo that convinces people. Someone reported an open model running on $8,000 of hardware at faster than API speeds. The engineer Gregor answered from having done exactly that. "Faster than API speeds" is nearly always q=1 latency, he said, meaning one request at a time with nobody queuing behind it. He had run self-hosted inference for transaction categorization. A single request was blazing. Then three concurrent calls arrived and he was hand-tuning batch settings at midnight, wondering why he had left the API. The demo is one user. Production is a queue.

The cheap per-million figures people quote carry the same flaw. Ricci Research points out that those numbers assume near-full GPU utilization, which is exactly the assumption that breaks for anyone with bursty traffic, because idle accelerators cost the same as busy ones. They also observe that the parties publishing break-even maths are often the parties selling the hardware. That is worth holding on to, because the break-even figures that do exist are largely vendor figures. One vendor puts the crossover past roughly 10 million requests a month for a single GPU running a smaller open model.

Then there is the operational tax, which rarely makes it into the spreadsheet:

• Cold start. Aidan Pak calls this one of the largest unsolved problems in inference, because traffic is spiky and can double in a matter of seconds, and warming a replica is far more than pulling a container.
• Cold prefill. Neko Legends puts cold prefill at about 1.5k tokens a second, which leaves a 100,000-token prompt waiting around 70 seconds for its first token.
• Batching. Mayank Singh notes that outputs have wildly different lengths, so under static batching a request that finishes in 20 tokens sits there fully computed, waiting on one that runs to 500, and utilization collapses.
• Running state. dylan calls the KV cache the silent monster of the inference era, and puts a long agent session at tens of gigabytes of state per user, per session. On your own hardware that is memory you are paying for by the hour.

Kirill Grishanin puts the summary plainly: running a local model on your own hardware does not by itself give you a production-ready system. It gives you a model. The system around it is the part that was never on the invoice.

It is also worth checking whether the bill has moved at all. On one vendor's self-hosted coding product, one engineer noticed that repos, secrets and artifacts stay on your own boxes while the prompts still go to the vendor's models. That moved execution, not inference. Self-hosted and "our data never leaves" are two different claims, and that product only made the second one. The inference bill never moved.

John Iosifov, who runs an agent product at Ender Turing, thinks the argument is in the wrong place. "The self-hosting question is wrong," he says. "Everyone is doing the break-even math ... That math is correct. The question is wrong." He has run one agent for 341 days, across more than 2,388 sessions and more than 10,000 API calls, and has never self-hosted. Not because the economics fail, but because his constraint is reliability and iteration speed, not the price of a token. His line for it: "Self-hosting buys cost certainty. It does not buy uptime."

Iosifov's diagnosis is specific. He says the teams burning 40 to 60 percent of revenue on tokens are not overpaying the provider. They are spending too much because their request architecture was never fixed, and caching, batching and prompt compression will cut more cost than a GPU rack will. He also warns that reserved capacity which pencils out today can reprice when the next model generation lands, so a twelve-month commitment can be to hardware two generations behind by month nine.

That leaves an order of operations, and buying comes last.

1. See the meter. Several people here found their own number only after it had run.
2. Check which meter you are on. A seat tier, a subscription and API rates are three bills for the same work.
3. Stop the loops. Retries and thinking tokens multiply everything downstream of them.
4. Fix the shape of the request. Caching and batching come before anything with a serial number.
5. Only then, and only if the volume is steady and large, look at owning the hardware. Bear in mind that the published crossover points come from the people selling the hardware.

A vendor selling routing, failover and caching tooling claims it took one client from $40,000 to $16,000 a month, a vendor's claim about its own product, pointing where Iosifov points.

Plenty of teams never see the meter at all. They buy a product that has a model inside it, and the token cost sits folded into somebody else's subscription. That is convenient, and it is a real choice. It also means the ceiling on what the product can do, and the margin it is run at, belong to someone else. Knowing what inference is, is what lets you tell the difference between paying for a model and paying for the wrapper around it.