Why Enterprise AI Bills Go Up When Per-Token Costs Go Down
While per-token prices have collapsed, companies are spending more than ever on enterprise AI. This is because metered inference turns efficiency gain into more consumption. Local AI breaks that loop by running inference on hardware you already own. This is the case we make at Icosa for on-prem AI.
Written by: Maddy Higgins
Published: August 5, 2026
The paradox
The cost of a million tokens from a frontier model fell roughly 80% between 2025 and 2026. Over a longer window, the decline is closer to two orders of magnitude. AI got dramatically cheaper in a very short window of time.
However, AI bills went the opposite way. Enterprise LLM API spend passed $8.4 billion in 2025 and is tracking to double again this year. The FinOps Foundation has documented organizations running three times over their 2026 token allocations, and its data shows roughly 85% of organizations misestimating AI costs by more than 10%, with close to a quarter off by 50% or more.
Where the money went
What ended up breaking AI budgets wasn't per-token cost, but the new ways it was being used.
The first is the introduction of agentic workflows. While a chatbot query is one model call, an agentic workflow is ten to twenty, chained together: retrieve, extract, cross-check, draft, critique, revise. Gartner's March 2026 analysis put agentic tasks at 4-30x the token consumption of a standard chatbot interaction. RAG pipelines inflate context windows another 3-5x on top of that. Long-horizon agents run overnight with no natural stopping point, ticking up the meter even further.
Almost every enterprise ROI model was built during the pilot phase, when the workload was single-turn Q&A. Then the same organizations shipped agents to production and discovered the per-task token count had moved an order of magnitude.
This is an example of a larger phenomenon called the Jevons paradox, where improvements in resource efficiency lead to a counterintuitive rise in consumption of that resource. This occurs because as per-unit costs decrease, people find ways of using that resource that weren't economically viable or even conceivable when it was still expensive.
Public examples
- Uber exhausted its entire 2026 AI coding budget in four months. Adoption of AI coding tools across its engineering organization jumped from roughly a third to 84% between December and March, and by the company's own account something like 70% of committed code was originating with AI. Uber's COO acknowledged publicly that token consumption did not appear to track with useful features actually shipped.
- Microsoft rolled an AI coding tool out broadly, watched adoption climb, then pulled the licenses across multiple divisions and redirected engineers to a cheaper internal tool. The tool worked fine. The bill at organic usage levels didn't.
- A major healthcare insurer increased AI token consumption from 3M to 150M in a year.
When the model costs more than the person
Nvidia's VP of applied deep learning, Bryan Catanzaro, said publicly that for his team "the cost of compute is far beyond the costs of the employees."
A Goldman Sachs analysis from May 2026 compared daily agent cost against daily human cost across several job types. For a coding agent, the raw API figure came in around $13 per day against roughly $300 for a software engineer, a 20x advantage on paper. For call-center work, AI and humans landed at near parity, around $90 a day each.
Reported figures from companies running unmetered rollouts put per-engineer AI charges at $500–$2,000 per month on top of salary. MIT CSAIL's analysis of vision-task automation found it economically viable to automate about 23% of roles studied; for the other 77%, humans remained cheaper.
These examples reveal that for many companies, rented AI has a real cost advantage on narrow, medium-judgement tasks. That advantage goes away when it is scaled across an organization, because unpredictable increases in token consumption have huge effects when companies don't own the AI they're using. Local AI systems allow companies to evade this metered pricing system altogether.
One pipeline, priced two ways
Here is a concrete pipeline comparison. Take a common enterprise workload: reviewing an internal document against a policy corpus and producing a memo.
Priced on the Claude API
| Step | Input tokens | Output tokens |
|---|---|---|
| 1. Route and classify | 2,000 | 200 |
| 2. Extract structured fields | 25,000 | 1,500 |
| 3. Cross-check against policy corpus | 30,000 | 2,000 |
| 4. Draft the memo | 12,000 | 3,000 |
| 5. Self-review and revise | 16,000 | 1,300 |
| Total per task | 85,000 | 8,000 |
Using Anthropic's published rates (July 2026):
| Configuration | Rate (in/out per MTok) | Cost per task | Per month | Per year |
|---|---|---|---|---|
| Haiku 4.5 | $1 / $5 | $0.125 | $275 | $3,300 |
| Sonnet 5 (introductory, through Aug 31) | $2 / $10 | $0.250 | $550 | $6,600 |
| Sonnet 5 + 60% prompt caching | $2 / $10 | $0.158 | $348 | $4,176 |
| Sonnet 5 (standard, from Sep 1) | $3 / $15 | $0.375 | $825 | $9,900 |
| Opus 4.8 | $5 / $25 | $0.625 | $1,375 | $16,500 |
Priced on local hardware
Now run the same pipeline on a fine-tuned small model on a workstation the firm already owns, a MacBook Pro-class M4 Max. The only variable cost is electricity.
| Parameter | Value |
|---|---|
| Prefill throughput | ~500 tok/s |
| Generation throughput | ~50 tok/s |
| Compute time per task | 330 s (0.092 h) |
| Incremental power draw under load | ~65 W |
| Energy per task | 5.96 Wh |
| U.S. commercial electricity (EIA, 2026) | ~$0.14/kWh |
| Cost per task | $0.00083 |
| Cost per month (2,200 tasks) | $1.84 |
| Cost per year (26,400 tasks) | $22.02 |
That is ~300x cheaper than Sonnet 5 at introductory pricing and ~750x cheaper than Opus 4.8 for this pipeline, saving over $10,000/yr in variable costs.
Icosa helps companies use AI while keeping their data and workflows on their own devices. Its LM Shop uses a company's documents to create specialized, efficient AI models without requiring code or an in-house machine learning team. Zeno provides a secure macOS workspace where employees can use these models. Icosa's proprietary Combinatorial Reasoning engine, developed with NASA and funded by the NSF, is designed to give these smaller local models reasoning capabilities comparable to leading AI systems.
The companies that overspent in 2026 didn't do anything wrong. They adopted quickly, their engineers used the tools enthusiastically, and their agents worked. The bill was a consequence of the pricing model, not the decisions.
You can respond to that by installing governance, capping usage, and asking engineers to think about cost before every prompt. Or you can move the workload onto hardware you already own, where the meter doesn't run.