The Real Cost of AI Agents Isn't the Tokens

Mon Oct 12 2026

Token prices have been falling for three years. Agent bills have not. If you run agents for real work, you already know these two facts don't contradict each other. This summer, three pieces I read, plus a paper from Stanford, explained why better than I could, and they all point at the same place: the price per token is the wrong number to optimize. The cost is in the path the agent takes, and in what it chooses to carry with it.

Cheap tokens, long trajectories

The Azure Architecture Blog piece Token Economics in Practice opens with the number everyone quotes: Stanford's AI Index measured a roughly 280-fold drop in the cost of GPT-3.5-level inference between late 2022 and late 2024. Then it names the trap. Cheaper inference doesn't make agents cheaper. It makes them longer: bigger context windows, repeated tool schemas on every call, retries, reflection loops, subagent fan-out.

The Stanford study it leans on, How Do AI Agents Spend Your Money?, is worth reading on its own. Across eight frontier models on SWE-bench Verified, agentic tasks consumed about 1,000x more tokens than code chat or code reasoning. Most of that cost is input, not output: the agent re-reading what it already has. And the part that should worry anyone with a budget: running the same agent on the same task produced costs that varied by up to 30x between runs. The models also predicted their own token usage poorly, and tended to underestimate it.

When one logical task can cost thirty times more depending on the path the agent happens to take, optimizing the average price per token makes the wrong system look efficient. The Microsoft post's answer is to change the unit: cost per accepted task, expected cost divided by the probability the output actually passes. A policy that halves the token bill but drops acceptance from 95% to 70% is more expensive per accepted task. Only that ratio makes it visible.

This matches what I see in our own agent-driven pipeline, where every task leaves a trace of how many iterations it took and what the review found. Per-token price has never been the number that moved. Iteration count is. A task that takes six round-trips instead of two costs three times more, and no model-pricing page tells you that in advance.

I've watched the same mistake play out with cloud costs during migrations. Nobody budgets the price per gigabyte wrong. They budget the chatty service-to-service calls and the retry storms wrong, because those depend on behavior, not on a price list. Agent cost is becoming the same discipline: you don't optimize the price of a call, you optimize the number and size of calls, and you accept that the number is a distribution, not an estimate.

The orchestrator's tax

Martin Fowler's site published The Orchestrator's Tax by Rahul Garg, and it's the most honest piece of multi-agent writing I've read this year. Garg was running four subagents against a .NET refactor in Claude Code when the session "started to feel harder to reason about than the code itself." So he stopped and asked the orchestrator to critique its own delegation.

The biggest cost didn't come from the four subagents. It came from the orchestrator checking on them. A throwaway prompt, "check on the agents", pulled the full raw transcript of a background agent, tens of thousands of tokens of intermediate reasoning and tool output, into the main thread. Then it happened again for a second status check. And that transcript didn't go away when the call finished. Every later turn carried it forward.

That's the distinction the article is built on, and it's the one I'd want every team running agents to internalize:

Tokens are spent once. Context shapes every decision that follows.

Two panels. Left: a single-threaded agent whose context stack grows every turn and keeps a large transcript dump from turn three onward. Right: an orchestrator whose context holds only the task and short summaries, while two subagents hold the files read, test output and failed attempts, and are discarded.
What the orchestrator carries forward. Left: one thread pays rent on the dump every turn. Right: noise stays in disposable workers; only summaries come back.

A token bill is one-time. Pollution left in context is a tax on every turn after it, and a bigger context window doesn't fix that. It just gives the noise more room to pile up before anyone notices. Garg's conclusion is that subagents get justified for the wrong reason. Teams pitch them as parallel execution. Their real value is what they keep out of the orchestrator's working memory: exploration, repeated file reads, failed approaches, noisy intermediate output. The orchestrator is the only part of the system that accumulates understanding across a long session. Everything else should be disposable.

He also names a second failure that I recognize from a different domain. Two of his four subagents worked in the same area of the pipeline, and each had to rebuild the same mental model of the code before starting. The work had been partitioned by task when it should have been partitioned by the knowledge each task required. He calls it cognitive locality: tasks that need the same mental model should stay together.

That's a service boundary rule. Every monolith-to-microservices migration I've been part of eventually learned that you split by what a team needs to hold in its head, not by what runs in parallel. Split for the wrong reason and you get two services that must both understand the same domain, a network hop, and a deployment pipeline to babysit. Garg's subagents paid the exact same price: two workers independently reconstructing one understanding, and an orchestrator stuck reconciling both.

Where I'd push back a little is on the fix. His CLAUDE.md rules are good ones: two to four agents per wave, don't poll for status when you already know the answer, don't fetch a transcript to answer a lightweight question, treat overlapping file ownership as a signal to merge, not to spawn. But his own caveat matters more than the rules: the ranking of what cost the most was the orchestrator grading its own mistake, with no per-call token accounting. The transcript dumps were real. Which cost was largest is a hypothesis. The tooling to measure this doesn't exist yet in most setups, and until it does, every orchestration rule is calibrated by feel, against one model, on one kind of work. He says as much. I'd be suspicious of any write-up that didn't.

Budgets are a people problem before they're a billing problem

The third piece, Mastering GitHub Copilot Budgets, is a plumbing guide, and the plumbing is instructive. Each Copilot Business or Enterprise seat adds credits to a shared pool. When the pool runs dry, overage is metered at $0.01 per credit, but only if the enterprise has explicitly enabled paid usage. User-level budgets are hard stops. Group and enterprise budgets are not: by default they keep charging past the limit unless an admin toggles "stop usage when budget limit is reached". Whichever budget has the least headroom wins. And when a developer gets blocked, chat, agents and reasoning models stop, while plain inline completions keep working.

The detail I'd underline is the default. The caps that protect the organization are off until someone turns them on. That's the same shape as every cloud-cost incident I've seen: the control existed, nobody owned it, and the first time anyone looked at the number was when finance asked about it. The article's advice to set a universal per-user baseline first, then refine per cost center, is right. What it can't say is that budgets fail as policy documents and work as habits. Someone has to own the number weekly, and an engineer burning far more than the team median should be able to explain why, as visibility, not as punishment. The same rule I'd apply to a cloud spend anomaly applies here. The tooling is the easy part.

What I'd actually track

Put the three pieces and the paper together and the message is consistent. Agent design decisions are now cost decisions, and cost decisions are now architecture decisions. That used to be true only at the infrastructure layer, in how you shard a database or size a cluster. Now it's true at the prompt and orchestration layer too. If I had to reduce it to what a team should measure, it's three things:

  1. Cost per accepted task, not per token. Count the retries and the rejected outputs in the denominator. If you can't tell which runs were accepted, you don't know your unit cost.
  2. Iterations per task, as a distribution. The Stanford 30x variance means the average lies. Track the P95, and treat a task whose cost spread is widening as a design smell, not a pricing problem.
  3. What the orchestrator carries forward. Not total context size, but what's in it. If a status check or a transcript dump lands in the main thread, that's debt you pay on every later turn. Delegate when it removes a concern from the orchestrator's head, not by default.

Treat your agent graph with the same scrutiny you'd give a service boundary, and most of the "surprise" bills stop being surprises. The tokens were never the cost. The path was.

References