What agent inference actually costs in production

Why this matters

Agent pilots are cheap in a way that is actively misleading. Fifty conversations against a small knowledge base costs less than lunch, which is why the budget conversation gets deferred. Then the same agent goes to the full contact center and the bill arrives with a different number of digits, and the team that shipped it spends a quarter explaining the variance instead of building the next thing.

The cost is knowable in advance. Most teams just measure the wrong unit.

Prerequisites

• An agent running against real traffic, even at low volume
• Access to token-level usage: Bedrock CloudWatch metrics, Anthropic Console usage, or Agentforce Einstein usage reporting
• Your actual conversation volume, including the seasonal peak rather than the monthly average

Step 1: Price the conversation, not the request

Cost per API call is the number vendors report and the number that misleads you. The unit that matters is cost per resolved conversation, because a single customer interaction is rarely a single call.

One conversation typically includes a retrieval call, a planning call, one or more tool invocations, a synthesis call, and often a retry. Six to ten model invocations for one customer question is normal. If you budgeted on one, you are off by most of an order of magnitude before traffic even arrives.

Measure it directly: take a hundred real conversations, sum the tokens across every call in each one, and divide. That number times your monthly volume is your bill.

Step 2: Find where the tokens actually go

In nearly every deployment we look at, input tokens dominate and the team is surprised, because attention goes to output length.

The usual distribution:

• System prompt and tool definitions. Sent on every single call. A 4,000-token system prompt on a ten-call conversation is 40,000 input tokens before the customer has said anything of substance.
• Retrieved context. Teams stuff the top twenty chunks in because it improves accuracy on the eval set. It does, and it also triples the bill.
• Conversation history. Resent in full on every turn. A ten-turn conversation resends turn one nine times.
• Output. Usually the smallest share, and the one everyone tries to optimize first.

Step 3: Apply the controls in order of impact

Prompt caching. The single biggest win. Your system prompt and tool definitions are identical on every call, so cache them. This is a configuration change, not an architecture change, and on a long system prompt it cuts input cost substantially. Do this before anything else.

Retrieval discipline. Go from twenty chunks to five and measure accuracy on your eval set. Most deployments lose very little and save a lot. If you do not have an eval set, you cannot tune this safely, which is one more reason to build one.

Model routing. Not every call needs your most capable model. Classification, routing, and extraction run fine on a smaller model. Reserve the frontier model for synthesis and judgment. Route by task, not by customer tier, and validate each route against the eval set.

History compaction. After a threshold, summarize older turns instead of resending them verbatim. Watch quality closely here, since this is the control most likely to degrade multi-turn behavior.

Step 4: Put a ceiling on it

Set a hard per-conversation call cap and a per-tenant daily spend cap, and alert before either is hit rather than after.

The failure this prevents is the loop: an agent that retries, fails, re-plans, and retries again against a malformed record or an unavailable tool. Without a cap, one bad record can generate thousands of calls overnight. This is the single most common cause of a surprise invoice, and it is entirely preventable with a counter.

Step 5: Report cost per resolution, not cost per call

Finance does not care about tokens. They care whether the agent is cheaper than the alternative.

Cost per resolved conversation compared against fully loaded cost per human-handled ticket is the only comparison that settles the argument. Report it monthly alongside deflection rate and satisfaction. An agent that resolves at a third of the human cost is a clear win even if the absolute spend is growing, and that framing is what keeps the program funded through the quarter where the bill goes up.

Gotchas

Retries are invisible until you look. Instrument them separately. A five percent retry rate on a ten-call conversation is not five percent more cost, because retries cluster on the expensive synthesis calls.

Caching has a time window. A cache that expires between calls in a slow conversation gives you none of the benefit and all of the complexity. Check the actual hit rate rather than assuming it.

Peak is not average. Contact centers are spiky. Size the budget and the rate limits against the worst hour, not the monthly mean, or the agent degrades exactly when it matters most.

When this breaks

Two situations where the arithmetic stops working.

Very long context windows, where you are passing an entire document set on every call. At that point retrieval architecture is the cost lever, not prompt tuning, and the fix is a real vector layer rather than a bigger prompt.

Agents with expensive tools, where the model call is rounding error next to the downstream system. If every agent turn triggers a query against a warehouse, the warehouse is your cost center and token optimization is theatre. Profile end to end before choosing where to spend effort.

Where this leaves you

Cost per resolved conversation, measured on real traffic, with caching on and a hard cap in place. That is the whole discipline. It takes a day to set up and it turns the agent budget from a quarterly surprise into a line item you can forecast.