Is Your AI Getting More Expensive for the Right Reasons?

Your compute costs can tell you a lot about the system behind it.

 

By Jeff Crate
Senior Forward Deployed Engineer, Foxtrot

As AI moves from experimentation into day-to-day operations, spend often rises for a good reason. More people are using it, and they’re using it across more of the business.

A workflow that began with one team gets picked up by another. A manual task gets automated. New use cases are added, more data moves through the system, and work that once happened occasionally starts running all day.

Some of that increase is exactly what you want to see. If AI is doing more valuable work across the business, higher spend may simply reflect higher adoption. The harder question is whether the economics are improving as the system scales.

When costs start climbing, the instinct is often to look for places to cut. We think it’s worth understanding what’s driving the increase first.

 

You can find meaningful savings earlier than you might expect

We saw this firsthand with a construction technology company.

Before the broader architecture was fully deployed, our team analyzed the company’s existing data infrastructure and reduced compute costs by 90%.

What matters just as much as the savings is how early they showed up. The company didn’t need to wait for a major redesign or a fully scaled AI environment to uncover them. There was already enough happening inside the existing infrastructure to materially improve the economics.

That’s one reason we think cost deserves attention earlier than it often gets. By the time AI is widely deployed and the expense becomes uncomfortable, inefficient choices may already be embedded in workflows the business relies on.

The broader engagement matters too. Cost efficiency sat alongside work to improve equipment reliability and build the technology needed for where the company wanted to go next. The goal was to create an operating foundation that could support the business as it grew.

Cost was part of that conversation from the beginning.

 

 
What looks small in a pilot can get expensive at scale

Pilots are forgiving.

An extra query may barely register when a workflow runs twenty times. Carrying more context than necessary may not seem consequential. Using a highly capable model for a relatively simple job may be perfectly manageable.

Repeat those choices thousands of times a day and the economics change.

Agents add another wrinkle. A single request may kick off several model calls, data queries, tools and actions before the user ever sees the result. All of that activity may be justified. The question is whether the work happening behind the scenes is necessary for the outcome you need.

That’s also why token usage rarely tells the whole story.

The model is only one part of what’s happening. Data may need to be queried and transformed. Logic runs. Context moves through the workflow. Other systems may be called along the way.

The more useful question is what the entire workflow costs to run and what the business gets in return.

 

Follow the cost back to the work

When compute spend starts moving, we want to understand what changed.

Did adoption genuinely grow? Did one workflow become much more active? Is the system repeatedly processing information it could reuse? Is it carrying more context than the task requires? Is the model appropriate for the work it’s doing? Can the team trace the spend back to the workflows responsible for it?

Those questions sound straightforward. Across a large enterprise environment, the answers can get complicated quickly.

That’s why our work around AI economics looks at the whole workflow. Depending on what the system is doing, the opportunity may sit in context management, caching, model routing, workload design, repeated processing or the way costs are attributed back to individual workflows.

There isn’t one cheapest architecture for AI because there isn’t one kind of AI workload.

A high-stakes decision that happens a few hundred times a month may justify a very different setup from a repetitive task running thousands of times an hour. A customer-facing workflow may have speed and reliability requirements that an overnight process doesn’t. Some work warrants the most capable model available. Other work may perform just as well with something lighter.

The work should determine the economics.

 

Cost management and agent-level ROI

Spend visibility. As AI adoption scales, companies need visibility into the full cost of agent execution and whether that spend translates into measurable business value.

Our AI Spend Optimization accelerator targets inefficiencies across data processing, context management, caching, model routing and workflow execution. The construction technology example shows how much value can sometimes be uncovered in infrastructure that already exists. As agentic systems scale, that same discipline can extend to the economics of individual agents and workflows.

Agent-level attribution. We can also build an agent-level economics layer using execution traces, token metering and infrastructure telemetry to attribute model, compute, retrieval, tool and human-review costs to individual agents and workflows. Connecting that telemetry to business outcome data makes it possible to measure cost per successful task, intervention rates and net value against an agreed baseline.

Shared-cost allocation. Multi-agent workflows add another layer of complexity. Several agents may share infrastructure, tools or contribute to the same outcome, so understanding ROI can require shared-cost allocation and controls to prevent the same value from being counted more than once.

The value is clearer agent-level economics. Teams can see which agents are creating enough value to scale, which need to be optimized and which may no longer justify their cost, while still maintaining quality, latency and reliability.

 

Good AI economics are about spending appropriately for the job

Using fewer tokens can absolutely lower costs. It can also create a poor trade if quality drops, people have to redo the work, or a workflow needs additional retries and human intervention to finish successfully.

There are times when spending more makes sense too.

A simpler task may be well served by a faster, less expensive model. Complex reasoning or a high-value decision may warrant heavier compute. In a higher-risk workflow, an additional model check may justify the extra token spend if it meaningfully improves quality or safety.

This is why optimizing consumption in isolation only gets you so far.

A more useful measure is the cost of completing the work successfully.

What does it cost to run the workflow from beginning to end? How often does someone need to step in? What happens to that cost as volume grows?

Those questions move the conversation much closer to business value.

A lower-cost model can still produce poor economics if it needs several attempts to get to the right answer. A more expensive model may be worth the additional spend when the work is complex or consequential. More context can improve a result when the information matters. Carrying unnecessary context through every interaction simply creates more work.

That’s why we increasingly think about cost at the level of the task, workflow or decision. It gives teams a much clearer way to judge whether AI is becoming more economical as it scales.

 

Flexibility has economic value too

Model choice is another part of the equation.

A model that makes sense for a workflow today may not be the best option a year from now. Another model may become faster, more capable or more economical for the job.

Companies should be able to take advantage of that without rebuilding the business logic, workflows and context around the model.

That flexibility is part of AI sovereignty. Our approach separates enterprise knowledge, workflow logic, evaluation criteria and orchestration from the underlying model so organizations can evaluate, switch or combine models based on what the work requires, including performance, security, latency, regulatory requirements and cost.

For AI economics, the implication is simple. Flexibility gives companies more choices as the market and the economics change.

 

Know what you’re paying for

The healthiest AI environment won’t always have the lowest compute bill.

A company putting AI to work across meaningful parts of its business may spend considerably more than one running a handful of pilots. That can be a very good trade.

What matters is being able to explain why.

Teams should know which workflows are driving spend, what those workflows cost to complete successfully and whether the value they’re creating justifies the expense. Some growth will reflect healthy adoption. Other growth may point to unnecessary work or an architecture that needs to evolve.

The companies that scale AI well will know the difference.

As AI becomes more embedded in the business, its economics should become easier to understand and manage.

If the system is becoming more useful, you should be able to explain what you’re paying for and why.