Why Your AI Bill Is Really a Context Bill

The Hidden Cost of Enterprise AI Is Context, Not Compute

Ask a finance leader’s favorite question of any modern AI assistant.

Fluent. Confident. Specific enough to paste straight into a board pack. And wrong.

Every correction was a business rule the model should have known and didn’t. It never asked. It answered anyway, confidently, on whatever default it happened to assume. By the time the numbers can be trusted, the interaction has burned five, ten, twenty times more tokens than the final answer required — not to mention the time. She did not prompt badly. She spent the entire exchange teaching the model her business.

This is the hidden cost of enterprise AI. The answer is cheap. The conversation is expensive.

Every inference has two costs. The tokens in the answer, and the tokens required to establish the context the model needs to produce that answer — the latter of which is increasing exponentially.

The evidence is stacking up. In June 2026, UBS reported that roughly 60% of the enterprises it surveyed are throttling AI spend. One user ran up $35,000 in a single month, and whole teams routinely blew past their token quotas by 100 to 200 percent.

Why are enterprise AI token costs rising?

Gartner called it out a month earlier. Neglect semantics, it warned, and AI agents turn inaccurate, inefficient, and expensive. By 2027 it predicts the reverse — organizations that prioritize semantics in AI-ready data will lift agentic AI accuracy by up to 80% and cut costs by up to 60%.

“Context with semantic coherence will become a cost-control and trust strategy, not a nice-to-have.” — Gartner Press Release, “Gartner Says Lack of Semantics Causes Inaccurate AI Agents and Wasted Spending,” 11 May 2026, gartner.com/en/newsroom/press-releases.

Your users are not prompting badly. They are compensating.

Words matter. Unfortunately, there are plenty of overlapping definitions for ordinary words across businesses. Revenue, customer, active employee, margin, churn, region, forecast, supplier risk, close status. A public model knows the general meaning of those words. It does not know how your company defines them, governs them, or changes them after a reorg. So users compensate. Every clarification our finance leader typed — contribution not gross, our EMEA not the legal rollup — is a business rule the model should have inherited before she asked. Instead, she has to type it in.

Making the context window larger does not fix this. It just gives the model more text to search through. Published research on how models actually use long context (Liu et al., “Lost in the Middle,” TACL 2024) showed that even capable models struggle when the relevant fact is buried mid-prompt. Frontier models have improved since, but the structural inefficiency remains. Meaning stuffed into a prompt is not meaning the system understands. It is just more text to parse.

A bigger context window is not business understanding. It is just more text to search through.

Keeping the chat open does not solve this. It hides it. Return to a long-running conversation tomorrow and the model appears to remember your definitions — but there is no memory, only replay. Every new message resends the entire history, clarifications included, so today’s question now carries yesterday’s corrections as overhead. The bill keeps growing even as the friction seems to disappear.

And the fix is personal. It works for you, in that one chat, until you close it. It does not reach the colleague in Berlin asking the same question in a fresh window, or the agent that spins up a new session at 3am with no history to inherit. Chat context is personal, temporary, and non-transferable. A semantic layer is organizational, persistent, and shared.

The context that only lives in your chat window is the context everyone else still has to rebuild from scratch, and the enterprise pays to rebuild it every time.

Agents make this problem worse, not better.

Everything so far assumed a human in the loop. Our finance leader put up with five clarifying turns before she got a usable number; an agent has no such patience, and grants none. Agentic workflows make dozens of model calls per task, and each one needs context. Rebuild that context in the prompt every time, and the enterprise pays for the same business definitions on every step of every run.

Gartner flags this directly, warning that agents need context “in each step of the agentic workflow to deliver accurate responses at an optimal cost.” It also projects the share of AI spending on data readiness will grow sevenfold from 2025 to 2029. And this is still the shallow end. UBS’s Evidence Lab found only 8% of roughly 130 companies have deployed agents at scale, so as agents reach production every ungrounded workflow becomes a repeating cost.

Every agent step without semantics is a step the enterprise pays for twice. Once in tokens, once in errors.

But what about prompt caching and RAG?

Every CIO I talk to raises these same two objections. Both deserve a straight answer. Prompt caching reduces the price of tokens that repeat, but it never establishes meaning. A cached wrong assumption is still wrong, just cheaper.

Retrieval-augmented generation retrieves documents. Semantics governs meaning. RAG can pull the policy PDF into the prompt, but it cannot tell the model that this policy replaced that policy after the January reorg, or that the EMEA rollup in your legal system does not match the EMEA definition your CFO reports on. Retrieval finds text. Semantics establishes what the text means in your business.

Retrieval finds text. Semantics establishes meaning. They are not the same layer.

Why pilots work and production stalls.

When AI cannot understand the business, users compensate in expensive ways. They ask follow-ups. They paste background. They correct assumptions. They rerun the request with different wording. They validate outputs manually. Or they give up and go back to the analyst team.

The cost is inference spend plus user time, analyst time, rework, governance risk, and stalled adoption.

Scale it up and this is the enterprise AI story of the last two years. In pilots, a small team handcurates the context. They pick a narrow use case, clean the data, define the terms, and guide the model. It works. In production, that manual context-setting has to spread across finance, HR, supply chain, sales, and operations. Nobody has the bandwidth.

McKinsey’s State of AI in 2025 survey put numbers on the gap. 88% of organizations regularly use AI. Nearly two-thirds have not begun scaling it across the enterprise. Only about 6% qualify as high performers with material EBIT (earnings before interest and taxes) impact. The high performers are the ones redesigning workflows around AI, not bolting AI onto existing ones.

In pilots, humans supply the context. In production, nobody can.

Semantic compression: what good looks like.

Enterprise AI needs a trusted context layer that gives the model business meaning before the user asks. In practice that takes several things working together:

• A governed data foundation, so the AI is grounded in the operational truth of the business, not whatever fragment lands in the prompt.
• Semantic definitions and hierarchies, centrally maintained, so terms like margin or region or active customer mean the same thing to every user and every agent.
• Workflow context, so the AI understands the process the data lives inside, not just the data.
• Lineage and access controls it inherits automatically, so nothing leaks and every answer traces back to source.
• And all of it must be available in the flow of work, not stranded in a separate analytics tool.

This is the critical shift. A semantic layer acts as compression for enterprise AI interactions. Instead of every business rule, exception, and definition being retyped into the prompt, the AI system inherits the right context automatically. Fewer clarifying turns. Fewer failed turns. Shorter interactions. Lower bills. And smarter answers in the moments that matter.

Business semantics are token compression. They replace repeated explanation with reusable meaning.

Most enterprises underestimate this part. Building the layer from scratch is not a weekend project. The data lives in dozens of systems, the definitions in analyst heads, the workflow logic in ERP configurations built up over a decade. Assembling it by hand, per use case, per team, is the very problem that made pilots work and production stall. But most of that context already exists. The systems that already run the business — ERP, finance, HR, supply chain — have encoded these definitions, hierarchies, and rules for years. The organizations that move fastest connect AI to that existing context instead of rebuilding it from nothing.

What should leaders do about AI context costs?

If AI usage is growing and token costs are rising, rationing usage is the wrong answer. The better question is why every interaction requires so much context in the first place. Four places to start:

• Which business terms do users keep redefining in prompts?
• Which AI use cases require the most follow-up clarification?
• Where do users distrust the first answer?
• Which workflows need AI to know your business rules before it can act?

Those are the domains where a semantic foundation will pay back fastest.

This is the problem Oracle AI Data Platform was built to solve. It grounds AI in trusted enterprise data, deep business context, and the flow of work, so users and agents inherit meaning instead of rebuilding it in every prompt. Fewer clarifying turns, fewer wrong answers, and an economic model for enterprise AI that actually scales.

The next AI cost frontier is meaning.

The first wave of AI cost management focused on model selection, caching, routing, and usage limits. Those matter. The next frontier is semantic efficiency.

Enterprises will not control AI economics by making people ask fewer questions. They will control it by making every answer arrive right the first time. Our finance leader asks why EMEA margin fell. This time the system already knows which margin she means, which EMEA, which forecast, and answers in a single clean pass. Nothing to clarify. Nothing to correct. An answer she can take straight into the board meeting.

The cheapest token is the one she never had to type.

For more information:

• Why enterprise AI needs deep business semantics — the deeper argument
• Oracle AI Data Platform — platform overview