How to Cut an Enterprise LLM Inference Bill in 90 Days

Many enterprise LLM programmes follow the same arc. A handful of assistants launch, every request goes straight to a frontier model API, nobody touches the serving stack after launch, and within a year inference has become one of the fastest-growing infrastructure lines in the budget. Then the finance team asks for the bill to be halved without anything users can feel.

This article is a worked example of how we would approach that brief over roughly 90 days. The scenario and its numbers are illustrative, chosen to show how each lever works; they do not describe a specific client engagement. Every optimisation described here applies to any enterprise running large-scale LLM inference in 2026.

The starting picture

Picture three workloads on one shared multi-tenant serving cluster: two customer-facing assistants and an internal merchandising copilot. All requests are routed to a single frontier model regardless of task. No caching layer. No routing. No batching beyond what the vendor gateway did on its own. A long system prompt is reloaded on every call, and completions are a few hundred tokens. Volume is in the tens of millions of requests a month and climbing.

The usual internal narrative is that the workload is as cheap as it can be because the team is "already using the API." That is a very common misreading in 2026 estates. Sitting on a vendor API removes the infrastructure work but does nothing to remove the optimisation work. Every lever we describe below applies whether you are self-hosting on H100s or paying per token to a hyperscaler.

Weeks 1 to 3: The audit and the boring wins

Most of the savings in a programme like this usually come from three unglamorous engineering moves that an audit can identify in the first fortnight. None of them requires a model change and none of them changes the user experience.

• Prompt caching for the system prompt. A system prompt that is identical across every call is the easiest saving available. Enabling the vendor's prompt caching feature cuts input token cost on that prefix sharply. Implementation is typically one flag and a cache-key discipline in the client library.
• Semantic caching for high-frequency queries. Analysing historical requests usually shows that a meaningful share of user prompts are near-duplicates of recent ones. A small semantic cache in front of the model, keyed on embedding similarity above a tuned threshold with a stale-check pass, returns those hits in milliseconds at effectively zero cost.
• Prompt compression on the merchandising copilot. Internal copilots often pass entire product catalogues into context on every turn. Replacing the raw dump with a two-stage retrieval pipeline that fetches only the most relevant SKUs per turn cuts input tokens dramatically, and users notice nothing except faster responses.

By the end of week three the run rate should already be visibly lower, which buys the platform team credibility with finance. This pattern is common in LLM cost work. The largest single savings come from disciplines that mature web engineering has had for twenty years — cache what does not change, do not send what you do not need, and measure before you assume.

Weeks 4 to 7: Routing and model tiering

The second wave of savings comes from admitting that a single frontier model is the wrong tool for much of the traffic. Spend two weeks classifying requests into intent buckets by running a lightweight labelling pass over historical traffic, for example:

• Simple lookup and formatting. Often the largest bucket. A well-tuned small model can usually handle these at parity on internal evals for a fraction of the cost per request.
• Standard product Q&A. A mid-tier model may handle these with a small eval drop that the business has to review and accept explicitly.
• Complex reasoning and multi-turn. Kept on the frontier model. No change.
• High-stakes or compliance-sensitive. Kept on the frontier model with an additional verification pass.

The router itself is a small classifier deployed in front of the serving cluster, retrained weekly on labelled production traffic. Its own inference cost is negligible relative to what it saves. Critically, routing should ship behind a feature flag with an A/B test against the all-frontier baseline, confirming no measurable impact on downstream conversion, resolution rate or CSAT before it becomes the default. This step alone is the one most enterprises skip because "the model is fine" — and that instinct is the single largest source of preventable LLM overspend we see in the field.

Weeks 8 to 12: Serving-tier and long-tail optimisations

The final wave is engineering-heavy and delivers the last increment of savings. This is the phase most teams never get to because they run out of runway before they reach it.

• Speculative decoding on the self-hosted tier. For workloads that can be self-hosted, vLLM with a small draft model doing speculative decoding raises effective throughput per GPU, and GPU-hour cost falls correspondingly.
• KV cache reuse across turns. Multi-turn sessions often re-encode conversation history from scratch on every turn. Session-aware KV cache reuse cuts per-turn input token cost on those workloads substantially.
• Batch backfill for asynchronous work. Internal batch processing that tolerates an SLA of hours belongs on the vendor's discounted batch tier, freeing real-time capacity for the workloads that actually need it.
• Guardrails and cost-per-tenant metering. A per-tenant token budget with hard caps and soft alerts does not deliver savings directly. It stops a repeat of the runaway spend that started the conversation in the first place. Every enterprise LLM programme in 2026 needs this and almost none of them have it.

How to measure it

Track four numbers from day one: annualised inference spend, median and P95 request latency, model output quality on an internal eval suite for each intent bucket, and the time to a first savings dashboard. If quality holds within an agreed tolerance on every bucket while spend and latency fall, the programme is working.

The takeaway

The pattern repeats. Enterprise LLM programmes overspend because they treat the first working version as the final version and because the vendor invoice is a monthly abstraction rather than a per-request lever. The biggest single move is deciding that inference cost is an engineering discipline in its own right, and that it deserves a named owner, weekly review, and a real optimisation roadmap. The tooling to do this is mature in 2026. What is scarce is the willingness to spend two engineers for a quarter on it.

If your LLM inference bill is running above six figures a month and you have not touched the serving stack since the workload shipped, there is very likely significant room to compress it with the disciplines described above. If you would like a second pair of eyes on where the leverage lives in your own estate, that is the conversation we most often start.