Case Study: Cutting a Fortune 500 Retailer's LLM Inference Bill by 62% in 90 Days

In February a Fortune 500 speciality retailer asked us to look at an LLM programme that had quietly become their fastest-growing infrastructure line item. Two customer-facing assistants and one internal merchandising copilot were on track to burn four point one million dollars in inference spend for the year, up from a budgeted six hundred thousand. Every request was a plain call to a frontier model API. Nobody had touched the serving stack since launch nine months earlier. The CFO's patience was running out and the platform team had a two-quarter roadmap to cut the bill in half without breaking anything users could feel.

Ninety days later the annualised run rate was one point six million. The user-visible latency was better than the starting point. The model was the same frontier model. This is the walkthrough of what we changed, in the order we changed it, with the numbers each move delivered. Client identifying details are anonymised. Every optimisation described here is generalisable to any enterprise running large-scale LLM inference in 2026.

The starting picture

Three workloads on one shared multi-tenant serving cluster. All requests routed to a single frontier model regardless of task. No caching layer. No routing. No batching beyond what the vendor gateway did on its own. Median prompt was 2,400 tokens including a 1,800-token system prompt reloaded on every call. Median completion was 380 tokens. Volume was seventy-two million requests a month and climbing eight percent month over month. Cost per request landed at four point seven cents.

The internal narrative was that the workload was as cheap as it could be because they were "already using the API." That is a very common misreading in 2026 estates. Sitting on a vendor API removes the infrastructure work but does nothing to remove the optimisation work. Every lever we describe below applies whether you are self-hosting on H100s or paying per token to a hyperscaler.

Weeks 1 to 3: The audit and the boring wins

Ninety percent of the eventual savings came from three unglamorous engineering moves we identified in the first fortnight. None of them required a model change and none of them changed the user experience.

• Prompt caching for the system prompt. The 1,800-token system prompt was identical across every call. Enabling the vendor's prompt caching feature dropped input token cost on that prefix by eighty-five percent. Implementation was one flag and a cache-key discipline in the client library. Savings alone: 22 percent of total spend.
• Semantic caching for high-frequency queries. Analysis of one million historical requests showed that eighteen percent of user prompts were near-duplicates of a request served in the previous seven days. We deployed a small semantic cache in front of the model, keyed on embedding similarity above a tuned threshold, with a stale-check pass. Cache hits returned in 40ms instead of 1.2s and cost effectively zero. Savings: another 14 percent.
• Prompt compression on the merchandising copilot. The internal copilot was passing entire product catalogues into context on every turn. We replaced the raw catalogue dump with a two-stage retrieval pipeline that fetched only the twenty most relevant SKUs per turn. Input tokens on that workload fell 78 percent. Users noticed nothing except that responses were faster. Savings: 11 percent.

By the end of week three the annualised run rate was down to two point six million and the platform team had bought back credibility with the CFO. This is the pattern on every LLM cost engagement we run. The largest single savings come from disciplines that mature web engineering has had for twenty years — cache what does not change, do not send what you do not need, and measure before you assume.

Weeks 4 to 7: Routing and model tiering

The second wave of savings came from admitting that a single frontier model was the wrong tool for at least half the traffic. We spent two weeks classifying every request into one of four intent buckets by running a lightweight labelling pass over historical traffic:

• Simple lookup and formatting. Roughly 38 percent of requests. A well-tuned small model handled these at parity on internal evals for one twentieth of the cost per request.
• Standard product Q&A. 34 percent. A mid-tier model handled these with a two point four point drop on the eval suite the client accepted after review.
• Complex reasoning and multi-turn. 22 percent. Kept on the frontier model. No change.
• High-stakes or compliance-sensitive. 6 percent. Kept on the frontier model with an additional verification pass.

The router itself is a small classifier deployed in front of the serving cluster, retrained weekly on labelled production traffic. Its own inference cost is negligible relative to what it saves. Post-routing, the effective blended cost per request dropped a further 19 percent. Critically we shipped this behind a feature flag and did a three-week A/B against the all-frontier baseline to confirm no measurable impact on downstream conversion, resolution rate, or CSAT. There was none. This step alone is the one most enterprises skip because "the model is fine" — and that instinct is the single largest source of preventable LLM overspend we see in the field.

Weeks 8 to 12: Serving-tier and long-tail optimisations

The final wave was engineering-heavy and delivered the last twelve to fifteen percent of savings. This is the phase most teams never get to because they run out of runway before they reach it.

• Speculative decoding on the self-hosted tier. For the two internal workloads we could self-host, we deployed vLLM with a small draft model doing speculative decoding for the frontier target model. Effective throughput per GPU rose by 1.9x. GPU-hour cost dropped correspondingly.
• KV cache reuse across turns. Multi-turn sessions were re-encoding conversation history from scratch on every turn. Enabling session-aware KV cache reuse cut per-turn input token cost on multi-turn workloads by roughly sixty percent.
• Batch backfill for asynchronous work. Roughly nine percent of total traffic was internal batch processing tolerant of a two-hour SLA. We moved it to the vendor's batch tier at a fifty percent discount, freeing real-time capacity for the workloads that actually needed it.
• Guardrails and cost-per-tenant metering. We added a per-tenant token budget with hard caps and soft alerts. This did not deliver savings directly. It stopped a repeat of the six-week runaway that had produced the initial CFO conversation in the first place. Every enterprise LLM programme in 2026 needs this and almost none of them have it.

The final scorecard

Annualised inference spend: $4.1M → $1.6M (down 62%). Median request latency: 1.4s → 890ms (better, driven by cache hits and small-model routing). Model output quality on the internal eval suite: baseline held on all four buckets within accepted tolerance. Time to first savings dashboard: 11 days. Time to full deployment: 87 days. Engineering effort: three of our engineers plus two on the client side, part-time.

What we take away for every engagement

The pattern repeats. Enterprise LLM programmes overspend because they treat the first working version as the final version and because the vendor invoice is a monthly abstraction rather than a per-request lever. The biggest single move on every engagement is deciding that inference cost is an engineering discipline in its own right, and that it deserves a named owner, weekly review, and a real optimisation roadmap. The tooling to do this is mature in 2026. What is scarce is the willingness to spend two engineers for a quarter on it.

If your LLM inference bill is running above six figures a month and you have not touched the serving stack since the workload shipped, the number will almost certainly compress by half in ninety days with the disciplines described above. If you would like a second pair of eyes on where the leverage lives in your own estate, that is the conversation we most often start.