Case Study: Cutting LLM Inference Latency 6x

A Fortune 500 customer-experience platform asked us to look at a generative AI copilot they had launched six months earlier into roughly 4,000 internal support agents. The feature was technically live. It was also, by their own measurement, unusable.

Agents had been asked to use the copilot for ticket summarisation, knowledge-base lookups, and draft responses. They were not using it. The reason was simple and unromantic: at a P95 of 4.8 seconds and a P99 north of nine seconds, the assistant arrived after the agent had already typed the answer themselves. Adoption sat at 11 percent. The internal verdict was that the model was the problem and they needed to upgrade to the next frontier release.

The model was not the problem. Eight weeks later, with no frontier model upgrade, the same workload was running at a P95 of 780 milliseconds and a P99 of 1.4 seconds. Adoption climbed past 60 percent inside the following month. This is what we found, what we changed, and what they had been quietly getting wrong.

What we walked into

The architecture was the standard 2025-era enterprise RAG stack, assembled by a competent team under time pressure. Requests flowed from the agent console to an API gateway, to an orchestration layer running on a managed serverless platform, into a retrieval step against a vector database, then into a frontier-tier LLM via the vendor's API, then back through a post-processing step that ran a separate moderation call. Nothing was unusual. Everything was slow.

The team's instinct had been to look at the model. We started where the time actually was — which, almost without exception in these reviews, is not where teams expect.

Where the 4.8 seconds was going

We instrumented the full call path with distributed tracing and ran two weeks of production traffic through it. The breakdown surprised the team:

• Cold starts on the orchestration layer. The serverless platform was provisioning a fresh container on roughly 18 percent of requests. Each cold start added 1.1 to 1.6 seconds before any meaningful work began.
• Sequential retrieval. Three separate retrieval calls — vector search, a keyword search, and a metadata lookup — ran one after the other. Together they added 900 milliseconds, of which the actual database work was around 280.
• Full-response generation. The frontend was waiting for the model to finish before rendering anything. The model was generating an average of 380 tokens at roughly 55 tokens per second. That is seven seconds of dead air for the agent on long responses.
• Post-processing moderation. A second LLM call to a separate moderation model ran serially after generation, adding another 600 to 900 milliseconds on most requests.
• The model itself. First-token latency from the vendor was 380 milliseconds on average. This was the smallest, not the largest, component of total latency.

The team had been preparing a procurement case for a faster model. That would have reduced one of the smaller components of the latency budget while leaving the larger ones intact.

What we changed

We worked through the call path in roughly the order the time was being spent. Each change was independently measurable; we shipped them in sequence so we could attribute the gains correctly.

1. Killed the cold starts

We moved the orchestration layer off the serverless platform and onto a small autoscaled service running on the existing AI deployment architecture cluster, with minimum replicas pinned to cover off-peak traffic plus headroom. The cost difference was negligible at this scale. The cold-start tail disappeared. P95 dropped by roughly 350 milliseconds immediately.

2. Parallelised retrieval

The three retrieval calls had no dependencies between them. We fanned them out concurrently and merged the results downstream. The retrieval step dropped from 900 milliseconds to roughly 320, dominated now by the slowest of the three rather than the sum.

3. Switched to streaming with incremental rendering

The single highest-impact change. We switched the model call to streaming, and the frontend to render tokens as they arrived with a subtle indicator that generation was still in progress. Time-to-first-useful-token became the perceived latency, not time-to-last-token. Agent-perceived latency on long responses fell from seven seconds to under one second.

This required real frontend work — handling partial markdown, partial code blocks, abort semantics, and the moderation step (more on that below). It is the kind of change that gets deferred because it touches three teams. It is also the change that mattered most.

4. Restructured moderation

The blocking post-generation moderation call was the second-largest opportunity. We split it into two paths: a fast input-side classifier that ran in parallel with retrieval, and a streaming output-side classifier that ran concurrently with generation and could interrupt the stream if a violation was detected. Median moderation overhead dropped from 750 milliseconds to roughly 40, while actually improving the catch rate on prompt-injection attempts because the input-side check now ran on every request rather than only on suspicious ones.

5. Added a semantic cache for the obvious cases

Support work has long tails of near-duplicate questions. We introduced a semantic cache with a tight similarity threshold and short TTL, sitting in front of the model call. It absorbed 31 percent of production traffic at less than 60 milliseconds end-to-end. Cache misses fell through to the normal path with no measurable overhead.

6. Routed by complexity

Roughly 70 percent of requests were short factual lookups against the knowledge base. We introduced a lightweight classifier that routed those to a smaller, faster model — same vendor, one tier down — while leaving the long-form drafting work on the original model. Evaluation scores were unchanged on the routed traffic. Roughly 40 percent faster on that segment, with a meaningful cost reduction as a side effect.

What the team had been getting wrong

Three patterns sit underneath this engagement and we see them constantly.

• Treating latency as a model problem. The model is usually 10 to 25 percent of the budget. The rest is system design. Teams default to upgrading the model because it feels like progress; it rarely is.
• Measuring the wrong latency. P95 of the API call is not the number that matters. Time-to-first-useful-token from the user's perspective is. The first metric was 4.8 seconds; the second became under one second.
• Sequential work that doesn't need to be sequential. Almost every slow AI system we AI services has retrieval, moderation, or enrichment steps running in series for no reason other than the order they were written.

The result

Eight weeks, no frontier model upgrade, no vendor change, no procurement cycle. P95 from 4.8 seconds to 780 milliseconds. P99 from 9.2 seconds to 1.4 seconds. Agent adoption from 11 percent to 64 percent inside the following four weeks. Per-request cost down roughly 35 percent as a side effect of the routing work.

Most enterprise AI systems are not slow because the models are slow. They are slow because they were assembled rather than designed. Treating the system as a system — with the same engineering discipline as any other latency-critical production service — is almost always the answer.

If your AI assistant has poor adoption and the working theory is "the model is too slow", it is worth one week of real instrumentation before you spend a quarter on a procurement decision. The bottleneck is usually somewhere quieter, and it is usually fixable.