GPU FinOps: Cutting Enterprise AI Infrastructure Spend

Every enterprise we've helped scale generative AI in the last eighteen months has hit the same wall: the bill. The pilots looked cheap. Production looks ruinous. CFOs are now asking pointed questions about what their AI infrastructure is actually doing during the 70 percent of the day it isn't doing anything useful — and engineering teams don't have great answers.

This is GPU FinOps, and most organisations are at year zero. The good news is that the wins are large and unglamorous. After more than a dozen enterprise cost reviews this year, we've come to a consistent conclusion: most AI spend can be cut 40 to 60 percent inside a quarter, with no measurable degradation in user-facing latency or quality. Below is the playbook we use.

Why AI infrastructure burns money differently

Classical cloud FinOps was built around a handful of well-understood failure modes: idle VMs, oversized databases, untagged resources, forgotten dev environments. The mental model was "find the thing nobody is using and turn it off." That model breaks down for AI workloads, for three reasons.

• GPU economics are extreme. A single H100 node costs more per hour than dozens of CPU instances. A small inefficiency on a CPU fleet is a rounding error; the same inefficiency on a GPU fleet is a budget event.
• Utilization is structurally low. Traffic to inference services is bursty, but GPUs are usually provisioned for peak. Average utilization across the enterprise inference deployments we have reviewed sits between 12 and 28 percent. The rest is paid-for silence.
• Cost is coupled to model choices, not just infrastructure. A 10x model choice can dwarf any cluster-level optimisation. You cannot FinOps your way out of running a 405B-parameter model when a 70B would do.

This last point is the most consequential and the most frequently ignored. AI FinOps is not just an infrastructure conversation — it is an architectural one.

The patterns we see burning the most money

The same handful of anti-patterns appear in nearly every enterprise environment we review. They are not exotic. They are boring, expensive, and entirely fixable.

• One model per use case, always on. Teams provision dedicated endpoints for every product surface, each running 24/7 against a state-of-the-art model. Half of them serve fewer than 200 requests an hour.
• No request batching. Inference servers process requests one at a time, leaving 80 to 90 percent of GPU throughput on the table. Modern serving stacks like vLLM and TensorRT-LLM support continuous batching out of the box; many teams have simply never enabled it.
• Production traffic on the largest available model. The team prototyped on the frontier model, shipped, and never revisited. We routinely see 80 percent of traffic that would be served equally well by a model an order of magnitude cheaper.
• No autoscaling, or autoscaling on the wrong signal. GPU clusters scaled on CPU utilization, scaled on queue length but with cold-start times that make scaling meaningless, or pinned at peak capacity because cold starts were too painful to debug.
• Untracked experimental spend. Fine-tuning jobs, hyperparameter sweeps, and notebook GPUs running over weekends with no owner. We have found single forgotten jobs costing more than five-figure sums per month.
• Commitment-free purchasing. Months of steady production traffic on pure on-demand pricing, with no reserved capacity or savings plans in place. This alone is typically a 30 to 50 percent overpayment versus committed alternatives.

The levers that actually move spend

We work through these in roughly this order, because the earlier ones tend to be cheap and the later ones tend to require architectural change.

1. Right-size the model before you right-size the cluster

For each production workload, ask one question: what is the smallest model that still satisfies the quality bar? "Quality bar" must be defined as an evaluation suite, not a vibe. We routinely move 60 to 80 percent of traffic from frontier models to mid-tier or fine-tuned open-weight models with no measurable loss on the metrics that matter to the business. The cost delta is usually 5 to 20x.

This is the single highest-leverage move in AI FinOps and it is almost always available.

2. Turn on continuous batching and KV-cache reuse

If you are serving open-weight models, you should be on vLLM, TensorRT-LLM, SGLang, or an equivalent. The throughput improvement over naive serving is routinely 5 to 10x for the same hardware. KV-cache reuse across requests with shared prefixes — common in RAG and agent workloads — adds another large multiplier on top.

3. Quantize aggressively, evaluate honestly

FP8 and INT8 quantization on modern GPUs frequently delivers near-identical quality at roughly half the memory footprint and meaningfully higher throughput. The catch is that quality regressions are workload-specific. Quantize, run your evals, and only ship if the deltas are within tolerance. We have seen teams refuse to quantize on principle and pay double for capacity they did not need.

4. Cache the obvious

Semantic caching of frequent or near-duplicate prompts can absorb 20 to 60 percent of inference traffic for many enterprise workloads — internal assistants, support copilots, structured extraction pipelines. The infrastructure is cheap; the savings are not.

5. Schedule for the workload shape

Inference is latency-sensitive and lives on dedicated GPU capacity. Training, fine-tuning, batch evaluation, and offline inference are not, and should not. We move all non-interactive work to spot GPUs with checkpointing — often at 60 to 80 percent discounts — and isolate them from production clusters so a spot reclaim never affects a user request.

6. Commit to capacity you are actually going to use

Once a workload is stable for 60 to 90 days, baseline its real consumption and buy committed capacity for the floor. Leave the peaks on on-demand or spot. This is unglamorous procurement work that consistently returns 25 to 40 percent on the committed portion.

7. Tag, attribute, and show people the bill

You cannot manage what you cannot attribute. Every GPU hour, every fine-tuning job, every inference endpoint should be tagged to a product, team, and environment. Then send the bill, weekly, to the team that owns the workload. Behaviour changes within a sprint.

Who should own this

AI FinOps sits in an uncomfortable seam between platform engineering, ML engineering, and finance. In our experience, the organisations that get this right share three traits:

• A single named owner for AI infrastructure cost. Not a committee. A person, usually inside platform engineering, with the authority to push back on workloads that are not paying their way.
• Cost as a first-class production metric. Dollars-per-thousand-requests sits next to latency and error rate on the same dashboard, with the same alerting discipline.
• Architectural review for new AI workloads. A lightweight review before any new endpoint is provisioned, focused on model choice, expected traffic, and serving strategy. Cheap to run, very expensive to skip.

What this looks like in practice

One recent engagement: a Fortune 500 enterprise running roughly 40 production inference endpoints across three clouds, with a monthly AI infrastructure bill north of seven figures. Inside a quarter we consolidated to nine endpoints, moved 70 percent of traffic to right-sized open-weight models on vLLM with FP8 quantization, introduced semantic caching on the three highest-volume workloads, shifted all non-interactive work to spot, and put a committed-use plan in place against the production floor.

Monthly spend fell 58 percent. P95 latency improved. No user-facing quality regression on any tracked evaluation. The work was almost entirely engineering hygiene applied with discipline — there was no silver bullet, and there rarely is.

The companies winning on AI economics are not the ones with the best models. They are the ones who treat inference as a production system and operate it with the same rigour as the rest of their platform.

If your AI spend has crossed the threshold where the CFO is asking questions, the answer is not to slow down. It is to operate the platform properly. The savings are sitting there, in plain sight, in clusters everyone has stopped looking at.