GPU Inference Autoscaling: Why Kubernetes HPA Fails and What to Deploy Instead

Every enterprise AI programme we work with in 2026 hits the same wall around the six-month mark. The first LLM workload ships. It runs on a handful of GPU nodes behind a Kubernetes Service. Traffic grows. Somebody points the standard Horizontal Pod Autoscaler at CPU utilisation, watches it thrash for a week, and quietly disables it. The platform team then hard-codes a replica count high enough to survive peak traffic, burns cash on idle GPUs the other twenty-two hours of the day, and adds "revisit autoscaling" to a backlog that never gets touched. We have inherited this exact configuration on more than a dozen client estates in the last twelve months.

The problem is not laziness. The problem is that the default Kubernetes autoscaling primitives were designed for stateless web tier services in the 2015 mould, and every assumption they make is wrong for large language model inference. This is a walk through what actually breaks, and the specific autoscaling stack we deploy when we are asked to fix it properly.

Why HPA on CPU or memory is the wrong signal

A vLLM or TensorRT-LLM server pinned to an H100 or B200 spends most of its wall clock time waiting on GPU compute. Node-level CPU sits at ten to fifteen percent while the GPU is saturated at ninety-five. Node memory looks flat while VRAM is completely pinned by the KV cache. HPA scaling on CPU utilisation will happily leave a pod at "healthy" while every request queues behind a full batch on the GPU. Users see latency degrade. HPA sees nothing.

Scaling on memory is worse. Serving frameworks pre-allocate GPU memory at startup and hold it constant. Node memory reported to the kubelet stays at whatever the model binary occupies. The signal is dead flat regardless of load. We have seen production estates where HPA scaled a Llama 70B deployment to fifty replicas because a sidecar cron job briefly touched a memory threshold. Fifty B200s idled for six hours before anyone noticed the invoice implication.

The lesson is simple. GPU inference is not CPU-bound and not memory-bound in the way HPA understands the words. The signals HPA can see are the wrong ones. The signals that matter live inside the inference server.

The four signals that actually matter

Every effective GPU autoscaler we deploy is driven by some combination of the following, exposed as Prometheus metrics from the inference server itself:

• Requests in the queue. The number of requests waiting for a slot in the current batch. This is the leading indicator. When it starts climbing, latency is about to degrade regardless of what any other signal says.
• GPU utilisation and SM occupancy. Pulled via DCGM exporter, not the kubelet. Distinct from node CPU. A sustained value above eighty percent with a growing queue is the definitive scale-up trigger.
• KV cache pressure. The percentage of VRAM committed to the KV cache. When this saturates the server rejects new sequences or evicts old ones, and tail latency spikes. Most teams do not monitor this and only learn about it during an incident.
• Time to first token and inter-token latency. The user-facing latency metrics the SLO is actually written against. Scale decisions that ignore these end up chasing internal utilisation while user experience quietly rots.

These are the signals a modern inference autoscaler consumes. HPA on its own cannot consume any of them without adapters. This is where the stack starts to get interesting.

The stack we deploy

There is no single component that solves this. GPU inference autoscaling is a small system of cooperating parts. The version we ship on most enterprise estates in 2026 looks like this.

Metrics layer

DCGM exporter on every GPU node for hardware metrics. vLLM, TGI, or TensorRT-LLM exposing native Prometheus endpoints for queue depth, KV cache utilisation, active sequences, and token throughput. Prometheus scraping both, with recording rules that pre-compute the queue-per-replica and utilisation-per-model ratios the autoscaler cares about. If a client is on Datadog or New Relic we mirror the same signals there, but the source of truth for scaling decisions stays inside the cluster to avoid a control-plane dependency on an external SaaS.

Autoscaler layer

KEDA, not vanilla HPA. KEDA lets us drive scaling from Prometheus queries directly, so we can scale on queue depth per replica greater than four for two minutes instead of on a CPU proxy. KEDA also supports scale-to-zero, which matters enormously for the long tail of internal workloads that get used a few hours per day. For workloads on KServe or Ray Serve we prefer their native autoscalers, which understand token-level throughput and concurrency directly, but we still wire in KEDA as the fallback for anything that lives outside a serving framework.

Node layer

Karpenter with GPU-aware provisioners, not the classic cluster autoscaler. GPU nodes take five to eight minutes to boot when you include image pull, model warm-up, and CUDA driver initialisation. Karpenter's ability to pre-provision on scheduling intent, batch node launches, and consolidate underutilised capacity is the difference between an autoscaler that saves money and one that fights itself. We also configure node overhead correctly so a B200 node is not blocked from scheduling by a mis-declared thousandth-of-a-GPU sidecar.

Model warm-up and cold start

The cold-start problem on GPU inference is not the node boot. It is the model load. A 70B model on B200 takes 90 seconds to warm the weights from container image into VRAM, and another 30 to warm the CUDA graphs. A scale-up that arrives after the queue has already saturated is worse than useless because the new pod cannot serve traffic in time. We ship every deployment with a warm-pool sidecar, a lazy weight loader that memory-maps directly from a shared PVC or S3-backed model store, and predictive scale-up rules that fire on queue trend rather than steady-state depth.

Cost guardrails

Every KEDA ScaledObject is deployed with a hard maxReplicaCount, a max-per-namespace GPU quota, and a Datadog or Grafana alert on any deployment holding more than N replicas for more than M minutes without traffic. Autoscaling saves money in the median case and generates six-figure invoices in the tail case. We have seen a misconfigured rule scale a fine-tuned Mistral deployment to twenty H100s overnight because a broken caller retried in a tight loop. The guardrails are not optional.

Scale-to-zero: worth it, with caveats

Scale-to-zero is the feature every finance team asks for and every SRE team is nervous about. The economics are real. Half of the LLM workloads we operate see zero traffic between 20:00 and 07:00 local. Scaling those to zero and back saves forty to sixty percent of GPU spend. But scale-to-zero on a 90-second cold start creates a user-visible seven-figure-per-second latency spike the first request after the idle window. The pattern we use:

• Scale to zero only for asynchronous or batch workloads. Anything user-facing scales to a minimum of one, always warm.
• Use a lightweight router in front. The router holds the first request, triggers scale-up, and streams a "warming" placeholder until the pod is ready. Users see a two-second delay, not a two-minute timeout.
• Warm on schedule for known peaks. A cron-based ScaledObject that pre-warms capacity at 07:45 for the 08:00 traffic ramp costs almost nothing and eliminates the morning cold-start hit entirely.

What we look for in an audit

When a client asks us to review an existing GPU inference platform, this is roughly the checklist we run in the first ninety minutes. If any of these fail, the autoscaling story is not real yet.

Is HPA driven by anything other than CPU or memory? Is DCGM exporter deployed and scraped? Does the serving framework expose queue depth as a Prometheus metric? Does the deployment have a maxReplicaCount guardrail? Does any workload have scale-to-zero configured, and is there a warm path in front of it? Is Karpenter or an equivalent provisioner batching node launches, or is every scale-up a fresh cold start? Are there any alerts on stuck-high replica counts?

The gap between an enterprise that has ticked all of these and one that has ticked none of them is usually twelve to eighteen months of infrastructure maturity, and in 2026 dollars, mid seven figures a year in avoidable GPU spend. Autoscaling done properly is one of the highest-leverage engineering investments an AI programme can make. Autoscaling done with the defaults is one of the fastest ways we know to burn the budget the CFO gave you for the model itself.

If your GPU inference tier is running a static replica count today and you know it should not be, we have done this migration on enough estates to make it boring. That is usually the goal.