Autoscaling LLM workloads

Web services autoscale on CPU. LLM servers need different thinking because the GPU is the bottleneck, replicas are expensive, and starting one is slow. Autoscaling is how you get a good cost per token without missing your SLO at peak, but only if you choose the right signal, set the right target and respect the cold start.

#What autoscaling can and cannot do

• It can match capacity to the daily curve, saving the money you would spend on peak-sized hardware that idles overnight (cost per token).
• It cannot react to a burst that arrives in seconds when a replica takes minutes to boot.
• It cannot fix a service with no limits: a spike can exhaust memory long before new capacity arrives.

So treat autoscaling as the slow loop (minutes to hours) and rate limits, queue caps and load shedding as the fast loop (milliseconds to seconds) (reliability patterns).

#Pick the right signal

A robust policy scales out quickly when queue depth or in-flight requests per replica exceeds a threshold and scales in slowly (for example only after 10 to 15 quiet minutes), because adding a replica takes minutes but removing one is instant, and traffic often bounces.

#Find the per-replica limit first

Load-test one replica and find the concurrency at which TTFT (or per-user speed) crosses your SLO (load testing). Say that is 12 concurrent streams. Set the autoscaler's target at roughly 60 to 70 percent of that number, here about 8 in-flight requests per replica.

desired_replicas = ceil( total_in_flight / target_per_replica )

With 100 in-flight requests and a target of 8 per replica: ceil(100 / 8) = 13 replicas. Without the measurement you are guessing, and an untested target is either wasteful (too low) or ineffective (too high).

#Cold starts

Time to a ready replica is a chain of delays:

Adding these up for an example: node 60 s + image 90 s + weights 45 s + init 25 s + warm-up 15 s = about 235 seconds, nearly four minutes.

Ways to shorten it:

• Pre-bake the weights into the image or a fast local volume, not a download at start.
• Use fast storage (local NVMe, or a regional cache) for model files.
• Keep a warm pool: a minimum replica count above zero.
• Use memory-mapped or streamed loading options your runtime offers.
• Warm up with a few dummy requests before marking ready, so the first real user does not pay for kernel compilation.
• Keep a readiness probe that only passes once the model can generate.

If your cold start is 4 minutes, your autoscaler must act on a forecast four minutes ahead, or keep headroom. That is why a minimum replica count and a scale-out buffer are normal.

#Headroom arithmetic

If traffic can grow by a factor g during one cold-start period T, you need spare capacity of about g times current load at the moment you decide to scale. Example: load rises 50 percent per 4 minutes at the steepest, so run at no more than 1 / 1.5 ≈ 67 percent of capacity, which is the same 60 to 70 percent target above. The two rules agree for a reason.

#Scale in carefully

• Use a stabilisation window (10 to 15 minutes) so short lulls do not remove replicas.
• Remove one replica at a time, and drain it: stop new requests, let in-flight streams finish, then terminate (graceful shutdown).
• Never scale below your minimum for availability (usually 2) even at 3 a.m.
• Watch for flapping: out, in, out within an hour means the thresholds are too close together; widen the gap.

#Scheduled and predictive scaling

Daily and weekly traffic patterns are the most predictable signal you have. Combine them with reactive scaling:

• Scheduled: raise the minimum replicas before the 9 a.m. ramp and lower it after 6 p.m. This hides the cold start entirely for predictable load.
• Predictive: use last week's same-hour traffic to pre-scale.
• Reactive: the queue-depth rule as the safety net for the unexpected.

A service busy only on weekdays from 8 to 18 uses 50 of 168 hours a week. Scheduling replicas to those hours instead of running 24/7 saves about 70 percent of the baseline bill, with no loss of service.

#Kubernetes sketch

# Scale on a queue metric exported by your gateway or the engine
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: llm }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: llm }
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric: { name: requests_in_flight }
        target: { type: AverageValue, averageValue: "8" }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies: [{ type: Pods, value: 2, periodSeconds: 120 }]
    scaleDown:
      stabilizationWindowSeconds: 900
      policies: [{ type: Pods, value: 1, periodSeconds: 300 }]

You need a metrics adapter (for example Prometheus Adapter or KEDA) to expose the custom metric to the HPA. KEDA can also scale on Prometheus queries directly and on schedules. Exact manifests differ by version, so treat this as the shape, not a copy-paste. Also set pod resource requests for the GPU, a readiness probe that waits for the model, a terminationGracePeriodSeconds longer than your longest stream, and a PodDisruptionBudget so updates cannot remove all replicas at once.

#Practicalities for GPUs

• Replicas are lumpy. You scale in whole GPUs (or groups, for tensor-parallel models). One more replica is a 25 percent step at four replicas, not a smooth increase.
• Spot or preemptible instances save money for baseline-plus-burst designs, but can disappear; keep on-demand capacity for the floor and handle preemption with draining.
• Quotas and availability: cloud GPU capacity is not infinite. Test that you can actually obtain maxReplicas in your region, and have a fallback.
• Multi-model services: scale each model's pool independently; small models can share GPUs, large ones need their own.
• Cost guard-rails: cap maxReplicas, alert when the cap is hit and when spend per hour exceeds a threshold, so a runaway client cannot bankrupt you.

#Testing autoscaling

Do not assume it works. In staging, with realistic load:

1. Ramp traffic from 20 to 100 percent of expected peak over 10 minutes. Does TTFT stay within the SLO? How many replicas appear and when?
2. Apply a sudden spike (double traffic in 30 seconds). What does the queue do, and do limits shed excess gracefully?
3. Drop traffic to near zero. Do replicas scale in slowly, and does the minimum hold?
4. Kill a replica during scale-out. Does the system recover?
5. Measure the actual cold-start time end to end, and compare it with your assumption.

Record the results; they feed your capacity plan and your runbook (monitored deployment lab).

#Metrics to watch

#Autoscaling is not a substitute for limits

Autoscaling reacts in minutes. A burst arrives in seconds. Keep rate limits and queue caps in place so a spike degrades gracefully instead of exhausting memory while new replicas boot.

#Common mistakes

1. Scaling on CPU, which stays flat while users wait.
2. No measured per-replica limit, so the target is a guess.
3. Ignoring cold start, then wondering why scale-out arrives after the spike has passed.
4. Scaling in too eagerly, causing oscillation.
5. No minimum replicas, so every quiet night ends with a slow first user.
6. No maximum or budget cap.
7. Never testing: the first real test is a customer-facing incident.

#Exercise

Your replica handles 10 concurrent streams within the SLO. Cold start is 3.5 minutes. Evening traffic ramps from 40 to 100 concurrent streams over 15 minutes. Minimum replicas are 2.

1. What target per replica and how many replicas at 100 streams?
2. Will pure reactive scaling keep up with the ramp?
3. What would you add?

Answers:

1. Target about 7 per replica (70 percent of 10). At 100 streams: ceil(100 / 7) = 15 replicas.
2. Probably not entirely: the ramp adds about 4 streams per minute, so about 14 extra streams (2 replicas) arrive during one cold start. Reactive scaling that begins when the first threshold is crossed will lag by 3.5 minutes, so users experience a degraded stretch unless the target headroom absorbs it.
3. A scheduled increase of the minimum replicas shortly before the usual evening ramp, a faster scale-out policy (add two replicas per step), queue caps and rate limits as the fast loop, and a pre-warmed spare to cut effective cold start.