Estimate the accelerator count needed for a prompt-cached inference workload. The calculation treats uncached prefill, cached-prefix handling, and output decoding as separate capacity demands, applies a target utilization limit, and adds optional redundancy. This makes the bottleneck visible instead of averaging incompatible processing stages into one token rate.
The total is an integer provisioning estimate. The largest stage load determines capacity because each stage must keep pace with incoming work. Redundancy is added after rounding the workload requirement.
This stage-bottleneck model assumes the stated GPU rates are compatible with your serving architecture. Memory capacity, model replicas, tensor parallelism, and cache residency may impose a higher minimum.
At 20 requests/s, 800 uncached, 3,200 cached, and 400 output tokens, with per-GPU rates of 50k, 500k, and 8k tok/s at 70% utilization:
The model treats stages as separate capacity constraints. If they share the same compute pool without overlap, a summed-work model may be more conservative.
No. Verify model weights, runtime overhead, batching, and KV-cache memory separately.
This calculator excludes them from required workload capacity so they remain available for failure or maintenance.
Use a level supported by your latency tests and operating policy; lower targets provide more queue and burst headroom.
Moving tokens from uncached prefill to faster cached handling can lower prefill load, but decoding may remain the bottleneck.
| Variable | Meaning | Unit |
|---|---|---|
| λ | Target request rate | requests/second |
| u | Target utilization | decimal |
| λ × tokens | Stage token demand | tokens/second |
| S | Stage throughput per GPU | tokens/second/GPU |
| N | Provisioned GPUs | whole GPUs |