Estimate sustainable request throughput for a GPU pool serving prompts with reusable cached prefixes. The calculator converts prefill, cache-handling, and decode capacity into requests per second, then uses the tightest stage as the end-to-end limit. It also reports hourly tasks, token throughput, and headroom against an optional traffic target.
The main result is the request rate that every active stage can sustain. Hourly tasks assumes steady demand. Token throughput multiplies this request rate by total tokens per request and includes cached tokens as processed workload.
Zero-token stages are excluded rather than treated as unlimited numeric results. Real throughput can be lower because of memory, batching, routing, or queueing constraints.
With 4 GPUs at 70% utilization, 800 uncached, 3,200 cached, and 400 output tokens, and rates of 50k, 500k, and 8k tok/s/GPU:
A request must pass every active stage, so the slowest stage limits the sustainable end-to-end rate.
The cache stage is excluded when it has no tokens, preventing division by zero and reflecting that no cache work is required.
Yes. It is sustainable requests per second multiplied by 3,600, assuming steady demand and the entered utilization.
Yes, as processed workload, even though their computational cost per token may be lower.
Memory limits, uneven request lengths, batching inefficiency, routing, and queueing can reduce observed throughput.
| Variable | Meaning | Unit |
|---|---|---|
| G | Workload GPUs | GPUs |
| u | Planned utilization | decimal |
| S stage | Per-GPU stage throughput | tokens/second |
| T stage | Stage tokens per request | tokens/request |
| Q | Sustainable throughput | requests/second |