Estimate the inference infrastructure for support conversations using workload-specific operating assumptions. Adjust the inputs to match measured production behavior, then use the primary estimate and supporting metrics to compare deployment choices, identify constraints, and document a repeatable planning baseline. The calculator runs locally in your browser and does not send operational data to a server.
The final result is at least the minimum replica count.
The result estimates GPU replicas needed for output generation at peak load. Raw load shows the unrounded requirement, while spare capacity shows the remaining token rate after rounding.
This throughput-based estimate does not prove the model fits in GPU memory. Benchmark the chosen model, precision, batching, context length, and hardware.
With 600 requests per minute, 300 output tokens, 1,200 effective tokens/second per GPU, 70% target utilization, and two minimum replicas:
The default formula sizes output generation. Use a benchmark that combines prompt and generation work, or add prompt demand separately, when prefill is material.
Effective throughput should come from a representative benchmark and therefore reflects the model, serving stack, and workload.
It is the share of benchmarked throughput you plan to consume at peak, leaving operational room below the measured maximum.
Teams may require more than one replica for availability even when the calculated throughput fits on a single GPU.
The input assumes one throughput unit per GPU. Tensor or pipeline parallel deployments require mapping the result to the actual serving topology.
| Variable | Meaning |
|---|---|
| Peak requests/min | Highest expected request rate |
| Output tokens/request | Average generated length |
| Effective GPU throughput | Measured tokens per second per GPU |
| Target utilization | Planned share of measured capacity |
| Minimum replicas | Availability floor |