Estimate the compute fleet required for a self-hosted AI code review model. The calculator converts peak review demand and tokens per review into required token throughput, adjusts usable GPU capacity for utilization, and applies a replica floor for availability. It also checks whether model weights, runtime overhead, and per-request KV cache fit in GPU memory at the planned concurrency. Use benchmark throughput from the exact model, precision, hardware, and serving stack.
Required throughput = reviews/minute × tokens/review ÷ 60
Usable GPU throughput = benchmark throughput × utilization target
GPU count = max(minimum replicas, ceiling(required ÷ usable))
VRAM headroom = VRAM − weights/runtime − KV cache × concurrency
The main result is the larger of the throughput-driven GPU count and the minimum replica count. The separate VRAM check prevents a throughput-feasible plan from silently exceeding memory at the entered concurrency.
This estimate assumes each GPU independently delivers the entered measured throughput. Tensor parallelism, batching, prompt caching, and interconnect overhead must already be reflected in the benchmark.
At 30 reviews per minute and 12,000 tokens each, demand is 6,000 tokens/second. A GPU measured at 120 tokens/second and targeted at 70% supplies 84 usable tokens/second, so compute demand is ceiling(6,000 ÷ 84) = 72 GPUs.
Token volume better captures how review size changes inference work than request count alone.
Yes. The replica floor can represent availability zones, maintenance, or redundancy requirements.
No. The separate VRAM headroom result checks the entered weights, runtime, KV cache, and concurrency assumptions.
Use a combined measured rate only if your benchmark reflects both phases; otherwise calculate prefill and generation capacity separately.
No. The result respects the positive minimum replica input even when workload demand is zero.
| Input | Purpose | Use measured? |
|---|---|---|
| Tokens per review | Work per request | Preferred |
| Throughput per GPU | Hardware capacity | Required |
| Utilization | Operating headroom | Policy choice |
| KV cache | Concurrency memory | Preferred |