Estimate how prompt caching changes end-to-end inference latency and the concurrency needed to sustain a request rate. Model uncached prefill, cached-prefix lookup, output decoding, and fixed network or queue overhead separately. The result highlights time to first token, full response latency, cached share, and the number of in-flight requests implied by Little’s Law.
Full response latency estimates when the complete answer is available. TTFT estimates when generation can begin, and average concurrency estimates simultaneous in-flight requests at the stated arrival rate.
Rates should be measured under representative batch size, model, hardware, and cache-hit conditions. Queueing under saturation is not automatically included.
At 8 requests/s with 600 uncached tokens, 2,400 cached tokens, 300 output tokens, rates of 12,000, 120,000, and 120 tokens/s, plus 80 ms overhead:
No. Uncached tokens still require normal prefill, and cached tokens can still incur lookup and transfer time.
Autoregressive decoding produces output sequentially, so long responses can take more time than cached prompt processing.
It is the average in-flight request count implied by arrival rate multiplied by response time, before extra headroom.
Only if queueing is included in fixed overhead. Run a higher-overhead scenario for saturated conditions.
Measured effective speed at your model, batch size, and workload is safer because peak specifications may not match production.
| Variable | Meaning | Unit |
|---|---|---|
| λ | Arrival rate | requests/second |
| U, C | Uncached and cached input | tokens/request |
| P, K | Prefill and cache speeds | tokens/second |
| D | Decode speed | tokens/second |
| L | Full response latency | seconds |