Estimate end-to-end latency and the concurrency needed to meet demand for an image generation service. The calculation separates queue, preprocessing, inference, and postprocessing time and compares service capacity with the requested arrival rate.
End-to-end latency = queue + preprocessing + inference + postprocessing. Capacity per hour = workers × 3,600 ÷ active service time. Workers needed = ceiling(demand × service time ÷ 3,600 ÷ 0.70).
Latency describes the experience of one request, while capacity describes the rate the worker pool can sustain. The 70% target leaves headroom for traffic variation; it is a planning assumption, not a universal standard.
Planning estimate only. Validate the result against production telemetry and current provider pricing before committing capacity or budget.
Using the defaults, end-to-end latency is 4.25 seconds and 4 workers provide about 3,512 tasks per hour before headroom.
Use recent production telemetry or a representative benchmark on the same model, hardware, input size, and quality settings.
No. It focuses on the entered operating assumptions; add fixed staffing, development, and platform overhead separately when building a full budget.
Use the observed fraction of attempts that must be repeated, and avoid adding the same retry overhead to more than one input.
A vendor maximum is a useful ceiling, but measured sustained performance is safer for capacity planning.
Input size, batching, model version, hardware, cold starts, safety checks, and traffic bursts can all change actual results.
| Input | Purpose |
|---|---|
| Workload values | Define the size and frequency of the task. |
| Performance values | Describe measured processing behavior. |
| Planning assumptions | Add utilization, retries, price, or headroom where applicable. |