GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Summary
- Researchers introduced GAUSS, a workload-aware analytical and Monte-Carlo simulator designed for large language model inference serving.
- GAUSS predicts full distributions of time-to-first-token and time-between-tokens, alongside sustainable request rates for specific workloads, engines, and accelerators.
- The simulator pairs calibrated latency surfaces for prefill and decode phases with a Lindley-recurrence model of queueing under stochastic arrivals.
- It uses ensemble Monte-Carlo sampling to capture latency tails and support operating-point selection, traffic segmentation, and scheduling capacity questions.
Why it matters
GAUSS addresses the limitations of trace-driven simulators by accurately predicting tail latencies and evaluating undeployed workloads, routing policies, and hardware configurations for LLM serving.