Answer extracted from the AI + a16z podcast — listen to the full episode below.
Why LLM serving requires a completely different infrastructure than traditional ML?
Serving a large language model requires running it on accelerators like GPUs or TPUs with computationally intensive processes that demand quick and efficient responses—fundamentally different from traditional machine learning workloads. The challenge lies in handling variable input lengths, non-deterministic output distributions, and the complex batching and scheduling requirements embedded in modern inference engines.
The Infrastructure Gap: GPUs, TPUs, and Real-Time Demands
Traditional machine learning models like BERT and ResNet are designed for standardized inputs and outputs. LLMs operate in an entirely different domain. Each request can vary dramatically in length—from a single sentence to thousands of tokens—and the model's response length is unpredictable. This variability forces inference engines to solve scheduling and batching problems that don't exist in conventional ML serving.
The computational intensity compounds this problem. GPUs and TPUs aren't just nice-to-have optimizations; they're absolutely essential because the models themselves are so large that they cannot run efficiently on standard processors. This is why vLLM was designed from the ground up as an inference engine specifically built for LLM workloads, not as a repurposed framework from traditional ML.
Non-Deterministic Outputs and Dynamic Batching
Traditional ML inference is deterministic: the same input always produces the same output, and batch sizes can be fixed. LLMs are fundamentally different. Each token generation is non-deterministic due to sampling and temperature parameters, meaning identical prompts can yield different results. Additionally, variable request lengths make fixed batching impossible—the inference engine must dynamically adjust how it groups requests to maximize throughput without sacrificing latency.
This dynamic behavior is why companies like NVIDIA, AMD, and Google are building inference optimization into their newest chip architectures, and why vLLM has become critical infrastructure that now runs on half a million GPUs at any moment.
"Every provider can offer potentially even 10 different levels of speed going from the slowest mode which can be a lot cheaper to 400 tokens per second almost up to 500 in many cases."
Simon Mo — Co-founder and CEO of Infraact and lead maintainer of vLLM. Mo's team discovered these serving challenges while trying to optimize an open-source demo in 2022, years before ChatGPT democratized LLMs. The resulting vLLM project has become a foundational layer powering modern AI infrastructure across multiple hardware vendors and cloud providers.
The quote reflects a concrete reality: speed optimization in LLM serving is far more complex than in traditional ML, requiring specialized scheduling algorithms that account for all these moving parts simultaneously. This is why traditional machine learning infrastructure simply cannot be adapted to handle LLMs—the problems are fundamentally different at the architectural level.