Answer extracted from the AI + a16z podcast — listen to the full episode below.
What economic advantages do open-weight models provide in terms of inference speed and customization compared to proprietary models?
Open-weight models unlock 10+ distinct speed tiers that proprietary solutions simply cannot match, enabling inference throughput of 400–500 tokens per second compared to 2–3x slower proprietary fast modes. This granular control over performance and cost is only possible when you run your own infrastructure instead of relying on API access.
The fundamental difference lies in how each model tier is packaged and delivered. Proprietary models typically offer two options: a standard mode and a fast mode. You pick one, accept its speed and cost trade-off, and move forward. Open-weight models work entirely differently because every provider running the same model weights can tune its own inference stack independently.
As Simon Mo explains in the episode, this flexibility is what transforms economics for enterprises. Instead of choosing between expensive (slow, accurate) or cheap (fast, slightly less accurate), you can provision anywhere along a spectrum of 10 different configurations, each optimized for a specific latency-cost balance.
Where the speed advantage compounds
The 2–3x speed advantage manifests concretely in real-world throughput. A proprietary fast mode might achieve a certain tokens-per-second rate, but an optimized open-weight inference setup can push 400–500 tokens per second on the same hardware. This isn't magic—it's the result of running dedicated inference engines like vLLM that handle the specific computational patterns of language models far more efficiently than general-purpose APIs.
When you control your own infrastructure, you're not paying for OpenAI's or Anthropic's operational overhead, their moderation systems, or their API gateway latency. Every millisecond saved directly translates to cost savings at scale. A 2–3x improvement in throughput means you can serve the same traffic volume on one-third the GPUs, or serve three times more traffic on the same hardware budget.
The customization angle runs deeper than speed alone. Enterprises want to add their own guardrails, extend the model pipeline, and understand exact performance characteristics, as discussed in detail in the podcast. Proprietary APIs give you none of that visibility or control. With open-weight models running on your own infrastructure, you audit every layer, inject your own safety filters, and modify the inference process to match your exact business logic.
"Every provider can offer potentially even 10 different levels of speed going from the slowest mode which can be a lot cheaper to 400 tokens per second almost up to 500 in many cases."
Simon Mo — Co-founder and CEO of Infraact, and lead maintainer of vLLM, the open source inference engine now running on half a million GPUs at any moment. Mo founded vLLM in 2022 when his team attempted to make a slow open source demo faster and uncovered a pile of unsolved problems in LLM serving infrastructure. Infraact, his company, builds the infrastructure layer that makes open-source AI models production-ready at enterprise scale.
The real-world implication is that this level of granular control is only achievable when running your own infrastructure—you cannot replicate it through an API, no matter how flexible the provider claims to be. That constraint alone reshapes the economics and strategic calculus for any organization making serious infrastructure bets.
See also
Why do enterprises choose to deploy open-weight models instead of relying on proprietary APIs, beyond cost considerations?
Control is a backbone motivation: enterprises want to control their own model infrastructure, extend it, add guardrails, and understand exact performance metrics—something proprietary APIs never permit.
At what point did open source AI models become critical infrastructure rather than an experimental curiosity?
Open source became critical infrastructure around 2023, when GitHub Copilot and ChatGPT became something people could not live without, and open-weight models proved viable for production workloads.
How does vLLM function within the modern AI infrastructure stack for enterprises using open-source models?
vLLM is an inference engine that turns available GPUs into running endpoints for intelligence, comparable to databases and operating systems. It supports over 1,000 model architectures and runs on half a million GPUs at any given moment.
Key takeaways
- Open-weight models support up to 10 distinct speed tiers versus proprietary solutions' binary fast/standard choice.
- Optimized open-weight inference achieves 400–500 tokens per second, typically 2–3x faster than proprietary fast modes.
- Running your own infrastructure eliminates API overhead, moderation latency, and operational costs, directly improving unit economics.
- Only self-hosted open-weight deployments permit true customization: guardrails, pipeline extensions, and full performance visibility.