Answer extracted from the AI + a16z podcast — listen to the full episode below.
Why do enterprises choose to deploy open-weight models instead of relying on proprietary APIs, beyond cost considerations?
Enterprises deploy open-weight models primarily for infrastructure control, data governance, and compliance assurance—not cost savings alone. With open-weight models, companies can guarantee response times, control data retention, enforce security policies, fine-tune models to their needs, and maintain SLA compliance in ways proprietary APIs do not permit.
The shift toward open-weight models reflects a fundamental change in how enterprises view AI dependencies. For voice agent companies and customer-facing services, controlling the inference engine itself is non-negotiable: they need guaranteed latency and reliability that a third-party API simply cannot promise.
As explained in this episode of AI + a16z, the performance profiles become transparent and reproducible when enterprises own their model infrastructure. This transparency extends beyond speed: companies gain the ability to understand exactly how many tokens their models consume, optimize cost per inference, and modify guardrails to match their specific compliance and safety requirements.
Control as the Core Enterprise Requirement
Data sovereignty and compliance drive a significant portion of enterprise adoption. Unlike proprietary APIs where data flows to third-party servers, open-weight models run entirely on an organization's own hardware, ensuring data never leaves the premises. This is especially critical in regulated industries: financial services, healthcare, and government agencies cannot afford to outsource their intelligence layer.
Performance predictability matters equally. Voice agent companies need guaranteed sub-second response times for phone calls; proprietary APIs offer no such guarantee because you are sharing infrastructure with thousands of other customers. With open-weight models deployed on dedicated hardware, the SLA becomes enforceable and transparent.
Fine-tuning and model customization represent another layer of control unavailable with proprietary APIs. Enterprises can adapt open-weight models to domain-specific language, industry jargon, or customer behavior patterns—then measure the impact of each change on accuracy and cost. This feedback loop is simply impossible with a black-box API.
"Every provider can offer potentially even 10 different levels of speed going from the slowest mode which can be a lot cheaper to 400 tokens per second almost up to 500 in many cases."
Simon Mo — Co-founder and CEO of Infraact and lead maintainer of vLLM. Mo's team created vLLM, an open source inference engine now running on half a million GPUs at any moment, after identifying unsolved problems in LLM serving infrastructure. His work spans the full spectrum of open-source model deployment challenges, from infrastructure optimization to enterprise adoption patterns.
This flexibility in speed-versus-cost tradeoffs is unique to open-weight deployments. With proprietary models, you typically get one "fast mode" and one "standard mode." Open-weight stacks allow operators to tune inference parameters, batch sizes, quantization levels, and hardware allocation to hit any point on the speed-cost curve. Enterprises can serve high-latency queries cheaply and reserve premium speed for customer-facing interactions—something impossible with fixed API pricing tiers.
For a deeper look at how vLLM powers this infrastructure layer, the episode discusses the full stack of considerations enterprises must weigh when building proprietary AI systems.
See also
At what point did open source AI models become critical infrastructure rather than an experimental curiosity?
Open source became critical infrastructure around 2023, when GitHub Copilot and ChatGPT became something people could not live without, and open-weight models emerged as trusted alternatives for enterprises seeking control and reliability.
How does vLLM function within the modern AI infrastructure stack for enterprises using open-source models?
vLLM is an inference engine that turns available GPUs into running endpoints for intelligence, comparable to databases and operating systems. It supports more than 1,000 model architectures and runs on half a million GPUs at any moment.
What makes serving a large language model fundamentally different from traditional machine learning workloads?
Serving a large language model requires running it on accelerators like GPUs or TPUs with computationally intensive processes that need quick and efficient token generation, unlike traditional ML workloads optimized for batch processing and latency tolerance.