Answer extracted from the AI + a16z podcast — listen to the full episode below.
When did open source AI become critical infrastructure rather than an experimental curiosity?
Open source AI crossed a critical threshold around 2023, when GitHub Copilot and ChatGPT became tools people could not live without, and open-weight models became foundational to daily productivity. The inflection came when application startups realized they could not build meaningful AI products using only closed-source models, forcing them to turn to open source for mid-training, post-training, and inference deployment.
The 2023 inflection: from experiment to necessity
The transition was sharp and driven by practical constraints. Startups discovered they needed control over model customization and deployment that proprietary alternatives could not provide. Closed-source models offered speed and reliability, but they lacked the flexibility required to build differentiated products at scale.
As discussed in the episode with Simon Mo, this shift forced the industry to build entirely new infrastructure layers to support open-source deployment. Open source went from a hobbyist curiosity to the backbone of production AI systems.
Infrastructure as the real bottleneck
Simply releasing model weights was never enough. Making open-source models viable at scale required solving problems that did not exist in the closed-source world—problems around inference efficiency, cost optimization, and multi-model orchestration across heterogeneous hardware.
This is where tools like vLLM, the inference engine now running on half a million GPUs at any moment, became essential. Without such infrastructure, open-source models would have remained a theoretical option, not a practical necessity.
"Every provider can offer potentially even 10 different levels of speed going from the slowest mode which can be a lot cheaper to 400 tokens per second almost up to 500 in many cases."
Simon Mo — Co-founder and CEO of Infraact and lead maintainer of vLLM. Mo and his team created vLLM in 2022, before ChatGPT even launched, when they set out to optimize a slow open-source demo and uncovered fundamental unsolved problems in LLM serving. Today, vLLM supports more than 1,000 model architectures and serves as critical infrastructure for thousands of enterprises.
This flexibility—the ability to trade speed against cost at granular levels—became a competitive advantage that proprietary platforms could not match. Open source enabled providers to offer speed tiers proprietary models reserved for their fastest mode alone.
The deeper reason open source became critical is that it solved a business problem proprietary models created: vendor lock-in and limited control. Enterprise teams discovered they could run, customize, and scale open-weight models independently, reducing dependency on a single provider and giving them the governance controls they needed for regulated industries.
Key takeaways
- The 2023 threshold was driven by startups needing customization and control that closed-source models could not provide.
- Infrastructure tools like vLLM became as critical as the models themselves—no weights-only release was sufficient.
- Open source enabled speed-vs-cost tradeoffs that proprietary platforms could not offer, making it cost-competitive for enterprises.
- The shift was fundamental: open source went from hobby project to non-negotiable infrastructure for AI-native startups.
See also
How does vLLM function within the modern AI infrastructure stack for enterprises using open-source models?
vLLM is an inference engine that turns available GPUs into running endpoints for intelligence, comparable to databases and operating systems. It supports more than 1,000 model architectures and runs on half a million GPUs at any moment, making it foundational infrastructure rather than just a developer tool.
What makes serving a large language model fundamentally different from traditional machine learning workloads?
Serving a large language model requires running it on accelerators like GPUs or TPUs with computationally intensive processes that need quick and efficient resource management, unlike traditional ML workloads that can tolerate batch processing delays and simpler hardware constraints.