How should teams approach starting with evaluations for AI agents?
Answered at 13:44 in Ep. #44, Beyond the Model: Observing AI Agents with Mikyo King · Heavybit Podcasts
How should teams approach starting with evaluations for AI agents?
You don't have to start with evals—it depends on how critical what you're building is. Evaluations work like layers of Swiss cheese with different layers and levels, and none of them will be perfect, but you want to guarantee that as the frontier shifts or model decisions change your system continues working. Evaluations can be a way to sift through data and don't always have to be perfect signals; people will use your platform in ways you didn't anticipate, so you can use LLM power to classify intent, segment data by region, or detect changing language patterns.
Full answer and detailsRead the whole answer, key facts and every other question from this episode
In short
You don't have to start with evals—it depends on how critical what you're building is.
Mikyo King, Head of Open Source at Arize · Ep. #44, Beyond the Model: Observing AI Agents with Mikyo King · Heavybit Podcasts · answer at 13:44
Key takeaways
- You don't have to start with evals—it depends on how critical what you're building is.
Quote to remember
“It's one thing to capture the information. It's another thing to kind of make that the fossil fuel for improving your system as the next step.”
— Mikyo King
The full answer · about 2 min
You don't have to start with evals—it depends on how critical what you're building is. Evaluations work like layers of Swiss cheese with different layers and levels, and none of them will be perfect, but you want to guarantee that as the frontier shifts or model decisions change your system continues working. Evaluations can be a way to sift through data and don't always have to be perfect signals; people will use your platform in ways you didn't anticipate, so you can use LLM power to classify intent, segment data by region, or detect changing language patterns.
Key facts
- Mikyo King has been a software developer for more than 20 years
- Last three years specifically focused on open source at Arize
- Arize started open source initiative around three years ago for observability and evaluation tooling
- Three years ago questions centered on LLM hallucinations, now focus shifted to agents as a service with higher levels of abstraction
People, companies and topics mentioned
More answers from this episode
- 3:26How does Arize use OpenTelemetry for AI agent observability?
- 5:27What is the difference between Open Telemetry semantics and OpenInference semantics for AI observability?
- 16:01Why did Arize choose to open source AI observability tooling instead of building a closed SaaS product?
- 19:05What security challenges does Arize face when open sourcing AI agent observability?
- 23:44How should open source maintainers handle pull requests from AI coding agents?