The Hallway Track
35 tracked signals on evals.
Nubank ships AI customer support agents 20x faster by generating eval data in simulation rather than waiting for production data.
“If you generate your eval data in sim instead of waiting on production data you can ship agents 20x faster and we'll give you evidence for that.”
Manually reviewing 100-1,000 real examples beats any automated benchmark for evaluating AI models.
“there's just no better eval than looking at 100 examples or 1,000 examples”
LangChain CEO argues companies must own their agent harness to truly own their AI intelligence
“The main job of a harness is to bring context to the model at the right point in time.”
Cognition built FrontierCode eval after SWE-bench saturated, focusing on code mergeability
“okay, this code is technically correct. But would you actually merge it? Would you feel happy? Would this improve the quality of your code base?”
AI evals must evolve from LLM-as-judge to agent-as-judge as systems shift to multi-agent long-horizon tasks
“we didn't just make the problem harder, we actually got a fundamentally different type of problem”
Salesforce AgentForce uses LangSmith to standardize evals across teams at scale
“LangSmith helps us to amplify our internal expertise, allowing us to run thousands and thousands of test cases.”
Agentic AI evaluation must shift from model benchmarking to production-grade system behavior testing.
“The question is no longer did the model generate the right answer? The question is did the system behave correctly?”
Chime built compliance evals for its AI co-pilot Jade by having legal teams co-author risk definitions.
“I would argue that evals are your alignment surface.”
Lyft scaled to seven+ production AI agents at 35% resolution by building an offline-eval quality gate before shipping.
“You don't want to use your users as test data.”
AI engineering evals are fundamentally data science, and teams keep repeating avoidable eval mistakes.
“I'm saying that using LLM judges kind of blindly without validating your validators is bad.”
Agent quality demands both pre-production evals and production observability to manage LLM non-determinism risks
“grading matters because LLMs are inherently non-deterministic”
Arize AI presents a 101 framework for evaluating AI agents from tracing to LLM-as-judge meta-evaluation
AI evals provide reliability in advance, not just post-hoc validation
“Evals provide reliability, but not just any reliability, but reliability in advance.”
Systematic evals beat gut-feel releases: 94% pass rates beat 'it looked good'
“I ran 200 different test scenarios, 94% of them passed, so we're releasing this feature.”
Clay runs 300M agent executions monthly and calls evals non-negotiable at that scale
“eval's became non-negotiable”
DoorDash evolved AI evals from an engineering task into a cross-functional platform for non-engineers.
“cost is a number one concern these days”
AI agents evolve rapidly but evaluation frameworks fail to keep pace with model changes
“building a demo with AI is really easy but making it production quality is really hard”
Agent loops amplify LLM hallucinations by compounding errors across downstream steps
“one hallucinated fact in step two can poison step three, step four and everything downstream”
Uber built a multimodal AI agent to enhance merchant food photos at scale without looking AI-generated
“we're threading the needle here. We need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same.”
Arize founder argues observability is evolving from human dashboards to autonomous self-healing agent systems
“how do I build systems that autonomously fix themselves?”
AWS and Motorway cut AI agent error rates from 1-in-8 to 1-in-50 with a production eval pipeline.
“The agent gives a confident-sounding response, but how do you prove it works reliably with real money on the line?”
Lyft applies ML model evaluation rigor to AI agents before production deployment.
“if we are running um offline evaluations for our machine learning model before that goes to productions, I think we should do the same for AI agents as well”
LangSmith positions itself as a full-stack, agnostic agent engineering platform with autonomous improvement.
“LangSmith Engine is your agent for agent engineering.”
Agent failures stem from static retrieval that never learns from eval and observability signals.
“We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn.”
OpenGov built and scaled OG Assist, a production AI agent embedded across its government ERP product suite.
Evals are broken and often misleading, but engineers should still use them well in agentic workflows.
“evals are broken and you should use them anyway”
Benchling's AI agents pair LLMs with structured lab data to roughly halve drug-discovery-to-patient time.
“What we've seen is when you kind of put these models on top of the right data, the quality answers go way up.”
Most developers building agents lack visibility into what their agent does under the hood; observability and evals close that gap.
“you probably don't know what your agent is actually doing under the hood”
LangChain defines a four-phase agent development lifecycle: build, test, deploy, monitor.
“What is the agent development lifecycle? So the agent development lifecycle is the process that agent engineers use to build and improve their agents.”
smevals is a new open-source eval framework for comparing AI model and prompt configurations
“I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me.”
Video generation eval methods lag far behind generation quality, missing narrative coherence entirely.
“the hard part was never how to make video. The hard part was how do we generate good enough video and how do we judge if the video is good enough?”
YouTube Ads team built agent reliability via layered evals, critique agents, and strong tool foundations.
“the reliability of your agent is basically a function of the capabilities of the agent uh the guard rails and the evals”
NearForm uses AI agents and spec-driven development to iteratively build more reliable, secure AI agents.
“an agent is an LLM inside a an agentic loop, which is connected to tools and can retrieve context. That's it.”
Braintrust workshop teaches isolating signal from AI agent trace noise using observability tooling
Systematic study finds no evidence AI labs train models to draw pelicans on bicycles better
“Pelicans aren't drawn any better than other animals. Bicycles aren't drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”