![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/9964d2cf12e70e8e_1784472302_inline_2.jpg\")
<\/p>\nNick Ung has a blunt diagnosis for the state of AI agent evaluation across the industry. “If your LM-as-a-judge is just floating out there and no one is really using that score as a meaningful gate,” he said, speaking on the AI Engineer podcast alongside product manager Ashe, “then that LM-as-a-judge is not available.” The two Lyft data science veterans have spent two years building the evaluation infrastructure behind the company’s customer support AI, a multi-agent system built on LangGraph. What they learned along the way amounts to a systematic critique of how most organizations test their AI agents \u2014 and a concrete blueprint for doing it right.<\/p>\n
The problem, as Ung and Ashe describe it, is not that companies skip evaluation entirely. It is that the evaluations they run do not drive decisions. Scores float in dashboards without triggering rollbacks. Metrics like “response helpfulness” produce numbers that nobody knows how to act on. And teams skip the foundational work of looking at raw production data before writing evaluation criteria. The pair’s core insight is deceptively simple: an evaluation system must be built from the ground up as a launch gate, not a post-hoc report card.<\/p>\n
The Three Ways Agent Evaluations Fail<\/p>\n
Ung distilled the failure modes into a clear taxonomy. First and most damaging: the absence of a launch gate. If a score exists but no process uses it to block or approve a release, the evaluation is decorative. Engineering teams ship changes, the score fluctuates, and nobody connects the two. Second: generic, noisy judges. Lyft’s own early attempts leaned on pre-built metrics from the open-source DeepEval library \u2014 response helpfulness, conversation naturalness, and similar dimensions. “Those metrics were not actionable,” Ung said. “If response helpfulness is 0.5, then what do we do with it?” The judge became noise that eroded trust across the organization. Third: no regression detection mechanism. Without a clear system to catch degradations in production and assign ownership, even well-designed evaluations decay in relevance.<\/p>\n
The dependency chain Ung laid out is unforgiving: “If you don’t look at the data, you won’t be able to create meaningful criteria or labels. And if you don’t have labels, you won’t be able to evaluate your judges. And if you are not evaluating your judges, you don’t know if your agentic pipeline is working as expected.” Every link in that chain is load-bearing, and Ung argued that most teams break it at the first step \u2014 skipping raw data analysis in favor of importing someone else’s evaluation template.<\/p>\n
Why Your Test Users Are Too Polite<\/p>\n
Even when teams build custom evaluations, the simulations themselves often lie. Lyft’s offline simulator was inspired by the Taobench paper from Sierra AI, which proposes a three-component architecture: a user language model that roleplays a customer, an agent language model that handles the interaction, and a judge that scores the full conversation trajectory. But Lyft’s first implementation produced results that were, as Ashe put it, “too good to be true.”<\/p>\n
The initial user simulator used a frontier model \u2014 GPT-4 class \u2014 prompted to behave as a Lyft customer. The bot it produced was unfailingly polite and verbose, patiently explaining issues in complete paragraphs. Real Lyft users, by contrast, send messages like: “I reported a lost item yesterday. Still no update. This is ridiculous. I need a human now.” The gap between simulated and real user behavior inflated the agent’s offline pass rate to roughly 90%, a number that bore no relationship to production performance.<\/p>\n
The fix was to fine-tune the user language model on actual production verbatims. After training on real customer messages \u2014 terse, impatient, frequently angry \u2014 the simulated users began producing realistic dialogue, and the pass rate dropped to an honest level. “If you have an eval that’s too easy, that doesn’t give you any real production insights into how your AI agent is actually going to perform,” Ashe noted.<\/p>\n
To capture behavioral diversity, the team also built user personas inspired by Microsoft’s UserAlien research:<\/p>\n
PersonaBehaviorBypasserImmediately tries to escalate; refuses to interact with AIRefund seekerDemands financial compensation; tests escalation boundariesAI skepticRepeatedly questions the bot’s ability; insists on human verification<\/p>\n
These personas ensure the simulator covers the long tail of real user behavior rather than defaulting to a single, unrealistically cooperative archetype. Lyft also deploys deterministic, code-based assertions alongside language model judges \u2014 verifying, for example, that a refund was actually granted by checking the agent’s tool calls against expected values. These function like unit tests for agent behavior: fast, unambiguous, and impossible for a language model to fudge.<\/p>\n
<\/p>\n
Binary Judges Beat Generic Scores<\/p>\n
The most consequential decision Lyft made was abandoning graded, multi-dimensional scores in favor of task-specific binary outcomes. Ashe and Ung argued that the only way to make a language-model judge consistently useful is to define pass\/fail rubrics in close partnership with domain experts. An “education rubric,” for instance, measures whether the agent escalated too soon or attempted to educate the user too many times. The judge outputs a single bit: pass or fail.<\/p>\n
This reframing transforms the evaluation problem into a familiar one: classifier validation. The process Lyft follows mirrors traditional machine learning workflows. Domain experts label roughly 100 examples with ground-truth pass\/fail judgments. The data is split into few-shot examples for the judge’s prompt, a development set, and a held-out test set. The team iterates the judge’s prompt against the dev set, measuring precision and recall at each step, then validates against the test set. “We can use these pre-built eval metrics as a baseline, but we shouldn’t use them as our core eval metrics because we want eval metrics to be actionable and tied to the business outcome,” Ashe said.<\/p>\n
There is a deeper point embedded in this methodology. Ashe stressed that evaluation criteria cannot be defined upfront and frozen. “We cannot define the criteria beforehand and then evaluate agents against them,” she said. The notion of quality evolves as more examples are graded and new failure modes emerge. Criteria must be co-developed alongside the model, refined continuously through observation rather than locked in at project kickoff. This principle cuts against the instinct to write a rubric once and automate everything downstream.<\/p>\n
Every Score Needs an Interval<\/p>\n
One of the most preventable errors Ashe identified in evaluation reporting is the bare point estimate. A four-point gain on only 50 samples \u2014 say, from 84% to 88% \u2014 tells leadership nothing without a confidence interval. The improvement could be real or entirely random. “We can reserve the expensive rigor for the moments when the number actually gates something,” Ashe said, “but every score needs an interval.” The team advocates for proper sampling and paired experimental designs to detect true improvements, reserving the deepest statistical work for launch-gating decisions while maintaining baseline rigor across the board.<\/p>\n
This discipline connects directly to the actionability principle. Ashe described a ruthless filtering process: “We only keep the metrics that change a decision. We remove all the noise.” If a metric fluctuates without ever triggering a different course of action \u2014 a rollback, a prompt change, a retraining cycle \u2014 it has no place in the dashboard. The goal is not comprehensive measurement; it is decision-quality signal.<\/p>\n
The Error Analysis Loop Never Stops<\/p>\n
Ashe was unequivocal on this point: “The error analysis loop is something which runs continuously. It’s not a one-off audit.” The loop follows a disciplined cadence \u2014 weekly or bi-weekly \u2014 starting with deep dives into raw production traces, pinpointing specific failure modes, and forming fresh premises about what needs to improve. Those premises feed back into three distinct improvement levers:<\/p>\n
LeverScopeExampleModel learningPost-training, updating model weightsFine-tuning a model for a specific customer-support taskContext learningImproving what the agent seesUpdating knowledge bases, memory storesHarness learningUpgrading system prompt, tool schemas, control flowChanging routing logic, retry policies<\/p>\n
The distinction between these levers matters because teams often collapse them into a single bucket \u2014 “make the model better” \u2014 when the fix might actually live in the context the agent receives or the control logic governing its behavior. Lyft’s error analysis loop is designed to route failure modes to the correct lever, not default to model retraining as the universal remedy.<\/p>\n
Underpinning the entire loop is a practical infrastructure requirement: annotation queues. Domain experts need a clean interface \u2014 the team mentioned LangSmith and Langfuse as examples \u2014 where they can label traces and those labels flow directly into the evaluation pipeline. Without this, the loop breaks. Experts cannot efficiently review data, labels stop arriving, judges degrade, and the whole system loses calibration.<\/p>\n
![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/9964d2cf12e70e8e_1784472388_inline_5.jpg\")
<\/p>\n
From Scattered Scripts to a Config-Driven Harness<\/p>\n
When Lyft first built its offline simulator, the codebase was, in Ashe’s description, “scattered as scripts across different notebooks.” The team is now building a systematic eval harness with several design principles. Configuration lives in YAML files that analysts and data scientists can edit without touching engineering code. The harness is built around primitives \u2014 task, dataset, persona, language model adapter, evaluator \u2014 that compose into test suites. It supports parallelism capable of handling “thousands if not tens of thousands of examples” in reasonable time. And it integrates at multiple gates: local development, pre-commit hooks, CI\/CD pipelines, and a dedicated regression test suite.<\/p>\n
Looking ahead, Lyft is exploring two frontiers. The first is post-training: with thousands of real user signals now collected, the team is investigating fine-tuned models for specific customer-support tasks and framing reward modeling to enable reinforcement learning. The second is cross-agent standardization \u2014 unifying evaluation for all sub-agents in the multi-agent system so that a regression in any component is immediately traceable and attributable.<\/p>\n
The cost question looms over this entire architecture. Fine-tuning a user language model, running thousands of multi-turn simulations, and maintaining annotation queues staffed by domain experts is expensive. Lyft’s bet is that the alternative \u2014 deploying agents without trustworthy evaluation \u2014 costs more in the long run, in degraded customer experience and engineering fire drills. For teams building user-facing AI agents, Ung and Ashe’s framework sets a new bar: if your evaluation does not gate a launch, surface a specific failure mode, and feed directly into an improvement loop, you are not really evaluating at all. You are generating numbers that nobody will ever use to make a decision.<\/p>\n","protected":false},"excerpt":{"rendered":"Nick Ung has a blunt diagnosis for the state of AI agent evaluation across the industry. “If your…\n","protected":false},"author":2,"featured_media":111219,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[405,53707,7537,57184,57186,9396,57189,57185,57188,20588,57190,57183,51680,57187],"class_list":["post-111218","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-ai-agents","tag-ai-engineer-podcast","tag-artificial-intelligence-agents","tag-ashe","tag-deepeval","tag-gpt-4","tag-langfuse","tag-langgraph","tag-langsmith","tag-lyft","tag-microsoft-useralien","tag-nick-ung","tag-sierra-ai","tag-taobench"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/111218","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=111218"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/111218\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/111219"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=111218"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=111218"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=111218"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}