As AI model capabilities advance at an exponential pace, the core methodology of the product manager role is undergoing a quiet but profound transformation. Dianne Penn, product lead for Anthropic’s AI Research & Labs team, recently offered a disruptive perspective on Lenny’s Podcast: for research-oriented product managers, evals have replaced product requirement documents (PRDs) as the core tool for defining user value and driving product iteration.
Penn joined Anthropic in 2023, when the entire product team consisted of just five engineers, with the API business run by a single engineer. Over three years, she has been involved in every model release from Claude 2 to Fable, and has been deeply engaged in incubating key products including Claude Code, MCP, Skills, tool use, and reasoning capabilities. Prior to that, she worked on Amazon’s Alexa AI team, and earlier in her career traded high-yield bonds at JPMorgan.
In Penn’s view, the underlying logic of product management in the AI era has fundamentally shifted. Under the traditional model, product managers write PRDs to describe product vision, align stakeholder expectations, and chart development paths. In a model-driven product system, the path to user value has become: identify user pain points, deeply understand failure trajectories, standardize them into reliably reproducible eval sets, then feed eval results back to the research team for targeted improvements.
“Evals are the new PRDs,” Penn said bluntly in the interview. She illustrated this shift with an early case study: during the Claude 2.0 era, the team repeatedly received user feedback that “Claude isn’t good at following instructions.” After drilling down layer by layer, they discovered that roughly 80% of so-called “instruction-following failures” were actually cases where the model couldn’t correctly output JSON in the required schema. The team then generated 30 to 40 failure cases to form an eval set, with each case containing a prompt, the model’s response, and a comparison against the expected golden answer. From then on, the eval ran automatically whenever a new model version was released. Today, that metric has stabilized above 99.9%—the pain point has effectively disappeared.
The core of this working method lies in converting vague user feedback into concrete, actionable signals for researchers. Penn gave an example: when users report that “Claude is hallucinating,” the team needs to determine further—is it a tool-use failure? A search or knowledge-integration failure? Or an alignment problem? Only by drilling down to that level of specificity can researchers design targeted evals and measure improvement.
Notably, Penn does not believe PRDs are dead. She pointed out that when the problem definition is very clear, evals can almost serve as a shorthand. But in scenarios involving ambiguous problems, cross-team alignment, and the need to reach consensus with stakeholders such as legal and safety, PRDs still hold irreplaceable value. For example, when the computer use feature had not yet launched, the team did not possess a clear set of user pain points; the product vision section of the PRD served the function of exploring “how to first enable a certain user group to use it effectively.”
Penn also emphasized the importance of leaders getting their hands dirty. She said that even senior product managers’ onboarding plans are identical to those of early-career employees: understand users, read user feedback (with consent), and talk to customers. “Someone who has never personally built an AI product can hardly judge what a great AI product or feature should look like,” she said. She herself maintains hands-on ownership of one or two model-related workflows to keep her mental model of users and model capability shifts current.
Reflecting on Anthropic’s trajectory, Penn identified several key turning points. The Golden Gate Claude project in early 2024, though live for only 24 hours and reaching about 2,000 people, gave the team its first realization that entirely new user experiences could be created and shipped at startup speed. The Opus 3.0 release established deep trust between the product and research teams; at the time the company had fewer than 200 people, and teams collaborated remotely over the holidays to complete training and release of that frontier model. Opus 4.5’s significance lay in being the first to combine frontier model capability with a frontier product experience—without a product vehicle like Claude Code, Opus 4.5’s intelligence level could not have been fully perceived by users.
On the emergent nature of model capabilities, Penn noted that the “discontinuous capability jumps” described in scaling laws papers mean a model may suddenly acquire an ability that the team may not detect in time. “If you don’t have evals, and you don’t have the corresponding testing systems, then these capability jumps may have already happened—and you wouldn’t even know it,” she warned.
On talent selection, Penn listed three core criteria: first-principles thinking, the ability to stay close to the details, and a grand long-term vision. She particularly emphasized that first-principles thinking means “figuring out what should be done to achieve the goal,” rather than “copying what has always been done in the past.” Anthropic’s most successful researchers tend to be both skilled at reasoning through problems and willing to dig into the underlying data from training runs and eval results.
Penn also shared her personal, distinctive way of using Claude. She built a Skill based on the book Crucial Conversations to help herself organize communication strategy before difficult conversations and judge the appropriate level of detail. “It’s almost like a personalized coach,” she said, stressing that AI’s value lies not only in boosting work-efficiency IQ but also in enhancing interpersonal EQ.
On the relationship between AI and human judgment, Penn believes that “hard-won judgment” will remain a core human value for the foreseeable future. Software engineering has already been profoundly transformed by AI, while fields like biology and life sciences are just at the starting point of the exponential curve. She also cautioned that forming one’s own opinions before deeply interacting with AI is critical; otherwise, one risks the degradation of independent thinking.
Notably, Penn revealed in the interview that Anthropic’s model release volume in the second quarter of this year has already surpassed the total of all four model families released in 2024. This accelerated pace places extremely high demands on team collaboration, which she described as a kind of “hive mind”—on the night before a release, even team members who are not core owners will voluntarily stay to help review blog posts, edit content, and design demos.
Against the backdrop of intensifying competition in AI products, Penn’s perspective offers the industry an important reference point: when model capability itself is no longer scarce, how to systematically convert capability into user value through a rigorous eval system is becoming the key variable that determines product success or failure.