Google DeepMind’s Philipp Schmid has a blunt message for the thousands of engineering teams racing to ship AI agents: you are almost certainly doing skills wrong. His team analyzed over 50,000 skills deployed in coding agents and found that the overwhelming majority were AI-generated, untested, and frequently made agent performance worse rather than better. Worse still, almost none included any kind of evaluation to detect the regression.
“Human-written skills are the best we can provide,” Schmid said, speaking on the AI Engineer podcast. “AI-generated skills can impact performance negatively.”
The finding lands at a moment when companies across the technology sector are betting heavily on agent-based products — even as Meta’s Mark Zuckerberg recently admitted his own company’s AI agent push “hasn’t really accelerated in the way that we expected,” triggering a $70 billion market value wipeout. Schmid’s argument is that the problem isn’t agent technology itself. It’s that teams are treating skills — the instruction files that tell agents how to behave — as documentation rather than as code. And code without tests fails.
The Two Types of Skills, and Why One of Them Should Die
Schmid draws a line through the skill landscape that most teams haven’t thought about. There are capability skills, which teach a model to do something it can’t yet handle consistently — scaffolding a React app, tracing a distributed log, generating a specific API call pattern. These skills exist because the underlying model has a gap. And because models improve with every release, these skills have an expiration date.
Then there are preference skills, which encode durable, company-specific knowledge: your team’s code review workflow, your deployment conventions, your style guide. The foundation model will never absorb these on its own because they’re specific to your organization. These skills are permanent investments.
Skill typePurposeLifespanRetirement triggerCapabilityTeaches model something it cannot do consistentlyTemporaryEval shows equal performance without skillPreferenceEncodes company-specific workflows, styles, conventionsDurableNever — replaced only if workflow itself changes
“Capability skills are temporary,” Schmid said. “As models improve, these skills will become unnecessary and can be removed.” The problem, he argues, is that most teams never remove them. They keep capability skills in production long after the model has learned the behavior natively, burning tokens on instructions that add zero value.
The only way to know when a capability skill has become dead weight is to run evaluations with and without it — what Schmid calls an ablation test. Without that test, teams are guessing. And the SkillBench data suggests they’re guessing wrong.
The SkillBench Finding: Most Skills Are AI-Generated and Unverified
The SkillBench analysis of those 50,000 skills revealed a structural problem. AI-generated skills — produced by asking one language model to write instructions for another — consistently underperformed human-written ones. The AI-generated files also suffered from bloat, drifting into territory Schmid described as “no-ops”: instructions that sound useful but change nothing about the model’s output. Phrases like “write clean code,” “ensure readability,” or “follow best practices” add token cost without altering behavior.
The fix is simple but rarely applied. “Run evals with your skill loaded and without your skill loaded,” Schmid said. “Only that way will you know when you can retire a skill.” An instruction that scores identically in both conditions is dead weight. Remove it.
Schmid also offered a concrete sizing rule: keep skill bodies under 500 words. Beyond that threshold, the signal-to-noise ratio degrades sharply, and the no-op problem compounds. The 500-word ceiling emerged from the SkillBench data and is now baked into Schmid’s team’s internal review process. Any skill file exceeding it gets flagged for scrutiny.
How Skills Actually Work — and Why Your Description Is Failing
The architecture Schmid described operates on a principle he calls “progressive disclosure”: the model only pays the token cost of the layer it actually needs to access, avoiding the waste of loading an entire 5,000-word instruction file into every context window.
The first layer is the title and description, which are always present in the model’s context. The description — roughly 100 to 200 tokens — must tell the model why to use the skill, how to use it, and critically, when not to use it. Schmid estimates that roughly half of all skill failures originate in this layer. A description that says “use for web development” will over-trigger in any JavaScript context, even when the skill was built for React-specific patterns. The fix is to include explicit negative cases in the description.
The second layer is the skill body, which is read into context only when the model decides to invoke the skill. This is where the actual instructions live. Schmid’s advice here cuts against common practice: favor directives over essays. “Use the Interactions API for multi-chat” outperforms “The Interactions API is recommended for multi-chat because it handles session state.” Models respond to instructions, not explanations.
The third and deepest layer is reference files — optional documents containing platform-specific or branch-specific detail that the model navigates to only when needed. This keeps the main skill body lean while still giving the agent access to deep context.
The architecture matters because Schmid sees a fundamental divide between how developers use agents and how customers use them. “When you build an agent inside your application for consumers or customers, they have no idea about what a skill is,” he said. “They don’t start their prompt with ‘use customer support skill to help me refund’ or ‘use refund skill to help me solve my problem.'” In production, only model-invoked skills — where the model decides based on the description — actually matter. If the description fails, the skill never fires.
A Concrete Eval Framework: 117 Tests That Block Bad Changes
Schmid walked through the evaluation harness his team built for a specific capability skill: teaching Gemini the Interactions API, which was released after the model’s training cutoff, meaning Gemini had zero native knowledge of the interface. The team constructed 117 test cases drawn from real user traces and synthetic edge cases.
The harness has three components. A JSON file stores test cases with fields for the user’s prompt, the programming language (TypeScript or Python), whether the skill should activate, and regex-based assertions that check for correct SDK imports, model IDs, method names, and the absence of deprecated patterns. A Python runner invokes the agent harness with each test case and captures the output. Assertions then validate the result.
The critical operational rule is that the eval runs on every change to the skill file. A merge is blocked unless the change improves scores or adds new test cases. Schmid noted that regex-based assertions are cheap and catch the majority of failures — wrong import paths, deprecated API calls, incorrect model versions. For more nuanced behavioral checks, an LLM-as-judge component with a rubric can be layered on, but the regex layer alone catches most regressions.
Ten Rules That Change How Teams Ship Skills
Schmid distilled his team’s experience into ten operational rules. Several are non-obvious and worth pulling out.
First, test across harnesses. A skill that works flawlessly with Gemini may fail with Claude Code or Codex. If your users operate across multiple agent frameworks, evaluate on all of them.
Second, run multiple trials. Agent outputs are non-deterministic. A single pass or fail is noise. Schmid recommends three to six trials per test case and measuring reliability as a percentage.
Third, use isolated workspaces. If the agent can find the skill context from prior chat history or cached files, it may “cheat” by never actually triggering the skill. The eval needs to simulate a fresh session.
Fourth, test outcomes, not paths. Do not require the model to invoke the skill on turn one. It can activate on turn five after exploring. The assertion should check whether the final output is correct, not whether the model followed a specific sequence of steps.
Fifth, keep the eval after retiring the skill. If a model improvement makes a capability skill redundant, don’t delete the test suite. Keep it as a regression detector. If the next model release regresses on that capability, the eval will catch it, and you can reintroduce the skill.
Sixth, retire aggressively. This is the rule Schmid emphasized most. Model releases improve fast. A capability skill that was essential six months ago may now be pure overhead. Evals make the retirement decision objective rather than gut-feel.
“If you have those type of use cases,” Schmid said, referring to rigid step-by-step workflows, “you should not use skills. You should write a script.” His point is that skills should define goals and constraints, not paths. The model knows how to change a port; tell it the constraints and let it navigate. Over-specification of workflows is a signal that the problem is better solved with deterministic code.
The Gaps Schmid Didn’t Close
The framework Schmid described is rigorous but assumes infrastructure that most teams lack. Running isolated agent invocations at scale, with CI/CD integration that blocks merges on eval failure, requires investment that many organizations have not made. Schmid’s suggestion to start with 10 to 20 manual test cases is honest about this gap — even a minimal test set will surface failures — but the distance between “run 10 manual tests” and “automated evals on every skill change” is where most teams will remain stuck.
There is also an open question about the 500-word ceiling on skill bodies. As context windows grow and token costs fall, the economics of longer skills may shift. But Schmid’s core insight — that longer AI-generated skills accumulate no-ops — suggests the problem is not cost but signal degradation. A longer instruction file full of filler is worse than a short one, regardless of whether the tokens are cheap.
The more immediate implication is for investors trying to separate signal from noise in the agent boom. Companies that have built internal eval infrastructure for their skills are operating with a fundamentally different quality bar than those that have not. The absence of evals — not the presence of flashy agent demos — is the variable that predicts whether a skill is helping or hurting. And on Schmid’s reading of 50,000 skills, most teams are on the wrong side of that line.
That gap will close, but the closure will be uneven. Teams building preference skills for internal developer tools have the incentive and the expertise to adopt eval-driven workflows. Teams building customer-facing agents where end users have no concept of skills face a harder problem: the description layer is the only interface, and if it fails, the skill never activates. In those products, description quality isn’t a nice-to-have. It’s the entire product. And right now, Schmid’s data suggests, it’s the layer most teams are getting wrong.