![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/12fccc7290a18160_1784828526_inline_2.jpg\")
<\/p>\nThe gap between a cool demo and a tool you actually trust is measured in failure rates. Antje Barth, a Member of Technical Staff at Amazon’s AGI Lab, put a number on that gap during her talk at the AI Engineer Worlds Fair in July 2026: end-to-end knowledge-work agents succeed “maybe 60 to 80 percent of the time.” At that rate, she noted, the math turns against adoption fast \u2014 “if your agent one in four times deletes a database, you will never touch that agent again.”<\/p>\n
The problem is not that agents lack ability. “What we figured out, clicking was actually the easy part,” Barth said. “What we didn’t solve is the actual work.” Agents can click, scroll, type, call APIs, and fill out forms with high reliability on isolated tasks. But hand them a cross-system workflow \u2014 onboarding a new employee, which means setting up accounts across five different applications, posting to Slack, booking introductory meetings, ordering a laptop \u2014 and they “still struggle to do this end to end because the real work lives within the seams of all of those different applications.”<\/p>\n
Barth’s core argument is that the industry has been solving the wrong problem. Capability is largely sufficient. What has not been built is a mechanism for trust, and trust requires verifiability.<\/p>\n
How Code Crossed the Trust Threshold \u2014 and Knowledge Work Did Not<\/p>\n
Barth traced a familiar arc for anyone who has watched AI coding tools evolve: autocomplete gave way to function generation, which gave way to agents that open pull requests. At each step, developers gradually stopped reviewing every line. “Once in a time we were able to just really read it and make sure it’s correct,” she recalled. “I think most in the audience here can still relate to that. These days, hardly anyone is still doing that.”<\/p>\n
The difference between code and knowledge work is structural.<\/p>\n
DomainVerifiabilityTrust achieved?Key failure modeCode generationHigh \u2014 unit tests, compilers, CI pipelinesYes \u2014 agents now trusted to open PRsNone; failures caught automaticallyKnowledge workLow \u2014 no unit test for “Did the report land?” or “Is the design on-brand?”NoAgents fire actions without confirming results; no shared context with user<\/p>\n
“Knowledge work is messy and the real world is really messy,” Barth said. “Did the report I created land? Is the design on brand? Did it get it what I actually meant? There is no unit test that can answer those questions.” Verification, she argued, “hits the wall right where most of our work lives.”<\/p>\n
Perception Agents: Stop Outsourcing Awareness to the User<\/p>\n
Humans handle messy cross-system work by collaborating \u2014 two people looking at the same screen, pointing at elements, discussing what needs to change. The missing property in current agents is that shared context. “You don’t necessarily need a bigger brain,” Barth said. “What you need is this shared context.”<\/p>\n
Today’s agents typically fire an action and move on. They do not watch the result. They cannot confirm whether a change actually took effect. Barth’s proposal, borrowed from robotics, is a perceive \u2192 plan \u2192 act loop adapted for screen-based work. The perceive step means an agent “takes in the screen the way you do, not scrape the code behind the page, but what’s actually rendered, the layout, the state, what just changed.” That lets the agent verify its own output instead of “firing off those actions and then hoping.”<\/p>\n
The contrast between current tool-use agents and the perception-agent target is stark.<\/p>\n
PropertyCurrent tool-use agentPerception agent (target)InputAPI calls, scraped DOM, text promptsRendered pixels plus shared context (annotations, in-meeting audio)OutputFires action, moves onActs in UI, observes result, verifies, optionally recoversVerificationNone \u2014 delegated to humanBuilt-in: visual and user-flow checksInteractionTurn-taking \u2014 human prompts, waits, re-promptsReactive \u2014 agent can suggest in real time while human worksAPI dependencyRequires backend API or scraped dataWorks off any rendered interface, even apps without APIs<\/p>\n
The shift eliminates two inefficiencies at once. Users no longer need to write long natural-language descriptions of what they want because they can point at elements on screen. And agents can keep pace with human work rather than operating as a separate, disjoint interlocutor.<\/p>\n
<\/p>\n
Two Open-Source Tools That Make the Idea Concrete<\/p>\n
Barth’s team released the first two building blocks of a perception-agent harness on GitHub. Both are designed to work on existing web applications without any backend integration.<\/p>\n
Annotation: Let the User Point Instead of Describe<\/p>\n
The annotation tool is a Chrome extension. A user selects an element on a rendered page by drawing a box around it, or hovers to find the right target. The extension captures the element’s location, style, and associated feedback \u2014 “change this heading to red,” “double the font size.” It produces a self-contained summary that the agent consumes. “There is no back and forth anymore because you captured exactly what you saw on screen and the agent can see the same thing,” Barth explained.<\/p>\n
Verification: Automated Checks Against a Design Spec<\/p>\n
The verification framework takes a design.md file that defines rules \u2014 colors, components, layout \u2014 and converts them into automated checks. It performs two types of verification. Visual checks ensure every element is on-brand and correctly laid out. User-flow checks have the agent walk through a realistic scenario, such as adding a task and then deleting it, to validate that the experience works as expected. The output is a report flagging which tests passed and which failed. The human then decides whether to update the specification or instruct the agent to fix a violation.<\/p>\n
Barth was candid about the early stage of the work. “These patterns can only get better if more people are using them, building on top of them, breaking things,” she said.<\/p>\n
Beyond the Screen: An Agent That Listens In<\/p>\n
A live experiment with a collaborator named Giovanni extended the perception concept beyond pixels. The pair sat in a meeting booth wearing B wearable devices, which provide real-time transcription. As they discussed design changes to a website \u2014 “turn the background to yellow, turn the heading to red, and change an emoji” \u2014 the transcript was summarized and sent directly to an agent through an “Apply” button. The agent implemented the changes and immediately kicked off verification. In this case, the color scheme happened to fall within the approved design rules. Had it not, the agent would have flagged the violation, leaving the human to decide whether to update the spec or reject the change.<\/p>\n
The implication is that perception agents can ingest context from multiple modalities \u2014 screen, speech, gesture \u2014 and execute with fully traceable verification, collapsing the latency and friction of the turn-taking chatbot model that still defines most AI assistants.<\/p>\n
The following flowchart maps the decision logic Barth described. The key addition compared to today’s agents is the verification gate: an action is not complete until the agent has confirmed its own output.<\/p>\n
![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/12fccc7290a18160_1784828700_inline_4.jpg\")
<\/p>\n
The Work Is Open-Source, the Context Is Pivotal<\/p>\n
Barth’s talk landed during a turbulent week for Amazon’s AGI division. The company confirmed job cuts within the unit on the same day, framing the move as a strategic sharpening under CEO Andy Jassy. The AGI group, which builds large-scale foundation models and develops autonomous AI tools like Amazon Nova Act, has also seen leadership departures: Rohit Prasad exited late last year, and AGI Lab head David Luan left in February. The division was consolidated under Senior Vice President Peter DeSantis in December.<\/p>\n
The restructuring underscores the pressure to deliver near-term customer impact from research that can feel years away from revenue. And yet Barth’s work points in the opposite direction: the unlock for agent adoption may not be a flashier model, but the unglamorous infrastructure of shared context and self-verification. Her lab’s open-source release is an invitation for the community to build on those patterns.<\/p>\n
Buying a bigger model will not fix an agent that cannot confirm whether it deleted a database. The frontier, Barth suggested, has shifted from what an agent can do to what an agent can prove it did.<\/p>\n","protected":false},"excerpt":{"rendered":"The gap between a cool demo and a tool you actually trust is measured in failure rates. Antje…\n","protected":false},"author":2,"featured_media":117146,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[6744,59531,53410,58939,1731,59530,3013,59535,59533,59532,53413,59534],"class_list":["post-117145","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agi","tag-agi","tag-ai-engineer-worlds-fair","tag-amazon-agi-lab","tag-amazon-nova-act","tag-andy-jassy","tag-antje-barth","tag-artificial-general-intelligence","tag-b-wearable-device","tag-chrome-extension","tag-perception-agents","tag-peter-desantis","tag-verification-framework"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/117145","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=117145"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/117145\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/117146"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=117145"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=117145"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=117145"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}