![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/d3b84bc9fd2ae6d7_1784537243_inline_1.jpg\")
<\/p>\nIf you handed a frontier AI coding agent a folder of 2,000 video files and asked it to find every clip containing a person, the agent would fail four times out of five. Not because the model is weak \u2014 but because it cannot see what is inside the files. That is the argument Dmitry Petrov, founder of the open-source DataChain project and creator of DVC, made on the AI Engineer podcast.<\/p>\n
The number is 21%. That is the baseline accuracy Anthropic measured when its coding agents attempted data projects without what Petrov calls a “data harness” \u2014 a structured metadata layer that translates the chaotic internal structure of video, sensor telemetry, and robot logs into something an agent can query. “The accuracy for data projects on their agents is only 21% until you add specific data harnesses to them and provide context,” Petrov said.<\/p>\n
The problem has nothing to do with model intelligence and everything to do with physics. Or rather, what Petrov calls “the other physics” \u2014 the peculiar topology of unstructured data.<\/p>\n
The Neutron Star Inside a Folder<\/p>\n
On the surface, a directory of 91 dashcam videos looks manageable. In Petrov’s live demo, those files generated 100,000 detection records after 24 minutes of YOLO processing. Scale that to thousands of videos, decompose each detected object into sub-objects with their own bounding boxes and confidence scores, and the numbers explode. “It looks like a neutron star \u2014 small size on the surface, but the mass is tremendous, more than our sun,” Petrov explained. “If you analyze thousands of videos, it easily goes into millions, and if you go deeper to the objects, it can easily be multiplied by ten, twenty, even hundreds.”<\/p>\n
Teams confronting this data typically fall into one of two traps. The first: dump metadata as millions of tiny JSON files into S3 alongside the raw images. The result is horrible latency and zero consistency. The second: move everything into a centralized database. That splits the stack into two programming languages \u2014 Python for compute, SQL for storage \u2014 and two incompatible systems that most researchers refuse to navigate. “This stack is useless for most of the researchers because they don’t want to deal with this complexity,” Petrov noted.<\/p>\n
His alternative keeps a single language, Python, for both code and schemas. Pydantic data models define the structure, and a transpiler converts them to SQL transparently. Researchers never write a JOIN statement.<\/p>\n
<\/p>\n
Six Layers Agents Need but Do Not Have<\/p>\n
Drawing on OpenAI’s published framework for data-agent context and Anthropic’s findings on agent failure modes, Petrov enumerated the six components that distinguish a working data harness from a naive integration.<\/p>\n
ComponentWhat It DoesWhy It MattersVisual layerUnderstands binary file formats \u2014 video containers, sensor logs, compressed archivesWithout it, agents cannot see the internal structure of files; they hallucinate or fail entirelyExecution engineDistributes Python functions across threads or machines based on file boundariesReplaces ad-hoc orchestration with a pattern that maps function \u2192 file \u2192 schema rowIncremental checkpointSaves partial results; on failure, catches up only the unprocessed filesPrevents losing expensive compute when a bug or API error kills a batchSelf-testing loopRuns fast validation queries before making results available”In data there is usually only one correct answer” \u2014 quality matters more than in softwareMetadata warehouseStores extracted information in dimensional models optimized for analytical SQLTurns expensive Python-on-raw-data questions into millisecond SQL queriesKnowledge baseGenerates Markdown files per dataset: context, schema, source code, statisticsShares investment so no teammate re-runs the same pipeline<\/p>\n
The checkpointing layer deserves special attention because it addresses a uniquely expensive failure mode. When a batch of LLM calls or object detection runs dies partway through \u2014 because of an API rate limit, a memory error, or a malformed file \u2014 the default behavior is to restart from zero. Petrov’s response is blunt: “You should be able to recover and catch up with all the data that it’s already processed. So you don’t want to waste your resources.” Every lost compute cycle is money burned.<\/p>\n
![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/d3b84bc9fd2ae6d7_1784537416_inline_2.jpg\")
<\/p>\n
A Function, a File, a Row<\/p>\n
Petrov demonstrated the harness in action with a deceptively simple code pattern. A Python function takes a file path pointing to S3, runs a YOLO model, and yields Pydantic objects \u2014 detections with bounding box, confidence score, class label, frame ID, and timestamp. No annotations. No boilerplate orchestration code.<\/p>\n
Under the hood, the execution engine distributes this function across machines by file boundary. “I just need to run it in 40 machines \u2014 that’s how easy it is,” Petrov said. The scheduler maps function to file, understands the schema output, and routes results into a DuckDB-based warehouse. The demo used a local database, but production deployments can target any SQL engine.<\/p>\n
When the demo finished processing 91 dashcam videos, Petrov asked the agent a simple question: “How many videos have people in them?” The answer \u2014 82 out of 91 clips \u2014 returned in under a second. The generated code was a single line: dataset.filter(label=”person”).count(). Transparent, debuggable, and auditable by anyone on the team.<\/p>\n
The Vertical Stack<\/p>\n
The architecture has three tiers. At the bottom sits the raw mass \u2014 terabytes of physical data in object storage that no one makes sense of directly. The middle layer runs expensive model calls to extract structured metadata, feeding a dataset governor that organizes slices of the warehouse. At the top, a knowledge base of Markdown files captures everything the team learned: the context of the original session, the Pydantic schema, preview rows, and the exact source code used.<\/p>\n
The loop is intentional. When an agent later encounters a directory that already has an associated dataset, it reads the Markdown file and reuses stored results instead of recomputing. “You’re paying double, triple, quadruple price to solving the same problem,” Petrov warned. “If you discover this neutron star\u2026 it’s better to share this information with people.”<\/p>\n
This lineage approach fundamentally changes how teams interact with unstructured data. Rather than treating each analysis as a throwaway script, the knowledge base makes the investment permanent. It also addresses a deeper truth Petrov surfaced about the nature of data work itself. “In software projects, there’s a lot of ways how to you can solve a particular problem,” he said. “In data, there is usually only one way and only one current correct answer to solve the problem.” That precision \u2014 that there is a right answer \u2014 means validation and reproducibility are not optional. They are the entire job.<\/p>\n
Where the Harness Stops<\/p>\n
Petrov was candid about the limits of his approach. The harness accelerates analytical queries \u2014 “how many videos contain people” \u2014 but it does not answer generative questions. Ask “describe what happened in this unusual video” and you still need expensive vision-to-text inference. “It doesn’t save you from heavy inference; it saves you from doing it twice,” he said.<\/p>\n
The approach also assumes Python-dominant workflows. Teams in C++ or Rust environments may find the Pydantic-centric stack foreign territory. And while the 21% accuracy baseline is a powerful opening statistic, Petrov did not present a controlled before-and-after comparison using his own harness \u2014 the live demo showed a successful end-to-end run, but listeners are left to trust the architecture rather than verify it against a benchmark.<\/p>\n
Three tensions remain unresolved. First, the metadata layer solves the query problem beautifully but leaves the inference problem untouched \u2014 and inference is where most of the money gets spent. Second, the knowledge base depends on team discipline; nobody forces a researcher to write the Markdown file. Third, DataChain enters a landscape already crowded with middleware projects \u2014 LangChain for LLM orchestration, Weights & Biases for ML experiments \u2014 each racing to define the primitives for agentic workflows.<\/p>\n
The wager Petrov is making is specific and defensible: the winning primitive for physical-data agents is not a stronger model or a bigger context window. It is a harness that makes the neutron star digestible. “Their intuition pushes them in a wrong direction because the laws of physics change,” he said. “You don’t use stronger models. Everyone uses frontiers. Instead, you build a data harness that understands the laws of this physical data.”<\/p>\n
For the tens of thousands of engineers now staring at folders full of robot telemetry, dashcam footage, and sensor logs, the message is clear. The bottleneck is not the model. It is the operating system the model is asked to run on. And until that operating system ships, even the smartest agent is running blind.<\/p>\n","protected":false},"excerpt":{"rendered":"If you handed a frontier AI coding agent a folder of 2,000 video files and asked it to…\n","protected":false},"author":2,"featured_media":111748,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[405,53,7537,2798,2317,416,57411,57410,57414,57412,11432,157,36192,57415,32003,57413],"class_list":["post-111747","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-ai-agents","tag-anthropic","tag-artificial-intelligence-agents","tag-claude-code","tag-codex","tag-copilot","tag-datachain","tag-dmitry-petrov","tag-duckdb","tag-dvc","tag-langchain","tag-openai","tag-pydantic","tag-s3","tag-weights-biases","tag-yolo"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/111747","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=111747"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/111747\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/111748"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=111747"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=111747"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=111747"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}