That assumption is what \u201cbuilt backwards\u201d means. And it\u2019s more common than most teams realize until something breaks.<\/p>\n
Agents Aren\u2019t Entities. They\u2019re Systems.<\/p>\n
A production AI agent isn\u2019t a single intelligent thing.<\/p>\n
Rather, there is a set of interacting pieces with different responsibilities, failure modes, and levels of observability.<\/p>\n
The LLM is one of those components, not the whole system. Just one piece of it.<\/p>\n
It may sound obvious when you say it out loud. But the \u201cautonomous agent\u201d framing that dominated 2023 and most of 2024 kept pulling engineers toward a different mental model: one entity, one reasoning loop, everything handled by the model.<\/p>\n
All you need is tools, a good system prompt, and a hope that everything will fall into place.<\/p>\n
In contrast, engineers who have shipped real AI-based products rarely describe their systems that way. What they actually describe sounds a lot more like distributed systems architecture. <\/p>\n
Not because they read a book about design patterns, but because they got burned enough times that they started putting structure more seriously in their workflow.<\/p>\n
Building top-down, starting from \u201cwhat should this agent do\u201d and working backwards into tools and prompts, is quick to get started.<\/p>\n
It\u2019s also how you end up with a system where the model is responsible for too much, and nothing is individually debuggable.<\/p>\n
The architecture was decided by the goal, not by the engineering requirements. <\/p>\n
That\u2019s the backwards part.<\/p>\n
So What Really Goes Into a Production System?<\/p>\n
The abstract version is easy to nod along to. Here\u2019s what it actually looks like.<\/p>\n
Every production AI system I have seen that works cleanly has something like a decision layer, whether the team named it like this or not. It\u2019s the part where the model lives and does its actual job.<\/p>\n
The instinct is to push everything into this layer: parsing requests, managing memory, handling retries, resolving tool failures.<\/p>\n
This is okay if you\u2019re working in a Jupyter notebook. In production, under load, with real users, this becomes the part of your system where everything is everyone\u2019s fault, and most times, nothing can be debugged.<\/p>\n
The decision layer should do one thing well, and that is deciding what to do next, given a certain context that is already prepared for it.<\/p>\n
That\u2019s the whole job.<\/p>\n
Who prepares the context? Something else. Who acts on the decision? Also something else.<\/p>\n
That \u201csomething else\u201d is the orchestration layer, and in most well-built systems, it\u2019s genuinely just code: conditionals, asynchronous runners, retry handling, queue routing, maybe even a state machine depending on how involved the workflow is.<\/p>\n
![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/05\/mermaid-diagram-2026-05-23-171820-1024x490.png\")
Instead of expecting the model to do everything, treat it like just another component. Here, standard code does the heavy lifting with state and tools, so the LLM only has to worry about making the next decision. Image by author.<\/p>\n
Many teams reach for frameworks here because bare orchestration code feels too simple, like surely there\u2019s supposed to be more infrastructure.<\/p>\n
There usually isn\u2019t. <\/p>\n
The less magic this layer contains, the faster you\u2019ll find bugs when they appear. And they will appear.<\/p>\n
From experience, I learned this the hard way on a project where the orchestration lived inside a framework\u2019s execution model. Something was retrying tool calls in a way that was corrupting state downstream. <\/p>\n
We spent two days finding the issue. Two days for a bug that could have been resolved in no time at all if the retry logic had been three lines of Python I wrote myself.<\/p>\n
This leads us to the tools and execution layer, where all communication happens.<\/p>\n
Now, the tools and execution layer is where things talk to the outside world. This layer usually has just one job, and that is to take a well-defined input and then produce a predictable output.<\/p>\n
But the failure I kept seeing, and kept repeating, honestly, was tools that tried to be helpful by doing more than one thing. A single function that calls an API, updates a cache, and does other things. <\/p>\n
In a setup like that, when it breaks, you don\u2019t know where. Even when you try to replace the API, you\u2019re untangling logic that shouldn\u2019t have been tangled in the first place.<\/p>\n
Memory and state is where I\u2019d push hardest, because it\u2019s where most teams are most underprepared. <\/p>\n
Most teams think about memory as \u201cwhat the model knows.\u201d The more important question is what the system knows, and whether that knowledge is current.<\/p>\n
I remember one day when it took me an afternoon to debug what seemed to be a simple \u201cmodel hallucination.\u201d The model had kept referring to user preferences, which, however, had been updated twenty minutes ago.<\/p>\n
That\u2019s not a model problem. <\/p>\n
That\u2019s a systems problem. <\/p>\n
And it\u2019s surprisingly common.<\/p>\n
In multi-agent systems, specifically, shared state is where subtle failures breed. One agent updates something. The others don\u2019t know.<\/p>\n
Everyone proceeds confidently in slightly different directions. The output looks almost right, which is almost worse than looking wrong<\/a>.<\/p>\n And then there\u2019s evaluation and observability, which almost everyone always puts off until something goes wrong. I\u2019ve been guilty of this, too.<\/p>\n