sitting closely with this topic, and it brought back some experiences from working on a couple of projects.<\/p>\n
Take this scenario: you ship an LLM-powered feature, the demo is clean, and all stakeholders are happy. Then three weeks into production, something breaks in a way nobody can explain.<\/p>\n
You spend an afternoon staring at logs that tell you what happened but not why.<\/p>\n
Then it turns out that the framework swallowed the context somewhere between step three and step four, and now you\u2019re reading source code you didn\u2019t write.<\/p>\n
That is not a bug report; it is a wakeup call about the architecture.<\/p>\n
Frameworks like LangChain let engineers build LLM-powered systems without first understanding how those systems work under pressure. At first, that sounds like the cavalry has arrived.<\/p>\n
But trust me, the cost doesn\u2019t show up until you\u2019re deep in a production incident, and now you\u2019re stuck wondering why your agent skipped the verification step it was supposed to run.<\/p>\n
This post is about that cost and why more engineers, after discovering it, are now building the orchestration layer by themselves.<\/p>\n
Give LangChain Its Credit<\/p>\n
I remember watching a colleague build a working RAG pipeline in about forty minutes sometime in early 2023.<\/p>\n
He went from the vector store through the retrieval chain, prompt templates, and the LLM call, all connected by lunchtime.<\/p>\n
Six months prior, that would have been at least a two-week project.<\/p>\n
Come to think of it, that\u2019s actually how and why LangChain spread so fast. <\/p>\n
Most engineers hadn\u2019t built LLM applications before. Nobody had strong opinions about the right way to structure a retrieval chain or manage conversation memory and other stuff like that. <\/p>\n
LangChain showed up with answers that were modular, composable, and documented, and of course, teams grabbed them immediately, including mine.<\/p>\n
So when I say it creates problems in production, I\u2019m not being dismissive. It was optimized for the phase most teams were in when they adopted it. The problems came later, when the phase changed.<\/p>\n
Where the Abstraction Breaks<\/p>\n
When I was learning object-oriented programming in my sophomore year, one of the first concepts that clicked was abstraction: hiding the internal details of how something works and only exposing what the user needs.<\/p>\n
LangChain applies that same idea to LLM orchestration. It hides a lot of what\u2019s happening inside your system so you can move faster.<\/p>\n
But production AI systems demand something that cuts against that: clarity. <\/p>\n
You need to know exactly what your system did, in what order, with what inputs, and why. Not roughly. Exactly.<\/p>\n
Abstractions trade that visibility for speed. That\u2019s a fair trade at first, until the hidden complexity becomes the very thing you need to understand.<\/p>\n
And it shows up in more ways than one.<\/p>\n
Debugging is worse than it sounds: When a multi-step chain gives the wrong output, you\u2019re not just debugging your own code. You\u2019re also trying to understand the framework\u2019s execution flow and what the callback layer was doing behind the scenes.<\/p>\n
I once spent three hours tracking down a failure that turned out to be a memory module silently cutting out context. The fix itself took four minutes. Finding what caused it took half a day because the abstraction made the actual behavior invisible.<\/p>\n
Observability hits a ceiling: You can integrate LangSmith<\/a> and get useful traces, but you\u2019re still seeing things through the framework\u2019s lens, limited to the spans it chooses to expose. When you need visibility into something specific to your business logic, you end up working around the framework\u2019s data model instead of just measuring what actually matters.<\/p>\n Multi-agent state is where things really fall apart: The moment you have agents coordinating<\/a>, one planning, others executing, and another verifying, shared state becomes the real problem.<\/p>\n Who created this information, when, and is it still valid?<\/p>\n One agent updates memory, another reads a stale version, and the coordinator makes a decision based on context that no longer matches reality.<\/p>\n Framework-managed state tends to work just fine for the happy path and quietly breaks down at the edge cases. Production systems live in those edge cases.<\/p>\n Latency accumulates: Every abstraction layer adds overhead through serialization, validation, callback firing, and internal routing that runs whether you need it or not.<\/p>\n In a prototype that overhead is invisible. Under real traffic, it shows up in percentile latency<\/a>, specifically in the p95 and p99 ranges where users actually feel it.<\/p>\n The cost per call might be small, but in an agentic system making four, five, or even six model calls per user request, those small costs compound quickly.<\/p>\n At some point, you have to ask whether that overhead is still worth what it buys you.<\/p>\n None of this is impossible to solve inside a framework. But the fixes start to look like working around the framework instead of working with it. And once you get there, it becomes harder to tell what the framework is still giving you.<\/p>\n So What Does \u201cBuilding It Yourself\u201d Actually Look Like?<\/p>\n \u201cNative agent architecture\u201d sounds more complex than it really is. It just means writing the orchestration logic yourself as code you own, instead of relying on a framework\u2019s abstraction of it.<\/p>\n State is something you define and update explicitly. Tools are clear functions you can test on their own. Memory is code you wrote, so it\u2019s easier to debug, control, and understand what gets stored and how it gets retrieved.<\/p>\n The model call is your code, which means you can instrument it directly and trace what matters.<\/p>\n Sure, there will be more code upfront. But when something breaks, the failure is in your code and not somewhere inside an execution model written by somebody else.<\/p>\n