{"id":141833,"date":"2026-08-17T02:17:15","date_gmt":"2026-08-17T02:17:15","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/141833\/"},"modified":"2026-08-17T02:17:15","modified_gmt":"2026-08-17T02:17:15","slug":"designing-a-persistent-knowledge-layer-that-refuses-to-guess","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/141833\/","title":{"rendered":"Designing a Persistent Knowledge Layer That Refuses to Guess"},"content":{"rendered":"

In my\u00a0RAG-ING Ahead\u00a0series<\/a>\u00a0I \u00a0cloud-native retrieval stack: speech and document processing, chunking, embeddings, Azure AI Search, and an assistant layer sitting on top of it. That series answered the question I had at the time, which was essentially \u2018how do I get a language model to answer questions about documents it was never trained on\u2019?\u00a0<\/p>\n

The stack still works. Retrieval-Augmented Generation remains the most practical way to ground a model in private, domain-specific or recently changed information without retraining anything.[1] If you have a corpus and you need answers from it, RAG is still where you start.\u00a0<\/p>\n

But I have been running that pattern for a while now, on projects that lasted longer than a demo, and a different question started to bother me:\u00a0<\/p>\n

The system retrieves the same paragraph, reasons over it, produces\u00a0a good answer\u00a0\u2014 and then throws\u00a0all\u00a0that reasoning away. Tomorrow, someone asks a related question, and it does the identical work again, from scratch, at the same cost, with no guarantee of reaching the same conclusion.\u00a0<\/p>\n

My first instinct was to tune the machinery rather than question it. I experimented with different caching mechanisms based on embeddings: recognizing that an incoming question was semantically close to one the system had already answered, and serving the earlier response instead of paying for the full retrieval-and generation pass again. Semantic caching genuinely helps with cost and latency, and I would still recommend it.\u00a0But it took me a while to admit what it actually is.\u00a0It\u00a0caches\u00a0answers, not understanding. The cached response is exactly as disposable as the original one. Nothing about the\u00a0system\u2019s\u00a0model of the domain has improved, and the moment a question falls outside the similarity threshold, the work starts from zero again. Whatever I tweaked, the main RAG architectural concept underneath\u00a0remained\u00a0the same.\u00a0<\/p>\n

\"\"Figure 1 \u2013 The semantic cache I was experimenting with. A hit is a shortcut past the pipeline; a miss starts from zero. Either way, nothing accumulates \u2014 the dashed box is the part that turned out to be missing. Image by\u00a0author.\u00a0<\/p>\n

That is not a retrieval problem. Retrieval is doing exactly what it was designed to do. It is an\u00a0architecture\u00a0problem.\u00a0There is nowhere in a standard RAG system for understanding to accumulate. No amount of caching, re-ranking\u00a0or chunking strategy fixes that, because they all\u00a0optimize\u00a0the\u00a0lookup,\u00a0none of them gives the system a memory.\u00a0<\/p>\n

This article is about\u00a0building\u00a0that\u00a0missing\u00a0place. It is the result of my latest work and experimentation around RAG,\u00a0GraphRAG\u00a0and agentic reasoning over a corpus of documents, the point where the incremental tweaks stopped being\u00a0enough\u00a0and the design itself had to change. I will present it in three parts.\u00a0<\/p>\n

Part I is\u00a0vendor-neutral.\u00a0It describes the architecture as a design pattern: the layers, the object model, the failure modes it exists to survive, and the governance it demands. None of it depends on Azure, or on any\u00a0particular database\u00a0or model provider. If you are on\u00a0AWS,\u00a0GCP, or running Postgres with\u00a0pgvector\u00a0and a local model, the design still holds, and I would like it to be useful to you.\u00a0<\/p>\n

Part II is the Azure implementation.\u00a0Service by service, with the reasoning for each choice, real infrastructure-as-code, and a running\u00a0FastAPI\u00a0application you can clone and deploy.\u00a0<\/p>\n

Part III is the demonstration.\u00a0A synthetic property insurer called\u00a0Ostermere\u00a0Mutual, twenty-one interconnected documents, and three walkthroughs that show the pattern doing something a retrieval system genuinely cannot.\u00a0<\/p>\n

Everything in the dataset is synthetic.\u00a0Ostermere\u00a0Mutual does not exist. Neither does the regulator, the policy, the claims, the people, the wind\u00a0zones\u00a0or the figures. Nothing here is insurance, legal, underwriting or\u00a0claims\u00a0advice, and no page in the demo\u00a0represents\u00a0a real interpretation of any real policy.\u00a0<\/p>\n

Naming note:\u00a0current Microsoft documentation uses\u00a0Microsoft Foundry\u00a0for the unified platform previously called Azure AI Foundry.\u00a0I use\u00a0Foundry\u00a0throughout, while keeping the familiar Azure service names where they make the architecture easier to follow.\u00a0<\/p>\n

Contents\u00a0<\/p>\n

Part I \u2013 The design:\u00a0<\/a><\/p>\n

1. Where classic RAG works, and where it stops<\/a><\/p>\n

2. Retrieval is not accumulated understanding<\/a><\/p>\n

3.\u00a0The three\u00a0layers<\/a><\/p>\n

4. What\u00a0actually lives\u00a0in the knowledge layer<\/a><\/p>\n

5.\u00a0The six\u00a0failure modes<\/a><\/p>\n

6. Writing knowledge is a different risk class<\/a><\/p>\n

7. Query routing<\/a><\/p>\n

8. When you should not build this\u00a0<\/a><\/p>\n

Part II \u2013 Implementing it on Azure:\u00a0<\/a><\/p>\n

9. Mapping the layers to services<\/a><\/p>\n

10. Blob Storage<\/a><\/p>\n

11. Document Intelligence<\/a><\/p>\n

12. Azure AI Search<\/a><\/p>\n

13. Cosmos DB<\/a><\/p>\n

14. Microsoft Foundry<\/a><\/p>\n

15.\u00a0FastAPI\u00a0on Container Apps<\/a><\/p>\n

16. Identity<\/a><\/p>\n

17. The ingestion lifecycle\u00a0<\/a><\/p>\n

Part III \u2013 The demonstration:\u00a0<\/a><\/p>\n

18.\u00a0Ostermere\u00a0Mutual<\/a><\/p>\n

19. The scoped rule<\/a><\/p>\n

20.\u00a0The contradiction<\/a><\/p>\n

21.\u00a0The date\u00a0of loss<\/a><\/p>\n

22.\u00a0The Obsidian\u00a0vault<\/a><\/p>\n

23.\u00a0The cost\u00a0argument<\/a><\/p>\n

24. Governance<\/a><\/p>\n

25. What I would build next<\/a><\/p>\n

26. To sum it all up\u00a0<\/a><\/p>\n

The complete project (application, infrastructure,\u00a0dataset\u00a0and a ready-to-open Obsidian vault) is available in the accompanying GitHub repository at\u00a0github.com\/mcekikj\/persistent-knowledge-layer<\/a>, under the MIT license.\u00a0<\/p>\n

Part I \u2013 The design\u00a0<\/p>\n

1. Where classic RAG works, and where it stops\u00a0<\/p>\n

A conventional RAG flow is simple enough to draw in one line.\u00a0<\/p>\n

\"\"Figure 2 \u2013 The classic RAG pipeline. Every question starts\u00a0at\u00a0the left. Nothing survives past the right. Image by\u00a0author.\u00a0<\/p>\n

Documents are chunked, embedded, and stored in a vector-capable index. A question\u00a0arrives,\u00a0the system finds semantically or lexically similar chunks, and hands them to the model as context. The model, which knows nothing about your business, is temporarily made to look like it does.\u00a0<\/p>\n

This solves a real and important problem, and I do not want to\u00a0undersell\u00a0it. The organization\u2019s private knowledge does not need to live in the model\u2019s parameters. It is fetched when needed. That is a genuinely good\u00a0idea\u00a0and it is why the pattern spread so fast.\u00a0<\/p>\n

But look at what the architecture is\u00a0optimized\u00a0for. It is\u00a0optimized\u00a0for lookup at query time. Every question is treated as the first question anyone has ever asked.\u00a0<\/p>\n

Now consider a real user, working on a real problem, over several weeks:\u00a0<\/p>\n

What is Actual Cash Value?\u00a0<\/p>\n

How does it differ from Replacement Cost Value?\u00a0<\/p>\n

When can recoverable depreciation be paid?\u00a0<\/p>\n

Which earlier decision defined how we treat depreciation?\u00a0<\/p>\n

Which document introduced the exception, and why?\u00a0<\/p>\n

A colleague told me the threshold is 15 years. Is it?\u00a0<\/p>\n

A simple RAG application answers each of these independently. It retrieves chunks again, rebuilds context again, and asks the model to reason again. Each answer\u00a0may well\u00a0be good. But the\u00a0synthesis\u00a0is disposable, meaning that when the response is delivered, the understanding evaporates. Nothing about the sixth question is easier because the system already answered the first five.\u00a0<\/p>\n

And on the last question \u2013\u00a0is the threshold 15 years?\u00a0\u2013 a retrieval system will do something worse than\u00a0fail. It will find the chunk that says 15 years, and it will confidently tell you yes.\u00a0<\/p>\n

2. Retrieval is not accumulated understanding\u00a0<\/p>\n

The analogy I keep coming back to is a researcher with a filing cabinet.\u00a0<\/p>\n

You ask a question. The researcher goes to the cabinet, pulls four documents, reads the relevant passages, and gives you a considered answer. This is genuinely useful. Then they put the documents back, throw away their notes, and forget the entire exercise.\u00a0Tomorrow\u00a0you\u00a0ask\u00a0a follow-up, and they start again at the cabinet.\u00a0<\/p>\n

A better researcher does something else while reading. They:\u00a0<\/p>\n

keep summaries of the important sources;\u00a0<\/p>\n

maintain a page for each recurring concept, and connect the concepts that turn out to be related;\u00a0<\/p>\n

write down decisions, and the reasoning behind them;\u00a0<\/p>\n

update a comparison when new evidence changes it;\u00a0<\/p>\n

record contradictions rather than quietly resolving them;\u00a0<\/p>\n

keep a running list of what they still cannot answer;\u00a0<\/p>\n

and keep a link from every claim back to the document it came from.\u00a0<\/p>\n

The first researcher is a query-time RAG system. The second is what I want to build.\u00a0<\/p>\n

This is close to the\u00a0LLM Wiki\u00a0pattern that Andrej Karpathy sketched out: raw sources stay where they are, while an agent\u00a0maintains\u00a0a set of Markdown pages (entities, concepts, comparisons, cross-references) that a human can read and navigate. Obsidian happens to be a convenient window onto the result.[2]\u00a0<\/p>\n

It is easy to get distracted by the Markdown here, so let me be precise about what the important idea\u00a0actually is.\u00a0<\/p>\n

The important idea is not Obsidian. It is not even Markdown. It is that\u00a0knowledge is compiled once into a durable artifact, instead of being reconstructed from raw chunks on every request.\u00a0<\/p>\n

That is an architectural claim, and it has architectural consequences.\u00a0<\/p>\n

3. The three layers\u00a0<\/p>\n

I do not want to replace RAG. I want to give it somewhere to put what it learns.\u00a0<\/p>\n

The raw source is still the strongest thing you have whenever you need exact wording, a quotation, a clause reference, a newly uploaded document, or verification of something disputed. A generated page,\u00a0however\u00a0carefully\u00a0maintained, is a derived interpretation. It is not evidence, and the moment you let it pretend to be evidence, you have built something dangerous.\u00a0<\/p>\n

So\u00a0the design has three layers, and they answer three different questions.\u00a0<\/p>\n

\"\"Figure 3 \u2013 The hybrid knowledge architecture. Both layers derive from the same sources; neither\u00a0replaces\u00a0the other. Image by\u00a0author.\u00a0<\/p>\n

Layer\u00a0The question it answers\u00a0Optimized\u00a0for\u00a0Evidence\u00a0What\u00a0source\u00a0material is relevant to this question, right now?\u00a0Recall, exact wording, citation, freshness\u00a0Knowledge\u00a0What has this system already worked out, and what does it currently believe?\u00a0Continuity, relationships, synthesis, reuse\u00a0Orchestrator\u00a0Which of those do I need to answer this safely?\u00a0Routing, risk, temporal scope\u00a0Table 1 \u2013 The three layers and the question each one exists to answer. Neither replaces the other; they are optimized for different things. Table by author.<\/p>\n

The distinction is practical, not philosophical. The evidence layer is a retrieval index.\u00a0The knowledge layer is a maintained, structured, human-readable model of the domain.\u00a0The orchestrator is the thing that knows a question about exact policy wording should go to the first, and a question about\u00a0why we decided this\u00a0should go to the second.\u00a0<\/p>\n

One rule holds the whole thing together, and it is worth\u00a0stating\u00a0on its own line:\u00a0<\/p>\n

A knowledge page is never a source. It is always traceable to one.\u00a0<\/p>\n

Break that\u00a0rule\u00a0and you no longer have a knowledge base. Instead,\u00a0you have a collection of confident claims that nobody can verify.\u00a0<\/p>\n

4. What\u00a0actually lives\u00a0in the knowledge layer\u00a0<\/p>\n

The word\u00a0wiki\u00a0invites people to picture a folder of Markdown files. For a personal knowledge base that is genuinely fine.\u00a0For an application, I want a structured store underneath, with Markdown as a\u00a0view\u00a0generated from it.\u00a0<\/p>\n

The object model I have settled on:\u00a0<\/p>\n

\"\"Figure 4 \u2013 The knowledge object model. Every derived object points back\u00a0at\u00a0the sources that justify it. Image by\u00a0author.\u00a0<\/p>\n

Most of these are unsurprising. Three of them are the whole point, and I want to dwell on them.\u00a0<\/p>\n

Decision \u2013 because the\u00a0why\u00a0dies first\u00a0<\/p>\n

A decision object holds the rule, its scope, its effective date, its accountable owner, and its rationale, with a pointer to where the rationale came from.\u00a0<\/p>\n

That last field matters more than it looks. In my synthetic corpus, an underwriting update says a roof inspection is triggered above 15 years in one specific wind zone. The update does not say\u00a0why\u00a015. The reasoning exists in exactly one place: an email thread between an analyst and a head of underwriting, which explains that between 15 and 20 years, roof displacement in the severe-wind band runs about 3.4x the standard band, and which explicitly warns that people will read the number without the qualifier and apply it to the entire book.\u00a0<\/p>\n

An email is not a policy document. No retrieval system ranks it highly. And in eighteen months, when someone\u00a0asks\u00a0\u201cwhy is it 15?\u201d, that reasoning is gone \u2014 unless something deliberately preserved it.\u00a0<\/p>\n

Contradiction \u2013 a first-class object, not an error\u00a0<\/p>\n

This is the idea I would most encourage you to take away, whatever else you use from this article.\u00a0<\/p>\n

Real corpora contradict themselves. Two teams write two documents, both current, neither superseding the other, and they disagree. In any document set\u00a0maintained\u00a0by more than one team for more than a year, this is the normal\u00a0state of affairs, not an edge case.\u00a0<\/p>\n

A retrieval system handles this catastrophically badly. It retrieves one chunk, or the other, or both, and then asks a language model to reconcile them in a single forward pass, under a system prompt that told it to be helpful. The model will produce an answer\u00a0that\u00a0will be\u00a0fluent,\u00a0confident,\u00a0but\u00a0it will have silently\u00a0picked\u00a0a side.\u00a0<\/p>\n

Worse, the heuristic it most naturally reaches for is\u00a0the more recent document wins. That sounds sensible, and it is wrong.\u00a0Recency is not applicability.\u00a0A newer document may have\u00a0narrower\u00a0scope, may address a different product, or may\u00a0have been written by a team with no authority over the question.\u00a0<\/p>\n

So,\u00a0the contradiction gets its own object, with a status, both statements verbatim, their effective dates, an accountable owner, and an explicit field:\u00a0why_not_resolved. The system\u2019s job is to detect the conflict and refuse to settle it.\u00a0<\/p>\n

Open question \u2013 knowing what you\u00a0don\u2019t\u00a0know\u00a0<\/p>\n

The natural companion. Some questions cannot be answered yet, often because they are blocked\u00a0by\u00a0contradiction. An open question that is\u00a0visible\u00a0is safe. The same question, quietly answered wrong, is the thing that ends up in a complaint file.\u00a0<\/p>\n

5. The six failure modes this architecture exists to survive\u00a0<\/p>\n

Here is the honest test of any architecture: what does it do that the simpler thing cannot?\u00a0<\/p>\n

I built the demonstration corpus specifically to answer that. It\u00a0contains\u00a0six distinct traps. A pure retrieval system falls into every one of them \u2014 and\u00a0falls\u00a0fluently, producing an answer that reads perfectly well.\u00a0<\/p>\n

5.1 Scoped supersession\u00a0<\/p>\n

A general rule says\u00a0inspect\u00a0roofs\u00a0above\u00a020 years. A later update says\u00a0inspect\u00a0above\u00a015 years, but only in one wind zone, and only for new business.\u00a0<\/p>\n

Retrieval returns the 15-year chunk. The model says the threshold is 15 years. It is now demanding inspections on tens of thousands of ordinary roofs, and the\u00a0broker\u00a0complaints are entirely justified.\u00a0<\/p>\n

The knowledge layer stores a decision with\u00a0scope: \u201cNew business only.\u00a0Zone H3 only.\u201d\u00a0The number never travels without its qualifier.\u00a0<\/p>\n

5.2 Genuine contradiction\u00a0<\/p>\n

The Claims Handling Manual says trace-and-access costs are covered\u00a0as standard\u00a0up to \u20ac5,000\u00a0and handlers may authorize without referral. The Endorsement Catalogue says trace and access is an\u00a0optional paid endorsement, not payable unless it is on the schedule.\u00a0<\/p>\n

Both documents are current. Neither supersedes the other. Different teams wrote them.\u00a0<\/p>\n

A RAG system picks one. The knowledge layer raises a contradiction, names an owner, and states that no answer is available.\u00a0<\/p>\n

5.3 Terminology drift\u00a0<\/p>\n

Across the corpus, the same concept appears as\u00a0actual cash value,\u00a0ACV,\u00a0cash settlement\u00a0basis\u00a0and\u00a0depreciated value. A\u00a0broker\u00a0email in the dataset\u00a0literally lists\u00a0six such terms and asks whether they are six things or one.\u00a0<\/p>\n

Without entity resolution, your wiki grows four separate pages that disagree with each other by omission. With it, one page, four aliases, and a query for any of them lands in the right place.\u00a0<\/p>\n

5.4 Effective-date scoping\u00a0<\/p>\n

A claim has a\u00a0date of loss\u00a0of 20 February 2026. A rule took effect on 1 March 2026. The rule cannot apply to that claim.\u00a0<\/p>\n

This one is my\u00a0favourite, because a retrieval system has no\u00a0defence\u00a0against it at all. Semantic similarity does not encode time. The chunk about the 15-year threshold is\u00a0maximally relevant\u00a0to a question about roof age on that claim \u2013 and\u00a0completely inapplicable. The system is not merely\u00a0wrong,\u00a0it is wrong in the most convincing\u00a0possible way.\u00a0<\/p>\n

The fix requires the orchestrator to know that the question is\u00a0about\u00a0a date, and to select the documentation in force on that date, including keeping a superseded document that was live at the time.\u00a0<\/p>\n

5.5 Rationale loss\u00a0<\/p>\n

Covered above. The reasoning lives in an\u00a0email,\u00a0the rule lives in a guideline, while\u00a0the connection between them lives nowhere.\u00a0<\/p>\n

5.6\u00a0Multi-hop\u00a0<\/p>\n

\u201cWhy was this claim triaged Level 1?\u201d\u00a0requires the claim notes, then the triage guideline, then the water-damage concept, then the policy clause.\u00a0Four hops. Top-k similarity search does not\u00a0traverse,\u00a0it ranks. Typed relationships\u00a0do\u00a0traverse.\u00a0<\/p>\n

Put together, these six are the argument \u2014 not that the wiki is nicer, but that there is a class of question retrieval answers confidently and wrongly, and the knowledge layer catches it.\u00a0<\/p>\n

6. Writing knowledge is a different risk class than answering\u00a0<\/p>\n

Here is the thing that took me longest to internalize, and it changed how I think about the whole design.\u00a0<\/p>\n

A wrong chat answer affects one conversation. A wrong\u00a0canonical\u00a0concept page affects every answer that is later built on top of it, for as long as it stays wrong, and nobody notices, because it looks like knowledge.\u00a0<\/p>\n

The moment your system starts\u00a0writing\u00a0persistent knowledge, it has crossed from\u00a0\u201cretrieval application\u201d\u00a0into\u00a0\u201csystem of record\u201d,\u00a0and it needs the controls that come with that.\u00a0<\/p>\n

Everything is a patch\u00a0<\/p>\n

The model never writes to the store. It proposes a patch. The application\u00a0validates\u00a0it and, where the change is consequential, a human approves it.\u00a0<\/p>\n

\"\"Figure 5 \u2013 The patch lifecycle. The model\u00a0proposes,\u00a0the application disposes. Image by\u00a0author.\u00a0<\/p>\n

Note where\u00a0\u201cresolve a contradiction\u201d\u00a0sits:\u00a0never automatic. If the system could resolve contradictions on its own authority, the contradiction object would be pointless.\u00a0<\/p>\n

Provenance is a chain, not a field\u00a0<\/p>\n

\"\"Figure 6 \u2013 If any link in this chain is missing, the page is a polished note, not a knowledge object. Image by\u00a0author.\u00a0<\/p>\n

An object whose sources cannot be\u00a0identified\u00a0should be\u00a0deleted, not corrected. You cannot fix something when you\u00a0don\u2019t\u00a0know where it came from.\u00a0<\/p>\n

Staleness is a property you must track\u00a0<\/p>\n

A page can be\u00a0correct\u00a0on Monday and wrong on Friday because a source was superseded underneath it.\u00a0So\u00a0every derived object carries\u00a0last_validated_at, and the source it was derived from carries\u00a0superseded_by. When a source is superseded, everything derived from it is marked stale and must not be presented as current until it is re-derived.\u00a0<\/p>\n

7. Routing:\u00a0which\u00a0layer answers this?\u00a0<\/p>\n

Not every question needs both layers, and sending everything to both is how you build something expensive and slow.\u00a0<\/p>\n

\"\"Figure\u00a07\u00a0\u2013\u00a0Query routing. The contradiction check is a gate, not a footnote.\u00a0Image by\u00a0author.\u00a0<\/p>\n

Two things about this diagram are deliberate.\u00a0<\/p>\n

First, the temporal check belongs before retrieval, not after. If you rank by similarity first and discard inapplicable results afterwards, the inapplicable documents have already consumed your top-k. In production, put the effective and superseded dates on the index and filter in the query itself, so the candidate set is constrained before ranking. The demo takes a shortcut here that I should own up to: it applies the date filter\u00a0immediately\u00a0after retrieval, which behaves identically at this corpus size but would quietly starve\u00a0top-k\u00a0on a large one. The principle stands\u00a0that\u00a0the demo trades it for a simpler index schema.\u00a0<\/p>\n

Second, the contradiction check is a gate on the way out. It does not matter which route the question took. If the topic is contested, the system\u00a0stops. In code, that is roughly:\u00a0<\/p>\n

def\u00a0query(self, question,\u00a0requested_mode,\u00a0top_k,\u00a0as_of=None):\u00a0
\n\u00a0\u00a0\u00a0 mode\u00a0=\u00a0self.choose_mode(question,\u00a0requested_mode)\u00a0
\n\u00a0
\n\u00a0\u00a0\u00a0\u00a0wiki_items\u00a0=\u00a0self._search_wiki(question,\u00a0top_k)\u00a0if\u00a0mode\u00a0in\u00a0{“wiki”,\u00a0“hybrid”}\u00a0else\u00a0[]\u00a0
\n\u00a0\u00a0\u00a0 evidence\u00a0=\u00a0self.evidence.search(question,\u00a0top_k)\u00a0if\u00a0mode\u00a0in\u00a0{“evidence”,\u00a0“hybrid”}\u00a0else\u00a0[]\u00a0
\n\u00a0
\n\u00a0\u00a0\u00a0\u00a0if\u00a0as_of:\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# The date the question is ABOUT – not the date it is asked.\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 evidence\u00a0=\u00a0self._filter_by_date(evidence,\u00a0as_of)\u00a0
\n\u00a0
\n\u00a0\u00a0\u00a0\u00a0# Pull in every contradiction touching a retrieved concept, even if the\u00a0
\n\u00a0\u00a0\u00a0\u00a0# contradiction object itself did not rank. Someone asking about trace and\u00a0
\n\u00a0\u00a0\u00a0\u00a0# access gets the conflict whether or not they used the word “contradiction”.\u00a0
\n\u00a0\u00a0\u00a0 contradictions\u00a0=\u00a0self._contradictions_for(wiki_items)\u00a0
\n\u00a0\u00a0\u00a0 warnings\u00a0=\u00a0self._warnings(evidence, contradictions,\u00a0as_of)\u00a0
\n\u00a0
\n\u00a0\u00a0\u00a0 context\u00a0=\u00a0self._build_context(wiki_items, evidence, contradictions,\u00a0as_of)\u00a0
\n\u00a0\u00a0\u00a0 answer\u00a0=\u00a0self.model.answer(question, context)\u00a0
\n\u00a0\u00a0\u00a0 …\u00a0<\/p>\n

And the instruction that goes to the model is unambiguous:\u00a0<\/p>\n

UNRESOLVED CONTRADICTIONS. You MUST present both positions with their sources and\u00a0state that the position is unresolved. You MUST NOT choose between them, and you\u00a0MUST NOT prefer the more recent document – recency is not applicability.\u00a0<\/p>\n

8. When you should\u00a0not\u00a0build this\u00a0<\/p>\n

I would rather you skip this architecture than misapply it, so let me be direct about the cases where simple RAG is the better engineering choice.\u00a0<\/p>\n

Stay with plain RAG when:\u00a0<\/p>\n

the corpus is small and queried rarely, so there is nothing to amortize;\u00a0<\/p>\n

users overwhelmingly want exact source lookup, not synthesis;\u00a0<\/p>\n

documents churn so fast that any derived synthesis is stale before it is used;\u00a0<\/p>\n

there is no cross-session knowledge worth preserving;\u00a0<\/p>\n

it is a prototype with a short life;\u00a0<\/p>\n

ingestion latency\u00a0has to\u00a0be minimal;\u00a0<\/p>\n

your\u00a0organization cannot yet govern AI-generated persistent knowledge. This one is not a technical\u00a0constraint\u00a0and it is the one people ignore.\u00a0<\/p>\n

Build the knowledge layer when:\u00a0<\/p>\n

the same domain is queried repeatedly, by people whose work continues across sessions;\u00a0<\/p>\n

cross-source synthesis is normal, not exceptional;\u00a0<\/p>\n

decisions and their rationale must survive staff turnover;\u00a0<\/p>\n

exceptions,\u00a0scopes\u00a0and contradictions\u00a0actually matter;\u00a0<\/p>\n

domain experts need to\u00a0see\u00a0and correct what the system believes;\u00a0<\/p>\n

an audit trail from answer to source is a requirement, not a nice-to-have.\u00a0<\/p>\n

This is a workload decision rather than a matter of\u00a0new good, old bad, and the honest answer for a lot of applications is that you do not need this.\u00a0<\/p>\n

Part II \u2013 Implementing it on Azure\u00a0<\/p>\n

Everything above is deliberately portable. Now let me build it properly on Azure, the platform I work with daily.\u00a0<\/p>\n

9. Mapping the layers to services\u00a0<\/p>\n

\"\"Figure 8 \u2013\u00a0The Azure-native architecture. The\u00a0colours\u00a0match Figure 3: red is evidence-at-rest, blue is the retrieval layer, green is the knowledge layer. Image by\u00a0author.\u00a0<\/p>\n

Responsibility\u00a0Azure service\u00a0Immutable raw sources\u00a0Azure Blob Storage\u00a0Scanned PDFs, tables, forms, layout\u00a0Azure AI Document Intelligence\u00a0Chunk, keyword,\u00a0vector\u00a0and hybrid retrieval\u00a0Azure AI Search\u00a0Chat and embedding model deployments\u00a0Microsoft Foundry\u00a0Structured wiki state\u00a0Azure Cosmos DB for NoSQL\u00a0API and orchestration\u00a0FastAPI\u00a0on Azure Container Apps\u00a0Event-driven ingestion\u00a0Event Grid \u2192 Container Apps Jobs\u00a0Identity and secrets\u00a0Entra ID, managed identity, Key Vault\u00a0Telemetry\u00a0Application Insights\u00a0Human inspection of the knowledge\u00a0Obsidian, over exported Markdown\u00a0Table 2 \u2013 Each responsibility mapped to the Azure service that carries it. Nothing here is exotic \u2014 the design is in how the pieces are wired together, not in the pieces themselves. Table by author.<\/p>\n

\"\"Figure 9 \u2013\u00a0The deployed resource group in the Azure portal\u2019s Resource visualizer: the Container App and its environment, Foundry, Cosmos DB, Application Insights, Key Vault, the managed identity, AI\u00a0Search\u00a0and the storage account. Screenshot by\u00a0author.\u00a0<\/p>\n

10. Blob Storage \u2013 the one thing you cannot regenerate\u00a0<\/p>\n

Everything else in this architecture is derived. Chunks can be re-chunked. Embeddings can be re-embedded. The entire wiki can, in principle, be recompiled from scratch. The original documents cannot be recovered from anything.\u00a0<\/p>\n

So\u00a0they get treated accordingly:\u00a0<\/p>\n

raw-sources\/\u00a0
\n\u00a0\u00a0\u00a0 {workspace-id}\/\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {document-id}\/\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 original-file.pdf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2190 never rewritten\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 extracted-content.json\u00a0\u00a0\u00a0\u00a0 \u2190 Document Intelligence output\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ingestion-metadata.json\u00a0\u00a0\u00a0 \u2190 what ran, when, which model version\u00a0
\n\u00a0
\nwiki-export\/\u00a0
\n\u00a0\u00a0\u00a0 {workspace-id}\/\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Home.md\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Concepts\/ \u00b7 Decisions\/ \u00b7 Contradictions\/ \u00b7 Sources\/\u00a0<\/p>\n

In Bicep, the part that matters\u00a0is\u00a0three properties:\u00a0<\/p>\n

resource\u00a0blobService\u00a0‘Microsoft.Storage\/storageAccounts\/blobServices@2023-05-01’ = {\u00a0
\n\u00a0 parent: storage\u00a0
\n\u00a0 name: ‘default’\u00a0
\n\u00a0 properties: {\u00a0
\n\u00a0\u00a0\u00a0 \/\/ Originals must survive an ingestion bug. Versioning and soft delete are the\u00a0
\n\u00a0\u00a0\u00a0 \/\/ cheapest insurance available on the one artifact the system cannot regenerate.\u00a0
\n\u00a0\u00a0\u00a0\u00a0isVersioningEnabled: true\u00a0
\n\u00a0\u00a0\u00a0\u00a0deleteRetentionPolicy:\u00a0{ enabled: true, days: 30 }\u00a0
\n\u00a0\u00a0\u00a0\u00a0containerDeleteRetentionPolicy:\u00a0{ enabled: true, days: 30 }\u00a0
\n\u00a0 }\u00a0
\n}\u00a0<\/p>\n

And on the storage account itself, one line that I would argue for in any production deployment:\u00a0<\/p>\n

allowSharedKeyAccess: false\u00a0\u00a0 \/\/ no connection strings, ever<\/p>\n

The wiki-writing process must never be able to touch the originals. That is a provenance, audit and deletion-workflow requirement, and it is much easier to enforce with separate containers and narrow role assignments than with good intentions.\u00a0<\/p>\n

11. Document Intelligence \u2013 used selectively\u00a0<\/p>\n

My demo corpus is\u00a0.txt\u00a0and\u00a0.md, so the application reads it directly. Real insurance documents are scans, forms, tables and signatures, and the layout itself is often load-bearing. A table of endorsement limits flattened into a paragraph is worse than useless.\u00a0<\/p>\n

Azure AI Document Intelligence gives you prebuilt and custom models that return text, tables, selection\u00a0marks\u00a0and structure.[3] My rule of thumb:\u00a0<\/p>\n

clean text and Markdown \u2192 a simple parser, no charge;\u00a0<\/p>\n

machine-readable PDFs \u2192 a PDF parser, if it is genuinely sufficient;\u00a0<\/p>\n

scans, forms, complex layout, tables \u2192 Document Intelligence;\u00a0<\/p>\n

always\u00a0persist\u00a0the extracted JSON next to the\u00a0original and\u00a0keep the page and span offsets so a citation can point at a location, not just a document.\u00a0<\/p>\n

That last point pays for itself the first time a compliance reviewer\u00a0asks\u00a0\u201cwhere exactly does it say that?\u201d\u00a0<\/p>\n

12. Azure AI Search \u2013 the evidence layer\u00a0<\/p>\n

Each indexed record carries both searchable text and its vector:\u00a0<\/p>\n

{\u00a0
\n\u00a0\u00a0“id”:\u00a0“INS-SYN-004-0”,\u00a0
\n\u00a0\u00a0“workspace_id”:\u00a0“ostermere-insurance-demo”,\u00a0
\n\u00a0\u00a0“source_id”:\u00a0“INS-SYN-004”,\u00a0
\n\u00a0\u00a0“title”:\u00a0“High-Wind Zone Underwriting Update”,\u00a0
\n\u00a0\u00a0“content”:\u00a0“For new\u00a0Hearthmere\u00a0business in zone H3…”,\u00a0
\n\u00a0\u00a0“chunk_number”:\u00a00,\u00a0
\n\u00a0\u00a0“content_vector”:\u00a0[0.012,\u00a0-0.008,\u00a00.031,\u00a0“…”]\u00a0
\n}\u00a0<\/p>\n

Azure AI Search fits this design well for one specific reason: text and vector fields coexist in a single index, and a hybrid query runs the full-text and vector queries in parallel, fusing the rankings with Reciprocal Rank Fusion. Semantic ranking can then reorder the top results.[4]\u00a0<\/p>\n

That matters more than it might sound. In insurance, half the queries are\u00a0conceptual\u00a0(\u201cwhat counts as sudden water damage\u201d) and half are\u00a0lexical\u00a0(\u201cwhat does HS-TA-01 cover\u201d). Vector search is good at\u00a0first\u00a0and unreliable at\u00a0second: embeddings are known to be weak with exact identifiers, clause\u00a0numbers\u00a0and product codes. BM25 handles those well but cannot handle paraphrasing. You want both, and you want them fused rather than chosen between.\u00a0<\/p>\n

from\u00a0azure.search.documents.models\u00a0import\u00a0VectorizedQuery\u00a0
\n\u00a0
\nvector_query\u00a0=\u00a0VectorizedQuery(\u00a0
\n\u00a0\u00a0\u00a0 vector=self.model.embed(query),\u00a0
\n\u00a0\u00a0\u00a0\u00a0k_nearest_neighbors=max(top_k,\u00a010),\u00a0
\n\u00a0\u00a0\u00a0 fields=”content_vector”,\u00a0
\n)\u00a0
\n\u00a0
\nresults\u00a0=\u00a0self.client.search(\u00a0
\n\u00a0\u00a0\u00a0\u00a0search_text=query,\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# BM25 leg\u00a0
\n\u00a0\u00a0\u00a0\u00a0vector_queries=[vector_query],\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# vector leg\u00a0
\n\u00a0\u00a0\u00a0\u00a0filter=f”workspace_id\u00a0eq ‘{self.workspace_id}'”,\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# security trim\u00a0
\n\u00a0\u00a0\u00a0 select=[“id”,\u00a0“source_id”,\u00a0“title”,\u00a0“content”,\u00a0“chunk_number”],\u00a0
\n\u00a0\u00a0\u00a0 top=top_k,\u00a0
\n)\u00a0<\/p>\n

Note the\u00a0filter, because it is doing security work, not relevance tuning.\u00a0<\/p>\n

\u26a0\ufe0f Vector similarity is not authorization.\u00a0Nothing about cosine distance respects your permission model. Security trimming must be a hard filter on an indexed field, applied at query time, on every single query. It must be applied identically to the knowledge layer and to the exported Markdown. A wiki page that synthesizes three documents the user cannot read is still a data leak, just a nicely formatted one.\u00a0<\/p>\n

For larger systems, integrated vectorization can move chunking and embedding into the indexer pipeline.[5] I keep the embedding calls in the application here\u00a0purely,\u00a0so the flow is visible and explainable in an article.\u00a0<\/p>\n

One sizing note from\u00a0actually deploying\u00a0this: the demo runs on the\u00a0free\u00a0Search tier, and vector plus hybrid search work fine there for a 21-document corpus. What the free tier gives up is the semantic ranker and managed-identity support on the service\u00a0itself,\u00a0both are conditionals in the Bicep,\u00a0so treat\u00a0basic\u00a0as the floor for production and\u00a0free\u00a0as a perfectly\u00a0good way\u00a0to\u00a0validate\u00a0the design for nothing.\u00a0<\/p>\n

\"\"Figure 10 \u2013\u00a0Search Explorer over the insurance-evidence index on the free tier: 33 chunk documents, 763.9 KB of vector storage, and a hybrid query ranking the H3 rationale email (INS-SYN-018) first by @search.score. Screenshot by\u00a0author.\u00a0<\/p>\n

13. Cosmos DB \u2013 the knowledge layer\u00a0<\/p>\n

Cosmos DB for NoSQL holds the structured wiki. The whole design fits in three decisions.\u00a0<\/p>\n

Partition\u00a0key is\u00a0\/workspace_id.\u00a0Every wiki object carries a\u00a0type\u00a0discriminator, so concepts, decisions,\u00a0contradictions\u00a0and open questions all live in one container. That means a single-partition query can pull an entire\u00a0workspace\u2019s\u00a0knowledge with no cross-partition fan-out, which is exactly the access pattern this system has.\u00a0<\/p>\n

partitionKey: {\u00a0
\n\u00a0 paths: [‘\/workspace_id’]\u00a0
\n\u00a0 kind: ‘Hash’\u00a0
\n}\u00a0<\/p>\n

No graph database \u2014 yet.\u00a0People reach for Gremlin or Neo4j the moment they hear\u00a0\u201crelationships\u201d. I would\u00a0push it\u00a0back. Explicit relationship records in a document store handle everything this system actually\u00a0does:\u00a0find\u00a0neighbours, follow a typed edge,\u00a0render\u00a0a concept page with its links. That is one or two hops.\u00a0<\/p>\n

A graph store earns its place when deep traversal is\u00a0itself\u00a0the workload: multi-hop impact analysis, centrality,\u00a0path-finding\u00a0across a large network. If you are not doing that, you are paying for a second database and a second query language to avoid writing\u00a0WHERE\u00a0c.source_id\u00a0= @id.\u00a0<\/p>\n

Serverless, for now.\u00a0Billing follows consumed request units, which suits a demo and a spiky early workload. Move to provisioned throughput once you have measured real RU consumption, not before, and not because the word\u00a0serverless\u00a0is fashionable.[6]\u00a0<\/p>\n

Data-plane access uses Cosmos\u2019s own RBAC system (SQL role assignments), which is separate from Azure RBAC and catches people out:\u00a0<\/p>\n

resource\u00a0cosmosDataRole\u00a0‘Microsoft.DocumentDB\/databaseAccounts\/sqlRoleAssignments@2024-11-15’ = {\u00a0
\n\u00a0 parent: cosmos\u00a0
\n\u00a0 name:\u00a0guid(cosmos.id, identity.id, ‘data-contributor’)\u00a0
\n\u00a0 properties: {\u00a0
\n\u00a0\u00a0\u00a0\u00a0principalId: identity.properties.principalId\u00a0
\n\u00a0\u00a0\u00a0 \/\/ 00000000-…-000000000002 is the built-in Cosmos DB Data Contributor\u00a0
\n\u00a0\u00a0\u00a0\u00a0roleDefinitionId: ‘${cosmos.id}\/sqlRoleDefinitions\/00000000-0000-0000-0000-000000000002’\u00a0
\n\u00a0\u00a0\u00a0 scope: cosmos.id\u00a0
\n\u00a0 }\u00a0
\n}\u00a0<\/p>\n

Combined with\u00a0disableLocalAuth: true, there is no key to leak.\u00a0<\/p>\n

\"\"Figure 11 \u2013\u00a0The con-001 contradiction as it lives in Cosmos DB: status unresolved, both statements with their sources and effective dates, an accountable owner, and the reasons it stays open. A contradiction is a stored,\u00a0queryable\u00a0object, not a footnote. Screenshot by\u00a0author.\u00a0<\/p>\n

14. Microsoft Foundry \u2013 models now, agents later\u00a0<\/p>\n

Foundry provides the chat and embedding deployments. The application talks to it through the Azure OpenAI\u00a0v1\u00a0interface using the standard OpenAI Python SDK, which means no\u00a0api-version\u00a0parameter to chase every few months:[7]\u00a0<\/p>\n

from\u00a0openai\u00a0import\u00a0OpenAI\u00a0
\nfrom\u00a0azure.identity\u00a0import\u00a0DefaultAzureCredential, get_bearer_token_provider\u00a0
\n\u00a0
\ncredential\u00a0=\u00a0get_bearer_token_provider(\u00a0
\n\u00a0\u00a0\u00a0\u00a0DefaultAzureCredential(),\u00a0“https:\/\/cognitiveservices.azure.com\/.default”\u00a0
\n)\u00a0
\nclient\u00a0=\u00a0OpenAI(base_url=”https:\/\/YOUR-RESOURCE.openai.azure.com\/openai\/v1\/”,\u00a0api_key=credential)\u00a0
\n\u00a0
\nresponse\u00a0=\u00a0client.responses.create(model=”YOUR-CHAT-DEPLOYMENT”,\u00a0input=prompt)\u00a0<\/p>\n

For this\u00a0demo,\u00a0the\u00a0FastAPI\u00a0app orchestrates explicitly: embed, retrieve, search the wiki, build context, call the model,\u00a0validate, write. I did that on purpose\u00a0since\u00a0every step is visible, and you can put a breakpoint in any of them.\u00a0<\/p>\n

The natural evolution is Foundry Agent Service, where the same orchestration becomes an agent with a constrained toolset:[8]\u00a0<\/p>\n

search_evidence()\u00a0
\nsearch_wiki()\u00a0
\nget_concept()\u00a0
\ncheck_contradictions()\u00a0\u00a0\u00a0\u00a0\u00a0 \u2190 the gate, as a tool\u00a0
\npropose_wiki_patch()\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2190 proposes only; cannot apply\u00a0
\napply_approved_patch()\u00a0\u00a0\u00a0\u00a0\u00a0 \u2190 requires an approval token\u00a0
\nexport_obsidian_vault()\u00a0<\/p>\n

I would not hand an\u00a0agent\u00a0unrestricted database access on day one. Each tool enforces its own validation, authorization,\u00a0logging\u00a0and narrow input schema. Note that\u00a0propose_wiki_patch\u00a0and\u00a0apply_approved_patch\u00a0are separate tools: the agent can reach the first and cannot reach the second without a human in between. That separation is the whole governance model, expressed as an API surface.\u00a0<\/p>\n

For extraction, use\u00a0structured outputs, which constrain the response to a JSON Schema instead of merely requesting valid JSON.[9] The difference becomes obvious the first time a production extraction returns almost-valid JSON.\u00a0<\/p>\n

I can report that from experience now, because deploying this stack produced two field notes worth passing on:\u00a0<\/p>\n

Truncated JSON, on the very first document.\u00a0gpt-5-mini is a reasoning model, and its reasoning tokens are spent from the same\u00a0max_output_tokens\u00a0budget as the answer. With a 5,000-token cap, the extraction JSON arrived cut off mid-string. The fix in the repo: a 16,000-token budget,\u00a0reasoning: {\u201ceffort\u201d: \u201clow\u201d}\u00a0for extraction work, JSON mode (text.format:\u00a0json_object) to constrain the decoder, and one retry. A detail that cost me two failed seeding runs: the first full seed succeeded without JSON mode, then two later runs failed on different documents. Prompt-only JSON does not fail reliably \u2013 it fails intermittently, which is worse. Schema-constrained structured outputs\u00a0remain\u00a0the real fix; this is the pragmatic one.\u00a0<\/p>\n

A deployment naming quirk.\u00a0An embedding deployment named identically to its model (text-embedding-3-small) came up healthy in every status check and then returned\u00a0unknown_model\u00a0on every single call, on both the v1 and the classic route. An identical deployment named\u00a0embed-3-small\u00a0worked on the first request. Chat deployments do not show the problem. The Bicep now keeps the deployment name and the\u00a0model\u00a0name as separate parameters, with this story in the description.\u00a0<\/p>\n

15.\u00a0FastAPI\u00a0on Container Apps\u00a0<\/p>\n

The API surface:\u00a0<\/p>\n

GET\u00a0 \/health\u00a0
\nGET\u00a0 \/wiki\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 list objects, filterable by type\u00a0
\nGET\u00a0 \/wiki\/contradictions\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 the contradiction register\u00a0
\nGET\u00a0 \/wiki\/{item_id}\u00a0
\nPOST \/query\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\u00a0question, mode,\u00a0top_k,\u00a0as_of\u00a0}\u00a0
\nPOST \/ingest\/text\u00a0
\nPOST \/ingest\/file\u00a0
\nPOST \/cost-estimate\u00a0
\nPOST \/export\/obsidian\u00a0
\nPOST \/export\/obsidian.zip\u00a0<\/p>\n

One implementation detail that cost me some debugging time and is worth passing on:\u00a0\/wiki\/contradictions\u00a0must\u00a0be declared before\u00a0\/wiki\/{item_id}, or\u00a0FastAPI\u00a0matches the path parameter first and cheerfully looks for a wiki object with the id\u00a0\u201ccontradictions\u201d. Route order is significant.\u00a0<\/p>\n

Container Apps is the right runtime here because it is container-based without asking me to run Kubernetes.[10] The scaling choice is worth a word:\u00a0<\/p>\n

scale: {\u00a0
\n\u00a0 \/\/ For production, 1 warm replica avoids cold starts; for a demo, 0 costs nothing.\u00a0
\n\u00a0\u00a0minReplicas: minReplicas\u00a0
\n\u00a0\u00a0maxReplicas: 3\u00a0
\n}\u00a0<\/p>\n

minReplicas\u00a0is a parameter (default\u00a01) because the right answer depends on what you are running. In production, scale-to-zero looks like free money and then charges you a cold start plus a model handshake on the first request after every scale-in. For a demo that is exercised from a developer machine, zero is exactly right \u2014 the validated deployment in this article ran at zero and cost nothing while idle.\u00a0<\/p>\n

16. Identity \u2013 no keys, anywhere\u00a0<\/p>\n

The whole deployment runs on a user-assigned managed identity with narrowly scoped roles, and every service has local auth disabled. The Container App gets\u00a0AZURE_CLIENT_ID\u00a0in its\u00a0environment,\u00a0DefaultAzureCredential\u00a0picks it up, and no secret is ever issued to the application.\u00a0<\/p>\n

{ name: ‘AZURE_CLIENT_ID’, value:\u00a0identity.properties.clientId\u00a0}\u00a0<\/p>\n

This matters more in this architecture than in a plain RAG app, and it is worth spelling out why. A retrieval system\u00a0reads. This system writes persistent knowledge that other answers will be built on. The blast radius of a compromised credential is not\u00a0\u201csomeone read your documents\u201d\u00a0\u2013 it is\u00a0\u201csomeone edited what your organization believes\u201d.\u00a0Treat the write path accordingly.\u00a0<\/p>\n

The full\u00a0infra\/main.bicep\u00a0in the repo provisions everything: storage with versioning, AI Search with\u00a0disableLocalAuth, Cosmos with serverless and SQL role assignments, Foundry with both model deployments, Key Vault, Log Analytics, Application Insights, the Container Apps environment and the app itself, plus the six role assignments that wire them together \u2013 and, when you pass your own\u00a0deployerPrincipalId, a mirrored set for your user account, which is what lets the seeding script and the tests run from a developer machine without a single key.\u00a0<\/p>\n

az\u00a0group create\u00a0-n\u00a0rg-wikirag-demo\u00a0-l\u00a0swedencentral\u00a0
\naz\u00a0deployment group create\u00a0-g\u00a0rg-wikirag-demo\u00a0-f\u00a0infra\/main.bicep\u00a0-p\u00a0namePrefix=wikirag\u00a0<\/p>\n

17. The ingestion lifecycle\u00a0<\/p>\n

\"\"Figure\u00a012\u00a0\u2013 Full ingestion. Note steps 11-13: the existing wiki is loaded\u00a0into the extraction context, which is what makes entity resolution possible. Image by\u00a0author.\u00a0<\/p>\n

That detail in the middle is the one people miss. When the model extracts concepts from a new document, it must see what the wiki already knows.\u00a0Otherwise,\u00a0it invents\u00a0\u201cCash Settlement Basis\u201d\u00a0as a brand-new concept, without knowing that\u00a0actual-cash-value\u00a0already exists with that exact alias, and your knowledge base quietly forks.\u00a0<\/p>\n

Entity resolution in the demo:\u00a0<\/p>\n

def\u00a0_resolve_concept(self, name, aliases):\u00a0
\n\u00a0\u00a0\u00a0\u00a0“””Exact id \u2192 title \u2192 alias.\u00a0
\n\u00a0
\n\u00a0\u00a0\u00a0 Without this, ‘cash settlement basis’, ‘depreciated value’ and ‘ACV’ each become\u00a0
\n\u00a0\u00a0\u00a0 their own\u00a0page\u00a0and the wiki fragments into synonyms. The demo stops at alias\u00a0
\n\u00a0\u00a0\u00a0 matching. A production system adds embedding similarity and an LLM adjudication\u00a0
\n\u00a0\u00a0\u00a0 step for the ambiguous middle, which is where the interesting failures live.\u00a0
\n\u00a0\u00a0\u00a0 “””\u00a0
\n\u00a0\u00a0\u00a0 candidates\u00a0=\u00a0{name.lower(),\u00a0*(a.lower()\u00a0for\u00a0a\u00a0in\u00a0aliases)}\u00a0
\n\u00a0\u00a0\u00a0\u00a0for\u00a0concept\u00a0in\u00a0self.repository.list_items(“concept”):\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if\u00a0concept[“id”]\u00a0==\u00a0slugify(name):\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return\u00a0concept\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 known\u00a0=\u00a0{concept[“title”].lower(),\u00a0*(a.lower()\u00a0for\u00a0a\u00a0in\u00a0concept.get(“aliases”, []))}\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if\u00a0candidates\u00a0&\u00a0known:\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return\u00a0concept\u00a0
\n\u00a0\u00a0\u00a0\u00a0return\u00a0None\u00a0<\/p>\n

The demo ships with a passing test that ingests a document using the phrase\u00a0\u201cdepreciated value\u201d\u00a0and asserts that no new concept page is created, but it resolves onto the existing\u00a0actual-cash-value.\u00a0<\/p>\n

And here is what happens without the stronger resolution steps, measured rather than argued. When I ran all twenty-one documents through live gpt-5-mini extraction on the deployed stack, the model proposed concepts that alias matching could resolve only partially. The result was\u00a0149 concept objects where the curated graph has 19\u00a0\u2013 a\u00a0roughly 7x\u00a0fragmentation factor. The machine-extracted concepts are not\u00a0wrong,\u00a0they are just named in ways no alias list\u00a0anticipated\u00a0(\u201cRoof Inspection Requirement\u201d,\u00a0\u201cEUR 5,000 authorization limit\u201d). That number is the concrete argument for embedding similarity and LLM adjudication in the resolution chain.\u00a0<\/p>\n

Part III \u2013 The demonstration\u00a0<\/p>\n

18.\u00a0Ostermere\u00a0Mutual\u00a0<\/p>\n

The synthetic corpus is twenty-one documents for an imaginary property insurer. It is small enough to read in an afternoon and deliberately engineered so that the six failure modes from\u00a0stage\u00a05 are all live in it.\u00a0<\/p>\n

Class\u00a0Documents\u00a0Policy & product\u00a0Policy overview v1.2,\u00a0superseded v1.1, Section 4 wording (escape of water), endorsement catalogue\u00a0Underwriting\u00a0Roof guideline (20 yrs), H3 update (15 yrs), wind-zone register, referral matrix, surveyor panel note\u00a0Claims\u00a0Triage guideline, claims manual ch.7,\u00a0three claim files\u00a0(CLM-1042, CLM-1108, CLM-1155)\u00a0Regulatory & compliance\u00a0Veyland\u00a0circular (fictional regulator), fair claims handling standard, knowledge provenance standard\u00a0Informal\u00a0Two email threads, working-group meeting minutes, customer FAQ\u00a0Table 3 \u2013 The twenty-one-document synthetic corpus by class. The informal sources at the bottom are where the contradiction is recorded and where the only rationale for the 15-year threshold exists. Table by author.<\/p>\n

Those informal sources are not\u00a0decoration. The meeting minutes are where the contradiction is formally recorded as unresolved. The email thread is where the\u00a0only\u00a0explanation of the 15-year threshold exists. In my\u00a0experience,\u00a0this is exactly how real organizations\u00a0work;\u00a0the rules are in the documents, and the reasons are in someone\u2019s inbox.\u00a0<\/p>\n

A note on data provenance: every document in the corpus is synthetic and written by me\u00a0(polished with AI)\u00a0for this article, so there are no licensing constraints on its use. The dataset ships with the repository under the same MIT license as the code.\u00a0<\/p>\n

Compiled, that corpus produces:\u00a0<\/p>\n

21 sources \u00b7 19 concepts \u00b7 28 typed relationships \u00b7 4 comparisons \u00b7 5 decisions \u00b7 2 contradictions \u00b7 4 open questions \u00b7 2 processes\u00a0<\/p>\n

19. Walkthrough one \u2013 the scoped rule\u00a0<\/p>\n

curl\u00a0-X\u00a0POST http:\/\/localhost:8000\/query\u00a0-H\u00a0‘Content-Type: application\/json’\u00a0-d\u00a0‘{\u00a0
\n\u00a0 “question”: “What is the roof inspection threshold?”,\u00a0
\n\u00a0 “mode”: “hybrid”\u00a0
\n}’\u00a0<\/p>\n

Plain retrieval finds the H3 update, which says 15 years, and says\u00a0\u201c15 years\u201d.\u00a0<\/p>\n

The knowledge layer returns the decision object:\u00a0<\/p>\n

{\u00a0
\n\u00a0\u00a0“id”:\u00a0“h3-roof-inspection-threshold”,\u00a0
\n\u00a0\u00a0“type”:\u00a0“decision”,\u00a0
\n\u00a0\u00a0“summary”:\u00a0“For new\u00a0Hearthmere\u00a0business in zone H3, request a roof inspection when the\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 primary roof covering is more than 15 years old. Outside H3, the general\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 threshold of more than 20 years\u00a0remains\u00a0in force…”,\u00a0
\n\u00a0\u00a0“scope”:\u00a0“New business only. Zone H3 only. Does not apply to in-force policies\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 written before the zone model existed.”,\u00a0
\n\u00a0\u00a0“rationale”:\u00a0“Between 15 and 20 years, the frequency of total or near-total covering\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 displacement in the severe-wind band runs at roughly 3.4x the standard\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 band at the same roof age…”,\u00a0
\n\u00a0\u00a0“rationale_source”:\u00a0“INS-SYN-018”,\u00a0
\n\u00a0\u00a0“accountable_owner”:\u00a0“D. Lindqvist (Head of Property Underwriting)”\u00a0
\n}<\/p>\n

The number never appears without its scope. And the rationale, recovered from an email, is right there, which means that in eighteen months, when someone asks\u00a0why 15, the answer exists.\u00a0<\/p>\n

The email\u2019s author, incidentally, predicted this exact failure in writing:\u00a0<\/p>\n

\u201cPlease do not let this become a general 15-year rule. If someone reads the update without the zone qualifier they will apply it to the whole book, we will demand inspections on tens of thousands of perfectly ordinary roofs, and the broker complaints will be entirely justified.\u201d\u00a0<\/p>\n

That is a synthetic quote I wrote to make a point, but I do not think it is an unrealistic one.\u00a0<\/p>\n

20. Walkthrough two \u2013 the contradiction\u00a0<\/p>\n

This is the one I would put in front of a\u00a0sceptical\u00a0architect.\u00a0<\/p>\n

curl\u00a0-X\u00a0POST http:\/\/localhost:8000\/query\u00a0-H\u00a0‘Content-Type: application\/json’\u00a0-d\u00a0‘{\u00a0
\n\u00a0 “question”: “Is trace and access covered under\u00a0Hearthmere?”,\u00a0
\n\u00a0 “mode”: “wiki”\u00a0
\n}’\u00a0<\/p>\n

A retrieval system answers this. It retrieves a chunk (from the claims manual, or from the endorsement catalogue) and tells you either\u00a0\u201cyes, up to \u20ac5,000 as standard\u201d\u00a0or\u00a0\u201conly if you bought HS-TA-01\u201d. Both answers are supported by a real document. Both are\u00a0wrong, because\u00a0the firm does not have a position.\u00a0<\/p>\n

The hybrid system responds:\u00a0<\/p>\n

{\u00a0
\n\u00a0\u00a0“answer”:\u00a0“The position is NOT SETTLED. The knowledge base holds an unresolved\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 contradiction covering this question, so no answer is given…”,\u00a0
\n\u00a0\u00a0“warnings”:\u00a0[\u00a0
\n\u00a0\u00a0\u00a0\u00a0“UNRESOLVED CONTRADICTION (con-001): Trace and Access – Standard Cover or Paid\u00a0
\n\u00a0\u00a0\u00a0\u00a0 Endorsement? The system will not choose between the conflicting sources.\u00a0
\n\u00a0\u00a0\u00a0\u00a0 Owner: Y. Tanaka (Product).”\u00a0
\n\u00a0\u00a0],\u00a0
\n\u00a0\u00a0“contradictions”:\u00a0[{\u00a0
\n\u00a0\u00a0\u00a0\u00a0“id”:\u00a0“con-001”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0“status”:\u00a0“unresolved”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0“accountable_owner”:\u00a0“Y. Tanaka (Product)”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0“statements”:\u00a0[\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\u00a0“source_id”:\u00a0“INS-SYN-012”,\u00a0“locator”:\u00a0“Chapter 7.3”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“effective_date”:\u00a0“2025-11-01”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“statement”:\u00a0“Trace-and-access costs are covered as standard under\u00a0Hearthmere,\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 up to EUR 5,000…”\u00a0},\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\u00a0“source_id”:\u00a0“INS-SYN-008”,\u00a0“locator”:\u00a0“HS-TA-01”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“effective_date”:\u00a0“2026-01-01”,\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“statement”:\u00a0“Trace and access is an optional endorsement (HS-TA-01)…\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Where it does not appear on the policy schedule, trace-and-access\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 costs are not payable.”\u00a0}\u00a0
\n\u00a0\u00a0\u00a0\u00a0],\u00a0
\n\u00a0\u00a0\u00a0\u00a0“why_not_resolved”:\u00a0“Both documents are current. Neither supersedes the other. They\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0 were written by different teams. The Endorsement Catalogue is the more recent\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0 document, but recency is not applicability…”\u00a0
\n\u00a0\u00a0}]\u00a0
\n}\u00a0<\/p>\n

Both statements. Both sources. Both dates. An accountable human being. And no\u00a0answer, because\u00a0there is\u00a0no\u00a0one.\u00a0<\/p>\n

That JSON\u00a0is demo mode\u2019s deterministic scaffold. Here is what the deployed stack does with the same question \u2013 gpt-5-mini, live, over Cosmos DB and AI Search:\u00a0<\/p>\n

There is an UNRESOLVED CONTRADICTION relevant to this question. Do not treat the issue as settled.\u00a0<\/p>\n

INS-SYN-012 (Chapter 7.3, effective 2025-11-01):\u00a0\u201cTrace-and-access costs are covered as standard under\u00a0Hearthmere, up to EUR 5,000. Handlers should authorize reasonable expenditure up to this limit without referral.\u201d\u00a0<\/p>\n

INS-SYN-008 (HS-TA-01, effective 2026-01-01):\u00a0\u201cTrace and access is an optional endorsement (HS-TA-01), limit EUR 5,000, indicative premium EUR 24. Where it does not appear on the policy schedule, trace-and-access costs are not payable.\u201d\u00a0<\/p>\n

This contradiction is UNRESOLVED. Accountable owner: Y. Tanaka (Product) [con-001].\u00a0<\/p>\n

I cannot resolve or state whether trace-and-access is covered under\u00a0Hearthmere\u00a0because of the unresolved conflict above.\u00a0<\/p>\n

The model was not asked to be cautious in general. It was\u00a0handed\u00a0the contradiction object and one rule, and it followed the rule.\u00a0<\/p>\n

The second-order effect is the part I find genuinely interesting. Once a contradiction is a stored object, other things can\u00a0depend\u00a0on it. The corpus\u00a0contains\u00a0a policyholder question on claim CLM-1155 (is the leak-detection survey cost payable?)\u00a0which\u00a0cannot be answered until this conflict is\u00a0resolved.\u00a0So,\u00a0it is stored as an open question with\u00a0blocked_by: \u201ccon-001\u201d.\u00a0<\/p>\n

The system knows why it cannot answer, and it knows who\u00a0has to\u00a0decide before it can.\u00a0<\/p>\n

21. Walkthrough three \u2013 the date of loss\u00a0<\/p>\n

Claim\u00a0CLM-1108: storm damage, roof covering displaced,\u00a0date of loss 20 February 2026. The roof is 18 years old. The property, under today\u2019s classification, would be in zone H3.\u00a0<\/p>\n

Ask a retrieval system whether an inspection was\u00a0required, and it will find the H3 rule (15 years, and 18 > 15) and tell you the roof was over\u00a0threshold,\u00a0and no inspection was on file. It sounds like a finding. It is the beginning of a wrongful declinature.\u00a0<\/p>\n

Because the H3 rule took effect on\u00a01 March 2026. Nine days\u00a0after\u00a0the loss. And the policy was written in September 2025, before the zone model existed at all, so the property carried no zone classification on the day the roof came off.\u00a0<\/p>\n

curl\u00a0-X\u00a0POST http:\/\/localhost:8000\/query\u00a0-H\u00a0‘Content-Type: application\/json’\u00a0-d\u00a0‘{\u00a0
\n\u00a0 “question”: “What roof inspection threshold applied to this property?”,\u00a0
\n\u00a0 “mode”: “evidence”,\u00a0
\n\u00a0 “as_of”: “2026-02-20”\u00a0
\n}’\u00a0<\/p>\n

The\u00a0as_of\u00a0parameter is the date the question is\u00a0about\u00a0\u2013 not the date it is asked. With it set,\u00a0INS-SYN-004\u00a0is excluded from the candidate set entirely. It was not in force. The applicable threshold was the general 20-year\u00a0rule,\u00a0the roof was 18 years old, it was\u00a0under\u00a0threshold, and no requirement was breached.\u00a0<\/p>\n

The same mechanism runs in the other direction. Ask about settlement basis with\u00a0as_of: \u201c2025-08-01\u201d\u00a0and the system retrieves the superseded policy v1.1, because that is the version that governed a loss on that date, and marks it:\u00a0<\/p>\n

{\u00a0
\n\u00a0\u00a0“source_id”:\u00a0“INS-SYN-009”,\u00a0
\n\u00a0\u00a0“in_force_at_as_of”:\u00a0true,\u00a0
\n\u00a0\u00a0“note”:\u00a0“superseded today, but in force on 2025-08-01 – this is the version that\u00a0
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 governs a loss on that date”\u00a0
\n}\u00a0<\/p>\n

Under v1.1, depreciation was\u00a0not\u00a0recoverable. Under today\u2019s v1.2, it is. Same term, same policy, opposite answer \u2013 and the only thing that distinguishes them is a date that similarity search cannot see.\u00a0<\/p>\n

Both\u00a0behaviours\u00a0are covered by tests in the repo, and both\u00a0reproduce on the\u00a0live deployment. At\u00a0as_of: 2026-02-20\u00a0the deployed model answered:\u00a0<\/p>\n

Short answer: The general >20-year inspection threshold applied. [INS-SYN-003]\u00a0<\/p>\n

Why: On 2026-02-20\u00a0the\u00a0Hearthmere\u00a0guideline required an inspection when the primary roof covering is more than 20 years old [INS-SYN-003]. The H3 15-year threshold \u2026 did not take effect until 1 March 2026; this policy was\u00a0written\u00a014 Sep 2025 (so the H3 rule did not apply to it) and the roof was ~18 years old at loss. [INS-SYN-018] [INS-SYN-013]\u00a0<\/p>\n

And at\u00a0as_of: 2025-08-01:\u00a0<\/p>\n

Settlement basis: Actual Cash Value (ACV) \u2013 defined as replacement cost less depreciation \u2013 for eligible building claims under\u00a0Ostermere\u00a0Hearthmere\u00a0v1.1. [INS-SYN-009]\u00a0<\/p>\n

Is depreciation recoverable? No.\u202fUnder v1.1 depreciation is not recoverable after repair. [INS-SYN-009]\u00a0<\/p>\n

They are the two tests I would keep if I had to\u00a0delete\u00a0every other one.\u00a0<\/p>\n

One more,\u00a0live\u00a0and unscripted: the multi-hop question\u00a0<\/p>\n

Section 5.6 claimed that typed relationships let the system traverse where similarity search only ranks. I had not scripted a demo for it, so I asked the deployed stack directly:\u00a0\u201cExplain why the CLM-1042 water loss was placed at triage Level 1 under the property claim triage process.\u201d\u00a0The answer walked the whole chain:\u00a0<\/p>\n

Governing\u00a0Level-1\u00a0criteria: Level 1 \u2013 Standard is defined as\u00a0\u201cMinor visible damage, no safety concern, no continuing water entry, and estimated exposure below EUR 5,000\u201d\u00a0[INS-SYN-002].\u00a0<\/p>\n

How CLM-1042 matches those criteria: property remained habitable, no reported electrical fault [INS-SYN-005]; water flow stopped within approximately 40 minutes [INS-SYN-005]; damage confined to the water path [INS-SYN-005; INS-SYN-014]; initial repair estimate EUR 3,800, under EUR 5,000 [INS-SYN-005; INS-SYN-002].\u00a0<\/p>\n

Supporting operational guidance: the Claims Handling Manual lists indicators of a sudden escape of water \u2026 which align with CLM-1042\u2019s facts [INS-SYN-012]. CLM-1155 explicitly contrasts CLM-1042 as a datable, sudden failure [INS-SYN-014].\u00a0<\/p>\n

Note on provisional status: coverage and exclusions\u00a0remain\u00a0subject to a formal coverage determination by an authorized claims professional [INS-SYN-005].\u00a0<\/p>\n

Four sources, four hops, and the human-decision boundary preserved at the end without being asked for.\u00a0<\/p>\n

One honest observation from the same session: my first phrasing of that question (\u201c\u2026and how does the water damage cover apply to it?\u201d) got\u00a0gated by the contradiction\u00a0\u2013 CLM-1042\u2019s evidence touches the contested trace-and-access concept, so the system presented con-001 and declined to make a settled determination instead of answering the triage part. The gate\u00a0errs\u00a0on the side of stopping. For a claims workflow I consider that the right default, but it is a real trade-off: an aggressive gate will sometimes hold back an answer the user legitimately\u00a0needed and\u00a0tuning that boundary is part of\u00a0operating\u00a0the system.\u00a0<\/p>\n

22. Seeing it: the Obsidian vault\u00a0<\/p>\n

The application exports the entire Cosmos DB state into a vault. Fifty-five pages, all generated, nothing hand-written:\u00a0<\/p>\n

obsidian_vault\/\u00a0
\n\u00a0\u00a0\u00a0 Home.md\u00a0
\n\u00a0\u00a0\u00a0 Open Questions.md\u00a0
\n\u00a0\u00a0\u00a0 Concepts\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 19 pages\u00a0
\n\u00a0\u00a0\u00a0 Sources\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 21 pages\u00a0
\n\u00a0\u00a0\u00a0 Decisions\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 5 pages\u00a0
\n\u00a0\u00a0\u00a0 Comparisons\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 4 pages\u00a0
\n\u00a0\u00a0\u00a0 Contradictions\/\u00a0\u00a0\u00a0\u00a0 2 pages\u00a0\u00a0 \u2190 con-001, con-002\u00a0
\n\u00a0\u00a0\u00a0 Processes\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2 pages\u00a0<\/p>\n

\"\"Figure 13\u00a0\u2013 A slice of the generated knowledge graph. The red cluster is the unresolved contradiction and everything it blocks. Image by\u00a0author.\u00a0<\/p>\n

\"\"Figure 14 \u2013\u00a0The generated\u00a0vault\u2019s\u00a0graph view: 55 pages, all produced by the exporter from the structured store. Concepts, comparisons,\u00a0contradictions\u00a0and sources form one connected map, with Home as the hub. con-001 and con-002 sit inside it as ordinary nodes, not footnotes. Screenshot by\u00a0author.\u00a0<\/p>\n

Open the folder in\u00a0Obsidian,\u00a0and you can navigate links, inspect backlinks, follow the graph, spot orphan pages, and, most importantly, see what the system believes and tell it that it is wrong.\u00a0<\/p>\n

That last capability is, I think, the strongest argument for this entire architecture. A vector index is operationally excellent and completely opaque to a domain expert. You cannot hand an underwriter a 1,536-dimension embedding and\u00a0ask,\u00a0\u201cdoes this look right to you?\u201d\u00a0You\u00a0can\u00a0hand them a Markdown page that says\u00a0the threshold is 15 years, but only in H3, and only for new business, and here is why, and here are the four documents it came from.\u00a0<\/p>\n

They will tell you within thirty seconds whether it is right.\u00a0Markdown makes the memory auditable by the people who actually know the domain.\u00a0No other part of the stack does that.\u00a0<\/p>\n

23. The cost argument, honestly\u00a0<\/p>\n

The knowledge layer costs\u00a0more at\u00a0ingestion. I am not going to pretend otherwise.\u00a0<\/p>\n

A RAG pipeline extracts and embeds each document once. The hybrid pipeline additionally summarizes, extracts concepts and claims, resolves entities against the existing wiki, generates relationship and comparison patches,\u00a0validates, and regenerates Markdown. The write amplification is\u00a0real,\u00a0and it is not small.\u00a0<\/p>\n

The argument is that this front-loaded cost\u00a0buys down\u00a0repeated query-time reasoning.\u00a0So,\u00a0the question is not whether the wiki costs more to build (it plainly does) but whether the compilation is amortized across enough future use.\u00a0<\/p>\n

The model in\u00a0cost_model\/cost_model.py:\u00a0<\/p>\n

Compilation\u00a0= D \u00d7 Td \u00d7 M\u00a0
\nSimple RAG\u00a0= Q \u00d7 Tr\u00a0
\nHybrid\u00a0= Q \u00d7 (Tw + V \u00d7 Tv)\u00a0<\/p>\n

With the illustrative defaults \u2013 50 documents, 6,000 tokens each, 1,000 questions, 6,000 retrieved context tokens per RAG question versus 1,350 wiki context tokens, raw verification on 25% of questions:\u00a0<\/p>\n

Measure\u00a0Tokens\u00a0Source corpus\u00a0300,000\u00a0Wiki compilation\u00a0501,000\u00a0Simple RAG, 1,000 questions\u00a06,000,000\u00a0Hybrid, 1,000 questions\u00a01,725,000\u00a0Context saved\u00a04,275,000\u00a0Break-even\u00a0~117 questions\u00a0Table 4 \u2013 Token volume for the illustrative defaults: 50 documents, 1,000 questions. Compilation costs 501,000 tokens up front and saves 4,275,000 at query time, breaking even at roughly 117 questions. Table by author.<\/p>\n

\"\"Figure 15\u00a0\u2013 The hybrid line starts above zero: that is the compilation cost. It\u00a0crosses at\u00a0roughly 117\u00a0questions. Image by\u00a0author.\u00a0<\/p>\n

Now the\u00a0caveats, because\u00a0a chart like this can easily mislead:\u00a0<\/p>\n

This is token volume, not price. It ignores output tokens, embedding costs, AI Search capacity, Cosmos RUs, Container Apps compute, and Document Intelligence pages.\u00a0<\/p>\n

It treats every token as equal. In practice, ingestion and answering can use different model classes, and that is where much of the real saving lives, because a small model can answer from concise wiki context while a stronger one is reserved for reconciliation and updates.\u00a0<\/p>\n

It does not guarantee anything. A badly governed agent that rewrites the whole wiki on every ingestion will erase any saving you modelled. The economics depend entirely on disciplined update policies.\u00a0<\/p>\n

One measured data point, from deploying this exact stack: ingesting all twenty-one documents through live gpt-5-mini extraction and embeddings, running every walkthrough in this article, and regenerating the vault from Cosmos DB cost\u00a0roughly\u00a0$0.20-0.30\u00a0in total. The idle stack \u2013 free-tier search, scale-to-zero Container App, serverless Cosmos \u2013 burns about $0.05 a day. At this corpus\u00a0size\u00a0the compilation cost is coffee money.\u00a0The economics\u00a0only become interesting at scale, which is what the model above is for.\u00a0<\/p>\n

\"\"Figure 16 \u2013\u00a0Cost analysis for the resource group over the full validation period: $0.16 in total, effectively all of it gpt-5-mini tokens. Cosmos DB, Storage and Log Analytics register in cents, and the free-tier search at zero. Screenshot by\u00a0author.\u00a0<\/p>\n

The context-size assumption also survived contact with the deployed system. Measured across a set of conceptual questions on the live stack, the wiki context averaged\u00a0roughly 500\u00a0tokens against\u00a0roughly 1,750\u00a0for the equivalent evidence context \u2013 a\u00a03.5x ratio, in the same range as the 4.4x the model assumes. The absolute numbers are smaller than the\u00a0model\u2019s, because the synthetic documents are short. The ratio is the part that transfers to a real corpus. If anything, the measured ratio is slightly more conservative than the assumed one, which moves the break-even later, not earlier \u2014 worth knowing before you quote the model at a budget meeting.\u00a0<\/p>\n

Run\u00a0\/cost-estimate\u00a0with your own assumptions. Then throw them away and use telemetry from your actual corpus and query mix, priced with the current Azure calculator.[11]\u00a0<\/p>\n

And honestly, the token argument is the\u00a0weakest\u00a0argument for this architecture. The real returns are:\u00a0<\/p>\n

consistent terminology across sessions and across people;\u00a0<\/p>\n

decisions and rationale that survive the person who made them\u00a0leave;\u00a0<\/p>\n

contradictions that are visible instead of silently resolved;\u00a0<\/p>\n

an audit trail from any answer to its source;\u00a0<\/p>\n

a knowledge artifact a domain expert can\u00a0review;\u00a0<\/p>\n

continuity across agent sessions and across model upgrades.\u00a0<\/p>\n

I would\u00a0take\u00a0over a few million input tokens.\u00a0<\/p>\n

24. Governance, restated\u00a0<\/p>\n

Because the system writes, it needs\u00a0to control\u00a0a read-only system does not.\u00a0<\/p>\n

Preserve provenance.\u00a0Every statement\u00a0traces to\u00a0a chunk, to a span, to a document, at the version it was derived from.\u00a0<\/p>\n

Patch, never write.\u00a0The model\u00a0proposes;\u00a0deterministic validation and, for anything consequential, a human disposes.\u00a0<\/p>\n

Detect staleness.\u00a0last_validated_at,\u00a0superseded_by,\u00a0source_effective_date. When a source is superseded, everything derived from it is stale until re-derived.\u00a0<\/p>\n

Trim security at every layer.\u00a0Blob paths, search records, Cosmos objects, Markdown exports, agent tools, caches, telemetry. Consistently. A synthesized page must never surface information the reader could not access in the source.\u00a0<\/p>\n

Keep\u00a0the human decision human.\u00a0In the demo, the AI may summarize an intake, retrieve policy evidence,\u00a0identify\u00a0missing information, propose a provisional triage\u00a0level\u00a0and flag conflicting rules. It may not\u00a0determine\u00a0coverage, decline a claim, assess fraud, rate a risk, resolve a contradiction, or tell a\u00a0customer\u00a0a decision has been made.\u00a0<\/p>\n

The synthetic working-group minutes\u00a0contain\u00a0the best articulation of this that I managed to write, and I will let it stand as the governance principle for the whole architecture:\u00a0<\/p>\n

Handlers currently adopt the assistant\u2019s proposed triage level in\u00a0roughly 90%\u00a0of cases, which is fine, but only because they are reading the intake themselves. Removing the handler from the loop removes the thing that makes the 90% trustworthy.\u00a0<\/p>\n

That is the trap in one sentence. An automated system earns credibility under human review, and then that credibility is used as the argument for removing the review.\u00a0<\/p>\n

25. What I would build next\u00a0<\/p>\n

The repo is a baseline, not a product. The gaps I am most conscious of:\u00a0<\/p>\n

Event Grid and queue-driven async ingestion (the demo ingests synchronously because it is easier to run locally).\u00a0<\/p>\n

Document Intelligence with page and span preservation, so citations point at locations.\u00a0<\/p>\n

Strict JSON Schema structured outputs on every extraction and patch.\u00a0<\/p>\n

Entity resolution with embedding similarity and LLM adjudication, not just\u00a0alias\u00a0matching.\u00a0<\/p>\n

A human approval UI for high-risk patches: right\u00a0now\u00a0the lifecycle exists in the\u00a0design\u00a0and the low-risk path exists in the code.\u00a0<\/p>\n

Foundry Agent Service tools, with\u00a0propose\u00a0and\u00a0apply\u00a0as\u00a0separately-permissioned\u00a0surfaces.\u00a0<\/p>\n

Evaluation sets for retrieval, synthesis and the hard one: update accuracy. How do you test that a knowledge base changed\u00a0correctly?\u00a0<\/p>\n

Freshness and\u00a0contradiction\u00a0dashboards. A contradiction\u00a0register\u00a0nobody looks at is just a log file.\u00a0<\/p>\n

Per-tenant security trimming, end to end.\u00a0<\/p>\n

Model routing by task complexity and risk.\u00a0<\/p>\n

Number 7 is the genuinely open research problem, and I do not have\u00a0a good answer\u00a0to it yet.\u00a0<\/p>\n

26. To sum it all up\u00a0<\/p>\n

RAG is the evidence engine of this architecture, and nothing here is its obituary. It gives the model access to original, relevant, current source material, and there is no substitute for that.\u00a0<\/p>\n

What it does not do is remember.\u00a0<\/p>\n

The knowledge layer adds the thing that was missing: a maintained, structured, inspectable representation of what the system has already worked out, with the scopes intact, the rationale preserved, the contradictions visible, and a line back to the evidence for every claim.\u00a0<\/p>\n

RAG asks:\u00a0What should I retrieve for this question?\u00a0<\/p>\n

The knowledge layer: What should be durably true after processing everything so far?\u00a0<\/p>\n

The orchestrator: Which of those do I need to answer this safely, right now?\u00a0<\/p>\n

On Azure that separation maps cleanly:\u00a0<\/p>\n

Blob Storage\u00a0preserves the originals \u2014 the only thing you cannot regenerate.\u00a0<\/p>\n

Document Intelligence\u00a0extracts the difficult content and keeps the spans.\u00a0<\/p>\n

Azure AI Search\u00a0stores retrievable, security-trimmed evidence.\u00a0<\/p>\n

Cosmos DB\u00a0stores the evolving concepts, relationships,\u00a0decisions\u00a0and contradictions.\u00a0<\/p>\n

Microsoft Foundry\u00a0provides the models, and the path to agents.\u00a0<\/p>\n

FastAPI\u00a0on Container Apps\u00a0runs the orchestration, the temporal\u00a0scoping\u00a0and the contradiction gate.\u00a0<\/p>\n

Obsidian\u00a0makes the whole thing visible to the people who know whether\u00a0it\u2019s\u00a0right.\u00a0<\/p>\n

It is more work than simple RAG, and it costs more to ingest. In exchange you get organizational memory, consistent synthesis, explicit relationships, a visible decision history, and, the part I keep coming back to, a system that will tell you\u00a0\u201ctwo\u00a0of our documents disagree and nobody has decided yet\u201d\u00a0instead of confidently making something up.\u00a0<\/p>\n

That last capability is not a feature. It is the reason to build it.\u00a0<\/p>\n

The application is no longer\u00a0searching\u00a0a pile of documents. It is slowly building a reviewable model of a domain, while keeping the original evidence close enough to check every important conclusion against.\u00a0<\/p>\n

Thank you for taking the time to explore this architecture with me. It turned into a longer piece than I intended, because the design kept having one more part worth explaining. The\u00a0FastAPI\u00a0project, the Bicep templates, the twenty-one synthetic documents and the Obsidian vault are all in\u00a0the repository<\/a>, and\u00a0I strongly believe they\u00a0give you a practical starting point for building a persistent knowledge layer of your own. Clone it, run it in demo mode without any Azure credentials, and try to break it \u2014 I would genuinely like to hear where it fails.\u00a0<\/p>\n

Disclosure: I am a Microsoft MVP. This article reflects my own independent work and opinions; Microsoft\u00a0had no involvement\u00a0in or review of its content. All Azure usage described is based on public documentation and my own deployment.\u00a0<\/p>\n

References\u00a0<\/p>\n

[1] P. Lewis et al.,\u00a0Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks<\/a>\u00a0(2020),\u00a0NeurIPS\u00a02020\u00a0<\/p>\n

[2] A. Karpathy,\u00a0LLM Wiki<\/a>\u00a0(2025), GitHub Gist\u00a0<\/p>\n

[3] Microsoft,\u00a0Document Intelligence Layout Model<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[4] Microsoft,\u00a0Hybrid Search Overview \u2013 Azure AI Search<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[5] Microsoft,\u00a0Integrated Vectorization in Azure AI Search<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[6] Microsoft,\u00a0Azure Cosmos DB Serverless<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[7] Microsoft,\u00a0Azure OpenAI v1 API Lifecycle<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[8] Microsoft,\u00a0Foundry Agent Service Overview<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[9] Microsoft,\u00a0Structured Outputs with Azure OpenAI<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[10] Microsoft,\u00a0Deploy a Flask or FastAPI Web App on Azure Container Apps<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n

[11] Microsoft,\u00a0Plan and Manage Costs of Azure AI Search<\/a>\u00a0(2026), Microsoft Learn\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"In my\u00a0RAG-ING Ahead\u00a0series\u00a0I \u00a0cloud-native retrieval stack: speech and document processing, chunking, embeddings, Azure AI Search, and an assistant…\n","protected":false},"author":2,"featured_media":141834,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[420,7829,23817,69952,223,34940,320,7828,30539],"class_list":["post-141833","post","type-post","status-publish","format-standard","has-post-thumbnail","category-microsoft","tag-azure","tag-azure-ai","tag-deep-dives","tag-document-retrieval","tag-generative-ai","tag-knowledge-graph","tag-microsoft","tag-microsoft-ai","tag-rag"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/141833","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=141833"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/141833\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/141834"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=141833"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=141833"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=141833"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}