Enterprise RAG Architecture
Grounding LLM answers in company knowledge at enterprise scale — with access control enforced at retrieval, an evaluation harness, freshness pipelines, and cost controls treated as first-class architecture.
By Jai Ganesh
What is enterprise RAG architecture?
Enterprise RAG (Retrieval-Augmented Generation) architecture grounds LLM answers in a company's own knowledge — documents, policies, and data — instead of the model's training data. What makes it enterprise is what surrounds retrieval: access control enforced before content reaches the model, hybrid search for exact identifiers, an evaluation harness that catches quality regressions, and a designed no-answer path. The result is an AI system that answers from approved knowledge, respects permissions, and admits when it doesn't know.
The Problem
Basic RAG is a weekend project: chunk documents, embed them, retrieve by similarity, stuff the prompt. Enterprise RAG fails on everything the weekend version ignores — an employee retrieving chunks from an HR document they were never allowed to read, answers built on a policy that was superseded last quarter, retrieval quality silently degrading as content grows, and a cloud bill that scales with every question asked.
The hard problems are not in the LLM at all. They are in the data plane: who may see which chunk, how knowledge stays current, how you measure whether retrieval is actually finding the right passages, and how you keep inference and storage costs from spiraling in production.
This pattern separates the system into an indexing pipeline, a query path, and a set of cross-cutting concerns — because the failures live in different places and need to be engineered independently.
The Architecture
How It Works
Indexing pipeline (ahead of time)
Sources — policies, SOPs, contracts, wikis, tickets — flow through ingestion and parsing (OCR and document intelligence for scanned material), then structure-aware chunking that respects headings, clauses, and tables rather than splitting blindly at a character count. Every chunk is enriched with metadata: source, version, timestamps, and the ACL tags that make security enforceable later. Embeddings are generated in versioned batches and stored in a vector database alongside a keyword index.
Query processing
The user question is classified for intent, rewritten for retrieval quality (expanding shorthand, resolving pronouns against conversation history), and embedded. Bad queries in, bad chunks out — this stage is cheap and disproportionately valuable.
Hybrid retrieval with ACL filtering
Vector similarity and keyword search run together — semantic search misses exact identifiers, part numbers, and clause references that keyword search catches. Results are filtered by the caller's permissions at query time, then reranked so only the strongest top-k chunks spend context budget.
Context assembly and grounded generation
Retrieved chunks are assembled with citations, conversation history, and domain instructions that constrain the model to answer only from the provided context. The response carries its sources, so every claim is traceable to a document and version.
The no-answer path
When retrieval confidence is low, the system says so and escalates — to a human, with full conversation context — instead of letting the model improvise. A grounded "I don't know, here's who can help" outperforms a fluent hallucination in every enterprise setting.
Cross-cutting concerns
Access control enforced in the retrieval layer; an evaluation harness scoring retrieval quality, groundedness, and drift against a golden set; a freshness pipeline doing incremental re-indexing and version cleanup as documents change; and cost controls — semantic caching for repeated questions, tiered storage for cold content, and top-k budgets tuned per use case.
Design Decisions
ACL enforcement at retrieval, never in the prompt
Asking the model to "not reveal restricted content" is not security. Filtering chunks by the caller's permissions before they ever reach the context window is — the model cannot leak what it never saw.
Hybrid retrieval over pure vector search
Embeddings are excellent at meaning and poor at exactness. Enterprise questions are full of policy numbers, clause references, and product codes where keyword match is the difference between the right document and a plausible neighbor.
Structure-aware chunking with rich metadata
A chunk that splits a contract clause in half is unanswerable, and a chunk without version metadata is unauditable. Chunking along document structure and tagging source, version, and ACL at write time is what makes citations, freshness, and security possible downstream.
An evaluation harness before scale, not after incidents
Retrieval quality degrades silently as the corpus grows. A golden set of questions with known-correct passages, scored on every index or model change, converts "the chatbot feels worse lately" into a regression you can bisect.
The no-answer path as a designed outcome
Every RAG system has questions it cannot answer; the only choice is whether it admits it. Confidence thresholds with human escalation preserve trust — one confident hallucination about a policy costs more than a hundred honest escalations.
Cost architecture as a first-class concern
The main cost drivers — large-model inference, high-dimensional vector storage, and retrieval hitting hot memory tiers — are all controllable: semantic caching, embedding-model right-sizing, tiered storage, and per-use-case top-k budgets. Designing them in beats discovering them on the invoice.
Trade-offs & Limits
- ›Freshness lag: content changes propagate on the re-indexing cadence — near-real-time indexing is possible but multiplies pipeline complexity and cost.
- ›Retrieval ceiling: RAG answers questions whose evidence exists in retrievable text. Multi-hop reasoning across many documents and aggregate questions ("how many contracts have clause X?") need knowledge-graph or analytical complements.
- ›Chunking is domain work: the right strategy differs between contracts, wikis, and tickets — expect iteration per corpus, not one global setting.
- ›The evaluation harness and golden set require ongoing curation by people who know the content — it is operational discipline, not a one-time setup.
When To Use It
- ✓Company-knowledge assistants over policies, SOPs, contracts, and documentation
- ✓Any corpus where different users hold different access rights
- ✓Domains where answers must be traceable to a source document and version
- ✓Grounding layers inside larger agent platforms — this pattern is the knowledge layer of the multi-agent architecture
When Not To
- ✕Aggregate or analytical questions over structured data — that is Text-to-SQL territory
- ✕Tiny, static, public corpora where stuffing documents into the context window is simpler and good enough
- ✕Reasoning that spans relationships across many documents — consider a knowledge-graph-enhanced approach
Enterprise architect and independent AI consultant — I help teams take agentic systems from deck to production, with the governance story intact.