RAG Architecture in Production: Enterprise Lessons

Retrieval-Augmented Generation has become the default architecture for enterprise LLM applications. The concept is simple: retrieve relevant documents, feed them to an LLM as context, and generate grounded responses. The implementation, however, is full of landmines. Here's what we've learned building RAG systems for enterprise clients.

The Gap Between RAG Demos and RAG in Production

Every RAG tutorial makes it look easy: chunk your documents, embed them, store in a vector database, retrieve top-K results, prompt the LLM. You can get a working demo in an afternoon. Getting that demo to production quality takes months.

The problems that don't show up in demos but dominate production:

• Documents with tables, images, headers, and mixed formatting that break naive chunking
• Users asking questions that span multiple documents or require reasoning across chunks
• Retrieval returning technically relevant but contextually wrong passages
• Stale data as source documents get updated
• Hallucinations that sound plausible but are subtly wrong
• Latency requirements that conflict with retrieval quality

Chunking Strategy Matters More Than Your LLM Choice

We've tested extensively, and chunking strategy has a bigger impact on answer quality than the choice of embedding model or LLM. Here's what works:

Semantic Chunking Over Fixed-Size

Fixed-size chunks (e.g., 512 tokens with overlap) are the default in tutorials. They're also terrible for real documents. A fixed-size chunk might split a paragraph mid-sentence, separate a table header from its data, or combine unrelated sections. Semantic chunking — splitting at natural boundaries like paragraphs, sections, and topic shifts — produces dramatically better retrieval results.

Hierarchical Chunking

For complex documents, we use a two-level approach: smaller chunks (200-400 tokens) for precise retrieval, with parent chunks (1000-2000 tokens) that provide surrounding context. The retrieval matches on small chunks for precision, then expands to the parent chunk before feeding to the LLM for context completeness.

Metadata Enrichment

Every chunk should carry metadata: source document, section heading, page number, document date, document type, and any relevant categorization. This metadata enables filtered retrieval (e.g., "only search financial reports from 2025") and helps the LLM ground its responses with proper citations.

Retrieval Is Not Just Vector Search

Pure vector similarity search fails in surprising ways. "What was our Q3 revenue?" might return passages about Q2 revenue or competitor revenue because the embeddings are semantically similar. We've found that hybrid retrieval consistently outperforms pure vector search:

• Vector search for semantic similarity (catches paraphrases and conceptual matches)
• BM25/keyword search for exact term matching (catches specific names, numbers, and technical terms)
• Metadata filtering for structural constraints (date ranges, document types, departments)
• Re-ranking with a cross-encoder to reorder the combined results by actual relevance to the query

The re-ranking step is particularly impactful. Initial retrieval casts a wide net; the re-ranker applies fine-grained relevance scoring that catches many of the "semantically similar but contextually wrong" cases.

Evaluation Is the Hardest Part

How do you know if your RAG system is actually good? This is where most teams struggle. You need:

• A golden dataset: 100-200 question-answer pairs created by domain experts, covering common queries, edge cases, and adversarial questions.
• Retrieval metrics: Are the right chunks being retrieved? Measure precision@K and recall@K against your golden dataset.
• Answer quality metrics: Faithfulness (does the answer match the retrieved context?), relevance (does it actually answer the question?), and completeness.
• Production monitoring: Track user feedback, query patterns, retrieval confidence scores, and LLM response latency in production.

Without rigorous evaluation, you're guessing. We've seen teams deploy RAG systems with 60% answer accuracy and not realize it because they only tested with the 10 questions they used during development.

Architecture Patterns That Work

The Query Pipeline

Don't send raw user queries directly to retrieval. Transform them first:

• Query expansion: Use an LLM to generate 2-3 alternative phrasings of the query, then retrieve for all of them.
• Query decomposition: For complex questions, break them into sub-questions that can each be answered independently.
• Query classification: Route different query types to different retrieval strategies (e.g., factual lookup vs. summarization vs. comparison).

The Response Pipeline

After retrieval and generation, post-process the response:

• Citation verification: Check that every claim in the response can be traced to a specific retrieved passage.
• Confidence scoring: If retrieval confidence is low, say "I'm not sure" rather than hallucinating.
• Guardrails: Filter responses for policy compliance, PII leakage, and off-topic content.

Common Mistakes to Avoid

• Embedding everything at once. Start with your most important document set, measure quality, and expand incrementally.
• Ignoring document freshness. Build pipelines that detect source document updates and re-embed automatically.
• Using the same retrieval strategy for all query types. Factual lookups, summaries, and comparisons need different approaches.
• Skipping the re-ranking step. The cost of a cross-encoder re-ranker is minimal compared to the quality improvement.
• Not setting user expectations. RAG systems are not omniscient. Design the UX to communicate confidence levels and encourage users to verify critical information.