Building a RAG Pipeline That Doesn't Hallucinate
Retrieval-Augmented Generation connects a language model to an organization's proprietary data, enabling grounded responses that reference specific documents rather than relying solely on the model's training data. The architecture is conceptually simple — retrieve relevant content, inject it into the prompt, generate a response — but the implementation decisions at each stage have outsized impact on output quality.
This post covers the key architectural decisions, common failure modes that produce hallucinated outputs, and the mitigation strategies we apply in production deployments on AWS Bedrock and Azure.
Pipeline architecture
A production RAG system involves four stages: document ingestion, chunking, embedding and indexing, and retrieval-augmented generation. Each stage introduces decisions that compound downstream.
During ingestion, source documents are parsed into clean text. PDF extraction quality varies significantly by library and document structure — scanned documents require OCR, tables require specialized parsing, and headers/footers must be stripped to avoid polluting the index. This preprocessing step is often underestimated in terms of engineering effort.
Chunking strategy
Chunk size is the single most impactful parameter in a RAG pipeline, yet it receives less attention than model selection in most implementations.
Chunks that are too small (100–200 tokens) lose contextual meaning. The retriever may surface a relevant fragment, but the LLM lacks sufficient surrounding context to generate a coherent answer. Chunks that are too large (2000+ tokens) dilute relevance — the retrieved passage contains the answer but also substantial irrelevant content that the model must filter through, increasing the probability of hallucination.
The effective range for most document types is 400–800 tokens, with 10–20% overlap between adjacent chunks to prevent information loss at boundaries. However, this is a starting point — optimal chunk size varies by document type, query patterns, and the embedding model's context window. Evaluation against representative queries is necessary to calibrate.
Embedding model selection
The embedding model determines how effectively the retriever can match queries to relevant content. The choice involves tradeoffs between cost, latency, multilingual support, and retrieval accuracy.
OpenAI's text-embedding-3-small offers a reasonable cost-quality balance for English-language corpora. Cohere's embed-v3 provides stronger multilingual performance. For organizations requiring self-hosted inference (data sovereignty requirements), open-source models like BGE-large or E5-large-v2 achieve competitive retrieval quality without external API dependencies.
Embedding model changes require full re-indexing of the document corpus. This makes the initial selection consequential — switching models later is operationally expensive for large document sets.
Hallucination failure modes
Irrelevant retrieval: The retriever surfaces passages that are topically adjacent but don't contain the answer. The model, pressured to respond, synthesizes an answer from insufficient evidence. Mitigation: implement a relevance score threshold (typically cosine similarity > 0.7) below which the system returns "insufficient information" rather than generating a response.
Context ignored: The model generates from its parametric knowledge rather than the retrieved context. This occurs more frequently with shorter contexts and less explicit prompting. Mitigation: structure the prompt with clear delineation between retrieved context and the generation instruction. Include explicit directives: "Answer based only on the provided context. If the context does not contain sufficient information, state that clearly."
Stale information: The indexed documents are outdated relative to the query. Mitigation: implement incremental re-indexing triggered by document modification timestamps. For rapidly changing data, consider hybrid approaches that combine vector retrieval with real-time database queries.
Platform considerations
AWS Bedrock Knowledge Bases provides a managed RAG implementation with S3 as the document source, OpenSearch Serverless as the vector store, and Bedrock foundation models for generation. The managed approach reduces operational overhead but constrains architectural flexibility.
Azure AI Search + Azure OpenAI offers tighter integration with the Microsoft ecosystem and supports hybrid search (combining vector similarity with keyword matching), which improves retrieval accuracy for certain query patterns.
Self-hosted vector databases (Chroma, Weaviate, Qdrant, pgvector) provide maximum architectural control at the cost of operational responsibility for scaling, backup, and availability.
Getting help
RAG pipeline architecture involves decisions that are difficult to reverse once documents are indexed and the system is in production. If you're planning a RAG deployment or troubleshooting quality issues in an existing pipeline, book a free 15-minute call and we'll discuss your specific requirements.