Source-linked AI summary
Seven Failure Points When Engineering a Retrieval Augmented Generation System
Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed Abdelrazek
TL;DR
RAG systems address gaps in LLM knowledge by retrieving domain artifacts for LLM-generated answers, but engineering them involves unresolved application-specific challenges. This paper reports lessons and seven failure points from three case studies, including empirical investigations, to guide practitioners and research.
Problem
LLMs have limitations with up-to-date or domain-specific knowledge, motivating RAG systems that use existing knowledge artifacts for answer generation.
Method
The paper presents lessons learned and seven failure points from three RAG implementation case studies, including a BioASQ experiment and validation of generated responses.
Results
The paper provides empirical insight into RAG engineering challenges and identifies failure points from three case studies for practitioner guidance.
Takeaways & Limitations
The paper proposes a practitioner reference, a research roadmap, and research directions concerning chunking and embeddings, RAG versus fine-tuning, and testing and monitoring.
Takeaways & Limitations
RAG answers can be incorrectly specific, and language models may ignore requested output formats such as tables or lists.
Abstract
from arXiv · showhide
Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community.
1 INTRODUCTION
The paper focuses on RAG as an alternative to fine-tuning for domain-specific knowledge, presenting seven failure points from three case studies and guidance for practitioners and researchers.
- Motivation: RAG uses existing knowledge artifacts with LLM generation, avoiding the need to manage or serve a fine-tuned model.The paper notes that both RAG and fine-tuning involve privacy, scalability, cost, and skills trade-offs.
- Engineering challenge: Engineers must preprocess artifacts, select storage, match queries to artifacts, rank results, and pass context documents to an LLM.The appropriate choices depend on the specific application context.
- Contributions: The study presents seven RAG failure points and lessons learned from three case studies.The stated purpose is to provide a practitioner reference and research road map.
- Research questions: The empirical experiment uses BioASQ with 15,000 documents and 1,000 question-and-answer pairs, generating responses with GPT-4 and validating them with OpenAI evaluations.Manual inspection analysed discrepancies, flagged inaccuracies, and a sample of correct labels.
- Research questions: A second research question examines key considerations when engineering RAG systems through lessons from three implementation case studies.The case studies report challenges faced and insights gained during implementation.
2 RELATED WORK
Related work identifies established information-retrieval challenges in RAG while distinguishing the additional evaluation and implementation issues introduced by LLM-based generation.
- RAG foundations: RAG uses documents to augment LLMs during pre-training and inference, making it attractive when training or fine-tuning is costly.The paper also notes that information extraction can suffer with long text.
- Research perspective: Existing surveys cover LLM use across the RAG pipeline, while this work focuses on software-engineering issues and research needs for current systems.The perspective is complementary rather than a replacement for the survey literature.
- Research gap: Benchmarking research has examined RAG performance, but not the failures occurring during implementation.The paper broadens attention beyond code-related tasks and emphasizes practitioners’ implementation challenges.
- Failure landscape: RAG inherits information-retrieval problems such as missing metrics for query rewriting, document re-ranking, and effective content summarisation.The paper reports factual-accuracy evaluation as a distinctive issue linked to semantic and generative LLM use.
3 RETRIEVAL AUGMENTED GENERATION
A RAG system indexes document chunks as embeddings, retrieves and re-ranks query-relevant chunks at runtime, then consolidates them before an LLM extracts the answer.
- Pipeline overview: RAG converts a natural-language query into an embedding, semantically searches documents, and passes retrieved documents to an LLM for answer generation.The pipeline is organized into separate Index and Query processes.
- Index process: During indexing, documents are split into chunks, embedded, and stored with their original text in a database.Chunk size and document-processing choices are engineering decisions that affect retrieval.
- Query process: At runtime, the system generalizes the question, incorporates conversational context, embeds the resulting query, and retrieves top-k similar documents.Similarity methods such as cosine similarity locate candidate documents.
- Query process: Retrieved documents are re-ranked so the chunk containing the answer is more likely to appear near the top.The design depends on the assumption that semantic similarity helps identify answer-bearing chunks.
- Query process: The Consolidator processes retrieved chunks to address LLM token and service rate limits.Reduction strategies chain prompts, creating trade-offs involving the amount of context and system latency.
- Answer extraction: Readers filter prompt noise, follow requested output formats, and extract the final answer from generated text.Implementing this stage requires customising multiple prompts for domain-relevant questions and answers.
4 CASE STUDIES
The paper studies RAG implementations in research, education, and biomedical question answering, combining deployed systems with a larger BioASQ experiment and evaluation.
- Case studies: Three case studies were conducted to identify implementation challenges, with BioASQ scripts, data, and failure-point examples made available online.The other two case studies were excluded from release because of confidentiality concerns.
- 4.1 Cognitive Reviewer: Cognitive Reviewer supports researchers by ranking uploaded scientific documents against a research objective and answering questions across those documents.It is used by Deakin University PhD students for literature reviews and indexes at runtime.
- 4.2 AI Tutor: The AI Tutor answers students’ questions from unit content and provides source access for verification.It indexes PDFs, videos, and text, transcribes videos with Whisper, and rewrites context-dependent queries using prior dialogue.
- 4.3 Biomedical Question and Answer: The BioASQ system uses biomedical questions with yes/no, summarisation, factoid, or list answers and documents prepared by domain experts.The experiment was designed to examine issues at a larger scale than the earlier case studies.
- 4.3 Biomedical Question and Answer: Automated evaluation was more pessimistic than human rating for the inspected BioASQ issues.The authors identify a validity threat because the dataset is domain-specific and reviewers were not experts.
5 FAILURE POINTS OF RAG SYSTEMS
The paper identifies seven RAG failure points spanning missing or mishandled information, unsuitable response specificity or format, and incomplete answers. These failures can occur during retrieval, consolidation, extraction, or interaction with users.
- 5 FAILURE POINTS OF RAG SYSTEMS: Missing content occurs when a question cannot be answered from the available documents, yet the system may still generate a response for related unanswered questions.The preferred behavior is to acknowledge that the answer is unavailable rather than provide a potentially misleading response.
- 5 FAILURE POINTS OF RAG SYSTEMS: Missed top-ranked documents occur when the document containing the answer is not ranked highly enough to enter the returned top K.The value of K is selected based on performance, so retrieval settings determine whether the relevant document reaches the user.
- 5 FAILURE POINTS OF RAG SYSTEMS: Not-in-context failures occur when retrieved documents containing the answer are removed during consolidation before answer generation.This can happen when many documents are returned and a consolidation process selects material for the generation context.
- 5 FAILURE POINTS OF RAG SYSTEMS: Not-extracted failures occur when the answer is present in context but the LLM does not extract it correctly, often amid noise or contradictory information.Wrong-format failures occur when the model ignores requested output structures such as tables or lists.
- 5 FAILURE POINTS OF RAG SYSTEMS: Incorrect specificity produces answers that are too general or too detailed, while incomplete answers omit information without necessarily being incorrect.For educational use, answers may need specific content rather than only the answer; multi-document questions may also cause the system to miss information, so separate questions can work better.
6 LESSONS AND FUTURE RESEARCH DIRECTIONS
The case studies identify practical failure points and research directions spanning chunking, embeddings, RAG customization, testing, and monitoring. Robust RAG engineering requires application-specific evaluation because relevant design choices and quality tradeoffs remain unsettled.
- 6.1 Chunking and Embeddings: Chunking choices affect embeddings, similarity matching, and downstream retrieval quality, motivating systematic comparisons of heuristic and semantic methods.Suggested evaluation metrics include query relevance and retrieval accuracy.
- 6.1 Chunking and Embeddings: Embedding strategies must account for document modality, chunk size, expected questions, content structure, and application domain.Changing the embedding strategy requires re-indexing all chunks.
- 6.2 RAG vs Finetuning: Fine-tuning and RAG provide distinct customization pathways: fine-tuning embeds curated internal data into the model, whereas RAG supplies relevant chunks as context.The passages frame these approaches as having different operational tradeoffs.
- 6.3 Testing and Monitoring RAG systems: Testing remains difficult because application-specific questions and answers are often unavailable for unstructured documents, while realistic domain-relevant test generation remains open.Quality metrics are also needed to help engineers make tradeoffs once suitable test data exists.
- 6.3 Testing and Monitoring RAG systems: Machine-learning adaptation ideas have not yet been applied to LLM-based RAG systems, motivating monitoring and adaptation research.The paper notes preliminary self-adaptive-systems work in other machine-learning applications.
7 CONCLUSION
The paper reports lessons from three RAG case studies and an empirical investigation involving 15,000 documents and 1000 questions. It presents practitioner guidance and research directions covering chunking and embeddings, RAG versus fine-tuning, and testing and monitoring.
- 7 CONCLUSION: The paper combines three case studies with an empirical investigation involving 15,000 documents and 1000 questions.Its findings are presented as a guide to challenges faced during RAG implementation.
- 7 CONCLUSION: Future research directions address chunking and embeddings, RAG versus fine-tuning, and testing and monitoring.These directions are linked to the implementation challenges identified in the paper.
- 7 CONCLUSION: The paper presents an investigation of RAG systems from a software engineering perspective.The stated focus is on implementation challenges and practitioner guidance.