PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbots

Philip Feldman, James R. Foulds, Shimei Pan

arXiv:2403.01193v3cs.CLcs.AI

TL;DR

The paper examines underexplored errors in context prompting and their implications for RAG reliability. It evaluates model responses with and without CV context, finding higher accuracy with context, while noting that credible hallucinations can still occur.

  • Problem

    Errors from context-prompting approaches and their effects on user experience and model reliability remain underexplored, despite the importance of reliable conversational AI.

    Source: Background · p. 2

  • Method

    The study compared responses to academic-profile prompts with and without participants’ CVs as context, using participant evaluations of model outputs.

    Source: Methods · p. 3

  • Results

    With context, responses were accurate approximately 94% of the time, versus 7.31% without context, an approximately 18-fold increase in likelihood of a correct answer.

    Source: Results · p. 4

  • Takeaways & Limitations

    Context improved response quality in this study, while the remaining error types point to the need to better understand context use and prompt engineering.

    Source: Conclusions · p. 7

    The authors flag credible hallucinations produced by synthesizing factual content, along with unusual formatting and incomplete context, as reliability concerns needing further attention.

    Source: §4.3 · p. 5, Conclusions · p. 7

Abstract

from arXiv · show

Large language models (LLMs) like ChatGPT demonstrate the remarkable progress of artificial intelligence. However, their tendency to hallucinate -- generate plausible but false information -- poses a significant challenge. This issue is critical, as seen in recent court cases where ChatGPT's use led to citations of non-existent legal rulings. This paper explores how Retrieval-Augmented Generation (RAG) can counter hallucinations by integrating external knowledge with prompts. We empirically evaluate RAG against standard LLMs using prompts designed to induce hallucinations. Our results show that RAG increases accuracy in some cases, but can still be misled when prompts directly contradict the model's pre-trained understanding. These findings highlight the complex nature of hallucinations and the need for more robust solutions to ensure LLM reliability in real-world applications. We offer practical recommendations for RAG deployment and discuss implications for the development of more trustworthy LLMs.

Introduction

LLMs can produce credible-seeming but inaccurate information, with legal cases illustrating the risks of hallucinations. This paper tests context prompting as used in RAG and examines when it has limited effectiveness.

Introduction

  • LLMs can generate credible-seeming but inaccurate information, and court filings citing non-existent cases illustrate the real-world stakes.The introduction describes a June 2023 case and a later instance involving cases that did not exist.Source: Introduction · pp. 1–2
  • User training to think critically about LLM output may not address how people interact with and trust these systems.Source: Introduction · p. 1
  • The authors favor terms such as “trajectory” and “navigate” over language implying that LLMs think or decide.They describe prompts as a self-influencing system acting on the LLM substrate.Source: Introduction · p. 1
  • RAG adds retrieved information as prompt context, but more context may improve relevance in unusual scenarios without guaranteeing accuracy.The paper illustrates this limit with an octopus analogy about language learned from intercepted communications.Source: Introduction · p. 2
  • The study compares context prompting with the same prompt without context and examines when context prompting has limited effectiveness.Source: Introduction · p. 2

Background

RAG systems use external information to enrich prompts, and prior work has reported reduced hallucinations. The section identifies context-prompting errors and their effects as underexplored questions.

Background

  • RAG systems use an external datastore at inference to enrich prompts with information that may include context, history, and recent knowledge.Source: Background · p. 2
  • Prior research reported that RAG can reliably reduce hallucinations, while other work links hallucinations to training-data biases and finds repeated responses can help assess factuality.Elements recurring across multiple responses were reported as more likely to be factual.Source: Background · p. 2
  • How context-prompting errors arise, including possible misrepresentation of retrieved data, and how they affect user experience and reliability remain underexplored.Source: Background · p. 2

Methods

The study used participants’ own CVs as a known source for evaluating responses to prompts with and without context. Participants rated model outputs across academic publications, work history, and education questions.

Methods

  • The study used academics’ CVs as a best-case proxy for RAG search results, allowing evaluators to know the source text behind model answers.The authors expected participants’ familiarity with their CVs to help them identify subtle inaccuracies or omissions.Source: Methods · pp. 2–3
  • CV diversity in length, ordering, and format provided variation in the documents supplied to the model.Source: Methods · p. 3
  • Participants saw prompts about publications, work history, and education, with a CV inserted as context in one condition and omitted in the other.The context-based prompt placed the CV at the {cv} marker.Source: Methods · p. 3
  • The model was OpenAI gpt-3.5-turbo-16k-0613, and participants rated each response’s hallucinated, partial, accurate, or unhelpful items.Figure 1 shows an unhelpful response in which the model says it lacks access to the data.Source: Methods · p. 3, Figure 1 · p. 4
  • The study randomly assigned participants to context or no-context conditions based on whether their database subject ID was odd or even.Source: Methods · p. 3
  • The study gathered 336 prompt-response pairs from 56 participants and produced 1,918 usable enumerated items.The prompt requested numbered lists to support itemized evaluation.Source: Methods · p. 3

Results

Context substantially improved user-rated accuracy, but remaining errors included navigation across CV sections, instruction-context mismatches, synthesis over missing context, and sensitivity to unusual formatting. In mismatch cases, responses could be unhelpful without being incorrect or hallucinated.

Results

  • With context, responses were accurate approximately 94% of the time versus 7.31% without context—an approximately 18-fold increase, as summarized in Table 1.These figures reflect participants’ assessments of responses.Source: Table 1 · p. 4, Results · p. 4
  • Of 1,125 items, users marked 68 incorrect; 21 of 168 context-provided responses were incorrect, grouped into five categories.Source: Results · p. 5

4.1 Noisy Context - 8 Responses (38.1%)

  • The most common context-based response-augmentation error was prompt navigation: responses based on one CV section sometimes continued into the next instead of stopping after relevant information was retrieved.Source: §4.1 · p. 5

4.2 Mismatch between instruction and context - 4 Responses (19.0%)

  • In several cases, query and context were too misaligned for successful answer extraction; some responses said information was absent from context or refused, but none were incorrect or hallucinated.Source: §4.2 · p. 5

4.3 Context-based Synthesis - 4 Responses (19.0%)

  • 4 CVs had missing or distorted context; one empty work-history section led the model to invent ten jobs, raising concern about more credible hallucinations.The model drew on education and publication histories to construct a plausible sequence.Source: §4.3 · p. 5

4.4 Unusual Formatting - 4 Responses (19.0%)

  • When CV context differed substantially from common structure—particularly when dates followed long job descriptions and sat near the next position and institution—the model could associate them incorrectly, producing subtly wrong responses.Source: §4.4 · p. 6

4.5 Incomplete Context - 1 Response (4.8%)

  • In one incomplete-context case, the model could not search online for a DOI and gave an incomplete answer limited to the supplied context.Source: §4.5 · p. 6

Discussion

Context prompting raised accuracy to 94% from 7.3% without context, but RAG can still mislead when retrieved material is complex or biased. The discussion contrasts LLM synthesis with information retrieval and warns that real-world retrieval errors and disclaimers do not eliminate the risks.

Discussion

  • Traditional information retrieval uses precision and recall, but the authors argue these measures are not useful for answers synthesized across an LLM’s training set.They describe such answers as requiring interpretation, sometimes through benchmarks including HellaSwag, TruthfulQA, and MMLU.Source: Discussion · p. 6
  • RAG combines retrieved external information with a prompt, yet the LLM may add biases or distort the response.The authors characterize RAG as a hybrid of an oracle-like LLM and a library-like retrieval system.Source: Discussion · p. 6
  • Missing sections can prompt hallucinations, while unconventional date placement can cause an event’s time to be attributed to another event.Source: Discussion · p. 6
  • Biased training data can make generated answers diverge substantially from ground truth, as in summaries that recast a Chinese spy balloon as a hypersonic spy plane.The authors connect this example to reporting that emphasized China’s technological prowess.Source: Discussion · p. 6
  • 94% accuracy with context, versus 7.3% for the un-augmented model, shows a substantial improvement in the experiments.The authors caution that RAG may still struggle when context falls beyond the model’s training data.Source: Discussion · pp. 6–7
  • The authors estimate that an equivalent RAG system could produce misleading or incorrect results in approximately 500 million of 8.5 billion daily Google searches, and say real retrieval errors could increase the number.This is presented as a potential extrapolation from the cited 2023 search estimate.Source: Discussion · p. 7
  • The authors argue that a disclaimer cannot wipe away these errors or the harm they may cause.Source: Discussion · p. 7

Conclusions

The conclusion reports an 18-fold accuracy improvement with context, while emphasizing that errors persist even with accurate context. It points to further work on understanding context use and prompt engineering, and flags credible hallucinations, unusual formatting, and incomplete context as reliability concerns.

Conclusions

  • An 18-fold improvement over situations without context accompanied 94% accuracy in navigating the text.Source: Conclusions · p. 7
  • The 6.04% of incorrect responses fell into five categories: noisy context, mismatched instructions and context, context-based synthesis, unusual formatting, and incomplete context.Noisy context was the most prevalent category.Source: Conclusions · p. 7
  • The error types can produce believable distortions that may be dangerous without close scrutiny.Source: Conclusions · p. 7
  • The authors flag credible hallucinations formed by interpolating between factual content, along with reliability problems involving unusual formatting and incomplete context.They call for further research attention to these issues.Source: Conclusions · p. 7
  • The authors identify managing user expectations and improving prompt engineering as opportunities for future research.This follows an instance in which a participant expected a DOI to lead to a comprehensive paper description.Source: Conclusions · p. 7
  • The conclusion presents better understanding of context use and prompt engineering as a path toward enhancing RAG systems.Source: Conclusions · p. 7
Loading 2403.01193v3…