Source-linked AI summary
Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries
Alden Do Rosario, Hussein Younes, Felipe Pires
TL;DR
Conventional RAG evaluation rewards systems for guessing when their knowledge bases lack support. This paper introduces penalty-aware scoring, knowledge-gap canaries, and failure attribution, then applies them to commercial RAG products. Accuracy when answering clusters at 97.0–98.0%, while canary violation rates range from 16.7% to 98.1%.
Problem
Conventional RAG metrics reward guessing and cannot distinguish grounded answers from parametric-memory answers on facts absent from the knowledge base.
Method
The paper combines asymmetric penalty scoring, knowledge-gap canaries, failure attribution, and blind cross-family judging in an audited product-level evaluation.
Results
Accuracy when answering clusters at 97.0–98.0% across RAG systems, while canary violation rates range from 16.7% to 98.1%.
Takeaways & Limitations
Abstention and canary behavior distinguish current RAG products more than answer accuracy, so evaluations should report both as first-class metrics.
Takeaways & Limitations
The benchmark does not test private corpora, and prompt delivery was asymmetric because CustomGPT.ai lacked the decline clause.
Abstract
from arXiv · showhide
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.
1 Introduction
The paper argues that conventional RAG evaluations reward guessing and cannot distinguish grounded answers from parametric-memory leakage. It introduces a penalty-aware framework and finds that commercial systems differ more in abstention and canary behavior than in accuracy when answering.
- Problem: Standard accuracy metrics treat correct grounded answers and lucky ungrounded guesses alike, while giving zero to both fabrications and abstentions.Under binary scoring, guessing strictly dominates abstaining, encouraging hallucination.
- Problem: Public-facts benchmarks can measure the backbone model rather than the retrieval product when parametric knowledge answers questions absent from the knowledge base.The product contract is to answer from documents or decline, but standard evaluations may not observe whether that contract is honored.
- Approach: The framework combines asymmetric penalty scoring, knowledge-gap canaries, and blind cross-family judging with quantified agreement and a pre-registered repair policy.The campaign releases provider calls, judge votes, and scoring decisions for audit.
- Results: 97.0–98.0% accuracy when answering clustered across the three RAG systems, while canary violation rates ranged from 16.7% to 98.1%.Abstention rates also differed substantially, from 0.2% to 11.1%.
- Contributions: The paper contributes a penalty-aware product-level scoring framework and a canary methodology that directly measures parametric-memory leakage.It also presents an audited three-repeat comparison with a no-RAG baseline on SimpleQA-Verified.
2 Related Work
The paper builds on work studying hallucination, factuality, self-knowledge, RAG quality, and judge bias. It positions its contribution as evaluating closed commercial RAG products while explicitly modeling abstention incentives and parametric-knowledge leakage.
- Hallucination and calibration: Prior work surveys generative hallucination and measures susceptibility to falsehoods or fabricated content through benchmarks such as TruthfulQA and HaluEval.Kadavath et al. additionally show that models carry usable self-knowledge about answer correctness.
- Hallucination and calibration: Kalai et al. explain that binary grading makes guessing dominant and propose confidence targets, which this paper operationalizes for RAG products.The connection supplies the paper’s motivation for penalty-aware scoring.
- Factuality benchmarks: SimpleQA-Verified re-verified gold answers and removed ambiguous items, while this paper adds per-question knowledge-base coverage flags for canary construction.The benchmark therefore supports short-form factuality measurement plus knowledge-gap identification.
- RAG evaluation: RAGAS, ARES, and FActScore decompose quality into faithfulness, relevance, and atomic factual precision but do not model answering-versus-declining incentives.They also do not control for parametric-knowledge leakage outside the corpus.
- LLM-as-judge: LLM judges can exhibit self-preference, motivating this paper’s cross-family panel, pivotal-vote analysis, and disclosure regime.The concern is concrete because one contestant shares a model family with any single judge selected.
3 Methodology
The methodology defines grounding and canary violations, applies asymmetric penalties that make abstention rational below a confidence threshold, and attributes failures across retrieval, generation, and judging. Responses are classified, blindly graded, scored, and logged for audit.
- Definitions: A grounding violation is an answer unsupported by the knowledge base, while parametric leakage describes production from backbone training knowledge rather than retrieved content.A canary violation operationally measures this behavior on a question whose answer is verified absent from the knowledge base.
- Definitions: On public-facts benchmarks, parametric memory can be correct even when grounding is absent, making direct violation measurement necessary.When parametric memory lacks the answer, an ungrounded response becomes a confident fabrication.
- Penalty-aware scoring: The quality score awards +1 for correct answers, −k for incorrect answers, and 0 for abstentions, unlike conventional volume scoring.This asymmetry prices incorrect attempts without penalizing abstention.
- Penalty-aware scoring: With k = 4, a system should answer iff its estimated correctness probability reaches 0.8; k = 0 makes guessing always rational.The threshold follows from comparing expected answering quality p − k(1 − p) with abstention’s score of zero.
- Knowledge-gap canaries: Canary answers receive Q = −k even when factually correct, because the answer is unsupported by construction; the no-retrieval baseline is exempt.The evaluation includes 18 canaries, producing 54 canary trials per provider across three repeats.
- Failure processing: The pipeline classifies responses as attempts or abstentions, sends attempts to panel grading, and then applies the scoring rules.Classifier failures use conservative phrase heuristics rather than a silent default.
- Failure processing: Blind grading uses anonymous provider IDs and shuffled question order, while technical API failures are excluded from scoring rather than counted as abstentions.Every exclusion is disclosed.
- Judging: A three-judge panel spanning gpt-5.4, Claude Sonnet 5, and Gemini 3 Flash assigns the majority verdict for each attempted answer.The design addresses model-family self-preference concerns.
4 Experimental Setup
The experiment compares three commercial RAG products and a no-retrieval baseline on identical knowledge-base content under frozen, documented configurations. It uses repeated runs, parity controls, disclosed prompt asymmetries, and an initial hard-set validation.
- Dataset and corpus: The benchmark uses 1,000 SimpleQA-Verified questions and a 1,000-document corpus with one web-sourced document per question, uploaded unchanged to each retrieval store.Identical content enables cross-system comparison of retrieval products.
- Systems under test: Each vendor’s shipped chunking, embedding, retrieval, and generation stack is evaluated through public APIs with a frozen roster of flagship-class backbones.The campaign was executed in August 2026 after freezing the roster in July 2026.
- Parity measures: All systems receive the same questions in fixed-seed order, fresh sessions, and temperature 0 where accepted, with latency and cost recorded per request.The reported abstention behavior uses a neutral standard prompt rather than the released stricter refusal variant.
- Prompting: OpenAI RAG and Gemini RAG receive grounding and decline instructions, whereas CustomGPT.ai uses a project-level persona because it rejects per-request system messages.The prompt asymmetry is disclosed and recorded in the run metadata.
- Prompting: CustomGPT.ai was not instructed to decline yet had the highest abstention rate, 11.1%, versus 3.3% and 0.2% for the other RAG systems.Its canary violation rate was also the lowest, so its abstention result should be read as shipped behavior while others are partly prompt-conditioned.
- Operational deviations: Gemini uses the Flash model because the Pro tier was unusable at the available API quota, and output budgets were capped at 1,024 tokens except for Gemini.The longest Gemini completion was 479 tokens.
- Validation: The validation hard set contained 107 questions drawn from repeated pilot errors and all canaries, with observed score drift ≤0.16 and no rank change.The frozen configuration was executed end-to-end twice to verify stability.
- Campaign design: The definitive campaign covers 1,000 questions × 4 systems × 3 repeats under a single-launch protocol with no result-conditioned reruns.It consumed about 16,000 provider calls and 33,000 judge votes at a total API cost under $900.
5 Results
Penalty-aware evaluation shows that deployed RAG systems are separated less by accuracy when answering than by abstention and knowledge-base violations. Rankings change because the scoring framework penalizes answers unsupported by the knowledge base, while the main ranking remains stable across penalty levels.
- Overall Comparison: About 2.7 points: every RAG system scores above the no-retrieval baseline on the penalty-aware metric, with wrong-answer rates falling from about 56% to 2–3%.OpenAI RAG has the highest Q among RAG systems; its advantage over CustomGPT.ai is significant after Bonferroni correction, while its difference from Gemini narrowly misses the corrected threshold.
- Penalty Sensitivity: k=1, 2, 4, and 9: OpenAI RAG remains first at every penalty level, while Gemini declines from second to last among RAG systems.The stricter the confidence requirement, the more a never-abstain policy costs; CustomGPT.ai and Gemini cross between k=4 and k=9.
- Accuracy and Penalty-Aware Score Disagree: 98.1%: Gemini’s canary violation rate, compared with 16.7% for CustomGPT.ai and 22.2% for OpenAI RAG, produces different conventional-accuracy and penalty-aware orderings.Gemini abstained on 2 of 3,000 answers and answered nearly all canary trials, whereas the other systems attempted far fewer canaries.
- Abstention Behavior Separates the Systems: 98.0%, 97.5%, and 97.0%: accuracy given an attempt for OpenAI RAG, CustomGPT.ai, and Gemini respectively, while abstention rates span 11.1% to 0.2%.The largest observed difference among RAG systems is abstention behavior rather than accuracy conditional on answering.
- Judge Reliability: 98.9%: the three judges were unanimous across 11,080 panel gradings, with no completed panel requiring a tie-break.Provider non-determinism changed roughly 1–3% of per-question outcomes between repeats, but rankings were unchanged in every repeat.
- Latency and Cost: 7–12s: all RAG systems share the same median latency class under identical parallel load, while per-query cost spans two orders of magnitude.The no-retrieval baseline’s 3.0s median indicates that retrieval dominates response time.
6 Why RAG Systems Hallucinate
The largest differences among these RAG systems arise from how they handle weak or absent grounding, not from generation accuracy alone. Canary violations primarily reflect answer-versus-decline policy, while retrieval-supported errors form a smaller, distinct failure mode.
- Policy-driven failures: 89% of CustomGPT.ai’s non-canary abstentions occurred when retrieval returned no citations, whereas Gemini declined only 4 times in 3,000 questions.These contrasting policies govern behavior when grounding is weak.
- Policy-driven failures: 16.7% vs. 98.1% canary violation rates show that systems differ sharply in answering questions their knowledge bases cannot support.Canary answers are measurable grounding violations because the answers are absent from the knowledge base.
- Retrieval-supported errors: 98% of CustomGPT.ai’s and 100% of Gemini’s non-canary wrong answers carried retrieval evidence, indicating retrieval-supported errors rather than answers produced from nothing.The dominant wrong-answer mode is retrieval surfacing content that does not settle the question, followed by generation answering anyway.
- Failure taxonomy: Parametric leakage on knowledge gaps is the largest and most system-dependent failure source, followed by retrieval-supported wrong answers and over-caution.The taxonomy reports canary violation rates from 0–98%, about 2% retrieval-supported wrong answers, and citation-bearing abstentions as over-caution.
7 Discussion
The discussion argues that RAG evaluation should measure abstention and knowledge-base violations alongside conventional accuracy. It also identifies cross-family judging as a practical safeguard against judge-family bias.
- Implications for buyers: Conventional accuracy could select the system with the highest rate of ungrounded answers, although private-corpus consequences remain untested.The benchmark uses public facts, where parametric memory may often supply correct answers; the private-corpus extrapolation is presented as a hypothesis.
- Implications for benchmark design: Controlled-corpus evaluations can flag absent answers and apply penalty-aware scoring post hoc, potentially reordering existing RAG rankings without changing queries.The proposed components require no provider cooperation.
- Implications for LLM judging: 98.9% judge agreement coexisted with systematic tie-break bias, while completing the three-judge panels reversed 12 of 12 degraded decisions.Cross-family majority voting with agreement reporting is presented as a low-cost defense concentrated on contested gradings.
8 Limitations and Conflict of Interest
The study discloses a vendor conflict and access asymmetry while describing mitigation and public audit materials. Its main empirical scope is constrained by the dataset, model, judge, canary, and procedural limitations reported by the authors.
- Conflict of interest: The authors are affiliated with CustomGPT.ai, one evaluated vendor, but use blind cross-family judging, disclosure, and released audit data to mitigate the conflict.The affiliated product does not lead the headline penalty-aware metric.
- Access parity: Administrative access to CustomGPT.ai was used to pin its backend model through a public settings API; all queries used public customer APIs.The passage distinguishes this access from private endpoints or internal configuration.
- Reproducibility: Code, configurations, audit logs, judge votes, statistics scripts, and the corpus manifest are publicly released for recomputation.The repository URL is provided in the paper.
- Limitations: The study uses one English short-form dataset, Gemini Flash rather than Pro, one shared-family judge, 18 canary questions, and several disclosed procedural exceptions.The canary intervals are wide, although the central contrast remains 16.7% vs. 98.1%.
9 Conclusion
The paper presents an audited penalty-aware framework showing that deployed RAG systems are separated more by when they decline than by accuracy when answering. The resulting ranking reversal persists across tested penalty settings, and the complete audit trail is released.
- Conclusion: Accuracy when answering clustered across RAG systems, while canary violation rates ranged from 16.7% to 98.1%.The conventional accuracy ranking ran in the opposite order from the canary-rate comparison.
- Conclusion: The ranking reordering persisted for every penalty setting tested, showing that the conclusion does not depend on one penalty choice.The framework, code, and complete audit trail are released for repetition and extension.