PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines

Prabhjot Singh, Bhushan Pawar

arXiv:2608.14588v1cs.AIcs.CLcs.MA

TL;DR

Sequential multi-agent LLM pipelines can transform hallucinations into progressively less verifiable claims without verification at handoffs. This paper models that propagation and finds that boundary gates reduce hallucination survival from 58.4% to 16.2%, whereas end-of-pipeline checking yields only a 2.3 pp improvement over no verification.

  • Problem

    Sequential multi-agent pipelines pass outputs between specialized agents without provenance, confidence scores, or source access, leaving hallucination propagation insufficiently addressed by detector-focused approaches.

  • Method

    The paper combines three experiments with a first-order Markov model of hallucination transformations across raw facts, derived computations, narrative, and invisible states.

  • Results

    Boundary gates reduce hallucination survival from 58.4% to 16.2%, a 42.2 pp improvement over end-checking, while 23.7% escape entirely.

  • Takeaways & Limitations

    Verification should occur before hallucinations become transformed, with the S1→S2 boundary prioritized because 75.4% remain catchable there.

  • Takeaways & Limitations

    The financial-domain experiments may not generalize to domains where facts are less crisply verifiable, and boundary intervention reduces internal consistency scores from 4.44 to 3.93.

Abstract

from arXiv · show

Sequential multi-agent LLM pipelines chain specialized agents without verification at handoffs, creating a structural flaw with measurable and severe consequences. We show that hallucinations injected at Stage 1 do not merely persist; they transform: raw numerical facts become derived computations, then narrative prose, then editorially approved conclusions. At each transformation, detectability degrades near-irreversibly. We formalize this as the hallucination snowball effect, a first-order Markov process over four states (Raw Fact $\to$ Derived $\to$ Narrative $\to$ Invisible) with empirically measured per-boundary escape probabilities of 24.6%, 48.3%, and 89.3%. Across 346 automatically injected hallucinations in a 4-agent financial analysis pipeline on FinanceBench, gpt-4o detection drops from 72.0% at Stage 1 to 50.9% at Stage 4, and 23.7% of hallucinations survive completely undetected in the final output. Even the strongest model tested (Qwen3.5-397B-A17B, 87.0% at Stage 1) faces a structural ceiling; projected Stage 4 detection is only ${\sim}$60--65%. Critically, boundary gates using identical RAG verification tools reduce hallucination survival from 58.4% to 16.2% versus end-of-pipeline checking (Cohen's $h = -0.911$, $p < 0.000001$), while end-checking alone achieves merely 2.3 pp improvement over no verification. When you verify matters more than whether you verify. Our model predicts survival for $n$-agent linear pipelines and prescribes optimal verification resource allocation: invest at $S_1{\to}S_2$ first, where 75.4% of hallucinations are still catchable, not at $S_3{\to}S_4$ where 89.3% have already escaped.

1. Introduction

Sequential multi-agent LLM pipelines pass text between trusting agents without provenance, confidence scores, or source access, allowing hallucinations to transform into increasingly difficult-to-detect claims. The paper shows that verification placement, rather than detection alone, determines whether these errors survive.

  • Architectural problem: Sequential agents receive only predecessor text and trust it without provenance metadata, confidence scores, or source documents.The pipeline architecture is Researcher → Analyst → Writer → Reviewer, with no verification at handoffs.
  • Detectability decay: 72.0% to 50.9%: gpt-4o detection declines by 21.1 pp from Stage 1 to Stage 4, regardless of detector sophistication.End-of-pipeline verification improves survival by only 2.3 pp over no verification, from 60.7% to 58.4%.
  • Verification placement: 58.4% to 16.2%: boundary gates reduce hallucination survival by 42.2 pp versus end-checking (Cohen’s h = −0.911, p < 0.000001).The reduction comes with a measured internal-consistency quality cost from 4.44 to 3.93.
  • Theoretical model: The paper models propagation as a four-state Markov process with escape probabilities rising from 24.6% to 89.3% across boundaries.By S3→S4, nearly 90% of hallucinations are structurally unrecoverable by downstream gates; the mechanism is presented as domain-agnostic.

2. Background and Related Work

Prior work largely evaluates hallucinations in isolated outputs or parallel multi-agent systems, leaving sequential error propagation insufficiently addressed. This work frames sequential hallucination transformation and handoff verification as distinct gaps in the literature.

  • Single-output hallucination evaluation: Existing methods such as FActScore, CoVe, VeriScore, and SAFE evaluate hallucinations at the single-output S1 level.These approaches decompose claims, self-verify, or use search-augmented evaluation.
  • Multi-agent architectures: Parallel debate systems share a factual foundation across agents, whereas sequential propagation exposes each agent only to the previous agent’s transformed output.The sequential topology is structurally absent from this line of work and reflects production architectures such as LangGraph, AutoGen, and CrewAI.
  • Adversarial versus structural failure: The hallucination snowball requires no adversary: well-intentioned agents can propagate errors because handoffs lack error correction.This distinguishes the failure mode from intentional manipulation studied in TrustAgent and rogue-agent research.
  • Handoff verification and research gap: The boundary gate repurposes RAG-style grounding for deterministic numeric verification at handoffs before downstream processing.The work addresses prior gaps by modeling sequential transformation, measuring detectability decay, and testing whether verification timing matters more than tooling.

3. Experiments

Experiments on a four-agent financial-analysis pipeline show that hallucination detectability declines sharply after early transformations, while boundary verification substantially reduces survival compared with end-of-pipeline checking. The results also indicate a structural detection ceiling that stronger models alone do not eliminate.

  • Experimental setup: 346 hallucinations were injected across 140 FinanceBench questions in a four-agent pipeline, with downstream agents receiving only upstream outputs and the Reviewer lacking source documents.The pipeline used gpt-4o-mini agents for extraction, computation, narrative writing, and internal consistency checking.
  • Detection decay: 23.7% of hallucinations survived completely undetected at the final output, although every injected hallucination was detectable at Stage 1.The result attributes invisibility to pipeline transformations rather than to hallucinations being intrinsically undetectable.
  • Detection decay: −11.6 pp was the largest gpt-4o detection drop, occurring at S1→S2 as raw numerical facts became derived computations; S3→S4 changed detection by only −2.3 pp.Retrieval detection similarly dropped 18.8 pp at S1→S2, while the Reviewer boundary was editorial rather than transformative.
  • Model capability: 87.0% was Qwen3.5-397B-A17B’s Stage 1 detection rate, yet no tested model approached 100%, indicating a structural rather than purely model-quality ceiling.Dollar amounts were detected at only 49–86% because plausible-but-wrong magnitudes can evade plausibility reasoning without ground truth.
  • Verification placement: 16.2% final survival under boundary gating compared with 60.7% for Vanilla, whereas end-of-pipeline checking improved survival by only 2.3 pp over no verification.Gate 1 alone captured 75.4% of hallucinations, nearly matching the 76.9% captured by all three gates.
  • Verification placement: 32.7% of caught cases ended with suppression without restoration because downstream agents failed to propagate corrections, while the gate’s false-positive rate was 0.4%.The quality penalty therefore reflected correction propagation rather than gate accuracy.

4. State-Transition Model

The model treats hallucination propagation as a first-order Markov process whose transformations irreversibly destroy the original verifiable claim. Measured escape probabilities rise from 24.6% to 48.3% to 89.3%, motivating early boundary verification and a predictive survival formula.

  • State Definitions: Once a raw fact becomes a derived computation, downstream agents cannot recover the original checkable claim because they receive only transformed text.This irreversibility explains why end-of-pipeline verification fails: the information needed to verify the original claim no longer exists in verifiable form.
  • Measured Transition Probabilities: 24.6% → 48.3% → 89.3% are the measured per-boundary escape probabilities, quantifying progressively irreversible hallucination transformation.The probabilities are derived from Experiment 3 gate detection rates as sk = 1 − gate detectionk.
  • Measured Transition Probabilities: 89.3% of narrative-embedded hallucinations escape at S3→S4, making nearly 90% structurally unrecoverable by the time an end gate runs.The near-unity S3→S4 escape probability provides the mathematical explanation for end-checking’s limited effectiveness.
  • Predictive Formula: 0.246 × 0.483 × 0.893 ≈ 10.6% predicted survival in the 4-agent setup versus 16.2% measured survival.The 5.6 pp gap reflects gate false negatives caused by value reformatting, rounding, and unit conversion.
  • Design Prescriptions: 75.4% detection at S1→S2 makes it the first verification priority, followed by 51.7% at S2→S3; S3→S4 contributes only 10.7% marginal detection.The model therefore prioritizes early gates in resource-constrained deployments.

5. Discussion

The discussion identifies limits to the current evidence, explains why quality scores miss factual benefits, and argues that verification should precede transformations that destroy verifiability across pipeline topologies and domains.

  • Limitations: The study’s rates come from financial experiments and a strictly linear four-agent topology, so domain-specific rates and nonlinear propagation dynamics remain uncertain.The S1→S4 transformation mechanism is presented as domain-agnostic, but its specific rates are not; branching, parallel, and cyclic extensions remain future work.
  • Measurement limitations: −0.51 quality drop (3.93 vs. 4.44) reflects an evaluator that measures internal consistency without ground-truth access, making it blind to factual accuracy.A fluent report containing a fabricated $71.2B figure can score higher than a report that correctly suppresses it.
  • Intervention limitations: 22.9 pp separates actionable detection (76.9%) from full correction (54.0%), because annotating refuted values does not always propagate the correct figures downstream.Boundary gates are deterministic, add approximately 0.9 seconds across three gates, and a stronger intervention would retrieve and inject correct values directly into agent context.
  • Generalization to other topologies: The Markov formulation extends to n-agent linear pipelines through a survival product formula, while non-linear topologies require different propagation formulations.The discussion states that the key verification principle generalizes even when the exact formula does not.
  • Generalization to other topologies: Verification value is highest immediately before a transformation that destroys verifiability and lowest after narrative embedding, extending the core insight beyond linear pipelines.For branching systems, gates should occur at merge points; for memory-augmented systems, verification should precede persistent memory writes.
  • Broader significance: The hallucination snowball is framed as a systems-architecture problem affecting sequential-agent pipelines in medical summarization, legal review, and other domains.These settings similarly transform outputs through extraction or parsing, synthesis or retrieval, and report or brief generation without structured verification.

6. Conclusion

The hallucination snowball effect describes how errors transform across sequential agent handoffs, progressively destroying verifiability and creating a structural ceiling on downstream detection. Earlier boundary intervention controls propagation more effectively than end-of-pipeline checking, though with a measurable output-quality tradeoff.

  • Hallucination snowball effect: 21.1 pp: gpt-4o detection degrades across four stages, while 23.7% of hallucinations escape entirely and the strongest model faces a Stage 4 survival ceiling.The strongest model tested reaches 87.0% at best-case conditions, with projected 35–40% survival at Stage 4.
  • Earlier intervention: 42.2 pp: boundary gates reduce hallucination survival, while the S3→S4 escape probability reaches 89.3%, making early intervention critical.Gate 1 at S1→S2 captures 75.4% of hallucinations, and the reduction is supported by five independent statistical tests.
  • Accuracy–coherence tradeoff: 4.44 to 3.93: boundary intervention lowers internal-consistency quality, primarily through suppression-without-restoration rather than false positives.Suppression-without-restoration accounts for 32.7% of caught cases, compared with 0.4% attributed to false positives.
  • Practical reframe: The paper reframes hallucination handling from detection to propagation control, presenting it as a measurable, reproducible, and addressable systems-engineering problem.For high-stakes multi-agent pipelines, the recommended principle is to act before transformation rather than after it.

A. Full Statistical Results · A.1. Hallucination Trajectory Distribution (Experiment 1, n=346)

In Experiment 1 (n=346), every injected hallucination was detectable at Stage 1 by at least one instrument. The all-missed trajectory count was exactly zero, indicating that invisibility emerged during the pipeline rather than at arrival.

  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): 0 hallucinations followed the all-missed trajectory across the pipeline.The ×→× →× →× count is exactly zero.
  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): 346 injected hallucinations were included in Experiment 1.The trajectory distribution is reported for Experiment 1, n=346.
  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): Every injected hallucination was detectable at Stage 1.At least one instrument detected each hallucination at S1.
  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): At least one instrument detected every hallucination at Stage 1.No hallucination arrived invisible at the pipeline’s first stage.
  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): The pipeline created invisibility during processing.The hallucination did not arrive invisible.
  • A.1. Hallucination Trajectory Distribution (Experiment 1, n=346): The hallucination did not arrive invisible.The all-missed trajectory count was exactly zero at Stage 1.

A.2. Experiment 2: All Six Pairwise McNemar Tests (n=346)

Across 346 cases, Experiment 2 reports all six pairwise McNemar contingency tables comparing hallucination detection among four models. The tables provide counts for both caught, only A caught, only B caught, and both missed outcomes.

  • Pairwise comparisons: Six pairwise McNemar tests compare Meta-Llama-3-70B-Instruct, gemini-2.5-flash, DeepSeek-V3.2, and Qwen3.5-397B-A17B across 346 cases.Each comparison reports four outcomes: both caught, only A caught, only B caught, and both missed.
  • Pairwise comparisons: 156/22/95/73, 166/12/96/72, and 171/7/130/38 are the contingency counts for Meta-Llama-3-70B-Instruct versus gemini-2.5-flash, DeepSeek-V3.2, and Qwen3.5-397B-A17B, respectively.The four-number sequences correspond to both caught / only A caught / only B caught / both missed.
  • Pairwise comparisons: 212/39/50/45, 236/15/65/30, and 242/20/59/25 are the contingency counts for gemini-2.5-flash versus DeepSeek-V3.2, Qwen3.5-397B-A17B, and DeepSeek-V3.2 versus Qwen3.5-397B-A17B, respectively.The four-number sequences correspond to both caught / only A caught / only B caught / both missed.

A.3. Experiment 3: Five Independent Statistical Tests

Across five mutually consistent statistical tests, Ours significantly reduced hallucination survival versus both Vanilla and End-Check, with large effects and p-values below 0.000001. The advantage was 44.5–42.2 percentage points.

  • Ours vs. Vanilla: Ours vs. Vanilla: 16.2% vs. 60.7%, Δ=−44.5 pp, with all five tests indicating a significant large effect.McNemar χ2 = 136.10, permutation test p < 0.000001, Cohen’s h = −0.958, unpaired χ2 = 142.95, and Fisher’s exact OR = 7.996.
  • Statistical consistency: All five tests were mutually consistent, with p-values below 0.000001 and therefore orders of magnitude below any Bonferroni-corrected threshold.The comparisons included McNemar, permutation, Cohen’s h, unpaired χ2, and Fisher’s exact tests.
  • Ours vs. End-Check: Ours vs. End-Check: 16.2% vs. 58.4%, Δ=−42.2 pp, with all five tests indicating a significant large effect.McNemar χ2 = 122.24, permutation test p < 0.000001, Cohen’s h = −0.911, unpaired χ2 = 129.94, and Fisher’s exact OR = 0.138.

A.4. Experiment 2: Detection by Injection Type … C. Infrastructure

Experiment 2 shows that detection depends strongly on injection type: percentage hallucinations are easiest to catch, while dollar amounts are hardest because plausible perturbations evade detection without ground truth. The injection protocol systematically targets numeric expressions using deterministic perturbations and a reproducible multi-model infrastructure.

  • A.4. Experiment 2: Detection by Injection Type: 71–98%: percentage hallucinations are caught at Stage 1 because large additive shifts often appear obviously implausible.This detection range contrasts with dollar-amount hallucinations, whose smaller relative perturbations can remain plausible.
  • A.4. Experiment 2: Detection by Injection Type: 49–86%: dollar-amount hallucinations are caught at Stage 1, making them the hardest injection type to detect.A 12.8% perturbation to a large company’s financials can fall within an industry-plausible range without ground-truth access.
  • B.1. Regex Patterns: Three numeric expression classes are detected and perturbed case-insensitively in priority order.The classes are dollar amounts, percentages, and large non-dollar, non-percentage numbers.
  • B.1. Regex Patterns: Year values are excluded contextually for 1900–2099 and unconditionally for four-digit values from 1950–2030.Contextual exclusions inspect nearby terms such as FY, FISCAL, YEAR, quarterly labels, 10-K, 10-Q, ANNUAL, and QUARTER.
  • B.2. Perturbation Rules and Seed Structure: 15–40%: dollar amounts and large numbers receive multiplicative shifts, while percentages receive additive shifts of 3–12 percentage points.Injection direction is randomized, each question receives 2–3 injections, and deterministic seeds govern question and injection-level randomness.
  • B.2. Perturbation Rules and Seed Structure: 2–3 injections: each question receives multiple perturbations, with retry-on-no-change logic and fully deterministic seed construction.The protocol uses RANDOM SEED=42 plus question-derived and injection-index-derived sub-seeds.
  • C. Infrastructure: Python and LangGraph StateGraph implement the pipeline, with APIs spanning OpenAI, Google GenAI, HuggingFace Router, and Novita.Tested models include gpt-4o, gpt-4o-mini, gemini-2.5-flash, Meta-Llama-3-70B-Instruct, DeepSeek-V3.2, and Qwen3.5-397B-A17B.
  • C. Infrastructure: 0.3-second delay: Experiment 3 rate-limits API calls, while all experiments use three retry attempts with a two-second base delay.These controls standardize API execution and failure handling across the experimental infrastructure.
Loading 2608.14588v1…