PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Augmented Generation Compositionality

Ido Levy, Segev Shlomov, Leshem Choshen

arXiv:2610.29722v1cs.CLcs.AI

TL;DR

RAG techniques are commonly evaluated one at a time, leaving unclear whether their gains persist in composed pipelines, whether retrieval and faithfulness favor the same choices, and whether small-corpus findings transfer to larger scales. This study uses LASAGNA, a stage-isolated framework, to evaluate 450 configurations across eight benchmarks and corpora from 5K to 200K documents. Results show that isolated gains do not reliably predict full-pipeline performance, and that metric and scale affect which composition performs best.

  • Problem

    RAG techniques are often evaluated one stage at a time, leaving unclear whether isolated gains persist under composition, whether retrieval and faithfulness favor the same choices, and whether small-corpus findings hold at larger scales.

    Source: §1 · pp. 1–2

  • Method

    The study introduces LASAGNA, a stage-isolated framework, and evaluates 450 configurations across eight benchmarks and corpora from 5K to 200K documents.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    +47% predicted gain from the three largest individual retrieval improvements yielded only +17% when composed, while retrieval and faithfulness favored different choices and composition rankings shifted with corpus scale.

    Source: §1 · p. 2

  • Takeaways & Limitations

    New RAG techniques should be tested in representative complete pipelines, on both retrieval and faithfulness, and at the intended corpus scale.

    Source: §6 · p. 9

    Headline profile comparisons rely on one constructed English multi-domain test bed with at most 93 retrieval and 200 generation queries per experiment, so profiles are candidates for validation rather than prescriptions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.

1 INTRODUCTION

LASAGNA studies whether isolated RAG gains persist when techniques are composed, whether retrieval and faithfulness favor the same choices, and whether rankings shift with corpus scale. Its results show that gains can diverge across metrics and scale, motivating evaluation within complete pipelines.

1 INTRODUCTION

  • The study targets gaps in whether isolated gains persist under composition, whether retrieval quality and answer faithfulness favor the same choices, and whether small-corpus findings hold at larger scale.Prior evaluations often varied one stage at a time or used representative pipelines, leaving component interactions and their scale dependence largely unquantified.Source: §1 · pp. 1–2
  • LASAGNA provides fixed interfaces for pipeline stages so a new technique can be tested in a complete modern pipeline without reimplementing the surrounding stack.The study uses a 167-configuration D-optimal design and 138 targeted ablation configurations across eight benchmarks and corpora from 5K to 200K documents.Source: §1 · p. 2
  • +47% predicted retrieval gain from the three largest individual improvements became only +17% when composed, while faithfulness gains transferred approximately near-additively.The study also found that a stronger generator helps the lightest pipeline more than the heaviest, so the heavier stack buys less when the generator is strong.Source: §1 · p. 2
  • At 200K documents, the small-corpus-leading design trails by 13.7% NDCG@10, with a rank flip visible from 25K.Across three other benchmarks at 100K–200K documents, lighter pipelines retain similar measured quality at 10–20× lower latency.Source: §1 · p. 2
  • The four operational profiles expose a Pareto tradeoff across retrieval quality, faithfulness, and response time, with no universal winner.The reported practical implication is to recheck small-corpus rankings at the intended corpus scale.Source: §1 · p. 2

2 RELATED WORK

Prior RAG work spans many techniques and stages, but existing evaluation approaches leave gaps in measuring controlled interactions across composed pipelines. LASAGNA’s workflow lets researchers add a component, declare a sweep, and run an end-to-end paired test.

2 RELATED WORK

  • RAG techniques span retrieval, reranking, compression, query augmentation, index-time enrichment, and generation-time controls, and each has shown benefits in isolation on some benchmark.The stages compose along more edges than a single-component ablation can enumerate.Source: §2 · p. 3
  • Best-practice searches identify strong end-to-end settings but do not estimate interactions, while benchmarking libraries do not enforce stage isolation needed for clean attribution.Automated search often reports only the winning setting, and evaluation suites or domain benchmarks do not provide a common controlled pipeline.Source: §2 · p. 3
  • LASAGNA’s research workflow registers a component, declares a factorial sweep in YAML, and runs end-to-end for a paired statistical test.Source: Figure 2 · p. 4

3 LASAGNA: A COMPOSITIONAL FRAMEWORK FOR RAG

LASAGNA is an open-source, modular framework for testing new RAG components inside a complete modern pipeline while keeping stages isolated. Its typed interfaces, registry, and evaluation harness support controlled comparisons against a consistent surrounding stack.

3 LASAGNA: A COMPOSITIONAL FRAMEWORK FOR RAG

  • LASAGNA provides a configurable end-to-end pipeline with a fixed interface per stage, allowing a new component to be measured against the same surrounding stack as other configurations.Behavioral tests check that swapping one implementation does not mutate a neighboring stage’s state.Source: §3 · p. 4
  • Eight component types are supported through typed dataclass input-output contracts, and a registry decorator makes new components discoverable at YAML configuration time.The framework ships with 53 built-in components, six vector backends, and adapters for the study’s eight benchmarks.Source: §3 · p. 4
  • Five automated isolation tests check that toggling one component does not silently change another, supporting attribution of interaction estimates to the varied components.Source: §3 · p. 4
  • Researchers can register a component, declare a factorial sweep in YAML, and run it end-to-end to receive a paired statistical test against the same surrounding stack.The complete study cost approximately $5,200 in cloud compute and 310 wall-clock hours.Source: §3 · p. 4

4 EXPERIMENTAL DESIGN

The experimental design tests RAG compositions across distinct retrieval and generation tracks, multiple benchmarks, and corpus scales. It pre-declares tests of additivity, profile generality, and whether rankings measured on small corpora persist at larger scales.

4 EXPERIMENTAL DESIGN

  • The study evaluates retrieval and end-to-end generation separately because retrieval quality and answer faithfulness measure distinct pipeline properties that applications may weight differently.The joint quality-latency envelope is reported as HV(Lref) = NDCG@10 × Faith × max(0, 1 − latency/Lref), with Lref=30 s.Source: §4 · p. 4
  • The full factorial spans eight dimensions and 30,720 configurations, so a 242-run D-optimal subset was drawn to estimate all main effects and 28 dimension-pair interactions within budget.Of the planned runs, 167 were executed; 63 GraphRAG-enriched runs were dropped after an ablation found no measurable NDCG@10 change despite roughly doubled latency, and 12 timed out.Source: §4 · p. 5
  • The evaluation unifies eight benchmarks into a multi-domain corpus and measures both retrieval over 5,132 documents and end-to-end generation over 5,566 documents.The study also includes 138 non-factorial ablation configurations across 24 experiments and a 5K–200K scaling sweep.Source: §4 · p. 5
  • Generation metrics include faithfulness, answer relevance, NDCG@10, exact match, token-level F1, latency, and cost; judge sensitivity checks preserve profile rankings despite shifting absolute faithfulness scores.A third-judge check reports Pearson r=0.44 and Cohen’s κ=0.35, with relative trends persisting although exact faithfulness scores depend on the judge.Source: §4 · p. 5
  • The pre-declared hypotheses test whether component gains add, whether one composition can serve multiple deployment targets, and whether rankings remain stable as corpora grow.H1 tests additivity, H2 profile generality, and H3 scale invariance.Source: §4 · p. 5
  • H3 compares composition rankings at 5K and 200K documents on the same corpus to test whether small-corpus rankings transfer across scale.The design uses per-composition log-linear fits, a slope-range threshold, bootstrap confidence intervals, and Kendall’s τ rank-flip test.Source: §4 · p. 6

5 RESULTS

Across factorial, ablation, and scaling results, component gains do not reliably carry into complete RAG pipelines, and the pattern differs by metric and corpus size. The results also show tradeoffs among retrieval quality, faithfulness, and response time.

H1: How RAG components compose

  • Three strongest retrieval gains predict +46.6% over DEFAULT_STANDARD but realize only +17.2% when combined in BALANCED, a 63% shortfall.For faithfulness, a +28% predicted gain is accompanied by a +32% gain in the best full pipeline over the same reference.Source: §5 · pp. 6–7
  • Retrieval quality and faithfulness favor different factors: embedding, enrichment, and reranking associate most with retrieval, while query augmentation and agentic gating associate most with faithfulness.Across matched configurations, the metrics are essentially unrelated (pooled Pearson r = −0.01, p = 0.91); the strongest retrieval interactions both involve embedding, while 10 of 20 query-augmentation/gating combinations depart from additivity by at least two percentage points.Source: §5 · pp. 6–7, Figure 3 · p. 7

H3: Scaling behavior

  • At 200K documents, BALANCED exceeds STANDARD by 13.7% relative (p = 0.003), after STANDARD leads at 5K and 15K.The regimes cross between 15K and 25K; BALANCED’s retrieval-stage additions include broader chunk retention, hierarchical partitioning, and post-retrieval merging.Source: §5 · p. 8, Figure 4 · p. 8
  • SPEED and STANDARD retrieval degrade log-linearly with corpus growth, while BALANCED and MAX-RELIABLE remain approximately flat.The authors attribute BALANCED’s flatter retrieval-NDCG degradation to retrieval-stage additions that sustain a denser, higher-signal top-k as distractors grow.Source: §5 · p. 8

H2: Complete-profile tradeoffs

  • BALANCED improves NDCG@10 by 4.5% and response time by 13% versus MAX-RELIABLE, with only a 0.005 faithfulness difference.The authors attribute part of this tradeoff to execution route: direct generation favors response time, while iterative verification favors faithfulness.Source: §5 · pp. 8–9
  • Across four complete profiles, no profile leads every metric; at a fixed 30 s reference, STANDARD has the highest joint hypervolume (0.523).SPEED is fastest, STANDARD leads NDCG@10, and agentic BALANCED and MAX-RELIABLE lead faithfulness; the profiles therefore expose a retrieval-quality, faithfulness, and response-time tradeoff.Source: §5 · pp. 8–9, Table 2 · p. 9

Supporting evidence: what helps and what does not

  • Targeted ablations show chunker choice spans 0.126 NDCG@10 across 20 configurations, while HyDE’s 3.1% NDCG@10 gain comes with doubled first-hop latency.Matryoshka truncation reduces vector storage and search cost by 25% while improving NDCG@10 by 2.3%; gate-triggered refinement can also remove supporting evidence.Source: §5 · p. 9

6 CONCLUSION

The paper studies RAG composition with LASAGNA and reports that component gains vary across pipeline combinations, metrics, and corpus scales. It also states the study’s validation boundary and documents its AI use and reproducibility provisions.

6 CONCLUSION

  • The study finds that isolated component gains do not necessarily persist in full pipelines, that retrieval and faithfulness favor different choices, and that the best composition changes with corpus scale.It presents LASAGNA as a stage-isolated framework and reports 450 configurations across eight benchmarks and corpus sizes from 5K to 200K documents.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI tools supported methodology, implementation, result interpretation, code and figure creation, and editing; authors reviewed AI-assisted work and re-derived results from raw files.LLMs also generated answers and acted as RAGAS judges in the studied system.Source: AI USE STATEMENT · pp. 9–10

ETHICS STATEMENT

  • Profile comparisons rely on one constructed English multi-domain test bed with at most 93 retrieval and 200 generation queries per experiment, so profiles are candidates for further validation, not prescriptions.The paper advises domain deployments to re-validate candidates on their own corpus.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The framework code is released, but paper-specific artifacts are available from the authors on request; the design and decision rules were hashed internally, without third-party timestamped preregistration.The paper cites a provenance manifest, stage-isolation conformance test, and compute accounting as reproducibility support.Source: REPRODUCIBILITY STATEMENT · p. 10

LIMITATIONS

This section documents limits on generalizing the findings, the study’s preregistered design, and how the executed study differed from it. It covers the GraphRAG test and exclusion, constraints and changes to planned analyses, and the reconciliation of the H2 and H3 claims.

LIMITATIONS

  • Evidence is limited to a constructed English test bed and proprietary generators; the small confirmatory faithfulness sample and domain-focused benchmarks warrant further validation.The confirmatory experiment used 43 unique queries rather than 200; raw faithfulness partly reflected answer length, so cross-regime claims use length-residualized scores too. Scaling degradation is an upper bound because distractors included some partially relevant documents.Source: LIMITATIONS · pp. 15–16

A.2 PRE-REGISTRATION DOCUMENT

  • The preregistration fixed hypotheses, tests, and success criteria before data collection, but had no third-party timestamp and labels its analyses as planned confirmatory.The document also states that deviations would be reported alongside both preregistered and modified analyses.Source: §A.2 · p. 16, §A.2.6 · p. 21

A.2.1 MOTIVATION AND RATIONALE

  • The study addresses a gap left by isolated evaluations: prior work notes coordinated component design and interaction effects but does not quantify their sign, magnitude, or domain dependence under controlled factorial conditions.The authors motivate preregistration as a way to distinguish confirmatory findings from exploratory analyses.Source: §A.2.1 · p. 17

A.2.2 PRE-REGISTERED HYPOTHESES

  • The study preregistered three questions: whether pipeline components interact, whether a balanced profile serves tradeoffs, and whether profile rankings change with corpus scale.The hypotheses target additivity, the faithfulness–latency–cost tradeoff, and scaling from 5K to 200K documents.Source: §A.2.2 · pp. 17–18

A.2.3 EXPERIMENTAL DESIGN

  • LASAGNA isolates pipeline stages so changing one stage does not alter another, supporting controlled interpretation of component effects.Each stage’s output is a deterministic function of the preceding stage’s output and its own configuration.Source: §A.2.3 · p. 19
  • A D-optimal fractional factorial reduced 30,720 candidate configurations to 242, estimating 232 model parameters with 10 residual degrees of freedom.The design includes 22 main effects and 209 two-way interaction terms.Source: §A.2.3 · p. 19, §A.4 · p. 21

A.5 ALIASING STRUCTURE

  • The design uses a diagnostic aliasing cutoff of |r| > 0.30 rather than guaranteeing strict orthogonality, and reports no three-factor alias above that threshold.The authors argue that residual three-way correlations at this level do not substantively confound interaction estimates.Source: §A.4 · p. 22, §A.5 · p. 22

A.4 D-OPTIMAL DESIGN GENERATION

  • NDCG@10 fell by 0.011 from 1-hop to 3-hop GraphRAG, and the approach was excluded from the factorial and scaling runs.The isolated ablation found no graph configuration improved over the baseline; the proposed faithfulness benefit was not tested.Source: Table 11 · p. 22, §A.4 · p. 22

A.6 PRE-REGISTRATION RECONCILIATION

  • The H3 rank crossover is visible from 25K and significant at 200K (Welch’s t, p=0.004), while intermediate 50K/100K comparisons are not significant (p > 0.17).Cross-profile slope range is approximately 0.10 per decade, clearing the ≥0.05 threshold.Source: §A.6 · p. 24
  • 167 of 242 planned factorial configurations (69%) were executed; GraphRAG was omitted, 12 timed-out configurations were excluded, and reduced power remains for enrichment interactions.API rate limits and Milvus Lite resets constrained execution; the 167-run subset estimates all 22 main effects and 28 two-way interactions, but its unbalanced cells affect individual interaction precision. A nested-SSR calculation also avoids double-counting shared variance that inflated η2 from 0.428 to 0.546.Source: §A.6 · p. 24, DEVIATIONS FROM THE PREREGISTRATION · pp. 24–25

DEVIATIONS FROM THE PREREGISTRATION

  • The executed H2 headline compares profiles on one SQuAD v2/MuSiQue/HotpotQA mixture, so it does not confirm the pre-registered multi-benchmark claim.MS MARCO and CUAD were replaced by TriviaQA and BEIR/FiQA; the BALANCED profile is therefore a candidate starting point pending validation on the target corpus.Source: DEVIATIONS FROM THE PREREGISTRATION · pp. 24–25
  • The pre-registered joint 28-interaction H1 test was unidentifiable on the 113-configuration data, so the study used per-pair Bonferroni correction and added a post-hoc focused-confirmatory experiment.The 232-parameter test exceeded the 113-configuration dataset and was rank-deficient.Source: DEVIATIONS FROM THE PREREGISTRATION · p. 25
  • Under the pre-registration’s variance-floor rule, η2 = 0.033 (E1) and η2 = 0.040 (per-sample joint) support additivity, but p < 0.05 rejects strict-zero additivity.The two readings support the practical conclusion that main effects dominate while cell-level residuals are small in aggregate.Source: DEVIATIONS FROM THE PREREGISTRATION · p. 25
  • A post-hoc audit of 100 LLM-audited pairs found a 0.0% false-negative rate (95% CI [0.0%, 0.0%]) and a 9.0% any-signal rate, all partial.The audit supports treating the H3 rank-flip as robust on audited splits and reported scaling degradations as an upper bound.Source: DEVIATIONS FROM THE PREREGISTRATION · p. 25
  • SHA-256 fingerprints for the 100K/200K corpus files are unavailable, so reconstruction relies on the seed-42 sampling recipe rather than content-addressed files.The sampling script is available from the authors on request.Source: DEVIATIONS FROM THE PREREGISTRATION · p. 25

A.7 COMPUTE BUDGET

The appendix details LASAGNA’s evaluation costs, three-track protocol, benchmark corpus construction, and scaling procedure. A distractor audit found no false negatives in its 100-pair sample, while the audit and corpus construction have stated scope and reproducibility limits.

A.7 COMPUTE BUDGET

  • $5,200 in API charges plus approximately $370 in local compute brought total experimental costs to approximately $5,600, with about 90% from LLM API usage.Generation evaluation cost $815 (32% of total), and RAGAS evaluation cost $590 (23%); concise prompts reduced RAGAS costs by approximately 15%.Source: §A.7 · pp. 26–27

B.1 EVALUATION DATASET DESIGN

  • Each configuration used 149 stratified queries across retrieval and generation, while the three-track protocol separated retrieval quality, answer generation, and hallucination resistance.The sample comprised 93 retrieval queries and 56 generation queries; the factorial design was reported as sufficient for main-effect and targeted-interaction testing at 80% power for η2 ≥0.03.Source: §B.1 · p. 27, §B.1.3 · p. 28

B.1.2 CORPUS ASSEMBLY

  • The retrieval track pools five BEIR domains into 5,132 documents; the E2E corpus adds QA gold passages, while scaling adds up to 45,000 distractors.The base corpus spans biomedical, financial, scientific, argumentative, and multi-hop domains; the E2E gold passages make answer evidence retrievable.Source: §B.1.2 · p. 27, Figure 5 · p. 28

B.1.5 INGESTION SIGNATURE AND CORPUS REUSE

  • LASAGNA reuses an indexed corpus when its ingestion signature matches, while changes to embedding or chunking parameters trigger fresh ingestion.Benchmark adapters implement query, corpus, and relevance-judgment loading; third-party adapters can be registered for automatic discovery.Source: §B.1.5 · p. 29, §B.2.5 · p. 32

B.2.1 BENCHMARKS USED IN THIS STUDY

  • Eight benchmarks were used across the retrieval and E2E tracks; seven other supported benchmarks were not evaluated because of infrastructure and time constraints.LASAGNA supports 15 benchmark adapters, and adding a benchmark typically requires approximately 50 lines of adapter code; datasets are downloaded from canonical sources rather than redistributed.Source: §B.2.1 · p. 29, §B.2 · p. 29, §B.2.5 · p. 32

B.3.4 100K AND 200K CORPUS CONSTRUCTION

  • Corpus scaling reaches 200K documents using held-out distractors, but the 100K and 200K compositions lack SHA-256 fingerprints and are reproducible from the construction recipe rather than corpus bytes.When the NFCorpus held-out pool constrained the draw, additional slots went to HotpotQA, slightly increasing its share in the largest corpora.Source: §B.2.7 · p. 32, §B.3.4 · p. 33, §B.3.6 · p. 34

B.3.5 DISTRACTOR-RELEVANCE AUDIT

  • 0.0% false-negative rate (95% bootstrap CI [0.0%, 0.0%], n=100) and 9.0% any-signal rate (95% bootstrap CI [4.0%, 15.0%]); the H3 rank-flip stands under the pre-specified robustness threshold.All nine partially relevant labels came from NFCorpus and SciFact; the audit’s conclusions are limited to the five audited BEIR sub-benchmarks.Source: §B.3.5 · p. 34
  • The distractor audit used a single first-pass LLM annotator and covered only the five BEIR sub-benchmarks in the scaling extension, so it does not certify additional benchmarks.The reported 0% false-negative rate is bounded above by the auditor’s own recall.Source: §B.3.5 · p. 34

B.3.8 END-TO-END SCALING RESULTS

The end-to-end scaling results compare retrieval quality and faithfulness across corpus sizes, while accompanying analyses examine scaling fits, evaluation choices, and faithfulness sensitivity to answer length.

B.3.8 END-TO-END SCALING RESULTS

  • The authors caution that rising faithfulness at scale may reflect safer, less specific answers or answer–context consistency rather than answer–truth consistency.They leave human evaluation of answer specificity at each corpus size for future work.Source: §B.3.8 · pp. 34, 36
  • At 50K documents, BALANCED overtakes STANDARD on both retrieval quality (NDCG@10) and generation faithfulness (RAGAS).Figure 6 shows both end-to-end scaling curves and marks the total change from 5K to 50K.Source: Figure 6 · p. 35

B.3.9 SCALING LAW FITS

  • SPEED and STANDARD have strong log-linear NDCG@10 fits (R2 ≥0.89; p < 0.01), while BALANCED and MAX-RELIABLE fits are weaker and their slopes are not significant.The reported fit is linear in NDCG@10 versus log10(N), not the pre-registered power-law form; numerical slope estimates and projections depend on that choice.Source: §B.3.9 · p. 36

B.4 REPRESENTATIVE QUERY EXAMPLES

  • The 56 generation queries test compositional reasoning and reading comprehension with unanswerable-question detection across three benchmarks.The appendix also describes retrieval queries spanning five diversity axes, from single-term medical lookups to multi-hop reasoning.Source: §B.4.2 · p. 36, §B.4 · p. 36

C.1 EVALUATION METRICS

  • The evaluation defines retrieval, end-to-end answer, and LLM-judged metrics; Recall@100 measures whether relevant material is surfaced before reranking, and faithfulness measures answer-claim support by retrieved context.RAGAS faithfulness uses atomic claims, and concise one-sentence prompts reduce claim-decomposition variance.Source: §C.1.1 · p. 38, §C.1.3 · p. 39

C.1.4 JUDGE BIAS AND MITIGATIONS

  • Pairwise judge comparisons swap answer order and count inconsistent preferences as ties; GPT-5.1 and Claude Sonnet 4.5 show moderate, significant agreement on sampled faithfulness scores.The cross-judge study reports Kendall’s τb = 0.38 and 66% of per-sample scores within 0.1 absolute points.Source: §C.1.4 · p. 40

C.2 GENERATION PROMPTS AND FAITHFULNESS REGIMES

  • Faithfulness values reported under different answer-generation prompt regimes are not directly comparable, while the observed gap is stable across profiles and corpus sizes, indicating a prompt/regime rather than corpus effect.Agentic-prompt faithfulness was 0.774 ± 0.031 (paired ∆= +0.161, p < 0.001).Source: §C.2 · p. 41

C.3 LENGTH-CONTROLLED FAITHFULNESS RERUN

  • A 50-question length-controlled study found that answer length is associated with faithfulness, and adjusting for length reverses the raw concise-versus-verbose ordering.The residualized means are verbose 0.824, neutral 0.791, and concise 0.714; the authors conclude that the apparent concise advantage is accounted for by shorter answers containing fewer claims to verify.Source: §C.3 · p. 42

C.4 HUMAN EVALUATION OF FAITHFULNESS

The validation and factorial analyses find that retrieval quality and faithfulness respond to different pipeline choices, while judge agreement supports per-answer scoring but not every fine-grained configuration ranking.

C.4 HUMAN EVALUATION OF FAITHFULNESS

  • The 50-example human validation is under-powered for configuration rankings and covers only three generation benchmarks, so it does not validate retrieval metrics or unseen profiles.The authors use it to establish judge validity at the per-answer level only.Source: §C.4 · p. 43
  • Two human annotators agreed strongly with each other (Pearson r = 0.96) and with RAGAS (r = 0.83 and 0.77), supporting per-answer faithfulness scoring.The authors scope the human validation to per-answer reliability, not configuration-ranking accuracy.Source: §C.4 · pp. 43–44

C.5 PAIRWISE FAITHFULNESS VALIDATION VIA BRADLEY–TERRY

  • BTL and RAGAS configuration rankings correlate poorly (Spearman ρ = −0.36), but the validation supports profile-tier separation; its sample excludes no-gate configurations.The BTL sample cannot independently test the no-gate-versus-agentic marginal claim.Source: §C.5 · pp. 44–45

D.1 FULL ABLATION RESULTS

  • Single-feature ablations found cross-encoder reranking (+0.084), disabling MMR (+0.046), and document awareness (+0.062) improved NDCG@10, while agentic RAG raised faithfulness (+0.05) but reduced NDCG@10 (−0.015).These exploratory one-feature comparisons do not account for interaction effects.Source: §D.1 · pp. 45–46, Table 32 · p. 46

D.2 FAITHFULNESS FACTORIAL ANOVA

  • Query augmentation and agentic gating each explain about 22% of faithfulness variance, while compression explains 11%; decomposition and LLMLingua reduce faithfulness, whereas Speculative-RAG gating scores highest.The reported faithfulness means are 0.645 for decomposition, 0.811 for Speculative-RAG, and 0.702 with LLMLingua compression versus 0.775 without.Source: §D.2 · p. 46
  • Query Augmentation × Agentic Gate is the only highly significant two-way faithfulness interaction (η2 = 0.191, p < 0.001); Enrichment × Reranking misses the Holm-corrected threshold.The authors describe competing effects of enrichment on reranker scoring and document-level coherence.Source: §D.2 · pp. 46–47, Table 33 · p. 47
  • Across 113 configurations, retrieval NDCG@10 and RAGAS faithfulness are statistically uncorrelated (Pearson r = −0.01), so optimizing either metric alone does not guarantee the other.Figure 7 plots the two metrics and marks configurations in off-diagonal and dual-threshold regions.Source: §D.2 · p. 47, Figure 7 · p. 48
  • A joint LASSO selected 31 interaction terms among 34 nonzero faithfulness features and confirmed Query Augmentation × Agentic Gate as the strongest interaction pair.For retrieval, it confirmed Embedding × Enrichment as the strongest interaction.Source: §D.2 · p. 48

D.3 H1 REPRODUCIBILITY AND JOINT-INTERACTION CORRECTION

This appendix examines reproducibility and interaction effects in RAG compositions, including how findings vary across component combinations, profiles, domains, corpus sizes, and analyses. It also covers retrieval composition penalties, latency trade-offs, and exploratory faithfulness findings.

D.3 H1 REPRODUCIBILITY AND JOINT-INTERACTION CORRECTION

  • The primary per-sample joint effect was η2 = 0.040, below H1’s 0.20 threshold; aggregation gave η2 = 0.428 only as sensitivity, and D5×D8 did not survive hierarchical reanalysis.The per-sample estimate was F(92, 4707)=2.11, p=4.8×10−9; the aggregated estimate was F(92, 226)=1.84, p=1.4×10−4, but aggregation collapses within-cell variance. The naive D5×D8 result was downgraded after query-level models yielded p=0.371 and p=0.168.Source: §D.3 · pp. 49–50

D.4 POWER-LAW REFIT OF SCALING CURVES

  • Power-law and log-linear fits give near-identical scaling slopes and fit quality, preserving the finding that BALANCED overtakes STANDARD at large corpus sizes.Across both fits, |b| is approximately 0.05 for STANDARD and 0.014 for BALANCED; the table reports BALANCED degrades about 3.6× slower under both models.Source: §D.4 · p. 50, Table 35 · p. 51, Figure 8 · p. 51

D.5 P-VALUE RECONCILIATION ACROSS STATISTICAL TESTS

  • Holm–Bonferroni is applied within test families, while the paired-t and Welch crossover p-values reflect different correlation assumptions and both survive adjustment.The passage says a global correction across logically independent hypothesis families would over-inflate Type II error on the preregistered H1 test.Source: §D.5 · p. 51

D.6 SCOPE OF H3 UNCERTAINTY ACROSS CORPUS REALIZATIONS

  • The H3 confidence interval conditions on one fixed corpus realization, but a distractor audit and query-level bootstrap bound concerns that the 100K-to-200K rank flip is a sampling artifact.The query bootstrap preserved the crossover sign in every resample; a fully corpus-marginalized interval is deferred because it requires independently shuffled corpus subsets and re-ingestion.Source: §D.6 · pp. 51–52

D.7 H2 CORPUS SCOPE AND CROSS-CORPUS GENERALIZATION STUDY

  • The H2 headline uses a SQuAD v2, MuSiQue, and HotpotQA mixture; the proposed TriviaQA/Natural Questions cross-corpus grid was not executed.The headline slice combines multi-hop benchmarks with a single-hop control; the deferred grid would compare BALANCED and MAX-RELIABLE across 12 runs and 300 queries per corpus.Source: §D.7 · pp. 52–53

D.8 CROSS-CORPUS H2 PATTERN REPLICATION (POST-HOC)

  • Across 336 paired response-time comparisons, BALANCED was faster on every sample, with 10.0–12.7× speedups at 100K and 14.6–20.0× at 200K, while quality differences were nonsignificant.This is descriptive corroboration on the same benchmark family, not a direct replication of the H2 headline test; joint-HV rankings also cross from STANDARD to BALANCED above Lref≈46 s.Source: §D.8 · pp. 54–55, Table 37 · p. 54, Table 38 · p. 54

D.9 FOCUSED CONFIRMATORY EXPERIMENT (E1): PRE-ANALYSIS DESIGN

  • The preregistered E1 design crossed five query-augmentation levels with four agentic-gate levels across 200 stratified queries and three seeds, for N=12,000 observations.The planned primary test was a hierarchical mixed-effects likelihood-ratio test with a query random intercept; the interpretation policy treated either confirmation or non-rejection as informative.Source: §D.9 · pp. 55–56

D.10 E1 EXECUTED RESULTS

  • The focused confirmatory experiment executed all 20 preregistered combinations of five query-augmentation levels and four agentic-gate levels.The supplied passage begins describing faithfulness but provides no further executed-design or result details.Source: §D.10 · p. 56
  • The Query Augmentation × Agentic Gate interaction was significant but small: χ2(12) = 27.28, p = 0.00704, partial η2 = 0.033, below the pre-registered η2 ≥0.20 threshold.The result confirms a directional interaction, not the pre-registered magnitude; the executed read used 43 queries × 3 seeds rather than 200 × 3, and held other pipeline dimensions at Standard-profile defaults.Source: §D.10 · pp. 56–57

D.11 EXPLORATORY FACTORIAL PROBES

  • Exploratory probes found a Reranking × Hybrid Alpha DiD of −0.110 and HyDE × Reranking DiD of −0.066, but held-out Reranking × Hybrid Alpha reversed to +0.027.Reranking × MMR replicated at test DiD = −0.058; none of the six probes survived Holm correction, so they are descriptive rather than confirmatory.Source: §D.11 · pp. 58–59, Table 43 · p. 58

D.12 FEATURE DROPOUT ANALYSIS

  • Feature dropout showed profile-dependent effects: HyDE helped Max-Reliable (−0.031) but hurt balanced (+0.008), while context reordering helped Max-Reliable (−0.003) but hurt balanced (+0.015).Removing MMR hurt across all four profiles, and document awareness produced stable gains where active (dropout deltas −0.041 to −0.055); query expansion and Agentic RAG removal also improved NDCG@10 in the reported tests.Source: §D.12 · p. 59, Table 44 · p. 59

D.13 PER-DOMAIN BREAKDOWN

  • Domain results varied: nfcorpus had NDCG@10 0.10–0.17 below the best domain, scifact gained +0.078 from reranking, and hotpotqa gained +0.088 from HyDE.Faithfulness exceeded 0.94 across profiles, with scifact at 0.991 for SPEED; the per-domain samples were only 11–19 queries, making comparisons exploratory.Source: §D.13 · pp. 59–60

D.14 THE COMPOSITION PENALTY

  • The composed SOTA_BALANCED system scored NDCG@10 = 0.483 versus an independently predicted 0.667 from the DEFAULT_STANDARD baseline of 0.412, losing 72% of expected improvement.HyDE and reranking overlap in query-passage alignment, while disabling MMR overlaps with reranking’s diversity effect; exploratory faithfulness marginals predict 0.886 from 0.692, versus a best observed 0.911, but these are not a matched stack.Source: §D.14 · pp. 60–61, Figure 9 · p. 61, Table 47 · p. 61

E.1 PER-SIZE PAIRED TESTS FOR THE RANK CROSSOVER

  • STANDARD led at 5K and 15K, while BALANCED led from 25K onward; paired tests reported p = 0.003 at 200K and p = 0.039† in the three-seed 50K run.The 5K and 15K comparisons were single-seed point estimates without tests.Source: §E.1 · p. 61, Table 48 · p. 62

E.2.1 RETRIEVAL LATENCY VS. CORPUS SIZE

  • From 5K to 50K, STANDARD retrieval P95 latency rose approximately 87%, while BALANCED grew 65% and SPEED stayed below 300 ms at every corpus size.At 50K, BALANCED offered 0.93 faithfulness at 2.6× STANDARD’s latency, while STANDARD had 0.79 faithfulness; BALANCED’s scaling advantage is attributed to context compression reducing reranking burden.Source: §E.2 · p. 61, §E.2.1 · p. 62, Figure 10 · p. 62

E.3 SCALING PROJECTIONS

The scaling projections extend measured retrieval quality beyond 200K documents, but their uncertainty varies by profile and the largest-scale estimates remain directional. The appendix also inventories targeted experiments and reports component-specific results across rerankers, embeddings, retrieval and generation ablations, and chunking.

E.3 SCALING PROJECTIONS

  • +0.059 NDCG@10: at 200K documents, BALANCED exceeds STANDARD, but their 1M prediction intervals overlap and the projected +0.077 is directional only.BALANCED’s slope is not distinguishable from zero, widening its projection band; the interpretation about implicit noise filters is supported by measured results through 200K, not the 1M extrapolation.Source: §E.3 · p. 63, Figure 11 · p. 63, Table 49 · p. 63

F EXPERIMENT INVENTORY

  • The study combines 167 retrieval-factorial runs, 122 faithfulness-factorial runs, 138 targeted ablation configurations, and 18 scaling cells into approximately 450 configuration runs.The scaling sweep comprises 18 cells and 72 runs; multi-seed replications are not counted as distinct configurations.Source: F EXPERIMENT INVENTORY · p. 64, §F.1 · p. 64, Table 50 · p. 64

F.2 RERANKER INVENTORY

  • +0.026 NDCG@10: ms-marco-MiniLM-L6 scores 0.496 versus 0.470 for the SOTA-profile default mxbai_rerank_large.The inventory covers 20 reranker configurations; newer SOTA-2025–2026 rerankers underperform on this benchmark, while the LLM-listwise reranker is also below older cross-encoders.Source: §F.2 · p. 65, Table 51 · p. 65

F.3 EMBEDDING INVENTORY

  • 0.489 NDCG@10: 768-dimensional Matryoshka embeddings slightly exceed the full 1024-dimensional embedding at 0.478, while late chunking helps BGE-large but hurts BGE-M3.Late chunking changes BGE-large from 0.460 to 0.493 (+0.033) and BGE-M3 from 0.459 to 0.361 (−0.098); the authors attribute the regression to shorter effective contexts from BGE-M3’s multilingual tokenizer.Source: §F.3 · pp. 65–66, Table 52 · p. 66

F.4 RETRIEVAL-SIDE SINGLE-FEATURE ABLATIONS

  • Nine retrieval-side ablations cover 51 configurations across hybrid search, HyDE, MMR, query enhancement, similarity threshold, sparse methods, contextual retrieval, and reranking.The supplied passages describe the ablation scope but do not provide the individual results.Source: §F.4 · p. 66

F.5 GENERATION-SIDE ABLATIONS

  • Agentic gates shift faithfulness without shifting retrieval NDCG@10, separating generation-side effects from retrieval quality in the 17-configuration agentic sweep.The generation-side ablations also include context compression and CRAG experiments.Source: §F.5 · p. 66, Table 54 · p. 68

F.6 CHUNKING STRATEGY SWEEP

  • 0.126 NDCG@10: the best chunker, docling_hier_1600_400 at 0.475, exceeds semantic_1200 at 0.349, motivating profile-level chunker defaults.The 20-configuration sweep spans semantic, fixed, sentence-window, and Docling-derived hierarchical chunkers.Source: §F.6 · p. 66, Table 55 · p. 69

F.7 JUDGE MODEL SENSITIVITY

F.8 THIRD-JUDGE ROBUSTNESS EXTENSION

The third-judge extension supports profile-level comparisons while showing that faithfulness scores are not judge-invariant. Generator substitutions preserve broad profile conclusions but not fine-grained ordering; the appendix also reports cross-cutting ablation findings and scope caveats.

F.8 THIRD-JUDGE ROBUSTNESS EXTENSION

  • 78.2% binary agreement with the canonical judge (Cohen’s κ=0.353) accompanies moderate correlation, indicating the faithfulness signal persists but is not judge-invariant.The third annotator is stricter on partial-support cases, assigning fewer maximum scores than RAGAS/GPT-5.1.Source: §F.8 · pp. 69–70

F.9 GENERATOR SUBSTITUTION: DOES THE FAITHFULNESS ORDERING HOLD?

  • Every profile scores higher under Claude Sonnet 4.5 than GPT-5.1, yet the two heavier agentic profiles gain no faithfulness advantage over the lighter direct-generation profile under either generator.STANDARD and BALANCED exchange ranks; with only two proprietary generators, this is a sensitivity check rather than a sweep.Source: §F.9 · p. 70, Table 57 · p. 70
  • +0.184: the stronger generator’s faithfulness benefit is largest for the lightest pipeline, versus +0.027 and not significant for the heaviest.The authors interpret this uneven benefit as evidence that pipeline machinery and generator capability partly substitute.Source: §F.9 · p. 70

F.10 CROSS-CUTTING FINDINGS FROM THE ABLATION SUITE

  • Agentic iterations move faithfulness rather than retrieval; chunking spans 0.126 absolute NDCG@10, while the one-document HyDE variant leads the clean sweep.The ablation summary also reports SPLADE below BM25 and contextual retrieval slightly below its off setting on the evaluated domain.Source: §F.10 · p. 71

F.11 CAVEATS AND SCOPE

  • The retrieval factorial executed 167 of 242 pre-registered configurations after infrastructure crashes and rate limits, retaining 99.3% D-efficiency; all faithfulness evaluations use GPT-5.1 as judge.Three environment-affected embedding runs are retained for transparency but do not inform main-text quantitative claims.Source: §F.11 · p. 71

F.12 PER-EXPERIMENT PROVENANCE MANIFEST

  • The provenance manifest records generator, judge, prompt, execution, decoding, and sample details to prevent silent mixing of those conditions across tables.Length-controlled prompt claims use residualized FAITHA because raw scores are confounded with answer length.Source: §F.12 · p. 71, Table 58 · p. 72

G.1 PRIOR-EVALUATION TABLE DEFINITIONS

  • Table 1’s methodology comparison distinguishes configuration count and mode, design strategy, reported effect sizes and interactions, scaling, Pareto objectives, preregistration, and benchmark coverage.LASAGNA uses eight of the 15 benchmarks supported by its framework.Source: §G.1 · p. 72

G.2 SOTA METHOD COMPARISON

The method comparison finds that retrieval systems trade quality against deployment cost, while external comparisons require careful attention to evaluation parity. The appendix’s framework smoke test is positional rather than a matched benchmark, and its platform tests and interfaces document practical implementation capabilities.

G.2 SOTA METHOD COMPARISON

  • ColBERT achieves the highest NDCG@10 at 0.4616, only +0.005 above the ms-marco-MiniLM-L6 cross-encoder reranker, but requires per-token index storage that scales linearly with corpus size.The lighter reranker offers comparable quality at lower indexing cost and is used in the composed profiles.Source: §G.2 · pp. 72–73, Table 59 · p. 72
  • SPLADE underperforms BM25 by −0.012 NDCG@10 on this domain-focused corpus; the authors attribute this to already-small vocabulary gaps and caution the result may not hold on open-domain collections.The larger mxbai-rerank-large model also scores −0.009 below ms-marco-MiniLM-L6 on the evaluated subset.Source: §G.2 · p. 73

G.3.1 EXECUTABLE BASELINES

  • +0.074 NDCG@10: on the same retrieval-isolation split, SOTA_STANDARD scores 0.510 versus 0.436 for BGE-M3 RECOMMENDED.The apparent table gap of 0.860 versus 0.436 is inflated because baseline and SOTA rows use different query splits and gold labels.Source: §G.3.1 · p. 73, Table 60 · p. 73, §G.3 · p. 73

G.3.2 PUBLISHED BEIR BASELINES (LITERATURE)

  • Published BEIR scores use full test sets and corpora, unlike this study’s stratified subsets from a combined 5,132-document corpus, so absolute values are not directly comparable.BGE-M3 has the highest published scores across the five evaluated domains; executable baselines estimate added-component gains on this study’s corpus.Source: §G.3.2 · pp. 73–74, Table 61 · p. 74

G.4 FRAMEWORK COMPARISON TO BERGEN AND FLASHRAG

  • The FlashRAG comparison is positional smoke evidence, not a head-to-head: its AI-topic corpus mismatches the Natural Questions queries, and the prompts differ in refusal behavior.The authors identify refusal rate as the only reliable signal and state that matched NDCG@10, BGE-M3, and cross-framework RAGAS results are not reported.Source: §G.4 · pp. 74–75, Table 62 · p. 74
  • A like-for-like factorial comparison in BERGEN or FlashRAG would require adding session-config-isolated boundaries, so the paper relies on LASAGNA-internal factorials and executable baselines for quantitative claims.The comparison is explicitly treated as positional rather than confirmatory.Source: §G.4 · p. 75

H.1 PLATFORM ISOLATION VERIFICATION

  • All five platform-isolation tests pass; separate ingestion is enforced when MiniLM-L6’s 384-dimensional embeddings cannot share a collection with BGE-M3’s 1024-dimensional embeddings.The embedding-dimension test was verified during scaling experiments.Source: Table 63 · p. 75, §H.1 · p. 75

H.2.1 CONFIGURATION UI

  • The Configurator lets users assemble profile-based pipeline settings without code or YAML and round-trips changes through the evaluation SDK’s YAML contract.Its Essential tab surfaces high-impact toggles, model selection, GPU control, and tuning sliders.Source: §H.2.1 · pp. 75–76, Figure 12 · p. 76

H.2.2 LAYERED PLUGIN ARCHITECTURE

This section describes LASAGNA’s extensible, configurable architecture and agent integrations, then compares MAX-RELIABLE and BALANCED as deployment profiles. It frames profile selection around retrieval quality, faithfulness, latency, and corpus size.

H.2.2 LAYERED PLUGIN ARCHITECTURE

  • LASAGNA supports eight typed component classes, with each component registered by decorator and integrated through its minimal ABC interface.The public workflow registers a component, declares a factorial sweep in YAML, and runs an end-to-end experiment with a paired statistical test.Source: §H.2.2 · pp. 76–77

H.2.3 BRING YOUR OWN DATABASE

  • LASAGNA includes six vector backends and supports custom databases through VectorBackendBase; declared capabilities enable software BM25 fallback when native sparse search is unavailable.After registration, setting vector_backend to the custom backend routes vector operations through that implementation.Source: §H.2.3 · pp. 77–78

H.2.4 BRING YOUR OWN AGENT

  • Researchers can replace or extend LASAGNA’s default multi-iteration retrieve-refine-generate agentic pipeline, or override an individual graph node while retaining its orchestration.The default graph includes hallucination checking and quality gates.Source: §H.2.4 · pp. 78–79

H.2.5 RESEARCH WORKFLOW

  • NDCG@10 was 0.487 for hyde_on versus 0.461 for hyde_off (p=0.031) in the example experiment.The runner automates ingestion, timing, RAGAS evaluation, checkpointing, cost tracking, and statistical testing, and saves per-sample results as JSON.Source: §H.2.5 · p. 79

H.2.6 AGENT INTEGRATION VIA MCP

  • LASAGNA exposes its knowledge base through an MCP server with 31 tools, letting compatible agents connect without custom integration code.The same indexed corpus can serve different agent frameworks and models without re-ingestion; the SDK also offers a typed Python client.Source: §H.2.6 · pp. 79–80

I.1.1 SOTA_SPEED

  • SOTA_SPEED targets latency-sensitive deployments, completing retrieval in ∼200 ms and end-to-end generation in ∼800 ms at p95.It omits reranking, HyDE, and agentic orchestration; the appendix advises against it when NDCG@10 > 0.7 or faithfulness > 0.6 is required.Source: §I.1.1 · p. 81

I.1.2 SOTA_STANDARD

  • SOTA_STANDARD achieves the highest NDCG@10 (0.860) on the SOTA-profile generation slice, using hybrid retrieval, HyDE, reranking, and pipeline-level quality gates.It follows the direct generation route and enables hallucination and answer-quality checks; per-query faithfulness above 0.8 requires application-level guardrails.Source: §I.1.2 · p. 82

I.1.3 SOTA_BALANCED_E2E; I.1.4 SOTA_MAX_RELIABLE

  • SOTA_BALANCED_E2E provides a stronger joint agentic tradeoff than MAX_RELIABLE, with higher NDCG@10, essentially tied faithfulness, and 13% lower mean latency.It uses extractive compression and document-awareness prefixes; MAX_RELIABLE differs mainly in retrieval choices, including hybrid_alpha 0.3 versus 0.7, contextual retrieval on, and max_chunks 30 versus 20. At 60 s, BALANCED has HV=0.609 versus 0.579; STANDARD leads at the fixed 30 s reference.Source: §I.1.3 · pp. 82–83, §I.1.4 · p. 83

I.1.4 SOTA_MAX_RELIABLE

  • MAX-RELIABLE’s faithfulness is essentially tied with BALANCED (0.916 vs 0.911), but NDCG@10 is 0.036 lower and mean latency is ∼2 s higher.The authors include MAX-RELIABLE for deployments requiring contextual retrieval and a wider chunk budget, despite its weaker joint tradeoff within the agentic-route class.Source: §I.1.4 · pp. 83–84

I.2 PROFILE SELECTION GUIDE

  • STANDARD is the recommended interactive-latency default, while BALANCED is recommended when longer response times and greater faithfulness are acceptable.STANDARD has HV=0.523 at the fixed 30 s reference; BALANCED takes the joint HV lead once Lref exceeds ∼46 s. The flowchart bases its choices on response-time target, faithfulness requirement, and corpus size.Source: §I.2 · p. 84, Figure 15 · p. 84
  • SPEED is the only viable profile under a hard sub-second response-time target, while MAX-RELIABLE suits high-stakes deployments that prioritize faithfulness over response time.SPEED uses MiniLM-L6 embeddings and dense-only retrieval without query-time LLM calls. MAX-RELIABLE uses up to three retrieval iterations with hallucination gating, and its broader retrieval envelope is justified when specifically required.Source: §I.1.4 · p. 83, §I.2 · p. 84
  • The guide may not generalize beyond its five BEIR domains: the MMR finding, approximate 25K corpus-size threshold, and measured latency depend on corpus and infrastructure.The authors note that MMR may be useful for genuinely redundant corpora and that the threshold is domain-dependent.Source: §I.2 · p. 85
  • When OFAT and hierarchical mixed-effects readings differ by at least one rank, the authors defer to hierarchical inference, except when query variance is independent of topic clustering.They cite ICC=0.284 on the H1 D5×D8 grid; the OFAT interaction estimate is η2=0.201 (p=0.018 Holm), whereas the hierarchical LRT is χ2(4)=4.27, p=0.371.Source: §I.2 · p. 85

I.3 HALLUCINATION RESISTANCE

This section examines hallucination resistance through faithfulness across deployment profiles, relating observed successes and failures to pipeline choices, interactions, and ablation evidence while distinguishing retrieval strength, latency, and scale robustness. It also considers when Max-Reliable’s faithfulness advantage may warrant its latency cost relative to balanced’s quality–response-time trade-off.

I.3 HALLUCINATION RESISTANCE

  • Across six historical profiles, mean faithfulness exceeded 0.96; SPEED and SOTA MAX-RELIABLE had median faithfulness of 1.000, while MAX QUALITY’s lowest observed per-query score was 0.667.These retrospective deployment-log patterns are descriptive and correlational; the study makes no causal claims about components, and isolating their contributions would require a dedicated factorial design.Source: §I.3 · p. 85

J ROOT CAUSE ANALYSIS OF PROFILE PERFORMANCE

  • The root-cause appendix explains each SOTA profile’s performance through architectural decisions, measured interactions, and ablation evidence from the 167-configuration factorial campaign.It uses a taxonomy and profile-level analysis to connect dominant success and failure modes to each profile’s stated purpose.Source: J ROOT CAUSE ANALYSIS OF PROFILE… · p. 86

J.1 FAILURE–SUCCESS TAXONOMY

  • The failure–success taxonomy separates pipeline location from the mechanism through which a cause affects performance.It distinguishes where a cause originates in the pipeline from how it changes outcomes.Source: §J.1 · p. 86

J.2 PROFILE-LEVEL ROOT CAUSE ANALYSIS

  • The profile-level analysis traces measured SOTA performance to taxonomy-based success and failure modes, quantifying contributions with ablation and dropout evidence.It also assesses whether each profile’s design choices fit its stated purpose.Source: §J.2 · p. 86

J.2.1 SOTA_SPEED: MINIMALIST RETRIEVAL, MAXIMAL FAITHFULNESS

  • SPEED had the lowest retrieval NDCG@10 among SOTA profiles (0.415 at 5K, 0.382 at 15K) but the highest mean faithfulness (0.980, median 1.000) in deployment evaluations.The analysis links its faithfulness to simple, unmodified context and concise prompting, while noting that concise answers reduce the claim surface and may make the advantage partly artifactual; its dense-only retrieval also degrades rapidly with scale.Source: §J.2.1 · pp. 88–89

J.2.2 SOTA_STANDARD: RETRIEVAL CHAMPION, SCALE-FRAGILE

  • STANDARD led SOTA-profile retrieval at 5K with NDCG@10 of 0.510 but fell to 0.432 at 200K, where BALANCED overtook it with a significant gap (∆= +0.059, p = 0.004).Its retrieval stack combines reranking, hybrid search, disabled MMR, and HyDE, but lacks generation-side defenses; the analysis links that retrieval-only design to scale fragility and lower faithfulness.Source: §J.2.2 · pp. 89–90

J.2.3 SOTA_BALANCED_E2E: STRONGEST JOINT TRADEOFF IN THE AGENTIC-ROUTE CLASS THROUGH COMPLEMENTARY DEFENCES

  • BALANCED’s NDCG@10 stayed near-flat from 0.504 at 5K to 0.491 at 200K and it overtook STANDARD at 200K; faithfulness was 0.911 versus 0.916 for MAX-RELIABLE, with 13.5% lower mean latency.The analysis attributes this tradeoff to complementary generation-side defenses: compression filters noisy context, while document awareness grounds it; their different pipeline roles limit composition interference and help preserve scale robustness.Source: §J.2.3 · pp. 90–92

J.2.4 SOTA_MAX_RELIABLE: MAXIMUM FEATURES, DIMINISHING RETURNS

  • MAX-RELIABLE had faithfulness of 0.916 versus BALANCED’s 0.911, but 13.5% higher mean latency; its E2E NDCG@10 was 0.804 versus STANDARD’s 0.860 at 5.5K.Hallucination gating raises its minimum observed faithfulness to 0.812, but aggressive refinement discarded 14 of 15 chunks in a multi-hop example, leaving an incomplete answer despite faithfulness of 1.0.Source: §J.2.4 · pp. 92–93
  • Max-Reliable adds 3–7 LLM calls per query and +2.0 s p95 latency over balanced for only +0.005 faithfulness gain.The calls cover planning, per-chunk relevance classification, hallucination checking, a quality gate, and optional re-retrieval.Source: §J.2.4 · p. 93
  • Removing agentic RAG from the Max-Reliable base improves NDCG@10 by +0.021, and the profile’s composition penalty exceeds 72%.Agentic refinement partly duplicates context compression by reducing context to high-relevance spans, diminishing marginal benefits.Source: §J.2.4 · p. 93
  • Max-Reliable inherits the suboptimal mxbai-large reranker, disables query expansion and RAG fusion, and query expansion hurts NDCG by −0.014 in single-feature ablation.Consequently, its agentic pipeline receives the same retrieval input as balanced, limiting its improvement potential to generation.Source: §J.2.4 · p. 93
  • Max-Reliable raises the faithfulness floor by +0.062 over balanced (0.812 vs. 0.750), while balanced offers a stronger joint trade-off outside floor-bound deployments.The joint disadvantage is driven by aggressive refinement that raises the floor while reducing answer completeness.Source: §J.2.4 · pp. 93–94

J.3 CROSS-PROFILE INTERACTION ANALYSIS

This section examines how RAG components interact, including how profile performance changes with corpus scale, and derives design principles for feature composition. It also describes retrieval-verification and refinement mechanisms and catalogs terminology, evaluation measures, and composition shortfall for LASAGNA.

J.3 CROSS-PROFILE INTERACTION ANALYSIS

  • Enrichment does not improve over the none baseline even with high-capacity embedders, and it harms weaker embedders more.Prepended synthetic context can overwhelm low-capacity models; the SOTA profiles therefore use no enrichment. Whether enrichment helps on open-domain corpora remains a question for future evaluation.Source: §J.3 · p. 94
  • Reranking has a compensatory interaction: its marginal gain is larger with weaker first-stage retrieval than with strong embeddings and hybrid retrieval.Strong first-stage retrieval leaves less room for reranking; weaker retrieval provides a noisier candidate set from which reranking can rescue relevant passages.Source: §J.3 · p. 94
  • Augmentation plus agentic gating is the only observed superadditive interaction, with the strongest faithfulness interaction near the preregistered H1 threshold.Augmentation alone does not improve faithfulness (Δ = 0.00 for RAG fusion), while gating alone improves it modestly (Δ = +0.05 for agentic RAG); together, augmentation supplies recall and gating filters noise. Max_reliable is the only SOTA profile using both, though aggressive refinement can discard passages that improve completeness.Source: §J.3 · pp. 94–95

J.4 SCALING ROOT CAUSES

  • At 200K documents, BALANCED overtakes STANDARD by Δ = +0.059 (p = 0.004), after STANDARD leads by +0.006 NDCG@10 at 5K.Retrieval-side defenses degrade as candidate sets grow, while compression and document awareness filter noise after retrieval; the crossover reverses between 15K and 25K and is not statistically significant through 100K.Source: §J.4 · p. 96

J.5 SUMMARY OF ROOT CAUSES BY MECHANISM

  • The analysis recommends complementing rather than duplicating features, validating composed systems end-to-end, and choosing features for the target corpus scale.Single-feature ablations overestimate composed performance, with composition penalties of 7–72%; retrieval-side features offer larger gains below 10K documents, while generation-side defenses retain effectiveness at scale.Source: §J.5 · p. 97

K RAG METHODS CATALOG

  • The methods catalog covers techniques across eight varied component dimensions, with document processing and the base answer generator held fixed and D8 varying agentic control.Implementations follow the original algorithms unless noted.Source: K RAG METHODS CATALOG · p. 97

None

  • Single-pass generation produces an answer without quality verification.Source: None · p. 100

CRAG (Corrective RAG)

Self-RAG

  • Self-RAG uses reflection tokens to gate retrieval, assess answer quality, and iteratively refine answers through retrieval and verification.The passage attributes this approach to Asai et al. (2024).Source: Self-RAG · p. 100

Speculative RAG

  • Speculative RAG generates drafts from different retrieved subsets and uses a verifier to select one, trading latency for answer diversity.The passage attributes this approach to Wang et al. (2025).Source: Speculative RAG · p. 100

Quality gates (hallucination filter)

  • Quality gates retain chunks an LLM judge deems supportive, raising the faithfulness floor but potentially starving generators on multi-hop queries.This shared mechanism appears in CRAG, Self-RAG, and Speculative RAG, and is used in MAX-RELIABLE.Source: Quality gates (hallucination filter) · p. 100

Multi-hop refinement loop

  • The multi-hop loop retrieves, drafts, identifies missing information, retrieves again for up to K iterations, then generates from aggregated context.The experiments use K = 3; the implementation shares an agentic graph with the quality-gate filter.Source: Multi-hop refinement loop · p. 100

L TERMS, COMMON QUESTIONS, AND RELEASED RESOURCES

L.1 TERMS

  • 63% was the composition shortfall for the three strongest retrieval techniques, measuring how far combined gains fell below the sum of isolated gains.The glossary defines composition shortfall as the gap between a combined pipeline’s gain and the gain predicted by adding techniques measured individually.Source: §L.1 · p. 101

L.2 COMMON QUESTIONS

The common questions show that isolated RAG gains do not reliably predict full-pipeline performance: retrieval and faithfulness favor different choices, rankings shift with corpus scale, and no complete profile wins on every metric. The paper situates its contribution as measuring how techniques compose, rather than proposing a new technique.

L.2 COMMON QUESTIONS

  • +47% predicted NDCG@10 improvement from the three strongest retrieval techniques becomes +17% when combined, a 63% shortfall; faithfulness gains are +28% predicted and +32% realized.Across 17 tested combinations, retrieval shortfalls range from 7% to 72%, with a median of 41%.Source: §L.2 · p. 101
  • Embedding, enrichment, and reranking dominate NDCG@10, while query augmentation and agentic gating dominate faithfulness, so optimizing one metric can hurt the other.The paper recommends evaluating new components on both retrieval quality and answer faithfulness.Source: §L.2 · p. 101
  • From 5K to 200K documents, STANDARD leads at 5K and 15K, but BALANCED leads from 25K through 200K, exceeding STANDARD by 13.7% relative NDCG@10.The large-scale comparison is reported with p = 0.003 across three seeds.Source: §L.2 · p. 101
  • No complete profile wins on every metric: SPEED is fastest, STANDARD leads NDCG@10 and joint hypervolume, while BALANCED and MAX-RELIABLE lead faithfulness at higher response time.The reported joint hypervolume for STANDARD is 0.523 at a 30 s reference.Source: §L.2 · p. 101
  • New RAG techniques should be evaluated inside a representative complete pipeline, on both retrieval and faithfulness, and at the intended corpus scale.LASAGNA measures components against the same surrounding stack and uses a paired statistical test.Source: §L.2 · p. 102
  • The evaluation uses one constructed English multi-domain test bed, with at most 93 retrieval queries and 200 generation queries per experiment, and does not test multilingual, private, or cross-organizational corpora.Faithfulness uses two generator models as a sensitivity check, and absolute scores rely on one judge’s calibration.Source: §L.2 · p. 102

L.3 WHERE THIS WORK SITS

  • The paper’s contribution is measuring how RAG techniques compose, situated among RAG methods, evaluation frameworks, retrieval benchmarks, pipeline toolkits, and experimental-design work.It presents this as a measurement contribution, not a new technique.Source: §L.3 · p. 102
Loading 2610.29722v1…