Post-Generation Verification Dominates Retrieval Optimization: A 2^4 Factorial Ablation of RAG Pipeline Features

1 Introduction

• Standard one-feature-at-a-time baseline tests miss feature interactions, diminishing returns, and structural dependencies, motivating factorial evaluation.The number of conditions grows exponentially with the number of features.Source: §1 · p. 1
• A 2^4 ablation tests four independent features in 16 configurations, 24 queries across eight interaction types, and two models, totaling 768 conditions.The corpus contains five public documents spanning 78–492 pages.Source: §1 · p. 1
• CC dominates quality outcomes, while ToC adds a zero-cost gain, AS varies by query type, and SE is neutral on this corpus.The results also show a six-configuration quality–latency Pareto frontier and query-type moderation of feature utility.Source: §1 · p. 1
• The study contributes a feature-interaction query taxonomy and evidence that aggregate effects can mask query-type-specific feature utility.It reports CC alone at 4.31/5 outperforming every configuration without CC.Source: §1 · pp. 1–2

RAG architectures and feature engineering

• RAG enhancements span pre-retrieval, retrieval, and post-retrieval stages, but factorial studies measuring feature interactions remain rare.Gao et al. (2024) survey the modular RAG landscape; the paper adopts the same stage taxonomy for its four features.Source: §2 · p. 2

Query decomposition

• Agentic search implements two-tier heuristic-plus-LLM query decomposition in the tradition of sub-query and query-rewriting methods.Related examples include Trivedi et al. (2023) and Ma et al. (2023).Source: §2 · p. 2

Structure-aware retrieval

• The paper’s ToC retrieval extracts headings without LLM calls, loads matched sections as full pages, and keeps heading and chunk indexes independent, unlike DocsRAG’s blended scores.DocsRAG (Jeong et al., 2025) generates a pseudo table of contents through multiphase LLM processing.Source: §2 · p. 2

Completeness verification

• Completeness checking decomposes questions into facets, checks each against the answer, and re-retrieves for missing facets, while holistic prompts risk false negatives.Related loops include self-reflection (Asai et al., 2024) and iterative retrieval–generation (Shao et al., 2023).Source: §2 · p. 2

LLM-as-judge and factorial designs

• The paper uses model-answer anchored scoring and notes that factorial designs are common in psychology but rare in NLP system evaluation.Shankar et al. (2024) raise reference-free evaluation bias; the authors report no known prior RAG-feature factorial ablation at this scale.Source: §2 · p. 2

3.1 Pipeline and Features

• The 14-stage pipeline combines hybrid cosine-similarity and BM25 retrieval with optional feature stages, answer generation, and optional verification.The four tested features operate independently, with no architectural dependencies or guard conditions.Source: §3.1 · p. 2, Table 1 · p. 3
• The completeness checker decomposes questions into facets, labels each as covered, missing, or partial, and checks numbered-sequence gaps.It marks an answer incomplete if any facet is missing or multiple facets are partial; holistic prompts produce high false-negative rates.Source: §3.1 · pp. 2–3
• ToC retrieves full page content by matching query embeddings against pre-embedded document headings, addressing gaps that chunk retrieval and section expansion cannot reach.Section expansion cannot reach cousin sections across hierarchy branches or distinguish subtopics in sections without subheadings.Source: §3.1 · p. 3

3.2 Corpus and Experimental Design

• The corpus covers five public documents spanning data protection, human resources, AI regulation, procurement, and cybersecurity.Document lengths range from 78 to 492 pages, with shallow-to-deep heading hierarchies and dense cross-referencing.Source: §3.2 · p. 3
• The 2^4 full factorial design tests 16 configurations across 24 queries and two comparable cloud-class models, yielding 768 conditions.The eight query interaction types target specific mechanisms or feature combinations, enabling feature effects to be examined where expected.Source: §3.2 · p. 3

3.3 Evaluation

• All responses are scored against manually verified gold-standard answers for accuracy, completeness, and usefulness; composite quality is their unweighted mean.Page-level precision and recall compare retrieved pages with labeled relevant pages, but are measurable only for ToC evidence because other chunk IDs lack page mappings.Source: §3.3 · p. 3

4.1 Overall Quality

• 4.31/5 for CC alone exceeds every configuration without CC, including the three-feature SE+AS+ToC configuration at 4.11.The all-feature configuration reaches 4.35/5, the highest overall quality.Source: §4.1 · p. 4, Table 3 · p. 5

4.2 Main Effects

• CC has the largest main effect (d = +0.48, p < 0.001), while ToC improves quality at zero LLM cost (d = +0.22, p = 0.003), AS is small, and SE is null.CC is significant on accuracy (+0.25), completeness (+0.47), and usefulness (+0.44); ToC is significant on all three dimensions.Source: §4.2 · p. 4, Table 4 · p. 5

4.4 Query-Type Moderation

• CC’s effect is strongest on completeness-demanding queries (cc_target completeness d = +0.83, p < 0.001), while AS helps some types and hurts others, including as_target (∆= −0.22).A single query type can therefore over- or undervalue AS; baseline queries average 4.71 while cc_target queries average 3.30.Source: §4.4 · pp. 4–5, Table 5 · p. 5

4.3 Feature Interactions

• CC dominates each interaction panel, and SE adds essentially nothing once CC is enabled; AS and ToC instead contribute additively.No feature pair shows a large synergistic boost; one-at-a-time testing would miss the SE–CC redundancy.Source: §4.3 · p. 4, Figure 2 · p. 5

4.5 Quality–Latency Pareto Frontier

• Six configurations are Pareto-optimal; the all-feature configuration has the highest quality (4.35) but takes 175s, more than twice baseline latency (78s).CC alone is a close second at 4.31 and 169s; no configuration reaches CC-level quality at non-CC latency.Source: §4.5 · pp. 5–6, Figure 4 · p. 6

4.6 Retrieval-Level Impact and Robustness

• ToC pages hit relevant content on 21% of runs, with 0% hit rates on baseline, cc_target, and se_target queries; SE’s null effect is consistent with deep document hierarchies and ToC overlap.ToC adds about 3.4 pages per query, while CC triggers fewer regenerations when paired with ToC (about 1.3–1.5) than alone (about 1.6).Source: §4.6 · p. 6, Figure 5 · p. 6

Verification versus retrieval

• CC checks generated answers against question facets and re-retrieves for gaps; on this corpus and query set, it improves accuracy as well as completeness.Its verification role differs from retrieval-stage features, which aim to improve the evidence in context.Source: §5 · p. 6

Operating constraints

• CC alone reaches 4.31, above every non-CC configuration; ToC is the best zero-LLM-cost lever, while AS+ToC scores 4.09 at 95s.For quality-critical settings the paper recommends CC; for latency-sensitive settings it recommends ToC or AS+ToC.Source: §5 · p. 7, Table 6 · p. 7

Feature selection

• CC alone (4.31) outperforms the three-feature SE+AS+ToC configuration (4.11), so feature identity matters more than feature count.Mean quality rises with feature count, but variance within each count is large; the highest-quality configurations share CC.Source: §5 · p. 7

Methodological implications

• Factorial testing reveals that adding SE to a CC-containing configuration yields negligible benefit, an interaction that one-at-a-time testing misses.CC’s aggregate d = +0.48 ranges from +0.16 on simple queries to +0.83 on completeness-demanding queries; AS changes sign across query types.Source: §5 · p. 7

Future work

• The proposed follow-up directions are dynamic CC triggering for likely-incomplete answers and ablations across structurally diverse corpora.A cost-aware feature selector could also choose features per query under a latency budget.Source: §5 · p. 7

Conclusion

• CC (d = +0.48) is the most impactful feature, improving accuracy, completeness, and usefulness; CC alone (4.31/5) beats every configuration without it.The experiment covers 768 conditions on five public documents.Source: §6 · p. 7
• ToC has a significant main effect (d = +0.22, p = 0.003) at zero LLM cost and is the only feature with observable page-level retrieval impact.CC is strongest across query types, while AS helps some types and harms others.Source: §6 · p. 7
• CC alone beats three-feature SE+AS+ToC, while six Pareto-optimal configurations span 0–4 features with CC common to all; the note argues answer verification is higher-leverage than retrieval optimization.Source: §6 · p. 7
• The authors conclude that answer verification is a higher-leverage investment than retrieval optimization, and that diverse query types and factorial designs are needed to evaluate RAG features.Source: §6 · p. 7

Limitations

• The study uses two similarly capable cloud-class models and public regulatory or policy documents; effects may differ with smaller or local models and other document types.The named out-of-scope document types include technical manuals, code documentation, and financial reports.Source: Limitations · p. 7
• Page-level precision and recall are measurable only for ToC-sourced evidence because other features lack page tracking at the chunk or generation level.Source: Limitations · p. 7
• Model-answer anchored scoring cannot measure valid novel perspectives or user-specific relevance when those dimensions are absent from the gold standard.Source: Limitations · p. 8

Retrieval and indexing

• Retrieval uses nomic-embed-text embeddings, 512-token chunks with 64-token overlap, hybrid cosine and BM25 search, and reranks 20 candidates to five.Source diversity is enabled, with an 8,192-token context window.Source: A Experiment Settings · p. 8

Feature settings

• SE expands 2–15 sibling chunks; AS uses up to four subqueries; CC allows two hops and up to 15 evidence items; ToC uses a page budget of five.CC runs independently of ToC; the settings also specify similarity and retrieval multipliers for AS and ToC.Source: A Experiment Settings · p. 8

Models and scoring

• DeepSeek-V4-Flash and grok-4.3 are answer models; claudeopus-4-6-2 scores accuracy, completeness, and usefulness from 1–5 at temperature 0 (maximum 500 tokens), with their unweighted mean forming the composite.Source: A Experiment Settings · p. 8

Statistics

• Main effects use Welch’s t-test and Cohen’s d, with 5,000-resample bootstrap confidence intervals and a p < 0.05 significance threshold.The reported confidence intervals are 95% percentile intervals.Source: A Experiment Settings · p. 8

Reported tables

• Main effects are reported by quality dimension for n = 759; the Pareto table notes that SE alone (3.62, 80s) is dominated by SE+ToC (3.90, 80s).Source: Table 8 · p. 9, Table 7 · p. 9

• Tables 7 and 8 provide the per-dimension main effects and the Pareto-optimal configurations, respectively.Source: B Additional Results · p. 9
• Figures 6–10 break results down by dimension, configuration, latency, feature count, and accuracy–completeness.Source: B Additional Results · p. 9
• Figure 11 presents page-level retrieval breakdowns, including metrics by ToC status and query type.Source: B Additional Results · p. 9

• Claude Opus 4.6 scored each answer against a verified reference generated from the source documents and manually reviewed for factual accuracy.Source: C Scoring Rubric and Reproducibility · p. 9
• Figure 6 reports accuracy, completeness, and usefulness for all 16 configurations by model, with SEM error bars and baseline means.Source: Figure 6 · p. 9
• Figure 7 shows CC alone exceeding every non-CC configuration, while Figure 9 shows feature-count means rising amid large within-group variance.Source: Figure 7 · p. 9, Figure 9 · p. 10
• Accuracy measures correctness, completeness measures reference content covered, and usefulness measures actionability on 1–5 scales.Source: C Scoring Rubric and Reproducibility · p. 9
• The evaluation covers 24 queries across five public documents; raw responses, telemetry, and query definitions with references are archived on Zenodo.Source: C Scoring Rubric and Reproducibility · p. 9
• Figure 8 reports latency by configuration and model, identifying CC as the primary latency driver.Source: Figure 8 · p. 10

• Table 9 lists all 24 queries with their interaction type, source document, and ground-truth relevant pages.Source: D Query Set · p. 10, Table 9 · p. 11
• Figure 10 plots mean accuracy against completeness by configuration, with color indicating feature count.Source: Figure 10 · p. 10
• Figure 11 breaks page-level retrieval metrics down by ToC status and query type, including ToC page-hit rate.Source: Figure 11 · p. 10