PaperlayerView exact revision on arXiv

Source-linked AI summary

FCPRAG: Fusion-Controller Parametric Retrieval-Augmented Generation for Stable Multi-Passage LoRA Injection

Jinchang Zhu, Jindong Li, Yi Ding, Xiaojian Nie, Rong Fu, Shuangyong Song, Haowei He, Menglin Yang

arXiv:2608.21750v1cs.CL

TL;DR

Multi-passage parametric RAG needs selective fusion because longer or conflicting evidence is costly and fragile, while global calibration can be expensive. FCPRAG adds a lightweight controller for retrieval-conditioned, sample-level adapter fusion and calibration, improving F1 over baselines across four datasets and three LLM backbones in most settings.

  • Problem

    Multi-passage RAG faces costly, fragile long contexts and expensive global calibration for passage-adapter fusion.

  • Method

    FCPRAG uses a lightweight controller to predict retrieval-conditioned passage fusion weights and sample-level calibration signals for adapter merging.

  • Results

    FCPRAG improves F1 over baseline methods across four datasets and three LLM backbones in most settings.

  • Takeaways & Limitations

    Sample-level calibration is a practical way to adapt evidence fusion to query-dependent retrieval uncertainty rather than relying on one global setting.

  • Takeaways & Limitations

    The method focuses on a small, fixed number of passages, and scaling to larger evidence sets may require sparse or hierarchical fusion policies.

Abstract

from arXiv · show

Parametric retrieval-augmented generation (PRAG) injects retrieved evidence into a large language model (LLM) through passage-specific LoRA adapters, reducing reliance on long in-context prompts. When multiple passages are retrieved for the same query, however, evidence-level fusion becomes a bottleneck: equal-weight merging can amplify weak or conflicting evidence, and translating retrieval signals into fusion weights often requires fragile global tuning. We propose FCPRAG, a fusion-controlled parametric RAG framework that adds a lightweight controller for retrieval-conditioned, sample-level adapter fusion. The controller predicts per-passage fusion scores together with sample-level calibration signals, including a mixing gate and an adaptive temperature, enabling fusion that stays selective under informative retrieval signals and conservative under uncertainty. FCPRAG is trained with merge-aware supervision derived from each adapter's marginal contribution within a multi-adapter merge, using training data only. We further show that a single dataset-level temperature is suboptimal under heteroscedastic retrieval uncertainty, motivating sample-level adaptation. Experiments on HotpotQA, 2WikiMultiHopQA, PopQA, and ComplexWebQuestions (CWQ) across three LLM backbones show that FCPRAG consistently improves F1 over standard RAG and parametric RAG baselines, with gains of up to 4.65% on 2WikiMultiHopQA and 7.55% on CWQ, while also reducing tuning cost and improving robustness under retrieval perturbations.

1 Introduction

FCPRAG addresses the multi-evidence fusion bottleneck in parametric RAG by learning retrieval-conditioned, sample-level LoRA fusion and calibration. It replaces fragile global merge heuristics with a lightweight, plug-in Fusion Controller trained using merge-aware supervision.

  • Motivation: Classical in-context RAG becomes costly and vulnerable to degradation as retrieved contexts grow longer or contain conflicts.Multi-evidence reasoning benefits from more context, but longer contexts increase inference cost and can propagate errors.
  • Motivation: PRAG reduces online context length by encoding each retrieved passage as a passage-specific LoRA adapter, but introduces a multi-evidence fusion bottleneck.Fusion weights determine which evidence is injected into the model’s effective parameters, making calibration mismatches consequential.
  • Motivation: Evidence utility is sample-dependent, so fixed global fusion heuristics cannot reliably distinguish supportive, redundant, weak, or conflicting passages.The marginal contribution of each adapter can be estimated by removing it from a multi-adapter merge and measuring the F1 drop.
  • FCPRAG: FCPRAG adds a Fusion Controller that predicts per-passage fusion weights and per-sample calibration signals, including selective behavior under confidence and conservative behavior under uncertainty.The controller is a plug-in that replaces the merge step without changing PRAG offline parameterization, retrieval, or generation, and adds < 1% trainable parameters relative to the backbone LLM.
  • Contributions: Under heteroscedastic retrieval uncertainty, any single dataset-level fusion temperature incurs strictly positive calibration regret, motivating sample-level (g, τ) adaptation.This result formally supports adapting calibration to individual samples rather than relying on one global temperature.
  • Contributions: FCPRAG supervises fusion weights with each passage adapter’s marginal contribution within a multi-LoRA merge while enforcing no-leak train/eval separation.Across HotpotQA and 2WikiMultiHopQA, it improves F1 across three LLM backbones, achieves the best Avg. Rank, and outperforms a random controller ablation.

2 Related Work

Parametric RAG avoids the long-context overhead of in-context retrieval by injecting retrieved knowledge through parameter updates, but multi-passage retrieval makes evidence combination an online bottleneck. FCPRAG targets this gap with retrieval-conditioned evidence-level fusion in parameter space, learning query-specific fusion and calibration signals.

  • Parametric RAG: Parametric RAG injects retrieved knowledge as parameter updates, avoiding long-context overhead but requiring multiple evidence-conditioned updates to be combined before generation.This multi-passage combination is identified as an online bottleneck.
  • PRAG-family systems: PRAG creates passage-specific LoRA weights through offline fine-tuning, while DyPRAG dynamically generates parametric knowledge and Poly-PRAG routes queries to latent LoRA experts.These systems differ in how they parameterize retrieved knowledge.
  • Fusion methods: Existing fusion methods operate mainly in token or prediction space, or through task-level parameter composition.Examples include Fusion-in-Decoder, RePlug, model soups, task arithmetic, TIES-Merging, and AdapterFusion.
  • FCPRAG: FCPRAG studies retrieval-conditioned evidence-level fusion in parameter space, where merge weights vary across queries and overweighting weak evidence can inject spurious knowledge.It addresses this setting by learning per-query fusion scores and calibration signals.

3 Methodology

FCPRAG extends parametric RAG with a lightweight Fusion Controller that predicts retrieval-conditioned adapter fusion scores and sample-level calibration signals. It trains these decisions using merge-aware marginal-contribution supervision derived exclusively from training data.

  • PRAG foundation: PRAG converts retrieved passages into passage-specific LoRA adapters, merges them, and injects the merged adapter into the LLM for answer generation.The Combine variant additionally supplies the retrieved passages in context.
  • Fusion controller: FCPRAG replaces uniform fusion with query-dependent weights that emphasize helpful evidence and suppress noise from irrelevant or conflicting passages.Its Fusion Controller feeds a Weighted Adapter Merger that applies the predicted weights to retrieved adapters.
  • Fusion controller: The controller uses query–passage embedding interactions to predict passage scores, a mixing gate, and an adaptive temperature for calibrated weight mapping.The temperature controls sharpness, while the gate controls deviation from uniform fusion under unreliable score signals.
  • Controller training: Merge-aware labels measure each adapter’s marginal contribution by comparing uniform fusion with and without that adapter, then convert contributions into a target distribution.Flat-contribution samples yield uniform targets and are down-weighted to avoid learning a trivial uniform mapping.
  • Efficiency: FCPRAG adds one query-and-passage embedding pass and lightweight MLP computation, while controller training avoids backpropagation through the LLM.Merge-aware label construction is the main offline cost and can be parallelized or reused across controller variants.
  • Calibration motivation: Under heteroscedastic retrieval uncertainty, no single dataset-level temperature is optimal for every query, motivating sample-level fusion sharpness adaptation.Sharp fusion is favored under low uncertainty, whereas conservative near-uniform fusion is safer under high uncertainty.

4 Experimental Setup

FCPRAG is evaluated across four QA benchmarks and three instruction-tuned LLM backbones using F1 as the primary metric. Comparisons control the retriever and top-K passages while varying knowledge injection and fusion mechanisms, with parametric methods tested in two settings.

  • Datasets and backbones: Experiments cover HotpotQA, 2WikiMultiHopQA, PopQA, and ComplexWebQuestions using Llama-3.2-1B, Qwen2.5-1.5B, and Llama-3-8B.
  • Evaluation metric: F1 (%) is the primary metric, with subtask-level reporting for HotpotQA and 2WikiMultiHopQA and a four-subtask macro average for 2WikiMultiHopQA.
  • Baselines and protocol: Vanilla, Standard RAG, PRAG, DyPRAG, and FCPRAG use the same retriever and top-K passages, isolating knowledge injection and fusion differences.
  • Baselines and protocol: PRAG, DyPRAG, and FCPRAG are evaluated in both Parametric-only and Combine settings, while a randomly initialized controller appears only in the Table 3 ablation.

5 Experiments

Experiments show that FCPRAG improves evidence fusion through learned retrieval-conditioned control, merge-aware supervision, and sample-wise calibration. The method remains effective across varying passage usefulness, tuning conditions, and retrieval perturbations.

  • Overall Results: FCPRAG generally improves over PRAG’s uniform merge, especially when multi-evidence retrieval contains passages with varying usefulness or distractors.These results frame retrieval-conditioned evidence fusion as a bottleneck in parametric RAG.
  • Overall Results: FCPRAG-Combine improves over PRAG-Combine at K = 4 and K = 5, although Parametric-only performance is not monotonic in K.The varying-K check is reported in Appendix B.6.
  • Ablations: The random untrained controller underperforms FCPRAG across subtasks under both Parametric-only and Combine settings.This attributes gains to learned retrieval-conditioned fusion rather than merely adding a module or changing the merge operator.
  • Ablations: Merge-aware labels raise macro F1 from 32.47 to 33.91 on HotpotQA, with the largest comparison-question gain of +2.63.The ablation holds the scoring projector fixed on Qwen2.5-1.5B in the Parametric-only setting.
  • Calibration: Sample-wise calibration predicts (g, τ) that remain in a stable high-performing basin, avoiding degenerate fusion and replacing exhaustive global tuning.Fixed dataset-level calibration is brittle under heteroscedastic retrieval uncertainty, while learned outputs need not match the single best global cell.
  • Retrieval Robustness: FCPRAG retains more relative performance than PRAG under replace-one and repeat-one retrieval perturbations at K = 3.The normalized results support conservative fusion when retrieved evidence is unreliable rather than applying a fixed global temperature.

6 Conclusion

FCPRAG is a learned controller for stable, retrieval-conditioned multi-passage LoRA fusion in parametric RAG. It predicts passage weights and sample-level calibration, improving robustness and reducing global calibration needs across datasets and backbones.

  • 6 Conclusion: FCPRAG provides stable, retrieval-conditioned multi-passage LoRA fusion for parametric retrieval-augmented generation.It is presented as a learned fusion controller.
  • 6 Conclusion: By predicting per-passage weights and per-sample calibration, FCPRAG improves robustness and reduces the need for expensive global calibration.The controller uses retrieval-conditioned fusion signals at the passage and sample levels.
  • 6 Conclusion: Across four datasets and three LLM backbones, FCPRAG improves F1 scores over baseline methods in most settings.The datasets are 2WikiMultiHopQA, HotpotQA, PopQA, and ComplexWebQuestions (CWQ).
  • 6 Conclusion: A random controller performs significantly worse than FCPRAG and the baseline methods.This comparison is reported across the evaluated settings.

Limitations

FCPRAG has limitations in offline training cost, dependence on embedding quality, and scalability beyond the small fixed number of passages studied. Merge-aware supervision becomes more expensive as the number of passages increases.

  • Offline label cost: Merge-aware supervision requires one all-merge and K leave-one-out merges per training sample, increasing offline computation cost.For K = 3, this requires four forward passes per sample.
  • Embedding dependence: Fusion predictions may degrade when query/passage embedding similarity is misaligned with actual answer utility.The controller operates on query/passage embeddings, making performance dependent on their quality.
  • Scaling to larger evidence sets: The approach focuses on a small, fixed number of passages per query, limiting demonstrated scaling to larger evidence sets.Larger K also increases the offline cost of merge-aware supervision.

Ethical Considerations

The work uses public benchmarks without collecting new human-subject data. Its main ethical concern is that parametric injection may make retrieved errors or biased evidence more persistent, despite FCPRAG’s mitigation efforts.

  • The study uses public benchmarks and does not collect new human-subject data.
  • Parametric injection may make retrieved errors or biased evidence more persistent in the model’s effective parameters during inference.
  • FCPRAG down-weights weak or conflicting evidence but does not guarantee factual correctness or fairness.

A Proof of Proposition 1 · A.1 Effect of Temperature and Retrieval Uncertainty

FCPRAG’s proof models fusion as a risk-aware mean–variance trade-off and shows that optimal temperature should rise with retrieval uncertainty. Under heteroscedastic query types, sample-level temperature adaptation can reduce expected calibration regret relative to a global temperature.

  • A Proof of Proposition 1: The proof assigns each retrieved passage a LoRA adapter and score, allowing score distributions to vary across query types while excluding degenerate equal-score cases.If all scores are equal, softmax fusion is uniform for every temperature, so temperature has no effect.
  • A Proof of Proposition 1: Although a global temperature already yields sample-dependent weights through scores, per-sample temperature adaptation can strictly reduce expected calibration regret under heteroscedastic uncertainty.This result motivates adapting temperature to each sample rather than using only a dataset-level constant.
  • A Proof of Proposition 1: The proof assumes independent passage noises, heteroscedastic query types, local score–utility alignment, and an interior optimum for temperature.These assumptions establish distinct uncertainty regimes and support the monotonicity argument around the optimum.
  • A Proof of Proposition 1: Correlated passage noises add cross-terms to the variance, but the monotonicity argument remains applicable and positive correlations can further favor near-uniform fusion under high uncertainty.This is presented as a robustness remark beyond the independence simplification.
  • A Proof of Proposition 1: The analysis uses a risk-aware mean–variance objective that captures the tension between evidence sharpness and amplification of retrieval noise.The paper treats this objective as a tractable proxy and reports that empirical QA experiments validate its conclusion.
  • A.1 Effect of Temperature and Retrieval Uncertainty: As T ↓0, softmax weights become peaked and increase fused-signal variance, whereas as T ↑∞, weights approach uniformity and variance decreases to 1/N.Lower temperature also improves alignment with high-µ_i passages, creating the central sharpness–variance trade-off.
  • A.1 Effect of Temperature and Retrieval Uncertainty: When σ²_t is small, the optimal temperature T⋆_t is small for sharp fusion; when σ²_t is large, the variance penalty shifts T⋆_t upward for flatter fusion.The first-order condition and implicit-function argument establish this shift under the stated alignment and interior-optimum assumptions.

A.2 Impossibility of a Single Dataset-level Temperature … B.4 Reproducibility Details

The theory shows that a single dataset-level temperature incurs strictly positive calibration regret under heteroscedastic retrieval uncertainty, motivating sample-level adaptation. FCPRAG operationalizes this principle through uncertainty-sensitive fusion, lightweight inference, and reproducible controller and evaluation settings.

  • A.2 Impossibility of a Single Dataset-level Temperature: A fixed dataset-level temperature differs from at least one type-specific optimum when both retrieval-uncertainty types occur, causing strictly positive expected calibration regret.This formally establishes that sample-level temperature adaptation is strictly beneficial under heteroscedastic retrieval uncertainty.
  • A.3 Oracle Weights and Connection to FCPRAG: The N = 2 oracle combines a uniform baseline with a noise-shrinking, utility-proportional deviation, motivating FCPRAG’s gate g and temperature τ.The theory provides qualitative guidance rather than quantitative calibration because λ has no direct architectural counterpart.
  • A.3 Oracle Weights and Connection to FCPRAG: FCPRAG should sharpen fusion by increasing g/τ at low uncertainty and become conservative by decreasing g/τ at high uncertainty.The controller infers uncertainty-correlated signals from query–passage interactions rather than observing σ_t^2 directly.
  • A.4 Discussion: Sample-level temperature and gate prediction adapts to heteroscedastic retrieval uncertainty and avoids limitations of dataset-level fusion heuristics.The theoretical argument uses a mean–variance proxy and a minimal two-type mixture, with the argument extending beyond that construction.
  • B.2 Extended Related Work: In-context RAG injects retrieved evidence through passage concatenation or decoder aggregation but remains sensitive to context length, retrieval noise, and evidence conflicts.This motivates parametric and fusion-controlled alternatives that reduce dependence on long, noisy contexts.
  • B.2 Extended Related Work: FCPRAG integrates calibration into PRAG fusion by learning a temperature and mixing gate alongside controller scores.This connects PRAG-family fusion with established calibration and input-dependent component weighting.
  • B.3 Inference Pseudocode: During inference, FCPRAG embeds the query and passages, constructs interaction features, predicts fusion scores with per-sample (g, τ), and supports combine or parametric-only modes.The interaction feature is xi = [eq; epi; eq ⊙epi; |eq −epi|].
  • B.4 Reproducibility Details: Experiments use three instruction-tuned backbones, top-K = 3 passages, bge-base-en-v1.5 embeddings, specified MLP training settings, and fixed-subset multi-hop evaluation.All experiments ran on 2× NVIDIA H100 GPUs; inference adds only a small MLP forward pass and avoids grid search or backbone backpropagation.

B.5 Merge-aware Label Construction and Analysis … B.8 Training-side Prompts

The appendix details merge-aware supervision, its offline construction cost and ranking rationale, sensitivity to retrieval-set size, dataset documentation, and training-side prompt templates for augmentation and passage-specific LoRA parameterization.

  • B.5 Merge-aware Label Construction and Analysis: Merge-aware labels softmax-normalize each adapter’s leave-one-out marginal contribution after comparing all-adapter and adapter-removed F1 evaluations, using training data only.The resulting soft label distribution supervises fusion-weight prediction.
  • B.5 Merge-aware Label Construction and Analysis: Merge-aware labels improve macro F1 over single-adapter labels under fixed scoring projector, backbone, and Parametric-only evaluation.This ablation is reported on HotpotQA with Qwen2.5-1.5B.
  • B.5 Merge-aware Label Construction and Analysis: For N retrieved adapters, label construction costs N + 1 merged evaluations per training sample, remains offline, and supports reuse across controller variants.Isolated adapter behavior is described as a weak proxy for the full merged ranking.
  • B.6 Varying Number of Retrieved Passages: Parametric-only performance on 2WikiMultiHopQA is not monotonic as retrieved-passage count K increases, reflecting the difficulty of fusing additional passage-specific updates and noise.The sensitivity analysis uses Qwen2.5-1.5B beyond the default K = 3 setting.
  • B.6 Varying Number of Retrieved Passages: In the Combine setting, FCPRAG improves over PRAG-Combine at K = 4 and K = 5.These results are part of the varying-K analysis on 2WikiMultiHopQA.
  • B.7 Dataset Overview: The dataset overview appendix provides a table covering datasets and subtasks.The supplied excerpt identifies this material as Table 7.
  • B.8 Training-side Prompts: Training-side prompts support passage rewriting and passage-to-QA generation, with runtime placeholders such as {passage} and {question}.The rewrite prompt preserves entities, proper nouns, key details, core meaning, and relevant information while changing wording.
  • B.8 Training-side Prompts: Passage-specific LoRA adapters are fine-tuned offline from passage-derived QA pairs using both passage-conditioned and question-only prompts.The templates include a passage-and-question format and a question-only format without passage context.

C Case Studies

The case studies show that FCPRAG answered representative questions correctly across 2WikiMultiHopQA, HotpotQA, PopQA, and ComplexWebQuestions. The examples also report per-passage fusion scores, mixing gates, temperatures, and adapter weights when available.

  • 2WikiMultiHopQA: 2WikiMultiHopQA produced the correct answer, identifying Morecambe Church Lads’ Brigade At Drill as earlier than Little Funny Guy, with EM/F1 of 1 / 1.0.The projector-enabled run reported scores [0.5088, 0.5350, 0.5331], g = 0.8357, tau = 0.0988, and weights [0.8644, 1.0767, 1.0588].
  • HotpotQA: HotpotQA correctly selected José Echegaray over Graham Swift for wider professional scope, achieving EM/F1 of 1 / 1.0.Fusion signals were scores [0.4886, 0.4649, 0.4761], g = 0.8178, tau = 0.0817, and weights [1.1235, 0.8864, 0.9901].
  • PopQA: PopQA correctly answered that Brigitte Bardot’s occupation was actress, matching the gold answer with EM/F1 of 1 / 1.0.The projector-enabled run reported scores [0.5344, 0.4883, 0.5225], g = 0.7373, tau = 0.1661, and weights [1.0854, 0.8860, 1.0286].
  • ComplexWebQuestions: ComplexWebQuestions correctly identified Brazil among the countries bordering Bolivia, achieving EM/F1 of 1 / 1.0.The gold answer listed Brazil, Argentina, Paraguay, and Chile, while the evaluated answer was Brazil.
Loading 2608.21750v1…