MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment

1 Introduction

• Single LLMs can produce narrow enrichments, while hand-crafted prompts transfer poorly across model families and make multi-LLM ensembles costly to scale.The cited examples include prompts that work well for Qwen2.5 but transfer poorly to Llama-3.3 or Mistral without further manual tuning.Source: §1 · p. 1
• MERGE independently generates candidate expansions with heterogeneous LLMs, then uses an LLM to synthesize them into one unified query expansion.The synthesis stage is prompted to filter likely hallucinations and merge complementary information.Source: §1 · pp. 1–2
• MERGE separates diverse candidate generation from synthesis, retaining generative flexibility rather than selecting among candidate outputs.The framework is an IR-focused two-stage ensemble combining heterogeneous generation with a generative ensemble step.Source: §1 · p. 2
• Task-grounded APO optimizes prompts for both stages using downstream retrieval performance rather than an LLM judge.The loop makes per-model prompt specialization automatic instead of a manual, non-scalable step.Source: §1 · p. 2
• MERGE improves BM25 retrieval over original queries on all five BEIR benchmarks and matches or outperforms strong LLM-based query-expansion baselines.BM25 is the shared retrieval backbone, so measured gains reflect query manipulation rather than a jointly trained retriever.Source: §1 · p. 2

LLM Ensembles and Output Fusion

• MERGE belongs to ensemble-after-inference methods but targets IR enrichment and automatically tunes every ensemble prompt with a retrieval objective.The related-work discussion distinguishes routing, token-level fusion, response-level selection, and regeneration approaches.Source: §2 · p. 2

Query Expansion for IR

• Unlike single-LLM Query2Doc, HyDE, and Exp4Fuse, which rewrite queries, MERGE uses multiple generators followed by generative synthesis; Exp4Fuse also fuses two BM25 ranked lists.Query2Doc and HyDE are described as generating pseudodocuments or hypothetical answers that clarify user intent.Source: §2 · p. 2, Figure 1 · p. 3
• Unlike supervised docT5query and corpus-side BEATS, MERGE works on user queries without supervision or corpus access and replaces BEATS’s human prompt-refinement step with retrieval scores.docT5query requires corpus access and re-indexing when the corpus changes.Source: §2 · p. 3

Automatic Prompt Optimization

• MERGE builds on APO approaches including textual-gradient, black-box, and evolutionary prompt optimization, but replaces SPO’s LLM evaluator with downstream IR scores and prompt history.The related-work passages identify Pro-TeGi, APE, OPRO, EvoPrompt, Promptbreeder, and SPO as prior approaches.Source: §2 · p. 3

3.1 Framework Overview

• MERGE passes an input query through two LLM stages and sends the resulting plain-text enriched query to a fixed BM25 retriever, leaving the corpus unmodified.The plain-text output can in principle feed other retrievers, though the work focuses on BM25.Source: §3.1 · p. 3

3.2 Stage 1: Multi-LLM Generation

• Stage 1 uses heterogeneous LLMs, each with its own APO-optimized prompt, to generate a candidate that clarifies intent and adds plausible synonyms or related concepts.The seed prompt discourages terms that could leak the answer rather than help locate a relevant passage.Source: §3.2 · p. 4

3.3 Stage 2: Generative Ensemble

• Stage 2 uses a distinct ensemble LLM and an APO-optimized prompt to identify shared intent and high-value entities, resolve redundancy, and synthesize a compact expansion.The ensemble model is kept distinct from Stage-1 models to avoid self-preference bias.Source: §3.3 · p. 4

3.4 Task-Grounded Automatic Prompt Optimization

• Task-grounded APO scores candidate prompts by end-to-end nDCG@10 on a fixed 20% training-data subset using the same BM25 retriever as at test time.It replaces SPO’s LLM evaluator with the downstream IR task and applies optimization stage-wise, freezing Stage-1 prompts before optimizing Stage 2.Source: §3.4 · pp. 4–5
• Each APO round compares two challenger expansions with the incumbent champion, and optimization ends when the champion survives two consecutive rounds.The optimizer receives score comparisons, confidence intervals and p-values, examples of retrieval wins and losses, and theme signals; the history variant also receives the last five rounds.Source: §3.4 · pp. 4–5

3.5 Dual-Level Ensemble

• MERGE combines horizontal diversity across Stage-1 LLMs with vertical Stage-2 synthesis, while APO tunes each model’s role without changing that structure.The paper analogizes the two levels to bagging and stacking, respectively.Source: §3.5 · p. 5

4.1 Setup

• Evaluation uses five heterogeneous BEIR tasks, unmodified corpora, and shared BM25; Stage 1 uses three 7–8B instruction models and Stage 2 Qwen2.5-14B.The baselines share BM25 and differ in how LLM-generated text is used to bridge the query–corpus vocabulary gap.Source: §4.1 · pp. 5–6

4.2 Main Results

• Across all five BEIR benchmarks, MERGE improves BM25 nDCG@10 over vanilla BM25, beats every unsupervised LLM-expansion baseline, and matches or exceeds the strongest supervised baseline on 3/5.All Table 1 methods share the BM25 retriever and differ in how queries are expanded.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• Without APO, seed prompts regress on two datasets; the better APO variant adds +10.9 (NQ), +3.6 (SciFact), +6.5 (FiQA), +13.0 (Touché-2020), and +4.4 (DBPedia) nDCG@10 points.APO turns those seed-prompt regressions into gains on all five datasets without additional manual engineering.Source: §4.2 · p. 7
• Default APO leads on NQ, SciFact, and DBPedia, while history-augmented APO leads on Touché-2020 and FiQA.The authors hypothesize that tournament history helps broader queries but can distract on entity-centric queries.Source: §4.2 · p. 7

4.3 Ablation: Both Stages Contribute

• The Stage-2 ensemble exceeds the best single Stage-1 generator by +4.4 (NQ), +0.9 (SciFact), and +0.6 (FiQA) nDCG@10.All three APO-tuned Stage-1 models improve over the original query, but their gains vary by model.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 Qualitative Analysis

• Optimized prompts add retrieval-friendly context to seed entity lists; Stage 1 contributes distinct signals that Stage 2 fuses, though discovered prompts may read like steering-token bags.The resulting query enrichments remain readable despite this prompt-level readability issue.Source: §4.4 · p. 8

• Across five BEIR benchmarks, MERGE improves BM25 nDCG@10 over original queries on all five (up to +14.9 on NQ), beats BM25+Exp4Fuse on every benchmark, and matches or exceeds docT5query+Exp4Fuse on 3/5.Source: §5 · p. 8
• MERGE couples three heterogeneous 7–8B generators with a 14B generative ensemble and retrieval-grounded APO in both stages.Each prompt is scored by BM25 nDCG@10 on a 20% dev subset; a history-augmented variant feeds the last five tournament rounds back to the optimizer.Source: §5 · p. 8
• Ablations show Stage 2 strictly dominates the best Stage-1 model, while APO turns −4 to −5 point seed-prompt regressions into consistent gains without manual tuning.Source: §5 · p. 9
• MERGE uses compact open-source models without supervised query–document pairs, and its enriched queries can also feed dense retrievers or passage-side enrichment.Source: §5 · p. 9