PaperlayerView exact revision on arXiv

Source-linked AI summary

An Analysis of Fusion Functions for Hybrid Retrieval

Sebastian Bruch, Siyu Gai, Amir Ingber

arXiv:2210.11934v2cs.IR

TL;DR

Hybrid retrieval combines lexical and semantic search, but how to fuse their scores is an open question. The paper analyzes convex combination and RRF, finding that convex combination outperforms RRF in-domain and out-of-domain, while requiring little training data to tune.

  • Problem

    Hybrid retrieval fuses lexical and semantic signals, but the relative behavior and properties of its fusion functions require further analysis.

    Source: §3.1 · p. 4, §4 · p. 6

  • Method

    The paper analyzes convex combination of lexical and semantic scores alongside RRF, examining normalization, parameters, and score distributions.

    Source: §3.1 · p. 4, §4 · p. 6, §5 · p. 15

  • Results

    Convex combination of normalized scores outperforms RRF on in-domain and out-of-domain datasets, and its normalization choice is less consequential than previously thought.

    Source: §1 · p. 2, §7 · p. 25

  • Takeaways & Limitations

    Convex combination’s parameter can be tuned sample-efficiently or set using domain knowledge, with α∈[0.6, 0.8] consistently improving results in the reported experiments.

    Source: §6 · p. 23, §7 · p. 25

    The analysis does not study cases where score variance is arbitrarily close to zero, which were not observed on the evaluated datasets.

    Source: §4 · p. 7

Abstract

from arXiv · show

We study hybrid search in text retrieval where lexical and semantic search are fused together with the intuition that the two are complementary in how they model relevance. In particular, we examine fusion by a convex combination (CC) of lexical and semantic scores, as well as the Reciprocal Rank Fusion (RRF) method, and identify their advantages and potential pitfalls. Contrary to existing studies, we find RRF to be sensitive to its parameters; that the learning of a CC fusion is generally agnostic to the choice of score normalization; that CC outperforms RRF in in-domain and out-of-domain settings; and finally, that CC is sample efficient, requiring only a small set of training examples to tune its only parameter to a target domain.

1 INTRODUCTION

The introduction motivates hybrid retrieval by the complementary relevance signals in lexical and semantic search, then asks how to fuse them. It argues that convex combination and Reciprocal Rank Fusion have different properties in normalization, parameter sensitivity, and sample efficiency.

1 INTRODUCTION

  • Hybrid retrieval combines lexical and semantic rankings because their relevance signals are complementary, while the fusion method itself remains an open question.Prior work reported gains, especially out of domain, but how to fuse the rankings warranted further investigation.Source: §1 · pp. 1–2
  • The paper re-examines prior empirical claims by analyzing why fusion methods work and what contributes to their relative performance.Its analysis focuses on the behavior of fusion functions.Source: §1 · p. 2
  • For convex combination, normalization is needed for bounded, consistent fusion, but linear normalization choices can yield rank-equivalent functions.In per-query learning, a solution under one normalization can be transformed to one under another.Source: §1 · p. 2
  • RRF is parameter-sensitive and generalizes poorly out of domain when tuned; because it uses ranks, it also ignores score distributions.The authors regard ignoring distances between raw scores as counter-intuitive in a metric space.Source: §1 · p. 2
  • With only a handful of labeled queries, convex-combination α can be tuned for a target domain regardless of the magnitude of distribution shift.RRF is described as less sample-efficient and as converging to a less effective retrieval system.Source: §1 · p. 2

2 RELATED WORK

Related work traces lexical and semantic retrieval approaches and their respective limitations, then motivates hybrid retrieval. It identifies a gap in comprehensive comparisons of fusion functions’ underlying behavior.

2 RELATED WORK

  • BM25 and related lexical methods match terms, so vocabulary mismatch, including a slight typo, can undermine retrieval.BM25 weights query-term importance and propensity, then adds the term scores.Source: §2 · p. 3
  • Dual encoders make semantic first-stage retrieval feasible by precomputing document vectors and retrieving nearest neighbors efficiently.Deep-model inference is otherwise often too computationally intensive for first-stage retrieval.Source: §2 · pp. 3–4
  • Prior hybrid-search studies [13] [14] [18] [20] [42] [47] mostly focus on in-domain tasks, and none comprehensively compares the main fusion ideas.Semantic retrieval can also underperform BM25 out of domain [5], motivating hybrid approaches.Source: §2 · p. 4

3 SETUP

The setup establishes notation for comparing fusion functions and describes the retrieval systems, datasets, and evaluation procedure used in the empirical analyses.

3 SETUP

  • The setup provides notation and empirical details needed to contextualize the paper’s theoretical and empirical analyses.It also specifies evaluation measures.Source: §3 · p. 4

3.1 Notation

  • A fusion function maps lexical and semantic retrieval scores into a hybrid metric, with the analysis restricted to Lex × Sem.The authors note that much of the analysis extends to fusing multiple retrieval systems.Source: §3.1 · p. 4

3.2 Empirical Setup

  • The evaluation uses MS MARCO Passage Retrieval v1 and eight BeIR datasets, with MS MARCO train queries for tuning and its 6,980-query dev set for evaluation.The BeIR datasets cover question answering, duplicate detection, medical, fact extraction, scientific claim verification, entity search, and financial tasks.Source: §3.2 · p. 5
  • The main experiments pair PISA BM25 lexical retrieval with all-MiniLM-L6-v2 semantic retrieval, while supplementary experiments test other model pairs.The main semantic model produces 384-dimensional vectors; supplementary pairs include BM25 with Splade or Tas-B, Splade with Tas-B, and Tas-B with all-MiniLM-L6-v2.Source: §3.2 · pp. 5–6
  • In supplementary experiments, MS MARCO is treated as in-domain for Splade and Tas-B because both were fine-tuned on that dataset.The remaining datasets are treated as out-of-domain.Source: §3.2 · p. 6
  • The authors form each query’s candidate union, compute missing scores where needed, fuse the candidates, and evaluate Recall and NDCG at specified cutoffs.They use cutoffs of 1,000 except for SciFact and NFCorpus, which use 100; those choices do not change the reported conclusions.Source: §3.2 · p. 6

4 ANALYSIS OF CONVEX COMBINATION OF RETRIEVAL SCORES

The section examines why convex combinations can fuse lexical and semantic retrieval scores and whether normalization materially affects their rankings. It finds the scores complementary across datasets and that reasonable normalization changes can often be offset by adjusting the combination weight.

4 ANALYSIS OF CONVEX COMBINATION OF RETRIEVAL SCORES

  • A convex combination weights normalized semantic and lexical scores, and its score-distribution sensitivity makes normalization important when the component ranges differ.The paper uses min-max scaling as a common choice and considers other monotone transformations.Source: §4 · pp. 6–7
  • The normalization analysis does not study cases where score variance in the union set is arbitrarily close to zero, although observed variance was reasonably large on the evaluated datasets.When a component's variance is zero, fusion can be skipped without affecting retrieval quality.Source: §4 · p. 7

4.1 Suitability of Convex Combination

  • In-domain score patterns align better with α = 0.8 than α = 0.6, while out-of-domain separating lines indicate a more balanced lexical-semantic weighting.The authors also observe that adding either score dimension can distinguish relevant documents that look statistically indistinguishable in the other one-dimensional view.Source: §4.1 · pp. 8–9, Figure 1 · p. 8

4.2 Role of Normalization

  • Across datasets, reasonable normalization changes can preserve rankings through a corresponding weight adjustment, while removing normalization can distort them.The analysis links this to relative expansion rates: stable rates support reproducing rankings, whereas unbounded fusion without normalization can produce rankings unavailable under normalized fusion.Source: §4.2 · pp. 12–14

5 ANALYSIS OF RECIPROCAL RANK FUSION

The analysis finds that TM2C2 outperforms RRF in NDCG across datasets, while RRF is sensitive to parameter choices and tuned parameters generalize poorly out of domain. It also tests whether smoothing RRF’s rank-based fusion affects ranking quality.

5 ANALYSIS OF RECIPROCAL RANK FUSION

  • RRF uses a separate free parameter for each retrieval function, unlike the non-parametric default η = 60 described in prior work.The paper notes that this gives RRF more parameters than a convex combination.Source: §5 · p. 15
  • TM2C2 significantly outperforms RRF on all datasets in NDCG and generally does better in Recall.The reported comparison uses TM2C2 with α = 0.8 and RRF with η = 60.Source: §5 · p. 15, Table 2 · p. 16

5.1 Effect of Parameters

  • Because RRF uses ranks rather than raw scores, documents with different score magnitudes can receive the same fusion treatment when their ranks match.Exact ranks generally require ranking the full collection; the authors approximate them within the union of retrieved top-k sets.Source: §5.1 · pp. 15–16
  • RRF’s reciprocal-rank values cluster near two poles, but the distribution of positive samples around poles and axes varies across datasets.The authors interpret this variation as evidence that adding reciprocal ranks does not behave consistently across domains.Source: §5.1 · p. 17, Figure 6 · p. 18
  • RRF’s NDCG varies sharply with its parameters, and the best direction reverses between MS MARCO and HotpotQA.Increasing a system’s η discounts its ranks; MS MARCO favors ηLex > ηSem, while HotpotQA shows the opposite pattern.Source: §5.1 · p. 17, Figure 7 · p. 19, Figure 8 · p. 19
  • Tuning RRF improves in-domain NDCG but does not generalize well to out-of-domain datasets.The authors attribute the pattern reversal to different parameter settings favoring in-domain and out-of-domain datasets.Source: §5.1 · p. 17, Table 3 · p. 20

5.2 Effect of Lipschitz Continuity

  • As SRRF’s β decreases from high values, its ranking quality can improve over RRF, especially for η = 60 and out-of-domain datasets.When β is too small, approximation error is large and ranking quality degrades; as β grows, results trend toward RRF.Source: §5.2 · pp. 20–21, Figure 9 · p. 21, Figure 10 · p. 21

6 DISCUSSION

The discussion frames fusion around boundedness and preservation of score information, then contrasts the behavior of convex combination and RRF. TM2C2’s weight can be tuned with little data, while fully parameterized RRF remains less effective.

6 DISCUSSION

  • The proposed fusion desiderata include monotonicity in positively correlated scores and rank invariance under positive rescaling of query or document vectors.The discussion also calls for boundedness and favors interpretable, sample-efficient tuning or expert-guided parameter setting when zero-shot applicability matters.Source: §6 · pp. 22–23
  • Bounded fusion scores address the inconsistency of unnormalized convex combinations, while normalization can provide boundedness.The discussion identifies boundedness as a desired property because lexical and semantic scores can have different scales and BM25 is unbounded.Source: §6 · pp. 22–23
  • With unnormalized scores, convex combination severely degrades in-domain ranking but often improves out-of-domain ranking, where semantic scores are small relative to lexical scores.The authors attribute the in-domain decline to lexical scores distorting strong semantic rankings, and the out-of-domain gains to lexical retrieval helping when semantic retrieval does poorly.Source: §6 · p. 22
  • RRF discards raw-score distribution information, while TM2C2’s smooth score combination preserves more of the underlying distribution’s characteristics.The discussion uses Lipschitz continuity to formalize this distinction: a larger Lipschitz constant corresponds to greater distortion of the retrieval-score distribution.Source: §6 · p. 23
  • With less than 5% of training data, TM2C2’s α converges across the depicted and other datasets, whereas fully parameterized RRF remains suboptimal to TM2C2.The experiments progressively sample training subsets and evaluate tuned functions on test splits; RRF’s performance more or less converges despite parameter sensitivity to the training sample.Source: §6 · pp. 23, 25, Figure 12 · p. 24
  • Adding a weight to the combination of RRF terms does not significantly affect overall performance in the reported comparison.The tested RRF-CC formulation tunes α, ηLex, and ηSem.Source: §6 · p. 25

7 CONCLUSION

The paper concludes that normalized convex combination outperforms RRF in both in-domain and out-of-domain settings, and that normalization choice is less consequential than previously thought. TM2C2 can be tuned efficiently, although the analysis leaves per-query nonlinear fusion and fusion across multiple retrieval systems for future work.

7 CONCLUSION

  • Normalized convex combination outperforms RRF on in-domain and out-of-domain datasets, contrary to [5].The paper also reports that RRF is sensitive to its parameters and that convex-combination performance is not sensitive to normalization choice when the transformation has reasonable properties.Source: §7 · p. 25
  • In the reported experiments, α values in [0.6, 0.8] consistently lead to improvements, and the parameter can be tuned sample-efficiently or set using domain knowledge.The conclusion identifies theoretical minimum-maximum normalization as the TM2C2 normalization.Source: §7 · p. 25
  • Whether per-query nonlinear fusion or normalization improves α prediction remains unresolved and is left for future work.The authors note that their analysis of query-document pairs may not hold on a per-query basis, which they describe as a non-trivial effort.Source: §7 · p. 25
  • The findings still need validation when fusion combines more than lexical and semantic retrieval systems.The paper leaves an extended analysis of fusion across multiple retrieval systems to future work.Source: §7 · p. 25

A FUSION OF SPLADE AND BM25

This section’s supplied materials describe normalization comparisons for Splade–BM25 fusion and identify the scope of a dataset-level NDCG table and an SRRF comparison.

A FUSION OF SPLADE AND BM25

  • Figure 13 plots the effect of normalization on Splade–BM25 convex combination as α varies, alongside the relative expansion rate of Splade scores with respect to BM25 scores.The right-hand plot reports 95% confidence intervals; φ-f_o denotes applying normalization only to f_o.Source: Figure 13 · p. 29
  • Table 5 reports test-split NDCG for individual systems and RRF and TM2C2 fusions across datasets, with significance markers defined relative to each fusion.SciFact and NFCorpus use cutoff 100; other datasets use NDCG@1000.Source: Table 5 · p. 30
  • Figure 14 plots the NDCG@1000 difference between SRRF and RRF with η=60 as β varies; positive values indicate better ranking quality for SRRF.Source: Figure 14 · p. 30

B FUSION OF TAS-B AND BM25

This section’s supplied materials describe normalization comparisons for Tas-B–BM25 fusion and identify the scope of a dataset-level NDCG table and an SRRF comparison.

B FUSION OF TAS-B AND BM25

  • Figure 15 plots the effect of normalization on Tas-B–BM25 convex combination as α varies, alongside the relative expansion rate of Tas-B scores with respect to BM25 scores.The right-hand plot reports 95% confidence intervals; φ-f_o denotes applying normalization only to f_o.Source: Figure 15 · p. 31
  • Table 6 reports test-split NDCG for individual systems and RRF and M2C2 fusions across datasets, with significance markers defined relative to each fusion.SciFact and NFCorpus use cutoff 100; other datasets use NDCG@1000.Source: Table 6 · p. 32
  • Figure 16 plots the NDCG@1000 difference between SRRF and RRF with η=60 as β varies; positive values indicate better ranking quality for SRRF.Source: Figure 16 · p. 32

C FUSION OF TAS-B AND SPLADE

This section examines Tas-B and Splade fusion through normalization effects, score expansion, and test-set NDCG@1000 comparisons of individual systems, RRF, and M2C2.

C FUSION OF TAS-B AND SPLADE

  • Figure 17 plots Tas-B/Splade convex-fusion behavior against α and the relative expansion rate of Tas-B scores with 95% confidence intervals.It also distinguishes applying normalization to only one fusion input from leaving the other unnormalized.Source: Figure 17 · p. 33
  • Table 7 compares test-split NDCG@1000 (cutoff 100 for SciFact and NFCorpus) for individual systems, RRF, and M2C2.M2C2 uses α=0.2 in the Tas-B/Splade convex combination; significance markers are defined relative to M2C2 and RRF.Source: Table 7 · p. 34
  • Figure 18 plots the NDCG@1000 difference between SRRF and RRF with η=60 as β varies.Positive values indicate better ranking quality by SRRF.Source: Figure 18 · p. 34

D FUSION OF TAS-B AND ALL-MINILM-L6-V2

This section examines Tas-B and All-MiniLM-L6-v2 fusion through normalization effects, score expansion, and test-set NDCG@1000 comparisons of individual systems, RRF, and M2C2.

D FUSION OF TAS-B AND ALL-MINILM-L6-V2

  • Figure 19 plots Tas-B/All-MiniLM-L6-v2 convex-fusion behavior against α and the relative expansion rate of Tas-B scores with 95% confidence intervals.It also distinguishes applying normalization to only one fusion input from leaving the other unnormalized.Source: Figure 19 · p. 35
  • Table 8 compares test-split NDCG@1000 (cutoff 100 for SciFact and NFCorpus) for individual systems, RRF, and M2C2.M2C2 uses α=0.8 in the Tas-B/All-MiniLM-L6-v2 convex combination; significance markers are defined relative to M2C2 and RRF.Source: Table 8 · p. 36
  • Figure 20 plots the NDCG@1000 difference between SRRF and RRF with η=60 as β varies.Positive values indicate better ranking quality by SRRF.Source: Figure 20 · p. 36

Related papers

Loading 2210.11934v2…