PaperlayerView exact revision on arXiv

Source-linked AI summary

Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models

Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, Marc Najork

arXiv:2201.10582v1cs.IR

TL;DR

The paper examines whether deep retrieval trained in one domain generalizes to new domains without in-domain training, where domain shift can weaken its performance. It combines lexical and deep retrieval through a non-parametric rank-fusion framework and finds gains over either model in out-of-domain datasets.

  • Problem

    Evidence on deep retrievers’ zero-shot generalization across domains is limited, despite the computational and labeled-data costs of training them.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The paper trains lexical and deep retrievers separately and combines their rankings with Reciprocal Rank Fusion without fine-tuning for new domains.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    In three out-of-domain datasets, the hybrid achieves an average 20.4% relative recall gain over the deep model and 9.54% over BM25.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Deep retrieval weakens under substantial domain shift while lexical retrieval is more robust, and combining them is effective across in-domain and out-of-domain datasets.

    Source: §1 · p. 2, §7 · p. 12

    The hybrid’s effectiveness for rare-entity question answering remains to be investigated.

    Source: §7 · p. 12

Abstract

from arXiv · show

The pre-trained language model (eg, BERT) based deep retrieval models achieved superior performance over lexical retrieval models (eg, BM25) in many passage retrieval tasks. However, limited work has been done to generalize a deep retrieval model to other tasks and domains. In this work, we carefully select five datasets, including two in-domain datasets and three out-of-domain datasets with different levels of domain shift, and study the generalization of a deep model in a zero-shot setting. Our findings show that the performance of a deep retrieval model is significantly deteriorated when the target domain is very different from the source domain that the model was trained on. On the contrary, lexical models are more robust across domains. We thus propose a simple yet effective framework to integrate lexical and deep retrieval models. Our experiments demonstrate that these two models are complementary, even when the deep model is weaker in the out-of-domain setting. The hybrid model obtains an average of 20.4% relative gain over the deep retrieval model, and an average of 9.54% over the lexical model in three out-of-domain datasets.

1 Introduciton

The paper asks whether deep retrieval models trained in one domain can be applied to new domains without in-domain training. It studies domain shifts and proposes combining lexical and deep retrieval after finding that their strengths are complementary.

1 Introduciton

  • The study evaluates zero-shot generalization across five datasets: two in-domain and three out-of-domain with different levels of domain shift.The authors ask whether a deep retrieval model trained in one domain can be applied directly to new datasets or domains without in-domain training data.Source: §1 · p. 2
  • Deep retrieval performs well on related domains but deteriorates when the target domain is distinct from its training domain, while lexical models are more robust across datasets and domains.The authors report that lexical and deep models retrieve different sets of relevant documents, making them complementary.Source: §1 · p. 2
  • The proposed zero-shot hybrid combines separately trained lexical and deep models through Reciprocal Rank Fusion, without fine-tuning on new datasets or domains.The paper also frames its questions around complementarity with query and document expansion and combining retrieval approaches non-parametrically.Source: §1 · p. 2
  • In three out-of-domain datasets, the hybrid gains an average 20.4% relative recall over the deep model and 9.54% over BM25.The paper also reports that the hybrid outperforms stronger baselines including query and document expansion.Source: §1 · p. 2
  • The paper presents its contribution as the first zero-shot hybrid retrieval model incorporating lexical matching, expansion, and deep retrieval.It describes the approach as simple and effective across a variety of datasets and domains.Source: §1 · p. 2

2 Related Work

Prior work covers lexical retrieval, deep semantic retrieval, and hybrids, but the cited hybrid evaluations leave cross-domain zero-shot performance unclear. This paper studies deep-model generalization and proposes a hybrid for that setting.

2 Related Work

  • Lexical first-stage retrieval traditionally uses BM25 to capture exact query–document lexical matches; vocabulary mismatch may be reduced by expanding queries via pseudo-relevance feedback and, or documents via related documents.Source: §2 · p. 3
  • Deep retrievers use lower-dimensional semantic representations to bridge vocabulary gaps, with dual encoders enabling nearest-neighbor search accelerated by quantization [14].The passage contrasts this with lexical matching and describes separate query and document vectors scored by dot product or cosine similarity.Source: §2 · p. 3
  • Earlier deep-retrieval studies cite expensive computation and labeled-data creation as reasons to consider direct zero-shot application, while noting limited evidence on generalization.BEIR is identified as one prior evaluation across diverse domains and tasks.Source: §2 · pp. 3–4
  • Zero-shot evaluations in BEIR found deep models generalized poorly and fell significantly below BM25 under large domain shifts.This paper reports similar deterioration and adds a hybrid model that uses lexical retrieval to alleviate domain shift.Source: §2 · p. 4
  • Prior hybrid work uses score interpolation [17] [24] [25] [21], RM3-based selection [18], or other combinations [45], but its evaluations are limited to single tasks and domains.The paper identifies cross-domain evaluation without fine-tuning as an unresolved question and aims to compare a zero-shot hybrid with either model alone.Source: §2 · p. 4

3 Method

The method trains lexical and deep retrievers separately, then combines their rankings non-parametrically for zero-shot use. It instantiates the lexical side with BM25 and expansion methods and the deep side with NPR.

3 Method

  • The framework trains lexical and deep retrieval models separately and combines them with a non-parametric method designed for simplicity and flexibility.It is intended to be applied to new domains without in-domain training data.Source: §3 · p. 4

3.1 Hybrid retrieval model

  • Reciprocal Rank Fusion combines candidate ranking positions rather than model scores, avoiding domain-specific score normalization and weight tuning for zero-shot use.The paper notes that interpolation can be sensitive to score scales and assigned weights; RRF uses rank positions induced by each model's scores.Source: §3.1 · p. 5

3.2 Lexical retrieval model

  • BM25 is the base lexical retriever, supplemented by pseudo-relevance-feedback query expansion with RM3 or Bo1 and T5-based document expansion.For document expansion, generated queries are appended to passages and aggregated as document expansions.Source: §3.2 · pp. 5–6

3.3 Deep retrieval model

  • The deep retriever is NPR, a dual encoder trained on MS-MARCO passage data; documents are split into overlapping passages and scored using the best passage score.NPR uses dot-product relevance scores and negative-sampling strategies; the framework can replace it with another deep model.Source: §3.3 · p. 6

4 Experimental Setup

The evaluation spans five datasets, from MS-MARCO in-domain tasks to ORCAS, Robust04, and TREC-COVID with increasing domain shifts. It benchmarks lexical and deep retrieval models using dataset-specific processing and tuned lexical setups.

4.1 Datasets

  • Five datasets cover MS-MARCO passage and document retrieval, ORCAS, Robust04 news, and TREC-COVID biomedical retrieval.ORCAS queries are sampled by query length; Robust04 has 528K news stories and 250 queries.Source: §4.1 · pp. 6–7
  • The study uses MS-MARCO passage dev for in-domain evaluation and MS-MARCO doc to test generalization to document retrieval.NPR and docT5query are trained on MS-MARCO passage training data.Source: §4.1 · p. 6
  • Robust04 and TREC-COVID provide out-of-domain tests in news and biomedical domains, with TREC-COVID described as distinct from MS-MARCO.ORCAS has wider-topic, shorter queries than MS-MARCO and draws candidate documents from its collection.Source: §4.1 · pp. 6–7

4.2 Data processing and benchmarking

  • NPR is trained on MS-MARCO passage and applied to the other four datasets without fine-tuning; long documents are split into overlapping passages and scored by their best passage.The passage windows contain ten sentences with a stride of five; SCaNN performs passage-level nearest-neighbor search.Source: §4.2 · p. 7
  • Lexical baselines use Terrier defaults, tuned Bo1 settings, and docT5query expansions generated from passages or abstracts.RM3 was also tuned but yielded lower performance than Bo1 on all five datasets, so only Bo1 results are reported.Source: §4.2 · pp. 7–8

5 Evaluation

The evaluation finds that NPR weakens as domain shift increases, while lexical expansion and deep retrieval contribute complementary results. The proposed hybrid consistently outperforms either model alone, with especially clear gains over NPR on the three out-of-domain datasets.

5 Evaluation

  • Recall@1K is the primary metric because the work focuses on first-stage retrieval, and MAP is also reported.The evaluation addresses research questions posed in Section 1.Source: §5 · p. 8

5.1 Generalization of the deep retrieval model

  • NPR beats BM25 on MS-MARCO passage by absolute 10.77 (relative 12.35%) in Recall@1K and 16.15 (relative 83.59%) in MAP.On MS-MARCO doc, NPR also beats BM25 by 4.55 (5.0%) at Recall@1K and 3.86 (14.57%) at MAP.Source: §5.1 · p. 8
  • NPR's advantage declines with domain shift: it beats BM25 on ORCAS, is mixed on Robust04, and loses to BM25 on TREC-COVID by 11.71 (23.76%) in Recall@1K and 10.72 (38.48%) in MAP.TREC-COVID has the largest domain shift; ORCAS has the least among the three out-of-domain datasets.Source: §5.1 · pp. 8–9

5.2 Utility of query and document expansion

  • Bo1 query expansion brings relative Recall@1K gains of 1% on MS-MARCO passage/doc and ORCAS, 8.48% on Robust04, and 6.67% on TREC-COVID.docT5query gives BM25 very large gains on MS-MARCO passage and more than 2.5% recall gains on each of the other four datasets.Source: §5.2 · p. 10

5.3 Complementarity of lexical and deep retrieval models

  • BM25+Bo1, BM25+docT5query, and NPR retrieve complementary relevant documents; NPR retrieves the largest number of unique relevant results, but fewer relevant results overall than either expansion method.Figure 1 shows overlaps for Robust04 and TREC-COVID; the other datasets have around one relevant document per query.Source: §5.3 · p. 10

5.4 Effectiveness of the proposed hybrid model

  • The best MS-MARCO passage hybrid, BM25+docT5query and NPR, reaches Recall@1K 98.65 and beats BM25 and NPR by relative 12.94% and 0.52%, respectively.It also outperforms coCondenser (Recall@1K=98.4) [12]; the best MS-MARCO document hybrid combines Bo1, docT5query, and NPR.Source: Table 3 · p. 9, §5.4 · p. 10
  • The hybrid improves over NPR by 6.11%, 14.16%, and 44.25% in ORCAS, Robust04, and TREC-COVID, respectively, and by almost 10% relatively over BM25 across all three.The reported improvements of hybrid models over baselines in Recall@1K are statistically significant except for comparisons 5/7 versus 2 in Robust04 and TREC-COVID.Source: Table 4 · p. 9, §5.4 · p. 10

6 Discussion

The discussion contrasts non-parametric reciprocal-rank fusion with weight-tuned interpolation and examines how NPR performance varies with query length. Spot-checks connect short-query strengths to subword handling and long-query weaknesses to exact-match needs.

Discussion

  • The comparison focuses solely on recall because the study does not apply a second re-ranking stage to optimize early precision.Source: §6 · p. 10
  • RRF(BM25, NPR) outperforms the best tuned BM25–NPR interpolation by a relative 3% on both Robust04 and TREC-COVID, while an oracle fusion suggests further headroom.Interpolation requires weight tuning and is weight-sensitive; Figure 2 compares interpolation, the hybrid, and the oracle.Source: §6 · p. 11, Figure 2 · p. 11
  • NPR largely beats BM25 and expansion models on very short ORCAS queries, but its performance deteriorates at seven or more tokens.Spot-checks associate short-query advantages with misspellings or compounds that lexical models may treat as OOV; long complex queries needing specific information can favor BM25's exact match.Source: §6 · p. 12, Table 5 · p. 12

7 Conclusion

The paper concludes that deep retrieval generalizes poorly under large domain shifts, whereas lexical matching and expansion are more robust across domains. It proposes a non-parametric zero-shot hybrid integrating lexical matching, expansion, and deep retrieval, and identifies further tasks and signals for investigation.

Conclusion

  • The conclusion reports that the non-parametric zero-shot hybrid is effective on both in-domain and out-of-domain datasets.The hybrid integrates lexical matching, expansion, and deep retrieval models.Source: §7 · p. 12
  • Future work includes testing the hybrid on entity-centric QA, parameterizing it with query and domain signals, and exploring domain adaptation for out-of-domain deep retrieval.The proposed signals include query structure, query length, and degree of domain shift.Source: §7 · p. 12

Related papers

Loading 2201.10582v1…