Source-linked AI summary
The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
Ronak Pradeep, Rodrigo Nogueira, Jimmy Lin
TL;DR
The paper addresses how to build effective multi-stage text-ranking systems across ad hoc retrieval tasks. It proposes the Expando-Mono-Duo pattern, using sequence-to-sequence models for document expansion and pointwise and pairwise reranking, and reports effectiveness at or near the state of the art across several benchmarks. The paper also presents the combined design as a foundation and reference for transformer-based multi-stage ranking architectures.
Problem
The paper addresses the need to combine effective document expansion and reranking components into a coherent multi-stage design for text ranking across tasks and domains.
Method
Expando-Mono-Duo uses sequence-to-sequence models for pre-indexing document expansion, keyword-based candidate retrieval, pointwise Mono reranking, and pairwise Duo reranking.
Results
Across MS MARCO, TREC 2020 Deep Learning Track, and TREC-COVID, the pattern achieves effectiveness close to or at the state of the art, with additive and cumulative benefits from its components.
Takeaways & Limitations
Expando-Mono-Duo provides a foundation and reference for transformer-based multi-stage ranking architectures, including settings that can be zero-shot.
Takeaways & Limitations
Solutions for texts longer than the models’ input sequences are corpus dependent, and duoT5’s longer inputs impose additional computational costs.
Abstract
from arXiv · showhide
We propose a design pattern for tackling text ranking problems, dubbed "Expando-Mono-Duo", that has been empirically validated for a number of ad hoc retrieval tasks in different domains. At the core, our design relies on pretrained sequence-to-sequence models within a standard multi-stage ranking architecture. "Expando" refers to the use of document expansion techniques to enrich keyword representations of texts prior to inverted indexing. "Mono" and "Duo" refer to components in a reranking pipeline based on a pointwise model and a pairwise model that rerank initial candidates retrieved using keyword search. We present experimental results from the MS MARCO passage and document ranking tasks, the TREC 2020 Deep Learning Track, and the TREC-COVID challenge that validate our design. In all these tasks, we achieve effectiveness that is at or near the state of the art, in some cases using a zero-shot approach that does not exploit any training data from the target task. To support replicability, implementations of our design pattern are open-sourced in the Pyserini IR toolkit and PyGaggle neural reranking library.
1 INTRODUCTION
The paper introduces Expando-Mono-Duo, a multi-stage text-ranking design pattern that uses pretrained sequence-to-sequence models for document expansion and reranking. It combines these components across several retrieval tasks and presents ablation analyses to quantify their contributions.
- 1 INTRODUCTION: Expando-Mono-Duo uses T5-based sequence-to-sequence models within a multi-stage ranking architecture instead of encoder-only transformers such as BERT.The approach can also extend to sequence-to-sequence models such as BART and Pegasus.
- 1 INTRODUCTION: The pattern expands documents before indexing, retrieves candidates with keyword search, then applies pointwise Mono and pairwise Duo rerankers.Document expansion enriches keyword representations, while the two rerankers process candidates from initial retrieval.
- 1 INTRODUCTION: Effectiveness is at or near the state of the art across ad hoc retrieval tasks in different domains, sometimes without target-task fine-tuning.The reported evaluations cover MS MARCO, the TREC 2020 Deep Learning Track, and TREC-COVID.
- 1 INTRODUCTION: The paper is the first to bring the separate Expando, Mono, and Duo research threads together in a coherent, self-contained account with component ablations.The ablation analyses quantify the impact of each component.
2 BACKGROUND AND RELATED WORK
The paper situates its design in established multi-stage retrieval systems, where keyword or dense retrieval generates candidates and transformer models rerank them. It combines prior component-level developments while shifting the framework toward sequence-to-sequence models and document expansion.
- 2 BACKGROUND AND RELATED WORK: Ad hoc retrieval ranks a top-k list of corpus texts for an information need using a quality metric such as nDCG or MRR.The paper uses “ad hoc retrieval” and “ranking” interchangeably.
- 2 BACKGROUND AND RELATED WORK: Multi-stage ranking decomposes retrieval into candidate generation followed by successive reranking stages until the final results are produced.The retrieved unit may be a document, passage, or hybrid construction depending on the task.
- 2 BACKGROUND AND RELATED WORK: Multi-stage architectures balance model complexity and search latency by controlling candidate-set sizes across stages.The design has documented deployments in Bing and Alibaba’s e-commerce search engine.
- 2 BACKGROUND AND RELATED WORK: Neural ranking evolved from manually engineered learning-to-rank features toward continuous vector representations and pretrained transformer rerankers.BERT-based reranking became dominant, while approximate nearest-neighbor search over dense representations emerged as another retrieval direction.
- 2 BACKGROUND AND RELATED WORK: The paper unifies earlier work on transformer multi-stage ranking, document expansion, Mono, and Duo, which had previously appeared piecewise.Its sequence-to-sequence formulation contrasts with earlier BERT-based work and adds document expansion to the combined design.
3 EXPANDO-MONO-DUO WITH T5
The Expando-Mono-Duo architecture combines document expansion before indexing with keyword retrieval and sequential pointwise and pairwise reranking using pretrained sequence-to-sequence models.
- Architecture: Multi-stage ranking passes progressively smaller candidate lists through stages H0 to HN, with each stage receiving and reranking the previous list.The candidate count cannot increase between stages: k_n ≤ k_n−1.
- Expando: Expando enriches documents before indexing by appending queries predicted from their content, potentially adding terms absent from the original text.The expanded corpus feeds keyword retrieval, while reranking uses the original document texts.
- H0: Keyword Retrieval: H0 retrieves top k0 candidates with BM25, optionally using pseudo-relevance feedback depending on the task setting.The implementation uses a standard inverted index through Pyserini.
- Mono: MonoT5 pointwise-reranks candidates by estimating each candidate’s relevance score for the query.It is a sequence-to-sequence adaptation of monoBERT and uses relevance classification terminology.
- Practical considerations: Transformer ranking must address texts longer than the models’ typical 512-token input limit, using corpus-dependent solutions discussed later in the paper.The paper defers detailed treatment of these solutions to Section 4.
- Duo: DuoT5 pairwise-reranks the Mono output by estimating whether one candidate is more relevant than another for the query.The pairwise model predicts “true” when the first document is more relevant than the second; aggregate scoring variants include Sum, Sum-Log, Sym-Sum, and Sym-Sum-Log.
4 EXPERIMENTAL SETTINGS
The experiments instantiate and evaluate Expando-Mono-Duo across MS MARCO passage and document ranking and TREC 2020 Deep Learning Track settings. They specify corpus characteristics, expansion and retrieval procedures, reranking configurations, and computational constraints.
- The design pattern was evaluated in five settings: MS MARCO passage and document ranking, TREC 2020 passage and document ranking, and TREC-COVID.
- MS MARCO passage ranking uses 8.8M passages with mean length 56 tokens, while document ranking uses 3.2M documents with mean length 1131 tokens.The document corpus has a median length of 584 tokens and a maximum of 333757 tokens.
- “Expando” and Keyword Retrieval Settings: For MS MARCO passage ranking, doc2query-T5 expanded every passage before BM25 indexing, using five sampled queries per passage and BM25 parameters k1 = 0.82 and b = 0.68.Expansion inference for the 8.8M-passage corpus took approximately 40 hours on one TPU; TREC topics additionally used BM25 + RM3.
5 RESULTS
Across MS MARCO and TREC evaluations, Expando-Mono-Duo generally delivered strong effectiveness, with document expansion and sequential reranking contributing distinct gains. Results also show that computational cost and evaluation conditions matter when interpreting these improvements.
- MS MARCO: The full Expando-Mono-Duo configuration was near the state of the art on MS MARCO passage and document ranking tasks.It was the best leaderboard result at submission time for both tasks, though later submissions surpassed it.
- MS MARCO passage ranking: A nearly 50% relative improvement followed from adding document expansion to first-stage retrieval on MS MARCO passage ranking.The gain carried over to the held-out test set without neural inference at query time, although retrieval latency increased modestly.
- MS MARCO: Adding monoT5-3B and duoT5-3B produced additional cumulative gains over doc2query-T5 on MS MARCO passage and document ranking.The reported development-set gains were statistically significant at p< 0.01, and the complete combination achieved the best score in each task.
- Model and reranking comparisons: Larger sequence-to-sequence models improved effectiveness, while Duo improved over Mono even when using the smaller model.A single larger reranker nevertheless beat the pointwise/pairwise combination with a smaller model in the reported ablation.
- Candidate-depth analysis: Pairwise reranking gains mostly came from reshuffling the highest-ranked candidates, while increasing k_1 generally reduced effectiveness slightly.The duo model requires a quadratic number of candidate-pair inferences; leaderboard submissions used k_1 = 50 with Sym-Sum.
- Query and document expansion: Pseudo-relevance feedback improved recall but did not consistently improve end-to-end effectiveness after neural reranking.With document expansion, query expansion did not appear to matter much, and the highest nDCG@10 run omitted it.
6 SUMMARY AND CONCLUSION
The paper proposes Expando-Mono-Duo T5 as a multi-stage ranking design pattern that adapts sequence-to-sequence models for document expansion, pointwise reranking, and pairwise reranking. Across several retrieval benchmarks, the components produce additive gains and effectiveness at or near the state of the art.
- Expando-Mono-Duo T5 combines document expansion, pointwise ranking, and pairwise ranking in a multi-stage architecture.The three components are called Expando, Mono, and Duo.
- Sequence-to-sequence models can support document expansion, pointwise ranking, and pairwise ranking through transformations, input templates, and output probabilities.Document expansion enriches text representations, while rerankers extract relevance probabilities from model outputs.
- Document expansion improves retrieval effectiveness without requiring computationally expensive neural inference at query time.
- Pointwise reranking with monoT5 is highly effective across first-stage retrieval results with or without document expansion or pseudo-relevance feedback.
- Pairwise reranking with duoT5 further improves monoT5, particularly on early-precision metrics, and the three components provide additive and cumulative benefits.
- Experiments on MS MARCO, TREC 2020 Deep Learning, and TREC-COVID achieve ranking results close to or at the state of the art.The design pattern is presented as a foundation and reference for transformer-based multi-stage ranking architectures.