Source-linked AI summary
Document Ranking with a Pretrained Sequence-to-Sequence Model
Rodrigo Nogueira, Zhiying Jiang, Jimmy Lin
TL;DR
Document ranking commonly uses classification-based encoder-only models, while this work investigates a generation-based alternative using pretrained sequence-to-sequence models. The approach performs competitively across ranking settings, with particular gains in data-poor and zero-shot transfer regimes, while model-size confounding remains a limitation.
Problem
The paper examines whether pretrained sequence-to-sequence models can provide a more data-efficient and fundamentally different alternative to classification-based ranking.
Method
The approach adapts T5 to document reranking by casting the task as sequence generation and using generated relevance target words.
Results
The approach outperforms classification-based models in data-poor settings, achieves zero-shot Robust04 transfer against models tuned on Robust04, and improves with larger T5 models.
Takeaways & Limitations
Target-word generation can exploit pretrained knowledge for document ranking, although the paper leaves the precise mechanism as an open research question.
Takeaways & Limitations
Comparable pretrained T5 and BERT model sizes were unavailable, so improvements may reflect model size rather than intrinsic architectural advantages.
Abstract
from arXiv · showhide
This work proposes a novel adaptation of a pretrained sequence-to-sequence model to the task of document ranking. Our approach is fundamentally different from a commonly-adopted classification-based formulation of ranking, based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words", and how the underlying logits of these target words can be interpreted as relevance probabilities for ranking. On the popular MS MARCO passage ranking task, experimental results show that our approach is at least on par with previous classification-based models and can surpass them with larger, more-recent models. On the test collection from the TREC 2004 Robust Track, we demonstrate a zero-shot transfer-based approach that outperforms previous state-of-the-art models requiring in-dataset cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only model in a data-poor regime (i.e., with few training examples). We investigate this observation further by varying target words to probe the model's use of latent knowledge.
1 Introduction
The paper adapts a pretrained sequence-to-sequence model to document reranking, departing from classification-based encoder-only approaches. It highlights stronger performance with abundant data and greater data efficiency when training examples are limited.
- Unlike common classification formulations, the approach uses a generative model rather than an encoder-only classifier for ranking.
- The work adapts pretrained sequence-to-sequence models, specifically T5, to document reranking.
- In data-rich settings, the method can outperform a pure classification-based encoder-only approach.
- The sequence-to-sequence model significantly outperforms BERT when fine-tuned with limited training examples.The authors attribute this advantage to connecting fine-tuned relevance representations with target words and pretrained latent knowledge.
2 Method
The method casts document relevance as sequence generation: T5 receives a query-document input and generates a binary target word. Ranking uses the relative probability assigned to the “true” token.
- T5 uses a pretrained encoder–decoder architecture in which the ranking task is cast as sequence-to-sequence prediction.
- The input combines the query, document, and a relevance prompt.The illustrated format is “Query: q Document: d Relevant: (1)”.
- The model generates “true” for relevant documents and “false” for non-relevant documents.These target words serve as the ground-truth sequence-to-sequence predictions.
- At inference, a softmax over the “true” and “false” logits produces the relevance probability used for reranking.The method ranks documents by the probability assigned to “true”.
- Single-token target words ensure that each relevance class is represented by one logit.The authors avoid the additional aggregation required when target words split into multiple subwords.
3 Experimental Setup
The experiments evaluate T5 reranking across MS MARCO and Robust04 using standard retrieval baselines and BERT-based comparisons. Robust04 is held out for zero-shot transfer, while training uses fixed T5 configurations and constrained inputs.
- 3.1 Datasets: The experiments use MS MARCO passage ranking and the Robust04 TREC 2004 Robust Track collection.
- 3.1 Datasets: MS MARCO contains 8.8M passages from 1M queries, with approximately 500k training query–relevant-document pairs.
- 3.1 Datasets: Robust04 is evaluated zero-shot because the models are trained on MS MARCO and its relevance judgments are used only as a held-out test set.
- 3.2 Training and inference: T5 base, large, and 3B are fine-tuned for 100k iterations with learning rate 10^-3 and class-balanced batches of 128.
- 3.2 Training and inference: Inputs are capped at 512 tokens with one output token, and greedy decoding is used because only one decoding step contributes logits.
- 3.2 Training and inference: For Robust04 full-length documents, sliding-window passages are classified independently and the highest passage probability becomes the document score.
- 3.3 Baselines: Baselines include BM25, BM25+RM3, and a BM25 followed by BERT-large reranking pipeline.
4 Results and Analysis
T5-based reranking performs strongly across in-domain, zero-shot, and data-poor evaluations, while target-word probing supports a role for pretrained linguistic knowledge. However, comparisons across architectures remain partly confounded by model size and training choices.
- Main Results: T5 models achieve competitive MS MARCO passage-ranking results, with larger models generally performing better than smaller ones.T5-3B had not fully converged at evaluation, so its reported effectiveness may underestimate its potential.
- Main Results: T5 improves over BM25 and BM25+RM3 on Robust04, achieving the highest known reported scores in a zero-shot transfer setting.The models are trained on MS MARCO and evaluated on Robust04 without using its relevance judgments for training.
- Main Results: The evidence does not establish an intrinsic architectural advantage over classification-based models because T5 and BERT lack comparable pretrained model sizes.The authors note that improvements may result from simply using a larger model.
- Effect of Model Size and Training Data: T5 is significantly more effective than BERT with limited training data, while BERT initially performs worse than the BM25 baseline.With 20k total training instances, BERT remains six points behind T5; T5 reaches roughly 45% of the possible BM25 gain using 4% of the training data.
- Target Word Probing Experiments: The authors investigate data efficiency through target-word manipulations that vary linguistic relatedness and polarity while preserving relevance-label prediction.The experiments compare baseline, reversed, antonym, and other target-word configurations on MS MARCO.
- Target Word Probing Experiments: In low-data settings, the baseline target mapping performs best, while manipulations significantly reduce effectiveness with 20k samples.Subword targets perform worse than BM25, whereas other manipulations generally remain above BM25; the experiments are inconclusive about the importance of polarity.
- Target Word Probing Experiments: The findings suggest that T5 exploits latent semantic or linguistic knowledge when predicting relevance, especially in data-poor regimes.The proposed explanation is that T5 can use pretrained output-generation machinery, unlike the classification layer in BERT.
5 Conclusion
The paper introduces a generation-based document-ranking approach using pretrained sequence-to-sequence models and examines why it outperforms classification-based methods, particularly with limited data.
- The paper introduces a novel generation-based approach to document ranking using pretrained sequence-to-sequence models.
- The approach outperforms a classification-based approach, especially in the data-poor regime with limited training data.
- Target-word probing experiments operationalize hypotheses about knowledge gained from pretraining versus task-specific fine-tuning.
- The experiments demonstrate that the model exploits knowledge from its ability to generate fluent natural-language text.
- The exact mechanism underlying this exploitation remains an open research question.