Source-linked AI summary
RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, Michael Bendersky
TL;DR
Text ranking has limited support for T5 because existing approaches commonly cast ranking as token classification and require postprocessing. RankT5 introduces encoder-decoder and encoder-only models that directly output scores and use ranking losses, producing substantial gains and stronger out-of-domain zero-shot performance in supported settings.
Problem
T5-based text ranking is under-explored because ranking requires real-valued query-document scores and ranking losses over multiple documents, unlike typical per-example generation losses.
Method
RankT5 uses encoder-decoder and encoder-only T5 structures that output real numbers and are fine-tuned with pairwise or listwise ranking losses.
Results
RankT5 ranking-loss models substantially outperform classification-loss T5 rankers on MS MARCO and Natural Question, while some ranking losses yield better zero-shot performance.
Takeaways & Limitations
Ranking-loss fine-tuning improves T5 text-ranking performance and can maintain improvement in zero-shot evaluation on out-of-domain data sets.
Takeaways & Limitations
The paper leaves open whether encoder-decoder and encoder-only structures differ when training data is insufficient and identifies further architecture and pretraining directions.
Abstract
from arXiv · showhide
Recently, substantial progress has been made in text ranking based on pretrained language models such as BERT. However, there are limited studies on how to leverage more powerful sequence-to-sequence models such as T5. Existing attempts usually formulate text ranking as classification and rely on postprocessing to obtain a ranked list. In this paper, we propose RankT5 and study two T5-based ranking model structures, an encoder-decoder and an encoder-only one, so that they not only can directly output ranking scores for each query-document pair, but also can be fine-tuned with "pairwise" or "listwise" ranking losses to optimize ranking performances. Our experiments show that the proposed models with ranking losses can achieve substantial ranking performance gains on different public text ranking data sets. Moreover, when fine-tuned with listwise ranking losses, the ranking model appears to have better zero-shot ranking performance on out-of-domain data sets compared to the model fine-tuned with classification losses.
1 Introduction
Text ranking has advanced through pretrained language models, but leveraging T5 remains under-explored because ranking requires numerical scores and ranking-specific objectives. RankT5 addresses this gap with two T5 structures and ranking-loss fine-tuning, achieving improved ranking and zero-shot performance.
- Text ranking orders documents by relevance to a query or context and supports applications including search and question answering.
- T5 is challenging to apply natively because ranking models must output numerical scores, whereas typical T5 fine-tuning uses text-generation losses for individual examples.
- Existing T5 ranking approaches formulate relevance as predicting “true” or “false” tokens and derive document rankings through inference-time postprocessing.
- RankT5 introduces encoder-decoder and encoder-only structures that output real-valued scores and support pairwise or listwise ranking losses.
- RankT5 ranking-loss models substantially outperform classification-loss T5 ranking models on MS MARCO and Natural Question, with better zero-shot performance for some ranking losses.
2 Related Work
Related work spans pretrained-language-model rankers, alternative model structures, generative-likelihood methods, retrieval-oriented architectures, and pointwise or listwise ranking losses. RankT5 is positioned as an early-stage reranking approach that directly supports ranking-loss optimization.
- Pretrained language models have become state-of-the-art tools for text ranking, extending earlier literature based on handcrafted features and ranking algorithms.
- Model structure: Cross-attention models concatenate a query and candidate document, while pair-input models score document pairs but are limited to late ranking because exhaustive pair scoring is not scalable.
- Generative-likelihood approaches rank documents by estimating the likelihood of generating the query from each candidate document.
- Dual encoders and late-interaction encoders prioritize efficiency for retrieval over corpora containing hundreds of millions of documents.
- Fine-tuning with ranking losses: Pointwise losses score query-document pairs independently, whereas listwise losses use one positive and multiple negatives to compute a softmax loss over ranking scores.
- Fine-tuning with ranking losses: This work focuses on reranking models rather than retriever models fine-tuned with pairwise or softmax losses.
3 Preliminaries
The paper defines ranking as scoring candidate documents for each query and sorting them to optimize ranking metrics. It also reviews T5’s encoder-decoder generation process and its component transformations.
- Problem definition: For each query, a candidate-document list and nonnegative relevance labels are provided as inputs to the formal ranking task.
- Problem definition: A ranking model maps each query-document pair to a real-valued score, after which documents are sorted to optimize ranking metrics.
- Problem definition: Unlike retrieval over millions of corpus documents, the ranking task usually scores a small candidate list and can use a high-capacity cross-attention model.
- T5: T5 is a text-to-text pretrained generative language model with an encoder-decoder Transformer architecture that autoregressively outputs token sequences.
- T5: During decoding, T5 receives an input sequence and previously generated tokens, producing a vocabulary-sized probability vector for the next token.
- T5: T5’s encoder produces embedding sequences, its decoder produces an embedding from encoder outputs and prior tokens, Dense projects to logits, and Softmax normalizes probabilities.
4 RankT5 Modeling
RankT5 adapts T5 to output numerical ranking scores directly and supports pointwise, pairwise, and listwise ranking losses. It studies encoder-decoder and encoder-only structures for directly optimizing ranking performance.
- T5 ranking is challenging because ranking requires numerical query-document scores, unlike the token or token-sequence outputs of typical T5 tasks.
- RankT5 directly outputs numerical ranking scores instead of deriving them through postprocessing after text-generation or classification fine-tuning.
- Encoder-decoder (EncDec): The encoder-decoder variant uses T5 decoder logits and an unused vocabulary token’s unnormalized logit as the ranking score.
- Encoder-only (Enc): The encoder-only variant removes autoregressive decoding, pools encoder embeddings, and projects the pooled vector directly to a ranking score.
- Ranking losses: RankT5 trains on predicted score lists using pointwise cross entropy, pairwise logistic loss, listwise softmax cross entropy, or listwise Poly-1 loss.
5 Experiment Setup
Experiments use MS MARCO and Natural Questions passage-ranking data with top-1000 retrieved candidates, while training samples lists of up to 36 documents. Evaluation uses MRR@10 and NDCG@5/10.
- MS MARCO: MS MARCO contains about 530,000 training queries, 6,800 development queries, and more than 8.8 million passages.
- A dual-encoder retriever supplies the top-1000 candidate passages for each query in both data sets.
- Natural Questions (NQ): Natural Questions contains more than 50,000 training queries and 8,000 development queries with binary relevance labels.
- Training data construction: Training selects one relevant document and uniformly samples up to 35 nonrelevant documents, producing lists of at most 36 documents.
- Evaluation: Evaluation ranks the top-1000 retrieved documents using MRR@10 and NDCG@5 and NDCG@10.
- Implementation: The experiments use pretrained T5 checkpoints, default to T5-Large, and implement the system with T5X and Rax in JAX.
6 Results
RankT5 models improve text-ranking performance across datasets, with listwise losses generally outperforming pointwise and pairwise alternatives. Performance also depends on model size and training-list size, while Softmax improves out-of-domain generalization.
- Overall comparison: RankT5 improves MS MARCO metrics by more than +1.8% and NQ metrics by +2.8% in MRR@10 and +2.0% in NDCG@5 and NDCG@10 over monoT5.These results support optimizing T5 ranking models with ranking losses rather than classification losses.
- Loss comparison: Listwise losses such as Softmax and Poly1 consistently outperform PointCE and Pair on both datasets.Poly1 significantly outperforms Softmax for the encoder-decoder model on MS MARCO and the encoder-only model on NQ across all metrics.
- Model structure: The encoder-decoder and encoder-only structures have similar performance when trained with the same ranking loss, with no consistent winner.The authors suggest the decoder may be less important when sufficient training data is available.
- Model size comparison: Larger T5 models consistently improve performance, with T5-3B outperforming T5-Base on NQ by nearly +7% in MRR@10.Figure 2 compares RankT5-EncDec models trained with Softmax and PointCE across Base, Large, and 3B checkpoints.
- Training list size: On both datasets, larger training lists improve performance; NQ appears to saturate above list size 30, while MS MARCO shows a continued uptrend.Training lists contain one positive and m −1 negatives, with tested sizes from 5 to 36; Softmax needs roughly 20 to 30 documents to beat PointCE.
7 Conclusion
RankT5 uses encoder-decoder and encoder-only T5 structures to output numerical ranking scores and support ranking-loss fine-tuning. Additional experiments examine target tokens and pooling strategies.
- 7 Conclusion: RankT5 introduces encoder-decoder and encoder-only structures that output numerical scores and support fine-tuning with ranking losses.The encoder-decoder uses a target token's unnormalized logit, while the encoder-only structure uses pooled encoder representations.
- 7 Conclusion: RankT5 improves ranking performance on MS MARCO and NQ, with improvement maintained in zero-shot evaluation on out-of-domain data sets.The conclusion attributes these experiments to ranking-loss fine-tuning of T5 models.
- A Choice of Target Tokens and Input Sequence: The encoder-decoder model uses a target token and its unnormalized logit as the ranking score.Experiments compare different target tokens and input formats under Softmax loss on MS MARCO.
- A Choice of Target Tokens and Input Sequence: Target-token choice and adding a “Relevant:” postfix do not seem to affect performance with sufficient training data.The observed insensitivity aligns with prior observations cited by the paper.
- B Pooling Strategies Comparison for RankT5-Enc: Different pooling strategies do not generate much different results for RankT5-Enc under Softmax loss on MS MARCO.The paper attributes this partly to transformer attention and sufficient training data.
C Ranking Performances with Additional Metrics
Additional metrics evaluate RankT5 beyond top-position precision, and the results preserve its advantage over other baselines.
- C Ranking Performances with Additional Metrics: NDCG, MAP, and Recall@5 are reported to assess ranking quality beyond metrics emphasizing top positions.These metrics also penalize ranking relevant documents too low in the list.
- C Ranking Performances with Additional Metrics: RankT5 models significantly outperform other baselines on the additional metrics.The conclusion is reported as similar to Table 1.
- C Ranking Performances with Additional Metrics: Softmax- and Poly1-fine-tuned models outperform models fine-tuned with pointwise or pairwise losses.The paper notes that both data sets have very few relevant documents per query; over 94% of dev queries have only one labeled relevant document.
D Additional Experiments on Model Size Comparison
Model-size experiments compare RankT5-Enc systems trained with Softmax and PointCE losses using MRR@10. Larger models perform better while the loss gap remains similar.
- D Additional Experiments on Model Size Comparison: RankT5-Enc models are compared across Base, Large, and 3B T5 sizes using Softmax and PointCE losses.Performance is plotted with MRR@10 and 95% confidence intervals.
- D Additional Experiments on Model Size Comparison: Performance increases with larger model sizes, while the gap between Softmax and PointCE remains similar.The results are described as similar to the RankT5-EncDec model-size results.
E Additional Zero-Shot Experiments
Zero-shot experiments evaluate RankT5-Enc across model sizes and against monoT5. Larger models perform better on average, and Softmax-trained RankT5 outperforms monoT5 on average.
- E Additional Zero-Shot Experiments: RankT5-Enc zero-shot performance is compared across Base, Large, and 3B models fine-tuned with Softmax on MS MARCO.The comparison uses NDCG@10 for the model-size experiments.
- E Additional Zero-Shot Experiments: Larger RankT5-Enc models perform better on average on out-of-domain data sets in the zero-shot setting.This conclusion is drawn from the model-size comparison.
- E Additional Zero-Shot Experiments: RankT5 fine-tuned with Softmax outperforms monoT5 on average at both compared model sizes.The comparison covers Base and 3B because those are the sizes available for both models.
F Limitations
The work is limited by binary relevance labels in both datasets and by its exclusive focus on T5; extending the approach remains open for other model structures.
- Both datasets use only binary relevance labels, leaving graded-relevance ranking datasets outside the evaluated setting.The authors note that different fine-tuning strategies might be needed for optimal performance on graded-relevance datasets.
- The study focuses exclusively on T5 rather than other sequence-to-sequence language models.The authors specifically identify GPT-2 as having a different structure because it lacks an encoder.
- Applying the proposed techniques to models with different structures remains an open direction.