PaperlayerView exact revision on arXiv

Source-linked AI summary

Sliding Windows Are Not the End: Exploring Full Ranking with Long-Context Large Language Models

Wenhan Liu, Xinyu Ma, Yutao Zhu, Ziliang Zhao, Shuaiqiang Wang, Dawei Yin, Zhicheng Dou

arXiv:2412.14574v1cs.IRcs.CL

TL;DR

Sliding-window ranking can repeatedly evaluate passages and serialize inference, while full ranking with long-context models handles them in one step but has uncertain effectiveness and efficiency. The paper studies both strategies and proposes complete-label construction with an importance-aware loss for fine-tuning full ranking. Fine-tuned full ranking outperforms sliding windows, while zero-shot full ranking is more efficient but less effective.

  • Problem

    Sliding-window ranking repeatedly evaluates passages and serializes inference, while the effectiveness and efficiency of single-step full ranking with long-context models remain unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs full ranking labels with multi-pass sliding windows and fine-tunes full ranking models using an importance-aware loss that weights passage IDs by rank.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Results

    Zero-shot full ranking is more efficient but less effective than sliding windows, whereas fine-tuned full ranking outperforms them; on TREC DL19, it improves NDCG@10 by 2.2 and reduces latency by 29.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported comparisons show that full ranking can outperform sliding windows in supervised fine-tuning, while zero-shot full ranking has lower effectiveness.

    Source: §1 · p. 2

    The study does not evaluate larger open-source long-context models such as 30B- or 70B-parameter models, and does not design a specialized long-context model for ranking.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown exciting performance in listwise passage ranking. Due to the limited input length, existing methods often adopt the sliding window strategy. Such a strategy, though effective, is inefficient as it involves repetitive and serialized processing, which usually re-evaluates relevant passages multiple times. As a result, it incurs redundant API costs, which are proportional to the number of inference tokens. The development of long-context LLMs enables the full ranking of all passages within a single inference, avoiding redundant API costs. In this paper, we conduct a comprehensive study of long-context LLMs for ranking tasks in terms of efficiency and effectiveness. Surprisingly, our experiments reveal that full ranking with long-context LLMs can deliver superior performance in the supervised fine-tuning setting with a huge efficiency improvement. Furthermore, we identify two limitations of fine-tuning the full ranking model based on existing methods: (1) sliding window strategy fails to produce a full ranking list as a training label, and (2) the language modeling loss cannot emphasize top-ranked passage IDs in the label. To alleviate these issues, we propose a new complete listwise label construction approach and a novel importance-aware learning objective for full ranking. Experiments show the superior performance of our method over baselines. Our codes are available at \url{https://github.com/8421BCD/fullrank}.

1 Introduction

The introduction contrasts redundant, serial sliding-window ranking with single-step full ranking enabled by long-context models, then studies their efficiency and effectiveness and proposes training changes for full ranking.

1 Introduction

  • 2.2 NDCG@10 improvement and 29.3% lower latency are reported for the fine-tuned full ranking model versus the sliding-window model on TREC DL19.Figure 1 compares the models on NDCG@10 and per-query latency.Source: Figure 1 · p. 1, §1 · p. 2
  • Sliding windows repeatedly evaluate passages and serialize processing, while long-context full ranking handles all retrieved passages in one step but leaves its effectiveness and efficiency uncertain.Longer inputs may increase encoding time even as full ranking reduces redundant inference and API costs.Source: §1 · pp. 1–2
  • Zero-shot full ranking is more efficient but less effective than sliding windows, whereas fine-tuned full ranking outperforms sliding windows.The study compares both strategies across efficiency and effectiveness.Source: §1 · p. 2
  • Existing sliding-window training labels may guarantee only top-passage order, while equal loss across hundreds of IDs can let irrelevant IDs overwhelm relevant top-ranked IDs.This imbalance conflicts with ranking evaluation, which prioritizes top-ranked passages.Source: §1 · p. 2
  • The paper addresses the training limitations with complete listwise labels and an importance-aware objective for fine-tuning full ranking models.The authors report that their fine-tuned full ranking model outperforms previous state-of-the-art models and their fine-tuned sliding-window model.Source: §1 · p. 2

2 Listwise Ranking with LLMs

This section defines listwise passage ranking and contrasts full ranking with sliding windows, then describes the distillation-based supervised fine-tuning setup.

2 Listwise Ranking with LLMs

  • Listwise ranking takes a query and passage list as input and outputs a ranked sequence of passage IDs.The section considers full ranking and sliding-window strategies.Source: §2.1 · p. 2

2.1 Zero-shot Ranking

  • Full ranking processes the entire passage list at once; sliding windows move from the list’s end to its beginning using a window and step size.The reported setup uses 100 passages, a window size of 20, and a step size of 10.Source: §2.1 · pp. 2–3

2.2 Supervised Fine-tuning

  • Prior fine-tuning distills a teacher’s passage-ID sequence as the label and trains by standard language-modeling loss; Figure 2 previews multi-pass label construction and importance-aware weighting.The labels are primarily obtained by reranking BM25’s top-20 passages.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Fine-tuning Full Ranking Model

The section addresses two training limitations: one-pass windows do not guarantee a full ranking list, and equal ID penalties can overwhelm top-ranked IDs.

3 Fine-tuning Full Ranking Model

  • One-pass sliding windows guarantee only top-ranked passages, while equal penalties across imbalanced full-ranking labels can overwhelm the top-ranked IDs.The proposed training approach pairs multi-pass label generation with an importance-aware objective.Source: §3 · p. 3

3.1 Multi-pass Sliding Window Approach

  • The multi-pass sliding-window approach repeatedly reranks the remaining passages, collecting successive groups of ten until all 100 retrieved candidates are ordered.The first pass yields the top 10; the next pass reranks the remaining 90.Source: §3.1 · p. 3

3.2 Importance-Aware Learning Objective

  • The importance-aware loss weights passage IDs by rank so higher-ranked IDs contribute more, aligning training emphasis with their greater importance in ranking metrics.Tokens belonging to a passage ID share its weight; the relation operator “>” receives weight α, with α <= 1.Source: §3.2 · pp. 3–4

4 Experiments

The experiments find that supervised fine-tuning lets full ranking outperform sliding-window ranking on nearly all datasets, while zero-shot full ranking is generally less effective. Full ranking also reduces latency, and its latency advantage persists across passage-list sizes.

4.1 Setting

  • Evaluation reranks BM25 top-100 passages on TREC DL19/20 and eight BEIR datasets using NDCG@10; fine-tuning compares Mistral-based RankMistral100 and RankMistral20.GPT-4o-mini and GPT-4o provide the fine-tuning labels.Source: §4.1 · p. 4

4.2 Effectiveness Analysis

  • In zero-shot evaluation, full ranking underperforms sliding windows on nearly all datasets, although it wins on Touche for Mistral-v0.3, LLaMA-3.1, and GPT-4o-mini.GPT-4o has the best full-ranking performance, with average NDCG@10 of 71.99 on TREC and 50.78 on BEIR.Source: §4.2 · p. 5
  • Fine-tuned RankMistral100 outperforms RankMistral20 on nearly all datasets, with about 4-point and 2-point average improvements on TREC and BEIR, respectively, using GPT-4o-mini.The authors also report the same comparative conclusion with Qwen2.5-7B-Instruct.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 Efficiency Analysis

  • 29.9 seconds versus 3.8 seconds: for Signal top-10 output, sliding windows take about 8x as long as full ranking.Full ranking also has lower latency across long-context LLMs when ranking the full list.Source: Figure 3 · p. 6, §4.3 · p. 6

4.2 Effectiveness Analysis

  • With standard language-modeling loss, removing Lia lowers RankMistral20 and RankMistral100 by about 0.7 points on BEIR Avg.RankMistral100 still outperforms RankMistral20 on DL19, DL20, and BEIR Avg under that loss.Source: §4.2 · p. 6

4.4 Impact of Passage Number N

  • Across passage counts from 20 to 100, full ranking has lower latency, while sliding windows remain more effective; GPT-4o’s effectiveness gap is smaller than Mistral’s.Full-ranking latency grows almost linearly as passage count increases.Source: §4.4 · p. 7

N Model DL19 DL20 BEIR Avg

RankMistral100 generalizes across passage-list sizes, outperforming RankMistral20 at the tested counts. For top-100 passage ranking, full ranking also reduces API cost relative to sliding windows.

N Model DL19 DL20 BEIR Avg

  • RankMistral100 outperforms RankMistral20 across tested passage counts from 20 to 100 on DL19, DL20, and BEIR, despite training on lists of length 100.The results are reported in Table 2.Source: N Model DL19 DL20 BEIR Avg · p. 7, Table 2 · p. 7

4.5 API Cost Comparison

  • For top-100 passage ranking, full ranking reduces API cost by about 50% versus sliding windows; longer passages raise costs across datasets.Costs are calculated from input and output tokens using OpenAI’s official price.Source: §4.5 · p. 8

5 Related Work

Prior passage-ranking work includes pointwise, pairwise, and listwise approaches, with listwise methods directly ranking passage lists. Long-context LLMs offer a less explored setting for listwise ranking.

5.1 LLMs for Passage Ranking

  • Listwise methods directly rank passage lists and support multi-passage comparisons; prior work also includes pointwise and pairwise approaches.Examples include prompt-based ranking and distillation of proprietary-model ranking ability into smaller LLMs.Source: §5.1 · p. 8
  • Existing listwise methods use sliding windows because of input-length limits, bringing efficiency and cost problems; long-context LLM ranking had received little study.The paper presents its work as the first attempt to investigate this direction.Source: §5.1 · p. 8

5.2 Long Context Large Language Models

  • Long-context LLM development has expanded input lengths through approaches including efficient attention and positional interpolation.The cited examples include Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and GPT-4o.Source: §5.2 · p. 8

6 Conclusion

Full ranking is more efficient than sliding-window ranking, and after fine-tuning it also achieves better ranking effectiveness. The paper proposes multi-pass label construction and an importance-aware objective for fine-tuning.

6 Conclusion

  • Fine-tuned full ranking is more effective than sliding-window ranking, while full ranking is more efficient.The conclusion reports both comparisons without specifying quantitative values.Source: §6 · p. 8
  • The paper studies passage ranking with long-context LLMs.Source: §6 · p. 8
  • For fine-tuning full ranking, the paper proposes multi-pass sliding windows for label generation and an importance-aware learning objective for optimization.Experiments demonstrate the effectiveness of the proposed method.Source: §6 · p. 9

Limitations

The authors could not test larger open-source long-context models or develop a specialized long-context ranking model. The appendix prompt instructs the model to rank all supplied passages by relevance.

Limitations

  • The study does not test larger open-source long-context LLMs, such as models with 30B or 70B parameters.The authors identify the impact of model size on full-ranking effectiveness and efficiency as future research.Source: Limitations · p. 9
  • The authors did not design a specialized long-context LLM for ranking; they leave this as future work.The reported efficiency advantage is attributed to existing long-context LLMs.Source: Limitations · p. 9

A Listwise Ranking Prompt

  • The prompt asks RankLLM to return every supplied passage identifier in descending order of relevance, with no additional explanation.The input includes a query and numbered passages.Source: A Listwise Ranking Prompt · p. 12

B Training Configuration

The training configuration uses Mistral-7B-Instruct-v0.3 and teacher labels generated from 1,000 randomly sampled MS MARCO training queries. Both ranking models are fine-tuned for four epochs.

B Training Configuration

  • RankMistral20 and RankMistral100 are fine-tuned for 4 epochs with learning rate 5e-6 and batch size 1.Source: B Training Configuration · p. 12
  • The authors randomly sample 1k MS MARCO training queries to generate teacher labels; more queries did not yield better results in their experiments.The passage points to Figure 6 for the query-count result.Source: B Training Configuration · p. 12
  • Mistral-7B-Instruct-v0.3 is the fine-tuning backbone, and experiments use noisy embeddings, bfloat16 precision, and 4 A100-40G GPUs.Source: B Training Configuration · p. 12

C The Impact of Passage Order and Ranking Number

On TREC DL19, passage order affects RankMistral100 performance, while gains from increasing ranking number converge at 3 or 4. The reported teacher-label costs are higher for full ranking than for sliding-window fine-tuning.

C The Impact of Passage Order and Ranking Number

  • NDCG@10 on BEIR Avg does not improve when RankMistral100 training queries increase from 1000 to 1500.Source: Figure 6 · p. 12
  • Initial passage order significantly affects RankMistral100 performance on TREC DL19, and ranking-number gains begin to converge at 3 or 4.The passage compares random and reverse initial orders; Table 3 covers passage order and ranking number.Source: C The Impact of Passage Order and… · p. 12, Table 3 · p. 12

D API Cost

  • For 1k queries, sliding-window RankMistral20 teacher labels cost about $1.7 with GPT-4o-mini and $29 with GPT-4o; full-ranking labels cost around $15.7 and $261, respectively.The full-ranking labels are generated with the multi-pass sliding-window strategy.Source: Table 4 · p. 13, D API Cost · p. 13

C The Impact of Passage Order and Ranking Number

  • Table 5 reports NDCG@10 results on TREC and BEIR under different passage numbers, using labels generated by GPT-4o.Source: Table 5 · p. 13

E Baselines Details

The section lists four reranking baselines and describes sampling-strategy and full-ranking comparisons, including a reported advantage for RankQwen100 on most datasets.

E Baselines Details

  • The comparison baselines are monoBERT (Nogueira and Cho, 2019), monoT5 (Nogueira et al., 2020), RankVicuna (Pradeep et al., 2023a), and RankZephyr.They include cross-encoder and sequence-to-sequence rerankers, plus listwise models fine-tuned or distilled from GPT-generated rankings.Source: E Baselines Details · p. 13
  • Table 6 reports RankMistral20 NDCG@10 on TREC and BEIR for top-20 BM25 passages versus 20 randomly sampled passages from the top 100.Source: Table 6 · p. 14
  • RankQwen100 outperforms RankQwen20 on most datasets and achieves an average 1-point improvement on BEIR Avg.Both use Qwen2.5-7B-Instruct, with GPT-4o-generated training labels.Source: Table 7 · p. 14
Loading 2412.14574v1…