PaperlayerView exact revision on arXiv

Source-linked AI summary

DecisionRank: On-Policy Distillation for Decision-Model Rerankers

Yanran Wu, Dian Yu, Chongyang Bai, Jieyi Zhang, Rui Liu, Tianyi Chen, Shike Mei, Hang Cui, Frank Shyu, Sandeep Pandey, Luke Simon, Chunwei Liu, Xi Liu

arXiv:2610.42128v1cs.CL

TL;DR

Generative teachers express ranking preferences through sequential document choices, while decision rerankers rank candidates from first-token scores. DecisionRank combines offline ranking supervision with item-space on-policy feedback on the student’s current ranking. It improves retrieval effectiveness across general and reasoning-intensive benchmarks while retaining faster inference than generative listwise rerankers.

  • Problem

    Transferring a generative teacher’s sequential ranking preferences to a single-token reranker is challenging because its first-token scores represent the whole ranking in one pass.

    Source: §1 · p. 2

  • Method

    DecisionRank combines offline teacher rankings with item-space on-policy distillation, using conditional teacher feedback on the student’s current ranking to supervise its first-token scores.

    Source: §1 · pp. 2–3

  • Results

    DecisionRank consistently improves ranking effectiveness over offline RankNet across four benchmarks and outperforms alternative training objectives; its 8B student runs 24.5× faster than DirectRank-8B on TREC DL.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DecisionRank distills sequential ranking decisions into single-token scoring that ranks the full candidate set without autoregressive generation.

    Source: §5 · p. 12

Abstract

from arXiv · show

Decision models target structured tasks by directly predicting choices or scores, avoiding autoregressive decoding latency. However, transferring a generative LLM's ranking knowledge to a decision model remains challenging: the teacher expresses preferences through a ranked list of document ID sequences, while the student determines the ranking using scores from the first-token position. To bridge this gap, we introduce DecisionRank, a decision model trained through item-space on-policy distillation (OPD). Item-space OPD constructs the student's rollout by sorting its document scores into a ranked sequence. A generative teacher takes the resulting ranking list as inputs and provides conditional feedback to directly supervise those scores. Combined with offline ranking supervision, DecisionRank achieves state-of-the-art performance on both general and reasoning-intensive retrieval benchmarks such as TREC DL, BEIR, and BRIGHT. Across four benchmarks, item-space OPD consistently outperforms alternative training objectives. Even at 1.7B parameters, DecisionRank surpasses its 27B generative teacher on TREC DL19--23. At 8B, it outperforms its generative listwise counterpart while running $24.5\times$ faster on TREC DL and $16.7\times$ faster on BRIGHT.

1 Introduction

DecisionRank transfers a generative teacher’s sequential ranking preferences to a single-pass decision model by combining offline rankings with conditional feedback on the student’s current ranking. It improves effectiveness across general and reasoning-intensive retrieval benchmarks while retaining lower inference latency than generative listwise rerankers.

1 Introduction

  • Single-token rerankers avoid the multi-step decoding latency of generative listwise methods, but transferring a teacher’s ranking preferences to first-token scores remains challenging.Fixed offline rankings cannot adapt to the student’s evolving predictions, while sequential teacher feedback does not directly match single-pass score-based ranking.Source: §1 · pp. 1–2
  • Item-space on-policy distillation constructs prefixes from the student’s current top-K ranking and uses conditional teacher feedback to supervise the first-token scores used at inference.The on-policy objective is jointly optimized with offline RankNet supervision from teacher-generated rankings.Source: §1 · pp. 2–3
  • All three distilled students (1.7B, 4B, and 8B) outperform the 27B generative teacher on TREC DL19–23, and the 8B student surpasses DirectRank-8B while running 24.5× faster on TREC DL.Under the same input-length settings, DecisionRank also improves over offline RankNet across all four benchmarks and outperforms the alternative training objectives reported.Source: §1 · p. 3
  • DecisionRank shows consistent improvements over zero-shot students across all four benchmarks and generalizes across teacher families and student sizes.The benchmarks cover both general and reasoning-intensive retrieval tasks.Source: §1 · p. 3

2 Method

DecisionRank ranks candidates using first-token scores, then trains those scores with student-conditioned teacher feedback and offline teacher rankings. The method combines conditional guidance on selected candidates with global ordering supervision and uses the same score-based ranking at inference.

2 Method

  • DecisionRank combines first-token ranking, on-policy conditional feedback, and offline supervision to train a decision model for document reranking.Both teacher and student choice distributions are aligned over candidate items, with the teacher remaining frozen.Source: §2 · p. 3

2.1 First-Token Ranking and Item-Space Alignment

  • Each candidate receives a distinct single-token document ID, and sorting its first-output-position logit yields the student’s ranking.The teacher and student predictions are aligned by candidate identity, so their token representations need not match.Source: §2.1 · pp. 3–4

2.2 On-Policy Conditional Feedback

  • The student’s current top-K ranking defines selection prefixes, and the teacher supplies conditional choices over remaining candidates at those states.The conditional distributions are extracted in one teacher-forced pass; reverse-KL feedback updates the first-token scores as the student’s ranking is recomputed.Source: §2.2 · p. 4

2.3 Offline Supervision and Joint Training

  • Offline RankNet trains global ordering preferences from teacher rankings, while on-policy distillation adds conditional feedback on the student’s top-K selections.The objectives jointly optimize the first-token scores used to rank the full candidate set in a single inference pass, without autoregressive generation.Source: §2.3 · pp. 4–5

3 Experiments

Experiments evaluate DecisionRank across four general and reasoning-intensive retrieval benchmarks, comparing ranking quality, inference efficiency, calibration, training objectives, supervision combinations, and teacher–student generalization. The results show strong ranking performance and efficiency, consistent gains from item-space OPD and joint supervision, and generalization across teacher families and student sizes.

3.1.1 Training / 3.1.2 Evaluation

  • Training uses 10,000 MS MARCO queries and 6,000 synthetic ReasonEmbed queries; evaluation covers TREC DL, BEIR, BRIGHT, and R2MED using top-100 candidate sets.Students are Qwen3 models at 1.7B, 4B, and 8B; the main teacher is frozen Qwen3.8-27B, with Gemma-4-31B and ReasonRank-32B used for generalization tests.Source: §3.1.1 · p. 5, §3.1.2 · p. 5

3.2.1 Ranking Effectiveness

  • DecisionRank students outperform compared baselines on TREC DL and BEIR; on reasoning-intensive retrieval, the 8B student leads BRIGHT while scoring 43.55 on R2MED versus 44.50 for listwise DirectRank.The 8B student improves over listwise DirectRank by 1.42 points on TREC DL and 2.20 on BEIR; the 1.7B student exceeds every compared baseline on both general benchmarks.Source: §3.2.1 · p. 7

3.2.2 Inference Efficiency

  • All three students form the local-model effectiveness–latency Pareto frontier on both benchmarks; the 8B student is 24.5× faster than listwise DirectRank on TREC DL and 16.7× faster on BRIGHT.The 1.7B student also has higher nDCG@10 and lower latency than Jev in both modes on TREC DL, and the lowest latency on both benchmarks.Source: §3.2.2 · p. 7, Figure 3 · p. 7

3.2.3 Score Calibration

  • DecisionRank students have lower ECE than their teachers on TREC DL and BRIGHT, and BRIGHT calibration is comparable to Jev without a dedicated calibration objective.Figure 4 reports ECE ×100 over all evaluation queries, with lower values indicating better calibration.Source: §3.2.3 · p. 8, Figure 4 · p. 8

3.3 RQ2: Which training objective works best for ranking with first-token logits?

  • Item-space OPD is the only tested objective that consistently improves RankNet across all four benchmarks, whereas offline KL and autoregressive objectives show benchmark-specific trade-offs.Item-space OPD uses the student’s current top-K ranking as teacher-forced prefixes and aligns conditional teacher and student distributions over candidate IDs.Source: §3.3 · pp. 9–10

3.4 RQ3: Does combining offline and on-policy supervision improve ranking?

  • Joint RankNet and Item-space OPD training achieves the highest student score on all four benchmarks and improves over either supervision alone.The results support complementary roles: offline rankings provide a consistent target, while on-policy feedback refines scores as the student ranking evolves.Source: §3.4 · p. 10, Table 3 · p. 10
  • Joint training improves over RankNet in every matched-seed comparison across 4B and 8B students on TREC DL and BRIGHT.Table 4 reports benchmark-average nDCG@10 ×100 as mean ± standard deviation across seeds 42, 43, and 44.Source: §3.4 · pp. 10–11, Table 4 · p. 11

3.5 RQ4: Does our method generalize across teachers and student sizes?

  • Distillation generalizes across Gemma, Qwen, and ReasonRank teachers: each improves all three student sizes over their zero-shot counterparts on all four benchmarks.Table 5 reports benchmark-average nDCG@10 ×100 for students trained with RankNet and Item-space OPD.Source: §3.5 · p. 11, Table 5 · p. 11

4 Related Work

Prior efficient rerankers reduce the cost of generative listwise ranking through grouped comparisons or first-token scoring. DecisionRank applies on-policy teacher feedback to first-token scores rather than relying only on static rankings or autoregressive student sequences.

4.1 Efficient LLM-based Reranking

  • DecisionRank builds on first-token ranking and learns its scores from a generative teacher.FIRST sorts first-token document-ID logits, while generative listwise rerankers generate rankings autoregressively.Source: §4.1 · pp. 11–12

4.2 Knowledge Distillation for Reranking

  • DecisionRank applies on-policy teacher feedback directly to first-token ranking scores, unlike prior offline ranking distillation and autoregressive on-policy approaches.Prior approaches include training on teacher-generated rankings and feedback on student-generated sequences.Source: §4.2 · p. 12

5 Conclusion

DecisionRank combines offline ranking supervision with item-space on-policy distillation, using conditional teacher feedback on the student's current ranking to train inference-time first-token scores. The conclusion reports improvements across general and reasoning-intensive retrieval and lower latency than a generative listwise counterpart on TREC DL and BRIGHT.

5 Conclusion

  • DecisionRank combines offline ranking supervision with item-space on-policy distillation to train first-token scores using conditional feedback on the student's current ranking.The trained scores are those used at inference.Source: §5 · p. 12
  • DecisionRank improves ranking across general and reasoning-intensive retrieval and generalizes across teacher families and student sizes.The conclusion reports these patterns without specifying values in the supplied passages.Source: §5 · p. 12
  • On TREC DL and BRIGHT, the 8B DecisionRank model outperforms its generative listwise counterpart with substantially lower latency.Source: §5 · p. 12

Training.

Training combines on-policy and offline ranking supervision, while teacher configurations and inference settings vary by model and benchmark. The appendix also gives abbreviated examples of the ranking prompts and Jev API requests.

Training.

  • The joint objective uses K = 20 student top candidates, supervises steps 2–19, and sets λ = 0.1; both losses update first-token scores.The OPD-only configuration uses loss weight 1 and supervises steps 1–19.Source: Training · p. 16
  • Teacher rankings and reference scores use sliding windows of size 20 and stride 10, with thinking disabled; on-policy feedback settings vary by teacher.On-policy feedback disables thinking for Gemma-4-31B and Qwen3.8-27B but enables it for ReasonRank-32B.Source: Training · p. 16
  • At inference, students score the full candidate set in one forward pass, with passages truncated to 128 tokens for TREC DL and 384 for BEIR, BRIGHT, and R2MED.Source: Training · p. 16
  • Jev Score assigns expected relevance from four ordered grades per query–document pair, while Choice ranks all 100 candidates by returned probabilities.Both modes use student passage-length limits and retrieval-order tie breaks; passages are shortened uniformly when needed for Choice.Source: Training · p. 16
  • Local-model latency tests rerank 100 candidates on one B200 worker per method, using median query latency across eight workers.Listwise timings include model calls and decoding; student timings cover the forward pass but exclude score sorting.Source: Training · p. 16

A.1 Prompt Examples

  • The abbreviated prompt examples show query-and-passage ranking instructions, first-document-ID logits as student scores, and teacher-forced evaluation of student selections.The Jev examples distinguish per-passage Score requests from a single Choice request containing all candidates.Source: §A.1 · p. 17

B Detailed Results under Shared Passage-Length Limits

Tables 6–9 present reranker comparisons under shared passage-length limits. Their captions specify benchmark, candidate-generation, model, and nDCG@10 reporting scopes, but the supplied passages do not include table values.

B Detailed Results under Shared Passage-Length Limits

  • Tables 6–9 use shared passage limits of 128 tokens for TREC DL and 384 for BEIR, BRIGHT, and R2MED.They also include model backbones and references, plus additional Qwen3-Reranker and TFRank configurations below 4B parameters.Source: B Detailed Results under Shared… · p. 18
  • Table 6 compares rerankers on BM25 top-100 TREC DL candidates, while Table 7 covers BM25 top-100 candidates across displayed BEIR datasets.Both captions specify nDCG@10 ×100 and an average across their reported years or datasets.Source: Table 6 · p. 18, Table 7 · p. 19
  • Table 8 covers twelve BRIGHT tasks using ReasonIR-8B candidates and original queries; Table 9 covers eight R2MED datasets using E5-Mistral-7B-Instruct candidates.Both captions report nDCG@10 ×100 and averages across their tasks or datasets.Source: Table 8 · p. 20, Table 9 · p. 21

C.1 Results across Training Seeds

The section reports individual runs for Qwen3-4B and Qwen3-8B with a Qwen3.8-27B teacher, using three seeds and paired significance tests.

C.1 Results across Training Seeds

  • Scores were averaged across seeds 42, 43, and 44 per query before paired significance tests compared joint training with RankNet.The runs use shared training data and evaluation settings; benchmark-average nDCG@10 ×100 is reported.Source: §C.1 · p. 21

C.2 Pairwise Score Calibration

Pairwise calibration evaluates whether confidence tracks the correctness of document preferences. DecisionRank students show lower ECE and higher pairwise accuracy than their corresponding teachers on TREC DL and BRIGHT.

Pairwise predictions and confidence

  • Pairs are formed from same-query documents with different annotated relevance grades, with unjudged documents treated as grade 0.A pairwise prediction is correct when its preferred document agrees with the relevance judgments.Source: §C.2 · p. 21

DecisionRank students

  • For DecisionRank, the sigmoid of the first-token logit difference gives pairwise preference probability, while the absolute difference sets confidence.A larger score gap yields a more confident preference.Source: §C.2 · p. 22

Generative teachers

  • For generative teachers, the initial document-ID probabilities are normalized into pairwise preferences, with log-probability differences defining the score gap.The probabilities represent the teacher’s preference at its initial selection.Source: §C.2 · p. 22

Pointwise rerankers and Jev

  • DirectRank and ERank are pointwise rerankers that score query–document pairs separately and use score differences for pairwise probabilities and confidence.Jev instead derives pairwise preferences from independent predicted grade distributions and uses expected grades as ranking scores.Source: §C.2 · pp. 22–23

Binning and ECE

  • ECE averages the absolute gaps between mean confidence and empirical accuracy across bins, weighted by each bin’s pair count.Valid pairs are pooled across queries within each benchmark, so queries with more eligible pairs contribute more weight.Source: §C.2 · p. 23

Full-benchmark score calibration

  • Both 4B and 8B students have lower ECE and higher pairwise accuracy than their respective teachers on TREC DL and BRIGHT.Comparisons cover Qwen3.8-27B and Gemma-4-31B teachers; calibration gains come from ranking supervision alone, without a dedicated calibration objective.Source: §C.2 · p. 23

(a) TREC DL (b) BRIGHT

On TREC DL and BRIGHT, 4B and 8B DecisionRank students achieve better pairwise calibration than their corresponding teachers. Figures 5–8 compare calibration and reliability across teacher families and student sizes.

(a) TREC DL (b) BRIGHT

  • Figures 5 and 7 plot calibration curves for 8B and 4B students against their respective teachers, with dotted curves marking ideal calibration.The figures report ECE (×100) as teacher → student.Source: Figure 5 · p. 23, Figure 7 · p. 24
  • Figures 6 and 8 show mean predicted confidence against empirical accuracy within score-gap bins, with the diagonal indicating ideal calibration.The 4B figures use the same binning, axis ranges, and layout as the 8B plots for direct size comparison.Source: (a) TREC DL (b) BRIGHT · p. 23, Figure 6 · p. 24, Figure 8 · p. 25
  • Both 4B and 8B DecisionRank students achieve lower aggregate ECE and higher pairwise accuracy than their corresponding teachers on TREC DL and BRIGHT.This pattern holds for students trained with either Qwen3.8-27B or Gemma-4-31B, using ranking supervision alone without a dedicated calibration objective.Source: (a) TREC DL (b) BRIGHT · pp. 23, 25

Related papers

Loading 2610.42128v1…