PaperlayerView exact revision on arXiv

Source-linked AI summary

System One models Are Not Better Rerankers

Michael Dinzinger, Kanishka Ghosh Dastidar, Michael Granitzer

arXiv:2610.32318v1cs.CLcs.IR

TL;DR

Prior reports do not isolate whether decision fine-tuning improves reranking over reranker fine-tuning of the same backbone, and calibration matters for uses of scores on an absolute scale. This paper compares these approaches across Qwen models and retrieval collections, finding that decision fine-tuning does not make the better reranker.

  • Problem

    Prior comparisons use decision models and rerankers built on different models, leaving the same-backbone fine-tuning comparison unresolved; calibration also matters when scores are used on an absolute scale.

    Source: §1 · p. 1

  • Method

    The study compares untuned models, decision fine-tunes, and reranker fine-tunes across six Qwen backbones and ten TREC DL and BEIR collections, also testing an alternative question wording.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    From 0.6B to 9B, reranker fine-tunes beat decision fine-tunes by 5 to 29 points nDCG@10, while they tie at 27B; up to 9B, decision fine-tuning adds nothing over the untuned model.

    Source: §1 · p. 2, §8 · p. 7

  • Takeaways & Limitations

    For reranking, fine-tune a reranker: up to 9B, calibration does not make up for its worse ranking at a fixed threshold.

    Source: §8 · p. 7

    The controlled reranker comparison relies on one training run and one recipe at 9B and 27B, without hyperparameter search.

    Source: §7 · p. 7

Abstract

from arXiv · show

System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.

1 Introduction

Decision models are being used as rerankers, but prior comparisons do not isolate the effect of decision fine-tuning from the underlying model. This study compares decision and reranker fine-tuning on the same Qwen backbones and finds that decision fine-tuning does not make the better reranker.

1 Introduction

  • Earlier comparisons of decision models and rerankers use different underlying models, leaving the effect of decision fine-tuning unresolved.The study compares decision fine-tuning with plain reranker fine-tuning of the same backbone.Source: §1 · pp. 1–2
  • Calibration is assessed separately because ranking depends on score order, whereas practical uses such as passage filtering read scores on an absolute scale.The study tests whether calibration holds for relevance and helps with such uses.Source: §1 · pp. 1–2
  • The experiments compare six Qwen backbones from 0.6B to 27B across ten TREC DL and BEIR collections, including zero-shot prompting and two in-house reranker fine-tunes.They also test an alternative question wording from S1Rank.Source: §1 · p. 2
  • From 0.6B to 9B, reranker fine-tunes win by 5 to 29 points nDCG@10, while the methods tie at 27B.Decision fine-tuning is never the better reranker; its gap persists on zero-shot collections, and up to 9B it adds nothing over asking the untuned model.Source: §1 · p. 2
  • Decision models from 4B on are better calibrated than published reranker fine-tunes when asked whether a passage answers the query.With S1Rank’s looser wording, they lose this calibration advantage while ranking changes little; at a fixed threshold, reranker fine-tunes retain better passages up to 9B.Source: §1 · p. 2

2 Related Work

Prior work establishes neural cross-encoders and open LLM rerankers as ranking approaches, while reports on decision models have not isolated their fine-tuning effect against rerankers built on the same backbone. Calibration research also distinguishes ranking order from the absolute score values used for thresholding.

LLM-based rerankers

  • Open LLMs fine-tuned for ranking are described as the current state of the art, building on earlier cross-encoder rerankers [17] [38] [39].Earlier neural rerankers used encoder or sequence-to-sequence models [19] [20].Source: §2 · p. 2
  • Prompted LLM ranking can work without ranking-specific fine-tuning, but its effectiveness is sensitive to prompt wording [26] [40] [30].Other approaches generate ranked lists [24, 31] or let passages attend to each other [28].Source: §2 · p. 2

Decision models in retrieval

  • Reports on decision models in retrieval compare Jev with rerankers built on other models, not decision and reranker fine-tunes of the same backbone.These reports include production search and relevance gating; none tests whether calibration pays off.Source: §2 · p. 2

Calibration of relevance scores

  • Ranking losses can yield well-ordered but uncalibrated scores because they are invariant to shifting a query’s scores.Calibration matters when scores are read on an absolute scale, such as cutting off a ranked list [2].Source: §2 · p. 2

3 Decision Models as Rerankers

A decision model returns probabilities for answers to questions about an input; reranking orders retrieved passages by the probability that each answers the query. The paper compares this scoring approach with untuned and fine-tuned variants that also return relevance probabilities.

Decision Models as Rerankers

  • Reranking orders a first-stage retriever’s passages by each passage’s probability of answering the query.For binary questions, the model returns the probability of “yes.”Source: §3 · p. 2
  • In isolated mode, each score conditions on the query and one passage; in joint mode, it can also depend on the other candidates.Both modes score passages individually rather than producing a ranked list directly.Source: §3 · p. 2
  • The comparison includes untuned chat models, decision fine-tunes, and reranker fine-tunes, all returning a relevance probability per query–passage pair.Untuned and decision models use the probability of “yes” to the decision question; reranker fine-tunes use their own prompt and probability of “yes” against “no.”Source: §3 · pp. 2–3
  • Table 1 compares the same-backbone variants on nDCG@10 over BM25 top-100 candidates, including all-collection and zero-shot means.Its caption explains the model markers and significance-test symbols.Source: Table 1 · p. 3

4 Experimental Setup

The experiments compare six Qwen backbones and multiple decision and reranker fine-tunes, using a fixed relevance question and shared BM25 candidates. Ranking, calibration, thresholding, and count estimation are evaluated with specified metrics and test procedures.

Backbones and variants

  • The study covers six Qwen backbones and decision fine-tunes, alongside Qwen3-Reranker, Prism, and two in-house reranker fine-tunes.The in-house models are trained at 9B and 27B; Qwen3-Reranker and Prism use the same pretrained checkpoints as Kev at their respective sizes.Source: §4 · p. 3

Our reranker fine-tunes

  • The in-house reranker recipe uses one MS MARCO epoch with LoRA, hard negatives, an InfoNCE ranking loss, and a yes/no classification loss.Training queries matching test or development topics are removed; the checkpoint is selected using 200 MS MARCO development queries, with no decision data or hyperparameter search.Source: §4 · p. 3

The question

  • The fixed question asks whether a document answers the query, matching the relevance labels used for evaluation.The alternative S1Rank wording is also tested; reranker fine-tunes retain their own prompts.Source: §4 · p. 4

Collections and candidates

  • Candidates are BM25 top-100 results from TREC DL 2019–2023 and five BEIR collections, with TREC-COVID, Touché, and DBPedia designated zero-shot.The comparison also reports other rerankers on these candidates and includes published results where systems were not rerun.Source: §4 · p. 4, Table 2 · p. 4

Ranking evaluation

  • Ranking uses nDCG@10, with paired query-level t-tests Holm-corrected across ten collections.Per backbone, the tests compare the reranker fine-tune with the strongest decision fine-tune, that decision model with the untuned model, and joint with isolated mode.Source: §4 · p. 4

Calibration and its uses

  • Calibration is measured on judged TREC DL pairs using ECE and Brier score, while thresholding and count estimation test practical uses of probabilities.The fixed cutoff is p≥0.5; count estimation sums probabilities over judged passages, and reranker fine-tunes are also calibrated post hoc using MS MARCO development queries.Source: §4 · p. 4

Sanity checks

  • Sanity checks reproduce BM25, monoT5-3B, and RankZephyr results within 0.2 nDCG@10 points.Jev in joint mode reproduces S1Rank’s results within 0.5 points on four of five collections, and is 1.5 points lower on TREC-COVID.Source: §4 · p. 4

5 RQ1: Decision vs. Reranker Fine-Tuning

Across Qwen backbones from 0.6B to 9B, reranker fine-tuning consistently outperforms decision fine-tuning, while the two tie at 27B. Up to 9B, decision fine-tuning adds little over asking the untuned model.

5 RQ1: Decision vs. Reranker Fine-Tuning

  • 5.2 to 28.8 points nDCG@10: reranker fine-tunes beat the better decision-fine-tune mode on every 0.6B–9B backbone, with significant gains on 48 of 60 collection pairs.The advantage also holds on zero-shot collections, by 5.4 to 23.9 points.Source: §5 · p. 4
  • At 27B, reranker and decision fine-tunes tie overall: the matched reranker scores 59.6 versus Kev-27B’s 60.1, and 54.9 versus 55.0 on zero-shot collections.They differ significantly on one collection only, DBPedia, where the reranker is better.Source: §5 · p. 5
  • −2.5 to +0.7 points: up to 9B, decision fine-tunes stay near their untuned backbones, while reranker fine-tuning adds 5.9 to 29.5 points.At 27B, Kev-27B adds 3.7 points to the untuned model and the reranker fine-tune adds 3.2; Clef adds nothing.Source: §5 · p. 5
  • A fine-tune of a strong open backbone reaches Jev’s 59.1 nDCG@10, with or without decision data; Laya’s 43.5 does not improve on BM25’s 43.2.Jev is comparable to 8B–9B reranker fine-tunes at 59.0–59.4 and the Set-Encoder at 59.0.Source: §5 · p. 5
  • About one point: joint scoring improves Kev-27B and Clef, but costs Kev-0.6B 12.2 points and Kev-8B 13.0.A 100-passage joint state is about 13k tokens on TREC DL and 40k or more on BEIR, compared with reported small-Kev training states of at most 7,552 tokens.Source: §5 · p. 5

6 RQ2: Are Decision Models Better Calibrated?

Decision models’ calibration advantage depends on the question used, even when rankings change little. The section also tests whether calibrated scores help select passages or estimate relevant-passage counts.

6 RQ2: Are Decision Models Better Calibrated?

  • From 4B on, decision fine-tunes are better calibrated than published reranker fine-tunes when asked whether a passage answers the query; on smaller backbones, they are not.For example, Kev-8B has ECE 0.113 versus Qwen3-Reranker-8B’s 0.257; the 9B reranker fine-tune is as calibrated as Clef-Flash (0.111 vs. 0.112).Source: §6 · p. 5
  • −2.5 to +2.6 points: changing the question shifts decision fine-tunes’ mean nDCG@10 little, but S1Rank’s wording removes their calibration edge over rerankers.With S1Rank’s question, almost every model is worse calibrated, often by a factor of two or three; one sentence of the prompt can change which model is better calibrated.Source: §6 · pp. 5–6
  • A question matching the labels’ relevance definition improves calibration with or without decision fine-tuning; users must express that definition in their wording.Untuned Qwen3-8B’s ECE changes from 0.383 with S1Rank’s question to 0.231 with the paper’s wording.Source: §6 · p. 6
  • 51.0 vs. 42.6 F1@.5: up to 9B, reranker fine-tunes keep the better passage set at threshold 0.5, despite often worse calibration.For decision models, 0.5 is close to each collection’s best threshold, with regret of 0.9 to 3.1 points; at 27B, better-calibrated Kev-27B keeps the better set (57.2 vs. 48.3).Source: §6 · p. 6
  • 5.8 passages: untuned Qwen3.8-27B has the lowest count error, while decision models from 4B on usually beat the median guess under the paper’s wording.With S1Rank’s wording, no decision model beats the median guess by more than a fraction of a passage.Source: §6 · p. 6
  • Temperature scaling lowers the worst rerankers’ ECE, but Platt scaling can fail at the fixed cutoff because MS MARCO dev has about one relevant passage per query versus many on TREC DL.Calibration holds only for the relevance notion and base rate on which it was fitted.Source: §6 · p. 6

7 Discussion and Limitations

The discussion attributes the small-backbone gap to the task-specific training of rerankers and cautions that the 27B comparison does not isolate why results differ. Calibration also depends on the relevance definition and how it is measured; the authors describe several limits on the comparison.

Why decision fine-tuning does not make a reranker

  • Up to 9B, decision fine-tuning transfers about as well as the untuned model’s instruction following, while reranker fine-tuning targets relevance with a ranking loss over hard negatives.The discussion notes that Kev’s training data contain no MS MARCO or BEIR; the performance gap also persists on collections neither fine-tune saw.Source: §7 · p. 6

Why 27B is different

  • At 27B, Kev is the only decision fine-tune that improves on its untuned model, but size alone cannot explain the difference from Clef.Kev-27B is a full fine-tune of a released chat model trained on longer states, unlike the smaller Kevs; the authors cannot tell which difference matters.Source: §7 · p. 6

What calibration means for relevance

  • Calibration depends on the relevance definition and base rate, and can shift with question wording, training labels, or the labels used to fit a scaling.The discussion says this matters for thresholds and counts, but not for ranking.Source: §7 · p. 6

Limitations

  • The comparison uses released models with differing training data and objectives; its own reranker fine-tunes were each trained once, with one recipe and no hyperparameter search.Calibration was measured only on TREC DL, at one relevance grade and threshold; the fixed question was not selected on development data, and only one alternative wording was tried.Source: §7 · p. 7

8 Conclusion

The conclusion finds that decision fine-tuning does not produce a better reranker than plain reranker fine-tuning on the same backbone. It also reports that calibration advantages vary with the question and do not overcome weaker ranking at a fixed threshold up to 9B.

8 Conclusion

  • 5 to 29 points nDCG@10: reranker fine-tuning beats decision fine-tuning from 0.6B to 9B, while the two tie at 27B.Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model.Source: §8 · p. 7
  • For a reranker, the authors’ conclusion is to fine-tune a reranker rather than rely on decision fine-tuning.This follows their reported ranking comparison on the same backbone.Source: §8 · p. 7
  • The calibration advantage of decision models depends on question wording: matching the relevance labels improves calibration, while a looser question removes their advantage over reranker fine-tunes.Up to 9B, reranker fine-tunes retain better passages at a fixed threshold.Source: §8 · p. 7

Related papers

Loading 2610.32318v1…