PaperlayerView exact revision on arXiv

Source-linked AI summary

Equal Ranking Quality, Different Decisions: Training Order-Consistent LLM Scorers

Markus Frohmann, Mahdiyar Alavi, Elizabeth Lingg, Navid Rekabsaz

arXiv:2608.26762v1cs.CLcs.IRcs.LG

TL;DR

LLM scorers can preserve ranking quality while changing threshold, reader, and preference decisions when candidate order changes. The paper compares serving-, label-, and training-time remedies, finding that OC-SFT attenuates order dependence without sacrificing ranking quality. It therefore argues that evaluations should report decision stability alongside ranking quality.

  • Problem

    Shared-prompt scorers are selected by ranking quality even though their scores determine decisions that can vary when candidates are reordered.

  • Method

    The paper measures consumer-output changes across permutations and trains OC-SFT to penalize the scorer’s own order residual.

  • Results

    OC-SFT holds ranking quality and leads the trained scorers on decision stability across passage reranking, multi-document QA, and response ranking.

  • Takeaways & Limitations

    Comparisons should report what thresholds retain and readers answer, not ranking quality alone.

  • Takeaways & Limitations

    Two evaluated legal collections are proprietary, so others cannot reproduce the full 18-collection reranking means.

Abstract

from arXiv · show

Rerankers, reward models and multi-document QA scorers score candidate documents or responses in one LLM prompt, so each score depends on their order. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or a preference model selects. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. A published reranker takes the highest retained-set F1 in our comparison and still overlaps by only 0.667. No prompt-time change we test removes that order dependence: the only one that gains ranking quality leaves all three decisions unchanged. Order-consistency SFT (OC-SFT) attenuates it in the weights, training a candidate's score not to depend on the order. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks: it flips the reader's answer on 0.125 of permutation pairs against 0.149-0.164 for three other objectives that target order. It is more stable than order-averaged distillation on 12 base models, and one OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.

1 INTRODUCTION

LLM scorers that score candidates jointly produce decisions whose stability is not guaranteed by ranking quality. The paper measures these decision changes across tasks and finds OC-SFT as the preferred training approach.

  • Thresholds, readers, and preference models consume scores as decisions, so identical ranking quality can still produce different outputs.
  • Presentation dependence arises because each candidate’s score is conditioned on the shared prompt and candidate order.
  • Prompt-construction fixes do not remove order dependence, motivating training that makes scores less order-sensitive.
  • OC-SFT penalizes disagreement across permutations and is the compared objective whose decisions change least.
  • The evaluation tests passage reranking, multi-document QA, and response ranking across collections and model bases.
  • Five trained scorers within 0.010 nDCG@10 retain sets overlapping only 0.656–0.835 under reshuffling.

2 RELATED WORK

Prior work identifies order, prompt composition, and other presentation choices as sources of scorer instability. Existing remedies place their cost at inference or in prompt construction, while this paper measures consumer-level stability and addresses it during training.

  • Candidate order and accompanying candidates can shift scores in retrieval, reward-model, and LLM-judge settings.
  • Prompt construction also varies through partitioning rules, slot markers, and answer-skeleton wording, each placing cost at a different stage.
  • Inference-time remedies include permutation averaging, tournament grouping, adaptive context sampling, and downstream decoding from several retrieval orders.
  • Rerankers and reward models are generally trained for ranking or preference quality rather than presentation dependence.
  • The paper measures changes in consumer outputs across permutations instead of only stability in rankings or accuracy under one ordering.

3 PRESENTATION DEPENDENCE AND HOW WE MEASURE IT

Presentation dependence is the score variation caused by how candidates are arranged and presented in a shared prompt. The paper isolates this order channel and measures both ranking movement and changes in downstream consumer outputs.

  • A scorer processes candidates in windows of B, and a candidate’s score depends on its window assignment and accompanying candidates.
  • Presentation dependence includes candidate order, window partitioning, slot markers, and answer-skeleton wording while holding the query and pool fixed.
  • τ-PSI averages pairwise Kendall correlations across M=10 random permutations and treats lower values as greater stability.
  • A permutation-invariant scorer has τ-PSI 0, while τ-PSI values of 0.5 and 1 correspond to uncorrelated and reversed rankings.
  • Consumer measures report both output quality and disagreement across permutation pairs because ranking movement need not match decision changes.
  • The score decomposes into an order-marginal µ_B(x) and residual δ_B, with τ-PSI measuring the residual’s effect on ranking.

4 WHERE TO ADDRESS ORDER DEPENDENCE

The paper compares addressing order dependence at serving, in labels, or during training while retaining a shared-prompt score readout. OC-SFT penalizes the scorer’s own order residual using multiple shuffled views.

  • The score readout extracts one expected grade per candidate from a shared prompt, producing continuous scores usable by decision thresholds.
  • Training variants differ in whether they encounter permutations at serving, in offline labels, or in OC-SFT’s multiple views.
  • OC-SFT preserves the single-order relevance target while scoring each window under N permutations and pulling each candidate’s views toward their mean.
  • The objective penalizes empirical score variance proportional in expectation to Eπ∥δ_B(x, π)∥2, targeting the within-window order residual.
  • Both the relevance anchor and consistency penalty are required; λ=0 reduces the objective to single-order distillation.

5 EXPERIMENTAL SETUP

The experiments evaluate batched candidate scoring across reranking, QA, and response ranking, using shared readouts, multiple models, baselines, and decision-oriented metrics.

  • Three tasks score candidate sets in one prompt, using task-specific data and teacher labels with the same readout and objective.
  • Passage reranking uses MS MARCO training data and evaluates 18 collections spanning TREC-DL, BEIR, and two legal collections.
  • The primary Qwen3-4B experiments extend to 11 dense bases across three model families, with self-distillation requiring no external teacher.
  • Baselines include batched self-consistency, logit calibration, published rerankers, GPT-5.4, and pointwise or pairwise reward models.
  • Quality uses nDCG@10 or nDCG@1, while order instability uses τ-PSI and consumers use retained-set overlap, answer flip, or pair flip.
  • Table 1 reports quality, instability, and consumer decisions over M=10 permutations, with trained results averaged across three seeds.

6 RESULTS

Ranking quality and downstream decisions diverge under reshuffling: prompt-time fixes leave decisions largely unchanged, whereas OC-SFT improves decision stability without sacrificing quality across tasks and models.

  • 6.1 DECISIONS AT MATCHED RANKING QUALITY: 0.656–0.835 retained-set overlap separates five trained scorers whose passage-reranking quality differs by at most 0.010 nDCG@10.
  • 6.1 DECISIONS AT MATCHED RANKING QUALITY: 0.667 retained-set overlap leaves jina-reranker-v3 behind OC-SFT despite leading it on retained-set F1.
  • 6.2 CHANNELS FIXED AT PROMPT CONSTRUCTION: 0.052 nDCG@10 is recovered by round-robin partitioning, but retained sets, reader verdicts, and selected response pairs do not measurably change.
  • 6.2 CHANNELS FIXED AT PROMPT CONSTRUCTION: Fixed Grade: 0 is used as the placeholder because it has the best average ranking quality, while no alternative lowers τ-PSI distinguishably.
  • 6.3 WHAT TRAINING CHANGES FOR THE CONSUMERS: 0.083 order instability under OC-SFT is lower than 0.209 under single-order distillation, with more reproducible retained sets, reader outputs, and selected pairs.
  • 6.3 WHAT TRAINING CHANGES FOR THE CONSUMERS: OC-SFT remains more stable across 11 dense bases and unseen pool replacement, where instability is 0.031 versus 0.099 for single-order distillation.
  • 6.3 WHAT TRAINING CHANGES FOR THE CONSUMERS: At K=10, serving-time self-consistency still fails to reach OC-SFT at K=1 on two of three consumers, while averaging further improves OC-SFT.
  • 6.3 WHAT TRAINING CHANGES FOR THE CONSUMERS: Permutation augmentation closes about two thirds of the τ-PSI gap, whereas OC-SFT closes the remaining third without the reader’s 0.023 exact-match cost.

7 CONCLUSION

Decision stability must be measured alongside ranking quality because training order-consistency can improve reproducibility while preserving ranking performance.

  • A consistency penalty attenuates order dependence in the weights using one teacher pass and one serving permutation, without ranking-quality cost.
  • The resulting scorer makes threshold retention, reader answers, and preference-model pair selection more reproducible.
  • Stability metrics over permutations provide the additional measurement needed to distinguish scorers that ranking metrics cannot separate.

ETHICS STATEMENT

The study examines scorer reliability using public benchmarks and two governed proprietary legal collections, without collecting new annotations or releasing sensitive raw data.

  • The evaluation combines public TREC-DL and BEIR collections with two proprietary legal collections containing professional content and attorney relevance judgments.
  • No new annotations were collected, and raw queries, documents, and annotator identifiers from the legal collections are not released.
  • GPT-5.4 comparisons are less reproducible because its weights and training data are unavailable and the served version may be retired.
  • The method itself requires no closed model because the default teacher is the student’s open-weight base model.

REPRODUCIBILITY STATEMENT

The paper releases its scoring and evaluation materials, configurations, code, labels, adapters, and commands, while documenting substantial computational requirements and reproducibility caveats.

  • The paper documents the scoring readout, reranking prompt template, silver-label protocol, consistency-weight selection, and shared training configuration.
  • The release includes the τ-PSI implementation, evaluation harness, scoring prompt, training and evaluation code, silver labels, LoRA adapters, configurations, and commands.The public evaluation suite runs in 60–90 hours on one L40S-48GB GPU; one collection takes about an hour.
  • Rescoring byte-identical prefixes in separate inference batches can return different numbers, so released outputs may not reproduce score-level figures exactly.This caveat applies to GPT-5.4 evaluation through the specified Azure OpenAI deployment.

AI USE STATEMENT

The paper treats large language models as objects of study and method components, while using generative AI tools for implementation, literature work, drafting, editing, and reference formatting.

  • Large language models serve as trained scorers, silver-label teachers, and prompted closed-model baselines in the study.
  • Generative AI tools were used for software implementation, data preparation, literature search, brainstorming, result interpretation, manuscript drafting, editing, and reference formatting.
  • The authors verified references against primary sources and checked numerical claims against the cited floats.

A SCOPE AND LIMITATIONS

The study specifies its scoring protocol, task configurations, evaluation collections, and tested alternatives, while bounding conclusions by serving conditions, data access, model behavior, and task scope.

  • A SCOPE AND LIMITATIONS: The serving order comes from the first-stage system, while evaluations permute candidates and freeze consumers rather than refitting them to each scorer.A co-adapted consumer could absorb some instability, so reported reductions may be smaller in a fully retrained pipeline.
  • A SCOPE AND LIMITATIONS: The conclusions exclude reasoning-intensive retrieval, reasoning rerankers, non-English collections, and long-context corpora exceeding the B = 20 prompt geometry.
  • B.1 SCORE READOUT AND PROMPT: The reranker scores B = 20 candidate documents in one prompt by reading four grade-token probabilities per candidate from a fixed Grades: skeleton.Documents are truncated to 1200 characters for TREC Deep Learning and non-reranking corpora, and 500 characters for BEIR and legal collections.
  • B.1 SCORE READOUT AND PROMPT: The selected expected-grade readout is held fixed after comparing argmax grade, P(g ≥2), and expected grade on the off-the-shelf Qwen3-4B scorer.P(g ≥2) leads expected grade by less than 0.001 nDCG@10, while argmax grade is lower on the aggregate grid.
  • B.1 SCORE READOUT AND PROMPT: Varying placeholders from Grade: 0 through Grade: 3 decreases quality monotonically from the selected Grade: 0, and no alternative improves both quality and instability.
  • B.3 CONSOLIDATED CONFIGURATION: Every task selects its consistency weight λ and checkpoint using held-out ranking quality, with matched tuning effort across OC-SFT and baselines.The consistency-weight grid is λ ∈{0.5, 1, 2, 3, 4, 5}.
  • A SCOPE AND LIMITATIONS: The study tests slot, window-index, companion-coupling, pool-membership, marker-renaming, measurement-artifact, and metric-robustness effects.

C.1 MECHANISMS AND PERTURBATION TRANSFER

The scorer’s residual order dependence is dominated by companion effects, while OC-SFT substantially attenuates these residuals and transfers robustness to changed candidate pools but not renamed markers.

  • Mechanisms: More than four fifths of off-the-shelf per-document score variance is residual variation consistent with companion coupling, and OC-SFT removes 96.8% of it.The decomposition removes each document’s mean score, then separates slot and window-index effects from the remaining residual.
  • Mechanisms: OC-SFT flattens both pooled and gold-only order-residual profiles, contrasting with the off-the-shelf scorer’s monotone early-over-scored pattern.The observed profile differs from the U-shaped long-context effect associated with “lost in the middle.”
  • Perturbation transfer: On all 15 collections, order-averaged distillation falls between single-order distillation and OC-SFT for Pool-PSI and top-10 churn, while mean nDCG@10 remains level.The drop-only control changes Pool-PSI by less than 0.001, isolating changed companions rather than adding rank 101.
  • Perturbation transfer: Order-consistency training transfers to changed pool membership but not marker renaming; a marker-consistency penalty lowers held-out marker τ-PSI across all six held-out cells.Marker identifiers do not change relevance but can move off-the-shelf scores, and varied-marker SFT alone does not remove this sensitivity.
  • Supervision and training cost: Increasing teacher permutations from T=5 to T=40 changes nDCG@10 by +0.0042 and τ-PSI by −0.0398 at 8x the label-generation cost.Additional teacher passes mainly improve stability, whereas increasing student views beyond N = 2 does not help.
  • Supervision and training cost: From K = 1 to K = 10, permutation averaging raises off-the-shelf nDCG@10 by 0.095 but each trained variant by at most 0.035, indicating training captures most of that quality gain.OC-SFT has the lowest single-pass τ-PSI among the four variants.

D.6 SERVING-TIME AVERAGING AFTER TRAINING

Serving-time permutation averaging improves reproducibility after OC-SFT, but contributes less to label correctness once the scorer is trained. OC-SFT remains the strongest trained variant across consumer decisions and matched retention settings.

  • Consumer stability: OC-SFT leads every measured consumer-stability budget and collection from K = 2 onward, with one TREC-COVID tie at K = 1.The consumers are retained sets, reader outputs, and selected response pairs.
  • Passage reranking: 0.832 retained-set overlap at K = 1 rises to 0.950 at K = 10 for OC-SFT, versus 0.663 and 0.869 for single-order distillation.Order-averaged distillation reaches 0.737 at K = 1 and 0.904 at K = 10; all values are three-seed means.
  • Response ranking: 0.654 of off-the-shelf queries agree with gold under some permutations but not others, and 0.682 of pair flips occur on those queries.The K = 10 average raises binary-gold mean label accuracy.
  • Response ranking: Averaging adds little to label correctness after OC-SFT, while reproducibility continues improving with additional presentations.Its gains concentrate in RewardBench-2 and PPE-MATH and add almost nothing on RM-Bench or PPE MMLU-Pro.
  • Multi-document QA: OC-SFT has the lowest τ-PSI on all 13 QA collections while remaining near order-averaged distillation in quality.It ties order-averaged distillation under BM25, trails by 0.004 under BGE, and leads by 0.002 under SPLADE.
  • Reader decisions: OC-SFT reduces verdict and answer flips under both frozen readers while reported reader-accuracy intervals cover zero.The same-evidence controls are positive, indicating the lower flip rates do not require changing which evidence reaches the reader.

G.3 TRAINING WIDTH AND TRAIN-SERVE MISMATCH

Training width affects how scorers transfer across serving widths, while OC-SFT remains stable across a broad width range and preserves per-document scores at lower latency. Its stability advantage transfers across bases and teacher choices, but quality depends on the training and serving geometry.

  • Train-serve mismatch: B = 20-trained students transfer to B = 1, whereas B = 1-trained students lose 0.050 nDCG@10 when served at wide width and recover it only by averaging ten evaluations.Training width therefore determines tolerance to serving-width changes.
  • Width effects: The +0.020 passage-reranking gain from order averaging survives estimator matching, while engine-call batching has little effect on retained-set overlap.Changing candidate order produces the larger effect than changing engine-call batching.
  • Score dependence: Training attenuates score-level width effects without making OC-SFT pointwise; the consistency penalty adds no distinguishable further reduction.OC-SFT remains above zero on the normalized score-width effect.
  • Serving width: OC-SFT quality remains stable from B = 1 through B = 30, while nDCG@10 and τ-PSI degrade at B = 34 and further at B = 40 and B = 50.The selected checkpoint was trained at B = 20.
  • Latency: Candidate batching reduces median per-query latency by 51.3% at B = 10 and 45.4% at B = 20 relative to B = 1.The reduction compounds with serving one permutation rather than ten.
  • Cross-base robustness: OC-SFT is more stable than order-averaged distillation across all twelve bases, with further instability reductions on ten dense bases and the sparse MoE.Qwen3-0.6B is omitted because its off-the-shelf output is degenerate and does not follow the required format.
Loading 2608.26762v1…