PaperlayerView exact revision on arXiv

Source-linked AI summary

The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express

Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep

arXiv:2610.17158v1cs.AIcs.CLcs.SE

TL;DR

Scoring formats for LLM-judged instruction-tuning data are rarely ablated despite their role in tie-breaking selection. The paper compares five formats under a controlled setup and finds that their effective resolution differs sharply, while τb can misrepresent selection-relevant ties.

  • Problem

    Scoring formats are usually chosen for convenience and rarely ablated with the selector held fixed, although curation consumes scores as an ordering and breaks ties at random.

    Source: §1 · p. 1

  • Method

    The study scores a fixed pool under five formats with three judges, then curates subsets, fine-tunes 66 models, and evaluates them with four judges.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Numeric-100 yields 7.0–9.9 effective levels at α = 2 versus 3.3–4.4 for Likert-5, while binary leaves the top score tied in 77–78% of prompts.

    Source: §3.2 · p. 3, §3.3 · p. 3

  • Takeaways & Limitations

    Curation pipelines should report the format and decoding rule, tie probability or D2, comparative-format slot controls and bias, and verdict parse rate.

    Source: §5 · pp. 4–5

    The structural results describe format together with greedy decoding, not format alone.

    Source: Limitations · p. 5

Abstract

from arXiv · show

When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty, and a binary verdict leaves the top score tied in 77--78\% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $\tau_b$ hides them: it ranks binary best while $\tau_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.

1 Introduction

LLM-judge scoring formats are usually chosen for convenience rather than ablated with the selector held fixed, even though selection breaks score ties at random. The paper isolates format in a controlled study and connects tie-aware diagnostics to data selection.

  • A format choice is rarely ablated with the selector fixed, although tie structure matters because selection breaks ties at random.With a binary verdict, all accepted candidates are indistinguishable and the frontier choice is made by chance.Source: §1 · p. 1
  • The study varies five formats across three judges on a fixed candidate pool and follows one judge’s scores through curation and fine-tuning to 66 models.Its contributions include format-driven tie structure, a selection-audit problem with τb, comparative-format slot bias, and downstream evaluation.Source: §1 · p. 1
  • Prior elicitation studies assess agreement or protocol bias without following scores into training, while data-selection studies follow scores using one elicitation protocol.Deutsch et al. (2023) show that ties distort Kendall-style meta-evaluation; this paper examines the consequence for selection.Source: §1 · p. 1

2 Method

The method compares five scoring formats while holding the candidate pool and decoding setup fixed, then uses curated subsets to train models for evaluation by four judges.

  • The study samples 4,000 UltraFeedback prompts, yielding 15,406 responses after a 20–6,000-character filter, and holds out 300 further prompts.UltraFeedback GPT-4 scores serve as both the agreement reference and the source for choosing the anchored-format candidate; they do not enter the prompt as numbers, selection, or training data.Source: §2 · pp. 1–2
  • Five formats share either pointwise or pairwise prompts, use greedy decoding, and preserve score ties through within-format ranking.The pointwise formats are binary, Likert-5, and Numeric-100; the comparative formats are anchored and tournament, with scores derived from verdicts.Source: §2 · p. 2
  • Three judges score the identical pool, and DeepSeek scores determine top-ranked selections at 10%, 25%, or 50% retention, with ties broken randomly.Random, length-based, and full-data subsets provide comparison conditions; each subset fine-tunes Qwen3-4B with LoRA across three seeds, for 66 models total.Source: §2 · p. 2
  • Four judges evaluate model responses against the highest-GPT-4-rated completion on 300 held-out prompts, with comparisons run in both orders.Win rates are comparable within a judge but not across judges because unresolved comparisons score 0.5 and judges resolve different fractions; the analysis uses Friedman tests and paired, seed-clustered bootstrap contrasts with Holm correction.Source: §2 · p. 2

3 Scoring-Stage Results

Across scoring-stage diagnostics, format explains nearly all structural score variation, while judge effects are small. Effective score resolution, tie-aware agreement measures, and comparative slot effects reveal why nominal scale size alone is insufficient.

3.1 Format Fixes Score Structure

  • Format explains 97–99% of structural diagnostic variance, while judge explains under 0.5%.After normalizing nominal capacity, D2 remains 96% format-driven; excess tie probability is 54% format and 41% residual.Source: §3.1 · p. 2
  • With DeepSeek and Qwen alone, format explains 99.0–99.7% of reference-agreement variance, while judge explains 0.1–0.4%.The full-grid judge share is partly carried by Nemotron; the authors caution that three judges do not establish a capability law.Source: §3.1 · p. 2, Table 2 · p. 3

3.2 Nominal Granularity is Not Effective Granularity

  • Numeric-100 yields 7.0–9.9 effective levels at α = 2, versus 3.3–4.4 for Likert-5—two to three times the effective resolution, not twenty.At one-third of the pool, distinct values fall from 37 to 28.0±2.0 while D2 moves by about 0.1 s.d. or less.Source: §3.2 · p. 3

3.3 Kendall’s τb Misleads a Selection Audit

  • τb ranks binary best under DeepSeek and Nemotron, although τa, Somers’ D, and top-1 rank it last under DeepSeek and Qwen.Binary leaves the top score tied in 77–78% of prompts, versus 40–47% for Numeric-100, so selection often breaks ties randomly.Source: §3.3 · p. 3

3.4 Comparative Formats Inherit Slot Bias

  • Anchored comparisons show opposite slot-bias signs across judges, while tournament listing order predicts selection despite a 0.05 correlation with GPT-4 quality.Anchored b is −0.33 for DeepSeek and Qwen and +0.89 for Nemotron; in four-candidate prompts, tournament selection favors the last-listed response 44.6% for DeepSeek and the first-listed 67.5% for Nemotron.Source: §3.4 · pp. 3–4

4 Downstream Consequences

Curated subsets outperform random selection across evaluation judges, and these gains are robust to length and markdown adjustment. No downstream ordering among formats is detected, even though the formats select largely different examples.

4 Downstream Consequences

  • Three curated-versus-baseline contrasts replicate under all four evaluation judges, and five more replicate under three; every replicated contrast favors curated data over a baseline.A +0.09 difference corresponds to about nine more wins against the reference per 100 prompts than RANDOM.Source: §4 · p. 4
  • At 10% retention, 18 of 20 format-by-judge differences versus FULL are positive, up to +0.059, but only two survive Holm correction.RANDOM at 10% stays within 0.02 of FULL.Source: §4 · p. 4
  • Adding length and markdown controls changes every curated-versus-RANDOM contrast by less than 0.02, so the gains are not a length or markdown effect.Curated students produce 1,044–1,257 characters, with markdown in 38–46% of responses; RANDOM produces 1,072–1,113 characters, with markdown in 36–40%.Source: §4 · p. 4
  • At 10% retention, format subsets have mean Jaccard overlap 0.02, yet their students are statistically indistinguishable, suggesting training compresses scoring-stage differences in this pool.Mean overlap rises to 0.06 at 25% and 0.15 at 50%.Source: §4 · p. 4
  • No format beats another under more than one evaluation judge; binary trails Likert-5 in all 12 judge-by-rate comparisons and Numeric-100 in 11 of 12, but the design cannot resolve the gaps.The missing ordering is undetected, not evidence that formats are equivalent.Source: §4 · p. 4

5 Conclusion

With a fixed decoding rule, format largely determines tie structure and which examples are selected, while comparative formats also carry judge-specific slot bias. The paper recommends reporting four pipeline properties and validating formats with measures that reflect selection ties.

5 Conclusion

  • Under fixed decoding, format sets tie structure almost independently of the judge, and that structure determines how often selection is random.The formats also largely change which examples are selected.Source: §5 · p. 4
  • Comparative formats import judge-specific slot bias, so report whether positions were swapped or balanced and the estimated bias.Source: §5 · pp. 4–5
  • Report the verbatim format with its decoding rule, tie probability or D2 alongside distinct-value count and pool size, comparative slot handling and bias, and verdict parse rate.The paper recommends Likert-5 or Numeric-100 over binary and τa, Somers’ D, or top-1 agreement rather than τb for validation.Source: §5 · p. 5

Limitations

The study uses greedy decoding, one curation judge for training data, one candidate pool, one student, and one language, with additional limits on reference and downstream comparisons.

Limitations

  • Structural findings describe format together with greedy decoding, not format alone; probability-weighted or repeated-sampling scores could make binary and Likert-5 effectively continuous.Source: Limitations · p. 5
  • Training data use only DeepSeek curation, so the study does not test whether a less accurate curation judge yields worse students.There is no untuned-student or GPT-4-oracle selector baseline, so better reference agreement cannot be linked to better training data.Source: Limitations · p. 5
  • Agreement is measured against UltraFeedback GPT-4 scores rather than human labels, and the same scores choose the anchored-format candidate and evaluation reference.Source: Limitations · p. 5
  • Length and markdown adjustment is a sensitivity check, not a causal control, because it conditions on properties of the trained model’s own outputs.The LENGTH baseline is confounded with truncation: its 1,024-token cap removes 11.4–25.2% of examples, versus 1.8–6.2% elsewhere.Source: Limitations · p. 5
  • Binary’s disadvantage is inseparable from random tie-breaking, while unswapped anchored comparisons include 26% self-comparisons and tournament selections inherit listing-order slot bias.The study’s scope is one candidate pool, one student, and one language.Source: Limitations · p. 5

A Method Details

The method compares pointwise and pairwise scoring formats, including anchored and tournament comparisons, then follows curation through model training and statistical evaluation.

A Method Details

  • The pointwise prompt asks judges to assess helpfulness, correctness, completeness, and clarity; its format line requests ACCEPT/REJECT, a 1–5 rating, or a 0–100 score.Comparative prompts ask which of two responses is better, with five verdicts mapped to +2, +1, 0, −1, or −2.Source: A Method Details · p. 6
  • Anchored comparisons use the upper-middle candidate by GPT-4 score as Response A, score the verdict as (v + 2)/4, and include 4,000 self-comparisons among 15,406 queries.Source: A Method Details · p. 6
  • Tournament scoring judges within-prompt pairs once in listing order; each candidate’s score is its mean win probability across its matches.Source: A Method Details · p. 6
  • Figure 1 plots Numeric-100 score distributions by curation judge, with blue bars marking multiples of five; Nemotron has 74.7% of scores on those multiples.Source: Figure 1 · p. 7
  • Models use LoRA for two epochs, with training sequences capped at 1,024 tokens; the cap truncates 4.3% of the full pool and 1.8–6.2% of curated subsets.LENGTH truncation is higher, at 11.4–25.2%.Source: A Method Details · p. 7
  • All formats are evaluated on the same 300 prompts using per-judge Friedman tests, paired bootstrap contrasts over prompts and seeds, and Holm correction across format pairs.Source: A Method Details · p. 7

B Anchored Order-Swap Decomposition

The anchored order-swap decomposition separates candidate preference from slot bias by comparing verdicts in both response orders. The reported slot-bias signs differ across judges, while three judges do not establish a monotone capability–bias relationship.

B Anchored Order-Swap Decomposition

  • The method estimates q and b from forward and reverse verdicts using q = (vfwd − vrev)/2 and b = (vfwd + vrev)/2.Candidate preference changes sign when the candidate moves slots; position preference remains with the slot.Source: B Anchored Order-Swap Decomposition · p. 7
  • DeepSeek and Qwen show slot-B bias (b = −0.33), while Nemotron shows slot-A bias (b = +0.89).The decomposition estimates candidate preference q separately from slot bias b; the table caption says DeepSeek and Qwen track quality, while Nemotron is dominated by slot bias.Source: Table 4 · p. 7
  • Three judges do not establish a monotone relationship between capability and position bias.DeepSeek and Qwen have indistinguishable slot bias; the reverse pass samples 500 candidates per judge, excluding self-comparisons, and reports 95% bootstrap confidence intervals.Source: B Anchored Order-Swap Decomposition · p. 7

C Additional Diagnostics

The additional diagnostics cover distinct emitted values and Somers’ D, leave-one-judge-out variance shares, pointwise-only variance shares, and length- and markdown-adjusted curated-versus-random contrasts.

C Additional Diagnostics

  • Table 5 reports distinct values emitted and mean within-prompt Somers’ D.The section identifies this table alongside the other diagnostics.Source: C Additional Diagnostics · p. 7, Table 5 · p. 8
  • Table 6 reports leave-one-judge-out variance shares, with top-1 evaluated using the exact expectation under random tie-breaking.Its columns identify the judge removed.Source: Table 6 · p. 8
  • Table 7 reports variance shares for binary, Likert-5, and numeric-100, excluding slot-assignment confounding.These are the three pointwise formats.Source: Table 7 · p. 8
  • At 50% retention, the largest adjustment to curated-minus-RANDOM win rate is 0.013 after controlling for log response length and markdown.The reported adjustment is smaller at 25% retention.Source: Table 8 · p. 8

D Effective Granularity

Effective granularity measures how many equally used levels a score distribution represents, while distinct-value counts treat rare and frequent scores alike. The reported comparisons vary across judges and measures.

D Effective Granularity

  • DeepSeek’s four most frequent numeric-100 scores—85, 75, 45, and 65—account for 70.2% of the score mass.A distinct-value count weights a score used twice the same as one used 3,791 times; Table 9 lists nominal, emitted, and effective levels.Source: D Effective Granularity · p. 8, Table 9 · p. 8
  • Anchored uses fewer effective levels than Likert-5 under DeepSeek and Nemotron, but Qwen reverses the order (D1 4.34 vs. 3.76).Dα = exp(Hα) is the number of equally used levels with the same Rényi entropy; D1 is exponentiated Shannon entropy and D2 is inverse tie probability.Source: D Effective Granularity · p. 8
  • Subsampling to one-third of the pool reduces distinct-value counts by 16–24% and collapses the judges’ counts to 27.8–28.0 from a full-pool difference of 37 vs. 33.D1 and D2 shift by at most 0.3% of their values and preserve the judge ordering.Source: D Effective Granularity · p. 9, Table 10 · p. 9
Loading 2610.17158v1…