One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

• The pooled selection-rate standard deviation measures how unevenly letters are selected, but can rise with either concentration on letters or variation in accuracy across them.Prior correction divides each letter’s probability by an estimate of its option-symbol prior; reversal aggregation selects the content with the largest product of probabilities under an ordering and its reverse.Source: §1 · p. 1
• Both mitigation methods roughly halve the accuracy range across correct-answer letters, but aggregation removes about a third of changing-answer questions while prior correction removes almost none.Aggregation also leaves fewer model-rank reversals, while partly reducing flips by settling questions on a wrong answer.Source: §1 · p. 2
• Under balanced orderings, pooled SD exactly equals the label-free flip rate multiplied by positional skew, so the same score can reflect many mild-lean flips or few strong-lean flips.Source: §1 · p. 2
• The authors propose reporting both factors alongside accuracy because pooled scores mix them and neither factor alone indicates whether mitigation improves predictions.Prior correction mainly moves skew, while aggregation mainly moves the flip rate.Source: §1 · p. 2
• Across 41 checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while positional skew tracks the estimated option-symbol prior.Source: §1 · p. 3

• Prior-correction studies estimate and divide out option-symbol preferences, while related interventions alter bias nodes, model components, or label priors.The paper distinguishes the quantities measured and operations applied, then identifies which pooled-score factor a correction moves.Source: §2 · p. 3
• Order-sensitivity studies count answer changes under selected perturbations and have used aggregation, majority voting, calibration, or reliability reporting.This paper relates flip rates to pooled scores under balanced designs and examines how the paired orderings affect skew.Source: §2 · p. 3
• Prior audits often report order effects beside favored positions, while this paper gives an exact factorization of pooled SD for balanced multiple-choice ordering designs.Source: §2 · p. 3

3.1 Cyclic reordering and the flip rate

• The flip rate is the share of items whose chosen content changes across cyclic orderings, and it needs no gold labels.Stable items are either correct in every ordering or wrong in every ordering; the flip rate bounds accuracy changes across cyclic orderings and is a lower bound on its value across all K! orderings.Source: §3.1 · p. 4

3.2 The pooling identity

• With balanced orderings, stable items contribute uniform position shares, so pooled SD factors exactly into flip rate times the positional skew of flipping items.The identity requires each content to appear at each position equally often; without balance, stable items can also skew pooled shares.Source: §3.2 · pp. 4–5
• The identity describes how pooled SD is composed, not why answers flip or lean; flips can occur without producing uneven pooled shares.Different combinations of flip rate and skew can yield nearly the same pooled SD.Source: §3.2 · p. 5

3.3 What each kind of statistic reads

• Balanced RStd also factors through flip rate but uses the gold-label-dependent accuracy distribution of flipping items, whereas pooled SD uses their position distribution without gold labels.Consistency counts changes but not where changing answers go; pooled SD and balanced RStd combine the flip rate with different measures of the changing items’ lean.Source: §3.3 · pp. 5–6

• Table 1 classifies statistics by what they read: consistency measures count changes, while pooled SD and balanced RStd factor through flip rate and forward KL is bounded through it.Only pooled SD splits into two label-free factors.Source: Table 1 · p. 6
• The evaluation covers 41 open-weight checkpoints from seven model families across 12 benchmarks with two, four, or ten options.A cell is one checkpoint–benchmark pair.Source: §4 · p. 6
• The primary answer is the option letter with the highest next-token probability; five-shot, direct-answer, and chain-of-thought prompting with greedy decoding serve as checks.Source: §4 · p. 6
• Prior correction divides letter probabilities by an estimated option-symbol prior, while reversal-product aggregation multiplies each content’s probabilities under an ordering and its reversal.The composition applies operations in sequence, chiefly correction followed by the product; all factors are computed over the resulting answers across cyclic orderings.Source: §4 · p. 6

5.1 Flips track accuracy and uncertainty

• Across cells, flip rate falls with accuracy (rs = −0.95) and concentrates on items whose leading options are close in probability, including items wrong under every ordering.The association also holds where the accuracy cap on flips restricts nothing; flipping rather than stable items is identified with median AUC 0.85.Source: §5.1 · pp. 6–7

5.2 The skew reflects option-symbol bias

• Dividing out the model’s estimated prior lowers positional skew while barely moving flip rate; dividing out a letter-permuted prior raises skew.The flipping answers lean toward letters favoured by the prior (median centred cosine 0.87 on four-option cells, versus −0.10 for the permuted prior).Source: §5.2 · p. 7, Figure 4 · p. 7
• Each factor ranks cells nearly the same way on two random item halves, and they carry comparable shares of variance in log pooled SD across cells.Source: §5.2 · p. 7

6 Mitigations move different factors

• Prior correction mainly lowers skew, while the reversal product mainly lowers flip rate and part of skew.The reversal product’s effects on added preferences depend on whether mirrored letters differ: opposite-sign preferences cancel, while same-sign preferences survive.Source: §6 · pp. 7–8, Figure 6 · p. 8
• Correction subtracts an estimated prior within each ordering but does not link answers across orderings, so skew can fall much more than flip rate.The small net flip-rate change can hide movement in both directions; the averaged prior does not separate estimation error, item-specific preferences, or other departures as causes of remaining flips.Source: §6 · pp. 7–8
• The reversal product combines content scores under an ordering and its reversal, pairing mirrored letters such as A with D and B with C.Opposite-sign mirrored preferences cancel exactly, while same-sign preferences survive; across cells, more surviving preference is associated with a smaller skew reduction.Source: §6 · p. 8
• Prior correction and the reversal product lower balanced RStd alike, from 0.064 to 0.036, although the product lowers flip rate much more.Balanced RStd alone makes the operations look similar, while flip rate alone misses correction’s placement-range reduction.Source: §6 · p. 8

7 Residual harms and composition

• Placement range measures how far accuracy moves when correct answers occupy different letters, while rank reversals count checkpoint pairs whose accuracy ordering changes across cyclic orderings.A checkpoint’s accuracies under two orderings differ by at most its flip rate on the same items.Source: §7 · p. 8
• The reversal product leaves fewer flips and rank reversals than correction while narrowing placement range by a similar amount; correction followed by the product lowers both factors and narrows the range most.The comparison covers shared held-out items in cells with at least three options.Source: §7 · pp. 8–9, Table 2 · p. 9
• Correction followed by the reversal product retains the product’s flip-rate reduction, lowers skew more than the product alone, and leaves rank reversals at the product’s level.Source: §7 · p. 9
• The reversal product lowers flip rate partly by settling items on a wrong option; stable-wrong share rises more than accuracy does.For nine of ten newly stable-wrong items, the settled option was already the model’s unique most frequent choice across cyclic and reversed queries.Source: §7 · p. 9

8 Conclusion

• Under balanced orderings, pooled SD is exactly flip rate multiplied by positional skew, and balanced RStd shares only the flip rate unless K = 2.Prior correction mainly lowers skew; aggregation with an ordering’s reversal mainly lowers flip rate, and applying correction before aggregation lowers both factors.Source: §8 · p. 9
• The factors leave different harms: aggregation leaves fewer rank reversals but lowers flips partly by settling items on a wrong answer, so accuracy should accompany both factors.The paper recommends judging mitigation by both factors, accuracy, and the harm of interest.Source: §8 · p. 9

Limitations

• The analysis characterizes observable effects of option order, not the model-internal mechanisms that produce them.The authors call for targeted interventions to investigate those mechanisms.Source: Limitations · p. 10
• Observed associations with accuracy and output uncertainty do not establish causal explanations.Source: Limitations · p. 10
• The pooling identity is exact under balanced orderings, but reducing either factor does not guarantee better predictions.Consistently wrong answers can remain stable, and positional-skew changes need not improve placement robustness or ranking stability; experiments focus on deterministic readouts of mostly English benchmarks and specific mitigation configurations.Source: Limitations · p. 10

A.1 Cohorts, models and benchmarks

• The evaluation spans 12 benchmarks and seven checkpoint families, filters items to those usable under every ordering, and uses a fixed prompt with first-token and generative readouts.Readouts also include five-shot prompting; chain-of-thought extraction uses the latest valid explicit answer or a standalone-letter fallback, and no generative readout includes MMLU-Pro.Source: §A.1 · pp. 12–13, Table 3 · p. 13, §A.2 · p. 14

A.4 Decoding

• Open-checkpoint readouts assume deterministic answers under repeated prompts, so sampled readouts are excluded and closed models that vary on repeats are analyzed separately.Factor and mitigation analyses use paired checkpoint-block bootstrap draws with nominal 95% percentile intervals.Source: §A.4 · p. 15, §A.5 · p. 15

B.1 The identity under any ordering design

• For any ordering design, pooling admits a corrected decomposition, while the original flip-rate identity holds for every prediction pattern exactly when the design is position-balanced.The corrected residual is at numerical precision in all-ordering cells and random-subset draws.Source: §B.1 · p. 16

B.2 Gold-label statistics and two-order disagreement

• Under balanced cyclic orderings, the placement-accuracy standard deviation and range factor into the flip rate and gold-letter accuracy variation among flipping items.Balanced RStd is the letter-balanced form of RStd (Zheng et al., 2024); across cells, the dispersions of gold-letter accuracy and option-letter skew have a Spearman correlation of 0.92.Source: §B.2 · p. 17
• A pair’s disagreement rate is bounded by the flip rate when both orderings belong to the design, but reversal may fall outside cyclic orderings for K ≥3.With additional orderings, pairwise disagreement and the overall flip rate need not determine one another, and neither indicates where changed answers move.Source: §B.2 · p. 18

B.3 The divergence bound

• On balanced designs, total variation and maximum deviation scale with the flip rate, chi-square scales with its square, and KL divergence is bounded by the flip rate times the flipping-item divergence.The KL bound is tight when the flip rate is 0 or 1, or the flipping-item histogram is uniform.Source: §B.3 · p. 18

B.4 The skew and the placement factor

• Option-letter skew does not determine placement harm for four-option items, although their centered vectors have a mean cosine of 0.92 in operation-evaluation cells.Across cells, skew and placement-range changes correlate at 0.92, yet skew falls while placement range rises in 3% of reversal-product cells and 6% under prior correction without aggregation.Source: §B.4 · p. 18

C.1 Canonical margins and the option-symbol prior

• The canonical top-two gap separates flipping from stable items across accuracy thirds, while prior correction lowers skew and raises accuracy and letter-permuted correction has the opposite effects.The gap distinguishes flips less sharply from stable-wrong than stable-correct items; at half strength, the estimated prior still lowers skew and the permuted prior still raises it, by smaller amounts.Source: §C.1 · p. 19, Table 5 · p. 19, Table 6 · p. 19
• Leaving the canonical answer out of the flip outcome lowers AUC but remains above chance; the gap also separates movers from stable items among those wrong under every ordering.Most separation remains when items are compared within strata of their own correctness and how often other checkpoints answer them correctly, indicating the gap measures option competition beyond accuracy alone.Source: §C.1 · p. 20

C.2 Variation and reliability of the factors

• Flip rate correlates strongly with accuracy (rs = −0.947), versus more weakly for positional skew std(g) (rs = −0.333); their variance contributions are comparable overall.Across option counts, the flip-rate contribution is larger on four-option cells, while the skew contribution is larger on two- and ten-option cells.Source: §C.2 · p. 20, Table 8 · p. 21
• Both factors rank cells nearly identically across random item halves, and a conservative cluster-aware test rejects uniform position choice in nearly two fifths of first-token cells after Holm correction.The test treats each item’s K answers as a cluster and items as independent draws within each checkpoint and benchmark.Source: §C.2 · p. 20, Table 9 · p. 21

C.3 A second prior estimator

• A held-out option-letter probability profile from disjoint items ranks cells closely with flipping-item skew, including after controlling cyclic accuracy and option count, though ten-option estimates have wide intervals.The held-out profile averages option-letter probabilities over items and cyclic orderings; its nonuniformity is the SD across option letters.Source: §C.3 · p. 21, Table 10 · p. 21
• Observed flip-rate/accuracy association exceeds the cap null where the cap does not bind, is compatible where it binds, and is not compatible on four-option cap-one cells.The null permutes the flip rate scaled by its cap within option-count and cap subsets.Source: Figure 8 · p. 22, Table 11 · p. 22

C.4 Controls on the flip rate’s association with accuracy

• Even where accuracy caps the flip rate, scaled-rate controls are compatible with the association; with four options and accuracy at most 0.75, flips still fall with accuracy (rs = −0.881).The cap is not a complete explanation, though correctness could act through other routes.Source: §C.4 · p. 22
• The cyclic flip rate remains negatively associated with accuracy measured on disjoint items, benchmarks, or orderings.The disjoint-item control finds it tracks accuracy about as closely as when measured on all items.Source: §C.4 · pp. 22–23, Table 12 · p. 23

C.5 Closed models

• For closed models, reordering changes answers about as often as repeating the same ordering, so these measurements cannot isolate reordering’s contribution to flips.The authors therefore do not place their factors beside the open-weight cells.Source: §C.5 · p. 23

D.1 Protocol

• Operations share first-token predictions and evaluate the remaining 95% of each cell after a 5% calibration split, with no gold labels used in the operations.The main cohort has at least three options and uses cyclic orderings and their reversals.Source: §D.1 · p. 24

D.2 What the reversal product cancels

• Reversal pairs retain the symmetric component of a letter preference and cancel its antisymmetric component, including front-to-back and alternating preferences for four options.The surviving skew also depends on model scores and which items still flip; the retained-share statistic correlates with log-skew change at 0.29 [0.17, 0.41].Source: §D.2 · pp. 24–25

D.3 Adding a known letter preference

• Adding a known one-logit preference leaves answers unchanged under front-to-back or alternating shapes after reversal aggregation, while PriDe largely removes every tested shape.The experiment intervenes on output scores and does not establish that models form preferences additively.Source: §D.3 · p. 25

D.4 Effects of each operation

• The reversal product resolves flips, about two in five onto a wrong option, while prior correction moves items both into and out of flipping.Among items made stably wrong, aggregation usually settles on the option already chosen most often; these outcome-selected items do not identify the cause.Source: §D.4 · p. 25, Table 14 · p. 25

D.5 PriDe on all cells and checkpoint comparisons

• For base and post-trained checkpoint pairs, pooled SD and flip rate move in opposite directions in about one pair in six when both changes are resolved; skew can outweigh a post-training flip-rate fall.The figure distinguishes opposite-direction changes from same-direction changes; filled points have both item-bootstrap intervals excluding zero.Source: Figure 9 · p. 26, §D.5 · p. 26
• Among cells where PriDe lowers pooled SD, the median share of log reduction attributed to skew is 0.96; when pooled SD rises, the median flip-rate share of log increase is under a tenth.In about a quarter of cells where pooled SD falls, the flip-rate point estimate does not fall.Source: §D.5 · p. 26

E Cyclic orderings against all K!

• The cyclic flip rate is a lower bound on the rate over all K! orderings because any item that flips cyclically also flips in the full set.This appendix measures the bound’s gap and compares mitigation operations across balanced ordering sets.Source: E Cyclic orderings against all K! · p. 27

E.1 How loose the bound is

• The cyclic protocol understates the full flip rate by 0.095 on average, yet orders cells almost identically and tracks accuracy almost equally; its flip rate is 7.2% higher per cell than an equally sized random subset.It finds 76.9% of items whose correctness changes across all K! orderings, pooled over cells from six checkpoints.Source: §E.1 · p. 27

E.2 Operations beyond the cyclic orderings

• Across balanced four-ordering sets, PriDe lowers placement range, the reversal product lowers flip rate by about ten points, and their composition lowers both.The cyclic set is typical in flip rate and placement range, but about one in ten items stable on it flips on another set.Source: §E.2 · p. 28

E.3 Other ordering designs

• The pooling identity is exact on position-balanced designs but departs from random four-ordering subsets; subtracting the stable-item deviation restores it under random draws.In the median draw, the corrected statistic tracks accuracy better than uncorrected pooled SD and nearly as well as the cyclic design.Source: §E.3 · p. 28, Table 19 · p. 28, Table 20 · p. 29

F Metric definitions

• The flip rate counts items whose chosen content changes across cyclic orderings, while the fluctuation rate compares two cyclic orderings; for four options, the compared rotations are ABCD and CDAB.The referenced fluctuation-rate definition instead compares a forward ordering with its reversal.Source: F Metric definitions · p. 29
• Selection-rate SD uses unconditional selection frequencies without gold labels, whereas RStd depends on gold labels and balanced placement of correct answers.The paper’s factorization requires every option of every item to visit every position equally often.Source: F Metric definitions · p. 29

G Closest related measurements and mitigations

• Consistency measures do not by themselves establish correctness, and the paper distinguishes stable answers from correct ones when evaluating mitigation.Related multiple-choice work reports differing mitigation effects across RStd, PBM, and CKLD; PriDe’s pooled-SD identity is exact under balanced designs.Source: G Closest related measurements and… · p. 30
• RStd and CKLD compare predictions with gold labels, unlike the paper’s label-free selection-frequency statistics; PriDe’s correction does not itself enforce agreement across orderings.The paper distinguishes PriDe’s skew reductions from permutation-aware training that rewards agreement.Source: G Closest related measurements and… · p. 31