PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Are Not Robust Multiple Choice Selectors

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang

arXiv:2309.03882v4cs.CL

TL;DR

LLMs are vulnerable to option-position changes in MCQs because they exhibit selection bias toward particular option IDs. The paper analyzes this bias across models and benchmarks, attributes it primarily to token bias, and proposes PriDe to estimate and remove the option-ID prior at inference time. PriDe provides effective, interpretable, and transferable debiasing with low computational cost, though performance may degrade across large domain gaps.

  • Problem

    LLMs’ selection bias toward specific option IDs makes their MCQ performance vulnerable to changes in option positions.

  • Method

    PriDe estimates the model’s prior bias for option IDs by permuting option contents on a small set of unlabeled test samples and applies it to debias remaining predictions.

  • Results

    Across 20 LLMs and three benchmarks, selection bias is prevalent and primarily linked to token bias; PriDe achieves superior debiasing effectiveness, especially at low cost.

  • Takeaways & Limitations

    Option-ID token preferences are a central robustness issue in MCQ evaluation, and PriDe offers an efficient label-free way to mitigate that bias.

  • Takeaways & Limitations

    PriDe may slightly degrade performance when transferred across large domain gaps, so the estimated prior should be updated with new samples under predictable domain shifts.

Abstract

from arXiv · show

Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent "selection bias", namely, they prefer to select specific option IDs as answers (like "Option A"). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasing method, called PriDe, which separates the model's prior bias for option IDs from the overall prediction distribution. PriDe first estimates the prior by permutating option contents on a small number of test samples, and then applies the estimated prior to debias the remaining samples. We demonstrate that it achieves interpretable and transferable debiasing with high computational efficiency. We hope this work can draw broader research attention to the bias and robustness of modern LLMs.

1 INTRODUCTION

MCQs are widely used to evaluate LLMs, but changing option positions can substantially alter model performance because models preferentially select particular option IDs. The paper identifies token bias as a primary cause and introduces PriDe, a label-free inference-time debiasing method.

  • Motivation: MCQs are prevalent in LLM benchmarks and automatic evaluation, requiring models to select the most suitable answer from multiple candidate options.The paper frames robust answer selection as an expected property of LLM use in MCQs.
  • Selection bias: 6.3 accuracy points: moving golden answers to position D reduces gpt-3.5-turbo’s 0-shot MMLU accuracy from 67.2 to 60.9.Moving golden answers to position A instead boosts llama-30b by 15.2 points, to 68.2 from its original 53.1.
  • Selection bias: 34.6% / 27.3% / 22.3% / 15.8%: llama-30B selects A/B/C/D at these rates despite balanced correct-answer positions.For gpt-3.5-turbo, the corresponding rates are 22.5% / 25.6% / 32.3% / 19.6%; both distributions are statistically nonuniform.
  • Empirical findings: Across 20 LLMs, three benchmarks, and two-to-five options, selection bias is prevalent and is not well mitigated by simple prompting such as Chain-of-Thought.The bias varies across models but shows cross-domain similarity within the same model.
  • Empirical findings: Token bias is identified as a more salient intrinsic cause than position bias: models assign more prior probability to specific option-ID tokens when predicting answers.The paper reports that position bias is less regular and depends on models and tasks.
  • PriDe: PriDe estimates option-ID priors by permuting option contents on a small fraction of test samples, then uses those priors to debias remaining samples without labels.The procedure runs at inference time with negligible extra computational cost and is reported to improve effectiveness, interpretability, and cross-domain transferability.

2 INVESTIGATION ON SELECTION BIAS

The study finds that selection bias is widespread across LLMs, persists across domains, and is driven primarily by option-ID token bias rather than ordering position. Removing option IDs reduces bias but usually harms performance, while simple prompting strategies do not reliably mitigate it.

  • Measurement: Selection bias is measured as the standard deviation of recalls across option IDs, with greater imbalance indicating stronger bias and weaker robustness to option-position changes.The recall-based measure is less susceptible to label imbalance than counting predictions.
  • Key observations: Selection bias is prevalent across LLMs and varies with model families, model sizes, and other training-related factors.The authors speculate that training-data composition and ordering, model capacity, and hyperparameters interact in producing the bias.
  • Key observations: The same model shows moderately similar option-ID preferences across domains, although exact preference rankings may differ by task.For example, llama-30B tends toward A/B, whereas gpt-3.5-turbo tends toward C/B.
  • Key observations: In-context examples can reduce selection bias while also introducing new biases that become intertwined with the model’s inherent bias.Llama-30B disfavors C in 0-shot evaluation but becomes biased toward C in 5-shot evaluation.
  • Causes: Shuffling option IDs leaves RStd little changed, whereas removing option IDs notably reduces it across models, benchmarks, and option counts, implicating token bias as a primary cause.Replacing A/B/C/D with alternative symbols also produces no remarkable RStd reduction.
  • Debiasing attempts: Removing option IDs usually degrades model performance, making likelihood-based cloze prompting less practical despite its lower selection bias.The remaining position bias varies irregularly with models and tasks, and simple debiasing prompts or Chain-of-Thought prompting do not mitigate selection bias well.

3 METHODOLOGY

The methodology develops permutation-based debiasing and PriDe, which separates option-ID prior bias from debiased option-content predictions and estimates transferable priors from a small sample subset.

  • 3.1 PERMUTATION-BASED DEBIASING BASELINE AND FORMULATION: Permutation-based debiasing averages prediction distributions across option permutations to cancel token and position bias.Full permutation requires ×n! inference costs, while Cyclic Permutation reduces this to ×n and preserves one pairing between each option ID and option content.
  • 3.2 PREDICTION PROBABILITY DECOMPOSITION: PriDe assumes observed predictions decompose into an option-ID prior and a debiased option-content distribution.The prior incorporates token and position bias, while the debiased distribution reflects belief about option content independently of its ID.
  • 3.2 PREDICTION PROBABILITY DECOMPOSITION: The derivation assumes debiased predictions are invariant to option ordering and the option-ID prior depends only on the question.These assumptions simplify the decomposition used to estimate and remove the prior.
  • 3.3 DEBIASING WITH PRIOR ESTIMATION: Because selection bias shows moderate cross-domain similarity within a model, priors estimated on partial test samples can transfer across remaining samples and domains.This transfer avoids additional computational overhead after prior estimation.
  • 3.3 DEBIASING WITH PRIOR ESTIMATION: PriDe estimates sample-specific priors by permutation-debiasing K test samples, then averages them into a global prior for the remaining samples without labels.When K is much smaller than the test set, prior-estimation overhead becomes negligible relative to total inference cost.
  • 3.3 DEBIASING WITH PRIOR ESTIMATION: Figure 4 varies computational cost using β for Cyclic/Full Perm and α for PriDe’s prior-estimation samples; PriDe is not plotted at α = 100% because it becomes Cyclic Permutation.The figure averages 0-shot results across 20 LLMs.

4 EXPERIMENTS

Experiments evaluate PriDe’s debiasing effectiveness, computational efficiency, prediction effects, and cross-domain transfer. PriDe generally improves debiasing with low overhead, but large domain gaps can slightly reduce model performance.

  • Main results: PriDe estimates option-ID priors from a small test-sample subset and applies them to remaining samples without labels during inference.The estimation ratio α controls computational overhead.
  • Main results: PriDe achieves superior debiasing effectiveness and performance improvement to Full/Cyclic Perm, especially in low-cost settings.The result also holds under the 5-shot setting.
  • Main results: Estimated priors correlate with empirical selection bias and remain stable when estimated from 2% to 20% of test samples.These patterns support interpreting and reliably estimating selection bias with limited computational budget.
  • Main results: PriDe can combine with two or three random permutations to improve debiasing effectiveness and efficiency at approximately ×2 or ×3 cost.These alternatives are cheaper than Cyclic Perm at ×n or Full Perm at ×n!.
  • Transferability analysis: Estimated priors exhibit reasonable transferability across the evaluated MMLU and ARC domains.The study estimates priors from source-domain samples and applies them to other domains.
  • Transferability analysis: Large domain gaps, such as transfer from STEM or Humanities to ARC, may cause slight model-performance degradation.The authors suggest updating the prior with new samples when predictable domain shifts occur.

5 RELATED WORK

The related work situates MCQs as widely used in LLM evaluation and distinguishes this study’s selection bias from broader bias research. It also contrasts PriDe with contemporaneous position-bias studies through broader empirical coverage and a token-bias analysis.

  • MCQs: MCQs are widely used in LLM benchmarks and automatic evaluation frameworks.Examples include MMLU, ARC, and frameworks that compare model-generated answers.
  • Bias and Robustness of LLMs: This study uses “bias” for systematic error in LLMs rather than cultural or gender prejudice studied in LLM safety.Prior work linked language-model bias to sensitivity to instructions, examples, and answer ordering.
  • Bias and Robustness of LLMs: Pezeshkpour and Hruschka observed sensitivity to option positions but did not ablate option-ID effects.Their evaluation covered fewer models and tasks than this study’s cross-model and cross-task analysis.
  • Bias and Robustness of LLMs: This study identifies token bias as a more salient intrinsic cause of selection bias than position bias, which varies with models and tasks.The authors use this broader evidence to motivate PriDe.

6 CONCLUSION

The conclusion identifies selection bias as a source of LLM vulnerability to option-position changes in MCQ evaluation. It attributes the bias primarily to token bias and partially to position bias.

  • Conclusion: Selection bias makes LLMs vulnerable to option-position changes in MCQ evaluation.The bias reflects preference for particular option IDs when predicting answers.
  • Conclusion: Token bias is identified as the primary source of selection bias, while position bias contributes partially.Token bias refers to assigning more prior probability to specific option-ID tokens.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by releasing preprocessed evaluation data, preprocessing scripts, experimental code, and evaluation results. It also documents the hardware and model-access settings used for the experiments.

  • Reproducibility statement: Preprocessed MMLU, ARC, and CSQA data, preprocessing scripts, experimental code, and evaluation results are released.The releases are intended to facilitate reproduction when running the experiments from scratch is expensive.
  • Reproducibility statement: Experiments used A100 40GB GPUs for 70B models and V100 32GB GPUs for other models.Open-source models were available through HuggingFace and gpt-3.5-turbo-0613 through the OpenAI API.

A PROMPTS USED IN EXPERIMENTS

The experiments use distinct prompt formats for open-source models and gpt-3.5-turbo, with option-ID probabilities serving as predictions when IDs are present. When option IDs are removed, models score options by length-normalized likelihood or generate the selected option for matching.

  • Prompt formats: Open-source models use templated inputs and predict the answer from next-token probabilities for option IDs at the designated output position.The prompts omit a beginning-of-sequence token for these models.
  • Prompt formats: gpt-3.5-turbo receives an API-style prompt containing in-context or test samples, with the expected returned answer explicitly marked.The MMLU subject is included as part of the request.
  • Option IDs removed: Without option IDs, open-source models select the option with the maximum length-normalized likelihood.This changes prediction from option-ID token selection to comparing option contents.
  • Option IDs removed: Without option IDs, gpt-3.5-turbo generates the complete selected option, which is compared with the golden answer.The model is evaluated by the generated option text rather than an option-ID token.
  • Prompt formats: The chain-of-thought prompt for gpt-3.5-turbo follows OpenAI Evals with decoding temperature set to 0.This provides a deterministic decoding setting for the experiment.

B EVALUATION RESULTS OF SELECTION BIAS

Supplementary evaluations examine selection bias across multiple model families and reduced-choice settings. The figures indicate that removing option IDs notably reduces selection bias for most models, with several model-specific exceptions.

  • Model evaluations: Llama and vicuna-v1.3 models show selection bias measured by recall scores normalized by subtracting overall accuracy.The figure uses recall imbalance rather than raw answer frequencies alone.
  • Model evaluations: Llama-2, llama-2-chat, and vicuna-v1.5 models are evaluated for selection bias across their option selections.The figure compares these model families under the same selection-bias analysis.
  • Model evaluations: Falcon and gpt-3.5-turbo are evaluated for selection bias, while the authors conjecture that falcon-7B and falcon-inst-7B are undertrained.The conjecture is supported by pronounced bias, near-random MMLU performance, and recalls for position D close to zero.
  • Reduced-choice settings: In the 3-option setting, removing option IDs notably reduces selection bias for most LLMs, except llama-30B, vicuna-v1.3-33B, and llama-2-13/70B.The setting randomly samples three options, including the golden answer.
  • Reduced-choice settings: In the 2-option setting, removing option IDs notably reduces selection bias for most LLMs, except llama-2-13B and falcon(-inst)-7B.The latter falcon models are described as likely undertrained.

C SUPPLEMENTARY EXPERIMENTAL RESULTS

Supplementary experiments examine permutation choices, prior estimation, computational cost, transferability, and prediction changes for PriDe and related debiasing methods. They also report methodological caveats and detailed evaluation breakdowns.

  • Permutation analyses: Different selected permutations in Cyclic Permutation produce no remarkable debiasing differences on 0-shot MMLU and ARC.The permutations preserve one pairing between each option ID and option content.
  • Prior estimation: PriDe-estimated priors are usually strongly correlated with empirical selection bias, but correlation is moderate for several llama-2 and vicuna models.The authors conjecture that ideal alignment may require well-calibrated probabilistic predictions.
  • Evaluation breakdowns: The supplementary tables break down debiasing results under 0-shot and 5-shot settings using recall standard deviation and accuracy.These tables report RStd and Acc for the debiasing evaluation.
  • Robustness checks: Shuffling default options produces no consistent and remarkable changes in recall standard deviation.The comparison uses dashed bars before shuffling and solid bars after shuffling.
  • Cost and transferability: The supplementary figures evaluate 5-shot debiasing under varying computational costs and cross-domain transfer settings.Cross-domain transferability is slightly degraded under 5-shot evaluation, plausibly because in-context examples are added or altered.
  • Low-cost debiasing: The 2/3 Perm alternatives use subsets of standard Cyclic Permutation and control cost through the ratio β of debiased test samples.The ratio β ranges from 0% to 100%.
  • Combined debiasing: Combining PriDe with 2/3 Perm first estimates a prior on 5% of test samples, then applies permutation and prior-based debiasing to the remaining samples.The procedure varies the total fraction α receiving the combined treatment.
  • Prediction changes: Prediction-change analyses count wrong-to-wrong, correct-to-wrong, and wrong-to-correct transitions after debiasing, alongside probability changes.Additional tables rank original and debiased predictions within the other distribution.

D SUPPLEMENTARY PROOF FOR PERMUTATION-BASED DEBIASING BASELINE

The supplementary proof derives the soundness of permutation-based debiasing from the probability decomposition assumption. The derivation rewrites inverse mappings, aggregates logarithms across permutations, and states an approximation condition.

  • Proof strategy: The probability decomposition assumption is used to provide a theoretical proof for the permutation-based debiasing baseline.The proof relies on the inverse mapping between option IDs and option contents.
  • Derivation: Summing logarithmic terms over the permutation set yields a relation involving the debiased probability and an additive constant.The displayed derivation includes the factor |I| multiplying log P_debiased(oi|q, x).
  • Derivation: The derivation aggregates observed probabilities after applying the inverse permutation mapping to option IDs and contents.The relevant terms are expressed using P_observed(d_gI(i)|q, x_I).
  • Assumption: The approximation holds when the variance of the observed probabilities across permutations is not too large.This condition limits when the proof’s approximation is expected to apply.

E EVALUATION DATA AND STATISTICS

The evaluation uses benchmark data with a documented sampling constraint and excludes a small set of MMLU items to preserve the validity of option-position manipulations.

  • gpt-3.5-turbo evaluation on MMLU uses 5,700 randomly sampled test samples, or 40% of the original size, because of budget constraints and rate limits.
  • About 3.2% of MMLU samples are excluded because their options refer to one another, such as “A and B” or “none of the above”.The exclusion applies across the experiments.
Loading 2309.03882v4…