PaperlayerView exact revision on arXiv

Source-linked AI summary

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Karl Hanna, Chen Feng

arXiv:2608.11947v1cs.CLcs.AI

TL;DR

MCQ scores conflate model knowledge with sensitivity to option order, motivating label-free strategies to separate answer commitment from option presentation. Across six models and two benchmarks, neither tested strategy reliably improved accuracy, while eliminating positional influence still did not reliably produce gains.

  • Problem

    MCQ accuracy does not distinguish model knowledge from sensitivity to option order, limiting the trustworthiness of this evaluation format.

  • Method

    The paper evaluates two label-free prompting strategies against established baselines across six models and two benchmarks, using accuracy and order-sensitivity diagnostics.

  • Results

    Neither strategy reliably improves accuracy; independent hypothesis eliminates positional influence by construction, yet accuracy still does not reliably improve.

  • Takeaways & Limitations

    Eliminating positional influence and improving accuracy come apart, while withholding options rather than matching is the bottleneck in two-stage prompting.

  • Takeaways & Limitations

    Results are limited to four-option benchmarks with fixed question counts, and single-run evaluation leaves run-to-run provider nondeterminism untested.

Abstract

from arXiv · show

Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.

1 Introduction

The paper tests whether hiding option labels can reduce order sensitivity without sacrificing multiple-choice accuracy. Across two label-free strategies, accuracy and positional robustness diverge: neither reliably improves accuracy, and the main bottleneck is withholding options rather than matching answers.

  • MCQ accuracy conflates model knowledge with sensitivity to option order, limiting its reliability as an evaluation measure.This motivates evaluating robustness alongside accuracy.
  • Prompt-based robustness is motivated by the cost of cyclic or full permutation and the logit-access requirement of PriDe.Cyclic permutation costs k calls, full permutation k!, while PriDe requires logits.
  • Neither two-stage prompting nor independent hypothesis scoring reliably improves accuracy; two-stage prompting usually reduces it, while gains for Llama-local concentrate on ARC-Challenge.Independent hypothesis scoring has mixed results, reducing accuracy for most models but improving Llama-local.
  • The two-stage decomposition shows that hiding options in Stage 1, rather than the matching step, is the primary bottleneck.Replacing the Stage 2 LLM matcher with semantic matching while hiding options causes a sharp performance drop, whereas visible options recover much of the loss.
  • Two-stage prompting does not reliably improve flip rate or recall standard deviation, and reduced positional sensitivity does not necessarily produce higher accuracy.The paper therefore treats order sensitivity and accuracy as separable outcomes.

2 Related Work

Prior work shows that multiple-choice evaluation is vulnerable to option-order sensitivity, selector bias, and other formatting artifacts. Related approaches seek robustness through debiasing, open-style answering, answer matching, and structurally order-invariant scoring, positioning this paper at their intersection.

  • Multiple-choice brittleness and option-order sensitivity: Multiple-choice benchmarks can reflect option-order sensitivity and formatting rather than stable knowledge, while models may select the least incorrect option.Reordering options can substantially change performance, and hidden or disguised biases may further distort option-based evaluation.
  • Debiasing and robustness-oriented prompting methods: Debiasing methods modify prompting or prediction distributions to reduce bias in multiple-choice answering.PriDe estimates a prior from a small calibration set and subtracts it from the prediction distribution; BiasPrompting proposes a prompting-based intervention.
  • Open-style answering and answer matching: Open-style answering and answer matching challenge forced-choice evaluation as the measurement interface for model knowledge.Prior work argues that multiple-choice constraints can distort measurement and reports that answer matching outperforms standard multiple-choice evaluation.
  • LLM-as-judge and positional bias: LLM-based answer matching may inherit positional bias because it functions as an LLM-as-judge comparison over labelled options.LLM judges have exhibited position, verbosity, and self-enhancement biases, including order-inconsistent verdicts when response quality is similar.
  • Isolated per-option scoring: Structurally isolated option scoring can eliminate order effects, but Set-Based Prompting typically leaves accuracy essentially unchanged.Its accuracy effect is small and generally remains within ordinary reordering variation.
  • Positioning of this work: This paper connects research on MCQ artifacts with proposals for open-form answering, answer matching, and order-robust evaluation.Its positioning follows prior findings on option-order effects, selector bias, formatting-sensitive artifacts, and alternative measurement interfaces.

3 Methodology

The methodology evaluates six language models on sampled MMLU and ARC-Challenge questions using two label-free strategies, baseline and permutation-based comparisons, and accuracy and positional-sensitivity diagnostics. It defines independent scoring to remove presentation dependence by construction while measuring end-to-end, conditional, and flip-rate outcomes.

  • Evaluation setup: The study evaluates six models, including API-served and local Qwen 2.5 and Llama 3.1 variants, on sampled benchmark questions.MMLU uses 20 questions from each of 50 subjects, sampled with seed 42; seven subjects are excluded.
  • Strategies: Two-Stage Prompting generates a free-text response without options, then reintroduces the options for prediction, so it is not positionally invariant.Its Stage 2 prediction remains dependent on the joint option presentation ϕ.
  • Strategies: Independent Hypothesis scores each option separately and selects the highest-scoring option, making the final prediction independent of option ordering and labels.Independent scoring can make questions such as “which of the following” underspecified when other options are hidden.
  • Comparison conditions: Comparisons include the direct multiple-choice baseline, cyclic permutation with majority voting, and PriDe positional-prior calibration.Cyclic permutation queries the model across rotated option positions, unpermutes responses, and resolves ties by defaulting to the original permutation.
  • Evaluation metrics: End-to-end accuracy counts unscorable outputs as incorrect, conditional accuracy excludes them, and flip rate counts questions with at least two distinct semantic answers across permutations.Questions receive four permutations, except ARC-Challenge questions with three options, which receive three.

4 Results

Label-free strategies generally reduced or failed to improve accuracy, while cyclic permutation often improved it. The main bottleneck was withholding options rather than matching, and neither aggregate nor per-question order-sensitivity measures showed reliable debiasing.

  • Accuracy: Two-stage reduced accuracy in 11 of 12 model–benchmark pairs and independent hypothesis in 8 of 11, whereas cyclic improved 5 of 6 pairs on each benchmark.The sole two-stage increase was Llama-local on ARC, from 58.0 to 58.3, within noise.
  • Parse failures: Two-stage Gemini had 82.0 conditional accuracy but only 68.3 end-to-end accuracy, a 13.7-point parse-failure gap; independent hypothesis scored 1,000/1,000 included questions.Fallback analysis found Gemini’s loss parse-driven, but its recovered score of 79.6 remained below the 84.9 baseline.
  • Order sensitivity: Two-stage did not reliably reduce either RStd or flip rate: RStd increased in 5 of 12 pairs, while flip rate increased in 7 of 12.For Qwen-local, flip rate rose by +5.2 pp on MMLU and +12.2 pp on ARC while RStd fell from 8.06 to 4.27 and 4.68 to 3.05, respectively.
  • Matching and options: Hiding options, not the matcher, caused the major loss: GPT-4.1 mini on MMLU rose from 45.8 to 81.7 when options were shown with semantic matching.Visible-options semantic matching generally matched baseline within noise, while Llama-local gained 6.5 pp on MMLU and 12.4 pp on ARC.
  • Accuracy gains: Llama-local’s apparent 14.4 pp independent-hypothesis gain on ARC was not unique: cyclic reached 72.9, independent hypothesis 72.4, and Visible+LLM 70.4 versus a 58.0 baseline.On MMLU, independent hypothesis reached 51.2 from a 50.2 baseline, while cyclic reached 57.0 and Visible+LLM 56.7.
  • Robustness: Changing either the Stage 1 or Stage 2 prompt preserved the negative pattern: Gemini scored 70.2 and 70.7 versus 68.3 originally and an 84.9 baseline.Thus, the result was robust to both prompt variations.

5 Discussion

The discussion attributes two-stage performance loss primarily to withholding options during Stage 1, while independent hypothesis may lose comparative context. Across strategies, reduced positional sensitivity does not reliably improve accuracy, and cyclic permutation offers the strongest model-agnostic results despite higher cost.

  • Two-stage prompting: 35.9 pp separates GPT-4.1 mini’s MMLU accuracy with semantic matching when options are hidden versus visible.This supports the claim that withholding options, rather than the matching step, drives two-stage performance loss.
  • Independent hypothesis: Independent hypothesis may perform poorly because scoring each option alone removes comparative context, while models exploit group dynamics among options.Prior work similarly finds that individual-choice priors do not fully explain choices-only accuracy.
  • Accuracy and order sensitivity: Reduced order sensitivity under two-stage does not translate into higher GPT-4.1 mini accuracy on MMLU, and cyclic permutation performs better.Removing option identifiers can reduce selection bias while usually degrading accuracy.
  • Cost and performance: Independent hypothesis loses to cyclic permutation in 10 of 11 pairs despite equal call counts, while cyclic is the best-performing model-agnostic method evaluated.Baseline uses one call per question, two-stage uses two, and cyclic and independent hypothesis each use k calls.

6 Conclusion

Across six models and two benchmarks, neither label-free strategy reliably improved accuracy, with two-stage prompting and independent hypothesis usually degrading performance. The bottleneck was withholding options rather than matching, while position-blind methods and cyclic permutation produced divergent accuracy outcomes.

  • Neither strategy reliably improves accuracy: two-stage prompting degrades performance in 11 of 12 model–benchmark pairs, while independent hypothesis degrades it in 8 of 11 valid pairs.
  • The 2 × 2 decomposition identifies withholding options, rather than matching, as the bottleneck; only visible Stage 1 options paired with an LLM matcher consistently match baseline.
  • Two-stage diagnostics show no reliable debiasing: RStd increases in 5 of 12 pairs, flip rate increases in 7 of 12, and they move together in 8 of 12.
  • Independent hypothesis removes positional influence by construction without reliable accuracy gains, semantic matching yields exactly zero flip rate but substantially lower accuracy, and cyclic permutation improves accuracy in 10 of 12 pairs.

Limitations

The study’s conclusions are limited by its narrow evaluation scope, single-run and single-instantiation designs, and incomplete testing of reasoning, matching, and calibration choices. Parse failures, provider-side failures, and data irregularities further restrict which questions and conditions support the reported metrics.

  • Evaluation scope and variance: The evaluation covers six models on two benchmarks in a four-option setting with fixed question counts, limiting generalization to other option counts, distributions, or larger models.Each condition was run once, except for the independent-hypothesis tie-break seed sweep, so run-to-run provider non-determinism was not assessed.
  • Evaluation scope and variance: The flip-rate experiment lacks repeated identical-order controls, leaving residual provider-side effects untested.The supplied passage ends before specifying the resulting residual effect.
  • Strategy design: Each strategy was evaluated in only one instantiation, and no condition allowed Stage 1 reasoning before answering in the two-stage strategy.The study used one free-text prompt, one matching prompt, and two stage-specific ablations; it also did not test a separate or stronger matcher model.
  • Strategy design: PriDe’s behavior was evaluated at only one calibration budget because calibration-set size was not varied.This leaves the effect of calibration budget unexamined.
  • Data quality and interpretation: Parse failures reduced scored-question counts in several cells, so flip-rate and RStd values reflect subsets and may inflate apparent position-blindness.The affected cells include semantic matching across all six models and Gemini’s two-stage cells on both benchmarks; the under-specification mechanism was supported by prior work rather than tested on the study’s own data.
  • Data quality and interpretation: Independent hypothesis on Gemini and ARC-Challenge scored only 313 of 1,000 questions because of extensive provider-side API failures, while three ARC-Challenge items had three options and a rendered placeholder fourth option.Parse-failure rates also depend on provider-specific serving behavior, and the supplied passage continues with an incomplete statement about Gemini.

Ethical Considerations · Appendix · A Prompt Templates

The ethical risk is minimal because the study uses public benchmarks without human subjects or new data collection, but ineffective debiasing methods should not be treated as safety guarantees. The appendix documents reproducible prompt templates for baseline, permutation, two-stage, ablation, and text-extraction methods.

  • Ethical Considerations: AI assistance supported code refactoring, debugging, feature implementation, manuscript wording, formatting, consistency checks, and claim clarification, while authors retained responsibility for the research and final claims.
  • Ethical Considerations: The study reports minimal ethical risk because it uses public benchmarks without human subjects or new data collection.The authors caution that ineffective debiasing methods should not be relied upon as safety guarantees.
  • A Prompt Templates: Prompts are stored under prompts/{version}/ and snapshotted into each run directory, with curly-brace tokens filled at runtime for reproducibility.
  • A Prompt Templates: Baseline, Cyclic Permutation, and PriDe use one prompt, while cyclic rotates label contents across four calls and PriDe uses four calibration calls plus one inference call.
  • A Prompt Templates: The two-stage method elicits a short free-text answer without options, then injects it into a matching prompt that selects the closest labeled option.
  • A Prompt Templates: A detail-elicitation ablation requests distinguishing information without reasoning explanations, whereas semantic matching prioritizes meaning and handles incomplete or ambiguous references.
  • A Prompt Templates: Text extraction exposes all options but forbids letter output, then matches generated answer text by exact match, substring containment, or embedding cosine similarity at a 0.30 threshold.The runtime path uses all-MiniLM-L6-v2 and excludes rapidfuzz, which is reserved for offline diagnostics.

B PriDe Implementation Details

PriDe estimates a positional-bias prior on disjoint calibration questions and transfers it to evaluation predictions. The implementation supports three logprob-exposing models and normalizes model outputs into four-option distributions.

  • Models: PriDe is evaluated on three logprob-exposing models: Qwen/Qwen2.5-7B-Instruct-Turbo, Qwen/Qwen2.5-7B-Instruct, and meta-llama/Llama-3.1-8B-Instruct.The first uses the Together AI API; the latter two run locally.
  • Logprob extraction: Logprob extraction prefills "The answer is " so the first generated token is usually an answer letter, with top_logprobs=20 requested for coverage.The implementation parses both standard OpenAI and Together AI response formats into option-letter log-probabilities.
  • Phase 1: Calibration: Calibration uses K = 50 questions sampled from a pool excluding every evaluation question ID, guaranteeing calibration/evaluation disjointness.With seed = 42, each calibration question receives four API calls, one per cyclic option rotation, producing a 4 × 4 probability matrix.
  • Phase 2: Transfer Debiasing: Per-question cyclic-rotation priors are averaged and renormalized into a cached global prior, then applied to each evaluation question’s baseline-prompt distribution.The debiased distribution uses an elementwise ratio clipped at ϵ = 10−12, renormalizes it, and selects its argmax letter.

C Experimental Configuration

The main evaluation used fixed sampling settings across models, with one longer-token exception, while provider-specific controls governed API calls and a specified software stack supported local inference and statistical analysis.

  • Model and API configuration: All models used temperature = 0.0 and seed = 42, while max_tokens was 500 except for independent_hypothesis, which used 4000.API calls were asynchronous, with provider-specific concurrency, delay, and retry settings to remain within rate limits.
  • Software environment: Local inference used Python 3.10.5, Py-Torch 2.12.0, transformers 5.9.0, and sentencetransformers 5.5.1; statistical analysis used statsmodels 0.14.6, rapidfuzz 3.14.5, numpy, and scipy.The sentence-transformer configuration used all-MiniLM-L6-v2 at revision 1110a243, with statsmodels supporting McNemar tests.

D Independent Hypothesis Implementation … H Fallback Analysis

The paper specifies an option-isolated hypothesis evaluator, a separately implemented visible-options LLM matcher, and diagnostics for order sensitivity, tie-break dependence, and extraction failures. These analyses define how predictions are generated, measured, and diagnostically repaired across the evaluated conditions.

  • D Independent Hypothesis Implementation: Independent-hypothesis evaluation scores each option as an isolated hypothesis rather than presenting options together.The model assesses whether each hypothesis answers the question and emits a 0–100 confidence score in <score>X</score> tags.
  • D Independent Hypothesis Implementation: The independent-hypothesis runner makes one parallel API call per real option, using four calls normally and three for ARC-Challenge questions missing a fourth option.No prompt contains multiple options, and missing options are not scored.
  • D Independent Hypothesis Implementation: Final independent-hypothesis predictions use the maximum confidence score, with exact ties resolved reproducibly by an RNG seeded from the run seed and question ID.The tie-break is independent of asynchronous completion order.
  • D Independent Hypothesis Implementation: The independent-hypothesis condition overrides the default token budget with max_tokens: 4000 because Gemini 2.5 Flash exhausted 500 tokens before emitting score tags on 976/1000 rows.At the smaller budget, Gemini’s initial-run accuracy collapsed to chance level; the override is shared across all four API jobs.
  • F Flip-Rate Protocol: Direct order sensitivity is measured by a per-question any-flip rate across cyclic option permutations for baseline, two_prompt, and twostage_semantic_match diagnostics.A question flips when its parsed answer takes at least two distinct values across n rotations, with n = 4 normally and n = 3 for three-option ARC questions.
  • F Flip-Rate Protocol: The cyclic condition has no separate flip-rate column because its predictions match baseline under each rotation and its majority-voted output cannot flip by construction.For two_prompt, reused Stage-1 completions were held fixed while only order-sensitive Stage-2 matching was rerun per permutation.
  • G Tie-Break Seed Sensitivity: Tie-break seed sensitivity is tested by recomputing predictions for 11 available independent-hypothesis files under 10 alternative seeds without new model inference.Gemini × ARC-Challenge is excluded because provider-side API failures affected 68.7% of rows.

I Full Results Tables

The appendix provides complete MMLU and ARC-Challenge results, including accuracy, robustness, diagnostic, and fallback analyses. It confirms that the main-text patterns generally hold across both benchmarks, while documenting exclusions and alternative evaluation treatments.

  • Accuracy results: Complete end-to-end accuracy tables report MMLU and ARC-Challenge results with 95% Clopper–Pearson confidence intervals, including limited PriDe evaluation where logprob access is available.PriDe is evaluated only for the three models with logprob access; unassessed cells are marked “–”.
  • Scoring and failures: The appendix compares end-to-end and conditional accuracy for methods producing unscorable outputs, counting such outputs as incorrect only in the end-to-end measure.Conditional accuracy is computed over scorable outputs only, for both MMLU and ARC-Challenge.
  • Robustness diagnostics: RStd, flip rates, broken/fixed counts, and McNemar p-values quantify positional recall uniformity, answer changes across cyclic rotations, and question-level changes relative to baseline.RStd uses 10,000-resample bootstrap confidence intervals; semantic matching has a 0.0% flip rate on both benchmarks and is omitted from the flip-rate table.
  • Diagnostic analyses: Stage-1/Stage-2 diagnostic grids cross option visibility with LLM-versus-embedding matching, while the independent-hypothesis table adds tie-break, seed-spread, and RStd analyses.The Gemini 2.5 Flash × ARC-Challenge independent-hypothesis cell is excluded because only 313 of 1,000 questions returned successfully.
  • Fallback analyses: Fallback tables compare original accuracy with scores obtained when unscorable outputs revert to the baseline prediction, excluding independent hypothesis and PriDe because their failure semantics differ.Fallback re-scoring is reported separately for MMLU and ARC-Challenge.
Loading 2608.11947v1…