Overwhelmed by Choice: Studying LLM Decision Making at Scale

1 Introduction

• Most multiple-choice evaluations use fewer than ten options, despite deployed tasks such as medical coding and tool selection involving large candidate spaces.Source: §1 · p. 1
• Accuracy consistently declines as candidate counts increase across datasets, prompting strategies, model scales, and decision-making tasks.The evaluated tasks include reranking, medical diagnosis, long-context classification, and information extraction.Source: §1 · p. 1
• Controlled analyses find that long-context interference and positional-bias explanations alone cannot fully account for degradation under large candidate sets.Large-option decisions require comparing and prioritizing candidates, beyond locating relevant information in a long context.Source: §1 · pp. 1–2
• Gold-margin collapse and early-option dominance are distinct failure patterns: correct-answer confidence weakens, while later candidates exert less influence on predictions.The score gap between the correct answer and strongest distractor shrinks primarily because confidence in the correct answer falls.Source: §1 · p. 2
• At N = 160, hierarchical partitioning and permutation-based inference improve accuracy by roughly 20 percentage points over standard single-pass inference on both HotpotQA and MIMIC.The interventions restructure comparison into smaller local stages or aggregate predictions across shuffled candidate orders.Source: §1 · p. 2

2.1 Experimental Setup

• The setup evaluates single-answer selection across reranking, scientific classification, medical diagnosis, and long-context reasoning tasks.Models are evaluated under standard zero-shot prompting and several reasoning settings, with candidate counts from N = 5 to 160.Source: §2.1 · pp. 2–3

2.2 Accuracy Degradation at Large N

• Accuracy consistently decreases as options grow across tasks and model families, with the same trend across additional datasets.Source: §2.2 · p. 3, Figure 2 · p. 3
• The degradation remains robust under bootstrap confidence intervals and 1,000-example evaluations, and persists at N = 640 and for a strong closed-source model.Its magnitude varies across models and tasks.Source: §2.2 · p. 3
• Chain-of-thought, self-critique, few-shot prompting, and pairwise comparison can improve absolute accuracy, but all continue to degrade as candidate sets grow.Source: §2.2 · p. 3, Figure 3 · p. 4
• As options grow, the gold margin decreases while the gold score collapses substantially faster than the distractor score.Source: Figure 4 · p. 4

3.1 Gold-Margin Collapse Under Large Candidate Sets

• As candidate sets grow, the gold margin shrinks and can become negative, primarily because the gold score falls while distractor scores remain relatively stable.The gold margin is the gold score minus the highest-scoring distractor; distractor mass is the log-sum-exp of distractor scores.Source: §3.1 · p. 4

3.2 Context Length Alone Does Not Fully Explain Large-Option Degradation

• Increasing irrelevant context length and distractor verbosity do not consistently reproduce the degradation induced by candidate-set expansion.The results suggest context length alone cannot explain the stable degradation under candidate-set scaling.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 Attention Distribution Does Not Directly Reflect Prediction Decisions

• Prediction distributions remain front-skewed across models despite substantially different attention patterns, suggesting attention allocation does not directly explain prediction-position behavior.The discrepancy also persists across candidate-set sizes and datasets.Source: §3.3 · p. 5, Figure 5 · p. 5

3.4 Early-Option Dominance in Large-Option Comparison

• As candidate sets grow, later-positioned gold answers become harder to recover and exert less influence, while earlier distractor preferences increasingly persist.Later gold insertions reduce gold probability and its suppressive effect on the pre-anchor distractor; the trend is consistent across datasets and models despite minor edge-position effects.Source: §3.4 · p. 6

4.1 Overview

• The interventions restructure comparison through staged selection over small groups or aggregation across shuffled candidate orders, with CoT included as a reasoning baseline.The overview describes these as inference-time interventions motivated by gold-margin collapse and early-option dominance.Source: §4.1 · p. 6, Figure 7 · p. 7

4.2 Intervention Results

• At N = 160, Hierarchy and Permutation each improve accuracy by roughly 20 percentage points over Single-Pass on HotpotQA and MIMIC.Their gains increase with option count; CoT improves absolute accuracy but still degrades substantially at larger option sizes.Source: §4.2 · p. 7, Figure 8 · p. 7

4.3 Mechanism Analysis

• Hierarchy and Permutation maintain larger gold margins and weaker early-position bias than Single-Pass as candidate sets grow, unlike CoT.Permutation has lower final-round gold margins at small N because early-agreement examples leave a more difficult subset for the final comparison.Source: §4.3 · p. 8

MCQ-based Benchmarking

• Traditional MCQ benchmarks typically use few answer choices [Hendrycks et al., 2020, Rein et al., 2023, Zhang et al., 2024], while prior work reports declines with more options, stronger distractors, or greater reasoning demands [Wang et al., 2024, Zhang and Nguyen, 2025, Elhady et al., 2025].Source: §5 · p. 8

Long-Context Reasoning and Positional Bias

• Prior long-context research studies interference, retrieval degradation, and positional bias [Hong et al., 2025, Shi et al., 2023, Du et al., 2025, Lee et al., 2025, Liu et al., 2024, Hsieh et al., 2024, Wu et al., 2025], chiefly in retrieval-oriented settings.Source: §5 · p. 8

Decision-Making Degradation and Mitigation

• Related work examines distractor competition and selection bias [Elhady et al., 2025, Amanlou et al., 2026, Zheng et al., 2023] alongside inference-time mitigation [Parekh et al., 2025, Gan and Sun, 2025, Zheng et al., 2026].This paper connects large-option degradation to comparison-level failure patterns and evaluates interventions that restructure comparison.Source: §5 · p. 8

• Candidate-set scale is an important evaluation variable because strong small-option performance may not reliably reflect scalable candidate-comparison ability.Source: §6 · p. 8

7 Limitations

• The scope is primarily English, text-only, single-answer candidate-selection tasks, with analysis focused on inference-time behavior.Extending the findings to multilingual, multimodal, and interactive settings remains future work.Source: §7 · p. 8
• The controlled experiments characterize systematic gold-margin collapse and early-option dominance but do not establish a complete causal account of underlying model mechanisms.Source: §7 · p. 8
• The proposed restructuring strategies require additional inference computation, and more compute-efficient interventions remain future work.Source: §7 · p. 8

8.1 Dataset Statistics

• Table 2 summarizes the appendix datasets' task types, average option lengths, and total prompt lengths at N=160.For each dataset, the large-option benchmark setting samples 200 evaluation instances.Source: §8.1 · pp. 10–11, Table 2 · p. 11

8.2 Additional Datasets for Accuracy Degradation

• Accuracy consistently declines as candidate options increase across diverse additional tasks and domains.The appendix dataset statistics report task type, average option length, and total prompt length at N=160.Source: §8.2 · p. 11

8.3 Additional Implementation Details and Results for System-2 Reasoning

• All evaluated System-2 reasoning strategies on Qwen3-4B still degrade substantially as option count increases, despite improving absolute accuracy in smaller candidate settings.The strategies include chain-of-thought, self-critique, few-shot prompting, and pairwise comparison.Source: §8.3 · p. 11, Figure 10 · p. 12

8.4.1 Statistical Robustness and Larger Evaluation Samples

• On HotpotQA, Phi-4-mini accuracy drops from 83.0% to 19.5%, and on MIMIC it drops from 85.0% to 9.0%, between N = 5 and N = 160.The observed degradation is substantially larger than bootstrap uncertainty and persists with a fivefold larger evaluation set.Source: §8.4.1 · pp. 11–13, Table 3 · p. 13

8.5 Robustness of Gold-Margin Analysis

• Gold-margin collapse remains under length-normalized scores and probability-space analysis, so it is not an artifact of summed token-level log-probabilities.Normalized margins shrink from N = 5 to N = 160 across all three HotpotQA models, becoming negative for Llama-3.1-8B and Phi-4-mini.Source: §8.5 · p. 13, Table 4 · p. 14

8.6 Irrelevant Contents

• Longer distractor options do not consistently reproduce the degradation caused by increasing the number of candidates, so context length alone cannot fully explain it.The control uses neutral background text to vary context length without adding task-relevant information.Source: §8.6 · pp. 13–14

8.4.2 Extended Candidate-Set Scaling and Closed-Source Models

• GPT-5.4 accuracy falls from 98.5% at N = 5 to 49.0% at N = 640, while all evaluated models continue to decline beyond N = 160.The degradation also persists in open-source models, which reach approximately 2%, 0%, and 9% accuracy at N = 640 for Llama-3.1-8B, Phi-4-mini, and Qwen3-30B, respectively.Source: §8.4.2 · p. 13, Figure 11 · p. 14

8.7 Distractor Verbosity Control

• Long distractors increase prompt length but produce less consistent degradation than increasing the number of candidate options.This supports that context length alone cannot fully explain large-option degradation.Source: §8.7 · p. 14, Figure 12 · p. 15

8.8 Attention-Prediction Discrepancies Across Candidate Sizes and Datasets

• Prediction distributions remain front-skewed even when attention allocation varies substantially and sometimes shifts toward later options.The observations support that prediction behavior cannot be directly explained by attention allocation patterns alone.Source: §8.8 · pp. 14–15, Figure 13 · p. 15

8.9 Prediction Positional Bias Analysis

• Predictions shift toward earlier candidate positions as options increase, despite stable gold-position distributions; the trend persists under reversed indexing.The results suggest the bias primarily follows candidate ordering rather than preferences for specific answer indices.Source: §8.9 · pp. 15–17, Figure 15 · p. 17, Table 5 · p. 17

8.10 Additional Probability Dynamics under Controlled Gold Insertion

• As options increase, prediction mass becomes less responsive to the gold insertion region and concentrates more toward earlier candidates.Source: §8.10 · p. 17, Figure 17 · p. 18

8.11 Early-Option Dominance Generalization

• On arXiv, early-position preference persists under reversed indexing, and later-inserted gold answers become progressively less able to overturn earlier preferences.With distractors fixed, predictions become less sensitive to the gold insertion position as the option count grows; stronger models may show a mild late-position rebound.Source: §8.11 · pp. 17–18, Figure 16 · p. 18

8.12 Additional Revision Dynamics Examples

• Across Llama-3.2-3B and Qwen3-4B, later-position gold answers progressively lose the ability to overturn earlier candidate preferences, with only occasional mild probability recovery near the final positions.Source: §8.12 · p. 18, Figure 22 · p. 21

8.13 Additional Implementation Details

• Hierarchical partitioning repeatedly selects winners from groups of size K until one candidate remains, while permutation inference aggregates selections from shuffled candidate orders.Single-pass inference instead asks the model to choose one answer from all N options in one prompt.Source: §8.13 · pp. 18–20

8.14 Mechanism Analysis of Intervention

• For hierarchical partitioning, the gold candidate's chance of surviving to the final round decreases as options grow, shifting risk to irreversible early elimination.Permutation inference increasingly preserves the gold among multiple remaining candidates, so final-stage comparison quality becomes a key determinant of accuracy.Source: §8.14 · p. 21
• Across five random seeds, permutation inference at N = 160 achieves 54.38 ± 1.00% accuracy, with accuracy standard deviations of 0.67–1.33 percentage points across candidate sizes.The low run-to-run variance indicates the reported improvement is not driven by a particular permutation choice.Source: §8.14 · p. 21
• On HotpotQA with Llama-3.1-8B at N = 160, permutation inference reaches 76.5% accuracy (+19.0 points) and hierarchical partitioning reaches 71.0% (+13.5 points) versus single-pass 57.5%.Permutation inference averages 5.76 calls and 8.74 seconds per example; hierarchical partitioning uses 42.0 calls and 6.04 seconds under the reported batched implementation. Hierarchical partitioning risks irreversible early elimination, while permutation inference can preserve the gold candidate alongside competitors, making final comparison important.Source: §8.14 · pp. 21–22, Table 6 · p. 22, Figure 23 · p. 22
• The interventions address distinct challenges: hierarchical partitioning reduces comparison complexity, while permutation inference reduces sensitivity to candidate ordering.Source: §8.14 · pp. 20–22

Empirical Validation Across Models

• Both methods consistently mitigate accuracy degradation as option counts rise, including across Qwen3-30B and Phi-4-mini.The trend indicates their effectiveness is not tied to model scale.Source: §8.15 · p. 22, Figure 24 · p. 23
• Both methods improve robustness by reducing comparison complexity or alleviating sensitivity to candidate ordering.Source: §8.15 · p. 23

Effect of Group Size and Number of Permutations

• Larger hierarchical groups generally perform better across option counts; Group-10 and Group-20 outperform smaller groups, which may lack candidate diversity for early selections.Source: §8.15 · p. 23, Figure 25 · p. 23
• Hierarchical partitioning only mitigates the comparison bottleneck: all tested configurations still degrade as the number of options increases.Source: §8.15 · p. 23
• Permutation-based inference benefits from a moderate number of shuffled views, while excessive permutations can add aggregation noise and slightly reduce performance in some cases.Source: §8.15 · p. 23, Figure 25 · p. 23
• The methods involve different hyperparameter trade-offs: larger groups preserve strong candidates, while permutation-based inference performs best with a moderate number of diversified views.Source: §8.15 · p. 23