PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements

Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Jinzhu Wu, Yuhan Wu, Change Jia, Lin Sun, Xiangzheng Zhang

arXiv:2502.12459v3cs.CLcs.AIcs.LG

TL;DR

The paper asks whether high benchmark scores reflect genuine LLM generalization or sensitivity to superficial cues, a question complicated by benchmark contamination. It introduces controlled stress tests that make minor, content-preserving changes to questions, and reports accuracy losses across option length, problem type, and irrelevant noun shifts.

  • Problem

    It remains unclear whether LLMs’ high benchmark scores reflect genuine generalization or exploitation of superficial cues, while data contamination can inflate benchmark estimates.

    Source: §1 · p. 1

  • Method

    Generalization Stress Tests assess LLMs under minor, content-preserving perturbations to option length, problem type, and irrelevant nouns.

    Source: §1 · p. 1

  • Results

    LLMs lose accuracy across the tested perturbations; Qwen 2.5 1.5B’s MMLU score drops from 89 to 36 when option lengths change without changing the question.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that LLMs rely on superficial patterns rather than robust, generalizable reasoning under these changes.

    Source: §5 · p. 5

    The work tests non-chain-of-thought LLMs such as GPT-4o and does not consider emerging O1.

    Source: Limitations · p. 6

Abstract

from arXiv · show

In this paper, we propose a ``Generalization Stress Test" to assess Large Language Models' (LLMs) generalization ability under slight and controlled perturbations, including option length, problem types, and irrelevant noun replacements. We achieve novel and significant findings that, despite high benchmark scores, LLMs exhibit severe accuracy drops and unexpected biases (e.g., preference for longer distractors) when faced with these minor but content-preserving modifications. For example, Qwen 2.5 1.5B's MMLU score rises from 60 to 89 and drops from 89 to 36 when option lengths are changed without altering the question. Even GPT4o experiences a 25-point accuracy loss when problem types are changed, with a 6-point drop across all three modification categories. These analyses suggest that LLMs rely heavily on superficial cues rather than forming robust, abstract representations that generalize across formats, lexical variations, and irrelevant content shifts.

1 Introduction

High benchmark scores leave open whether LLMs generalize or exploit superficial cues, and contamination can inflate benchmark estimates. The paper introduces stress tests using minor content-preserving changes and finds substantial accuracy losses across those changes.

1 Introduction

  • Although LLMs score highly on established benchmarks, it remains unclear whether those scores reflect genuine generalization or reliance on superficial cues.The introduction notes claims of human-like generalization alongside this unresolved question.Source: §1 · p. 1
  • Existing benchmarks may expose patterns rather than assess generalization, and data contamination can inflate performance estimates.Prior perturbation studies also changed original problems, increased evaluation complexity, or focused on limited formatting changes such as option IDs.Source: §1 · p. 1
  • The Generalization Stress Tests examine three minor, content-preserving perturbations: option length, problem type, and irrelevant noun replacement.Examples include lengthening correct or distractor options, converting multiple-choice questions to Boolean questions, and substituting irrelevant nouns.Source: §1 · pp. 1–2
  • Qwen 2.5 1.5B’s MMLU score falls from 89 to 36 after option-length changes, while GPT4o loses 25 points after question-type changes and 6 points across all three categories.The paper reports these results as evidence that models struggle to generalize across option lengths, problem types, and noun replacements.Source: §1 · p. 2

2 Methods: Generalization Stress Tests

The methods apply minor changes to benchmark questions while preserving their content, covering option length, scoring type, and irrelevant nouns. The tests include multiple-choice and open-ended question answering.

2 Methods: Generalization Stress Tests

  • The framework applies minor modifications to benchmark questions across option length, scoring type, and irrelevant nouns, using multiple-choice and open-ended QA tasks.The paper tests GSM-8K for noun replacement because some MMLU cases lack irrelevant nouns.Source: §2 · p. 2

2.1 Alter Option Length to Analyze LLMs’ Length Bias

  • The option-length tests compare lengthening one wrong option (WL) with lengthening the correct option (RL), using controlled length ranges and human verification.GPT4o generates the longer options; WL lengths are below 10, 10–20, or above 20 tokens, and experts check for unintended bias or hints.Source: §2.1 · pp. 2–3

2.2 Change Problem Type to Fairly Analyze LLMs’ Scoring Bias

  • The scoring-type method converts multiple-choice questions to Boolean questions, using one true proposition based on the correct option and one false proposition based on a randomly selected wrong option.The authors choose Boolean questions because cloze questions reduce the expected value of selecting the correct answer.Source: §2.2 · p. 3

2.3 Replace Irrelevant Nouns to Analyze Bias towards Irrelevant Content

  • The noun-replacement method substitutes unrelated nouns, such as people’s or animals’ names, while preserving human decision-making and varying replacement semantic similarity.GPT-4o mini performs replacements with varying degrees of semantic similarity.Source: §2.3 · p. 3

3 Experiments

Across the evaluated benchmarks and models, slight changes to option length, scoring type, and irrelevant nouns expose substantial generalization weaknesses. The experiments include multiple-choice and open-ended QA tasks, with smaller models generally faring worse on the reported perturbations.

3 Experiments

  • The experiments evaluate Llama3.1 and Qwen2.5 models and GPT4o on MMLU and ARC-Challenge multiple-choice tasks and open-ended GSM8K QA.Evaluations use the harness framework’s default setting.Source: §3 · p. 3

LLMs struggle to generalize across option

  • Lengthening the correct option raises scores, while lengthening a wrong option lowers them across models; RL changes can exceed 10 points.The pattern persists when all options are lengthened, and LLMs also tend to select the right answer when all incorrect options are longer.Source: LLMs struggle to generalize across… · p. 4

3.2 Results of Altering Scoring Type

  • Models score lower when MCQs are converted to boolean questions, and smaller models generalize worse across scoring types.Qwen2.5 1.5B and Llama3.1 8B score only half the points in MMLU’s “both” setting.Source: §3.2 · p. 4, Table 3 · p. 4

3.3 Results of Replacing Irrelevant Nouns

  • Renaming irrelevant nouns lowers every model’s score, with similar declines across models; semantically distant replacements reduce effectiveness further.The reported noun-replacement results use GSM8K because it contains irrelevant nouns.Source: §3.3 · p. 4, Table 5 · p. 4, Table 4 · p. 4

4 Discussion

The discussion examines why perturbations affect accuracy, whether simple mitigations help, and how broadly the findings apply. It reports attention shifts toward lengthened options, partial mitigation with accuracy trade-offs, and continued sensitivity in newer models.

4 Discussion

  • The MCQ setting differs slightly from the Origin setting because the instruction to output the answer directly was removed to accommodate boolean questions.Source: §4 · p. 4

4.1 Reasons Behind Accuracy Drops

  • Lengthening an option shifts attention toward it; increasing distractor C’s length shifts more attention to C, consistent with a length-induced attention bias.The attention scores are from layer 0, averaged across 32 heads and 24 option orders.Source: §4.1 · p. 5, Table 6 · p. 5

4.2 Generalization of Results

  • Fine-tuning and CoT partially mitigate length bias but do not resolve the vulnerability; CoT narrows the RL–WL gap from 34.5% to 15.4% while lowering overall accuracy by 10.7%.On Qwen2.5-7B MMLU, fine-tuning improves WL by 7.8% and decreases RL by 12.3% versus the base model.Source: §4.2 · p. 5, Table 7 · p. 5
  • Gemini-2.0-flash, Claude-3.5-sonnet, and DeepSeek-v3 remain sensitive to changes, including under short chain-of-thought prompting.The results are reported for short-CoT models on MMLU.Source: §4.2 · p. 5, Table 8 · p. 5

4.3 Compare to Work in the Pre-LLM Era

  • The study extends earlier robustness analyses, which were limited to models below 1B parameters, by examining state-of-the-art models on comprehensive, premium-quality benchmarks.The paper reports significant limitations in these capable models.Source: §4.3 · p. 5

4.4 Rank Changes of Models

  • Model rankings on the MMLU leaderboard change across settings.Source: §4.4 · p. 5

5 Conclusion

The paper introduces Generalization Stress Tests for evaluating LLMs under minor, content-preserving changes. It reports performance degradation across these changes and concludes that LLMs rely on superficial patterns rather than robust generalizable reasoning.

5 Conclusion

  • LLMs show significant performance degradation under slight changes to question format, option length, or irrelevant content.The paper presents Generalization Stress Tests to assess these perturbations.Source: §5 · p. 5
  • The findings suggest LLMs rely on superficial patterns rather than robust, generalizable reasoning.Source: §5 · p. 5

Limitations

The work evaluates non-chain-of-thought LLMs and does not consider emerging O1.

Limitations

  • The study focuses on non-chain-of-thought LLMs, such as GPT-4o, and excludes emerging O1.Source: Limitations · p. 6

Ethnic Statement

This material reports an ethical statement and describes prompts and verification procedures for option-length and noun-replacement modifications.

Ethnic Statement

A.1 Prompts

  • The option-length prompt asks GPT-4o to lengthen choices without changing the question or hinting at the answer, using three word-count ranges.The ranges are no more than 10 words, 10–20 words, and at least 20 words.Source: §A.1 · pp. 7–8

A.2 Verification Process

  • Manual checks of 100 MMLU rewrites found 99 without issues and one changed question meaning, for 99% rewriting accuracy.Source: §A.2 · p. 8

B Prompts in Replacing Irrelevant Nouns

  • The noun-replacement prompts specify three levels of increasingly imaginative substitutions while retaining the original word order and sentence structure.The instructions also prohibit adding or removing sentences.Source: B Prompts in Replacing Irrelevant Nouns · p. 8

C Experiment Setup Details

This section describes the foundational setup of the experiments and analyses.

C Experiment Setup Details

C.1 Evaluation Protocol

Evaluations use the Harness framework with its default parameters and, unless specified otherwise, five-shot examples from benchmark training sets. The evaluated models span Llama3.1, Qwen2, and GPT4o families.

C.1 Evaluation Protocol

  • Unless specified otherwise, evaluations use five-shot examples drawn from the corresponding benchmark training sets.The experiments use the Harness framework and its default parameters unless otherwise specified.Source: §C.1 · p. 8

C.2 Models

  • The evaluated model families include Llama3.1, Qwen2, and GPT4o.Source: §C.2 · p. 8
  • The evaluation includes Llama3.1 8B and 70B, Qwen2.5 1.5B, 7B, and 72B, and GPT4o and GPT4o mini.Source: §C.2 · p. 9

C.3 Benchmarks

The study evaluates several benchmarks, uses about 1,000 GPU hours, and checks whether the reported option-length result persists across temperature and seed settings.

C.3 Benchmarks

  • Evaluations cover MMLU, ARC, GSM-MCQ, and GSM8k; the first four are multiple-choice benchmarks, while GSM8k uses open-ended questions.Source: §C.3 · p. 9
  • MMLU covers 57 tasks and 14,042 test examples, and its reported score averages accuracy across tasks.Source: §C.3 · p. 9
  • ARC includes eight task types and 3,548 samples, with ARC-Challenge and ARC-Easy subsets; the challenge subset excludes items answerable by retrieval and word co-occurrence methods.Source: §C.3 · p. 9
  • GSM8K contains 1,319 open-ended multistep math word problems designed to be solvable by middle school students.Source: §C.3 · p. 9

C.4 Budget

  • The experiments used an H800 GPU and cost about 1,000 GPU hours.Source: §C.4 · p. 9

C.5 Random Seeds

  • The RL/WL gap persists across tested temperatures and Python, NumPy, and Torch seed settings for Qwen-2.5 7B on MMLU.The authors report that their conclusions remain unchanged.Source: §C.5 · p. 9

D Additional Results

Additional analyses find option-length robustness differences across models, reveal a correct-option selection pattern when all distractors are lengthened, and show that evaluation protocol changes model rankings.

D.1 Making All Options longer

  • LLaMa is more robust than Qwen, and larger models are more robust than smaller models when all options are lengthened.The conclusion that LLMs are vulnerable to option length and biased toward long options remains unchanged under AL.Source: §D.1 · p. 9

D.2 Make All Wrong Options Longer

D.3 Rank Changes of Models

  • Kendall τ=0.39 between WL and RL model rankings indicates that switching the evaluation protocol substantially changes rank ordering.The passage notes that τ below 0.4 is not considered strongly correlated.Source: §D.3 · p. 10, Table 12 · p. 10

Related papers

Loading 2502.12459v3…