Revisiting Generalization Across Difficulty Levels: It's Not So Easy

• Prior findings on cross-difficulty generalization are mixed, while human judgments can limit the scalability and resolution of difficulty assessment.Source: §1 · p. 1
• Limited cross-difficulty generalization makes curating an appropriate range of example difficulties important for LLM training.Source: §1 · p. 1
• The study uses IRT with evaluation results from thousands of LLMs to estimate example difficulty across six datasets, treating LLMs as students.Source: §1 · p. 2
• Each dataset is divided into ten equally sized bins ordered by increasing IRT difficulty for systematic generalization tests.Source: §1 · p. 2
• The analysis estimates difficulty for examples in six widely used benchmarks and studies generalization across difficulty levels.Source: §1 · p. 2
• Cross-difficulty transfer is limited: training on easy or hard bins does not generalize consistently, and larger train–test gaps can degrade performance, sometimes below zero-shot.Source: §1 · p. 2

Easy-to-hard Generalization

• Easy-to-hard studies disagree: Hase et al. (2024) report near parity on hard tests, while Yang et al. (2024), Chen et al. (2024), and Pikus et al. (2025) favor hard data.Source: §2 · p. 3
• Prior evaluations often use coarse easy-versus-hard splits or simple tasks, whereas this work studies fine-grained splits created with item-response theory.Source: §2 · p. 3

• Unlike Ding et al. (2024), this study estimates difficulty directly from model behavior without calibrating scores to human judgments.Source: §2 · p. 3

Difficulty Estimation

• Human-based difficulty approaches use grade levels, expert ratings, cognitive skills, and length or reasoning-step proxies, which do not consistently correlate with model-based difficulty.Source: §2 · p. 3
• Existing model-based methods may be computationally costly, depend on few models, or misrepresent task difficulty; this work uses responses from thousands of LLMs with IRT.Source: §2 · p. 3

3.1 Difficulty Estimation using IRT

• The Rasch 1PL model treats language models as subjects and estimates each task’s difficulty from their observed responses.The authors estimate posterior distributions with stochastic variational inference and use expected values as point estimates.Source: §3.1 · p. 3

3.2 Data Collection

• Leaderboard responses from six datasets are scored with IRT, sorted by difficulty, and divided into ten equally sized bins for cross-difficulty evaluation.The collected responses generally correspond to benchmark test sets, and model coverage varies by task.Source: §3.2 · p. 4

3.3 Difficulty Validation

• Zero-shot accuracy from Qwen3 models not used to estimate difficulty decreases consistently as IRT difficulty bins increase, supporting generalization of the scores to unseen models.The validation uses evaluations across all difficulty bins.Source: §3.3 · p. 4

3.4 Differences between IRT and Human-Based Metrics

• Most datasets show little or no positive correlation between IRT difficulty and human-based metrics, with some metric–dataset pairs even negatively correlated.The strongest positive correlations reported are answer length in MATH (0.56) and reasoning steps in GSM8K (0.49); the figure reports weak positive correlations for ARC and mostly no or negative correlations for MMLU-Pro.Source: §3.4 · p. 5, Figure 2 · p. 5
• The weak correlations suggest that what makes tasks difficult for language models differs from how human-based metrics measure difficulty.Some datasets show stronger alignment for reasoning steps or question length, but the pattern is weak for most human-based metrics.Source: §3.4 · p. 5

Experimental models

• Experiments use seven instruction-tuned models from the Qwen 2.5 and Llama 3 families, with results for Qwen2.5 14B Instruct presented in the main paper.Additional models range from Qwen2.5 1.5B to 7B and Llama 3.2 1B/3B and Llama 3.1 8B; the reported patterns are broadly consistent.Source: §4 · p. 6

Training

• Each model is fine-tuned separately on each single difficulty bin using supervised fine-tuning for five epochs.All model parameters are trained on response tokens using instruction-response templates.Source: §4 · p. 6

Evaluation

• Evaluation compares accuracy on non-training bins with zero-shot performance and reports the difference to assess generalization beyond the fine-tuning difficulty.The study follows the standard lm-eval-harness protocols and metrics.Source: §4 · p. 6

Easy-to-hard transfer

• Models trained on easier bins fail to generalize consistently to harder ones; on MMLU-Pro, training on the easiest bins degrades quickly on test bins five and above.This challenges claims that easy-only supervision can recover hard-task performance.Source: §5 · pp. 6, 8

Main cross-difficulty results

• Across six benchmarks, cross-difficulty gains are limited, weaken as train–test gaps widen, and can fall below zero-shot performance in either direction.The strongest performance clusters near the diagonal, where training and test difficulties are similar.Source: §5 · pp. 6, 8, Figure 4 · p. 7

Hard-to-easy transfer

• Hard-to-easy transfer is also limited: on BBH, models trained on the hardest bins perform worse on easy questions.The lower triangle of the heatmap shows negative values throughout for this case.Source: §5 · p. 8

Variation across datasets

• Transfer differs by dataset and model family: ARC shows near-zero cross-difficulty gains, while GSM8K shows moderate generalization for Qwen2.5 but not Llama models.The Llama results show no generalization for most bins.Source: §5 · p. 8

Evaluation implications

• Because strong performance on challenging questions does not establish performance on easier ones, benchmarking should cover a broad spectrum of difficulty levels.The paper specifically notes that benchmarks focused on the hardest problems provide limited information about behavior on easier questions.Source: §5 · p. 8

Consistency across models

• Weak cross-difficulty generalization persists across Qwen and Llama model sizes from 1B to 14B parameters, although larger models achieve higher absolute accuracy.The authors report similar patterns across the evaluated model families and sizes.Source: §5 · p. 8

Discussion

• Generalization is highest when training and testing difficulties are similar, and decreases as their gap grows across model families, scales, and datasets.The authors describe cross-difficulty generalization as limited in scope.Source: §6 · p. 8
• The authors argue prior work may have overestimated transfer by using human-based difficulty metrics, only a few difficulty splits, or narrow benchmark coverage.Their analysis uses ten difficulty bins and six benchmarks spanning reasoning, factual recall, and instruction following.Source: §6 · p. 8
• Training and evaluation datasets should account for difficulty alongside diversity and coverage, given the limited cross-difficulty generalization observed.Whether a curriculum structured by model-based difficulty can improve transfer remains an open question.Source: §6 · p. 8
• The authors call for precise difficulty measurement and difficulty-aware data design to support models that extend reasoning beyond training distributions.They identify training objectives and selection strategies targeting stability across difficulty levels as future work.Source: §6 · pp. 8–9

• Across model families, scales, and six benchmarks, generalization from both easy-to-hard and hard-to-easy training decreases significantly as the train-test difficulty gap increases.The analysis used fine-grained, model-based IRT difficulty estimates.Source: §7 · p. 9

• The six publicly available benchmarks are English-only, limiting direct extension to multilingual tasks and domains without reliable ground-truth labels.Source: Limitations · p. 9
• Model-based difficulty rankings may shift with changes in model capabilities, data distributions, or evaluation practices, affecting the observed patterns.Source: Limitations · p. 9
• The authors cannot ensure evaluation questions were unseen during pretraining, and undetected overlap might inflate accuracy and partially reduce true generalization.Eliminating this concern would require test sets collected after all evaluated models’ pretraining cut-offs.Source: Limitations · p. 9
• Because the study uses single-bin training to isolate difficulty effects, it does not cover mixtures of bins or adaptive sampling strategies.How such curricula interact with fine-grained difficulty remains future work.Source: Limitations · p. 9

• Full-parameter fine-tuning covered seven instruction-tuned models from the Llama 3 and Qwen 2.5 families.The models ranged from Llama3.2 1B Instruct to Qwen2.5 14B Instruct.Source: A Training Configuration · p. 12
• Training used five epochs, effective batch size 8, learning rate 5e-6, cosine decay, and gradient clipping at L2 norm 0.1.Runs used mixed precision and a maximum input length of 4096 tokens.Source: A Training Configuration · p. 12
• Training loss curves showed consistent convergence across difficulty bins, with final training accuracy averaging 0.84 across models and datasets.Training scripts, configurations, and requirements files were slated for release with the code repository.Source: A Training Configuration · p. 12

B Evaluation Setup

• Evaluation used greedy decoding with each model’s chat template, and reported mean accuracy across difficulty bins and evaluation tasks.Models were evaluated with lm-eval-harness and vLLM.Source: B Evaluation Setup · p. 12

C IRT Model Selection

• More parameterized IRT models produced unstable or counterintuitive difficulty estimates, so the authors chose 1PL for lower error and more interpretable scores.Examples included infeasible questions placed near middle difficulty under 4PL and easy questions mislabeled as guessable when adding guessing parameters.Source: C IRT Model Selection · p. 12
• Spearman rank correlations of 0.998–1.000 across datasets showed near-perfect agreement between difficulty rankings from LLM subsets and the full-model baseline.The ablation fit IRT using random subsets containing 25%, 50%, and 75% of available LLMs; reducing the model count did not meaningfully change relative difficulty ordering.Source: C IRT Model Selection · pp. 12–13
• Qwen 3 4B and 30B-A3B Instruct 2507 models showed lower zero-shot performance in more difficult IRT bins despite not being calibrated using their responses.Source: Figure 5 · p. 13

D Datasets

• Eight public datasets span elementary science and math, expert knowledge, instruction following, and specialized reasoning challenges.The datasets provide a range of task difficulties for studying cross-difficulty generalization.Source: D Datasets · p. 13
• GSM8K tests multistep arithmetic, MMLU-Pro covers expert-level multiple-choice subjects, and BBH focuses on difficult reasoning and comprehension.Source: D Datasets · p. 13
• IFEval received one annotated correct answer per question so it could support training and evaluation on the same set without distribution shift.Source: D Datasets · p. 13
• For MATH Level 5 problems, the authors replaced provided solutions with sampled correct responses in separate Qwen- and Llama-style training datasets.Directly fine-tuning on the original final answers or solutions reduced zero-shot performance for all models.Source: D Datasets · pp. 13–14
• Manual review found no systematic mislabeling in the hardest bins; five GSM8K errors were under 4% of that bin and insufficient to explain its difficulty distribution.Source: D Datasets · p. 14

E Supplementary Results

• IFEval and GPQA-Extended showed no meaningful learning or consistent cross-difficulty patterns, with fewer than 55 examples in each fine-tuning bin.Both datasets contain fewer than 550 examples and are excluded from the main paper.Source: E Supplementary Results · p. 14

F Correlation

• Across the eight datasets, the extended correlation heatmaps show weak overall alignment between IRT difficulty scores and human annotations.The appendix provides the full set of heatmaps to complement the main-text summary.Source: F Correlation · p. 14

G Examples

• Figure 6 compares SFT with zero-shot performance for Qwen2.5 7B, while Table 3 gives examples where dataset difficulty labels disagree with IRT.Source: G Examples · p. 14, Figure 6 · p. 15, Table 3 · p. 26

B ARC IRT: bin 2

• Tables 4 and 5 provide questions from IRT difficulty bins 0–4 and 5–9.Source: Table 4 · p. 27, Table 5 · p. 28

H Use of Large Language Models

• The authors used ChatGPT, Grammarly, and GitHub Copilot for writing assistance and codebase development.Source: H Use of Large Language Models · p. 29