PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction

Ming Li, Han Chen, Yunze Xiao, Jian Chen, Hong Jiao, Tianyi Zhou

arXiv:2512.18880v2cs.CLcs.AIcs.CY

TL;DR

Item difficulty estimation matters for educational assessment, but cold-start items lack the historical response data used by existing approaches. The paper evaluates difficulty alignment across LLMs and finds that model scale, proficiency simulation, and problem-solving ability do not ensure alignment with human difficulty.

  • Problem

    Cold-start items lack historical response data for difficulty estimation, and it remains unclear whether off-the-shelf LLMs can align difficulty judgments with human perception without task-specific fine-tuning.

    Source: §1 · p. 1

  • Method

    The study benchmarks over 20 LLMs across educational domains, analyzing model consensus, IRT-based machine difficulty, metacognitive alignment, and proficiency simulation.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    Scaling does not guarantee human-aligned difficulty estimates: models converge toward machine consensus, while proficiency simulation and prediction of their own limitations are unreliable.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Estimating human difficulty requires more than stronger problem-solving models or proficiency prompting, according to the study’s findings.

    Source: §5 · p. 8

    The study uses zero-shot proficiency prompts and does not test few-shot examples of student errors or fine-tuning on student response logs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Accurate estimation of item (question or task) difficulty is critical for educational assessment but suffers from the cold start problem. While Large Language Models demonstrate superhuman problem-solving capabilities, it remains an open question whether they can perceive the cognitive struggles of human learners. In this work, we present a large-scale empirical analysis of Human-AI Difficulty Alignment for over 20 models across diverse domains such as medical knowledge and mathematical reasoning. Our findings reveal a systematic misalignment where scaling up model size is not reliably helpful; instead of aligning with humans, models converge toward a shared machine consensus. We observe that high performance often impedes accurate difficulty estimation, as models struggle to simulate the capability limitations of students even when being explicitly prompted to adopt specific proficiency levels. Furthermore, we identify a critical lack of introspection, as models fail to predict their own limitations. These results suggest that general problem-solving capability does not imply an understanding of human cognitive struggles, highlighting the challenge of using current models for automated difficulty prediction.

1 Introduction

Because cold-start items lack response histories and existing difficulty-prediction methods depend on such data, the study asks whether off-the-shelf LLMs can align difficulty estimates with field-tested student struggles. It finds that scaling and proficiency prompting do not reliably close the gap.

1 Introduction

  • Cold-start items lack the historical response data needed for statistical difficulty estimates, while prior supervised methods depend on such data and LLM alignment with human difficulty remains an open question.The study compares LLM judgments with difficulty values obtained from real student field testing.Source: §1 · p. 1
  • The study benchmarks over 20 models across four educational domains and examines model consensus, IRT-based machine difficulty, metacognitive alignment, and proficiency simulation.It treats models as observers predicting others’ difficulty and actors experiencing difficulty themselves.Source: §1 · pp. 1–2
  • Increasing model scale does not improve difficulty predictions reliably; models align more strongly with one another than with human reality.The paper calls this shared pattern a Machine Consensus.Source: §1 · p. 2
  • Neither ensembling nor proficiency simulation reliably repairs the misalignment, and models struggle to mimic different proficiency levels.Ensemble performance is strictly bounded by weaker models.Source: §1 · p. 2
  • Models’ actual correctness yields difficulty estimates that correlate with humans even worse than their explicit perceptions, as human-difficult items are often trivial for models.This capability persists even under weak student prompts.Source: §1 · p. 2
  • Models fail to predict their own limitations, with AUROC near random guessing and explicit difficulty estimates decoupled from actual correctness.The authors identify this as a lack of introspection.Source: §1 · p. 2

2 Formulation and Evaluation Metrics

The framework separates models’ explicit difficulty judgments from their performance as test-takers, comparing each with human difficulty. It also tests proficiency-conditioned estimates across datasets and model families.

2.1 Task Formulation

  • The task represents each item with its context, correct answer, and a difficulty label derived from real-student field testing.Difficulty labels may be continuous or discrete, depending on the domain.Source: §2.1 · p. 2
  • In the observer mode, models receive the item context, correct answer, and optional proficiency prompt, then generate a response parsed into a numerical difficulty estimate.In the actor mode, they solve items zero-shot with the correct answer hidden; an extracted final answer is checked for correctness.Source: §2.1 · pp. 2–3

2.2 Evaluation Framework

  • The evaluation compares perceived difficulty with human labels and empirical machine difficulty with human difficulty, using Spearman rank correlation for both.For empirical machine difficulty, the model pool is treated as synthetic examinees and a Rasch model estimates item difficulty from answer correctness.Source: §2.2 · p. 3

2.3 Proficiency Simulation

  • Four proficiency configurations—baseline, low, average, and high—condition generation to test whether models can simulate different student cognitive states.Prompts vary by domain, and descriptions of proficiency are intentionally limited because understanding those levels is part of the investigation.Source: §2.3 · p. 3

2.4 Experimental Setup

  • The evaluation uses four educational domains spanning reading comprehension, verbal reasoning, math reasoning, and specialized professional knowledge.The setup prioritizes datasets with difficulty values grounded in real-student field testing.Source: §2.4 · pp. 3–4
  • USMLE includes 667 items field-tested with over 300 medical students per item; Cambridge contains 793 questions across 120 passages and uses rescaled IRT b-parameters.USMLE difficulty values are transformed p-values ranging from [0, 1.3], while Cambridge values range from [0, 100].Source: §2.4 · p. 4
  • The SAT Reading & Writing set has 1,338 questions and SAT Math has 1,385, with difficulty labeled Easy, Medium, or Hard.Figure-dependent SAT Reading & Writing questions were removed.Source: §2.4 · p. 4
  • The study evaluates over 20 LLMs, including proprietary and open-weight general-purpose models as well as reasoning-focused models.The setup also includes reasoning-specialized variants such as DeepSeek-R1, QWQ-32B, and Qwen3-32B in reasoning mode.Source: §2.4 · pp. 4–5

3 The Landscape of Explicit Perception

The section evaluates zero-shot difficulty prediction and finds that stronger models do not reliably align with human difficulty judgments. Model consensus and persona-based methods do not provide a dependable substitute for that alignment.

3 The Landscape of Explicit Perception

  • The section establishes a zero-shot item-difficulty prediction baseline without student response data and summarizes three limitations of current human-AI alignment.Source: §3 · p. 5

3.1 Systematic Misalignment

  • Across domains, LLM difficulty rankings remain misaligned with humans: scaling does not reliably improve alignment, while advanced models converge on a shared non-human consensus.Source: §3.1 · pp. 5–6
  • Spearman’s ρ averages below 0.50, with SAT Math at ≈0.41 and USMLE at ≈0.13; GPT-5 scores 0.34 versus GPT-4.1’s 0.44.Source: §3.1 · p. 5
  • Predictions cluster narrowly toward lower difficulty despite a broader human difficulty range, indicating distribution shift and variance collapse.Source: §3.1 · p. 5
  • Machine IRT diagnoses implicit alignment through saturation (90% model solve rate), Savant (human-hard, model-easy items), and brittleness (human-easy, model-fail items).Source: Table 3 · p. 6

3.2 Ensemble and Proficiency Simulation

  • In SAT Math, ensembling raises correlation from 0.56 to 0.66, but adding models beyond K = 14 dilutes the signal; dilution starts immediately in USMLE.Source: §3.2 · p. 6
  • Individual persona effects are noisy, but averaging personas raises GPT-5 correlation to 0.47 from baseline 0.34, smoothing inconsistent prompt responses rather than simulating students faithfully.Source: Figure 3 · p. 5, §3.2 · p. 6, Table 2 · p. 6
  • Persona prompts change problem-solving accuracy only marginally, indicating models struggle to suppress or enhance intrinsic capabilities on command.Source: Table 4 · p. 7

4 Capability vs. Perception

The section compares models’ explicit difficulty judgments with difficulty inferred from their own correctness, finding capability-driven divergence from human judgments and weak introspection. It also tests whether proficiency prompts can counter that divergence.

4.1 The Curse of Knowledge

  • Conceptually difficult human items are often trivial for models, and proficiency prompts shift accuracy by typically less than 1%, failing to suppress their problem-solving capability meaningfully.The authors describe this persistence as the Curse of Knowledge, which prevents authentic simulation of struggling students’ specific misconceptions.Source: §4.1 · p. 7
  • The analysis treats 21 models as a student cohort to estimate intrinsic machine difficulty with IRT, defining Savant and Brittleness Rates within human-difficulty tiers.The Savant Rate concerns the top 33% of human-difficulty items solved by over 90% of models; the Brittleness Rate concerns the bottom 33% with model pass rates below 50%.Source: §4.1 · p. 7
  • USMLE model-IRT difficulty aligns worse than explicit perception (Savant Rate 70.4%; saturation 75.6%), while SAT Math reaches ρ = 0.386 but retains a 32.2% Savant Rate.In USMLE, the Savant Rate means that 70.4% of items in the top third of human difficulty are solved by over 90% of models; saturation is 75.6% dataset-wide.Source: §4.1 · p. 7

4.2 Metacognitive Blindness.

  • Metacognition is assessed by whether predicted difficulty separates items a model answers incorrectly from those it answers correctly, using AUROC, where about 0.5 indicates random alignment.The analysis formulates this as binary classification of the model’s own correctness.Source: §4.2 · p. 8
  • Most models show weak metacognitive alignment: AUROC scores are typically 0.50–0.60, and even frontier models’ overall discrimination remains poor, near or below 0.67.GPT-5 reaches 0.73 on Cambridge and DeepSeek-R1 reaches 0.64 on USMLE, but these are localized highs.Source: §4.2 · p. 8

5 Conclusion

The conclusion reports that LLMs struggle to align difficulty judgments with human perception despite advanced problem-solving ability. Scaling, proficiency prompting, and models’ limited ability to suppress knowledge or identify their own limitations do not resolve the gap.

5 Conclusion

  • The study finds that increasing model scale does not guarantee better human alignment and is associated with machine consensus that diverges from student reality.The authors attribute the gap to difficulty suppressing knowledge to simulate struggling students and limited metacognitive insight into model limitations.Source: §5 · p. 8

Limitations

The proficiency-simulation investigation uses zero-shot prompting and does not test few-shot prompts with real student error patterns or fine-tuning on student response logs.

Limitations

  • The study tests zero-shot proficiency simulation, not few-shot prompting with real student errors or fine-tuning on response logs.The authors note that few-shot scenarios might improve Human-AI alignment, but focus here on intrinsic capability unaffected by further learning or training.Source: Limitations · p. 9

Table of Contents for Appendix

The appendix reviews item-difficulty prediction approaches, from field-tested response data and text-based modeling to neural and LLM-assisted methods.

A.1 Item Difficulty Prediction

  • Conventional CTT and IRT difficulty calibration uses field-test response data, but collection and calibration can be costly and time-consuming.The passage notes that IRT calibration may require several thousand examinees, while field data collection can take months.Source: §A.1 · p. 14
  • Text-based prediction avoids collecting response data, but feature-engineering approaches can require substantial manual extraction and generalize poorly across domains.Expert ratings are also described as subjective and weakly aligned with psychometric difficulty.Source: §A.1 · p. 14
  • Transformer models can outperform traditional linguistic or embedding baselines in some settings, while LLMs also supply difficulty predictions or auxiliary features for downstream models.The reviewed work notes a recurring trade-off between predictive power and interpretability.Source: §A.1 · p. 14

A.2 Student Simulation

Student simulation uses artificial agents to generate learner-like behavior, while prior work explores prompting, learner traces, and agentic designs. This study positions its inquiry around whether models’ difficulty judgments reflect human struggles and their own likelihood of failure.

A.2 Student Simulation

  • Student simulation generates learner-like behavior or data, distinct from student modeling systems that infer latent states for adaptation.Earlier approaches used hand-crafted tutoring systems or human role-play data; LLM-based work also uses prompting, fine-tuning, and agents with memory and planning.Source: §A.2 · pp. 14–15
  • Prompting, learner-trace fine-tuning, and agentic designs model student traits, performance, or learning progression, but fluent output does not ensure behavioral fidelity.Some approaches add knowledge tracing or IRT to control simulated ability over time; prior work also notes that many simulated-learner studies lack empirical realism validation.Source: §A.2 · p. 15
  • Prior studies examine synthetic examinees and learner-mimicking prompts, but whether LLMs align with human difficulty without ability calibration remains open.Related findings caution that weakened prompts, grade enforcement, and temperature scaling do not reliably reproduce learner-like errors or alignment across settings.Source: §A.2 · p. 15

A.3 LLM Self-Awareness

  • Self-knowledge benchmarks often test abstention under particular failure modes, while verbalized uncertainty can be brittle and generalize poorly.Other work reports improvements from fine-tuning or prompting, including increasingly calibrated correctness probabilities with scale.Source: §A.3 · pp. 15–16
  • In educational difficulty estimation, explicit difficulty judgments are only weakly predictive of a model’s own failure cases, indicating a metacognitive gap.The study tests whether judgments about item difficulty provide an introspective signal of the model’s likelihood of error on the same items.Source: §A.3 · p. 16

B Consensus of Machines

Across four datasets, correlations among models are relatively higher than correlations between models and human judgments.

B Consensus of Machines

C Case Study

GPT-5’s Low-Proficiency persona broadens its difficulty-prediction distribution toward the Ground Truth, but systematic underestimation remains.

C Case Study

  • GPT-5 still systematically underestimates difficulty, remaining optimistic about student performance relative to reality.This underestimation persists despite the Low-Proficiency persona’s closer distributional resemblance to the Ground Truth.Source: C Case Study · p. 22, Figure 8 · p. 22
  • The Low-Proficiency persona expands GPT-5’s prediction distribution, bringing it closer to the Ground Truth.Figure 8 presents violin plots of GPT-5 difficulty predictions across personas.Source: Figure 8 · p. 22, C Case Study · p. 22
  • Figure 8 compares GPT-5 difficulty-prediction distributions under different personas with the Ground Truth.The comparison includes the Low-Proficiency simulation, whose distribution expands toward the Ground Truth.Source: Figure 8 · p. 22, C Case Study · p. 22

D Experimental Prompts

The experimental prompts cover difficulty prediction and question answering across four tasks, with Low-, Medium-, and High-Proficiency Student personas.

D Experimental Prompts

Loading 2512.18880v2…