PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

arXiv:2610.38402v1cs.CLcs.AI

TL;DR

Predicting medical-item difficulty from text alone has remained challenging, particularly when new items lack historical response data. This study combines open-weight LLM judgments and simulated-examinee response signals to predict difficulty, finding complementary information and strong results on USMLE and PANCE.

  • Problem

    Predicting new medical-item difficulty before historical response data are available remains challenging, and text-based approaches have shown limited performance.

    Source: Introduction · p. 2

  • Method

    The study evaluates open-weight LLMs as item-difficulty judges and simulated examinees, then combines judgment-based and response-based signals in prediction models.

    Source: Introduction · pp. 2–3, Method · p. 6

  • Results

    Pairwise judgment was the most consistent individual signal, and combined configurations reached ρ = .447 on USMLE and ρ = .494 on PANCE.

    Source: Introduction · p. 3, Discussion · p. 14

  • Takeaways & Limitations

    Open-weight LLM judgment and response signals captured difficulty-related information beyond text representations and may offer practical value for high-security items.

    Source: Introduction · p. 3

    The relatively small test sets introduce sampling uncertainty, so closely spaced correlation differences should be interpreted cautiously.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets, while simulated-examinee signals and absolute judgments provided complementary information in some feature combinations. The best configurations reached a Spearman correlation of $\rho$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($\rho$ = .25), and $\rho$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($\rho$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.

Introduction

Predicting new medical-item difficulty before response data are available remains challenging, while text-based approaches have achieved modest results. This study tests whether LLM judgments and response behavior offer useful difficulty signals.

Introduction

  • Estimating new-item difficulty before historical responses are available matters because pretesting takes time and resources and can expose unreleased high-stakes items.Pretesting also depends on collecting enough examinee responses.Source: Introduction · p. 2
  • Text-based prediction remains limited, and standardized item-writing practices constrain surface cues while similarly themed items can differ in empirical difficulty.These constraints motivate seeking signals beyond text representations alone.Source: Introduction · p. 2
  • The study examines LLMs as difficulty judges and simulated examinees, treating their judgments and response behavior as complementary signals.Judges rate or compare item difficulty; simulated examinees answer items and provide response and option-probability signals.Source: Introduction · pp. 2–3
  • ρ = .447 on USMLE and ρ = .494 on PANCE were the best configuration results, with pairwise judgment the most consistent individual signal across both datasets.The USMLE result exceeded the best BEA 2024 shared-task system's approximately .25; the combined open-weight signals were comparable to Claude Opus 5 absolute judgments.Source: Introduction · p. 3

Medical Item Difficulty Prediction

Medical item difficulty prediction from text has remained limited, motivating approaches that use LLM interactions with items. Prior work also suggests pairwise comparisons can support ranking when direct score mapping is difficult.

Medical Item Difficulty Prediction

  • Across submissions from 17 teams, BEA yielded RMSE = 0.299, r = 0.27, ρ = 0.25, only slightly beating a mean-predicting dummy (RMSE = 0.311; rank 16/43).Separately, a full-feature model using 12,038 USMLE Clinical Knowledge MCQs reached only r = 0.32.Source: Medical Item Difficulty Prediction · p. 4
  • Standardized item-writing practices limit surface-level variation, and similar medical content can still have substantially different empirical difficulty.The supplied discussion argues richer text representations may leave difficulty-related variation unexplained.Source: Medical Item Difficulty Prediction · pp. 4–5

LLM Pairwise Comparison

  • On reading-comprehension MCQs, pairwise prompting with ChatGPT-3.5 reached Spearman .404 versus .183 for absolute judgments from the same model.Prior work describes relative judgment as an alternative when direct item-to-score mapping is limited.Source: LLM Pairwise Comparison · p. 5
  • A sparse comparison design with Bradley-Terry aggregation offers a scalable way to estimate a shared latent difficulty scale from limited pairwise judgments.The model jointly estimates relative item scores from observed comparisons, avoiding the O(N^2) comparisons required for exhaustive pairing.Source: LLM Pairwise Comparison · p. 5

LLM Simulated Examinees

Prior work has used LLM response behavior as a performance-based signal, but whether it adds information beyond difficulty judgments has remained unclear. This study evaluates response-derived signals alongside judgment-based signals.

LLM Simulated Examinees

  • Simulating different proficiency levels with one LLM can be sensitive to model and prompt design, and intended ability differences may not match actual problem-solving performance.Studies cited report that some models do not preserve the expected higher-ability-at-higher-grade pattern.Source: LLM Simulated Examinees · p. 6
  • Whether response-based signals add information beyond judgment-based estimates, or largely overlap with them, remained unresolved.Source: LLM Simulated Examinees · p. 6

Method

  • This study treats LLMs as simulated examinees by deriving response-based signals from direct answers and evaluates them alongside judgments for incremental and complementary value.The approach compares variation across models, elicitation methods, response readouts, and pairwise-comparison density.Source: Method · p. 6

Datasets

The study evaluates difficulty prediction independently on USMLE and PANCE datasets, using open-weight models for item judgments and responses. The signals are derived from ratings, pairwise comparisons, and answer-option distributions.

Datasets

  • The evaluation used 528 PANCE items and 667 retired USMLE items, with separate dataset analyses and distinct difficulty definitions and splits.PANCE difficulty was Rasch b; USMLE difficulty was a linear transformation of proportion correct, and the datasets used a 70/30 stratified split and the official BEA split, respectively.Source: Datasets · p. 7

LLM-as-a-judge

  • Six open-weight models rated item difficulty on a 0–100 scale or compared item pairs, with pairwise outcomes aggregated into relative difficulty scores.The comparison graph gave each item at least 40 comparisons; lower-density graphs were also evaluated.Source: LLM-as-a-judge · pp. 7–8

LLM-as-an-examinee

  • Thirteen candidate models answered items, yielding correctness, keyed-option probability, response entropy, and top-two-option probability margin.Correctness indicates whether the highest-probability option matches the answer key; entropy measures distribution uncertainty, while margin measures confidence.Source: LLM-as-an-examinee · p. 8

Feature Selection and Model Development

The study selected the strongest training-set source for each LLM signal type, then combined selected signals with baseline item features in downstream prediction models.

Feature Selection and Model Development

  • The strongest training-set model was selected separately for pairwise judgment, absolute judgment, and each of four simulated-examinee readouts.The four readouts could come from different examinee models.Source: Feature Selection and Model Development · p. 9
  • Selected signals were combined with baseline item features and evaluated using ridge regression, random forest, and XGBoost.Hyperparameters were chosen by repeated five-fold cross-validation on the training set; models were refit there and evaluated on a held-out test set.Source: Feature Selection and Model Development · p. 9
  • Eight prespecified feature configurations were compared to assess the incremental contribution and complementarity of LLM-derived signals.Table 2 presents the configurations evaluated in the downstream models.Source: Table 2 · p. 9, Feature Selection and Model Development · p. 9

Results

Pairwise judgments provided stronger difficulty signals than absolute ratings, while model architecture, difficulty gaps, and comparison density shaped judgment performance. Pairwise accuracy and order consistency did not always move together.

LLM-as-a-judge

  • Gemma 4 31B had the highest pairwise accuracy on USMLE and PANCE, outperforming two models with nominal parameter counts of 120B.Total parameter count is not directly comparable across architectures.Source: LLM-as-a-judge · pp. 9–10, Table 3 · p. 10
  • At the same 27B scale, MedGemma consistently outperformed Gemma 3 across both datasets, suggesting medical-domain adaptation may support difficulty judgments.Gemma 4’s strong performance may also reflect architectural and reasoning changes rather than parameter count alone.Source: LLM-as-a-judge · p. 10
  • Order consistency did not map simply to accuracy: Gemma 3 remained highly consistent despite substantially lower accuracy.Gemma 4 combined the highest pairwise accuracy with the highest PANCE consistency and relatively high USMLE consistency.Source: LLM-as-a-judge · p. 10
  • Pairwise accuracy generally rose with larger empirical difficulty gaps, exceeding .80 in the largest gap interval; two-way agreement showed no similar systematic rise.Gemma 4 was strongest across nearly all gap intervals, while agreement appeared more tied to model-specific stability than gap size.Source: LLM-as-a-judge · pp. 10–11, Figure 1 · p. 11
  • Most models reached about 95% of their k = 40 correlation with empirical difficulty using 10 to 15 comparisons per item, with diminishing returns as comparisons increased.The correlation generally improved with comparison density.Source: LLM-as-a-judge · p. 11, Table 4 · p. 12
  • More comparisons did little to compensate for weaker judgment quality: Gemma 3 27B rose only from .158 to .174 on USMLE and stayed near .06 on PANCE.These values compare k = 5 with k = 40 on USMLE; PANCE remained approximately .06 across densities.Source: LLM-as-a-judge · p. 11
  • Pairwise judgments achieved higher best training-set Spearman correlations than absolute judgments: .424 vs .349 on USMLE and .439 vs .411 on PANCE.The relative model ordering was broadly consistent across the two elicitation formats.Source: LLM-as-a-judge · pp. 11–12, Table 5 · p. 12

LLM-as-an-examinee

Across both datasets, probability-based examinee readouts correlated more strongly with empirical difficulty than binary correctness. The strongest models and readouts differed by dataset.

LLM-as-an-examinee

  • On USMLE, Mistral Small 3.1 24B supplied the strongest keyed-option probability, entropy, and margin signals (.287–.327), while Nemotron 3 Nano 30B led binary correctness (ρ = .270).On PANCE, Llama 3.1 70B led the three continuous readouts, each above .380, while Granite 3.3 8B led correctness (ρ = .269).Source: LLM-as-an-examinee · p. 13
  • Across both datasets, probability-based readouts correlated more highly with empirical difficulty than binary correctness; keyed-option probability led PANCE (ρ = .386) and tied margin on USMLE (ρ = .327).The three continuous readouts differed little on PANCE, ranging from .384 to .386.Source: LLM-as-an-examinee · p. 13

Combining LLM-Derived Signals for Difficulty Prediction

Combining LLM judgment and simulated-examinee signals improved medical-item difficulty prediction, with the best configurations comparable to Claude Opus 5 and above the 2024 BEA system on USMLE. Pairwise judgments were the most consistent individual signal; discussion findings qualify how comparison density and model behavior affect these signals.

Combining LLM-Derived Signals for Difficulty Prediction

  • Across eight configurations, adding pairwise judgment raised Spearman from .228 to .425 on USMLE and .100 to .450 on PANCE; examinee signals also improved prediction, while absolute judgment varied.For combination analyses, the strongest training-set Spearman signal from each source was selected.Source: Combining LLM-Derived Signals for… · p. 13, Discussion · p. 14
  • On USMLE, B+P+S led Spearman (ρ = .447), while B+P+A+S led Pearson (r = .443) and RMSE (.278).Adding absolute judgment did not further improve rank-order performance.Source: Combining LLM-Derived Signals for… · p. 13
  • On PANCE, B+P+A+S led Spearman (ρ = .494), while B+P+S led Pearson (r = .500) and RMSE (1.006).Adding absolute judgment to B+P+S increased Spearman from .480 to .494.Source: Combining LLM-Derived Signals for… · p. 13
  • ρ = .447 on USMLE and .494 on PANCE were comparable to Claude Opus 5 (.443 and .488); USMLE exceeded BEA’s winning system (ρ = .25).The best-performing configurations used ridge regression.Source: Combining LLM-Derived Signals for… · p. 13, Table 6 · p. 14

Discussion

  • Pairwise comparison was less dependent on absolute-scale calibration because it required only a local relative judgment.The prompts gave minimal information about target populations, such as ability levels or known-difficulty anchor items.Source: Discussion · pp. 14–15
  • About 10 to 15 comparisons per item reached roughly 95% of the external predictive performance observed at k = 40 for most models.The authors note that prior reliability criteria are not directly equivalent; more comparisons could not compensate for weak judgment quality.Source: Discussion · p. 15
  • Nominal parameter count did not straightforwardly predict judge performance; architecture and active parameters complicate comparisons between dense and mixture-of-experts models.The authors also point to model generation, training procedures, architecture, and domain adaptation as possible contributors.Source: Discussion · p. 15
  • Simulated examinees may need both adequate task competence and enough item-level response variation for their behavior to reflect real examinee difficulty ordering.Models answering nearly all items correctly may provide too little variation, while insufficiently capable models may err without a systematic difficulty pattern.Source: Discussion · pp. 15–16

Limitations

The study’s findings are bounded by prompt design, small test sets, and possible exposure of publicly released USMLE items during model development.

Limitations

  • The shared prompts were relatively simple, and unobservable system-level processing in commercial models may differ from the more directly controlled inputs of open-weight models.Future work could test richer population descriptions, examples, or model-specific prompt optimization.Source: Limitations · p. 16
  • The relatively small test sets introduce sampling uncertainty into correlations, so closely spaced configuration differences warrant caution.Source: Limitations · p. 16
  • USMLE items were publicly released and may have appeared in some models’ development data; PANCE provided an independent evaluation with nonpublic empirical difficulty parameters.Similar patterns across datasets provide additional support for the findings.Source: Limitations · p. 16

Appendix A Supplementary Tables

The supplementary tables cover correlations between simulated-examinee readouts or selected LLM-derived signals and empirical item difficulty, using training-set Spearman correlations.

Appendix A Supplementary Tables

  • Table A1 reports correlations between LLM examinee readouts and empirical item difficulty.The readouts are keyed-option probability, binary correctness, response entropy, and top-two margin.Source: Table A1 · p. 19, Appendix A · p. 19
  • Table A2 presents selected LLM-derived signals and their training-set Spearman correlations.Within each signal type or response readout, the selected model had the largest absolute correlation with empirical item difficulty.Source: Table A2 · p. 20, Appendix A · p. 20
  • Keyed-option probability, binary correctness, and top-two margin correlate negatively with difficulty, whereas response entropy correlates positively.The selected-signal table reports these correlations as absolute values.Source: Appendix A · p. 20

Appendix B Prompts

The prompts elicit absolute difficulty estimates, pairwise difficulty judgments, and simulated-examinee answers. They specify exam-specific framing and output formats for each task.

Appendix B Prompts

  • Simulated-examinee prompts ask models to answer a multiple-choice item without showing the keyed answer.Non-reasoning prompts request only the option letter, while reasoning-model prompts ask for brief thought before a final answer line.Source: Appendix B · p. 21, Simulated Examinee · pp. 22–23
  • The pairwise prompt templates identify the two presented items as A and B, with placeholders for their respective text.Absolute-judgment templates use the item stem, response options, and keyed answer; examinee prompts do not show the keyed answer.Source: Appendix B · p. 21, Pairwise Judgment · p. 21
  • Pairwise-judgment prompts present two items with their stems, options, and keyed answers, and ask which is harder for examinees.The prompts define harder as fewer examinees answering correctly and request a direct comparison of why one item is harder.Source: Pairwise Judgment · p. 21
  • Absolute-judgment prompts ask for an item's difficulty percentile on an evenly distributed exam reference scale.They define percentile 0 as the item most examinees answer correctly and percentile 100 as the item fewest answer correctly.Source: Absolute Judgment · p. 22
Loading 2610.38402v1…