PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

arXiv:2610.39373v1cs.CLcs.AI

TL;DR

Medical-item difficulty research has emphasized prediction more than interpretable explanations of why items differ. This study induces difficulty rubrics from empirical labels for USMLE and PANCE items and tests rubric-guided LLM judgments. Guidance changed scale use but did not consistently improve judgments, suggesting rubric induction may be more useful for generating testable structural hypotheses than as a reliable performance aid.

  • Problem

    Medical-item difficulty work has mainly predicted empirical difficulty, with comparatively little focus on readable accounts of the structures associated with it.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    Claude Opus 5 induced absolute and pairwise difficulty rubrics from empirical labels on USMLE and PANCE training items, which models then used to judge held-out items with or without guidance.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Rubric guidance changed absolute scale use but did not consistently improve fine-grained ranking or pairwise performance across models and examinations.

    Source: §5 · pp. 16–17

  • Takeaways & Limitations

    Label-guided rubric induction may be more valuable for producing interpretable, testable hypotheses about empirical difficulty than for reliably improving LLM judgments.

    Source: §5 · p. 17

    All four rubrics were induced by Claude Opus 5 and were not independently evaluated by medical education or item-writing experts; generalization beyond USMLE and PANCE remains unclear.

    Source: §5 · p. 17

Abstract

from arXiv · show

Research on medical item difficulty has largely focused on prediction rather than on identifying interpretable patterns associated with empirical difficulty. This study used empirical difficulty labels to guide large language models (LLMs) in inducing difficulty rubrics and examined how rubric guidance affected LLM difficulty judgments. Using training items and their empirical difficulty estimates from the United States Medical Licensing Examination (USMLE) and the Physician Assistant National Certifying Examination (PANCE), Claude Opus 5 induced separate rubrics for absolute and pairwise difficulty judgment. None of the induced rubrics treated surface features, such as stem length, option count, or images, as major sources of difficulty. Several patterns emerged across both examinations, including cue convergence, misleading cues, and option competition, although their specific forms differed. Retrieval of knowledge that could not be derived from the stem was particularly prominent in the PANCE rubrics, whereas the USMLE rubrics placed greater emphasis on integrating multiple clinical cues. We compared commercial and open-weight LLMs on held-out items under unguided and rubric-guided conditions. Rubric guidance changed how models used the difficulty scale but did not consistently improve performance. In absolute judgments, guidance encouraged broader use of the scale and generally improved coarse-grained agreement, without consistently improving rank ordering. In pairwise comparisons, its effects varied across models and examinations, and judgments became more sensitive to presentation order. These findings suggest that the value of label-guided rubric induction may lie more in generating interpretable, testable hypotheses from empirical difficulty data than in reliably improving difficulty judgments.

1 Introduction

Prior research has focused on predicting empirical medical-item difficulty, but prediction alone does not explain why items differ or yield principles item writers can inspect. This study uses empirical difficulty to guide LLM induction of interpretable rubrics and tests their effect on LLM judgments.

1 Introduction

  • Prior work has predicted empirical item difficulty from text, features, representations, or simulated responses, but predictive performance alone does not explain why items are difficult.A representation can capture difficulty-associated signals without yielding principles item writers can inspect, interpret, and apply.Source: §1 · p. 2
  • The study treats empirical difficulty labels as supervision for inducing interpretable principles that distinguish easier from harder items.LLMs can synthesize examples and summarize recurring patterns in natural language.Source: §1 · p. 2
  • The study asks which difficulty patterns LLM-induced rubrics identify across USMLE and PANCE, and how rubric guidance affects absolute and pairwise judgments.It compares unguided and rubric-guided judgments of new items across commercial and open-weight models.Source: §1 · p. 2

2 Related Work

Prior work has predicted medical-item difficulty but has focused less on deriving readable accounts of its structure. This study introduces empirical difficulty labels during rubric induction to generate and test interpretable principles.

2.1 Medical Item Difficulty Prediction

  • Existing prediction results show limited recovery of empirical difficulty from item information, including a best BEA 2024 system with Pearson correlation .27, Spearman correlation .25, and RMSE .299.Its RMSE modestly improved on the mean-prediction baseline of .311; Ha et al. (2019) reported r = .32 for a full model and approximately .11 for its strongest individual feature.Source: §2.1 · p. 3
  • Medical difficulty-prediction work has used linguistic features, text representations, and simulated examinee responses, but mainly targets prediction rather than explanation.Prior approaches include a shared benchmark and methods using item content or simulated performance.Source: §2.1 · p. 3

2.2 Interpretable Modeling of Item Difficulty

  • Interpretable modeling of item difficulty has used expert-developed feature sets in mathematics and reading, but no comparable feature framework is established for medical items.Prior medical work found surface complexity and semantic representations to be relatively weak predictors.Source: §2.2 · p. 3
  • Unlike text-first concept induction, this study supplies empirical difficulty labels during rubric induction so principles can capture structures associated with examinee performance.The induced structures are then explicitly provided to assess how they change LLM judgments of new items.Source: §2.2 · p. 4

3 Method

The study uses PANCE and USMLE training items to induce separate absolute and pairwise difficulty rubrics, then evaluates unguided and rubric-guided LLM judgments on held-out items. Pairwise induction uses clearly separated difficulty pairs, while evaluation also tests order sensitivity.

3.1 Datasets

  • The study analyzes 528 PANCE items and 667 USMLE items, using stratified 70/30 splits for PANCE and the official training/test split for USMLE.PANCE difficulty is the Rasch b parameter; USMLE difficulty is a linear transformation of the proportion answering correctly.Source: §3.1 · pp. 4–5

3.2 LLM Rubric Generation

  • Claude Opus 5 induces separate absolute and pairwise rubrics from training items and empirical difficulty, without seeing test items.Absolute induction uses the training items and five quintile-based bands; pairwise induction selects 250 item pairs per dataset with difficulty gaps of at least 1.5 standard deviations.Source: §3.2 · pp. 5–6
  • For pairwise induction, Opus 5 first derives a working account from stable-correct pairs, then revises it using stable errors and order-dependent pairs.Order dependence constrains rubric application rather than generating new difficulty principles; the final rubric is fixed for testing.Source: §3.2 · p. 6

3.3 LLM Judges

  • Twelve models rate absolute difficulty on a 0–100 scale, while seven open-weight models compare pairs in both presentation orders.The study evaluates held-out absolute judgments using scale use, Spearman correlation, and QWK; pairwise evaluation uses 1,200 non-induction pairs and measures accuracy and order consistency.Source: §3.3 · pp. 6–7, §3.4 · p. 7

4 Results

Rubric guidance broadened absolute scale use and generally improved coarse-band agreement, but it did not consistently improve fine-grained ranking. Pairwise effects varied by model and examination, while the induced rubrics described difficulty through relations among cues, answers, and competing options.

4.1 LLM-Induced Rubrics

  • Across induced rubrics, stem length, option count, and image presence were not primary difficulty sources; shared patterns included converging cues, misleading cues, and option competition.PANCE emphasized non-derivable knowledge retrieval, whereas USMLE emphasized integrating multiple clinical cues.Source: §4.1 · p. 8

4.1.1 PANCE

  • In PANCE, difficulty can arise when salient cues misdirect attention, the question shifts its target, or an uncommon fact cannot be inferred from the case.PANCE rubrics also describe direct retrieval and matching descriptions to adjacent terms; multiple defensible options can require choosing the most appropriate or highest-priority answer.Source: §4.1.1 · pp. 9–10

4.1.2 USMLE

  • USMLE rubrics describe misleading stem facts that support wrong answers, near-neighbor options within a knowledge family, and increasing inferential demand.Difficulty can rise from one extra inferential step to integrating multiple clinical cues before eliminating options.Source: §4.1.2 · pp. 10–11

4.2 Effect of Rubric Guidance on Absolute Difficulty Judgments

  • Guidance broadened use of the absolute difficulty scale and mostly improved coarse-band agreement, but did not consistently improve rank ordering.QWK changes varied across models and datasets; Opus 5 reached the highest guided QWK on both examinations, at .326 for USMLE and .245 for PANCE.Source: §4.2 · pp. 11, 12, 13, 15, Table 5 · p. 14
  • In unguided absolute judgments, some commercial models ranked USMLE items with Spearman correlations near or above .39, exceeding cited customized-embedding and simulated-student feature benchmarks.The comparison supports the reported observation that frontier LLMs recover difficulty information through direct judgment without task-specific training.Source: §4.2 · p. 13, Table 6 · p. 14
  • Rubric guidance did not improve fine-grained ranking consistently: all five commercial models’ Spearman correlations fell on both datasets, while open-weight results varied.Claude Opus 5 changed from .443 to .436 on USMLE and from .488 to .476 on PANCE.Source: Table 6 · p. 14, §4.2 · p. 15

4.3 Effect of Rubric Guidance on Pairwise Comparison

  • For pairwise judgments, Gemma 4 31B led unguided accuracy on both datasets, while rubric effects varied and order consistency generally declined under guidance.All four 27B–31B models improved USMLE accuracy by .006 to .023, but three of those four declined on PANCE; the authors suggest competing rubric criteria may shift with presentation order.Source: §4.3 · p. 15

5 Discussion

Medical item difficulty appears relational: how stem information shapes the path to an answer matters more than isolated item components. Rubric guidance made difficulty structure explicit but did not consistently improve judgments, and its benefits must be interpreted within the study’s scope.

5 Discussion

  • Difficulty appears relational: how stem information structures the path to an answer matters more than isolated item components.A cue can narrow the answer space or support a plausible competing option; recognizing the condition may still leave substantial work before the final answer.Source: §5 · pp. 15–16
  • Stem length, option count, and images did not consistently explain difficulty, while the two examinations showed broad similarities with some differences in how information structures were expressed.Source: §5 · p. 16
  • Rubric guidance did not consistently improve difficulty judgments: absolute-scale use changed more clearly than ranking, while pairwise effects varied by model and examination.On USMLE, all four 27B–31B models improved in pairwise accuracy, but this pattern did not replicate on PANCE.Source: §5 · p. 16, Table 7 · p. 16
  • Explicit rubrics may externalize readable, inspectable principles without capturing all fine-grained information in models’ original judgments.Even Claude Opus 5 did not improve its ranking using its own rubric.Source: §5 · p. 17
  • Pairwise judgments became less order-consistent under rubric guidance, suggesting that weighing multiple explicit rules may make evidence salience sensitive to presentation order.Source: §5 · p. 17
  • Rubric induction may be more valuable for turning empirical difficulty differences into interpretable, testable structural hypotheses than for reliably improving judgments.Source: §5 · p. 17
  • The rubrics were induced by one model, lacked expert evaluation, covered only USMLE and PANCE, and used pairwise examples with relatively large difficulty differences.The principles may therefore be better suited to clearly different items than to subtle difficulty distinctions; they describe associations, not experimentally established causal mechanisms.Source: §5 · p. 17

Appendix A Prompt Templates

Appendix A documents the prompts used to induce difficulty rubrics and judge items in absolute and pairwise formats. Guided prompts inserted the induced rubric into an otherwise shared base prompt.

Appendix A Prompt Templates

  • The prompts identify the examination and target population: US medical-school students for USMLE and graduating US physician-assistant students for PANCE.Source: Appendix A · p. 19

A.1 Absolute Difficulty Rubric Induction

  • For absolute rubric induction, Claude Opus 5 received training items with empirical percentiles and five equal-sized difficulty bands, then characterized each band using shared attributes, exceptions, within-band variation, and representative items.The prompt asked for concrete descriptions grounded in the items and difficulty data, without inventing abstract band names.Source: §A.1 · p. 19

A.2 Pairwise Difficulty Rubric Induction

  • The pairwise rubric was induced in two stages: correct judgments under both presentation orders informed an initial account, then stable errors and order-dependent pairs tested and revised it.Source: §A.2 · p. 20

A.3 Absolute Difficulty Judgment

  • Absolute judgments used the same base prompt in both conditions, inserting the induced rubric only in the guided condition.Models estimated a percentile from 0 to 100 within the examination reference set; 0 denotes the item most examinees answered correctly and 100 the item fewest answered correctly.Source: §A.3 · p. 22
  • The absolute prompt instructed models to treat figures or tables visible to examinees as available evidence, even when those materials were omitted from the model’s text.Source: §A.3 · p. 22

A.4 Pairwise Difficulty Judgment

  • Pairwise judgments likewise shared a base prompt, with induced guidance inserted only in the guided condition.Models compared two items for which they received stems, options, and correct answers, then identified which was harder for examinees and explained the direct comparison.Source: §A.4 · p. 23
  • The pairwise prompt also treated figures and tables available to examinees as evidence, although those materials might not appear in the model’s text.Source: §A.4 · p. 23

Appendix B Examples of Model Reasoning

The appendix gives illustrative excerpts of model reasoning under unguided and rubric-guided conditions, not a systematic qualitative analysis.

Appendix B Examples of Model Reasoning

  • The reasoning excerpts illustrate how rubric guidance changed model responses, but they are not intended as a systematic qualitative analysis of all responses.Source: Appendix B · p. 24

B.1 Absolute Difficulty Judgment

For one USMLE item, Claude Opus 5 placed the unguided estimate at percentile 17 and the rubric-guided estimate at percentile 22. Both explanations describe a straightforward urge-incontinence item with mild competing cues.

B.1 Absolute Difficulty Judgment

  • Unguided assessment placed the item at percentile 17, while rubric-guided assessment placed it at percentile 22; the vignette was a straightforward urge-incontinence case with normal labs excluding infection, glucose, and renal issues.Source: §B.1 · p. 24
  • Both explanations treat the urge-incontinence presentation as straightforward, with normal findings excluding infection and other distractors.Source: §B.1 · p. 24
  • The rubric-guided explanation places the item in band 2 territory, citing recognition plus mild exclusion work and a nondiagnostic rival cue.Source: §B.1 · p. 24

B.2 Pairwise Difficulty Comparison

In the compared USMLE pair, item 392 was harder than item 103; unguided and rubric-guided judgments both selected it. The rubric-guided comparison emphasized the specific retrieval and less direct inference required by item 392.

B.2 Pairwise Difficulty Comparison

  • Item 392 was harder than item 103, with an empirical difficulty gap of 1.88 SD.Item 103 was presented as Question A and item 392 as Question B.Source: §B.2 · p. 24
  • The unguided judgment correctly selected item 392, contrasting its septic-arthritis reasoning with item 103’s more classic heat-illness scenario.Item 392 requires integrating immunosuppression risk, synovial fluid findings, and empiric antibiotic coverage; item 103’s management options were described as less ambiguous.Source: §B.2 · p. 24
  • The rubric-guided judgment correctly selected item 392 because it requires retrieving a specific drug and making a less direct clinical inference.Both items contain a stem fact pointing toward a wrong answer, but item 392 requires choosing vancomycin rather than a broad intervention.Source: §B.2 · p. 25

Appendix C Induced Absolute Judgment Rubrics

The appendix note states that representative PANCE items are omitted while the other rubric text is preserved.

C.2 PANCE Difficulty Rubric

  • Representative PANCE items are omitted from this version, while the remaining rubric text is preserved.The note identifies this as the PANCE Difficulty Rubric appendix subsection.Source: §C.2 · p. 32

Appendix D Induced Pairwise Comparison Rubrics

The pairwise rubrics direct comparisons toward item-specific cues and answer demands rather than superficial features. They distinguish patterns that tend to make items harder from those that tend to make them easier, while recommending consistent comparison across both items.

D.1 USMLE

  • For each item, identify what the final step asks and whether the answer could be produced before reading the options; a stated fact supporting a specific wrong answer that must be overridden is the strongest hard-item signal.The rubric says to work through each item alone first, then compare.Source: §D.1 · p. 36
  • Price the final step separately from an easy diagnosis; when items remain level, favor specific-name retrieval over reasoning from the stem, then near-neighbour options.The named answer may be a drug, regimen, design, or other entity.Source: §D.1 · pp. 36–37
  • Specific drug or regimen retrieval, near-neighbour options, and choosing which test comes first are costly; a lone plausible option among clearly different alternatives is comparatively cheap.The rubric also treats superlatives among real risk factors as costly unless only one option is a cause.Source: §D.1 · p. 36
  • Statistics interpretation that requires distinguishing individual spread from mean precision is costly, while open-ended counselling answers are cheap and rule-based or end-of-life drug choices are expensive.The rubric also contrasts cheap restraint answers with cases where objective abnormalities must be overridden.Source: §D.1 · pp. 36–37
  • Apply the same basis to both items and disregard considerations that change with presentation order.The PANCE rubric also warns against relying on supposedly strong distractors or the number of reasoning steps.Source: §D.1 · p. 37, §D.2 · p. 39

D.2 PANCE

  • PANCE items default harder when they ask for mechanisms, criteria, labels, or thresholds, or when multiple options fit and the keyed choice wins by degree.Matching findings to adjacent terms and reversing a reflex without an explicit conflicting stem fact are also identified as hard patterns.Source: §D.2 · pp. 37–38
  • A distinctive finding explained only by the keyed answer, an obvious emergency action, or deterministic interpretation defaults easier; ambiguous diagnosis options push difficulty upward.A pathognomonic cue helps only when the topic is one everyone drills.Source: §D.2 · p. 38
  • Do not treat stem length, laboratory volume, images, option count, or management-versus-recall framing as decisive evidence of pairwise difficulty.The USMLE rubric likewise rejects stem length and pictures that merely illustrate what the text already states.Source: §D.2 · pp. 37, 39, §D.1 · p. 37
Loading 2610.39373v1…