PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique

arXiv:2610.06170v1cs.AIcs.CL

TL;DR

Broad medical benchmarks provide limited evidence about whether LLMs can be audited against narrow, evolving MS-MRI workflow knowledge. MS-Exam-Gen builds and audits a source-grounded candidate MCQ bank; 12 primary endpoints separated across a 46.9%–89.7% accuracy range.

  • Problem

    Broad medical benchmarks provide limited evidence that LLMs can be audited against narrow, evolving, source-grounded MS-MRI workflow knowledge.

    Source: §I · p. 1

  • Method

    MS-Exam-Gen builds an auditable candidate MCQ bank from indexed expert sources using seeded topic induction, evidence-grounded generation, automated audits, and model-panel calibration.

    Source: §I · pp. 1–2

  • Results

    12 non-construction-overlap primary endpoints separated accuracy from 46.9% to 89.7% on the fixed candidate bank.

    Source: §VII · p. 7

  • Takeaways & Limitations

    Generated construction labels remain metadata, while model-panel disagreement is a provisional calibration signal rather than psychometric validation.

    Source: D. Evaluation and Empirical Difficulty · p. 4, §VI · p. 6

    The benchmark tests source-grounded textual MCQs, not image interpretation or clinical competence, and lacks human or free-response baselines.

    Source: §VI · p. 6

Abstract

from arXiv · show

Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.

I. INTRODUCTION

Broad medical benchmarks provide limited evidence that LLMs can be audited against narrow, evolving MS-MRI textual knowledge. MS-Exam-Gen proposes source-grounded MCQ evaluation of that knowledge, not direct image interpretation.

I. INTRODUCTION

  • MS-MRI textual reasoning spans dissemination rules, lesion morphology, acquisition protocols, longitudinal activity, treatment-safety surveillance, and clinically important mimics.Diagnostic criteria and MRI recommendations periodically revise how MS-MRI findings should be interpreted and reported –.Source: §I · p. 1
  • Broad medical QA and MS imaging benchmarks do not test auditable, source-grounded textual MS-MRI criteria, protocols, reporting, monitoring, and differential knowledge –,,.The target knowledge is narrow, evolving, and tied to a specific clinical workflow.Source: §I · p. 1
  • MS-Exam-Gen indexes expert sources, induces an exam-oriented taxonomy with seeded retrieval, generates grounded MCQs, and applies schema, evidence, redundancy, anti-cue, and consistency checks.The workflow also uses a same-family consistency screen and stable item identifiers.Source: §I · pp. 1–2
  • The locked candidate bank contains 3,058 MCQs across 16 topics and 53 subtopics, with item-level empirical difficulty estimated from model-panel disagreement.The evaluation assesses discriminative utility as well as item-level difficulty.Source: §I · p. 2
  • Without human expert item review, the resource remains an automatically filtered candidate set, and generated difficulty labels require post-generation cue and option-order analyses.The paper does not frame the benchmark as a clinically certified examination.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior medical QA and MS imaging benchmarks address broad medical knowledge or image-analysis tasks, rather than auditable textual MS-MRI subspecialty knowledge. MS-Exam-Gen is positioned as a complementary, source-grounded benchmark family.

II. BACKGROUND AND RELATED WORK

  • Broad medical QA datasets measure broad knowledge but are not organized around a subspecialty imaging corpus or item-level MS-MRI source provenance,,,,.The cited resources include MedQA, PubMedQA, MMLU, MedMCQA, and MultiMedQA-style evaluations.Source: §II · p. 2
  • MS imaging challenges such as MSSEG and longitudinal lesion segmentation evaluate image-analysis tasks, unlike source-grounded textual reasoning over MS-MRI guidelines and differential diagnosis,.The related work also distinguishes visual understanding, report generation, and general radiology capability from this textual focus.Source: §II · p. 2
  • LLM-assisted generation can scale construction, while judge bias and option-order or answer-label sensitivity motivate candidate framing, quality audits, raw-response records, and panel-disagreement calibration –,,.Model-panel disagreement is treated as a provisional review-prioritization signal rather than relying on nominal construction labels.Source: §II · p. 2
  • MS-Exam-Gen complements prior benchmark families with source-grounded textual MS-MRI items, item-level evidence, preserved raw responses, anti-cue filtering, and empirical difficulty calibration.The positioning is distinct from general medical knowledge, radiology images or reports, and MS lesion segmentation.Source: §II · p. 2

III. MS-EXAM-GEN FRAMEWORK

MS-Exam-Gen turns retrieved MS-MRI evidence into an auditable candidate MCQ bank and evaluates models using stable item records. Its source-grounded design does not imply expert confirmation of clinical claims.

III. MS-EXAM-GEN FRAMEWORK

  • “Source-grounded” means generated content uses retrieved evidence with the original source and text section recorded, not that experts confirmed the clinical claims.The design goal is auditability across source indexing, topic induction, item acceptance or rejection, and model measurement.Source: §III · p. 2
  • Table I entries mark benchmark-family design goals, not feature absence from every dataset; Partial and Variable denote support not specific to source-grounded MS-MRI textual MCQs.The table caption identifies its scope as positioning relative to related benchmark families.Source: §III · p. 2, Table I · p. 2
  • Figure 1 traces source indexing and seeded retrieval through taxonomy, MCQ generation, automated audits, and same-family consistency screening, with regeneration of malformed or cueable items.Algorithm 1 additionally details benchmark locking, model evaluation, and empirical calibration.Source: Figure 1 · p. 3
  • GPT-5 mini handled dataset-construction taxonomy, generation, repair, and consistency screening; its score is excluded from primary analyses as construction-overlap.Taxonomy and screening used temperature 0.0, generation 0.3, and repair 0.1.Source: §III · p. 3
  • Algorithm 1 repairs or rejects items, stores accepted items with stable IDs and provenance, evaluates models with a standardized A–D prompt, and summarizes results by model, topic, label, and empirical difficulty.The stored per-item records include raw responses, parsed options, parse status, and correctness.Source: §III · p. 3, Figure 1 · p. 3

A. Corpus Indexing and Topic Induction

MS-Exam-Gen indexes a curated MS-MRI source corpus with provenance metadata and uses seeded retrieval to induce an exam-oriented topic taxonomy. Its checklist audit does not guarantee exhaustive coverage, and character-based chunking may split related material.

A. Corpus Indexing and Topic Induction

  • The curated corpus includes consensus guidelines, diagnostic criteria, reviews, protocols, and educational sources selected for authority, recency, and concept coverage.A source manifest records inclusion status, document identifiers, metadata, and chunk counts.Source: A. Corpus Indexing and Topic Induction · p. 3
  • Documents are split into 1,200-character chunks with 200-character overlap, provenance offsets, and content hashes before embedding into Chroma.The corpus is embedded with text-embedding-3-large.Source: A. Corpus Indexing and Topic Induction · p. 3
  • Eighteen seeded queries retrieve passages that are summarized into an exam-oriented taxonomy of topics and subtopics.Up to four passages per query and 60 distinct passages are used; a checklist compares the taxonomy with predefined seed concepts.Source: A. Corpus Indexing and Topic Induction · p. 3
  • The checklist-based coverage audit does not guarantee exhaustive MS-MRI topic coverage, and character-based chunking may split tables, lists, or neighboring criteria..Source: A. Corpus Indexing and Topic Induction · p. 3

B. Source-Grounded MCQ Generation

Generation grounds each four-option single-best-answer item in retrieved subtopic evidence, with distinct retrieval and prompt constraints for standard difficulty and multi-source synthesis.

B. Source-Grounded MCQ Generation

  • Each item includes topic and subtopic, a generated construction label, question type, correct option, explanation, and source evidence.The generator retrieves supporting passages for each subtopic, and options are shuffled after generation.Source: B. Source-Grounded MCQ Generation · p. 3
  • Standard Easy/Medium/Hard items use k = 3 chunks, while multi-source synthesis uses k = 30, at most two chunks per source, and ten snippets.The synthesis stream is stored under the “Very Hard” generator label, which the paper does not treat as validated difficulty.Source: B. Source-Grounded MCQ Generation · p. 3
  • Easy/Medium prompts target concept recognition and interpretation; Hard and synthesis prompts require non-recall stems, balanced options, and plausible near-miss distractors.Construction labels remain metadata and are calibrated empirically after model evaluation.Source: B. Source-Grounded MCQ Generation · p. 3

C. Automated Quality Auditing

Automated audits screen candidate items for schema, redundancy, cueing, and consistency problems, while treating their thresholds and model-family judgments as construction safeguards rather than clinical validation.

C. Automated Quality Auditing

  • Deterministic audits require a stem, four A–D options, one correct option, metadata, explanation, and source evidence, and flag phrasing, label, contradiction, duplicate, and trivia issues.Near-duplicate flags are retained in the final audit for transparency.Source: C. Automated Quality Auditing · pp. 3–4
  • Generation-time near-duplicate filtering uses normalized string similarity ≥0.90; synthesis candidates also receive semantic duplicate checks at embedding cosine similarity ≥0.90.The final audit reports remaining near-duplicate flags even when they do not meet exclusion criteria.Source: C. Automated Quality Auditing · p. 4
  • Hard-item cue thresholds require stems ≥18 words, options 8–60 words, length advantage ≤4 words, and correct/distractor ratio ≤1.15.Synthesis thresholds are stricter: stems ≥35 words, options 14–70 words, advantage ≤2 words, ratio ≤1.08, and no uniquely longest correct option; these are construction heuristics, not validated psychometric criteria.Source: C. Automated Quality Auditing · p. 4
  • The same-family consistency screen checks answerability, key and explanation consistency, and uniqueness; major issues reject an item, but passing does not establish clinical correctness.It uses stored gold context or, when unavailable, a fresh k = 3 retrieval context saved with the verdict.Source: C. Automated Quality Auditing · p. 4
  • A leave-one-topic-out logistic selector tests residual cueing from structural option features alone and with TF–IDF stem overlap, fitting the same pipeline to both banks.The selector uses C = 0.5.Source: C. Automated Quality Auditing · p. 4

D. Evaluation and Empirical Difficulty

Evaluation aggregates parsed model answers by stable item identifier and reports overall and stratified accuracy, while defining empirical difficulty as a panel-dependent review signal rather than psychometric validation.

D. Evaluation and Empirical Difficulty

  • Model predictions are joined by stable item ID; parsed responses are scored, while unresolved failures are recorded as parse failures rather than scored correct.Reported results include overall, topic-level, construction-label, and empirical-difficulty analyses.Source: D. Evaluation and Empirical Difficulty · p. 4
  • Empirical difficulty is the number of 12 non-construction-overlap endpoints that miss an item, a panel-dependent calibration signal rather than psychometric validation.It may reflect ambiguity, source disagreement, prompt sensitivity, or model-panel composition.Source: D. Evaluation and Empirical Difficulty · p. 4
  • Uncertainty is estimated with 4,000-replicate hierarchical bootstrapping over 16 construction batches and then items, with a parallel analysis over 13 semantic parents testing taxonomy-overlap sensitivity.Additional analyses test key disagreement and option order.Source: D. Evaluation and Empirical Difficulty · p. 4
  • Standard Hard and the flagged legacy pool were excluded and replaced by refreshed strata; resulting review queues prioritize adjudication, while the stratified pilot does not estimate full-bank option-order effect prevalence.The supplied passages identify no item-level or model-level outcomes for these analyses.Source: D. Evaluation and Empirical Difficulty · p. 4

IV. EXPERIMENTAL SETUP

The experimental setup uses a curated MS-MRI source corpus, a locked candidate benchmark, and a reproducible multi-model evaluation with a stratified option-order pilot.

IV. EXPERIMENTAL SETUP

  • The non-exhaustive corpus comprises 66 indexed sources and 4,289 chunks, covering diagnostic criteria, MRI recommendations, mimics, lesion markers, and imaging and monitoring topics.The corpus began with 72 raw files, and copyrighted source PDFs are not redistributed.Source: §IV · p. 4
  • The locked benchmark contains 3,058 schema-valid MCQs across 16 topics and 53 subtopics: 646 Easy, 646 Medium, 883 Hard, and 883 multi-source synthesis items.The legacy Hard/multi-source pool had 1,217 of 1,293 items flagged as cueable or weak and was replaced by refreshed streams.Source: §IV · p. 4
  • Thirteen LLMs were evaluated, with GPT-5 mini marked as construction-overlap and excluded from the 12-endpoint primary panel of 36,696 responses.Models used a standardized A–D-only prompt and temperature zero where supported.Source: §IV · pp. 4–5
  • Evaluation records preserve item-level responses, parsing, correctness, and run metadata, and a 100-item pilot tested four cyclic option orders with 11 available endpoints.Full-bank option-order counterbalancing remains future validation.Source: §IV · p. 5

V. RESULTS

Across the 12 primary endpoints, the locked benchmark separated model accuracy, while audits found reduced measurable cueing but residual answer cues and position sensitivity. Item disagreement and topic-level results also varied across the benchmark.

Overall endpoint separation

  • 89.70% accuracy for Gemini 2.5 Flash contrasted with 46.89% for Phi-4 mini across the 12 primary endpoints.Table IV reports the results on the locked 3,058-item benchmark.Source: Table IV · p. 5, §V · p. 5

Benchmark auditing and empirical calibration

  • Structure-only key prediction fell from 65.7% to 36.4%, structure-plus-stem prediction from 64.4% to 43.8%, and uniquely-longest-key prevalence from 68.1% to 6.6%.These comparisons concern the replaced legacy and locked Hard/synthesis strata; the refreshed construction reduced measurable cueing.Source: §V · p. 5, Figure 2 · p. 5

Empirical difficulty

  • 25.5% of items were missed by at least four primary endpoints, while 53.0% were missed by zero or one.Leaving out any endpoint retained miss-count correlation ≥0.934 and ≥96.2% agreement on the four-endpoint flag.Source: §V · p. 5, Figure 2 · p. 5

Topic-level variation

  • Accuracy-independent parent means were lowest for RIS/CIS (67.9%), mimics/differential diagnosis (72.5%), diagnostic criteria (74.3%), and reporting/acquisition/safety (74.5%).Figure 3 retains all 16 topic labels and item counts, with cells showing percentages and the final column showing each original topic’s mean.Source: §V · p. 5, Figure 3 · p. 6

Quality and audit outcomes

  • The locked full bank remained above chance for structure-only key prediction (38.3%) and structure-plus-stem prediction (41.2%), so it was not cue-free.The audit flagged 1,217 items from the legacy Hard/multi-source synthesis pool before refreshed generation; the compiled evaluation had zero unresolved parse failures.Source: §V · pp. 5–6

Key-risk and option-order sensitivities

  • The option-order pilot retained coarse ordering (ρ = 0.991), but mean absolute score change was 2.82 points, semantic stability was 56.9%, and 9/11 endpoints showed position effects.Removing the majority and stricter non-key queues changed scores without erasing broad endpoint separation.Source: §V · p. 6

VI. DISCUSSION

MS-Exam-Gen measures source-grounded MS-MRI textual MCQ performance, not image interpretation or clinical authority. Discussion findings support empirical calibration of construction labels, heterogeneous model panels, and cautious use of lower-accuracy groups for review.

What the Benchmark Measures

  • Scores measure source-grounded MS-MRI textual MCQs, not direct image interpretation, clinical authority, or performance relative to human readers.Without human or free-response baselines, scores are not measures of clinical competence or human-relative difficulty.Source: §VI · p. 6

Why Construction Labels Need Calibration

  • Hard and synthesis accuracies fell by 4.6 and 6.4 percentage points after refreshed generation, but construction labels remain metadata rather than validated difficulty categories.The identical shallow-cue diagnostic more directly supports reduced measurable cueing; empirical calibration uses the number of models incorrect per item.Source: §VI · p. 6

Why Retain the Synthesis Stratum

  • Source count was 7.96 versus 2.24 per item for synthesis and Hard, respectively, but was weakly associated with accuracy (ρ = 0.054).The synthesis stratum represents multi-source construction, not validated difficulty.Source: §VI · p. 6

Value of a Heterogeneous Model Panel

  • 46.9–89.7% accuracy across the heterogeneous panel exposed discriminative items and performance bands, while seven API-served endpoints retained a 9.09-point range.Adding models with varied performance profiles reduced ceiling effects.Source: §VI · p. 6

Interpretation of Lower-Accuracy Item Groups

  • Lower-accuracy groups cover reporting standards, MS mimics, RIS/CIS presentations, and diagnostic criteria, so their descriptive differences prioritize review rather than establish intrinsic clinical difficulty.Three overlapping pairs and their parent means are marked.Source: §VI · pp. 6–7

Audit Lessons for Automatic Benchmark Construction

  • Schema-validity checks alone missed cueable items; refreshed prompts and anti-cue filters improved the problem, but stricter prompts did not guarantee psychometric ordering.Residual cues, key-risk queues, and order effects show automated screens do not certify validity.Source: §VI · p. 7

Threats to Validity

  • The automatically filtered benchmark lacks human MS-imaging-expert adjudication, so answer-key errors, ambiguous distractors, or clinically weak items may remain.Expert adjudication is required before educational or clinical deployment.Source: §VI · p. 7

Prospective Validation Roadmap

  • Prospective work should add expert adjudication, option-order counterbalancing, domain-transfer testing, and separate stable, stress-test, and refreshed item sets.The roadmap also calls for validation across topics and empirical-difficulty bins.Source: §VI · p. 7

VII. CONCLUSION

MS-Exam-Gen provides an auditable candidate benchmark for source-grounded MS-MRI textual knowledge. Across primary endpoints, performance varied broadly; the benchmark supports model auditing and targeted review, not clinical deployment without expert adjudication.

VII. CONCLUSION

  • The 3,058-item benchmark spans 16 topics and 53 subtopics, and 12 primary endpoints scored from 46.9% to 89.7%.Topic-level analysis identifies lower-accuracy item groups for targeted review.Source: §VII · p. 7
  • An auditable subspecialty workflow separates endpoint performance on a fixed bank, while generated construction labels require empirical calibration.The conclusion states that labels must be calibrated empirically rather than accepted at face value.Source: §VII · p. 7
  • The candidate benchmark is intended for model auditing and construction research, not clinical education or credentialing without expert adjudication.Future work includes expert adjudication, full option-order counterbalancing, domain-transfer experiments, and stronger prospective validation.Source: §VII · p. 7

Related papers

Loading 2610.06170v1…