PaperlayerView exact revision on arXiv

Source-linked AI summary

SpineBench: A Clinically Salient, Level-Aware Benchmark Powered by the SpineMed-450k Corpus

Ming Zhao, Wenhui Dong, Yang Zhang, Xiang Zheng, Zhonghao Zhang, Zian Zhou, Yunzhi Guan, Liukun Xu, Wei Peng, Zhaoyang Gong, Zhicheng Zhang, Dachuan Li, Xiaosheng Ma, Yuli Ma, Jianing Ni, Changjiang Jiang, Lixia Tian, Qixin Chen, Kaishun Xia, Pingping Liu, Tongshun Zhang, Zhiqiang Liu, Zhongyan Bi, Chenyang Si, Tiansheng Sun, Caifeng Shan

arXiv:2510.03160v3cs.CVcs.AI

TL;DR

Spine diagnosis requires level-aware, multimodal reasoning, but traceable instruction data and standardized spine benchmarks are lacking. SpineMed pairs a clinician-co-designed corpus with SpineBench; evaluations report weaknesses in contemporary LVLMs and consistent gains from SpineGPT.

  • Problem

    Spine diagnosis requires integrating X-ray, CT, and MRI at specific vertebral levels, yet traceable instruction data and standardized, clinically grounded spine benchmarks are lacking.

    Source: §1 · p. 2

  • Method

    The paper introduces clinician-co-designed SpineMed-450k and SpineBench, using a clinician-in-the-loop pipeline to curate multimodal questions, consultations, and reports.

    Source: §1 · pp. 2–3, §2.2 · p. 4

  • Results

    SpineGPT achieves 87.44% performance on SpineBench, substantially outperforming open-source alternatives, while contemporary open-source LVLMs show consistent weaknesses.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    Clinician feedback on exemplar outputs supports their diagnostic clarity and utility for planning.

    Source: §1 · p. 3

    The authors identify comprehensive direct comparisons with leading proprietary models, including GPT-4 and Gemini, as future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Spine disorders affect 619 million people globally and are a leading cause of disability, yet AI-assisted diagnosis remains limited by the lack of level-aware, multimodal datasets. Clinical decision-making for spine disorders requires sophisticated reasoning across X-ray, CT, and MRI at specific vertebral levels. However, progress has been constrained by the absence of traceable, clinically-grounded instruction data and standardized, spine-specific benchmarks. To address this, we introduce SpineMed, an ecosystem co-designed with practicing spine surgeons. It features SpineMed-450k, the first large-scale dataset explicitly designed for vertebral-level reasoning across imaging modalities with over 450,000 instruction instances, and SpineBench, a clinically-grounded evaluation framework. SpineMed-450k is curated from diverse sources, including textbooks, guidelines, open datasets, and ~1,000 de-identified hospital cases, using a clinician-in-the-loop pipeline with a two-stage LLM generation method (draft and revision) to ensure high-quality, traceable data for question-answering, multi-turn consultations, and report generation. SpineBench evaluates models on clinically salient axes, including level identification, pathology assessment, and surgical planning. Our comprehensive evaluation of several recently advanced large vision-language models (LVLMs) on SpineBench reveals systematic weaknesses in fine-grained, level-specific reasoning. In contrast, our model fine-tuned on SpineMed-450k demonstrates consistent and significant improvements across all tasks. Clinician assessments confirm the diagnostic clarity and practical utility of our model's outputs.

1 INTRODUCTION

Spine diagnosis requires level-aware integration of X-ray, CT, and MRI, but progress is constrained by missing traceable instruction data and clinically grounded benchmarks. SpineMed pairs a clinician-co-designed instruction corpus with SpineBench, and SpineGPT shows consistent benchmark gains despite weaknesses in contemporary LVLMs.

1 INTRODUCTION

  • Spinal diagnosis requires integrating X-ray, CT, and MRI findings at specific vertebral levels to identify pathology, grade severity, and plan interventions.The introduction identifies the absence of traceable instruction data and standardized, clinically validated spine benchmarks as constraints on progress.Source: §1 · p. 2
  • SpineBench evaluates clinically relevant axes including diagnosis, treatment, risk prognosis, coverage, granularity, and interpretability using clinician-developed items, error taxonomies, and rubrics.Its design emphasizes fine-grained, anatomy-centric reasoning and mistakes that matter in practice.Source: §1 · p. 2
  • Contemporary LVLMs show weaknesses in fine-grained, level-specific diagnosis and open-ended reasoning, while SpineGPT delivers consistent SpineBench improvements.Clinicians assessed exemplar outputs for decision relevance; Figure 1 presents SpineGPT benchmark performance.Source: Figure 1 · p. 1, §1 · p. 3
  • SpineMed-450k contains more than 450,000 instruction instances across multiple-choice and open-ended QA, multi-turn consultations, and report generation, alongside clinician co-designed SpineBench.The corpus uses a specialist-supported pipeline with anatomical integration and two-stage report refinement, and the benchmark includes approximately 1,000 real hospital cases.Source: §1 · p. 3

2 SPINEMED-450K DATASET

SpineMed-450k combines diverse orthopedic and spine sources with clinician-guided curation to produce multimodal, level-aware instruction data. Its workflow covers dataset collection and processing, task-specific generation, structured diagnostic reports, and clinical reasoning beyond perception-focused benchmarks.

2 SPINEMED-450K DATASET

  • SpineMed-450k is constructed through a clinician-in-the-loop pipeline covering collection, structured information extraction, de-identification and cleaning, dataset generation, and spinal diagnostic report annotation.The overview describes these as core stages, although it enumerates five.Source: §2 · p. 3

2.1 DATA COLLECTION

  • Sources include textbooks, guidelines, expert consensuses, question banks, open datasets, case reports, and approximately 1,000 de-identified multimodal hospital cases, with provenance tracked for derived items.Clinicians set inclusion criteria and selected decision-informative hospital images; the data includes text, CT, MRI, X-ray, and tables.Source: §2.1 · p. 3

2.2 DATASET CURATION

  • PaddleOCR extracts text and layout from academic sources, while Picture Context Matching links figures to captions and contextual descriptions before a GPT-5-mini consistency check filters mismatches.The extraction output is exported to Markdown while preserving document structure.Source: §2.2 · pp. 3–4
  • Expert-designed annotation schemas generate bilingual, multimodal questions from external sources, simulated consultations from sampled open-dataset image slices, and clinical-record tasks using locally deployed GLM-4.5V.The open-dataset process adaptively samples 25 slices per case and produces over 300 simulated consultations; clinical-record tasks include questions, interviews, and diagnostic reports.Source: §2.2 · p. 4
  • Spinal diagnostic reports use hospital reports and physician recommendations to simulate a complete clinical workflow across six dimensions, including imaging findings, diagnosis, treatment, risk, postoperative management, and diagnostic rationale.Treatment recommendations include patient-facing advice and a physician-facing, guideline-based decision tree.Source: §2.2 · pp. 4–5

2.3 COMPARISON WITH EXISTING BENCHMARKS

  • Unlike predominantly unimodal perception datasets such as VerSe and RSNA, SpineMed-450k targets multimodal synthesis, level-aware reasoning, and workflow-complete clinical instructions.Its workflow includes structured imaging findings, AI diagnosis, treatment recommendations, and risk assessment.Source: §2.3 · p. 5, Table 1 · p. 5

3 DATA STATISTICS

SpineMed-450K is a traceable, multimodal training dataset for orthopedic spine knowledge, with disease, hospital, source, and data-type diversity. Its statistics describe broad spinal-disease coverage and real clinical cases drawn from multiple hospitals.

3 DATA STATISTICS

  • SpineMed-450K is a multimodal training dataset for orthopedic spine knowledge in large language models, with traceability, broad coverage, diverse question types, and rich modalities.Its statistics include disease, hospital, source, and data-type diversity.Source: §3 · p. 5

3.1 DISEASE DIVERSITY COVERAGE

  • 47% of the orthopedic data is spinal diagnostic data, spanning 14 spine subconditions within seven orthopedic subspecialties.The spinal diagnostic data was sampled across disease categories to ensure uniform distribution.Source: §3.1 · p. 5, Figure 4 · p. 6

3.2 PATIENT SOURCE DIVERSITY

  • 1,000 real clinical cases from 11 hospitals span three recent years, with the largest hospital contributing 33% and the smallest 1%.The cases include patients of different genders, age groups, and physical conditions, and personal information was de-identified.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

3.3 DATA SOURCE AND QUESTION TYPE DIVERSITY

  • Tables 2 and 3 categorize dataset statistics by source and split, and by domain and task type, respectively.The supplied captions identify these table groupings but do not provide their underlying values.Source: Table 2 · p. 6, Table 3 · p. 6

3.4 DATA TYPE DIVERSITY

  • The collected cases include physical examination information, consultation records, X-rays, CT scans, and MRI images; case-level variation presents modeling challenges while approximating clinical scenarios.Data varies with hospital facilities and patient conditions.Source: §3.4 · p. 6

4 SPINEBENCH

SpineBench is constructed from sampled SpineMed-450k questions and reports, then reviewed by orthopedic surgeons. Its evaluation framework combines text-only and multimodal multiple-choice performance with diagnostic-report scoring.

4.1 BENCHMARK CONSTRUCTION

  • SpineBench was sampled from SpineMed-450k using 500 multiple-choice questions and 100 medical reports, covering 14 spinal sub-diseases and multiple sources.The final validated benchmark contains fewer items than the initial sample.Source: §4.1 · p. 7
  • 17 board-certified orthopedic surgeons reviewed the sampled benchmark items, correcting erroneous pairs and removing unsuitable questions; the final set contains 487 multiple-choice questions and 87 report prompts.The surgeons worked in three independent groups.Source: §4.1 · p. 7

4.2 EVALUATION METRICS

  • The overall evaluation framework combines text-only multiple-choice, multimodal multiple-choice, and diagnostic-report performance, with weights determined dynamically from category sample sizes.The report-generation criteria are presented across five key dimensions in Table 4.Source: §4.2 · p. 7, Table 4 · p. 7
  • The diagnostic-report score uses an expert-calibrated framework, with scores normalized to a 0–100 scale and dimensions indexed by section.The unified scoring system is intended to enable comparison across clinical tasks, from basic diagnostic reasoning to report generation.Source: §4.2 · p. 7

5 SPINEGPT: A SPECIALIZED CLINICAL COLLABORATOR

SpineGPT is introduced to validate SpineMed-450k through a three-stage curriculum for orthopedic spine care. Training progresses from general medical foundations to spinal specialization and then dialogue, report-generation, and long-reasoning tasks.

5 SPINEGPT: A SPECIALIZED CLINICAL COLLABORATOR

  • SpineGPT is introduced to rigorously validate the efficacy of SpineMed-450k, and Table 5 presents training configurations across stages.The supplied table caption identifies its scope but does not provide the configurations.Source: §5 · p. 8, Table 5 · p. 8

5.2 CURRICULUM LEARNING

  • The curriculum has three training stages that integrate distinct datasets and strategies to progressively strengthen performance in spinal health.It is designed to improve the model’s applicability and proficiency in orthopedic spine care.Source: §5.2 · p. 8
  • The foundational stage uses medical text and multimodal instruction data, followed by non-spinal SpineMed-450k data; the authors report that the latter significantly improved SpineBench performance.The cited text names medical-o1-reasoning-SFT, Medical-R1-Distill-Data, MedThoughts-8K, and 150,000 PubMedVision samples.Source: §5.2 · p. 8
  • The spinal-health stage trains on spine-related data and uses selected multiple-choice and open-ended questions to construct long reasoning chains for spine-surgery proficiency.Source: §5.2 · p. 8
  • The final stage trains on multi-turn dialogues, report generation, and long-chain reasoning instructions to develop dialogue and report-creation abilities.Source: §5.2 · p. 8

6 EXPERIMENTS

SpineBench results expose weaknesses in current LVLMs, while SpineGPT performs strongly among open-source models and is competitive with proprietary systems. Expert scoring correlates with automated LLM scoring, and Table 7 covers close-ended QA ablations.

6.1 COMPARISON AND ANALYSIS ON SPINEBENCH

  • Qwen2.5-VL-72B, despite having 72B parameters, achieves 79.88% average performance and a 63.80 cumulative medical-report-generation score; Medgemma-27B achieves a 76.66% average.Source: §6.1 · pp. 8–9
  • A 7.44-point text-to-image gap appears for GPT5 (87.41% on text; 79.97% on images), and open-source models also show modality gaps.GLM-4.5V has an 85.71% text score versus 81.35% on images; Qwen2.5-VL-72B has a 4.90-point gap.Source: §6.1 · p. 9
  • Our method achieves an 87.44% average score, outperforming all open-source models by 4.18+ points; on close-ended QA, it scores 87.89% versus Claude4’s 79.67% and GPT-4o’s 84.74%.Source: §6.1 · p. 9
  • A 7B-parameter SpineGPT reaches ∼98% of Gemini-2.5-Pro’s performance while using less than 7% of its parameters.The authors describe the model as suitable for local hospital deployment behind firewalls without external cloud APIs.Source: §6.1 · p. 9

6.2 HUMAN-EXPERT AGREEMENT ANALYSIS

  • Pearson correlations between LLM and expert scores range from 0.382 to 0.949, with most evaluation dimensions above 0.7.The authors interpret this alignment as supporting automated LLM scoring as a proxy for expert judgment.Source: §6.2 · pp. 9–10, Figure 5 · p. 10

6.3 ABLATIONS OF SPINEGPT

  • Table 7 presents a performance comparison of models on close-ended QA tasks for SpineGPT ablations.The supplied passage identifies the table’s scope but does not state its results.Source: Table 7 · p. 10

7 CONCLUSIONS, LIMITATIONS, AND FUTURE WORK

The authors conclude that SpineMed-450k and SpineBench support level-aware spine diagnosis and planning, and report that SpineGPT outperforms open-source alternatives. They also identify future work on dataset expansion and broader proprietary-model comparisons.

7 CONCLUSIONS, LIMITATIONS, AND FUTURE WORK

  • SpineGPT achieves 87.44% performance and substantially outperforms open-source alternatives, according to the conclusion.The authors say specialized instruction data enables clinically relevant AI capabilities for complex anatomical reasoning tasks.Source: §7 · p. 10
  • The authors plan to expand datasets, train models beyond 7B parameters, use reinforcement learning, and compare directly with GPT-4 and Gemini.They describe the direct comparisons as necessary to establish clear performance benchmarks.Source: §7 · p. 10

A.1 ETHICS STATEMENT

  • The paper states that patient privacy is protected through de-identification and that raw public images, copyrighted paragraphs, figures, and sensitive records are not redistributed.Its data-collection approach is described as adhering to institutional, legal, and copyright frameworks.Source: §A.1 · p. 17, Table 8 · p. 17

A.2 REPRODUCIBILITY STATEMENT

The reproducibility plan combines open materials and data pointers with controlled access to a de-identified hospital-data subset. The remaining supplied sections describe prior work, LLM use in manuscript editing, medical-report evaluation coverage, and an image-context matching algorithm.

A.2 REPRODUCIBILITY STATEMENT

  • The release plan opens SpineBench, code, and SpineMed-450k annotations, while using metadata pointers rather than redistributing raw public-benchmark images.A representative de-identified subset of sensitive hospital data is to be available under a Data Use Agreement; together with public pointers, it is intended to support reproduction of reported SpineBench metrics.Source: §A.2 · pp. 17–18

A.3 LLM USAGE

  • The authors used an LLM to polish manuscript language and improve readability, not to develop the research concepts, methods, experiments, or analyses.The authors retain responsibility for the manuscript content.Source: §A.3 · p. 18

C RELATED WORK

  • The related-work discussion frames level-aware diagnosis as combining X-ray, CT, and MRI findings and localizing them to vertebral levels.It argues that this integrative reasoning is not acquired from static descriptive datasets alone.Source: C RELATED WORK · pp. 18–19
  • Prior spine datasets are described as largely supporting single-modality perception, classification, or segmentation rather than integrated clinical reasoning.The paper contrasts those resources with the need to synthesize multimodal information into a diagnosis and treatment plan.Source: C RELATED WORK · p. 19
  • Earlier spine AI tools handle discrete segmentation and measurement tasks, leaving synthesis and planning to clinicians; SpineMed-450k and SpineBench target that gap.The paper presents them as infrastructure for AI systems acting as clinical collaborators in spine surgery.Source: C RELATED WORK · p. 19

D PERFORMANCE COMPARISON ON MEDICAL REPORT GENERATION

  • Table 9 compares LVLM performance on ten medical-report-generation dimensions, including imaging report, diagnosis, patient guidance, treatment planning, and risk prognosis.The caption also lists coverage, relevance, granularity, and explanation.Source: Table 9 · p. 20

E PICTURE CONTEXT MATCHING ALGORITHM

F DETAILED DISTRIBUTION ANALYSIS OF SPINEBENCH

The supplied distribution analysis describes the SpineBench testing QA set as covering diverse pathologies and data sources.

F DETAILED DISTRIBUTION ANALYSIS OF SPINEBENCH

  • The testing QA set contains 487 items with subspecialty tags, and the authors report coverage across diverse pathologies and data sources.The passage frames these statistics as evidence of clinical representativeness and unbiasedness.Source: F DETAILED DISTRIBUTION ANALYSIS OF… · p. 21

F.1 DISEASE SUBSPECIALTY DISTRIBUTION

SpineBench spans 14 spinal subspecialties, with substantial representation of tumor and osteomyelitis and complications.

F.1 DISEASE SUBSPECIALTY DISTRIBUTION

F.2 DATA SOURCE DIVERSITY

The testing data balances academic resources with real-world clinical data, including private hospital cases. The accompanying figures compare SpineGPT and ChatGPT-4o reports and catalog prompts for report quality and medical question generation.

F.2 DATA SOURCE DIVERSITY

  • 54.2% of testing data is Real-world Clinical Data and 45.8% Academic Resources; Hospital Databases contribute 33.5% in a zero-leakage evaluation environment.Source: §F.2 · p. 21, Table 11 · p. 21

G QUANTITATIVE COMPARISON OF SPINEGPT WITH GPT-4O

  • For adolescent idiopathic scoliosis, SpineGPT’s report is contrasted with ChatGPT-4o’s basic diagnostic and treatment recommendations, with differences in diagnostic depth, reasoning, and treatment specificity.Source: Figure 7 · p. 22, Figure 8 · p. 23, Figure 9 · p. 24

H PROMPTS

Loading 2510.03160v3…