PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis

Jing Hao, Yuxuan Fan, Yanpeng Sun, Kaixin Guo, Lizhuo Lin, Jinrong Yang, Qi Yong H. Ai, Lun M. Wong, Hao Tang, Kuo Feng Hung

arXiv:2509.09254v1cs.CVcs.MM

TL;DR

Panoramic X-rays are challenging to interpret, yet oral radiology is largely missing from general medical benchmarks and instruction resources. The paper introduces MMOral and MMOral-Bench, evaluates current LVLMs, and reports both limited benchmark performance and gains from MMOral-based fine-tuning.

  • Problem

    Oral radiology, especially panoramic X-ray interpretation, remains largely absent from medical benchmarks and instruction resources despite the modality’s role in assessing oral health.

    Source: §1 · p. 2

  • Method

    The authors build MMOral, a 20,563-image dataset with 1.3 million instruction instances, and MMOral-Bench, a five-dimension evaluation suite, then fine-tune Qwen2.5-VL-7B as OralGPT.

    Source: §1 · pp. 2–3

  • Results

    GPT-4o reaches only a 41.45% average score on MMOral-Bench, while one epoch of SFT with MMOral instruction data improves OralGPT’s average score by 24.73%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MMOral and OralGPT are proposed as resources for advancing digital dentistry and multimodal AI in oral healthcare.

    Source: §6 · p. 10

    MMOral is limited in imaging-modality diversity because it focuses on panoramic X-rays; future work aims to add other 2D and 3D oral imaging modalities.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent advances in large vision-language models (LVLMs) have demonstrated strong performance on general-purpose medical tasks. However, their effectiveness in specialized domains such as dentistry remains underexplored. In particular, panoramic X-rays, a widely used imaging modality in oral radiology, pose interpretative challenges due to dense anatomical structures and subtle pathological cues, which are not captured by existing medical benchmarks or instruction datasets. To this end, we introduce MMOral, the first large-scale multimodal instruction dataset and benchmark tailored for panoramic X-ray interpretation. MMOral consists of 20,563 annotated images paired with 1.3 million instruction-following instances across diverse task types, including attribute extraction, report generation, visual question answering, and image-grounded dialogue. In addition, we present MMOral-Bench, a comprehensive evaluation suite covering five key diagnostic dimensions in dentistry. We evaluate 64 LVLMs on MMOral-Bench and find that even the best-performing model, i.e., GPT-4o, only achieves 41.45% accuracy, revealing significant limitations of current models in this domain. To promote the progress of this specific domain, we also propose OralGPT, which conducts supervised fine-tuning (SFT) upon Qwen2.5-VL-7B with our meticulously curated MMOral instruction dataset. Remarkably, a single epoch of SFT yields substantial performance enhancements for LVLMs, e.g., OralGPT demonstrates a 24.73% improvement. Both MMOral and OralGPT hold significant potential as a critical foundation for intelligent dentistry and enable more clinically impactful multimodal AI systems in the dental field. The dataset, model, benchmark, and evaluation suite are available at https://github.com/isbrycee/OralGPT.

1 Introduction

The paper addresses the lack of dental-specific resources for interpreting panoramic X-rays and introduces MMOral, its benchmark, and OralGPT. Evaluations reveal substantial limitations in current LVLMs, alongside improvement from fine-tuning on MMOral data.

MMOral and MMOral-Bench

  • MMOral provides 20,563 annotated panoramic X-rays and 1.3 million instruction instances across four task formats, while MMOral-Bench covers five diagnostic dimensions.The formats include attribute extraction, report generation, visual question answering, and image-grounded dialogue.Source: §1 · p. 2, Figure 1 · p. 2

Motivation

  • Panoramic X-rays pose interpretive challenges because dense anatomy and fine-grained pathological cues require dental-specific instruction data and evaluation.The paper identifies oral radiology as largely absent from general-purpose medical benchmarks and instruction resources.Source: §1 · p. 2

Evaluation findings

  • GPT-4o achieves only a 41.45% average score on MMOral-Bench, and current LVLMs also struggle with open-ended questions and fine-grained teeth-related questions.The evaluation also finds no clear advantage for existing medical LVLMs over general-purpose models in dentistry.Source: §1 · p. 3

OralGPT

  • 24.73% improvement in OralGPT’s average MMOral-Bench score follows one epoch of SFT on MMOral instruction data.OralGPT is based on Qwen2.5-VL-7B; the SFT data include MMOral-Report, MMOral-VQA, and MMOral-Chat.Source: §1 · p. 3

Contributions

  • The authors assess 64 LVLMs on MMOral-Bench and present OralGPT as a fine-tuned model for panoramic X-ray analysis.The evaluation includes 53 publicly available models and 11 proprietary models.Source: §1 · p. 3

2 MMOral Dataset Curation

MMOral is curated through a sequence that turns panoramic X-rays into extracted anatomical information, reports, and instruction-following examples. Its construction combines specialist-model extraction, post-processing, LLM-assisted report revision, and question-and-dialogue generation.

2 MMOral Dataset Curation

  • 20,563 panoramic X-rays were collected from TED3 and the dataset proposed by Hoang Viet Do, then filtered for duplicates within a four-step curation pipeline.The pipeline is described as four sequential steps.Source: §2.1 · p. 3, §2 · p. 3

2.1 Images and Visual Specialists Construction

  • Ten visual specialist models recognize 49 anatomical categories, with overlapping category spaces intended to support reliable extraction.An off-the-shelf OpenOCR model detects image acquisition times.Source: §2.1 · pp. 3–4

2.2 Anatomical Structure Extraction

  • The post-processing pipeline deduplicates overlapping detections, links findings and treatments to tooth numbers through spatial relationships, and inserts domain-specific clinical knowledge.The pipeline comprises eight systematic steps.Source: §2.2 · p. 4

2.3 Report Generation

  • Grounding captions preserve coordinates, categories, and confidence scores, while medical reports summarize findings and recommendations in three specialist-informed sections.Reports are generated from captions with an LLM, then corrected by GPT-4-turbo using manually prepared error rules and revision logs.Source: §2.3 · p. 4

2.4 Instruction Data Generation

  • Instruction data include template- and LLM-generated closed- and open-ended QA pairs mapped to five clinical dimensions, plus multi-turn patient-friendly dialogue with definite answers.The dimensions are Teeth, Patho, HisT, Jaw, and SumRec.Source: §2.4 · pp. 4–5

3 MMOral Dataset Analysis

MMOral combines a large, varied panoramic X-ray instruction resource with several quality-control strategies. These include overlapping specialist detections, report correction, and evaluation by professional dentists.

3.1 Data Statistics

  • MMOral contains 20,563 images and 1.3 million instruction instances across four sub-datasets, with an average of 44 bounding boxes per image and 49 anatomical subcategories.The four sub-datasets address visual perception, report generation, visual question answering, and image-grounded dialogue.Source: §3.1 · p. 5

3.2 Strategies for Ensuring Data Quality

  • Ten visual specialist models have overlapping category spaces, and post-processing combines their detections to provide metadata for report and instruction construction.The passage describes ten structures validated by at least two specialists, with final results obtained through post-processing.Source: §3.2 · pp. 5–6
  • 95.45% of preliminary medical reports were successfully corrected after eight error-based rules guided GPT-4-turbo revisions.Revision logs were analyzed to assess whether reports were corrected.Source: §3.2 · p. 6
  • Two professional dentists rated image and report quality, and the reported average scores for all aspects exceeded 3.5 on a five-level scale.Report aspects included correctness, completeness, relevance, and clarity.Source: Figure 3 · p. 5, §3.2 · p. 6

4 MMOral-Bench

MMOral-Bench is built from manually selected and checked panoramic X-ray cases, with questions organized for multi-dimensional clinical analysis. It evaluates closed-ended and open-ended answers using different scoring procedures.

4.1 Benchmark Construction

  • MMOral-Bench contains 100 manually selected images, 500 closed-ended questions, and 600 open-ended questions, with cases checked for quality and answerability.The source images are selected from the dataset proposed by Hoang Viet Do, and incorrect answers are re-annotated.Source: §4.1 · p. 6
  • Questions cover five clinical dimensions—Teeth, Patho, HisT, Jaw, and SumRec—and each QA pair can map to one or more dimensions by clinical intent.The multi-dimensional mapping supports analysis across diagnostic categories.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • Closed-ended questions are scored by accuracy, while open-ended answers receive GPT-4-turbo scores from 0 to 1 using few-shot examples.The evaluation uses nine in-context examples and is integrated into VLMEvalKit.Source: §4.2 · p. 6

5 Experiments

The experiments benchmark LVLMs on MMOral-Bench, expose weaknesses across model types and question categories, and test whether MMOral instruction data improves performance through fine-tuning. Case studies illustrate errors on both closed- and open-ended questions.

5.1 Experimental Setups

  • The study evaluates 64 LVLMs zero-shot, showing 36 results in the main text, and applies one-epoch SFT to Qwen2.5-VL-7B and LLaVA-Next-13B.Results for LLaVA-Next-13B SFT are provided in the supplementary materials.Source: §5.1 · pp. 6–7

5.2 Evaluation Results

  • On MMOral-Bench, even GPT-4o achieves only 41.45% overall performance, illustrating the significant challenges current LVLMs face interpreting complex panoramic X-rays with anatomically dense structures and fine-grained pathological patterns.Source: §5.2 · p. 7
  • Nearly all LVLMs performed worse on open-ended than closed-ended QA, and 33 of 53 open-source models scored below 25% on open-ended QA.Figure 4 compares performance on both question types across LVLMs.Source: Figure 4 · p. 8, §5.2 · p. 8
  • Medical-specific LVLMs did not outperform general-purpose models; HealthGPT had the strongest medical-specific average score at 39.59%.That score remained below those of Ovis2, GPT-4o, and Claude-3-7-Sonnet.Source: §5.2 · p. 8
  • Models performed relatively well on Jaw questions but worse on fine-grained Teeth, HisT, and Patho categories.These results indicate a remaining need for fine-grained dental visual understanding and reasoning.Source: §5.2 · p. 8

5.4 Case Study

  • Case studies show errors in FDI tooth numbering, caries and abscess detection, and recognition of crowns and root-canal treatment; some proprietary models also refuse dental questions.In the open-ended example, GPT-4-Turbo missed root-canal treatment while MedDr missed crown restoration on teeth #36 and #46.Source: Figure 5 · p. 9, §5.4 · p. 9

5.3 Efficacy Validation of MMOral Instruction Data

  • 44.53% average score was reached from a 21.46% baseline after SFT with MMOral-report and MMOral-VQA; adding MMOral-Chat improved the score by another 1.66%.Including MMOral-Chat improved open-ended QA while slightly reducing closed-ended QA performance.Source: Table 3 · p. 8, §5.3 · p. 9

6 Conclusion

The paper presents MMOral and OralGPT for panoramic X-ray analysis, where current medical LVLMs perform unsatisfactorily. The authors identify limited imaging-modality diversity as a dataset limitation.

6 Conclusion

  • MMOral is a panoramic X-ray dataset and benchmark, and OralGPT is a multimodal vision-language model designed for panoramic X-ray analysis.The authors report that current universal medical-specific LVLMs perform unsatisfactorily on panoramic X-rays, which contain dense anatomical structures and fine-grained pathological cues.Source: §6 · p. 10
  • MMOral’s imaging-modality coverage is narrower than that of existing medical instruction datasets.Future work aims to include additional 2D oral images and 3D modalities, including CBCT and MRI.Source: §6 · p. 10

Supplementary Materials

The supplementary material details MMOral’s image sources and processing, report and instruction generation, and related work. It also describes the dataset’s visualization materials.

A Related works

  • Existing general-purpose and specialized medical benchmarks cover diverse domains and modalities, but oral radiology, particularly panoramic X-ray understanding, remains overlooked.Panoramic X-rays are widely used as a primary diagnostic source for assessing oral health.Source: A Related works · p. 16

B.1 Image and Visual specialists construction

  • MMOral’s 20,563 images come from TED3 and Hoang Viet Do’s dataset; ten visual specialist models detect 49 structures across diverse dental conditions.The source images are deduplicated, and specialist models are fine-tuned on public panoramic-radiograph datasets.Source: §B.1 · pp. 16–17

B.2 Anatomical structure Extraction

  • Ten overlapping visual specialists detect anatomical and pathological structures, which a post-processing pipeline integrates and associates with tooth notations using spatial relationships.The integration and relationship-generation pipeline comprises eight steps.Source: §B.2 · p. 17

B.3 Report generation

  • Reports follow three dentist-informed sections, and a two-stage process uses DeepSeek to draft reports from grounding captions before GPT-4-Turbo revises them.The authors note that generating accurate reports from location captions involves multiple cognitive subtasks; revision logs support human verification of changes.Source: §B.3 · pp. 17–20

B.4 Instruction Data Generation

  • GPT-4-Turbo generates visual question-answering and image-grounded conversation data from refined medical reports.The prompts for MMOral-VQA and MMOral-Chat are shown in Figures 9 and 10, respectively.Source: §B.4 · p. 20

C MMOral Visualization

  • Word clouds visualize frequent words in MMOral-Report, MMOral-VQA, and MMOral-Chat, alongside examples from MMOral-Attribute and MMOral-Report.The examples are referenced in Figures 14–17.Source: C MMOral Visualization · p. 20

D.1 Evaluation Metrics

Closed-ended responses are evaluated with accuracy, while open-ended answers receive GPT-4-Turbo scores against the question and ground truth. The section also describes score aggregation and the evaluation framework.

D.1 Evaluation Metrics

  • Closed-ended questions are scored by accuracy after regular expressions extract the answer option; if no valid answer is found, a multiple-choice option is selected randomly.The extraction pipeline selects the response option appearing most often.Source: §D.1 · p. 20
  • Open-ended responses are scored by GPT-4-Turbo on a 0-to-1 scale using the question, ground truth, and model prediction.The few-shot prompt includes examples of fully correct, incorrect, and partially correct answers.Source: §D.1 · p. 21
  • The evaluation script is prepared for MMOral-Bench, with future support planned for the VLMEvalKit framework.The text defines subcategory membership as set C and its sample count as Nc.Source: §D.1 · p. 22

D.2 LLM as the Evaluator for Open-ended Questions: A Feasibility Analysis

Dentist scores broadly align with GPT-4-Turbo’s evaluation of open-ended answers, though dentists score more strictly overall. Repeated evaluations also show low score variation.

D.2 LLM as the Evaluator for Open-ended Questions: A Feasibility Analysis

  • Dentist A’s score differences are smaller for Teeth, Patho, HisT, and SumRec but larger for Jaw and Report; Dentist B’s overall difference is 0.37 points.The authors associate subcategory score fluctuations with subjective evaluator perceptions.Source: §D.2 · p. 25
  • Dentist-assigned absolute differences on the Overall metric are approximately 1 point lower than LLM-based evaluation differences for GPT-4o and HealthGPT-XL32.The authors interpret the scoring trends as aligned, while noting that dentists are more stringent.Source: §D.2 · p. 24, Table 6 · p. 25
  • Across five GPT-4-Turbo evaluations, overall-score standard deviations stay below 0.45 and coefficients of variation are around 1% for four LVLMs.Reported standard deviations include 0.434 for Ovis2-34B and 0.096 for Qwen2.5-VL-7B.Source: Table 7 · p. 25, §D.2 · p. 25

D.3 Evaluation results

The section reports zero-shot evaluation of 64 LVLMs on MMOral-Bench and one-epoch supervised fine-tuning results for two models using MMOral instruction data.

D.3 Evaluation results

  • Zero-shot evaluation was conducted across 64 LVLMs on MMOral-Bench.The results are presented in Table 8.Source: §D.3 · p. 25
  • Figure 13 reports category means and standard deviations across five repeated evaluations of four LVLMs’ predictions.Source: Figure 13 · p. 26

E Efficacy Validation of MMOral Instruction Data

  • 24.73% and 18.42% average-score improvements were reported for Qwen2.5-VL-7B and LLaVA-Next-13B-hf, respectively, after one epoch of fine-tuning on combined MMOral instruction data.The data combined MMOral-Report, MMOral-VQA, and MMOral-Chat.Source: E Efficacy Validation of MMOral… · p. 26

F Limitations

The authors flag unverified ground-truth labels as a limitation and describe overlapping-model checks and report revision as safeguards.

F Limitations

  • Ground-truth label accuracy was not independently validated, and one centre could not manually verify the annotated data quickly.The cited scale is 10 datasets comprising 28,375 images.Source: F Limitations · p. 26
  • Overlapping visual-specialist models validate at least ten anatomical or pathological structures, with final results obtained through post-processing.The examples include caries, periapical lesions, implants, and crowns.Source: F Limitations · pp. 26–27
  • A two-stage report-revision process used generated drafts, manual review, and eight prompting rules; revision logs show 95.45% of reports were successfully corrected.The authors call for third-party validation of ground-truth accuracy.Source: F Limitations · pp. 27–28

G Experiments Compute Resources

The paper reports experiment costs and hardware, then provides closed- and open-ended QA examples in its case study.

G Experiments Compute Resources

  • The experiments cost approximately 1000 USD, including about 600 USD for dataset construction and 400 USD for LVLM evaluation; SFT used 4×NVIDIA A100 80G GPUs.Source: G Experiments Compute Resources · p. 28

H Case Study

Loading 2509.09254v1…