Source-linked AI summary
A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer
Haengbok Chung, SunGyu Kim, Joo hyun Lee, Sangjin Bae, Min Jeong Cho, Minseok Suh, Jae Sung Lee
TL;DR
Expert PET/CT interpretation for head and neck cancer is challenging, and generalist multimodal models have limitations in medical imaging. This study fine-tunes a specialized model using a two-level curriculum and autoregressive training; it outperformed generalist baselines, with primary-tumor accuracy of 69.03 ± 0.81% externally.
Problem
Generalist multimodal models have limited domain knowledge and other barriers to clinical adoption, while PET/CT-specific LMMs remain scarce despite the modality's clinical use.
Source: Introduction · p. 3Method
The authors fine-tuned a specialized LMM for head and neck [18F]FDG PET/CT using a two-level curriculum and autoregressive training.
Source: Introduction · p. 3, Network Training and Loss Function · pp. 6–7Results
The specialized model significantly outperformed generalist models; primary-tumor accuracy was 69.03 ± 0.81% on external validation.
Source: Results · pp. 8–9Takeaways & Limitations
The findings indicate that domain-specific LMMs have potential for clinical implementation in head and neck cancer PET/CT workflows.
Source: Conclusion · p. 10The model relies on 2D slices rather than fully interpreting 3D volumetric data.
Source: Discussion · p. 10
Abstract
from arXiv · showhide
Background: Diagnosing head and neck cancer using PET/CT is clinically challenging and time-consuming due to the anatomical complexity of the region, motivating computer-aided diagnosis (CAD). Generalist Large Multimodal Models (LMMs) remain limited in medical contexts by insufficient domain-specific knowledge, privacy and security concerns, and verbosity, motivating specialized standalone LMMs. Purpose: We evaluated the feasibility of a specialized LMM for automated PET/CT interpretation in head and neck cancer using a large-scale multi-institutional PET/CT dataset, a tailored training curriculum, and autoregressive training. Methods: LLaVA-NeXT was fine-tuned using a two-level curriculum with image-conversation pairs curated by two radiologists from public data. The dataset included clinically important annotations such as primary tumor presence and metastatic lymph node location. Level 1 used 28,000 image-conversation pairs to learn basic information, including modality type and hypermetabolism. Level 2 used 12,975 pairs to learn primary tumor presence and the existence and anatomical location of cervical lymph node metastases. External validation included four institutions with diverse imaging devices. Results: The specialized LMM substantially outperformed ChatGPT and LLaVA-NeXT. In Level-2 external validation, ROUGE-L, ROUGE-S, Cosine Similarity, Precision, Recall, and F1 were 0.8751, 0.8794, 0.8324, 0.8794, 0.8711, and 0.8751, while generalist models consistently scored below 0.1. Primary tumor classification accuracy was 83.14 +/- 1.15% internally and 69.03 +/- 0.81% externally. For lymph node localization, the corresponding scores were 0.6389, 0.6257, 0.5287, 0.5782, 0.6371, and 0.6648. Conclusion: Specialized LMMs show promising results for fast, accurate PET/CT-based diagnostic support and medical education, highlighting their potential for clinical translation.
Introduction
Head and neck PET/CT interpretation is clinically important but challenging, while expert availability and generalist-model limitations constrain diagnostic support. This study develops a specialized LMM for interpreting these scans and detecting clinically relevant findings.
Introduction
- Expert interpretation of PET/CT is limited by a worldwide shortage of trained nuclear medicine physicians, which may delay care and increase interpretive errors.The authors identify assistive technologies as a way to augment expertise and broaden access to diagnostic services.Source: Introduction · p. 3
- Generalist LMMs face barriers including insufficient domain knowledge, privacy concerns, computational demands, and variable outputs, while PET/CT-specific LMMs remain limited.The paper attributes the scarcity of PET/CT models largely to limited large-scale datasets and PET/CT's multimodal characteristics.Source: Introduction · p. 3
- The study develops and evaluates a specialized LMM for [18F]FDG PET/CT interpretation in head and neck cancer, targeting primary tumors, metastatic lymph nodes, and hypermetabolic regions.It uses a multi-center, multi-instrument dataset and a two-level curriculum that progresses from foundational imaging concepts to clinically salient diagnostic inference.Source: Introduction · p. 3
Materials and Methods
The methods combine multi-institutional PET/CT data, a two-level diagnostic curriculum, and autoregressive fine-tuning of LLaVA-NeXT. Evaluation uses held-out internal and external cohorts and text-generation and clinically focused measures.
Dataset
- The dataset paired images with questions and expert answers, with two nuclear medicine experts annotating primary tumors and metastatic lesions.Training data came from HECKTOR2021 and AutoPET2024 and included 28,000 Level 1 pairs from 1,333 patients and 12,975 Level 2 pairs from 55 patients.Source: Dataset · p. 5
- Validation used unseen data from four independent institutions, with separate internal and external sets for both curriculum levels.The Level 2 external set comprised 1,000 pairs from 138 patients; the cohorts used diverse scanners, modalities, views, and scan coverage.Source: Dataset · pp. 5–6
Network Model
- LLaVA-NeXT, a 7-billion-parameter model with a CLIP vision encoder and Vicuna language model, was fine-tuned using LoRA.Training used AdamW with a learning rate of 1e-5; LoRA rank and alpha were 128, with dropout 0.1.Source: Network Model · p. 6
Network Training and Loss Function
- The two-level curriculum moves from foundational imaging questions to clinically focused assessment of primary tumors and cervical lymph-node metastases and their stations.Level 1 covers modality, hypermetabolic lesions, view, and contrast-agent use; Level 2 asks whether a primary tumor or suspicious cervical nodes are visible and requests node station when present.Source: Network Training and Loss Function · pp. 6–7
Evaluations and Statistical Analyses
- Fine-tuning used autoregressive generation rather than teacher forcing, conditioning on the model's own prior outputs to approximate inference conditions.Evaluation covered text-generation metrics and clinically focused primary-tumor and lymph-node predictions; performance metrics were averaged across three random-seed runs.Source: Network Training and Loss Function · p. 7, Evaluations and Statistical Analyses · p. 7
Results
The specialized LMM outperformed generalist baselines across both task levels. Primary-tumor accuracy was higher internally than externally, while lymph-node localization remained comparatively more challenging and station-level performance was stable despite class imbalance.
Results
- The proposed LMM significantly outperformed generalist baselines on both Level 1 and Level 2 across six metrics.The comparison includes ROUGE-L, ROUGE-S, Cosine Similarity, Precision, Recall, and F1.Source: Results · p. 8, Figure 3 · p. 8
- Lymph-node station identification scored lower on both validation sets than holistic performance metrics, though the authors judged it high enough to support potential clinical utility.Source: Table 5 · p. 8
- 83.14 ± 1.15% internal and 69.03 ± 0.81% external validation accuracy were achieved for primary-tumor detection.Source: Results · p. 9
- Stable, uniform F1 performance across lymph-node stations persisted despite highly imbalanced data distributions.The authors attribute this consistency to the structured output format focusing classification on clinically significant information rather than grammatical variation.Source: Results · p. 9, Figure 4 · p. 9
- Detailed ablations of hyperparameters, PubMed pretraining, scalability, and sentence structure are reported in the Supplementary Notes.Source: Results · p. 9
Discussion
The authors present quantitative evaluation of clinically salient outputs as evidence of feasibility, while acknowledging that direct physician assessment was not performed. They also outline model limitations and proposed directions for improving clinical usability and validation.
Discussion
- Quantitative evaluation of tumor and lymph-node outputs is offered as a proxy for clinical feasibility, but direct physician assessments were not performed.Source: Discussion · p. 9
- Separate CNN or Transformer experts may perform well on narrow tasks but scale poorly across clinical questions and struggle with free-text generation.Source: Discussion · p. 10
- Reported limitations include low clinical-information extraction, overfitting and security vulnerabilities, nonstandard abbreviations, 2D-only interpretation, limited interaction, and costly training.Source: Discussion · p. 10
- Proposed improvements include expert-model priors, personalized medical abbreviations, and standardized output formatting.Source: Discussion · p. 10
- Future work calls for more diverse data, additional cancers and tracers, prospective independent studies with nuclear-medicine specialists, and direct 3D volumetric operation.Source: Discussion · p. 10
Conclusion
The authors conclude that a specialized LMM for head-and-neck cancer diagnosis from PET/CT is feasible and significantly outperforms generalist models. They see potential for its use in diagnostic workflows.
Conclusion
- The specialized LMM significantly outperformed generalist models such as GPT-4o, supporting its potential for head-and-neck PET/CT diagnostic workflows.The model was developed using a large-scale dataset curated from seven centers with diverse imaging devices.Source: Conclusion · p. 10
Supplementary Materials
The supplementary materials describe datasets for diagnostic and educational tasks, ablation tests of training and answer choices, and validation results for the model. The ablations report effects from PubMed pretraining, curriculum progression, and answer structure.
Dataset Construction
- HECKTOR2021 and AutoPET2024 images were preprocessed into resized 2D slices, with annotations covering scan properties and hypermetabolism; the combined datasets were split into training and testing subsets.Source: Dataset Construction · p. 13
- Level 1 covers scan modality and basic image findings, while Level 2 uses radiologist annotations for primary-tumor presence and cervical-node presence and location.Source: Dataset Construction · p. 13
- The PubMed dataset pairs nuclear-medicine images with article-derived descriptions and abstracts, converted into instruction-following examples for explanation and education.Source: Dataset Construction · p. 13
Ablation Study
- Temperature and top-p settings were varied together; 0.8 was selected because 0.5 produced overly similar answers despite superior Level 1 performance.Source: Figure 5 · p. 14, Ablation Study · p. 14
- PubMed pretraining generally degraded Level 1 performance; the authors attributed this to task mismatch and the dataset’s small size and noise.Source: Figure 6 · p. 15, Ablation Study · p. 15
- After Level 2 training, Level 1 performance changed little in most cases, indicating versatility across the staged tasks.Source: Figure 7 · p. 16, Ablation Study · p. 16
- After mixed answer templates were introduced, performance for answers with station information severely degraded, prompting a unified structure using “station none” when no lymph node was present.Source: Figure 8 · p. 16, Ablation Study · pp. 16–17
Performance of Level 1 and Level 2
- The model significantly outperformed other baselines on Level 1 and Level 2 tasks across internal and external validations.Source: Table 6 · p. 17, Performance of Level 1 and Level 2 · p. 17, Table 8 · p. 18