Source-linked AI summary
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
Yang Xing, Jiong Wu, Savas Ozdemir, Yang Zhou, Boxiao Yu, Ying Zhang, Zheren Zhu, Chenyu You, Wei Shao, Yang Lu, Kang Wang, Tinsu Pan, Yang Yang, Kuang Gong
TL;DR
PSMA PET/CT interpretation is important in prostate cancer management, yet clinical interpretation and reporting remain manual and inconsistently standardized, and existing PET models do not combine language tasks with 3D lesion segmentation. This paper presents a unified PSMA PET/CT vision-language model for report generation, VQA, and lesion segmentation. It reports improved language-task performance and voxel-level lesion localization, supporting the feasibility of unified PSMA PET/CT analysis.
Problem
PSMA PET/CT interpretation and reporting remain manual and inconsistently standardized, while existing PET VLMs do not jointly support clinical language tasks and 3D lesion segmentation.
Source: §1 · pp. 2–3Method
The authors develop a unified PSMA PET/CT VLM for report generation, VQA, and semantic and language-guided lesion segmentation, using a PET/CT vision encoder and an nnUNet-style 3D segmentation module.
Source: §1 · p. 3, §4.2.2 · p. 15, §4.2.3 · p. 16Results
The proposed model outperformed baselines across report-generation metrics and VQA tasks, while its 3D module enabled voxel-level lesion localization.
Source: §2.3 · p. 7, §3 · pp. 9, 12Takeaways & Limitations
The findings support the feasibility of a unified framework for structured, interactive, and interpretable PSMA PET/CT analysis with voxel-level lesion localization.
Source: §3 · p. 12The report-generation and segmentation datasets were not paired case by case, limiting direct supervision and evaluation of sentence-level lesion grounding.
Source: §3 · p. 11
Abstract
from arXiv · showhide
Accurate PSMA PET/CT interpretation is central to prostate cancer management, yet existing PET/CT AI models typically address isolated tasks. We propose a unified PSMA PET/CT vision-language model for report generation, visual question answering, and lesion segmentation. The framework adopts an LLaVA-style architecture, comprising a PET/CT vision encoder, an MLP-Mixer projection module, a LoRA-tuned large language model, and a 3D segmentation branch. Training followed a four-stage strategy: vision encoder pretraining, projection-layer alignment, VLM fine-tuning, and final multitask tuning. Language tasks used 5,747 PSMA PET/CT datasets with paired reports, while segmentation used the PSMA subset of AutoPET. The model outperformed PET2REP and a CT-based baseline across standard report-generation metrics, improved performance across VQA question types, and achieved higher Dice and lesion-level overlap F1 than SegAnyPET and nnUNet. These results support the feasibility of a unified framework for structured, interactive, interpretable PSMA PET/CT analysis with voxel-level grounding within a single multitask model architecture.
1 Introduction
PSMA PET/CT is central to prostate cancer assessment, but interpretation and reporting remain manual and inconsistently standardized. The paper addresses the limited scope of existing PET vision-language models with a unified system for clinical language tasks and 3D lesion segmentation.
1 Introduction
- PSMA PET interpretation requires lesion localization, uptake assessment, and whole-body disease-burden estimation across staging, recurrence monitoring, and treatment assessment.Consensus guidelines increasingly emphasize tumor burden, lesion distribution, and response patterns in treatment eligibility and patient outcomes.Source: §1 · p. 2
- Only 11% of clinical PSMA PET reports adhered to a standardized format in a cited study, while interpretation remains largely manual and reader-dependent.The authors note that such variability may contribute to misinterpretation, inconsistent follow-up or work-up recommendations, and management delays.Source: §1 · p. 2
- Existing PET vision-language models support image-text alignment and instruction-following but do not jointly support PSMA PET/CT language tasks and 3D lesion segmentation.The paper places this gap against broader medical VLM use in classification, retrieval, report generation, VQA, and interactive annotation.Source: §1 · p. 3
- The proposed unified architecture supports report generation, VQA, semantic lesion segmentation, and language-guided lesion segmentation for whole-body PSMA PET/CT.It uses a LISA-like language-guided interface, an nnUNet-style 3D module, and TotalSegmentator-derived organ masks as auxiliary anatomical supervision; training uses image-report pairs, report-derived QA pairs, and lesion masks.Source: §1 · pp. 3–4
2 Results
The results compare the proposed model with PET/CT retrieval, report-generation, VQA, and lesion-segmentation baselines. Across these tasks, the reported comparisons favor the proposed model, with segmentation evaluated under both semantic and language-guided prompting.
2 Results
- The language-task evaluation used 5,747 whole-body PSMA PET/CT image-report studies; segmentation used 597 image-mask studies from the PSMA subset of AutoPET.The language-task studies used 18F-DCFPyL and were acquired between 2022 and 2025; the AutoPET studies came from 378 male patients at LMU University Hospital Munich, acquired between 2014 and 2022.Source: §2 · p. 4
2.1 PET/CT image-text retrieval
- Recall@100 rose from 14.17% to 35.57% for image-to-text retrieval and from 8.52% to 38.87% for text-to-image retrieval, compared with CT-CLIP.PET/CT-CLIP achieved the highest Recall@K values across all evaluated retrieval settings.Source: §2.1 · p. 4, Table 1 · p. 5
2.2 Report generation
- The proposed model improved all five report-generation metrics over PET2REP, including ROUGE from 16.73 to 23.42 and BLEU from 27.40 to 36.17.The metrics were ROUGE, BLEU, METEOR, CIDEr, and BERTScore; the proposed model achieved the best performance across all evaluated metrics.Source: §2.2 · p. 5, Table 2 · p. 6
2.3 VQA
- Multiple-choice VQA accuracy increased from 70.89% to 83.60%, and the proposed model outperformed MedVL-SAM2 across all three VQA task types.For free-response VQA, ROUGE increased from 18.91 to 33.20 and CIDEr from 17.48 to 91.55.Source: Table 3 · p. 7, §2.3 · p. 7
2.4 Lesion Segmentation
- Dice was 0.5858 for semantic and 0.5875 for language-guided segmentation, exceeding SegAnyPET (0.4653) and nnUNet (0.5498).The proposed model also achieved lesion-level F1 scores of 0.7354 and 0.7356 under “Any overlap” for semantic and language-guided segmentation, respectively.Source: Table 4 · p. 9, §2.4 · p. 9
3 Discussion
PET/CT-specific vision-language training improved report-generation and VQA performance, while semantic and language-conditioned segmentation enabled voxel-level lesion localization. The findings support the feasibility of a unified framework for PSMA PET/CT analysis.
3 Discussion
- Report drafts summarize major findings, VQA supports interactive queries, and 3D segmentation supports semantic or interactive localization and quantitative assessment.These are described as potential clinical uses of the framework.Source: §3 · pp. 9–10
- The framework extends PET-oriented VLMs beyond alignment, report generation, or instruction following by adding explicit 3D lesion segmentation.PSMA interpretation draws on tracer-specific biodistribution, disease-related uptake, and anatomical correlates on CT.Source: §3 · p. 10
- Language specifies segmentation targets through [SEG], an nnUNet-style decoder produces dense volumetric predictions, and auxiliary organ masks add anatomical context.The organ context may help distinguish lesions from physiologic uptake in organs with characteristic PSMA expression.Source: §3 · p. 11
- The tasks assess complementary outputs: whole-body findings in text, targeted answers about an examination, and voxel-level localization of PSMA-avid lesions.The shared instruction-driven interface preserves the distinct output required by each task.Source: §3 · p. 11
- The report-generation and lesion-segmentation datasets were not paired by case, limiting direct supervision and evaluation of sentence-level lesion grounding.The VQA dataset may inherit report omissions or interpretation bias, and AutoPET differs from the in-house cohort in institution, acquisition protocol, and tracer distribution.Source: §3 · p. 11
- PET/CT-specific vision-language training improved report-generation and VQA performance, while semantic and language-conditioned segmentation enabled voxel-level lesion localization, supporting unified analysis feasibility.The framework combines clinical language tasks and 3D lesion segmentation in one instruction-driven model.Source: §3 · pp. 9, 12
4.1 Dataset
The dataset section describes separate in-house report-linked PSMA PET/CT studies for language tasks and AutoPET lesion masks for segmentation, alongside report-derived VQA and organ pseudo-labels.
4.1.1 PSMA PET/CT datasets with radiology reports
- 5,747 whole-body 18F-DCFPyL studies acquired from 2022 to 2025 supplied paired reports, standardized to fixed spacing and matrix size with structured JSON findings and impressions.PET intensities were converted to standardized uptake values, and examinations were cropped to the upper-thigh field of view.Source: §4.1.1 · p. 12
4.1.2 VQA data construction
- Report-derived VQA covered disease presence and distribution, recurrence, nodal disease, and metastases, with nine questions per case and evidence-checking quality control.Case-level splits yielded 4,023 training, 574 validation, and 1,150 testing cases.Source: §4.1.2 · pp. 12–13
4.1.3 AutoPET PSMA lesion segmentation dataset
- AutoPET PSMA supplied 597 studies from 378 men with manually annotated voxel-level lesion masks and patient-level splits of 419, 59, and 119 studies.The dataset supported semantic tumor segmentation training and quantitative lesion-level evaluation.Source: §4.1.3 · p. 13
4.1.4 TotalSegmentator-derived organ masks
- Ten CT-derived organ groups served as auxiliary pseudo-labels to help distinguish physiologic uptake patterns from suspicious lesions.Selected structures included the kidneys, bladder, liver, spleen, brain, lungs, heart, prostate, stomach, and salivary glands.Source: §4.1.4 · p. 14
4.2 Network architecture
The architecture unifies text-generation and mask-prediction tasks from PET/CT images and text prompts. It combines a language-model pathway with a language-conditioned 3D segmentation branch.
4.2.1 Overview
- A unified 3D medical VLM takes PET/CT volumes and text tokens and returns text tokens and a segmentation mask through a LLaVA-style architecture.Its components include a 3D PET/CT encoder, multimodal projection layer, LLM, and language-conditioned nnUNet-style segmentation module.Source: §4.2.1 · pp. 14–15
4.2.2 Vision-language alignment
- The adaptive MLP-Mixer compresses 3D visual tokens and aligns them with the LLM embedding space before they are combined with text instructions.LoRA layers are applied to the language backbone for parameter-efficient fine-tuning while preserving the pretrained model’s general language capability.Source: §4.2.2 · pp. 15–16
4.2.3 Semantic and language-guided segmentation module
- For segmentation, the [SEG] token representation is projected and fused with multiresolution PET/CT features through cross-attention before mask decoding.This conditions voxel-level prediction on the text instruction while using an nnUNet-style 3D encoder-decoder.Source: §4.2.3 · pp. 16–17
4.3 Implementation details
Training progresses from PET/CT visual encoding and language alignment to task-specific generation and joint multitask tuning.
4.3 Implementation details
- Training followed four stages: vision-encoder pretraining, projection-layer alignment, VLM fine-tuning for report generation and VQA, and final multitask tuning.The final stage jointly optimized the VLM branch, segmentation-token interface, cross-attention module, and segmentation branch.Source: §4.3 · p. 17
- Contrastive learning aligned 3D image features with report-level text, then the projection layer was trained on report-generation and VQA samples while the encoder and language model were frozen.The next stage optimized the vision encoder, projection layer, and language-model LoRA parameters, without updating the segmentation branch.Source: §4.3 · p. 17
- Final multitask tuning used mixed instructions for report generation, VQA, semantic segmentation, and language-guided segmentation.Text samples used autoregressive language-modeling loss; segmentation samples used cross-entropy and Dice losses, with task-specific prompts at testing.Source: §4.3 · p. 17
5 Statistical analysis
Statistical analysis quantified uncertainty and robustness using bootstrap confidence intervals and task-specific evaluation metrics.
5 Statistical analysis
- Bootstrap resampling estimated 95% confidence intervals for all evaluation metrics.Source: §5 · p. 17
- Image-text retrieval was evaluated with Recall@K for both image-to-text and text-to-image retrieval.Source: §5 · p. 17
- Report generation used ROUGE, BLEU, METEOR, CIDEr, and BERTScore; VQA used language-generation metrics or multiple-choice accuracy, and segmentation used case-level Dice and lesion-level F1 criteria.Lesion-level F1 criteria were SUVmax-match, any-overlap, and Dice (>0.5).Source: §5 · pp. 17–18
7 Code availability
The implementation details identify the optimizer, comparison baseline, software versions, and computing hardware.
7 Code availability
- Model training used the AdamW optimizer as implemented in PyTorch.Source: §7 · p. 18
- CTCLIP was used via the HuggingFace hub for baseline comparisons.Source: §7 · p. 18
- Experiments used PyTorch v2.7 and Transformers v4.42 on a cluster with eight NVIDIA B200 GPUs, each with 180 GB.Source: §7 · p. 18