Volumetric Radiology AI in the Era of Multimodal Large Language Models

1 Introduction

• Clinical evidence in radiology is distributed across 3D space, acquisition settings, and longitudinal examinations, so reliable interpretation must preserve task-relevant spatial and contextual information.Findings may be small or sparse, depend on continuity across slices or anatomical relationships, and require quantitative measurements, prior examinations, and clinical context.Source: §1 · p. 3
• Selected 2D images, compressed representations, or report-derived text may suffice when they capture decisive evidence, but tasks requiring volume-wide information warrant native 3D modeling.Needs include lesion extent, anatomical relationships, quantitative burden, phase- or sequence-specific characteristics, and longitudinal change.Source: §1 · p. 3
• Volumetric foundation models and MLLMs provide complementary strengths, while agentic systems extend their integration through iterative evidence acquisition, specialized tools, context management, and workflow interaction.Agents can support targeted inspection, tool-mediated analysis, evidence verification, and human-supervised workflow coordination rather than serving primarily as autonomous replacements for radiologists.Source: §1 · pp. 3–4
• Clinical credibility requires task-aligned evaluation that assesses spatial grounding, appropriate tool use, traceable evidence and provenance, and utility in realistic workflows with human oversight.Fluent reports, isolated benchmark improvements, or successful tool invocation alone do not establish faithful image interpretation or clinical usefulness; clinicians must be able to examine, challenge, and override AI-supported conclusions.Source: §1 · p. 4
• The review synthesizes more than 200 publications through July 2026, connecting model-level volumetric representation and multimodal interpretation with agentic orchestration, clinical applications, and evaluation.It examines when 2D or report-mediated reasoning is sufficient, when native volumetric modeling is necessary, and how systems retrieve, verify, and act on spatial evidence.Source: §1 · pp. 5–6, Figure 2 · p. 5
• The Claim–Design–Validation framework appraises whether a study’s stated technical, workflow, or clinical claim matches its design and validation setting.It considers task requirements such as native volumetric modeling, multimodal grounding, tool-mediated analysis, memory, planning, and human handoff.Source: §1 · p. 6, Figure 3 · p. 6

2.1 Traditional Volumetric Radiology Analysis Methods

• Radiomics shifted volumetric analysis toward quantitative biomarkers by extracting intensity, shape, texture, and spatial-heterogeneity descriptors for diagnosis, prognosis, or treatment response.Typical pipelines combined acquisition and reconstruction, registration or normalization, region-of-interest delineation, handcrafted features, and a downstream statistical or machine-learning model.Source: §2.1 · p. 8
• Traditional feature-engineering pipelines were sensitive to segmentation and acquisition choices, transferred poorly across settings, and produced fixed outputs rather than interactive, context-aware support.Supervised 3D networks also remained label-dependent and task-specific, motivating reusable representation learning and multimodal models.Source: §2.1 · p. 8
• Deep learning replaced handcrafted descriptors with learned features, including multi-scale 3D CNNs, volumetric encoder-decoders, self-configuring segmentation, and transformer-based models [12] [13] [15].This shift matters because isolated 2D slices may miss information in volumetric context.Source: §2.1 · p. 8

2.2 Large Language Models

• For radiology, LLMs support language understanding, report generation, question answering, differential-diagnosis explanations, retrieval, and reasoning over clinical context rather than replacing image interpretation.That context can include anatomy, prior reports, clinical history, guideline thresholds, uncertainty, and downstream decisions.Source: §2.2 · p. 9
• Because LLMs accept text while volumetric evidence is dense, spatial, and quantitative, reducing a CT to a few sentences or slices can lose subtle, longitudinal, or lesion-level findings.The LLM is therefore a reasoning and communication layer built on reliable spatial grounding, not a replacement for volumetric perception.Source: §2.2 · p. 9

2.3 AI Agents in Healthcare and Radiology

• Medical agents maintain task state, plan steps, use tools, update memory, and answer or defer; general systems explore adaptive collaboration, tool selection, and evidence-based diagnostic workflows [32] [33] [34].These capabilities address clinical reasoning distributed across images, reports, laboratory values, prior examinations, guidelines, measurements, and specialist tools.Source: §2.3 · p. 9
• Radiology agents must ground language in image regions and tool provenance, access full-volume evidence when continuity matters, and coordinate traceable, human-supervised workflows [214] [35] [36] [37] [38] [39].Examples span MedRAX for chest X-rays, CT-Agent and 3DMedAgent for volumetric analysis, RadAgent and CT-Flow for stepwise interpretation, and systems emphasizing human-in-the-loop assistance. Representation learning supplies perceptual priors, vision–language alignment makes semantics addressable, and MLLMs support interactive interpretation.Source: §2.3 · p. 9

3 Foundation Models in Volumetric Radiology

• Foundation models are broadly pretrained on volumetric, visual, textual, or multimodal medical data and adapt across tasks through reusable perceptual, semantic, or instruction-following capabilities.Volumetric data pose challenges from complex anatomy, 3D structure, and scarce high-quality expert annotations.Source: §3 · p. 10
• The functional progression runs from SSL anatomical and spatial priors, through clinical-language alignment, to MLLM context for generation, dialogue, and instruction following.Before supporting generation or multi-step understanding, models must acquire reliable volumetric representations, connect them to clinical semantics, and expose visual information for language-model reasoning and communication.Source: §3 · p. 10
• Across stages, 2D, native 3D, and hybrid encoders recur; compression becomes most explicit in MLLMs, which tokenize and expose volumes to autoregressive decoders.Figure 5 summarizes the three-stage foundation-model layer and its shared constraint of preserving clinically decisive 3D information for grounded reporting.Source: §3 · p. 10

3.1 Volumetric Self-Supervised Representation Learning

• Unlabeled 3D scans provide scalable pretext supervision for anatomical, spatial, and modality-specific priors before language alignment or task specialization.The subsection focuses on masked modeling, context restoration, deformation prediction, and contrastive learning as sources of reusable 3D visual representations.Source: §3.1 · p. 10

3.1.1 Self-Supervised Objectives

• Masked reconstruction encourages long-range spatial modeling, whereas contrastive objectives learn semantic features but require careful positive and negative sampling for shared tissues and sparse abnormalities.These objectives establish visual perception before linguistic supervision.Source: §3.1.1 · p. 11

3.1.2 Representative Approaches

• Representative methods restore anatomy, model hierarchy and deformation, encode geometric invariance, bridge 2D and 3D data, and scale across modalities and corpora.Models Genesis and DAE [47] use restoration; HU-based masking [49], HybridMIM, MiM [51], and MDM address meaningful tissue, structural scales, or deformation, while VoCo [16], geometric visual similarity [53], sequence-invariant MRI [54], and GzPT add contextual, topological, protocol, or human-prior constraints. UniMiSS [56], CDSSL-P3D [57], PCRLv2, M3AE [59], BrainMVP [60], MedCoSS [61], BrainSTORM, FM-HCT [62], SPECTRE [63], 3DINO [64], and CoralBay [65] extend cross-dimensional, multimodal, or foundation-scale learning.Source: §3.1.2 · pp. 11–12

3.2.1 Alignment Objectives

• CLIP-style contrastive pre-training aligns each image or volume with its report and separates mismatched pairs, creating clinically meaningful representations and bidirectional image–text retrieval.The objective averages image-to-text and text-to-image losses; shared image–text representations also support textual prompts for zero-shot recognition and downstream adaptation.Source: §3.2.1 · pp. 12–13

3.2.2 Representative Approaches

• Early progress focused predominantly on 2D imaging, where BioMedCLIP [4], PMC-CLIP, and PubMedCLIP achieved robust alignment using millions of image-caption pairs; representative 3D methods aligned global volume-report embeddings with 25k–400k pairs.MPS-CT augments pre-training with LLM-extracted “silver-standard” labels, while MR-CLIP [76] aligns brain MRIs with structured DICOM metadata without textual reports.Source: §3.2.2 · pp. 13–14
• Because global pooling can dilute sparse findings such as small lesions, fine-grained methods enforce anatomical, slice-, patch-, chunk-, or lesion-level correspondences.Examples include fVLM [81] for anatomy-level alignment, RadFinder [82] for weak axial-depth anchors, and GLeVE [88] for correspondence between report descriptions and 3D lesion proposals.Source: §3.2.2 · p. 14
• Knowledge-enhanced and hierarchical methods address flat, noisy clinical text by modeling report levels and concepts, including negation, uncertainty, synonyms, and modality gaps.Approaches include multi-level contrastive losses in UniBrain and VELVET-Med [92], and knowledge-informed similarity targets in SCALE-VLP [96].Source: §3.2.2 · pp. 14–15
• These text-aligned feature spaces provide the semantic substrate for later MLLMs to support language-mediated interpretation.Source: §3.2.2 · p. 15

3.3 Multimodal Large Language Models

• The MLLM layer transforms volumetric representations into visual tokens or cross-attended states for report generation, multi-turn question answering, localization, uncertainty communication, and instruction following.Source: §3.3 · p. 15

3.3.1 Vision Encoders

• 2D encoders provide transferable biomedical semantic priors, but selected views or slice sequences can weaken through-plane continuity, lesion extent, and volumetric morphology; native 3D and hybrid designs address this trade-off.Source: §3.3.1 · pp. 18–20
• For tasks requiring spatial continuity or subtle morphology, native volumetric priors raise evidence-preservation potential, while localized representations and task-adaptive compression help protect findings before language integration.Source: §3.3.1 · pp. 18–20

3.3.2 Vision-Language Interfaces: Projection, Resampling, and Fusion

• Interfaces must compress volumetric tokens while preserving sparse findings and spatial provenance, using query pooling, structured fusion, region-aware conditioning, or question-conditioned routing rather than mere dimensional projection.Source: §3.3.2 · pp. 20–22

3.3.3 Generative Backbones: Text Decoders and Multimodal Foundation Stacks

• Report decoders, text-oriented instruction models, and inherited multimodal stacks offer different generation and context capabilities, but visual grounding depends on the encoder–interface pipeline and output fidelity should be assessed end to end.Source: §3.3.3 · pp. 22–23

3.3.4 Training Strategies

• Training follows a curriculum: initialize volumetric priors, learn to expose visual tokens, then instruction-tune; study-level reports provide weak supervision because they rarely label every slice, organ, lesion, or negative finding.Source: §3.3.4 · p. 24
• 3D instruction construction determines whether supervision teaches only global impressions or also laterality, location, lesion extent, modality-specific cues, and uncertainty.CT supervision spans diagnosis, segmentation, report generation, and multiple-choice reasoning [110]; CT2Rep, BrainGPT [118], MEPNet, Brain3D [129], and PETRG-3D [128] emphasize translating volumes into structured radiological language.Source: §3.3.4 · p. 25
• Grounding-aware training combines report generation with classification, localization, segmentation, or region description so generated claims remain tied to spatial evidence.Examples include MedRegion-CT’s pseudo-mask-derived region tokens [125], PETAR’s PET, CT, and lesion-mask supervision [127], and RadSight’s staged evidence hierarchy [112].Source: §3.3.4 · p. 25
• Token-budget-aware training teaches models which visual tokens to retain for each instruction, using pruning, scheduling, compact tokenization, or question- and modality-dependent routing.MedPruner learns hierarchical slice and patch pruning [228], while Photon learns instruction-conditioned token scheduling [108].Source: §3.3.4 · p. 25
• With limited 3D instruction data, many systems freeze major components and use lightweight adaptation; emerging reward-driven post-training targets diagnostic correctness and spatial grounding.Med3D-R1 applies RL-style optimization to volumetric abnormality diagnosis [29], while E-MRL rewards diagnosis-localization-verification trajectories with cross-view consistency [132].Source: §3.3.4 · p. 25
• Volumetric training must preserve 3D anatomical and semantic priors, compress them into a visual–language interface, and support appropriate spatial detail before optimizing correctness and grounding.What the model learns is constrained by which slices, regions, modalities, and masks the pipeline teaches it to preserve.Source: §3.3.4 · p. 26
• The model landscape is moving toward anatomy-aware alignment, native or hybrid 3D tokenization, richer interfaces, and grounding-aware post-training, with ClinFusion [113] illustrating a bridge to agentic workflows.That bridge depends on representing, compressing, grounding, and discussing volumetric information before orchestrating it into multi-step clinical workflows.Source: §3.3.4 · p. 26

4 Agentic Systems in Volumetric Radiology

• Agentic systems combine planning, grounded tool use, memory, and workflow coordination to select evidence, perform spatial or quantitative operations, integrate context, and support human review.These capabilities form an iterative analysis loop, complementing volumetric MLLMs by dynamically accessing and coordinating evidence across stages.Source: §4 · p. 26, §4.4 · p. 31

4.1 Reasoning and Planning

• Planning agents organize targeted inspection and tool use, but current evaluations do not establish that final statements are causally supported by intermediate steps.Designs range from anatomy-aware decomposition, as in CT-Agent [35] and 3DMedAgent [36], to policy-constrained phase acquisition in PD-CTAgent [136] and planner–executor workflows. Evaluations include VQA accuracy, report quality, expert review, task completion, and tool trajectories; trace fidelity and clinical reasoning validity remain incompletely tested.Source: §4.1 · pp. 27–28

4.2 Tool-Augmented Perception and Grounded Action

• Tool-augmented perception translates clinical intent into spatial or quantitative outputs, but tool validity must be assessed at both module and workflow levels.Tools include segmentation, registration, measurement, viewer manipulation, retrieval, and dose simulation; checkable outputs such as masks and measurements are more directly verifiable than viewer actions or retrieved context. Errors in tool choice, parameters, series selection, or segmentation can propagate into clinical claims.Source: §4.2 · p. 29

4.3 Memory and Dynamic Context Management

• Memory supports within-study aggregation, longitudinal change assessment, and retrieval of external cases or guidance, but its value depends on visible provenance and comparability.Working-memory examples include CT-Agent [35] and 3DMedAgent [36]; longitudinal systems include Neuro-Oracle [148] and BT-RADS Agent [151]. Stored or retrieved information may be inaccurate, outdated, or non-comparable, and memory effects are rarely isolated from perception, retrieval, or decision components.Source: §4.3 · pp. 29–30

4.4 Workflow Interaction and Multi-Agent Collaboration

• Workflow agents coordinate tools, artifacts, and people as bounded, supervised assistance systems, while human experts retain clinical authority.Validation should match workflow purpose, including report accuracy and review efficiency, dose agreement and segmentation reliability, planning constraint satisfaction, or research reproducibility. Few studies assess correct tool use, failure detection, intermediate-observation preservation, or clinician workload under realistic constraints.Source: §4.4 · pp. 30–31

5 Clinical Applications and Evaluation in Volumetric Radiology

• The Claim–Design–Validation framework links claim strength to validation: internal benchmarks establish feasibility, external studies test robustness, workflow studies assess usability, and prospective evaluation supports deployment claims.The four application families differ in clinical question and therefore in the validation needed to support their claims.Source: §5 · p. 31

5.1 Diagnostic Interpretation and Reporting

• Diagnostic evaluation must jointly establish clinical correctness and traceability to volumetric evidence, because text similarity or spatial accuracy alone may not ensure supported, accurately communicated findings.Calibration, reader or workflow studies, and external validation across institutions and acquisition settings remain limited; expert review is necessary when automated metrics cannot establish correctness and grounding.Source: §5.1 · pp. 32–34

5.2 Prognosis and Treatment Response

• Prognostic validation must define endpoints, cohorts, treatment exposure, and prediction horizons; discrimination alone does not establish calibrated risk or reliable change assessment.Temporal validation remains incomplete, including calibration at the intended horizon, lesion correspondence across serial studies, and external validation; retrieved historical evidence must be temporally aligned and traceable.Source: §5.2 · p. 34

5.3 Image-Derived Planning and Clinical Decision Support

• Planning systems require end-to-end technical and workflow validation, since accurate components do not establish constraint satisfaction, integration, or reliable operation across the pipeline.Studies rarely assess error propagation, detection of upstream failures, or required expert correction; external and prospective workflow studies are needed to support clinical-utility or deployment-readiness claims.Source: §5.3 · p. 37

5.4 Cross-Task Evaluation and Validation

• Cross-task validation must assess case-level coherence among outputs and the validity of agents’ intermediate actions, because final-answer accuracy cannot validate unsupported workflow steps.Current benchmarks often report task, grounding, trace-fidelity, workflow, and safety measures separately, while integrated validation, contamination control, subgroup analysis, and external evaluation remain limited.Source: §5.4 · pp. 37–38

6.1 From 2D Surrogates to Volumetric and Acquisition-Aware Intelligence

• Native volumetric intelligence should preserve spatial continuity while compressing data by clinical salience and representing acquisition physics and protocol context.Compression should retain small findings, spatial relations, and task-relevant regions; representations should account for factors such as CT phase, MRI sequence, slice thickness, and scanner protocol.Source: §6.1 · pp. 39–40

6.2 From Static Perception to Workflow-centric Clinical Intelligence

• Future clinical intelligence should ground claims in image regions, orchestrate multi-step workflows, and keep decisions under bounded human supervision.Agents should expand beyond isolated tasks to coordinate care across diagnosis, treatment planning, and longitudinal management, with inspectable results and clinician handoffs.Source: §6.2 · pp. 40–41

6.3 From Digital Boundaries to Self-Evolving and Medical Embodied Intelligence

• The research agenda extends agents from static prediction to clinical world models, self-evolving planning, and embodied physical intervention.It proposes perception-dynamics-planning loops, verifiable workflow steps with feedback, and later reinforcement learning and continuous strategy improvement; embodiment links real-time imaging to robotic procedures.Source: §6.3 · pp. 41–42

6.4 From Benchmarks to Traceable and Clinical-Grade Intelligence

• Clinical-grade translation requires spatially grounded data, workflow-specific evaluation, traceable evidence, staged validation, and implementation science.Validation should progress from retrospective tests through reader studies and simulations to prospective trials and monitoring, linking outputs to supporting studies, series, or slices. Hybrid effectiveness–implementation studies should assess adoption, fidelity, scalability, and sustainability alongside clinical outcomes.Source: §6.4 · pp. 42–43

6.5 Scope and Limitations

• The review covers volumetric radiology through July 2026, but excludes broader 2D imaging, non-imaging medical LLMs, and legal, economic, and reimbursement issues.Because agentic work is rapidly evolving and several systems are preprints or early reports, they indicate emerging design patterns rather than deployment readiness.Source: §6.5 · p. 43

• Selected 2D views suffice when decisive evidence is limited to a few images, whereas native volumetric modeling is essential when interpretation depends on full-volume spatial relationships.Source: §7 · p. 43
• The field advances through clinically aligned volumetric representations and agentic workflows that acquire, verify, and coordinate evidence using tools, memory, and iterative orchestration.Source: §7 · p. 43
• Clinical credibility requires aligning intended claims, preserved evidence, system actions, and validation; the Claim–Design–Validation framework identifies traceability and clearly defined human oversight as cross-cutting requirements.Source: §7 · p. 43
• Future progress depends on stronger native volumetric data resources, more reliable grounding and validation, and tighter integration with clinical tools and longitudinal context.As these components mature, traceable, human-supervised systems may support radiologists across diagnostic, planning, and longitudinal workflows.Source: §7 · p. 43