Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?

1. Introduction

• Unified systems have been proposed in medical AI, while recent work argues that vision models may move from task-specific architectures toward general-purpose foundation models.The introduction cites MedGemma [27] and Wiedemer et al. [36] in this context.Source: §1 · p. 1
• Medical vision models remain underexplored, and no existing framework unifies tasks such as segmentation, registration, and motion prediction.The introduction contrasts this fragmented setting with the potential for cross-task generalization without retraining.Source: §1 · p. 1
• The study uses LVM, a purely visual autoregressive Transformer trained on large-scale image and video data, as its medical video-modeling backbone.Its visual tasks are adapted through prompting rather than task-specific retraining.Source: §1 · pp. 1–2
• The Figure 1 examples span low-level restoration, including denoising and super-resolution, and higher-level segmentation and motion tasks performed directly from CT sequences.The caption presents these examples as zero-shot tasks without task-specific training.Source: Figure 1 · p. 2
• Despite no medical-data exposure, LVM achieves competitive performance across five medical imaging tasks and accurately forecasts future 3D CT phases from prior phases in radiotherapy motion prediction.The introduction says this result indicates patient-specific motion patterns and temporal dynamics can be modeled.Source: §1 · p. 2

1. Unified video models can address diverse medical imaging tasks.

• A single large video model can perform segmentation, detection, denoising, super-resolution, and motion prediction without task-specific retraining.The paper frames this as evidence for the feasibility of a unified medical imaging framework through video modeling.Source: §1 · p. 2
• The model’s performance does not yet reach state-of-the-art levels across all tasks.The authors nevertheless describe the results as supporting a unified framework that could reduce reliance on specialized architectures and handcrafted task designs.Source: §1 · p. 2
• The proposed unified framework could reduce reliance on highly specialized architectures and handcrafted task designs.This is presented as a feasibility claim, not as evidence that the model already matches specialized methods across tasks.Source: §1 · p. 2

2. Video models are particularly effective for sequential prediction tasks.

• In zero-shot motion prediction, LVM outperforms task-specific DVF-based and generative baselines.The passage identifies Figure 2 and Table 1 as presenting this comparison.Source: §2 · p. 2
• The authors suggest autoregressive video models inherently capture temporal dependencies and dynamic patterns relevant to sequential medical data.They identify 4D CT, functional MRI, and dynamic ultrasound as examples where temporal coherence and motion reasoning are critical.Source: §2 · p. 2
• The passage presents temporal coherence and motion reasoning as important for medical applications involving sequential data, including 4D CT, functional MRI, and dynamic ultrasound.Source: §2 · p. 2

3. Emergent general visual reasoning.

• The model handles out-of-distribution medical data and novel tasks without task-specific retraining, which the authors describe as signs of general visual reasoning.Source: §3 · p. 3
• The authors say further investigation is needed to clarify how this generalization arises and whether medical sequential-data pretraining could improve reasoning and clinical applicability.Source: §3 · p. 3
• The passage reports generalization despite the model never having been exposed to medical images during pretraining.Source: §3 · p. 3

Video Foundation Models

• Autoregressive and diffusion video models trained on large datasets have shown zero- and few-shot abilities across visual tasks, including segmentation, tracking, and physical reasoning [1, 11, 37].The cited work describes next-frame prediction, masked-frame reconstruction, and video continuation as common training objectives [36].Source: §2 · p. 3

Medical Foundation Models

• Medical foundation models support multimodal tasks, but most remain focused on static imaging rather than temporal data such as 4D CT, cine MRI, and ultrasound video [17] [20] [27] [38].The section frames zero-shot generalization to these modalities as a step toward general-purpose medical AI.Source: §2 · p. 3

Medical Motion Modeling

• Motion methods range from PCA-based DVF models to VAEs, diffusion, ConvLSTM, and hybrid simulators, with limitations including linear-motion assumptions, registration dependence, or handcrafted supervision.The passage also notes that generative methods depend on precomputed DVFs [31] and [25], while sequential models remain domain-specific [28] and [18].Source: §2 · p. 3

Zero-shot Video Models for Medical Motion

• Zero-shot video models are proposed as a domain-agnostic way to process medical 4D CT sequences and model respiratory motion, organ deformation, and temporal coherence without retraining.The passage presents this as potential for a shared pipeline spanning segmentation, tracking, and motion prediction.Source: §2 · p. 3

3. Motion Modeling and Reasoning

• Respiration moves tumors and nearby organs, motivating patient-specific prediction of future 3D CT phases from earlier phases in a 4D CT sequence.The static radiation field must cover the motion track to avoid geographic miss, increasing irradiation of surrounding healthy tissue.Source: §3 · p. 3, Figure 2 · p. 4
• Autoregressive modeling treats the temporal trajectory as joint probability, conditioning each predicted CT phase on previously observed or generated phases.The formulation is intended to capture short- and long-range dependencies, promote temporal continuity, and avoid intermediate DVFs.Source: §3 · p. 4

3.1. Pipeline

• Organ masks from nnUNet provide geometric priors for motion modeling and labels for evaluating predicted organ shape, location, and boundary continuity.The pipeline extracts lungs, heart, and liver masks for these purposes.Source: §3.1 · pp. 4–5
• VQ tokenization represents each CT slice as discrete visual tokens, which a causal decoder-only transformer predicts phase by phase from preceding tokens.At inference, the model receives an initial phase prefix and autoregressively generates subsequent phases, allowing multi-phase prediction.Source: §3.1 · p. 5

3.2. Implementation Details

• Implementation uses nnUNet-extracted organ masks, a VQGAN tokenizer fine-tuned on 4D CT data, and an approximately 120-million-parameter, 12-layer Transformer.The tokenizer maps each 256×256 slice to a 16×16 grid of 256 tokens; the model context supports up to 16 CT phases.Source: §3.2 · p. 5

4.2.2. Next-Phase Motion Prediction.

• Across public and private data, LVM outperformed all baselines; public lung prediction reached 90.75% IoU and 95.15% DSC, exceeding RMSim by 5.6% IoU and 5.1% DSC.On private data, lung gains were 6.3% IoU and 5.9% DSC over the best baseline; liver gains were 8.7% (IoU) and 7.0% (DSC) over the next-best model.Source: Table 1 · p. 6, §4.2.2 · p. 7

4.1. Experiment Setting

• The experiments use public and private 4D CT cohorts, assess motion spatially with IoU, DSC, SD, and HD, and also test segmentation, denoising, and super-resolution.The evaluation focuses on organ-position and shape accuracy; nnUNet-generated masks are used for each generated CT phase.Source: §4.1 · pp. 6–7

4.2.1. Segmentation, Denoising, and Super-Resolution

• Zero-shot LVM scores 91.52% Dice, 34.4 dB denoising PSNR, and 29.6 dB super-resolution PSNR, with SSIM around 0.8 in both enhancement tasks.Despite no medical-data training, it demonstrates spatial-prior and texture consistency, but remains below the best supervised or fine-tuned models.Source: §4.2.1 · p. 7

4.2.3. Multi-phase Motion Prediction.

• LVM maintains DSC above 85% at the fifth predicted phase, while public and private dataset figures show its highest DSC and smallest decline across forecast phases.The model conditions predictions on past CT phases and feeds predictions into subsequent steps, supporting patient-specific motion patterns and temporal consistency.Source: §4.2.3 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

4.2.4. Ablation Study.

• Combining CT and masks yields the best performance across metrics; CT supplies texture and density continuity, while masks constrain motion learning to anatomically relevant regions.The ablation evaluates CT-only, mask-only, and combined inputs across lung, heart, and liver datasets.Source: Table 3 · p. 9, §4.2.4 · p. 9

4.2.5. Visualization

• Qualitative examples show reduced lung prediction errors and capture of diaphragm-driven liver deformation, while segmentation examples delineate organs without task-specific fine-tuning.For lung and liver, the visualizations compare predictions for the next five phases with the ground truth; the segmentation examples cover liver, heart, and lung.Source: §4.2.5 · pp. 10–11

5. Limitations & Future Directions

• Reliable integration into clinical radiotherapy workflows remains challenging despite evidence of clinically meaningful zero-shot reasoning.Source: §5 · p. 11

Beyond intra-fraction motion

• Current motion prediction focuses on intra-fraction respiratory motion; inter-fraction anatomical changes require longitudinal imaging and reasoning across extended temporal contexts.Source: §5 · p. 11

Generalization across fundamental imaging tasks

• Zero-shot segmentation, denoising, and super-resolution show transferable spatial reasoning, but performance does not yet match specialized, domain-tuned networks.Source: §5 · p. 11

Toward a unified medical foundation model

• The long-term aim is a large-video-architecture medical foundation model for diverse imaging tasks and temporal reasoning without task-specific retraining.Source: §5 · p. 11

• Without medical-domain training, a video model predicts patient-specific respiratory motion from sequential CT with coherent, anatomically consistent outputs, surpassing specialized pipelines.Source: §6 · p. 11
• The same model performs segmentation, denoising, and super-resolution zero-shot, though it does not yet match fully supervised approaches.Source: §6 · p. 11
• The authors envision video architectures supporting medical foundation models that perform diverse imaging tasks with minimal supervision.Source: §6 · p. 11