PaperlayerView exact revision on arXiv

Source-linked AI summary

MedVTok: A General-Purpose Medical Visual Tokenizer

Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, Shaohao Rui, Chaoyang Zhang, Wenjie Li, Wenhao Tang, Cheng Tang, Wei Li, Shujian Gao, Yujie Zhang, Ying Chen, Zilong Li, Zongyuan Ge, Diping Song, Jin Ye, Ming Hu, Junjun He, Hongming Shan

arXiv:2610.35842v1eess.IVcs.CV

TL;DR

Existing medical visual tokenizers often specialize by imaging dimension or modality, leaving a need for a reusable visual interface for heterogeneous clinical imaging. MedVTok combines inter-slice structural regularization with multi-expert semantic alignment in a unified 2D/3D token space, achieving state-of-the-art results across reconstruction and six downstream tasks.

  • Problem

    Medical visual tokenizers are often tied to specific dimensions and modalities, producing fragmented representations that are difficult to reuse in multimodal systems.

    Source: §1 · p. 2

  • Method

    MedVTok maps 2D images and 3D volumes into a unified token space, using slice differential regularization and multi-expert alignment to preserve structure and clinical semantics.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    MedVTok achieves state-of-the-art results across reconstruction and six downstream tasks, spanning pixel-level fidelity to report generation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results demonstrate that structural fidelity and semantic richness can be jointly optimized within a single tokenizer.

    Source: §5 · p. 10

    The training data currently focus on common imaging modalities, leaving other modalities, clinical photos, and medical-publication illustrations unexplored.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multimodal medical AI requires a common visual interface for encoding heterogeneous imaging evidence for patient condition modeling. However, existing medical visual tokenizers are often tied to a specific dimension and imaging modality, forcing multimodal systems to rely on fragmented and poorly reusable representations. We introduce MedVTok, a general-purpose visual tokenizer that maps 2D images and 3D volumes from diverse imaging modalities into a unified token space. Building such a tokenizer poses two key challenges: preserving anatomical structure across dimensions and capturing clinical semantics across modalities. To this end, MedVTok introduces (i) slice differential regularization, which explicitly models inter-slice anatomical coherence missing in previous per-pixel/per-voxel supervision, and (ii) multi-expert representation alignment, which integrates knowledge from multiple medical expert encoders while retaining modality-specific complementary knowledge. Trained on 28M images and 70K volumes across 9 imaging modalities, MedVTok supports classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation through a single tokenizer. Across 24 benchmarks, MedVTok achieves state-of-the-art performance, demonstrating a scalable token interface that unifies medical image dimension, modality, and task usage.

1 Introduction

Multimodal medical AI needs a shared interface for heterogeneous clinical evidence, but existing visual tokenizers often produce fragmented representations across imaging dimensions and modalities. MedVTok combines inter-slice regularization and multi-expert semantic alignment in a unified 2D/3D token space, and reports state-of-the-art results across 24 benchmarks.

1 Introduction

  • Across 24 2D/3D benchmarks, MedVTok achieves state-of-the-art performance across classification, segmentation, retrieval, synthesis, visual question answering, and medical report generation.The evaluation corpus spans 9 imaging modalities, with over 28M 2D images and 70K 3D volumes.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • Existing medical visual tokenizers are often tied to a specific modality and dimension, producing fragmented representations that are difficult to reuse in multimodal systems.The paper identifies preserving inter-slice structure and capturing modality-specific clinical semantics as two central challenges.Source: §1 · p. 2
  • MedVTok maps diverse 2D images and 3D volumes into a unified token space, using SliDR for inter-slice continuity and MERA to integrate multiple experts while retaining complementary semantics.The contribution combines per-pixel/per-voxel supervision with slice differential regularization and aligns shared representations with modality-specialized experts.Source: §1 · p. 2

2 Related Work

Prior medical tokenizers tend to specialize by input dimension or rely on a single semantic encoder, limiting how they represent volumetric continuity and complementary modality knowledge. MedVTok pairs a unified 2D–3D interface with slice differential regularization and multi-expert alignment.

2 Related Work

  • Earlier approaches adapt tokenizers across 2D and 3D by pooling 3D weights or duplicating 2D images, but do not explicitly account for through-plane anatomical continuity.MedVTok instead combines Slice Differential Regularization with dimension-agnostic patchification for a unified 2D–3D interface.Source: §2 · p. 3
  • Prior semantic approaches either offer broad but potentially incomplete coverage or isolate specialty-specific encoders; MedVTok integrates multiple experts while retaining modality-specific semantics.The related-work discussion notes that aligning to a single encoder can inherit single-expert bias.Source: §2 · p. 3
  • The training framework pairs reconstruction and slice differential losses for structural fidelity with MERA’s shared and expert losses for cross-modality clinical semantics.The figure caption presents these as the framework’s two learning objectives.Source: Figure 2 · p. 3, §2 · p. 3

3 Method

MedVTok learns a shared latent space for 2D images and 3D volumes using complementary structural and semantic supervision. Its method combines patch-based tokenization, inter-slice differential losses, multi-expert alignment, and a two-stage training schedule.

3.1 Overview of MedVTok

  • MedVTok uses a shared latent space for 2D and 3D data, with patch tokens and coordinates supporting variable-resolution, mixed-dimension batching and reconstruction plus semantic alignment.An encoder produces per-patch tokens for decoding and structural supervision, while an aggregation head produces a representation for MERA.Source: §3.1 · pp. 3–4

3.2 Slice Differential Regularization

  • Slice Differential Regularization matches first- and second-order slice changes between input and reconstruction to preserve volumetric continuity beyond voxelwise supervision.The first term matches inter-slice transitions; the second preserves through-plane curvature.Source: §3.2 · p. 4

3.3 Multi-Expert Representation Alignment

  • MERA aligns aggregated MedVTok representations with projected modality-specific expert features, then backprojects them to preserve complementary knowledge in each expert space.Mean pooling with a linear layer improves dense tasks while retaining performance on other tasks.Source: §3.3 · pp. 4–5

3.4 Training and Inference

  • Training uses two stages: 2D images alone with SliDR disabled, followed by joint 2D-image and 3D-volume training with SliDR weighted at 0.1.At inference, encoded tokens are consumed by downstream models without further fine-tuning of the tokenizer.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate MedVTok’s token fidelity and downstream utility across multiple tasks, then ablate its alignment, pooling, data-scaling, and volumetric-continuity components. The reported results favor MedVTok across the evaluated tasks and show improvements from the tested design choices.

4.1 Pretraining

  • Training uses 28,769,502 images and 70,144 volumes, with 2D-only training in Stage 1 followed by both 2D images and 3D volumes in Stage 2.Stage 2 alternates 2D and 3D data iterations.Source: §4.1 · p. 5

4.2 Evaluation on Downstream Tasks

  • MedVTok ranks first on all nine modalities in 2D reconstruction PSNR and SSIM, and achieves the best PSNR and SSIM on both MRI and CT volumes.The 3D results are reported against PUMIT and MedVAE; the 2D results are supported by Table 1.Source: Table 1 · p. 6, §4.2 · p. 6
  • MedVTok achieves the highest average on image classification and retrieval, surpassing other tokenizers and dedicated semantic encoders.The evaluation reports accuracy for classification and Recall@1 for retrieval.Source: §4.2 · p. 6, Table 2 · p. 7
  • MedVTok achieves the best average Dice score for segmentation, outperforming all baselines, including semantic encoders.The reported comparison uses a linear segmentation protocol on frozen patch tokens.Source: §4.2 · p. 7, Table 3 · p. 7
  • MedVTok achieves the best average FID for image synthesis, outperforming specialized and previous unified tokenizers.Lower FID indicates higher fidelity; the comparison trains diffusion models from scratch in frozen tokenizers’ latent spaces using identical architecture and data.Source: §4.2 · p. 8, Table 4 · p. 8
  • MedVTok has the best average VQA accuracy and the best report-generation scores across all reported metrics and anatomical regions.For report generation, MedVTok’s GREEN score is 0.413 versus 0.386 for MedITok.Source: §4.2 · p. 8, Table 5 · p. 8, Table 6 · p. 9

4.3 Ablation Studies

  • Multi-expert alignment improves classification and segmentation over single-expert alignment, while expert-space complementary learning adds a 1.53% classification gain.Replacing attention pooling with mean pooling improves segmentation while retaining competitive classification performance.Source: §4.3 · p. 9
  • Scaling training from 3M to 28M images yields consistent improvements, while adding SliDR improves 3D reconstruction PSNR and SSIM.The SliDR ablation reports improved through-plane continuity.Source: §4.3 · p. 9

5 Conclusion

MedVTok encodes 2D images and 3D volumes into a shared space using one transformer, with supervision for inter-slice fidelity and modality-specific semantics. The appendix details its architecture and training-data handling, including patch sequences that can mix image dimensions and resolutions.

5 Conclusion

  • MedVTok encodes 2D images and 3D volumes into a shared representational space through a single transformer architecture.Its conclusion reports state-of-the-art results across reconstruction and six downstream tasks, from pixel-level fidelity to report generation.Source: §5 · p. 10

Appendix

  • The appendix outlines architecture, training settings, evaluation protocols, additional results and analyses, and societal impacts.It identifies Appendix A as architecture, B as training, C as evaluation, and D as broader societal impacts.Source: Appendix · p. 25

A Tokenizer Overview

  • The encoder and decoder each use 27 transformer blocks, while 3D volumes use depth patches of 4 and spatial patches of 16 × 16.Encoded tokens are projected to a 96-dimensional output split into mean and log-variance, yielding latent codes of dimension 48.Source: A Tokenizer Overview · p. 25
  • Inputs are patchified into a common concatenated format; 2D images are zero-padded along the temporal axis, while 3D volumes are patchified across depth and space.A 256 × 256 image yields 256 patches; a 32-slice 256 × 256 CT volume yields 2,048 patches.Source: A Tokenizer Overview · p. 25
  • Patches from mixed resolutions, depths, and 2D or 3D inputs are concatenated into one sequence, avoiding dense padding and using O(N) memory.Coordinates and per-sample sequence lengths support batch-wise slicing; block-diagonal attention masks prevent cross-sample information leakage.Source: A Tokenizer Overview · p. 25
  • MERA uses a modality-assigned expert with dedicated projectors, aligning representations in a shared token space while retaining modality-specific information.The expert encoders are precomputed offline, and the listed 3D experts are included only in Stage 2.Source: A Tokenizer Overview · p. 25, Table 7 · p. 26

B.1 Training Datasets

The training corpus combines publicly available medical imaging data across nine modalities, with dataset-size thresholds and quality filters applied during curation.

B.1 Training Datasets

  • The curated training set contains 28,769,502 images and 70,144 volumes spanning nine imaging modalities.Before filtering, the corpus includes only 2D datasets with at least 9,000 images and 3D datasets with at least 180 volumes.Source: §B.1 · p. 26
  • Minimum dataset-size thresholds exclude small benchmark datasets to preserve evaluation integrity and avoid highly fragmented sources with little training signal relative to integration overhead.Source: §B.1 · p. 26
  • Automated filters screen 2D images for quality and clinical relevance, followed by manual exclusion of non-clinical illustrations such as tables and plots.Automated checks include dynamic range, minimum resolution, pixel-value diversity, and information content.Source: §B.1 · p. 26

B.2 Implementation Details

MedVTok is trained with reconstruction, KL, semantic-alignment, and—during volumetric training—slice-differential losses. Training proceeds in two stages, moving from 2D-only data to alternating 2D and 3D batches.

B.2 Implementation Details

  • The total training objective combines reconstruction, slice-differential, KL, and MERA losses, weighted by their respective coefficients.The stated defaults are λKL = 10^-6, λMERA = 1, and λSliDR = 0.1.Source: §B.2 · p. 26
  • The reconstruction loss combines pixel-level, perceptual, and adversarial terms, with an adaptive adversarial weight balancing reconstruction and adversarial gradients on the decoder’s last layer.The pixel-level terms are mean absolute error and mean squared error; the perceptual term is LPIPS.Source: §B.2 · p. 26
  • KL regularization projects patch tokens to a diagonal Gaussian posterior and penalizes its deviation from a standard normal prior.The latent code is sampled by reparameterization, with latent dimension 48.Source: §B.2 · p. 26
  • Stage 1 trains on 2D images with reconstruction, KL, and MERA losses; Stage 2 initializes from that checkpoint, adds 3D experts and SliDR, and alternates 2D and 3D batches.The 3D batches use an approximately equal ratio of CT and MR volumes.Source: §B.2 · pp. 26–27
  • Training uses AdamW with cosine annealing, and the reported full-scale training runs take approximately 15.75 hours for Stage 1 and 43.5 hours for Stage 2 on 16 NVIDIA H200 GPUs.Peak GPU memory usage is 132.54 GB per device; the full project required more compute than reported because of preliminary and failed experiments.Source: §B.2 · p. 27
  • The code and model weights are slated for release under the Apache 2.0 license.Source: §B.2 · p. 27

C.1 Evaluation Datasets

The evaluation datasets are grouped by downstream task, and the authors report that all listed datasets are excluded from the training corpus.

C.1 Evaluation Datasets

  • All evaluation datasets listed in the task-grouped benchmark summary are excluded from the training corpus.Source: §C.1 · p. 27

C.2 Evaluation Setup and Results

The evaluation tests frozen tokenizers across reconstruction, classification, retrieval, segmentation, synthesis, VQA, and report generation. MedVTok leads reported comparisons across segmentation, synthesis, and report generation, with additional evidence from expert comparisons and VQA controls.

Evaluation protocols

  • Evaluation uses frozen-tokenizer reconstructions, k-NN classification and retrieval, and lightweight linear segmentation heads across task-grouped datasets.Reconstruction reports PSNR and SSIM; retrieval reports Recall@1, and segmentation reports Dice and HD95.Source: §C.2 · pp. 27–28, Table 8 · p. 28

Segmentation and modality experts

  • MedVTok leads average performance across modalities on HD95 and exceeds several modality experts in classification and segmentation, while Conch scores higher on pathology.It exceeds MedImageInsight on seven of nine classification benchmarks and all eight segmentation benchmarks.Source: §C.2 · p. 28, Table 9 · p. 31

3D segmentation

  • MedVTok achieves the highest overall Dice on both AMOS-MM CT and MRISegmenter MRI, and the highest transfer Dice on AMOS-MM MRI without target-set training or tuning.The 3D segmentation datasets were excluded from tokenizer pretraining; transfer evaluation used ten shared anatomical labels.Source: §C.2 · p. 28, Table 12 · p. 32, Table 13 · p. 33

Image synthesis protocol

  • For synthesis, each tokenizer is the only variable in matched latent-diffusion models, with a frozen CLIP text encoder and controlled template captions.The setup is intended to compare latent-space generative suitability, not to build state-of-the-art text-to-image models or benchmark fine-grained synthesis.Source: §C.2 · pp. 28–29

Image synthesis

  • MedVTok consistently achieves the best average performance across KID, FID, and IS in the image-synthesis evaluation.The comparison uses frozen tokenizers in otherwise matched latent-diffusion setups.Source: §C.2 · pp. 28–29, Table 14 · p. 33

VQA and report-generation protocols

  • VQA uses frozen visual tokenizers as LLaVA-Med backbones; report generation uses frozen tokenizers in LaMed with CT volumes and reports from AMOS-MM.LaMed evaluation covers abdomen, chest, and pelvis; representative LLaVA-Med responses are shown in Figure 12.Source: §C.2 · pp. 29–30

Visual question answering

  • MedVTok leads VQA under connector-only training, and substituting GPT-5 for the Qwen3 judge preserves the tokenizer ranking.The VQA controls hold the connector, language model, data, LoRA settings, decoding, and judge fixed across tokenizers.Source: §C.2 · p. 30, Table 15 · p. 33

Medical report generation

  • The LaMed model with MedVTok achieves superior average scores on each reported metric for medical report generation.The evaluation reports BLEU-2, ROUGE-L, METEOR, RaTEScore, and GREEN by anatomical region and globally.Source: §C.2 · p. 32

C.3 Additional Analyses

Additional analyses examine the balance between semantic performance and reconstruction, test alternative depth regularizers, and evaluate transfer to unseen microscopy and matched-data comparisons. MedVTok shows gains under the reported settings, with trade-offs when regularization weights increase.

Effect of λMERA

  • At λMERA = 2, PSNR drops by nearly 3 dB and SSIM falls to 0.868; λMERA = 1 is selected to balance semantic performance and reconstruction.Increasing MERA weight monotonically improves semantic metrics, while λMERA = 0 leaves classification and segmentation low despite strong reconstruction.Source: §C.3 · p. 33, Figure 9 · p. 34

Effect of λSliDR

  • Enabling SliDR substantially improves 3D PSNR and SSIM without affecting 2D reconstruction; raising λSliDR to 0.5 degrades classification accuracy by 2.6%.The authors choose λSliDR = 0.1 to capture most 3D gains without compromising other objectives.Source: §C.3 · p. 33, Figure 10 · p. 34

Alternative depth regularizers

  • SliDR achieves the highest mean PSNR and SSIM on AMOS-MM CT and MRISegmenter MRI in a controlled 10K-volume comparison of depth regularizers.The alternatives are direct gradient matching, total variation, and Laplacian regularization.Source: §C.3 · p. 33, Table 17 · p. 35
  • Unlike TV and Laplacian penalties, SliDR matches first- and second-order slice differentials to the ground-truth volume, targeting anatomical transitions.TV and Laplacian encourage smoothness without reference to ground truth, including where genuine inter-slice changes occur.Source: §C.3 · p. 34

Transfer to an unseen modality

  • 94.78% k-NN accuracy and 96.84% linear-classifier accuracy are the highest reported on unseen PBC microscopy, absent from tokenizer pretraining.The evaluation uses frozen tokenizer features.Source: §C.3 · pp. 34–35, Table 18 · p. 35

Comparison on the same training data

  • On the same 2M training images, MedVTok achieves the highest reconstruction PSNR, classification accuracy, and segmentation Dice, despite using images alone while competitors use text supervision.AToken, MedITok, and MedVTok are retrained with the same settings.Source: §C.3 · p. 35, Table 19 · p. 35

t-SNE analysis

  • MedVTok forms clearly separated PBC cell-type clusters and tighter JSIEC retinal-condition clusters than the compared tokenizers in t-SNE visualizations.PBC microscopy is an out-of-distribution modality not seen during training.Source: §C.3 · p. 35

D Broader Impact

The broader-impact discussion presents a shared medical visual interface as a potential aid to multimodal research and patient-level modeling. It also cautions that downstream clinical use requires oversight, target-population validation, and regulatory compliance.

Potential impact

  • A shared, reusable visual interface across modalities may accelerate multimodal medical AI research and patient-level modeling.The authors say it may lower barriers for under-resourced specialties or institutions lacking data and compute.Source: D Broader Impact · p. 35

Use and safety

  • Downstream systems using MedVTok could produce incorrect predictions that may harm patients if acted on without appropriate clinical oversight.The authors state that MedVTok is intended for research use and encourage validation on target populations and compliance with applicable regulations before clinical deployment.Source: D Broader Impact · p. 36

Examples and corpus

  • The paper presents representative LLaVA-Med responses and LaMed reports, alongside 2D and 3D corpus-distribution figures.The corpus figures describe 28.7M images across nine modalities and 70K volumes across two modalities.Source: Figure 12 · p. 37, Figure 13 · p. 38, Figure 14 · p. 39

Related papers

Loading 2610.35842v1…