Source-linked AI summary
Pix2Rep-v2: Data-Efficient Representation Learning for Dense Medical Imaging Applications
S. Sifaoui, E. Angelini, S. Toupin, T. Pezel, L. Le Folgoc
TL;DR
Dense medical imaging needs useful pixel- and voxel-level representations despite scarce annotations. Pix2Rep-v2 combines equivariant redundancy-reduction SSL with multiscale and implicit 3D designs, then evaluates fine-tuned and training-free prototype-based use across settings, reporting strong few-shot data-efficiency and competitive task results.
Problem
Scarce pixel-level annotations limit development of dense medical imaging solutions for new applications, modalities, and scanners.
Method
Pix2Rep-v2 learns dense representations with pixel-level redundancy reduction, spatial equivariance, multiscale sampling, 3D support, and dense-prototype in-context inference.
Results
Pix2Rep-v2 outperforms competing methods across cardiac MRI and 3D abdominal CT settings and achieves +9.3 Dice points over Pix2Rep in one-shot M&Ms-2 segmentation.
Takeaways & Limitations
Pix2Rep-v2 supports data-efficient dense medical imaging with strong few-shot and in-context capabilities across multiple tasks, modalities, structures, backbones, and 2D/3D settings.
Takeaways & Limitations
Patch-level SSL baselines train only the encoder, leaving the decoder to be trained from scratch during downstream fine-tuning, potentially reducing few-shot performance.
Abstract
from arXiv · showhide
Dense self-supervised learning (SSL) is a powerful paradigm for learning without annotations the local descriptors required to solve dense medical imaging tasks. We present Pix2Rep-v2, a framework for SSL of pixel- and voxel-level representations suitable for few-shot downstream applications. Pix2Rep-v2 addresses the main challenges of dense SSL by leveraging a redundancy reduction objective at the pixel-level with a principle of equivariance of dense representations, that scales efficiently to 3D or wide field-of-view applications. We evaluate our method on four datasets, across multiple tasks, multiple modalities and anatomical structures using multiple backbones in 2D and 3D, and under various data regimes. As an alternative to linear probing or full fine-tuning on the downstream task, we also propose an in-context variant, without downstream training, based on a dense prototype approach. Pix2Rep-v2 shows substantially higher data-efficiency in few-shot scenarios compared to fully supervised baselines, and is competitive with the state-of-the-art e.g., +9.3 Dice points in one-shot segmentation on the M&Ms-2 dataset. Our code and pre-trained models are publicly available at https://github.com/BioMedTP/pix2rep-v2.
1 Introduction
Pix2Rep-v2 targets the annotation bottleneck in dense medical imaging by learning pixel-level representations without labels, including for new applications, modalities, scanners, and data-limited settings. It combines efficient redundancy reduction, equivariance, multiscale sampling, broad evaluation, and a training-free in-context variant.
- Pixel-level annotations remain scarce for new medical imaging applications, modalities, and deployment data from new scanners.
- Foundation models may require thousands of densely annotated scans, motivating scalable self-supervised pre-training and small task-specific models trained with minimal annotation.
- Pix2Rep-v2 addresses dense contrastive learning challenges with an equivariance-based formulation using a single arbitrary spatial augmentation and non-contrastive redundancy reduction.
- The framework uses an efficient pixel-level redundancy reduction objective, aggressive multiscale sampling, and evaluation across tasks, modalities, structures, data regimes, backbones, and 2D/3D settings.
- A parameter-free, training-free in-context version complements linear probing and full fine-tuning to assess the intrinsic quality of Pix2Rep-v2 representations.
2 Related Work
Dense SSL adapts self-supervised objectives to pixel- or patch-level representations for dense tasks, but existing approaches face limitations in spatial augmentation, computational cost, and downstream decoder initialization. Pix2Rep-v2 is positioned as a redundancy-reduction alternative with multiscale, 3D, and in-context capabilities.
- Dense SSL learns pixel-level or patch-level representations for dense downstream tasks using contrastive, masked-image-modeling, joint-embedding, or redundancy-reduction objectives.
- Pixel-level contrastive methods align representations of corresponding anatomical points, but partial crop overlap limits spatial augmentation and sparse sampling avoids exploding negative-set size.
- 3D contrastive approaches address computational demands with dedicated coarse-to-fine feature-pyramid representations.
- Patch-level SSL trains only the encoder, leaving the backbone decoder to be trained from scratch during downstream fine-tuning, potentially reducing few-shot performance.
- Pix2Rep-v2 extends dense SSL with redundancy reduction, aggressive multiscale sampling, 3D support, and a scalable dense-prototype in-context alternative.
3 Methods
Pix2Rep-v2 learns dense representations by enforcing photometric invariance, spatial equivariance, informativeness, and component-wise non-redundancy across augmented views. Its multiscale and implicit 3D designs support efficient pre-training, while dense prototypes enable segmentation and propagation without task-specific fine-tuning.
- Pix2Rep-v2 extends dense SSL with redundancy reduction, aggressive multiscale patch sampling, 3D support, and in-context capabilities.
- The framework enforces invariance to photometric augmentations, equivariance to spatial transformations, and component-wise informativeness and non-redundancy.
- Two photometric views and one spatial transformation are processed asymmetrically so transformed representations from one view are paired with representations extracted from the transformed other view.
- Paired projected representations are normalized and used to compute a d×d cross-correlation matrix, avoiding the pixel-count-squared scaling of contrastive similarity matrices.
- Multiscale sampling resizes randomly sized patches from whole scans, exposing the model to varied spatial scales while preserving aspect ratio.
- 3D Backbone: The implicit 3D U-Net computes representations at sampled coordinates rather than storing full-resolution feature maps, sampling 2^17 coordinates per 3D patch.
- In-Context Segmentation: In-context segmentation gathers support-pixel representations into prototypes and assigns query-pixel labels by nearest-neighbor cosine similarity without task-specific fine-tuning.
- Zero-Shot 3D+t Video Propagation: Zero-shot 3D+t propagation uses prototypes from the previous frame and the labeled reference anchor frame to limit error accumulation across time.
4 Experiments and Results
Pix2Rep-v2 is evaluated across cardiac MRI segmentation and video propagation, plus 3D abdominal CT segmentation, using varied backbones, data regimes, and comparison methods. It achieves strong data efficiency and competitive results across these settings.
- Experimental scope: Pix2Rep-v2 is evaluated on cardiac MRI segmentation, cine MRI video propagation, and multi-organ abdominal CT segmentation.Experiments use 2D U-Net and Swin-UNETR backbones for cardiac MRI and an implicit 3D U-Net for abdominal CT.
- Experimental setup: Pix2Rep-v2 uses unlabeled raw training data for pre-training and labeled subsets representing 1%, 5%, 10%, 25%, and 100% training regimes for downstream evaluation.Performance is quantified with 3D Dice averaged over segmented structures and the test set.
- Cardiac MRI segmentation: +9.3 Dice points over best-of-the-rest Pix2Rep and +15.0 Dice points over next-best BT-UNet are achieved by fine-tuned Pix2Rep-v2 with U-Net at |Xtr| = 1% on M&Ms-2.The result is reported across cardiac MRI cohorts and data regimes, with in-context Pix2Rep-v2 also outperforming fine-tuned baselines at |XS| = 1%.
- Cardiac MRI segmentation: ×25 data-efficiency in few-shot and ×5-10 in large data regimes is obtained with Pix2Rep-v2 pretraining versus training from scratch under identical experimental setups.The comparisons keep backbone, pre-processing, and training iterations identical.
- Video propagation: Pix2Rep-v2 outperforms SAM 3 in zero-shot cine MRI video propagation and almost reaches MedSAM2 despite MedSAM2 training on the evaluated scans and annotations.Its redundancy-reduction representations slightly outperform Pix2Rep’s contrastive representations with the same propagation mechanism.
- 3D abdominal CT segmentation: Pix2Rep-v2 outperforms natively 3D self-supervised methods on AMOS and shows ×5 data-efficiency in low-data regimes versus an implicit U-Net trained from scratch.It reaches equivalent performance with ×5 fewer annotated scans for fine-tuning.
5 Discussion and Conclusion
The paper presents Pix2Rep-v2 as a framework for data-efficient pixel-level task solving with strong few-shot and in-context capabilities. It identifies future extensions to additional tasks and joint image- and pixel-level representations.
- Conclusion: Pix2Rep-v2 is presented as a dense representation learning framework for data-efficient pixel-level tasks with strong few-shot and in-context capabilities.The authors connect these capabilities to potential development of medical imaging foundation models and task-specific systems.
- Future work: Future work will investigate landmark detection, registration, and coupling image-level with pixel-level representations.