Source-linked AI summary
B-MIM: Biased Masked Image Modeling for Generalizable Segmentation of Fine-Grained Anatomical Structures
Sebastián González, Karen Sanchez, José M. Saavedra, Marcelo Pizarro, Bernard Ghanem
TL;DR
Existing medical-imaging encoders emphasize coarse-grained tasks, leaving fine-grained structures such as vessels and small tumors less addressed. The paper introduces B-MIM, a biased masked pretraining objective for CT, and reports improved vessel topological fidelity with parameter-efficient adaptation and challenging tumor segmentation remaining variable.
Problem
Existing medical-imaging encoders focus mainly on coarse-grained tasks, with comparatively less emphasis on fine-grained structures such as vessels or small tumors.
Method
B-MIM stochastically reduces global semantic alignment to prioritize local patch reconstruction, using a 3D Swin Transformer pretrained on a standardized cohort of 9,955 CT studies from 17 public sources.
Results
B-MIM consistently enhances vessel topological fidelity across datasets, with B-MIM-C improving clDice from 0.491 to 0.582 under CRLM-to-IRCAD domain shift, while lightweight adaptation achieves competitive segmentation scores.
Takeaways & Limitations
Reducing global semantic pressure during pretraining supports cross-dataset transfer for vessel structures while remaining parameter-efficient during downstream adaptation.
Abstract
from arXiv · showhide
Self-supervised pretraining enables transferable representations for medical imaging, yet most CT encoders remain biased toward coarse semantic understanding, limiting their sensitivity to fine-grained anatomical structures such as vessels or small tumors. In this paper, we introduce Biased Masked Image Modeling (B-MIM), a modification of the iBOT objective that stochastically reduces global semantic alignment to prioritize local patch reconstruction. This bias encourages the encoder to capture high-frequency morphological details and structural continuity. We curate a multi-institutional CT abdominal dataset of 9,955 filtered studies from 17 public sources and pretrain a 3D Swin Transformer backbone using B-MIM. Across inter-dataset experiments on liver vessel segmentation, the proposed encoder improves topological fidelity (clDice) and achieves competitive Dice scores in tumor segmentation, compared to fully fine-tuned baselines, despite updating only a fraction of the parameters. Our results suggest that reducing global semantic pressure during pretraining enhances generalization to intricate anatomical structures.
1 Introduction
The paper targets limited sensitivity of CT encoders to fine-grained anatomical structures by proposing a CT encoder that emphasizes local morphological detail and supports parameter-efficient downstream segmentation.
- Existing CT encoders mainly emphasize coarse-grained tasks, leaving vessels and small tumors comparatively underrepresented despite their clinical importance.These structures support liver resection planning, cancer treatment, disease prevention, early detection, monitoring, and treatment planning.
- An out-of-distribution evaluation observed up to an 18% gain in vessel segmentation.The encoder was evaluated on liver vessel and small-tumor segmentation with inter-dataset transfer.
- The curated pretraining dataset contains 9,955 CT studies from 17 public sources.The cohort was standardized for 3D self-supervised learning.
- The proposed B-MIM objective prioritizes local patch reconstruction over global semantic alignment.It modifies masked-image pretraining to reduce global semantic pressure during representation learning.
- The 3D Swin-based strategy encourages high-frequency morphological representations and topological continuity for fine-grained structures.The intended focus is especially relevant to vessels and small tumors.
- The pretrained backbone achieves competitive vessel and tumor segmentation results while updating only 16k trainable parameters.This supports parameter-efficient adaptation of the encoder for specialized tasks.
2 Related Works
Related work establishes self-supervised encoders as transferable representations, while identifying a gap in CT methods for fine-grained anatomical structures.
- Self-supervision enables encoders to learn representations from unlabeled data and supports adaptation across diverse tasks.The discussion situates this approach among MoCo, SimCLR, BYOL, and DINO-family methods.
- iBOT reconstructs masked patch encodings using contextual information, providing a basis for focusing on small image structures.B-MIM leverages this local-patch reconstruction principle.
- Medical visual encoders have been developed predominantly for X-ray imaging, including RayDINO, RadDINO, and CXR.RayDINO is described as trained self-supervised on 873,000 chest X-rays.
- VoCo provides a pretrained 3D medical-imaging methodology using contextual position priors but is not focused on fine-grained tasks.Its stated scope differs from the paper’s emphasis on vessels and small tumors.
- The paper addresses a gap in pretrained encoders for small anatomical structures in non-X-ray modalities by proposing a CT encoder for vessels and small tumors.The motivation combines the transferability of pretrained encoders with the limited fine-grained coverage identified in CT.
3 Methodology
The methodology builds a standardized multi-institutional abdominal CT cohort, prepares it for 3D self-supervised learning, and evaluates cross-dataset segmentation generalization.
- The methodology comprises dataset curation, encoder architecture design, and inter-dataset evaluation.These stages are organized around cross-dataset segmentation of fine-grained abdominal structures.
- The curated dataset aggregates 17 public DICOM and NIfTI sources for CT encoder pretraining.Automated filtering and normalization standardize orientation and abdominal anatomy.
- The filtering protocol excludes unsuitable scans, restricts cases by adult status and volumetric depth, and requires specified inclusion conditions.The passage explicitly describes exclusions for localizers, null or inconsistent orientations, and series with fewer than 20 post-cropping slices.
- The final cohort contains 9,955 abdominal CT studies and 1,993,194 two-dimensional slices.It comprises 5,613 DICOM studies and 4,342 NIfTI studies.
- All volumes are converted to LPS orientation and automatically cropped to the abdominal region using TotalSegmentator landmarks.The crop extends from the superior liver to the inferior L4 pole or kidneys, whichever is lower.
- The curated abdominal CT cohort is planned for public release upon acceptance.The complete source-dataset list is identified as available in the repository.
3.2 Biased Masked Image Modeling (B-MIM)
Figure 1 presents the proposed CT encoder and its integration into a downstream segmentation model.
- Figure 1 shows the general scheme of the proposed encoder and its integration into a downstream segmentation task.
Backbone Architecture.
The method uses a 3D Swin Transformer with iBOT-style cross-view self-distillation, modified to prioritize masked-patch reconstruction over global semantic alignment. This design targets local textures, long-range dependencies, and fine-grained structural continuity.
- Backbone Architecture: The 3D Swin Transformer partitions CT volumes into non-overlapping patches and computes hierarchical self-attention within shifting windows.This captures local textures and long-range spatial dependencies for detailed anatomical representation.
- Pre-training Objective: The encoder is pretrained with iBOT as a cross-view self-distillation task using student and exponentially averaged teacher networks.The framework combines global [CLS]-token and masked-token distillation losses.
- Biased Masked Image Modeling: B-MIM stochastically determines whether the global [CLS]-token loss is applied, increasing emphasis on masked-patch distillation.A Bernoulli variable samples this omission independently for each training batch.
- Biased Masked Image Modeling: The resulting objective reduces global semantic alignment in favor of local patch reconstruction for fine-grained structure learning.The approach is described as promoting robustness to fine-scale morphological variation.
Rationale for the Bias.
The bias addresses the tendency of standard iBOT to give equal priority to global and local objectives. Setting p < 1 reduces global feature alignment so the encoder focuses more on high-resolution geometries and continuity.
- Rationale for the Bias: Standard iBOT uses p = 1, whereas B-MIM sets p < 1 to reduce the frequency of global feature alignment.Global supervision is omitted stochastically rather than deterministically reweighted.
- Rationale for the Bias: The bias encourages reconstruction of masked tokens through LMIM instead of reliance on coarse global semantic features.The intended targets are high-resolution local geometries and continuity needed for thin structures.
Parameter-Efficient Downstream Adaptation
Downstream adaptation attaches a lightweight decoder to the pretrained encoder and compares mostly frozen, fully fine-tuned, and LoRA configurations. The primary feature-concatenation setting updates only 16,344 parameters.
- Parameter-Efficient Downstream Adaptation: The parameter-efficient strategy attaches a lightweight nnU-Net decoder to the frozen Swin-3D encoder.This avoids conventional full fine-tuning during representational evaluation.
- Feature Concatenation: B-MIM-A concatenates multi-scale pretrained embeddings into corresponding decoder levels while updating only 16,344 parameters.The encoder remains largely frozen in this configuration.
- Full Finetuning: B-MIM-B fully fine-tunes the same architecture to assess the effect of updating all encoder parameters.This provides the comparison with parameter-efficient adaptation.
- LoRA Adaptation: B-MIM-C adds LoRA to refine the bottleneck without unfreezing the backbone.This configuration is optional and comparative.
3.3 Generalization-based Evaluation
The encoder is evaluated on liver vessel and small-tumor segmentation using abdominal CTs, with training and validation on CRLM and testing on two unseen datasets. Pretraining uses the curated cohort without vessel or tumor annotations.
- 3.3 Generalization-based Evaluation: The evaluation covers liver vessel segmentation and small-tumor segmentation on abdominal CTs.Both tasks use the CRLM dataset for training and validation.
- 3.3 Generalization-based Evaluation: Generalization is tested on IRCAD and MSD, which were not used during pretraining.The setup is designed to assess inter-dataset transfer.
- 3.3 Generalization-based Evaluation: The encoder is pretrained self-supervised on the curated abdominal CT cohort without access to vessel or tumor annotations.This separates pretraining from the downstream segmentation labels.
4 Results and Discussion
B-MIM improves cross-dataset vessel topology while retaining competitive performance with minimal adaptation, whereas tumor segmentation remains more variable. Qualitative results show better preservation of thin vascular branches and closer tumor-boundary delineation.
- Evaluation Protocol: Downstream models use five-fold cross-validation and report mean Dice and clDice across folds, with the encoder frozen unless otherwise specified.Pretraining used 200 epochs, while downstream training followed the default nnU-Net protocol.
- Vessel Segmentation: 0.582 clDice under CRLM-to-IRCAD transfer, up from 0.491 for Swin-nnUNet, a +0.091 absolute gain.This corresponds to an 18.5% relative improvement and exceeds the corresponding Dice variation.
- Parameter Efficiency: 16,344 trainable encoder parameters in B-MIM-A still yield competitive Dice and clDice scores on IRCAD and MSD.B-MIM-A keeps the encoder largely frozen during adaptation.
- Qualitative Comparison: B-MIM-C better preserves thin vascular-branch continuity and reduces fragmentation compared with Swin-UNETR and VoComni.The qualitative comparison focuses on local differences in fine-grained structure delineation.
- Parameter Efficiency: B-MIM-B updates 5,716,410 parameters, compared with 16,344 for B-MIM-A, reflecting a trade-off between adaptation size and generalization performance.B-MIM-B achieves strong tumor performance, whereas B-MIM-A maintains competitive vessel segmentation.
- Tumor Segmentation: B-MIM-B achieves the best tumor Dice in this setting, while tumor transfer remains more variable because morphology, size, and contrast patterns vary across patients.Tumor morphology is described as more variable than the consistent tubular structure of vessels.
5 Conclusions
B-MIM is a biased masked pretraining objective for fine-grained CT anatomy. It improves vessel topological fidelity and parameter-efficient cross-dataset transfer, while tumor segmentation remains more challenging and broader evaluation is planned.
- Conclusion: B-MIM reduces global semantic pressure during self-supervised training to enhance fine-grained anatomical representations in CT.The method is designed for structures such as vessels and small tumors.
- Conclusion: Vessel structures benefit noticeably from the representation bias and achieve improved cross-dataset transfer in the evaluation.The conclusion attributes this pattern to vessels' consistent tubular morphology.
- Conclusion: Tumor segmentation remains more challenging because tumor size, morphology, and appearance are more variable.The authors propose future evaluation stratified by lesion size and complexity.
- Future Work: Future work includes sensitivity analysis for p and extension to additional modalities such as MRI.The authors also call for broader evaluation across fine-grained tasks.