Source-linked AI summary
Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
Zongwei Zhou, Vatsal Sodha, Md Mahfuzur Rahman Siddiquee, Ruibin Feng, Nima Tajbakhsh, Michael B. Gotway, Jianming Liang
TL;DR
3D medical imaging transfer learning often reformulates volumetric tasks in 2D, losing 3D anatomical information. The paper introduces annotation-free Models Genesis, trained by unified self-supervision on medical images, and reports stronger transfer performance than scratch-trained 3D and evaluated 2D approaches.
Problem
The paper asks whether abundant, largely unannotated Chest CT images can train transferable 3D source models for high-performance target tasks.
Method
Models Genesis use unified self-supervised image restoration on unlabeled medical images to learn generic source representations for transfer to 3D classification and segmentation.
Results
Models Genesis significantly outperform learning from scratch across five target 3D applications and consistently top the evaluated 2D approaches.
Takeaways & Limitations
The results support Models Genesis as generic source models for transfer learning across diseases, organs, datasets, and modalities.
Takeaways & Limitations
The study focuses on Chest CT, while extending Models Genesis to MRI, Ultrasound, brain, and heart models remains future work.
Abstract
from arXiv · showhide
Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of our Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated yet recurrent anatomy in medical images can serve as strong supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.
1 Introduction
The paper asks whether abundant, largely unannotated Chest CT data can support transferable 3D source models, and introduces Models Genesis to answer that question through unified self-supervision.
- Chest CT is chosen because it supports lung-disease diagnosis, has substantial clinical importance, and is represented by large databases such as LIDC-IDRI and NLST.
- The central question is whether unannotated Chest CT images can train source models that produce high-performance target models through transfer learning.
- Models Genesis are generic source models for 3D imaging, with Genesis Chest CT trained on Chest CT scans and additional 2D variants developed for ablation and 2D applications.
- Models Genesis significantly outperform learning from scratch in all seven reported target tasks, while Genesis Chest CT consistently surpasses the evaluated 2D approaches.
- Medical images provide recurrent anatomical patterns that self-supervision can use to discover common representations of particular body parts.
- The framework trains an encoder-decoder to restore original patches from transformed patches, learning common visual representations through self-supervision.
2 Models Genesis
Models Genesis combine multiple image transformations into one restoration-based self-supervised framework. The resulting models are annotation-free, scalable, and usable across diverse target tasks.
- The framework uses an encoder-decoder to restore transformed patches, then fine-tunes the encoder for classification or the encoder-decoder for segmentation.
- Learning appearance: Non-linear intensity transformations preserve relative anatomical intensity information while training restoration of distorted patches.
- Learning texture: Local pixel shuffling requires recovery of local boundaries and texture without changing the image's global content.
- Learning context: Out-painting and in-painting alter contextual regions by replacing pixels outside or inside generated windows, respectively.
- Autodidactic: Models Genesis learn from scratch on unlabeled images, requiring no manual expert annotation.
- Eclectic: The unified approach combines multiple perspectives in one restoration task, producing more comprehensive representations than individual schemes.
- Scalable: Sharing one encoder and decoder removes proxy-task-specific heads, allowing favorable transformations to be added without the memory burden of separate task decoders.
- Generic: Models Genesis initialize encoders for classification and encoder-decoders for segmentation across diseases, organs, and modalities.
3 Experiments and Results
Experiments evaluate Models Genesis across 3D and 2D medical-imaging tasks, showing strong transfer from self-supervised medical-image pretraining. Genesis Chest CT improves over scratch training and outperforms evaluated 2D alternatives, while the unified framework remains robust across tasks.
- 3D transfer learning: Genesis Chest CT improves same-domain performance by 4 AUC points for NCC and 3 IoU points for NCS versus training from scratch.For ECC, it also increases AUC by 8 points despite differences in disease and dataset.
- Self-supervised framework: The unified self-supervised framework is more robust across five 3D tasks, producing either the best result or performance comparable to the best individual scheme.The comparison reports no clear single winner among individual schemes.
- Generalization: Genesis Chest CT generalizes across diseases, organs, datasets, and modalities despite pretraining on chest CT.The experiments deliberately reserve images to avoid test-image leakage between proxy and target tasks.
- 3D versus 2D: Genesis Chest CT significantly outperforms all evaluated 2D solutions on three 3D target tasks, whereas 3D scratch models do not consistently beat 2D counterparts.The comparison is reported as statistically significant at p < 0.05.
- 2D transfer learning: In 2D tasks, Models Genesis 2D outperform scratch training and offer equivalent performance to ImageNet fine-tuning in same-domain settings.They also show fairly robust cross-domain transfer, though performance tends to decline when domain distance is large.
4 Conclusion and Future Work
Models Genesis are generic source models built directly from unlabeled 3D medical images for transfer learning. The authors report strong empirical performance and plan extensions to additional modalities and organs while releasing the work openly.
- Models Genesis are generic source models built from unlabeled 3D image data to generate application-specific target models through transfer learning.
- The authors report performance surpassing state-of-the-art results in most applications.
- Future work includes modality-oriented models such as Genesis MRI and Genesis Ultrasound and organ-oriented models such as Genesis Brain and Genesis Heart.
- The project releases its code and models publicly and invites broader contributions.
Supplementary Material Models Genesis: Generic Autodidactic Models
The supplementary material documents Models Genesis and its unified self-supervised framework. The framework extracts random patches and learns representations by restoring original patches from transformed ones.
- The supplementary material covers an overview, detailed transformations, qualitative visualizations, and transfer-learning results.
- The framework extracts patches of arbitrary sizes from random locations before applying transformations.
- Models Genesis learns visual representations by restoring original patches X from transformed patches ˜X.
A Models Genesis
Models Genesis uses a unified self-supervised framework that transforms image patches and trains an encoder-decoder to restore the originals. The resulting representations are intended to transfer across diseases, organs, and modalities, with reported advantages over scratch training and 2D approaches.
- The framework uses four individual transformations: non-linear, local-shuffling, out-painting, and in-painting.
- The framework combines image transformation with image restoration, training an encoder-decoder to recover original patch X from transformed patch ˜X.
- Models Genesis aims to learn representations transferable across diseases, organs, and modalities.
- Models Genesis significantly outperform learning from scratch in all five target 3D applications covering segmentation and classification.
- Models Genesis consistently top 2D approaches, including fine-tuned ImageNet models and 2D Models Genesis, highlighting the importance of 3D anatomical information.
B Non-linear Intensity Transformation Visualization
The non-linear intensity transformation scheme modifies medical-image intensity relationships for self-supervised restoration. It uses a Bézier curve controlled by four points and applies modality-specific preprocessing before transformation.
- Non-linear intensity transformation is introduced so models learn absolute or relative appearance of medical-image structures.
- A Bézier curve uses four control points P0–P3 to modify the transformation function's shape.
- The Bézier transformation is defined from two endpoints and two control points using B(t) over t ∈ [0,1].
- For CT images, Hounsfield Units are clipped to [-1000,1000] and normalized to [0,1] per slice, whereas X-rays are directly normalized to [0,1].
C Local Pixel Shuffling Visualization
Local pixel shuffling enriches patch-level variation while preserving enough global structure for Models Genesis to learn anatomical shape, boundaries, layout, and texture.
- C Local Pixel Shuffling Visualization: Local pixel shuffling randomly selects windows from each input patch and sequentially shuffles the pixels inside them.The method selects 1,000 windows per patch.
- C Local Pixel Shuffling Visualization: The transformation encourages learning local boundaries and texture together with object shapes and the relative layout of object parts.The visualization applies multiple window sizes to CT and X-ray images.
- C Local Pixel Shuffling Visualization: The transformed window is produced by independently permuting its rows and columns with permutation matrices.Pre-multiplication permutes rows, while post-multiplication permutes columns.
- C Local Pixel Shuffling Visualization: Window sizes remain smaller than the network’s receptive field and control the degree of distortion.This preserves global image content while varying the transformation difficulty.
D Out-painting Visualization
Out-painting masks the exterior of a complex window while exposing its interior, training the network to extrapolate global organ geometry and spatial layout.
- D Out-painting Visualization: Out-painting generates multiple windows with varied sizes and aspect ratios, superimposing them into one complex-shaped window.The framework allows an arbitrary number of windows, with the method description specifying ď 10.
- D Out-painting Visualization: The transformed patch retains original pixels inside the merged window and assigns random values outside it.This creates an extrapolation-based restoration task.
- D Out-painting Visualization: Out-painting is designed to teach global geometry and the spatial layout of organs in medical images.The visualization shows varied window configurations on CT and X-ray images.
E In-painting Visualization
The restoration visualizations compare individual and unified self-supervised transformations across CT, X-ray, datasets, organs, and modalities.
- E In-painting Visualization: The unified framework significantly outperforms individual training schemes on CT restoration across seven combined transformations.This result is presented as evidence for the framework and pre-trained Models Genesis.
- E In-painting Visualization: Genesis Chest CT can directly restore images from CT, X-ray, ultrasound, and MRI datasets despite being trained only on CT data.The qualitative assessment examines restoration across datasets, organs, and modalities.
- E In-painting Visualization: Models trained with a single transformation fail to handle other transformations, whereas the unified framework restores combined transformations more completely.The comparison uses identity, individual, and seven combined transformations.
- E In-painting Visualization: Genesis Chest X-ray is similarly evaluated for restoration across X-ray, CT, ultrasound, and MRI datasets.The figure notes that restoration and target-task performance across modalities may be positively correlated.
H Models Genesis vs. Models ImageNet
Models Genesis are compared with ImageNet-pre-trained models and scratch training in 2D and 3D settings using performance and stability indicators.
- H Models Genesis vs. Models ImageNet: Genesis Chest CT achieves 98.20 ± 0.51% for NCC, 77.62 ± 0.64% for NCS, and 88.04 ± 1.40% for ECC in 3D.These values are compared with 2D scratch and ImageNet results in the reported table.
- H Models Genesis vs. Models ImageNet: Shorter vertical error bars indicate more consistent and stable fine-tuning, while square and diamond markers distinguish 3D volumes from 2D slices.The figure uses 95% confidence intervals for training from scratch and fine-tuning.
- H Models Genesis vs. Models ImageNet: 3D training from scratch does not necessarily outperform 2D counterparts, while Genesis Chest CT fine-tuning significantly outperforms slice-based 2D models.The reported significance threshold is p < 0.05.
- H Models Genesis vs. Models ImageNet: The unified framework is more robust across target tasks, producing either the best result or performance comparable to the best model.The paper attributes this to consolidating multiple self-supervised training schemes.
I The NiftyNet Transfer Learning Capability
Fine-tuning NiftyNet’s supervised 3D pre-trained weights provides no statistically significant benefit over training from scratch on three target segmentation tasks. Models Genesis instead show stronger performance across these tasks.
- 75.86±0.90 Dice on NCS, 91.13±1.51 on LCS, and 92.58±0.30 on BMS were achieved by Models Genesis.NiftyNet scratch scored 69.65±2.56, 91.09±0.76, and 90.68±0.24, respectively.
- The p-values compare NiftyNet trained from scratch with the NiftyNet model-zoo initialization.The reported p-values were 0.3433 for NCS, 0.2214 for LCS, and 0.4301 for BMS.
- Fine-tuning NiftyNet’s supervised pre-trained weights had no benefit over random initialization across the three segmentation tasks.The comparison used Dice coefficient and reported p-values greater than 0.05.
- Learning curves further corroborated that fine-tuning pre-trained NiftyNet did not outperform training from scratch.