PaperlayerView exact revision on arXiv

Source-linked AI summary

Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning

Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Zongwei Zhou, Michael B. Gotway, Jianming Liang

arXiv:2102.10680v1cs.CVeess.IV

TL;DR

Medical imaging contains recurring anatomical patterns associated with human-anatomy semantics, but CNNs need annotated images while conventional bags of visual words lack transferability. TransVW harvests anatomical visual words from unlabeled images and uses self-classification and self-restoration for representation learning. The authors report higher performance, faster convergence, and lower annotation cost across applications, and say visual-word learning can boost existing self-supervised methods.

  • Problem

    CNN features transfer across applications but require many annotated images, whereas bags of visual words need no expert annotation but lack transferability; the paper also asks how to use semantics in recurring anatomical patterns for representation learning.

    Source: §I · p. 1

  • Method

    TransVW harvests semantic pseudo-labeled anatomical visual words from unlabeled medical images and trains representations through self-classification and self-restoration.

    Source: §I · p. 2, A. TransVW learns semantics-enriched… · p. 3

  • Results

    The authors report higher performance, faster convergence, and reduced annotation cost, and find that adding visual words consistently improves performance across five 3D target tasks.

    Source: §I · p. 2, A. TransVW is an add-on scheme · p. 6, §VII · p. 11

  • Takeaways & Limitations

    Visual-word learning can complement existing self-supervised methods and improve their performance, according to the authors.

    Source: §I · p. 2

Abstract

from arXiv · show

This paper introduces a new concept called "transferable visual words" (TransVW), aiming to achieve annotation efficiency for deep learning in medical image analysis. Medical imaging--focusing on particular parts of the body for defined clinical purposes--generates images of great similarity in anatomy across patients and yields sophisticated anatomical patterns across images, which are associated with rich semantics about human anatomy and which are natural visual words. We show that these visual words can be automatically harvested according to anatomical consistency via self-discovery, and that the self-discovered visual words can serve as strong yet free supervision signals for deep models to learn semantics-enriched generic image representation via self-supervision (self-classification and self-restoration). Our extensive experiments demonstrate the annotation efficiency of TransVW by offering higher performance and faster convergence with reduced annotation cost in several applications. Our TransVW has several important advantages, including (1) TransVW is a fully autodidactic scheme, which exploits the semantics of visual words for self-supervised learning, requiring no expert annotation; (2) visual word learning is an add-on strategy, which complements existing self-supervised methods, boosting their performance; and (3) the learned image representation is semantics-enriched models, which have proven to be more robust and generalizable, saving annotation efforts for a variety of applications through transfer learning. Our code, pre-trained models, and curated visual words are available at https://github.com/JLiangLab/TransVW.

I. INTRODUCTION

TransVW treats recurring anatomical patterns in medical images as semantic visual words that can provide supervision without expert annotations. It combines their discovery with self-supervised learning to produce transferable representations, reporting higher performance, faster convergence, and reduced annotation cost.

I. INTRODUCTION

  • Recurring anatomical patterns across medical images are treated as visual words associated with human-anatomy semantics.The paper motivates this with anatomical similarity across patients and the sophisticated patterns found in clinically focused imaging.Source: §I · pp. 1–2
  • TransVW automatically harvests visual words from unlabeled images and assigns each a semantic pseudo-label for self-supervised representation learning.The introduction identifies unsupervised clustering as the way to curate the visual-word dataset.Source: §I · p. 2
  • The framework combines self-discovery, self-classification, and self-restoration to learn representations from anatomical consistency and diversity.The figure describes the three stages as discovering instances, classifying them, and restoring perturbed visual words.Source: §I · p. 2, Figure 2 · p. 2
  • TransVW is reported to improve performance, accelerate convergence, and reduce annotation cost on applications with few annotated images.The authors also describe visual-word learning as an add-on that complements existing self-supervised methods.Source: §I · p. 2

II. TRANSFERABLE VISUAL WORDS

TransVW defines visual words as recurring segments of consistent anatomy and trains representations using their semantic pseudo-labels and anatomical variation. Its self-discovery, classification, and restoration components jointly encode common patterns and finer details.

II. TRANSFERABLE VISUAL WORDS

  • A visual word is a segment of consistent anatomy recurring across images; its instances are patches extracted across resembling images and share a semantic pseudo-label.The label represents the corresponding anatomical part.Source: §II · p. 3

A. TransVW learns semantics-enriched representation

  • Self-discovery selects similar patient scans, crops patches at a shared random coordinate, and assigns each visual word a unique pseudo-label.Repeating the process at C coordinates yields C visual words, each with K instances.Source: A. TransVW learns semantics-enriched… · p. 3
  • Self-classification predicts visual-word pseudo-labels, pulling same-word instances together while distinguishing instances from different words.The classification branch uses an encoder and classification head, trained with categorical cross-entropy.Source: A. TransVW learns semantics-enriched… · p. 3
  • Self-restoration reconstructs original visual words from perturbed inputs, using variation in organ shape, boundaries, and texture to encode anatomical diversity.Perturbations include non-linear, local-shuffling, out-painting, inpainting, and identity transformations.Source: A. TransVW learns semantics-enriched… · p. 3
  • The framework integrates classification and restoration through a shared encoder and a weighted joint objective.The two weights adjust the classification and restoration losses.Source: A. TransVW learns semantics-enriched… · p. 4

B. TransVW has several unique properties

TransVW uses anatomically meaningful visual words discovered in unlabeled medical images as free supervision for semantics-enriched representation learning. Its properties derive from balancing anatomical consistency with diversity, discouraging superficial solutions, and complementing existing self-supervised methods.

Autodidactic

  • With zero annotation cost, TransVW uses semantic pseudo-labels from discovered anatomical visual words for self-classification and self-restoration, and is reported to outperform other self-supervised methods and publicly available supervised pre-trained models.The cited supervised models include I3D, NiftyNet, and MedicalNet [17].Source: B. TransVW has several unique properties · p. 4

Comprehensive

  • Patient-level nearest-neighbor selection and fixed-coordinate extraction balance consistency and diversity, enabling class-level separation through self-classification and instance-level separation through fine-grained self-restoration.Intensity, shape, boundary, and texture differences provide fine-grained anatomical information for restoration.Source: B. TransVW has several unique properties · p. 4

Robust

  • Substantial within-class diversity, multi-scale jittered instances, and image perturbations discourage easy feature matching and are intended to promote semantic-bearing, robust representations.The paper links this design to concerns that self-supervised pretext tasks can learn shortcuts,.Source: B. TransVW has several unique properties · p. 4

Versatile

  • Visual word learning can be added to existing self-supervised methods, using anatomical similarities across images to encode common semantic structures and boost their performance.The paper points to Fig. 3 as evidence of this add-on effect.Source: B. TransVW has several unique properties · p. 4

III. EXPERIMENTS

The experiments pretrain TransVW on 3D chest CT and 2D chest X-ray images, then evaluate transfer across seven varied medical-imaging applications. Target validation and test images are excluded from pretraining.

A. Pre-training TransVW

  • Target validation and test images are excluded from pretraining to prevent test-case leakage from pretext tasks.Source: A. Pre-training TransVW · p. 4
  • The released models use 623 LUNA16 chest CT scans for 3D pretraining and 76K ChestX-ray14 images for 2D pretraining.Source: A. Pre-training TransVW · p. 4
  • Evaluation spans seven 2D and 3D classification and segmentation applications, including tasks on pretraining-matched and unseen organs, datasets, or modalities.Source: Table I · p. 5
  • Task codes use the first letter for object, the second for modality, and the last for task.Source: A. Pre-training TransVW · p. 5
  • Visual-word instances use multi-scale crops resized to 64×64×32 for chest CT and 224×224 for chest X-rays; K is 200 and 1000, respectively.The paper selects these K values to balance visual-word diversity and consistency.Source: A. Pre-training TransVW · p. 5

B. Fine-tuning TransVW

TransVW is evaluated through transfer learning on diverse 2D and 3D medical-imaging tasks against self-supervised and supervised baselines. The reported results show strong or comparable benchmark performance, faster convergence, and gains when visual-word learning is added to existing self-supervised methods.

B. Fine-tuning TransVW

  • TransVW is fine-tuned on seven diverse classification and segmentation tasks in 2D and 3D, covering varied diseases, organs, modalities, and domain shifts.Source: B. Fine-tuning TransVW · p. 5
  • Target classification uses the encoder with a task-specific head, while segmentation uses the encoder and decoder with a replaced final layer.Source: B. Fine-tuning TransVW · p. 5

C. Benchmarking TransVW

  • The comparison includes training from scratch, five self-supervised transfer baselines, and publicly available supervised models, with selected 2D methods extended to 3D.Source: C. Benchmarking TransVW · p. 5

IV. RESULTS

  • Adding visual-word self-discovery and self-classification improves four self-supervised methods across five 3D target tasks (p < 0.05).Source: Figure 3 · p. 6

A. TransVW is an add-on scheme

TransVW adds cross-image visual-word learning to existing self-supervised objectives and assesses its transfer performance, convergence speed, and label efficiency. The reported experiments find improved performance from the add-on and faster convergence from TransVW initialization.

A. TransVW is an add-on scheme

  • For reconstruction-based methods, added fully connected layers let the encoder learn from classification and restoration together; Rotation uses separate classification heads.Source: A. TransVW is an add-on scheme · p. 6
  • Adding visual words significantly improves four existing self-supervised methods across five 3D target tasks, with gains varying by method and application.Source: A. TransVW is an add-on scheme · p. 6
  • Self-discovery assigns pseudo-labels to recurring anatomical patterns, and self-classification learns similarities among those visual words across images.Source: A. TransVW is an add-on scheme · p. 6

B. TransVW is an annotation-efficient method

  • Fine-tuning from TransVW accelerates training versus scratch and Models Genesis across all five 3D target tasks.Source: Figure 4 · p. 7
  • Annotation efficiency is evaluated as better performance or shorter training at equal label counts, or equal performance with fewer labels.Source: B. TransVW is an annotation-efficient… · p. 7

1) TransVW provides superior transfer learning performance:

TransVW transfers effectively across five 3D target tasks, outperforming training from scratch and multiple pretrained alternatives. Its chest-CT pretraining also benefits tasks across organs, datasets, and modalities, while reducing annotation requirements.

1) TransVW provides superior transfer learning performance:

  • TransVW significantly outperforms training from scratch across all five target tasks and stabilizes overall performance.Source: 1) TransVW provides superior transfer… · p. 7
  • TransVW surpasses all self-supervised counterparts in five target tasks and achieves superior performance to publicly available fully supervised pre-trained 3D models—NiftyNet, MedicalNet, and I3D—in all five tasks.TransVW uses a basic 3D U-Net with 23M parameters; MedicalNet has 85.75M parameters, and I3D's encoder has 25.35M.Source: 1) TransVW provides superior transfer… · p. 7
  • 50%, 50%, 57%, 60%, and 80% less annotation are needed than training from scratch in NCC, NCS, ECC, LCS, and BMS, respectively; savings versus Models Genesis are 17%, 24%, and 50% in NCC, LCS, and BMS.Source: Figure 5 · p. 8
  • Chest-CT pretraining transfers across organs, datasets, and modalities, including liver CT and brain-tumor MRI, and beats Models Genesis in the distant liver and brain tasks.The paper attributes cross-domain benefits to reusable low- and mid-level features.Source: 1) TransVW provides superior transfer… · p. 8

2) TransVW accelerates the training process:

Across five 3D tasks, TransVW delivers better or equivalent performance in less training time than its counterparts. It also reduces annotation needs and improves liver-tumor segmentation when adopted by nnU-Net.

2) TransVW accelerates the training process:

  • TransVW initialization accelerates training in all five 3D target tasks compared with both training from scratch and fine-tuning Models Genesis.The authors interpret faster convergence as evidence that TransVW captures representations more aligned with subsequent target tasks.Source: 2) TransVW accelerates the training… · p. 8
  • TransVW achieves significantly better or equivalent performance with remarkably less training time than its 3D counterparts, including Models Genesis in all five tasks.It outperforms Models Genesis in both performance and training time in NCC, LCS, and BMS, and matches its performance in NCS and ECC.Source: 2) TransVW accelerates the training… · p. 8

3) TransVW reduces the annotation cost:

  • Annotation cost falls versus scratch by 50%, 50%, 57%, 60%, and 80% in NCC, NCS, ECC, LCS, and BMS; versus Models Genesis, it falls 17%, 24%, and 50% in NCC, LCS, and BMS.TransVW and Models Genesis require equivalent data in NCS and ECC to achieve comparable performance.Source: 3) TransVW reduces the annotation cost · p. 9
  • With nnU-Net on Medical Segmentation Decathlon liver-tumor segmentation, accuracy improves 2.5% over training from scratch and 1% over Models Genesis.Source: 3) TransVW reduces the annotation cost · p. 9

V. ABLATION EXPERIMENTS

The ablations compare TransVW’s self-classification and self-restoration components, then evaluate a 2D model on classification and segmentation tasks. The results indicate complementary component benefits and strong transfer performance for TransVW 2D.

V. ABLATION EXPERIMENTS

  • The 2D experiments train TransVW on chest X-rays and assess transfer on DXC, PXS, and slice-based NCC across classification and segmentation.Source: §V · p. 9

A. Comparing individual self-supervised tasks

  • Joint self-classification and self-restoration outperform either isolated task in NCC, ECC, LCC, and BMS, and match self-restoration in NCS.Self-restoration performs significantly better in NCC and BMS, while the two isolated tasks show no significant difference in NCS, ECC, and LCS.Source: A. Comparing individual… · p. 9

B. Evaluating 2D applications

VI. RELATED WORKS

The related-work discussion positions TransVW as a semantic, transferable alternative to prior visual-word and task-specific self-supervised approaches. The conclusion reports annotation-efficient transfer and identifies recurrent anatomical patterns as the source of semantics-enriched representations.

VI. RELATED WORKS

  • Prior BoVW-CNN approaches may yield medically unintuitive words and imbalanced clusters, while TransVW discovers explainable words, bypasses clustering, and uses a different pretext task.The related-work discussion also notes that medical self-supervised methods designed for individual tasks have limited generalization across tasks, whereas TransVW uses visual-word semantic supervision for a generic model.Source: §VI · pp. 10–11
  • The current work extends an earlier anatomical-pattern approach with transferable visual words, add-on self-discovery and self-classification, expanded 3D evaluation, and tests of transfer, convergence, annotation efficiency, linear evaluation, and class-count choices.The authors report that transfer learning from TransVW improves optimization and accelerates training, and that it reduces annotation effort.Source: §VI · p. 11

VII. CONCLUSION

  • TransVW combines learning common image representations with semantics from recurrent anatomical patterns to produce generic, semantics-enriched representations.The authors describe these patterns as naturally embedded in medical images.Source: §VII · p. 11
  • TransVW offers higher performance and faster convergence with reduced annotation cost than publicly available 3D models pretrained with self-supervision or full supervision, and can boost other self-supervised methods.The authors attribute the results to semantics from recurrent visual words arising from consistent anatomy.Source: §VII · p. 11

APPENDIX I SUPPLEMENTAL MATERIALS

The experiments cover seven target tasks across medical-image modalities, including classification and segmentation. In an ablation over five 3D tasks, 45 visual-word classes gave the best performance in every application.

A. Target tasks and datasets

  • The seven target tasks span CT classification and segmentation, including lung-nodule and pulmonary-embolism tasks, and liver segmentation.The task descriptions include lung-nodule false-positive reduction and segmentation, pulmonary-embolism false-positive reduction, and liver segmentation.Source: A. Target tasks and datasets · p. 13
  • The task set also includes brain-tumor MRI segmentation and chest-X-ray disease classification and pneumothorax segmentation.Together, the tasks cover classification and segmentation across CT, MRI, and X-ray modalities.Source: A. Target tasks and datasets · p. 13

B. Impact of number of visual words classes

  • C = 45 achieved the best performance in all five 3D target applications in averages over ten runs.The authors suggest balancing visual-word diversity and overlap to achieve the best transfer-learning performance.Source: B. Impact of number of visual words… · p. 13

C. Visual Words Visualization

TransVW discovers recurring anatomical patterns as visual words, illustrated with word instances and a patient-level neighbor-mining process. In nnU-Net liver-tumor segmentation, TransVW achieved the highest reported five-fold cross-validation Dice score among the compared methods.

C. Visual Words Visualization

E. Incorporating TransVW with nnU-Net framework

  • C = 45 produced the best performance across the five 3D target tasks in the class-count ablation.The figure reports average performance over ten runs for each model and task.Source: Figure 7 · p. 14

D. Visualizing the self-discovery process

  • Patient-level similarity from unsupervised latent features selects neighbors for visual-word extraction, balancing consistency with diversity in pattern appearance.Patches at the same coordinate across patients may differ substantially, motivating neighbor selection before patch extraction.Source: Figure 9 · p. 15

E. Incorporating TransVW with nnU-Net framework

  • 66.06%±0.98% Dice for TransVW exceeded 65.03%±0.70% for Models Genesis and 63.52%±0.28% for training from scratch in five-fold evaluation.The evaluation used the same data augmentation and nnU-Net architecture across methods.Source: E. Incorporating TransVW with nnU-Net… · p. 15

Related papers

Loading 2102.10680v1…