PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Weakly and Semi-Supervised Deep Learning for Localization and Classification of Masses in Breast Ultrasound Images

Seung Yeon Shin, Soochahn Lee, Il Dong Yun, Sun Mi Kim, Kyoung Mu Lee

arXiv:1710.03778v2cs.CV

TL;DR

Breast-ultrasound mass detection must balance weak labels, which are difficult to use effectively, against strong labels, whose scarcity can cause overfitting. The paper combines both annotation types in a weakly and semi-supervised CNN framework for joint localization and classification, achieving comparable localization with far fewer strong annotations and improving CorLoc when weak data are added.

  • Problem

    Large weakly annotated datasets are difficult to use for mass detection, while small strongly annotated datasets may lead to overfitting.

  • Method

    A CNN jointly localizes and classifies BUS masses using strongly annotated DXLoc images, weakly annotated DX images, shared network streams, and selected training losses.

  • Results

    The method successfully localizes and classifies masses with less annotation effort, with experiments comparing annotation amounts and training variants.

  • Takeaways & Limitations

    Combining weak and strong annotations provides a framework for reducing annotation requirements while maintaining effective BUS mass localization and classification.

Abstract

from arXiv · show

We propose a framework for localization and classification of masses in breast ultrasound (BUS) images. We have experimentally found that training convolutional neural network based mass detectors with large, weakly annotated datasets presents a non-trivial problem, while overfitting may occur with those trained with small, strongly annotated datasets. To overcome these problems, we use a weakly annotated dataset together with a smaller strongly annotated dataset in a hybrid manner. We propose a systematic weakly and semi-supervised training scenario with appropriate training loss selection. Experimental results show that the proposed method can successfully localize and classify masses with less annotation effort. The results trained with only 10 strongly annotated images along with weakly annotated images were comparable to results trained from 800 strongly annotated images, with the 95% confidence interval of difference -3.00%--5.00%, in terms of the correct localization (CorLoc) measure, which is the ratio of images with intersection over union with ground truth higher than 0.5. With the same number of strongly annotated images, additional weakly annotated images can be incorporated to give a 4.5% point increase in CorLoc, from 80.00% to 84.50% (with 95% confidence intervals 76.00%--83.75% and 81.00%--88.00%). The effects of different algorithmic details and varied amount of data are presented through ablative analysis.

I. INTRODUCTION

The paper targets simultaneous mass localization and classification in breast ultrasound while reducing reliance on costly strong annotations. It combines weakly annotated images with a smaller strongly annotated set through a shared CNN and supervision-specific losses.

  • I. INTRODUCTION: Strongly supervised CNN detectors may overfit on small datasets, while weakly annotated datasets may remain insufficient for training regardless of size.The motivation is to address annotation cost and the limitations of either supervision level alone.
  • I. INTRODUCTION: The framework builds on weakly supervised learning and mixed-supervision methods while adapting them to concurrent localization and classification of breast ultrasound masses.Prior work includes multiple-instance learning and methods combining weakly and strongly labeled images.
  • I. INTRODUCTION: The proposed framework jointly localizes and classifies breast ultrasound masses using weakly and strongly annotated datasets in a hybrid training scenario.Strong annotations contain mass bounding boxes and diagnostic labels, whereas weak annotations contain image-level diagnostic labels.
  • I. INTRODUCTION: The approach is designed to improve accuracy while reducing annotation cost in settings where physicians can provide strong annotations for only part of the dataset.The paper presents this as a typical medical-imaging setting with limited annotation resources.
  • I. INTRODUCTION: Two data streams feed a shared network: DX images use a MIL-based image-level loss, while DXLoc images use losses derived from mass-level ground-truth labels.The framework is illustrated with Faster R-CNN, although other supervised approaches can also be adopted.

A. Datasets

The study uses two BUS datasets distinguished by annotation strength, with DXLoc providing bounding boxes and DX providing image-level labels; their image and mass-area statistics differ.

  • A. Datasets: DXLoc images include diagnostic labels and mass bounding boxes, whereas DX images provide only image-level diagnostic labels.The datasets are split into training and test subsets by patient, with SNUBH-DX used only for training.
  • A. Datasets: SNUBH contains thousands of images from multiple ultrasound vendors, while UDIAT contains 163 images from different patients acquired with a Siemens system.SNUBH includes pathologically proven benignancy or malignancy labels for images containing masses.
  • A. Datasets: The datasets differ in whole-image size, mass bounding-box size, and the ratio between mass area and whole-image area.These comparisons are summarized in Table I and visualized in Fig. 3.

B. Strongly Supervised Learning Using the DXLoc Subset

The strongly supervised baseline jointly localizes and classifies breast masses with Faster R-CNN using DXLoc images and bounding-box annotations.

  • B. Strongly Supervised Learning Using the DXLoc Subset: Faster R-CNN jointly performs mass localization and classification through a region proposal network and a Fast R-CNN detector.The two components share convolutional layers for region proposal generation and detection.
  • B. Strongly Supervised Learning Using the DXLoc Subset: The training objective combines classification and regression losses from both the region proposal network and the Fast R-CNN detector.The losses compare network outputs with ground-truth boxes defined by overlap relationships.
  • B. Strongly Supervised Learning Using the DXLoc Subset: Positive proposals require IoU above 0.7 with a ground-truth mass box, while negative-proposal assignment differs between SNUBH and UDIAT.The UDIAT classifier additionally uses class weights to address dataset imbalance.
  • B. Strongly Supervised Learning Using the DXLoc Subset: Regression losses are applied only to positive proposals and to detected benign or malignant boxes, rather than background proposals.These rules adapt the detector to mass localization and benign-versus-malignant classification.
  • B. Strongly Supervised Learning Using the DXLoc Subset: Bounding-box similarities in both regression losses are measured against the ground-truth boxes with the highest IoU.

C. Weakly Supervised Learning Using the DX Subset

The DX subset supplies image-level labels without region-level ground truth, so the method uses multiple-instance learning to train region classification from detected mass regions. Image-level predictions are inherited from a selected mass-of-interest region.

  • Multiple-instance learning treats detected mass regions as instances in an image-level bag, using the bag label to define weak supervision.A bag is positive when at least one instance is positive and negative when all instances are negative.
  • The weakly supervised loss is a class-weighted cross entropy between each image’s ground-truth label and its predicted image-level label.Class weights address class imbalance in the DX set.
  • Image-level labels distinguish normal, benign, and malignant cases, while the prediction is inherited from the selected mass-of-interest region.
  • For benign images, four mass-of-interest criteria are tested; for malignant images, the most malignant region is always selected.
  • The region-selection definitions assume that a clinician focuses on one mass-of-interest region in each image.

D. Joint Weakly and Semi-Supervised Learning Using the DXLoc and DX Subsets

The framework jointly streams strongly annotated DXLoc images and weakly annotated DX images through a shared CNN. It combines their losses during training while restricting Fast R-CNN bounding-box regression to strongly annotated data.

  • Strongly and weakly supervised components are combined to train the shared convolutional, RPN, and Fast R-CNN parameters.Fast R-CNN bounding-box regression is trained only with DXLoc data.
  • Two training variants use either combined mini-batches or alternating strongly and weakly supervised mini-batches.
  • The combined-mini-batch variant sums strong and weak losses as L = Ls + αLws, while α scales the MIL loss.
  • In the alternating variant, strongly supervised updates train the full parameter set, whereas weakly supervised updates exclude Fast R-CNN bounding-box regression parameters.

III. RESULTS

The study evaluates localization and classification using CorLoc and FROC, with statistical uncertainty assessed through bootstrapped confidence intervals and paired tests. It also examines algorithmic variants through an SNUBH ablation study.

  • Table II presents an ablation study of algorithmic variants of the proposed method on the SNUBH dataset.
  • CorLoc measures the percentage of images with a correctly localized target object under the PASCAL criterion, requiring class probability and IoU both above 0.5.FROC is generated by varying the final class-probability threshold from 0 to 1.
  • Evaluations use the SNUBH-DXLoc-Ts or UDIAT-DXLoc-Ts test sets with CorLoc and FROC measures.
  • The analysis reports 95% confidence intervals and paired-test p-values computed by bootstrapping test images with replacement.

B. Experiments on the SNUBH Dataset

Experiments on SNUBH examine algorithmic choices, backbone depth, supervision-data allocation, and self-training for the proposed joint weakly and semi-supervised method. The analyses show benefits from gradually increasing MIL-loss weighting and combined mini-batches, while limited data constrains very deep residual networks.

  • a) The Effect of Algorithmic Details:: The gradually increasing MIL-loss scale factor improved performance by 2.25%, while combined DXLoc and DX mini-batches improved performance by 2%.The scale factor helps avoid undesired local minima, and combined mini-batches jointly probe weakly and strongly supervised data during updates.
  • b) The Effect of Network Structures:: Increasing residual-network depth improved strongly supervised performance with 800 images, but reduced performance with 600 or 400 images.The authors conclude that the available limited data were insufficient to realize the discriminative power of very deep residual networks.
  • c) The Effect of Dataset Sizes:: Using 10 strongly supervised images with 5014 weakly supervised images produced better or comparable results to training with the same total images only from strongly supervised data.The tested allocations included 800(+4224), 600(+4424), 400(+4624), 200(+4824), 50(+4974), and 10(+5014), with comparisons shown in Fig. 4.
  • d) The Effect of Using Self-training:: The proposed method was further evaluated with self-training by selecting high-confidence DX detections for movement into the strongly supervised training set.The initial network predicts detections on DX images, and selected images are used to retrain the network.
  • d) The Effect of Using Self-training:: Reclassifying an excessive proportion of the weakly supervised set may reduce the performance increase obtained through self-training.The reported retraining results used reclassification of 75% of the DX set, and the outcome was noted as dependent on the initial network model's quality.

2) Comparison with Previous Methods:

On the SNUBH dataset, the proposed joint weakly and semi-supervised method outperformed comparison methods while using weak and strong annotations together, improving localization with a trade-off in false positives.

  • The proposed joint weakly and semi-supervised approach clearly outperformed other methods with statistical significance on the SNUBH dataset.
  • 10 strongly and 5014 weakly supervised images achieved a slightly higher CorLoc than a method trained on 800 strongly supervised images, with a 95% CI of differences of -3.00%–5.00%.
  • Adding weakly supervised data helped detect more difficult masses, but increased false positives relative to training only on strongly supervised data.
  • On normal images, the proposed method produced 0.25 average false-positive detections per image, a 19.0% increase over, alongside a 5.63% CorLoc increase.
  • Qualitative results showed more precise detection and classification, while unclear mass boundaries caused failures across methods.

C. Experiments on the UDIAT Dataset

The study constructs a cascaded comparison system by combining separate localization and classification methods, each trained on strongly annotated UDIAT data.

  • The cascaded comparison system first localizes masses and then classifies them, with each constituent network trained separately using UDIAT-DXLoc-Tr.

1) Comparison with Previous Methods:

Across datasets, the proposed joint weakly and semi-supervised method outperformed comparison methods and remained effective when trained with weak data from the target environment.

  • The proposed approach clearly outperformed the cascaded system and other methods on UDIAT-DXLoc-Ts.
  • Using SNUBH-DX weak annotations with the small UDIAT-DXLoc-Tr strong dataset greatly boosted performance on SNUBH-DXLoc-Ts, where other methods almost failed.
  • The proposed training detected more difficult masses at the expense of increased false positives.
  • Qualitative results showed more precise localization and classification of both benign and malignant masses than the fully supervised methods.

IV. CONCLUSION

The paper concludes that joint weakly and semi-supervised training can localize and classify breast-ultrasound masses using few strong annotations and more weak annotations. Its findings support reduced annotation effort, while future work targets 3D automated ultrasound and broader applications.

  • The proposed algorithm jointly localizes and classifies breast-ultrasound masses using a small strongly annotated dataset and a larger weakly annotated dataset.
  • Self-training with under 100 strongly annotated images can achieve comparable performance to using many more strong annotations, while adding weak data improves true-positive detection.
  • The framework is intended for extension to 3D automated breast ultrasound, which offers higher reproducibility and less physician acquisition time but makes interpretation more time-consuming.
  • The authors suggest applying the framework to other image modalities, organs, and diseases for computer-aided detection and diagnosis.
Loading 1710.03778v2…