PaperlayerView exact revision on arXiv

Source-linked AI summary

SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound

Christian F. Baumgartner, Konstantinos Kamnitsas, Jacqueline Matthew, Tara P. Fletcher, Sandra Smith, Lisa M. Koch, Bernhard Kainz, Daniel Rueckert

arXiv:1612.05601v2cs.CV

TL;DR

The paper addresses the difficulty of detecting fetal standard scan planes and relevant anatomy in freehand 2D ultrasound. It presents a real-time convolutional framework for detecting 13 views and localising their structures using image-level labels, with strong detection, retrieval, and localisation results on challenging clinical data.

  • Problem

    Detecting fetal standard scan planes and relevant anatomy is difficult for non-experts, while 2D ultrasound is affected by poor signal-to-noise ratio, artefacts, and unfavourable fetal pose.

  • Method

    SonoNet uses a deep convolutional network to detect 13 fetal standard views and localise associated structures through category-specific saliency maps and bounding boxes, trained with image-level labels only.

  • Results

    The method achieved real-time detection, retrospective retrieval, and robust localisation of fetal standard views on a challenging clinical dataset.

  • Takeaways & Limitations

    Weakly supervised saliency-based localisation can support real-time annotation and localisation of fetal standard views without requiring bounding-box annotations during training.

  • Takeaways & Limitations

    Localisation errors mainly arose from inaccurate bounding boxes because saliency maps sometimes focused on only the most important feature or included regions outside the object.

Abstract

from arXiv · show

Identifying and interpreting fetal standard scan planes during 2D ultrasound mid-pregnancy examinations are highly complex tasks which require years of training. Apart from guiding the probe to the correct location, it can be equally difficult for a non-expert to identify relevant structures within the image. Automatic image processing can provide tools to help experienced as well as inexperienced operators with these tasks. In this paper, we propose a novel method based on convolutional neural networks which can automatically detect 13 fetal standard views in freehand 2D ultrasound data as well as provide a localisation of the fetal structures via a bounding box. An important contribution is that the network learns to localise the target anatomy using weak supervision based on image-level labels only. The network architecture is designed to operate in real-time while providing optimal output for the localisation task. We present results for real-time annotation, retrospective frame retrieval from saved videos, and localisation on a very large and challenging dataset consisting of images and video recordings of full clinical anomaly screenings. We found that the proposed method achieved an average F1-score of 0.798 in a realistic classification experiment modelling real-time detection, and obtained a 90.09% accuracy for retrospective frame retrieval. Moreover, an accuracy of 77.8% was achieved on the localisation task.

I. INTRODUCTION

Mid-pregnancy fetal ultrasound is clinically valuable but difficult to acquire and interpret consistently, motivating automated assistance. SonoNet addresses this by detecting 13 standard views and localising their fetal structures from image-level labels, with potential applications in real-time feedback and retrospective analysis.

  • 2D ultrasound is widely used because it is inexpensive, available, real-time, and radiation-free, but image quality is limited by noise, artefacts, and fetal pose.
  • Standard scan planes support biometric measurements and abnormality identification during routine mid-pregnancy examinations.
  • Guiding the probe and identifying relevant structures require extensive training and suffer from low reproducibility and operator bias.
  • SonoNet detects 13 fetal standard scan planes and localises associated structures with bounding boxes using only image-level scan-plane labels during training.
  • The system can provide real-time operator feedback, retrieve standard views retrospectively, support non-experts, and serve as preprocessing for automated measurement or segmentation.
  • The study extends preliminary work with category-specific saliency maps, a deeper architecture, and broader quantitative and qualitative evaluation.

B. Related work

Prior fetal ultrasound methods largely targeted anatomy detection, standard-plane categorisation, or a small number of planes in short sweeps. This work instead targets real-time detection and weakly supervised localisation across lengthy, unconstrained freehand clinical examinations.

  • Earlier video methods detected structures such as the skull, heart, or abdomen rather than specific standardised scan planes.
  • Standard-plane categorisation assumes every image is a standard plane, whereas detection must reject abundant background frames in real-time streams or videos.
  • Earlier automated detection methods covered only 1–3 standard planes in short videos and often used Haar-like features with AdaBoost or random forests.
  • A related CNN detected the abdominal view by repeatedly classifying overlapping patches, preventing real-time operation.
  • Unlike prior sweep-based studies, this work uses freehand clinical screenings lasting up to 30 minutes and containing over 20,000 frames per case.
  • The authors position the study as the first to perform fetal-ultrasound localisation with weak supervision and note that real-time weakly supervised localisation was not previously established.

II. METHODS

The study uses a large dataset of labelled fetal ultrasound examinations, standard-view images, and full-exam videos to evaluate detection under realistic conditions.

  • 2694 examinations from 45 expert sonographers covered gestational ages of 18–22 weeks and followed UK FASP guidelines.
  • The dataset included 13 standard views, four common cardiac views, and a facial profile view, with an additional “other” category.
  • 27731 standard-view images and 6856 “other” images were collected, with class counts ranging from 543 to 4868 examples.
  • 2638 videos averaged over 13 minutes and more than 20000 frames, with 2438 corresponding to cases that also had image data.
  • Freeze-frame images were matched to corresponding video frames to augment training and bridge the image–video domain gap.
  • All evaluations used video data containing motion and many irrelevant background frames to test realistic detection.

B. Preprocessing

Preprocessing standardised the ultrasound inputs, removed confounding overlays and annotations, and constructed a highly varied background class for realistic detection.

  • Colour Doppler overlays and split views were removed to retain structural images showing single fetal locations.
  • Sonographer annotations were removed with inpainting so the algorithm could not learn the annotations instead of image content.
  • Images were cropped to 224x288 regions, excluding vendor controls, then normalised using mean intensity and pixel standard deviation.
  • Background images were sampled from videos because the “other” classes did not capture the highly varied non-standard content.
  • The test-set standard-view-to-background ratio ranged from 1:138 to 1:1148, creating substantial intentional class imbalance.
  • Cases were split 80% for training and 20% for testing at the case level, preventing test-video frames from entering training.

C. Network architecture

SonoNet adapts VGG16 into a fully convolutional architecture that preserves spatial correspondence for localisation while supporting efficient inference on rectangular images.

  • SonoNet is inspired by VGG16 but introduces architectural changes for real-time detection and localisation.
  • Fully convolutional networks process arbitrary image sizes more efficiently in a single forward pass than overlapping-patch evaluation.
  • The architecture removes VGG16’s final max-pooling layer and replaces fully connected layers with two 1x1 adaptation convolutions.
  • For 224x288 inputs, the network produces 14x18 class score maps for 14 classes, preserving receptive-field correspondence with the input image.
  • Mean aggregation uses the entire image context for classification, whereas max aggregation considers only the maximally activated receptive field.
  • SonoNet-64, SonoNet-32, and SonoNet-16 vary kernel counts to examine the trade-off between inference speed and detection accuracy.
  • Batch normalisation was added to convolutional layers and produced substantially better results across the examined networks.

D. Training

Training combines optimisation, augmentation, balanced minibatch sampling, and image–video alignment to support classification and retrospective retrieval.

  • Networks used Nesterov momentum of 0.9, categorical cross-entropy, and an initial learning rate of 0.1, with reductions when validation error stopped decreasing.
  • A warm-up learning rate of 0.01 for 500 iterations was used when the initial rate was too aggressive for convergence.
  • Training combined annotated images with corresponding video frames to bridge their small domain gap, using 20% of the combined data for validation.
  • Scale augmentation sampled 174x174 to 224x224 patches, resized them to 224x224, and applied random horizontal flips and rotations from −25° to 25°.
  • Class imbalance was addressed by randomly sampling minibatches, integrating the sampling strategy with mini-batch gradient descent.
  • Optimisation typically converged after around 2 days on an Nvidia GeForce GTX 1080 GPU.
  • For detection, 14x18 class score maps were mean-pooled, converted to class confidences by softmax, and assigned by highest confidence.
  • For retrieval, the system recorded each class confidence across the video and selected the frame with the highest confidence for each class.

F. Weakly supervised localisation

The method uses category-specific saliency maps to localise fetal anatomy from image-level labels, exploiting spatial information in class score maps and a backward pass. The resulting maps are less noisy and easier to use for localisation than the compared guided backpropagation approach.

  • Saliency-map computation: The network obtains saliency and confidence maps through an additional backward pass after determining class score maps and the image category.Bounding boxes are then extracted from post-processed confidence maps.
  • Saliency-map computation: Category-specific saliency maps measure how strongly each input pixel influences the current prediction.They are computed as the gradient of the pre-softmax activation for the predicted category with respect to the input image.
  • Comparison with guided backpropagation: Guided backpropagation can highlight fetal anatomy but also background features, adversely affecting automated localisation.It back-propagates errors through ReLU units only when both the neuron input and higher-layer error are positive.
  • Proposed localisation method: The proposed method weights receptive-field influences by class-score activations, prioritising regions associated with highly activated neurons without heuristic percentile selection.Each class-score-map neuron corresponds to a receptive field in the original image, and negative activations are excluded.
  • Comparison with guided backpropagation: The proposed saliency maps are significantly less noisy, making fetal structures easier to localise than with the approach of Springenberg et al.Figure 3 compares input frames, class score maps, guided-backpropagation saliency maps, and the proposed saliency maps.

2) Bounding box extraction:

Saliency maps are converted into confidence maps and then bounding boxes through smoothing, thresholding, connected-component selection, and rectangular fitting. The procedure is illustrated on challenging views containing shadows and similar-looking structures.

  • Confidence-map construction: The method converts saliency maps into confidence maps by taking their absolute values and applying a 5x5 Gaussian blur.The resulting confidence maps represent the estimated location of the target structure.
  • Domain-specific processing: Domain-specific sign selection retains positive saliencies for the femur, spine, and lips, but negative saliencies for cardiac views.This uses the typical gradient signs associated with bright bones, lips, spine, femur, and dark cardiac vessels.
  • Bounding-box extraction: The confidence maps are thresholded with Isodata, reduced to their largest connected component, and enclosed by a minimum rectangular bounding box.The resulting bounding boxes are shown in Figure 4c.

III. EXPERIMENTS AND RESULTS

The experiments evaluate real-time detection under a highly imbalanced clinical-like stream, compare network architectures, and examine classification errors and label quality.

  • Real-time scan plane detection: Precision, recall, and F1-score were measured on freeze-frames containing many randomly sampled background frames.F1-score is the harmonic mean of precision and recall; averages were unweighted so the background class could not dominate.
  • Real-time scan plane detection: SonoNet-32 provided the best balance among architectures achieving real-time performance for both detection and localisation.SonoNet-64 had slightly better F1-scores but failed to perform localisation at more than 25 fps, whereas smaller networks showed greater accuracy deterioration.
  • Real-time scan plane detection: The realistic evaluation used approximately one standard-plane frame for every 24 background frames, reflecting clinical video conditions.Detailed SonoNet-32 classification scores and a class confusion matrix were reported for all modelled categories.
  • Real-time scan plane detection: Brain, abdominal, and femur views reached F1-scores of at least 0.9, while profile, RVOT, and 3VV views had the lowest detection accuracies.The two cardiac views were often confused, and some high-recall, low-precision predictions were actually correct detections with erroneous background labels.
  • Real-time scan plane detection: Many apparent false positives were correct predictions mislabeled as background, making the reported precision and F1-scores lower bounds on true performance.Examples included profile, sagittal spine, and Brain (Cb.) views; exhaustive manual verification was infeasible.

B. Retrospective scan plane retrieval

SonoNet-32 was evaluated for retrieving standard scan planes retrospectively from long clinical videos, with expert validation and qualitative comparison against saved sonographer frames.

  • Retrospective scan plane retrieval: Retrospective retrieval was evaluated on 110 random test-cohort videos averaging 13 min 33 sec and 20,321 frames.Retrieved frames were manually validated by two obstetric clinical experts with 11 and 3 years of experience.
  • Retrospective scan plane retrieval: The manually validated retrieval results were obtained directly from full videos and therefore reflected performance in a real scenario.The time-consuming validation procedure prevented evaluation on a larger number of videos.
  • Retrospective scan plane retrieval: Retrieved frames generally matched the saved ground-truth frames or had equivalent quality, although occasional wrong retrievals occurred.Examples showed complete retrieval success for one subject and several incorrect retrievals for another.

C. Weakly supervised localisation

Weakly supervised localisation was tested against manually annotated bounding boxes across all 13 standard planes, combining quantitative IOU evaluation with qualitative inspection.

  • Weakly supervised localisation: Localisation was quantitatively evaluated on 650 images, comprising 50 manually annotated images from each of 13 standard scan planes.Intersection over union measured similarity between automatically estimated and ground-truth bounding boxes.
  • Weakly supervised localisation: The method localised views across substantial scale and appearance variation, including small structures such as lips and femur.Qualitatively strong localisation was not always reflected quantitatively because IOU is more sensitive to small deviations in small boxes.
  • Weakly supervised localisation: The method was relatively robust to artefacts and correctly localised difficult examples including lips and RVOT views.These examples included anatomy that may be difficult for non-experts to localise.
  • Weakly supervised localisation: Localisation errors mainly involved inaccurate box boundaries rather than complete failure to identify the view.Saliency maps could focus on only part of the target or include nearby structures, such as both femurs.

IV. DISCUSSION AND CONCLUSION

The paper presents a real-time framework for detecting and localising 13 fetal standard views using weak supervision, and reports strong performance on challenging clinical data.

  • Discussion and conclusion: SonoNet detects standard views and localises fetal structures with bounding boxes in freehand fetal ultrasound, without bounding-box annotations during training.The architecture is based on VGG16 and uses category-specific saliency maps while being optimised for real-time operation.
  • Discussion and conclusion: The network achieved excellent results for real-time frame annotation and retrospective retrieval on a very challenging dataset.The conclusion covers both real-time annotation and retrieval rather than a single evaluation setting.
  • Discussion and conclusion: Future work will incorporate temporal information into training and prediction, particularly to benefit cardiac-view detection.The paper identifies temporal modelling as a direction for extending the current framework.
  • Discussion and conclusion: Current localisation relies purely on confidence maps, while incorporating original-image intensities or multi-instance learning may improve bounding-box prediction.The confidence maps may also serve as a data term for graphical-model semantic segmentation.
Loading 1612.05601v2…