PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography

Sarah Leclerc, Erik Smistad, João Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, Carole Lartizien, Jan D'hooge, Lasse Lovstakken, Olivier Bernard

arXiv:1908.06948v2eess.IV

TL;DR

The paper asks how effectively encoder-decoder CNNs can segment cardiac structures and estimate clinical indices in 2D echocardiography. It introduces CAMUS, a large annotated dataset, and evaluates several architectures, finding accurate segmentation and clinical estimates while identifying unresolved cases.

  • Problem

    Limited large, publicly available, well-annotated echocardiographic datasets constrain evaluation of CNNs for segmentation and clinical-index estimation.

  • Method

    The study introduces CAMUS and evaluates encoder-decoder CNNs, including two U-Net implementations and more elaborate architectures, for 2D echocardiographic segmentation.

  • Results

    Encoder-decoder networks produced highly accurate segmentation, with U-Net offering the best parameter–performance trade-off and robustness to image-quality variability.

  • Takeaways & Limitations

    Encoder-decoder techniques are presented as a solution of choice for 2D echocardiographic image segmentation, while their clinical results remained below inter-observer scores.

  • Takeaways & Limitations

    Outliers comprised 18% of the dataset, including 1.8% of cases whose predicted segmentation could not be assimilated to a heart shape.

Abstract

from arXiv · show

Delineation of the cardiac structures from 2D echocardiographic images is a common clinical task to establish a diagnosis. Over the past decades, the automation of this task has been the subject of intense research. In this paper, we evaluate how far the state-of-the-art encoder-decoder deep convolutional neural network methods can go at assessing 2D echocardiographic images, i.e segmenting cardiac structures as well as estimating clinical indices, on a dataset especially designed to answer this objective. We therefore introduce the Cardiac Acquisitions for Multi-structure Ultrasound Segmentation (CAMUS) dataset, the largest publicly-available and fully-annotated dataset for the purpose of echocardiographic assessment. The dataset contains two and four-chamber acquisitions from 500 patients with reference measurements from one cardiologist on the full dataset and from three cardiologists on a fold of 50 patients. Results show that encoder-decoder based architectures outperform state-of-the-art non-deep learning methods and faithfully reproduce the expert analysis for the end-diastolic and end-systolic left ventricular volumes, with a mean correlation of 0.95 and an absolute mean error of 9.5 ml. Concerning the ejection fraction of the left ventricle, results are more contrasted with a mean correlation coefficient of 0.80 and an absolute mean error of 5.6 %. Although these results are below the inter-observer scores, they remain slightly worse than the intra-observer's ones. Based on this observation, areas for improvement are defined, which open the door for accurate and fully-automatic analysis of 2D echocardiographic images.

I. INTRODUCTION

The paper addresses the limited automation and evaluation of cardiac-structure segmentation and clinical-index estimation in 2D echocardiography by introducing CAMUS and evaluating encoder-decoder CNNs.

  • 2D echocardiographic analysis supports diagnosis through clinical indices derived from segmentation and tracking.
  • Fully automatic segmentation remains insufficiently accurate and reproducible, leaving clinicians with time-consuming manual or semi-automatic annotation.
  • The lack of large, publicly available, well-annotated echocardiographic datasets has limited thorough evaluation of deep-learning methods for clinical-index estimation.
  • The study asks how CNNs compare with non-deep-learning methods, how much data they need, how accurately they estimate volumes and ejection fraction, and whether sophisticated architectures improve segmentation.
  • CAMUS provides two- and four-chamber echocardiographic sequences from 500 patients with expert annotations for the left-ventricular endocardium, myocardium, and left atrium.
  • Prior work included limited cardiac-ultrasound datasets and earlier deep-learning studies focused mainly on left-ventricular endocardium segmentation.

III. CAMUS DATASET

CAMUS is a large, fully annotated 2D echocardiographic dataset designed for clinically realistic assessment, with standardized reference annotations and explicit limitations in frame selection and image coverage.

  • III. CAMUS DATASET: CAMUS is described as the largest publicly available and fully annotated dataset for 2D echocardiographic assessment, containing two- and four-chamber sequences from 500 patients.
  • A. Dataset: The dataset preserves clinical realism by avoiding prerequisite or image-selection criteria, resulting in difficult cases, varied acquisition settings, and occasionally invisible wall segments.
  • A. Dataset: Ten-fold cross-validation uses 50 patients per test fold, with 400 patients for training and 50 for validation in each split.
  • A. Dataset: All patients have standard apical four- and two-chamber sequences acquired during routine practice using GE Vivid E95 scanners and an M5S probe.
  • B. Reference segmentation and contouring protocol: Three cardiologists established the reference annotations, including repeated annotations by one cardiologist on fold 5 to measure inter- and intra-observer variability.
  • B. Reference segmentation and contouring protocol: ED and ES were selected by largest and smallest LV dimensions because reliable ECG was unavailable, so reported volumes and EF require this qualification.

IV. EVALUATED METHODS

The study evaluates encoder-decoder networks for echocardiographic segmentation, comparing U-Net variants and more elaborate architectures under shared processing conditions.

  • A. CNN techniques based on an encoder-decoder architecture: Encoder-decoder networks extract compressed image features with an encoder and progressively reconstruct segmentation maps with a decoder.
  • 1) U-Net:: U-Net uses skip connections between encoder and decoder to recover details lost during downsampling and stabilize gradients.
  • 1) U-Net:: Two independent U-Net implementations were compared: U-Net 1 optimized for speed and U-Net 2 optimized for accuracy.
  • 1) U-Net:: U-Net 1 and U-Net 2 contain 2.0M and 17.5M parameters, respectively, enabling assessment of hyperparameter effects on segmentation quality.
  • The evaluated alternatives include anatomically constrained networks, stacked hourglasses with deep supervision, and U-Net++ with dense skip connections.
  • All evaluated methods used the same connected-component and hole-removal post-processing strategies.

B. Non-deep learning state-of-the-art techniques

The study compares encoder-decoder networks with established non-deep-learning methods for cardiac segmentation, using geometric metrics across cardiac structures and time points. Deep-learning implementations achieved the strongest overall segmentation results, while simpler U-Nets matched more sophisticated architectures and offered a favorable performance–complexity trade-off.

  • EDN implementations achieved the overall best segmentation scores across all metrics for both ED and ES.
  • U-Net 1 and U-Net 2 performed equivalently to more sophisticated encoder-decoder architectures across the compared metrics.
  • U-Net 1 and U-Net 2 differed statistically for most metrics, although their mean dm and dH differences were each only 0.1 mm.
  • The choice of normalization scheme materially influenced U-Net performance.
  • BEASM-auto achieved mean dH values of 9.9 mm at ED and 10.5 mm at ES, whereas SRF achieved mean dm values of 3.0 mm at ED and 3.5 mm at ES.
  • SRF showed large standard deviations, while BEASM-semi improved mean dm by 0.8 mm and mean dH by 2.4 mm through manual initialization.

2) Clinical scores:

Clinical indices were computed from algorithmic segmentations and compared across methods using volume and ejection-fraction metrics. EDN methods obtained the best tested clinical scores, with volume errors lower than 13.2 ml and more contrasted LVEF performance.

  • EDN methods obtained the best clinical scores across all tested metrics for LVEDV, LVESV, and LVEF.
  • LVEDV and LVESV estimation produced mean absolute errors of at most 13.2 ml.
  • LVEF estimation yielded correlation scores of at most 0.82, biases below 1.8%, standard deviations of at most 7.8%, and mean absolute errors below 5.7%.
  • Average EDN clinical scores remained below inter-observer scores, indicating clinical interest alongside a stated need for improvement.
  • U-Net methods achieved clinical results similar to more sophisticated EDN architectures.

2) The effect of poor quality images:

Additional experiments examined image-quality inclusion, training-set size, and annotator-specific supervision for U-Net 1. Poor-quality images produced similar results when included or excluded, while larger training sets generally improved segmentation and models learned annotator-specific contouring patterns.

  • 2) The effect of poor quality images: Including 19% poor-quality images, representing 94 patients, produced results close to training only on good- and medium-quality images.
  • 3) Influence of the size of the training dataset: Segmentation metrics generally improved as the training set increased from 50 to 400 patients across LVEndo, LVEpi, and LA.
  • 3) Influence of the size of the training dataset: Performance worsened slightly at 250 patients in one experiment, while later gains from 250 to 400 patients were 1.1 mm for LVEpi and 1.0 mm for LA.
  • 3) Influence of the size of the training dataset: U-Net 1 outperformed non-deep-learning methods after training with 50 patients and needed at least 250 patients to reach highly competitive results.
  • 4) Influence of the expert annotations: The model trained on cardiologist O1a’s annotations achieved the best scores, supporting consistent annotator-specific contouring differences learned by EDNs.

2) Inter and intra-observer variability:

Expert agreement was strong for ventricular volume estimation but weaker for ejection fraction, while encoder-decoder results approached intra-observer performance and sometimes exceeded inter-observer scores. U-Net offered a compact performance trade-off, although outliers and remaining gaps indicate room for improvement.

  • Clinical variability: 0.92 and 0.91 mean correlations were achieved by experts for LVEDV and LVESV, respectively, compared with 0.67 for LVEF.These values were computed on fold 5, restricted to 40 patients with good or medium image quality.
  • Architecture comparison: U-Net results were close to more sophisticated architectures, suggesting a performance plateau that favors U-Net's parameter and training-time efficiency.The authors attribute the similar performance partly to the simple truncated ellipse-like shapes in 2D echocardiography.
  • Segmentation performance: The five encoder-decoder implementations clearly outperformed fully and semi-automatic non-deep-learning methods for segmentation.U-Net 1 produced results close to U-Net 2 with nine times fewer parameters.
  • Outlier analysis: 18% of segmentations from U-Net 1, U-Net 2, ACNN, and SHG were outside inter-observer variability, compared with 30% for U-Net++.Outliers were defined using dm and dH thresholds at end diastole and end systole.
  • Agreement with experts: Encoder-decoder segmentation and clinical results were close to but slightly worse than intra-observer scores, leaving room to reproduce an individual expert's annotations more faithfully.The authors also report that EDN segmentation results were better than inter-observer scores, while clinical results were below them.

Supplementary materials for:

The supplementary materials document the paper's architectures, hyperparameter tests, geometric comparisons, and additional experiments. They also provide supplementary segmentation and clinical analyses.

  • Architecture and hyperparameters: The supplementary materials include details of the U-Net architectures and their implementation in Tables I and II.They also cover the impact of main hyperparameters in Table III.
  • Additional experiments: Additional supplementary analyses examine overlapping U-Net 1 and U-Net 2 geometric-result densities, ACNN behavior, and U-Net++ tests.The materials also include mono/multi/GM segmentation results in Figure 1.
  • Supplementary evaluation: Supplementary materials provide fold-5 geometric and clinical metrics restricted to good and medium image quality, alongside further implementation and result information.They include geometrical results in Table VII and clinical metrics for the same restricted fold.

A. Detailed architectures

The supplementary architecture description contrasts a compact U-Net 1 with a wider U-Net 2 and records shared preprocessing choices for fair comparison. Both designs use encoder-decoder convolutional processing with segmentation output layers.

  • Architecture comparison: U-Net 1 uses a more compact architecture with an additional downsampling level, whereas U-Net 2 uses wider layers beginning with 48 filters.The small performance gap makes U-Net 1 more efficient on the robustness-performance trade-off.
  • U-Net 1 design: Both implementations use repeated convolutional blocks, 2×2 max-pooling for downsampling, and a final four-class softmax segmentation layer.The tabulated layers specify convolutional feature-map widths and the segmentation output.
  • U-Net 2 design: U-Net 2 expands and contracts feature widths from 48 through 384 filters and uses transposed convolutions for upsampling.Batch normalization and ReLU activations are included throughout the listed convolutional blocks.
  • Common processing: The two U-Net implementations share resizing to 256×256 pixels, density normalization, no augmentation, padding before 3×3 convolutions, Glorot initialization, max-pooling, and ReLU activations.These shared choices were used for a fair comparison.

C. From U-Net 1 to U-Net 2

The supplementary experiments examine training-set effects, initialization, architecture width, shape regularization, and geometric-result distributions. They generally support compact U-Net designs and show that shape regularization helps mainly when training data are scarce.

  • U-Net 1 to U-Net 2: Batch normalization was the only tested modification consistently improving LV performance beyond initialization effects.Loss, batch size, and upsampling changes produced no consistent differences, while a lower learning rate was slightly detrimental.
  • U-Net 1 to U-Net 2: Adding more filters was not meaningfully beneficial, suggesting that U-Net 1 already has sufficient parameters and that extra filters learn redundant information.The experiment changed the original number of filters while evaluating LVEndo and LVEpi accuracy.
  • Training-set size: Shape regularization improved ACNN results for a five-patient training set but not for a 50-patient training set.An auto-encoder trained on 15 patients produced results close to one trained on 400 patients.
  • Training-set size: For this 2D echocardiography task, the authors estimate that the necessary number of cases for shape learning is below 50 patients because shape variability is low.This conclusion is presented as specific to the task and linked to the ability of an auto-encoder to infer shapes from few cases.

IX. TESTS ON U-NET ++

The U-Net++ experiments show that added architectural complexity does not improve segmentation on CAMUS, while learning strategies produce close Dice results. Encoder-decoder networks also retain strong performance across image-quality subsets and clinical evaluation.

  • U-Net++ design: The best U-Net++ variant retained only its last output and replaced baseline layers with the optimized U-Net 1 design.Averaging intermediate output maps worsened results, especially standard deviation; dropout was detrimental.
  • Learning strategy: Mono-structure and multi-structure learning produced close but statistically different Dice results, while excluding 19% of poor-quality images did not significantly change performance.Outliers were more numerous for the left atrium, whose shape and acquisitions were more variable.
  • Image quality: Encoder-decoder implementations achieved the best segmentation performance on poor-quality images except for LVEndo Hausdorff distance at end diastole.The supplementary evaluation covered 94 poor-quality patients.
  • Clinical evaluation: Bland–Altman analyses compared LVEF errors among cardiologists, repeated annotations, and eight evaluated methods using errors from -40 to 40.The plots represent mean differences and 95% confidence intervals where specified.

XI. ADDITIONAL INSIGHTS ON OUTLIERS

Additional analyses show that U-Net 1 needs roughly 200 training patients to reach highly competitive results, after which its outlier rate stabilizes. Visual examples compare predictions across architectures, classical methods, and cardiologists under good and medium image quality.

  • Training-set size: 34% to 13%: U-Net 1’s outlier rate fell sharply as training increased from 50 to 200 patients.Beyond 200 patients, the outlier rate stabilized around 12–13%, supporting at least 200 patients for highly competitive results.
  • Outlier analysis: The outlier-rate comparison on fold 5 included eight methods and cardiologists, using observer O1a as the reference.The analysis was restricted to 40 patients with good or medium image quality.
  • Visual examples: U-Net 1’s medium-quality example was selected to reflect median full-dataset scores of 1.6 mm for endocardium and 1.7 mm for epicardium.The figure overlays dotted ground-truth contours and solid prediction contours.

A. Unsolved cases and limitations

U-Net 1 still produces outliers in difficult acquisition contexts, although most remain anatomically plausible. Rare implausible shapes and context-sensitive failures identify robustness as an unresolved limitation.

  • Unsolved cases and limitations: 18% of U-Net 1 predictions were outliers, and about 90% of those remained anatomically plausible LV and myocardium shapes.The remaining 10% of outliers, representing 1.8% of the full dataset, could not be assimilated to a heart shape.
  • Failure contexts: Non-frequent zoom and probe tilt can mislead the network in outlier cases.These acquisition conditions are identified as peculiar contexts associated with prediction errors.
  • Failure contexts: Artifacts including shadowed zones, reverberation, and fuzzy textures can mislead segmentation, particularly around the epicardium.Figure 23 illustrates an artifact-related epicardial failure.
  • Failure contexts: Extremely low local or global contrast can mislead the network in a patient-dependent manner.The endocardium in Figure 23 is identified as an example.
  • Potential improvement: Artificial data augmentation that recreates deployment conditions is proposed as a possible route to improve robustness to context variations.This is presented as a lead rather than an established result.
Loading 1908.06948v2…