Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation

Ozan Oktay; Enzo Ferrante; Konstantinos Kamnitsas; Mattias Heinrich; Wenjia Bai; Jose Caballero; Stuart Cook; Antonio de Marvao; Timothy Dawes; Declan O'Regan; Bernhard Kainz; Ben Glocker; Daniel Rueckert

Source-linked AI summary

Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation

Ozan Oktay, Enzo Ferrante, Konstantinos Kamnitsas, Mattias Heinrich, Wenjia Bai, Jose Caballero, Stuart Cook, Antonio de Marvao, Timothy Dawes, Declan O'Regan, Bernhard Kainz, Ben Glocker, Daniel Rueckert

arXiv:1705.08302v4May 22, 2017cs.CV

TL;DR

CNN-based image analysis often lacks a mechanism for incorporating global anatomical structure, which is especially problematic for corrupted cardiac images. The paper introduces ACNN, an end-to-end regularisation strategy using learned nonlinear anatomical representations, and reports improved robustness and state-of-the-art cardiac results across MR and US tasks.

Problem
Pixel-level CNN objectives do not fully capture global shape, label structure, or output dependencies, while cardiac images can contain acquisition artefacts.
Method
ACNN couples neural-network models with autoencoder- and T-L-based regularisers that learn compact anatomical representations from intensity and segmentation images.
Results
Across cardiac MR and US datasets, ACNN supports artefact-robust MR segmentation and super-resolution, pathology classification from latent shape codes, and state-of-the-art 3D-US segmentation.
Takeaways & Limitations
Learned anatomical priors can guide neural-network predictions toward anatomically meaningful outputs when cardiac image data are corrupted or insufficiently informative.
Takeaways & Limitations
Adversarial regularisation can produce worse PSNR and SSIM, while its interpretability and training stability remain open problems for medical imaging.

Abstract

from arXiv · show

Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches. In particular, priors can be useful in cases where images are corrupted and contain artefacts due to limitations in image acquisition. The highly constrained nature of anatomical objects can be well captured with learning based techniques. However, in most recent and promising techniques such as CNN based segmentation it is not obvious how to incorporate such prior knowledge. State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not incorporate the structure and inter-dependencies of the output. To overcome this limitation, we propose a generic training strategy that incorporates anatomical prior knowledge into CNNs through a new regularisation model, which is trained end-to-end. The new framework encourages models to follow the global anatomical properties of the underlying anatomy (e.g. shape, label structure) via learned non-linear representations of the shape. We show that the proposed approach can be easily adapted to different analysis tasks (e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models. The applicability of our approach is shown on multi-modal cardiac datasets and public benchmarks. Additionally, we demonstrate how the learned deep models of 3D shapes can be interpreted and used as biomarkers for classification of cardiac pathologies.

I. INTRODUCTION

Medical-image artefacts make anatomical priors valuable, yet CNN objectives often model outputs pixel-wise without global shape and label dependencies. ACNN addresses this gap with a learned compact anatomical representation that regularises neural-network predictions across applications.

Imaging artefacts such as respiratory motion, bias fields, shadowing, signal drop-out, and low resolution can corrupt US and MR images.
Anatomical priors can improve segmentation by constraining the solution space when image evidence is corrupted or incomplete.
Medical-image neural networks have rarely incorporated anatomical priors deeply, despite anatomy being constrained in shape and location.
Most classification and regression models use pixel-level losses that do not fully represent semantic information and output dependencies.
ACNN learns a nonlinear compact representation with a stacked convolutional autoencoder and constrains predictions to the resulting shape and label manifold.
For super-resolution, the global objective acts as a prior over feasible high-resolution solutions and encourages anatomically meaningful images alongside reconstruction loss.

A. Clinical Motivation

Cardiac MR and US are clinically useful but remain vulnerable to acquisition artefacts that can prolong examinations and impair quantitative measurements. The paper applies ACNN to improve robustness and anatomical plausibility across cardiac enhancement and segmentation tasks.

Cardiac MR and US provide detailed assessment of cardiac function and morphology, with each modality serving distinct clinical uses.
Acquisition artefacts and operator errors can obscure anatomy, leading to repeated acquisitions, longer examinations, inaccurate measurements, or discarded images.
The evaluation covers cardiac MR and US datasets, including 1200 MR and 45 US cardiac image sequences.
ACNN constrains neural-network training to guide anatomically meaningful predictions when image data are insufficient or inconsistent.
The reported evaluation includes robust sub-pixel MR segmentation, pathology classification from latent shape codes, artefact-robust SR, and state-of-the-art 3D-US segmentation.
The framework uses autoencoder and T-L networks as regularisers and extends CNN-based segmentation and super-resolution models with a global anatomical objective.

A. Medical Image Segmentation with CNN Models

CNN segmentation estimates tissue labels from image intensities, but standard pixel-level supervision does not enforce global anatomical structure. The proposed framework adds autoencoder- and T-L-based priors to regularise segmentation predictions.

CNN segmentation estimates tissue labels ys from observed intensities x by learning the conditional distribution P(ys|x).
The model assigns each pixel probabilities across C tissue classes, producing class feature maps through learned nonlinear functions.
Softmax converts extracted class feature maps into final class-label decisions in cross-entropy-based segmentation.
The baseline mapping φ(x) is trained with average cross-entropy using convolutional layers, nonlinearities, multiscale processing, and reconstruction stages.
Pixel-level losses such as cross-entropy and Dice do not exploit global dependencies, so predictions may violate shape, label, or atlas priors.
Autoencoder and T-L networks provide a nonlinear compact anatomical representation used as a prior for segmentation.

B. Convolutional Autoencoder Model and ACNN-Seg

The ACNN-Seg model integrates an autoencoder-based anatomical regulariser into segmentation training to constrain pixel-level predictions toward globally consistent, anatomically plausible label structures. The same prior-based strategy is also adapted to super-resolution by combining reconstruction with global anatomical constraints.

ACNN-Seg: The autoencoder regulariser constrains segmentation labels toward anatomically meaningful and accurate outputs beyond pixel-wise cross-entropy.Cross-entropy alone does not guarantee global consistency or plausible anatomical shapes.
ACNN-Seg: Predicted and ground-truth label maps are projected into a lower-dimensional representation, enabling shape-based regularisation.The representation can use, for example, 64 dimensions.
ACNN-Seg: The ACNN-Seg objective combines cross-entropy, shape regularisation, and weight decay, with λ1 and λ2 controlling the latter terms.The shape loss encourages predictions to occupy a similar low-dimensional shape space as ground-truth labels and promotes label consistency.
Super-resolution adaptation: The SR formulation combines smooth ℓ1 reconstruction with a global prior on feasible high-resolution anatomical solutions.The global objective is useful because super-resolution is ill-posed and admits many possible outputs.
Super-resolution adaptation: The SR architecture reduces memory and computation by generating features on the low-resolution grid before reconstruction.It also uses a larger receptive field and multiscale feature maps to capture anatomy and enforce global shape constraints.

D. T-L Network Model and SR-ACNN

The T-L network extends autoencoder-based shape regularisation from segmentation masks to intensity images, allowing super-resolution models to use shared anatomical representations. It is trained jointly with predictor and autoencoder components, then applied only during training as a global regulariser.

T-L Network Model: Shape-encoding autoencoders are limited to segmentation masks, so the method extends them to intensity-based super-resolution through a T-L regularisation model.The T-L model combines an autoencoder with a predictor that maps images to low-dimensional anatomical codes.
T-L Network Model: Jointly learned codes can be reached from both label and intensity spaces, supporting anatomical regularisation for segmentation and super-resolution.The authors expect these codes to represent more variation because they incorporate information from both manual annotations and intensity images.
Training strategy: The T-L network is trained in stages: the autoencoder first learns from ground-truth masks, the predictor matches its latent codes, and both are then trained jointly.The encoder receives gradients encouraging predictor-compatible codes while preserving segmentation reconstruction.
Latent representations: The latent codes generally follow smooth, normal distributions across randomly selected components, a property described as important for training ACNN models.The figure displays histograms for 16 randomly selected codes.
SR-ACNN objective: The SR objective combines weight decay, pixel-wise reconstruction, and a global loss measuring Euclidean distance between codes from synthesised and ground-truth high-resolution images.The T-L model acts only during training as a compact anatomical representation rather than an expanded feature-map perceptual space.

E. Learnt Hidden Representations

The learned low-dimensional codes constrain neural-network models with global anatomical characteristics and are encouraged to represent a smooth underlying data manifold.

The learned low-dimensional representation h constrains neural-network models through global characteristics of the underlying anatomy.The representation is used during training through gradients based on Euclidean distances between codes.
Small Gaussian input noise during T-L training helps explain the smooth distribution of the learned codes.The text relates denoising reconstruction to a contractive penalty that resists input perturbations.
The penalty promotes learning the underlying low-dimensional data manifold and its local smooth structure.The extracted codes may be correlated because the decoder merges codes across spatial dimensions, which is not considered a limitation here.

III. APPLICATIONS AND EXPERIMENTS

The paper applies ACNN models to cardiac MR and ultrasound segmentation and evaluates them against several neural and shape-based baselines using public and paired high-resolution datasets.

The applications include 3D-US segmentation, cardiac MR segmentation, cardiac MR super-resolution, and pathology classification using learned representations.The experiments examine the importance of shape and label priors and their potential biomarker value.
The paired cardiac MR dataset contains 1200 cine 2D-stack and 3D high-resolution image pairs from healthy subjects.The 2D stacks may contain motion artefacts, while 3D imaging can require long acquisition times.
The CETUS’14 benchmark contains 3D+time ultrasound sequences from 15 healthy subjects and 30 patients with cardiac disease.The dataset focuses on left-ventricle chamber analysis from apical windows.
The evaluation compares ACNN-Seg with 2D-FCN, 3D-Seg, 3D-UNet, AE-Seg, and motion-augmented 3D-Seg models.The table evaluates cardiac MR segmentation against these alternative architectures.
ACNN-Seg achieves 1.37±0.42, 7.89±3.83, and .939±.017 for one evaluation group, with 1.14±0.22, 7.31±3.59, and .811±.027 for another.These values are reported in the ACNN-Seg row alongside model capacity 1.60 × 10^6; the metric labels are not included in the supplied row.

3) ACDC MICCAI’17 Challenge Dataset:

The ACDC’17 experiments use latent representations from the proposed T-L network for cardiac MR examination classification and apply augmentation and joint optimization during training.

The ACDC’17 dataset contains 20 cine 2D MR image stacks per class for healthy, infarction, dilated cardiomyopathy, and hypertrophic cardiomyopathy cases.Clinical annotations are provided at end-diastolic and end-systolic phases.
Affine transformations augment segmentation and T-L training samples, while rigid transformations simulate respiratory motion artefacts for super-resolution models.The augmentation strategies differ according to the task.
Gaussian noise improves robustness to image noise, and randomly swapping tissue labels encourages smoothness in the autoencoder latent manifold.The label-swapping probability is 0.1.
Joint T-L training updates the encoder using gradients from both cross-entropy and Euclidean-distance losses.The authors report better convergence than applying the two gradient updates sequentially.

C. Cardiac Cine-MR Image Segmentation

The cardiac cine-MR segmentation experiments test ACNN-Seg on low-resolution 2D stacks against multiple baselines, showing benefits from shape priors for artefact robustness without extra post-processing or model complexity.

C. Cardiac Cine-MR Image Segmentation: The experiment segments 1200 low-resolution cardiac MR images at end diastole using high-resolution projected ground-truth labels.The data are split into 900 training, 100 validation, and 200 testing images after landmark-based cropping.
C. Cardiac Cine-MR Image Segmentation: ACNN-Seg is compared with 2D-FCN, 3D-UNet, AE-Seg, 3D-Seg, and motion-augmented 3D-Seg under controlled model capacities.The comparisons include both slice-wise and volumetric approaches.
C. Cardiac Cine-MR Image Segmentation: ACNN-Seg reduces errors in basal and apical slices and is reported to be insensitive to slice misalignments compared with 2D-FCN.Low-resolution predictions are better correlated with high-resolution ground-truth annotations in the illustrated cases.
C. Cardiac Cine-MR Image Segmentation: Shape priors help address false-positive detections and motion artefacts, contributing to a large Hausdorff-distance difference.The Dice-score difference is smaller for endocardium labels because the LV blood pool is larger than the myocardium.
C. Cardiac Cine-MR Image Segmentation: ACNN-Seg learns shape constraints within a standard segmentation model without additional parameters, computational complexity, or post-processing.The cascaded AE-Seg model is more strongly regularised but can overlook fine myocardium details and adds computational complexity.
C. Cardiac Cine-MR Image Segmentation: Learned shape priors guide anatomically correct delineations when cavity noise limits apical LV segmentation or ventricle boundaries are occluded.The figure presents shape-prior guidance as a mechanism for handling corrupted or incomplete boundaries.

D. Cardiac 3D Ultrasound Image Segmentation

The cardiac 3D ultrasound experiment evaluates ACNN-based segmentation against CNN and deformable-model baselines, using endocardial boundaries and derived volumetric indices. The broader experiments also assess anatomically constrained super-resolution and its integration with segmentation.

3D cardiac ultrasound models delineate endocardial boundaries at end-diastolic and end-systolic frames to measure ejection fraction and surface-distance errors.
15 image sequences supported a blinded public-benchmark evaluation comparing neural networks with a fully convolutional baseline and the BEAS challenge winner.
Neural network models outperformed previous state-of-the-art approaches on the public benchmark despite the limited training-data size.
ACNN-SR matched the comparison model's filter count and capacity while achieving fivefold lower runtime through low-dimensional image-space feature extraction.
The proposed 3D-Seg model performed better than concatenating super-resolution and U-Net segmentation, avoiding reconstruction of a high-dimensional intensity image.

F. Learnt Latent Representations and Pathology Classification

The study analyzes latent representations learned from cardiac segmentation images to determine whether they encode anatomical variation and distinguish cardiac pathologies. It also discusses the interpretability and scope of these anatomy-specific priors.

The jointly trained T-L model produces 64-dimensional codes from cardiac MR segmentation images for latent-space analysis and pathology classification.
The first two latent dimensions were observed to capture myocardial wall-thickness and ventricular lateral-wall variation.
The classification experiment asks whether enforced prior distributions contain anatomical information beyond representations meaningful only to the autoencoder decoder.
Adversarial regularization may produce more realistic super-resolution appearance but can worsen PSNR and SSIM and has less interpretable, unstable training behavior.
Learned codes were demonstrated as biomarkers for classifying cardiac-related pathologies, while the latent-space distribution was also analyzed.

VI. SUPPLEMENTARY MATERIAL

The supplementary material specifies training hyperparameters, synthetic super-resolution sampling, and the architectures of the predictor and autoencoder components of the T-L network.

Both super-resolution and segmentation models use λ2 = 5 × 10^-6 for weight decay, learning rate 0.001, global-prior weight λ1 = 0.01, and batch size 8.
Synthetic low-resolution super-resolution samples use through-plane factor K = 5, Gaussian blurring with σ = 4.0 mm, and subsequent decimation.
The models use 10–18 layers without residual connections, and omit nonlinear layers on lower-dimensional latent representations.
The T-L predictor maps a 120x120x60 high-resolution intensity image to a 64-dimensional latent representation using multiscale convolutional layers and ReLU units.
The autoencoder maps a 120x120x60 segmentation map through convolutional and fully connected encoder layers, then reconstructs a segmentation map from the low-dimensional code.