Source-linked AI summary
Roto-Translation Covariant Convolutional Networks for Medical Image Analysis
Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, Remco Duits
TL;DR
Medical image analysis needs models that handle structures appearing at arbitrary orientations without relying on rotation augmentation. The paper builds SE(2) group convolutions into CNNs, and reports state-of-the-art performance across three medical-imaging tasks with consistent improvement over CNN baselines.
Problem
Medical image analysis seeks models invariant to shape and appearance variability, including arbitrary orientations.
Method
The paper generalizes R2 CNNs to SE(2) G-CNNs, using roto-translation-based lifting and group convolution layers on position-orientation space.
Results
The authors report consistent performance improvements over corresponding CNN baselines across mitosis detection, vessel segmentation, and cell boundary segmentation, with state-of-the-art results.
Takeaways & Limitations
SE(2) group convolution layers can provide rotation-aware CNN designs that avoid rotation augmentation while retaining a common basic network design across the three tasks.
Abstract
from arXiv · showhide
We propose a framework for rotation and translation covariant deep learning using $SE(2)$ group convolutions. The group product of the special Euclidean motion group $SE(2)$ describes how a concatenation of two roto-translations results in a net roto-translation. We encode this geometric structure into convolutional neural networks (CNNs) via $SE(2)$ group convolutional layers, which fit into the standard 2D CNN framework, and which allow to generically deal with rotated input samples without the need for data augmentation. We introduce three layers: a lifting layer which lifts a 2D (vector valued) image to an $SE(2)$-image, i.e., 3D (vector valued) data whose domain is $SE(2)$; a group convolution layer from and to an $SE(2)$-image; and a projection layer from an $SE(2)$-image to a 2D image. The lifting and group convolution layers are $SE(2)$ covariant (the output roto-translates with the input). The final projection layer, a maximum intensity projection over rotations, makes the full CNN rotation invariant. We show with three different problems in histopathology, retinal imaging, and electron microscopy that with the proposed group CNNs, state-of-the-art performance can be achieved, without the need for data augmentation by rotation and with increased performance compared to standard CNNs that do rely on augmentation.
1 Introduction
The paper extends standard CNNs with SE(2) group convolutions so rotation covariance is built into the architecture rather than learned separately for each orientation. This design targets arbitrary orientations in medical images and is positioned as a straightforward alternative to related symmetry-aware approaches.
- Proposed framework: SE(2) G-CNNs replace translations of a kernel with roto-translations on position-orientation space.The construction generalizes R2 CNNs by defining convolution layers through representations of SE(2).
- Proposed framework: Rotation covariance is encoded in the network, so features appearing at multiple orientations need to be learned only once.The authors state that this removes the need for rotation-based data augmentation and frees kernel weights to increase expressive capacity.
- Proposed framework: The group convolution layers remain compatible with standard CNN modules, enabling integration into popular CNN designs.
- Motivation and evaluation scope: The framework addresses medical-image variability in structure shape, appearance, and arbitrary orientation, with reported improvements over standard 2D CNNs across three tasks.The tasks are mitosis detection, retinal vessel segmentation, and electron-microscopy cell-boundary segmentation.
- Relation to prior work: Compared with related rotation-aware methods, group convolutions most naturally extend standard CNNs by replacing the convolution operators.The cited alternatives include harmonic networks, local transformation invariance learning, deep symmetry nets, scattering CNNs, and warped convolutions.
- Relation to prior work: Unlike steerable-kernel approaches constrained to specific complex basis functions, the interpolation approach rotates base weights through a sparse matrix-vector multiplication.
- Relation to prior work: Relative to scattering networks and template matching, the approach learns unrestricted, non-separable filters using common ReLU activations within a trainable CNN framework.Scattering uses hand-crafted separable filters and SVMs, while the cited template method optimizes B-spline-based SE(2) templates with logistic regression.
2 SE(2) convolutional neural networks
The paper constructs CNN layers on SE(2), encoding roto-translation structure so features transform covariantly with the input while final projection yields rotation-invariant features.
- Group theoretical preliminaries: SE(2) combines planar translations and rotations, with group multiplication describing the net roto-translation of successive motions.Elements are identified with position-orientation pairs in R2 × S1.
- The SE(2) group convolution layers: The implementation uses cross-correlations, which are equivalent to convolutions up to kernel reflection and remain compatible with standard CNN modules.On R2, cross-correlation is defined through inner products of translated kernels.
- The SE(2) group convolution layers: The lifting layer replaces translated kernels with roto-translated kernels and maps a vector-valued 2D image to data on the SE(2) position-orientation space.The output has domain R2 × S1 ≡ SE(2) and multiple channels.
- The SE(2) group convolution layers: Group convolution layers operate on SE(2)-images by correlating kernels and features over the group, mapping between SE(2) feature representations.The resulting representation performs a shift-twist, combining spatial rotation with an orientation shift.
- The SE(2) group convolution layers: The projection layer maps a multi-channel SE(2) image back to R2 and produces a rotation-invariant feature vector.The figure description contrasts activations for the original and rotated inputs and identifies the final projection as the invariant stage.
- The SE(2) group convolution layers: Discrete SE(2) kernels sample N rotation angles, while sparse matrix multiplication constructs all rotated kernels from shared base weights using bilinear interpolation.Lifting kernels have spatial and channel dimensions, whereas group kernels additionally include the sampled rotation dimension.
3 Experiments and Results
Experiments evaluate SE(2, N) networks across three medical-imaging tasks and orientation samplings, using matched network sizes and repeated runs. The tasks cover mitosis detection, retinal vessel segmentation, and electron-microscopy cell-boundary segmentation.
- Experimental setup: Networks were evaluated with SE(2, N) orientation samplings N ∈ {1, 2, 4, 8, 16} across three medical-imaging tasks.Experiments also included transpose augmentation, plus transpose and 90° rotation augmentation for N = 1.
- Experimental setup: Overall network weights and the number of 2D activations in the final three layers were matched across N and against the R2 approach.Each experiment was repeated three times to estimate mean performance and variance.
- Results presentation: The figure presents crop-outs and class probabilities in the top row, followed by mean results with ±1 standard deviation in the bottom row.The results summarize the three tasks evaluated in the experiments.
- Vessel segmentation: Retinal vessel segmentation used DRIVE images, with sensitivity and specificity summarized by the area under the ROC curve.Training used 17 × 17 patches sampled for vessel and background classes.
4 Discussion and Conclusions
Across three medical-image analysis tasks, G-CNNs consistently improved performance over corresponding CNN baselines. The strongest results occurred with orientation capacity N ≥4, while larger N improved stability in two tasks.
- Results: G-CNNs consistently improved performance over corresponding CNN baselines across all three medical-image analysis tasks.The reported results were in line with the benchmark of each dataset.
- Results: The best performances were obtained with orientation capacity N ≥4, indicating benefits from learning rotation-invariant representations.This pattern was observed across the evaluated tasks.
- Stability: Repeated experiments showed improved stability in mitosis detection and vessel segmentation for N = 8 and N = 16.The authors suggest this may reflect regularization from increased weight sharing as N increases.
- Conclusions: The authors conclude that SE(2) layers avoid rotation augmentation while improving overall performance and achieving state-of-the-art results across three medical-imaging problems.They expect further gains when embedding the layers in more complex designs such as UNets and ResNets.