Source-linked AI summary
Capsules for Object Segmentation
Rodney LaLonde, Ulas Bagci
TL;DR
Object segmentation requires accurate pixel-level delineation, while existing segmentation networks can be complex and capsule architectures are computationally expensive. The paper introduces SegCaps, a convolutional-deconvolutional capsule network with locally constrained routing and deconvolutional capsules, and evaluates it on pathological lung CT segmentation. SegCaps handles 512 × 512 images and achieves slightly better dice performance with substantially fewer parameters than U-Net and Tiramisu.
Problem
Object segmentation needs joint recognition and pixel-level delineation, while complex deep networks and original capsule routing impose optimization, memory, and runtime burdens.
Method
SegCaps extends capsule networks for segmentation using locally constrained routing, shared transformations, deconvolutional capsules, and masked reconstruction of the target class.
Results
SegCaps slightly outperforms U-Net and Tiramisu on pathological lung segmentation, with an average dice score of 98.479% and 95.4% fewer parameters than U-Net.
Takeaways & Limitations
SegCaps is the first capsule-network architecture for object segmentation and operates on large 512 × 512 images, beyond prior capsule inputs typically below 32 × 32.
Takeaways & Limitations
Original capsule networks require substantial memory and runtime because dynamic routing stores intermediate child-capsule representations across possible parent connections.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have shown remarkable results over the last several years for a wide range of computer vision tasks. A new architecture recently introduced by Sabour et al., referred to as a capsule networks with dynamic routing, has shown great initial results for digit recognition and small image classification. The success of capsule networks lies in their ability to preserve more information about the input by replacing max-pooling layers with convolutional strides and dynamic routing, allowing for preservation of part-whole relationships in the data. This preservation of the input is demonstrated by reconstructing the input from the output capsule vectors. Our work expands the use of capsule networks to the task of object segmentation for the first time in the literature. We extend the idea of convolutional capsules with locally-connected routing and propose the concept of deconvolutional capsules. Further, we extend the masked reconstruction to reconstruct the positive input class. The proposed convolutional-deconvolutional capsule network, called SegCaps, shows strong results for the task of object segmentation with substantial decrease in parameter space. As an example application, we applied the proposed SegCaps to segment pathological lungs from low dose CT scans and compared its accuracy and efficiency with other U-Net-based architectures. SegCaps is able to handle large image sizes (512 x 512) as opposed to baseline capsules (typically less than 32 x 32). The proposed SegCaps reduced the number of parameters of U-Net architecture by 95.4% while still providing a better segmentation accuracy.
1 Introduction
Object segmentation has advanced through CNNs, but increasingly complex architectures remain difficult to optimize, while capsules offer vector representations and dynamic routing that preserve part–whole relationships. This study extends capsule networks to object segmentation and evaluates them on pathological lung CT scans.
- Deep segmentation architectures improve image analysis but require many hyperparameters and can become too complex to optimize.
- CNN neurons encode scalar, additive responses and are ambivalent to spatial relationships within their receptive fields.
- Capsule networks store neuron-level information as vectors describing spatial orientation, magnitude, prevalence, and other feature attributes.
- Dynamic routing uses agreement between capsule vectors to form meaningful part-to-whole relationships.
- The study extends capsules and dynamic routing to object segmentation and tests them on pathological lung segmentation from CT scans.
2 Background and Related Works
Object segmentation jointly requires recognizing objects and delineating their pixel-level extent, making it difficult and clinically consequential. Deep learning methods dominate the field, but pathological lung segmentation remains challenging because CT scans contain diverse abnormalities.
- Object segmentation combines object recognition with delineation of an object's spatial extent and composition at the pixel level.
- U-Net and related encoder-decoder CNNs have become common approaches for medical image segmentation.
- Pathological lung segmentation from CT scans remains extremely challenging because lung abnormalities span diverse appearances, including fibrosis, nodules, and ground glass opacities.
- Precise segmentation matters clinically because small errors can affect diagnosis, severity estimation, prognosis, and other evaluations.
3 Building Blocks of Capsules for Image Segmentation
This section introduces SegCaps as the first capsule-network approach to object segmentation, addressing capsule routing costs through local connectivity and a convolutional-deconvolutional design. The architecture supports large images and achieves slightly improved lung-segmentation results with substantially fewer parameters than U-Net and Tiramisu.
- Capsule-based object segmentation had not previously been addressed in the literature.
- Local routing restricts children to spatially local parents, while sharing transformation matrices across grid members within each capsule type.
- 512 × 512 pixel inputs become feasible, whereas previous capsule architectures did not exceed 32 × 32 pixels.
- Deconvolutional capsules and a deep convolutional-deconvolutional architecture extend capsule networks beyond the original three-layer design.
- 95.4% fewer parameters than U-Net and 38.4% fewer than Tiramisu accompany slightly improved lung-segmentation Dice results.
4 SegCaps: Capsules for Object Segmentation
SegCaps defines convolutional capsules over spatial grids and routes capsule outputs locally, using shared transformations to reduce parameters while preserving segmentation-relevant context. Its pipeline applies dynamic routing, nonlinear squashing, positive-class reconstruction, and capsule-length thresholding to produce segmentation masks.
- SegCaps architecture: SegCaps processes a 512 × 512 CT image into 16-dimensional capsules arranged on a 512 × 512 spatial grid.The initial convolution produces 16 feature maps of the same spatial dimensions, forming the first capsule set.
- Convolutional capsules: Each parent capsule receives prediction vectors generated from child capsule types within a user-defined spatial kernel.The kernel dimensions are kh × kw, and prediction vectors are formed using learned transformation matrices.
- Parameter sharing: Transformation matrices are shared across spatial locations within each capsule type, reducing the number of parameters learned.The matrices remain distinct across capsule types while being independent of spatial location.
- Dynamic routing: Locally constrained dynamic routing restricts each child capsule to parents within a defined kernel instead of routing every child to every parent.Routing coefficients are computed by a routing softmax, and parent inputs are weighted sums of prediction vectors.
- Segmentation output: The network applies a nonlinear squashing function and measures agreement through a scalar product before creating the final segmentation mask from capsule lengths.Pixels whose final capsule magnitude exceeds a threshold receive the positive class; the others receive the negative class.
- Reconstruction regularization: Masked reconstruction regularizes the model by reconstructing only the positive input class and computing weighted mean-squared error over positive pixels.A three-layer 1 × 1 convolutional network performs the reconstruction.
5 Experiments and Results
On the LUNA16 lung-segmentation experiments, SegCaps slightly outperformed the compared methods while using substantially fewer parameters. Qualitative comparisons also examined U-Net segmentation errors on matched CT slices.
- Quantitative results: 98.479% average Dice: SegCaps slightly outperformed all compared approaches on the LUNA16 experiments.It used over 95% fewer parameters than U-Net and over 38% fewer than Tiramisu.
- Capsule representations: Segmentation capsule vectors progressively capture regions with different textural properties, including small and large homogeneous regions.The analysis considered selected visual attributes from the 16-dimensional final segmentation capsule vectors.
- Quantitative results: Table 1 reports Dice Coefficient results from a 4-fold cross-validation split of the LUNA16 dataset.For SegCaps (R1), dynamic routing is performed only on layers that change spatial dimensions; other layers use equal-weight coupling coefficients.
- Qualitative results: Figure 3 compares U-Net and SegCaps outputs on the same two CT slices, with green arrows marking U-Net segmentation errors.The U-Net results appear in the left column and SegCaps results in the right column.
6 Conclusion
The paper introduces SegCaps, a convolutional-deconvolutional capsule network for object segmentation, and evaluates it on pathological lung segmentation from CT scans. Its design uses locally routed capsules and reconstruction extensions to reduce computational burden while supporting pixel-level predictions.
- Conclusion: SegCaps is proposed as a capsule-based object-segmentation algorithm evaluated on pathological lung segmentation from CT scans.The paper presents it as the first use of capsule networks for object segmentation.
- Conclusion: Locally routed capsules and shared transformation matrices reduce memory and parameter demands, enabling operation on large image sizes.The framework also introduces convolutional-deconvolutional capsules for pixel-level object-label predictions.
- Conclusion: Masked reconstruction is extended as a regularization strategy for the segmentation problem.This extends reconstruction of the target class within the proposed capsule framework.
- Conclusion: Capsule segmentation vectors represent different visual attributes across their dimensions, including regions with distinct textural properties.Figure 4 presents selected visual attributes from 16-dimensional capsule segmentation vectors.