Source-linked AI summary
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, Yoshua Bengio
TL;DR
Semantic segmentation requires recovering full-resolution predictions without making dense connectivity computationally impractical. The paper extends DenseNets into fully convolutional networks with an upsampling design that mitigates feature-map explosion. The resulting networks improve state-of-the-art performance on CamVid and Gatech with about 10-fold fewer parameters and without pretraining or post-processing.
Problem
Naively extending DenseNets to segmentation can produce too many high-resolution feature maps, greatly increasing computation and parameter counts.
Method
The paper builds fully convolutional DenseNets using dense-block upsampling, limiting transition-up inputs to the preceding dense block while using skip connections for higher-resolution information.
Results
The resulting 56-to-103-layer networks improve state-of-the-art performance on CamVid and Gatech with about 10-fold fewer parameters, without pretraining, post-processing, or temporal information.
Takeaways & Limitations
Dense connectivity can support deep, parameter-efficient semantic-segmentation networks whose upsampling path avoids feature-map explosion.
Takeaways & Limitations
The models were not pretrained on large datasets such as ImageNet and could most likely benefit from such pretraining.
Abstract
from arXiv · showhide
State-of-the-art approaches for semantic image segmentation are built on Convolutional Neural Networks (CNNs). The typical segmentation architecture is composed of (a) a downsampling path responsible for extracting coarse semantic features, followed by (b) an upsampling path trained to recover the input image resolution at the output of the model and, optionally, (c) a post-processing module (e.g. Conditional Random Fields) to refine the model predictions. Recently, a new CNN architecture, Densely Connected Convolutional Networks (DenseNets), has shown excellent results on image classification tasks. The idea of DenseNets is based on the observation that if each layer is directly connected to every other layer in a feed-forward fashion then the network will be more accurate and easier to train. In this paper, we extend DenseNets to deal with the problem of semantic segmentation. We achieve state-of-the-art results on urban scene benchmark datasets such as CamVid and Gatech, without any further post-processing module nor pretraining. Moreover, due to smart construction of the model, our approach has much less parameters than currently published best entries for these datasets. Code to reproduce the experiments is available here : https://github.com/SimJeg/FC-DenseNet/blob/master/train.py
1. Introduction
The paper extends DenseNets to fully convolutional semantic segmentation, designing an upsampling path that limits feature-map growth. The resulting architecture targets accurate urban-scene segmentation without pretraining or post-processing.
- Motivation: FCNs recover input resolution through an upsampling path and use skip connections to restore fine-grained information lost during downsampling.The architecture combines a coarse semantic downsampling path with a resolution-recovering upsampling path.
- Motivation: DenseNets connect layers through iterative feature-map concatenation, enabling parameter efficiency, implicit deep supervision, and feature reuse.These properties naturally provide skip connections and multi-scale supervision for semantic segmentation.
- Design challenge: A naive fully convolutional DenseNet would create excessive high-resolution feature maps, causing very large computation and parameter counts.The problem arises because high-resolution maps would be multiplied by filters from all preceding layers.
- Proposed architecture: The proposed extension mitigates feature-map explosion by upsampling only maps from the preceding dense block and combining them with same-resolution skip features.This design allows dense blocks at each upsampling resolution independently of the number of pooling layers.
- Contributions: The upsampling path uses dense blocks and performs better than paths built from more standard operations.The paper identifies the dense-block upsampling path as a specific performance contribution.
- Results: The network outperforms state-of-the-art urban-scene benchmarks without pretrained parameters or further post-processing.The reported claim covers standard benchmarks for urban scene understanding.
2. Related Work
Prior semantic-segmentation work improves resolution recovery, context modeling, or output structure, often using specialized modules and pretrained backbones. Dense-block connectivity provides a distinct architectural basis for feature reuse and multi-scale supervision.
- Research directions: Recent segmentation architectures improve upsampling connectivity, broader context understanding, or structured prediction.These directions include stronger upsampling paths, context modules, and structured-output mechanisms.
- Resolution recovery: FCN upsampling ranges from bilinear interpolation to unpooling and transposed convolutions, with skip connections supporting finer information recovery.Prior work also analyzes combinations of identity mappings and long skip connections.
- Context modeling: Context-aware approaches add global descriptors, recurrent sweeps, or dilated convolutions to enlarge the network’s effective context.These methods seek broader context without necessarily reducing image resolution.
- Structured prediction: Conditional Random Fields and recurrent approximations to mean-field inference have been used to enforce structural consistency in FCN outputs.The recurrent formulation enables end-to-end training of the FCN and recurrent module.
- Pretraining: Many state-of-the-art FCN architectures rely on pretrained VGG or ResNet101 models to improve segmentation results.The paper positions its no-pretraining setting against this common practice.
3. Fully Convolutional DenseNets
This section reviews DenseNet connectivity and presents FC-DenseNet, which extends dense blocks with an upsampling path while avoiding feature-map explosion.
- Feature-map control: This design enables very deep FC-DenseNets without feature-map explosion while retaining information through skip connections.The architecture is intended to exploit feature reuse while avoiding the computational and parameter costs of naive upsampling.
- DenseNet review: DenseNets concatenate all preceding feature maps, encouraging feature reuse and giving every layer a direct supervision path.Each layer produces k feature maps, so the number of feature maps grows linearly with depth.
- FC-DenseNet architecture: FC-DenseNet uses a DenseNet downsampling path, a bottleneck, and dense-block-based upsampling with skip connections.Transition-down operations reduce spatial resolution, while transition-up modules use transposed convolutions before concatenating skip-connected features.
- Feature-map control: The upsampling path applies transposed convolutions only to feature maps from the preceding dense block rather than to all accumulated maps.The preceding block summarizes information from earlier blocks at the same resolution, limiting the number of maps processed at high resolution.
- FC-DenseNet103: FC-DenseNet103 contains 103 convolutional layers, five transition-down modules, five transition-up modules, and a per-pixel softmax output.Its layers are distributed across the input, downsampling path, bottleneck, and upsampling path.
- FC-DenseNet103: The proposed architecture has a reasonable pre-softmax feature-map count of 256 and is trained with pixel-wise cross-entropy loss.The feature-map count is attributed to mitigation of the DenseNet explosion in the upsampling path.
4. Experiments
The experiments evaluate FC-DenseNets on CamVid and Gatech using IoU and global accuracy, finding strong benchmark performance without pretraining or post-processing. The proposed architecture also improves segmentation over classic upsampling and achieves gains with substantially fewer parameters.
- Evaluation: IoU measures class-wise intersection over union, while global accuracy measures pixel-wise accuracy across the dataset.For class c, IoU compares pixels whose predictions and targets both equal c with pixels where either does.
- Training and scope: The experiments train models from scratch without extra data or post-processing, but the authors state that pretraining could most likely improve FC-DenseNet performance.Gatech training uses one of every ten redundant frames and evaluates all full-resolution test frames.
- CamVid results: 10 times more parameters: FC-DenseNet architectures improve upon published methods while achieving state-of-the-art results; FC-DenseNet56 exceeds architectures with at least 100 times more parameters.The results include FC-DenseNet56, FC-DenseNet67, and FC-DenseNet103, with deeper models benefiting from greater depth and parameter count.
- CamVid results: The model outperforms a temporally regularized CamVid method without temporal smoothing, while temporal regularization remains complementary and could provide additional improvements.Qualitative results show sharp segmentations, although thin tree details, vehicles, and shop signs can still be confused with other classes.
- Gatech results: 23.7%: the method improves global accuracy over prior 2D-convolution state of the art on Gatech, and gains 3.4% over spatio-temporal 3D-convolution models.The model uses only 2D convolutions; Gatech contains pixel-wise labels for eight classes across urban-scene videos.
5. Discussion
The discussion attributes FC-DenseNet’s advantages to parameter efficiency, implicit deep supervision, and feature reuse. It also connects dense connectivity to gradient paths spanning networks of different depths.
- Architectural advantages: FC-DenseNets use fewer parameters than other published CamVid segmentation architectures while retaining parameter efficiency inherited from DenseNets.The discussion also identifies feature reuse because layers can access preceding layers through iterative concatenation.
- Architectural advantages: Additional supervision levels produced no noticeable performance change, supporting the paper’s characterization of implicit rather than explicitly added deep supervision.
- Gradient flow: Dense connectivity passes gradients through networks of different depths, suggesting a distinct gradient-flow behavior from the short paths associated with ResNets.The authors propose revisiting whether fully convolutional DenseNets behave like ensembles of relatively shallow networks.
6. Conclusion
The paper extends DenseNets into fully convolutional networks for semantic segmentation and designs an upsampling path that limits feature-map growth. The resulting deep, parameter-efficient networks improve urban-scene segmentation on CamVid and Gatech without pretraining, post-processing, or temporal information.
- The proposed fully convolutional DenseNets extend DenseNets to semantic image segmentation.Their dense blocks iteratively concatenate feature maps.
- The upsampling path mitigates the linear feature-map growth of a naive DenseNet extension.
- 56 to 103 layers and about 10 fold fewer parameters characterize the resulting networks relative to state-of-the-art models.
- The networks improve state-of-the-art performance on CamVid and Gatech urban scene understanding datasets.
- The reported improvements require neither additional post-processing, pretraining, nor temporal information.