Source-linked AI summary
Convolutional Random Walk Networks for Semantic Image Segmentation
Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, Jianbo Shi
TL;DR
FCNs can blur object boundaries, while CRF post-processing can produce spatially fragmented segments; integrating CRFs or MRFs into FCNs can add complexity. RWN jointly learns pixel affinity and segmentation through a random-walk layer, improving accuracy over FCNs at the same model complexity.
Problem
FCNs have poor boundary localization, Dense-CRF post-processing can produce spatially fragmented segments, and integrated CRF/MRF methods can increase model and training complexity.
Source: §1 · p. 1Method
RWN jointly optimizes pixelwise affinity and semantic segmentation through a random-walk layer that enforces spatial consistency in deep network layers.
Source: §1 · p. 2Results
RWN improves accuracy over traditional FCNs at the same model complexity, with 131 additional learnable parameters (0.0008% of the network’s parameters).
Source: §6 · p. 8Takeaways & Limitations
RWN can be integrated into the standard FCN learning framework for joint end-to-end training.
Source: §6 · p. 8Some prior approaches adapted to specific tasks and datasets can be difficult to adapt to new tasks and datasets.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Most current semantic segmentation methods rely on fully convolutional networks (FCNs). However, their use of large receptive fields and many pooling layers cause low spatial resolution inside the deep layers. This leads to predictions with poor localization around the boundaries. Prior work has attempted to address this issue by post-processing predictions with CRFs or MRFs. But such models often fail to capture semantic relationships between objects, which causes spatially disjoint predictions. To overcome these problems, recent methods integrated CRFs or MRFs into an FCN framework. The downside of these new models is that they have much higher complexity than traditional FCNs, which renders training and testing more challenging. In this work we introduce a simple, yet effective Convolutional Random Walk Network (RWN) that addresses the issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives of pixelwise affinity and semantic segmentation. It combines these two objectives via a novel random walk layer that enforces consistent spatial grouping in the deep layers of the network. Our RWN is implemented using standard convolution and matrix multiplication. This allows an easy integration into existing FCN frameworks and it enables end-to-end training of the whole network via standard back-propagation. Our implementation of RWN requires just $131$ additional parameters compared to the traditional FCNs, and yet it consistently produces an improvement over the FCNs on semantic segmentation and scene labeling.
1. Introduction
FCNs produce poorly localized boundaries, while Dense-CRF post-processing can yield spatially fragmented segments; integrated CRF/MRF approaches add model or training complexity. RWN links affinity and segmentation learning through a random-walk layer with little added complexity.
1. Introduction
- FCNs lose fine boundary detail, Dense-CRF post-processing can fragment segments, and integrated CRF/MRF models often complicate training or increase model complexity.Large receptive fields and pooling reduce spatial resolution in deep FCN layers; post-processing may fail to capture semantic relationships between objects.Source: §1 · p. 1
- RWN addresses FCN boundary localization and Dense-CRF fragmentation without significantly increasing model complexity.The paper presents it as an alternative to traditional FCNs and recent segmentation approaches.Source: §1 · p. 2
- RWN combines pixelwise-affinity and semantic-segmentation objectives through a random-walk layer that enforces spatial consistency in deep network layers.The layer is implemented with matrix multiplication and supports end-to-end training with standard back-propagation.Source: §1 · p. 2
- RWN compares favorably with DeepLab variants on semantic segmentation and scene labeling, using only 131 additional parameters.Its implementation uses standard convolution and matrix multiplication and requires minimal modifications to an FCN framework.Source: §1 · p. 2
2. Related Work
Prior work refines FCN predictions through separate globalization models, integrated CRF/MRFs, or deconvolutional networks, each with trade-offs in learning, complexity, or model size. RWN jointly learns pixel affinity and segmentation with a small parameter increase.
2. Related Work
- Separate globalization models refine FCN predictions using region or boundary similarities, but disjoint learning can miss semantic relationships and produce spatially disjoint segments.The cited methods are [4].Source: §2 · p. 2
- Integrated CRF/MRF methods enable joint end-to-end training but can increase model size or require recurrent layers and complex training procedures.The approaches discussed include [27] [19] [17] and [5].Source: §2 · p. 2
- Deconvolutional networks use deconvolution and unpooling to recover fine details, but effective recovery can require nearly as many deconvolutional as convolutional layers.This yields a large increase in parameters; cited methods include [23] [12].Source: §2 · p. 2
- RWN jointly optimizes pixel affinity and semantic segmentation end-to-end, using 131 additional parameters over the base FCN.The added parameters are 0.0008% of the network total; experiments report spatially smoother segmentations.Source: §2 · p. 2
3. Background
The background describes random walks as graph-based information propagation using transition probabilities and matrix multiplication. Unlike fixed-grid CRF/MRF graphs, random walks can use arbitrary graph structures.
3. Background
- The RWN architecture has segmentation and pixel-affinity branches joined by a random-walk layer for spatially coherent predictions.It is jointly optimized end-to-end; random walks propagate information through matrix multiplication, unlike CRF/MRF approaches that use approximate inference.Source: Figure 2 · p. 3, §3 · p. 3
- A random-walk graph represents pixels as nodes, uses a symmetric affinity matrix W for pairwise similarity, and defines transition probabilities with A = D^-1W.D stores node degrees, computed from affinities to the other nodes.Source: §3 · p. 3
- A one-step random walk propagates a node distribution by multiplying it by the transition matrix, yt+1 = Ayt; the process can repeat until convergence.The passage illustrates this with rank distributions in PageRank.Source: §3 · p. 3
- Random walks can specify arbitrary graph structures through the affinity matrix, whereas CRF/MRF methods typically use fixed-grid graphs.The passage notes that different problems may benefit from different graph structures.Source: §3 · p. 3
4. Convolutional Random Walk Networks
The RWN combines semantic segmentation and pixel-level affinity branches through a random-walk layer that encourages spatially coherent predictions. Its design integrates these branches for end-to-end training while adding 131 parameters.
4. Convolutional Random Walk Networks
- The RWN combines semantic segmentation potentials and pixel-level affinities through a random-walk layer that encourages spatially coherent segmentation and supports joint end-to-end optimization.The network has separate segmentation and affinity branches, merged at the random-walk layer.Source: §4 · p. 4
4.1. Semantic Segmentation Branch
- The semantic segmentation branch uses DeepLab-LargeFOV, DeepLab-attention, or DeepLab-v2, though other architectures could also be integrated.The tested variants are based on VGG or ResNet architectures.Source: §4.1 · p. 4
4.2. Pixel-Level Affinity Branch
- The affinity branch uses RGB and conv1_1/conv1_2 features to estimate nearby pixel affinities, with same-label pairs within radius R assigned ground-truth affinity 1.The affinity input stores channel-wise L1 distances, and deeper features or additional affinity-branch layers yielded little improvement.Source: §4.2 · p. 4
4.3. Random Walk Layer
- The random-walk layer diffuses fc8 segmentation potentials through the learned transition matrix, with matrix multiplication enabling gradients to reach both branches.The transition matrix is sparse, so gradient computation considers pixel pairs corresponding to its nonzero entries.Source: §4.3 · pp. 4–5
- On SBD per-pixel Intersection over Union, RWN consistently outperforms DeepLab-LargeFOV, DeepLab-attention, and DeepLab-v2.Table 2 reports this comparison without providing specific scores in the supplied passage.Source: Table 2 · p. 5
4.4. Random Walk Prediction at Testing
- At testing, repeated random-walk updates balance propagating segmentation information with staying close to initial FCN predictions, and are applied until convergence.The authors describe the converged prediction as the result of repeated random-walk steps.Source: §4.4 · p. 5
- Qualitative comparisons report smoother RWN segmentations with fewer false positives than DeepLab-v2, and more accurate fine details than DeepLab-v2-CRF without post-processing.Examples include bike wheels and plane wings.Source: Figure 3 · p. 4, Figure 4 · p. 6
4.5. Implementation Details
- Training uses a single random-walk step with R = 40, while testing uses R = 5 and iterations until convergence; the test-time scheme takes about 1 second per image.The larger training radius avoids multiple steps and speeds training; the converged test-time scheme yields slightly better results.Source: §4.5 · p. 6, §4.4 · p. 6
5. Experimental Results
On semantic segmentation and scene labeling, RWN improves on DeepLab baselines, with additional comparisons showing stronger boundary localization, smoother predictions, and faster inference than dense-CRF inference. The experiments also examine random-walk steps and graph radius, finding benefits from iteration and flexibility across graph structures.
5.1. Semantic Segmentation Task
- RWN outperforms all three DeepLab baselines on Pascal SBD, matches or exceeds post-processed DeepLab, and achieves lower boundary error at every trimap width.Boundary comparisons also favor RWN over DeepLab-CRF on MF and AP; qualitative results show smoother regions, fewer false positives, and finer object details.Source: §5.1 · pp. 6–7, Figure 5 · p. 7, Table 4 · p. 7
5.2. Scene Labeling
- RWN exceeds the DeepLab baseline by 2.57% on Stanford Background and 2.54% on Sift Flow under overall IOU.Both scene-labeling experiments use DeepLab-largeFOV as the base model.Source: §5.2 · p. 7, Table 5 · p. 8
5.3. Runtime Comparisons
- A random-walk iteration takes 0.032 seconds per Pascal SBD image versus 3.301 seconds for dense-CRF inference; RWN_v2 reaches 82.2% IOU versus 81.9% for DeepLab_v2 with dense CRF.The runtime comparison uses a single core of a 2.7GHz Intel Core i7 processor.Source: §5.3 · pp. 7–8
5.4. Ablation Experiments
- IOU accuracy increases with additional random-walk steps and peaks at convergence, while predicted boundaries become better localized as iterations increase.Figure 7 reports accuracy by step count; Figure 6 illustrates changes in predicted probabilities.Source: §5.4 · p. 8, Figure 7 · p. 8, Figure 6 · p. 8
- With convergence, RWN performs similarly for radii between 3 and 20; radius 40 with one step reaches 75.5% and 77.6%, while radius 5 to convergence reaches 75.8% and 77.9%.Applying multiple steps with radius 40 does not improve accuracy; the model can use different graph structures, unlike the fixed-grid structures typical of MRF or CRF models.Source: §5.4 · p. 8
6. Conclusion
Random Walk Networks (RWNs) improve accuracy over traditional FCNs at the same model complexity while addressing poor boundary localization and spatially disjoint segmentations. The authors identify alternative architectures and new application domains as future work.
6. Conclusion
- RWNs produce improved accuracy over traditional FCNs at the same model complexity.The conclusion also describes RWN as a more efficient alternative to denseCRF approaches.Source: §6 · p. 8
- RWN addresses poor localization around segmentation boundaries and spatially disjoint segmentations.Source: §6 · p. 8
- The implementation adds 131 learnable parameters, or 0.0008% of the network's original parameters, and supports joint end-to-end training in the standard FCN framework.Source: §6 · p. 8
- Future work will explore alternative RWN architectures and applications in language processing or speech recognition.Source: §6 · p. 8