Source-linked AI summary
Learning Affinity via Spatial Propagation Networks
Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, Jan Kautz
TL;DR
The paper addresses the challenge of learning task-specific dense affinity matrices without hand-designed similarity kernels. It introduces spatial propagation networks that use CNN-predicted spatial transformations and three-way propagation, achieving refined segmentation results on HELEN and PASCAL VOC tasks. The paper also identifies a stability condition for the propagation model.
Problem
Affinity construction is often based on hand-designed distance metrics or kernels that may not describe mid-to-high-level pairwise relations, making effective learning-based affinity difficult for complex visual analysis.
Method
SPN learns spatially varying row/column transformation matrices with a deep CNN and uses three-way linear propagation to produce dense affinity relationships efficiently.
Results
SPNs improve segmentation over coarse baselines on HELEN and PASCAL VOC; on PASCAL VOC, DeepNet ResNet-101 SPN reaches mean IoU 80.22 versus dense CRF 79.7.
Takeaways & Limitations
The framework provides a general, effective, and efficient approach for generating high-quality segmentation results and is presented as applicable to numerous affinity-related vision tasks.
Takeaways & Limitations
Stable propagation requires each transformation matrix to satisfy λmax ≤1, a sufficient condition that constrains the learned transformations.
Abstract
from arXiv · showhide
In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, including but not limited to image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of the deep neural network classifier. We validate the framework on the task of refinement for image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.
1 Introduction
The paper treats affinity learning as a data-driven problem and introduces SPN, which combines CNN-predicted affinities with efficient spatial propagation for vision tasks.
- 1 Introduction: Affinity matrices encode task-specific pairwise similarity between image pixels, from low-level color and texture coherence to high-level semantic relationships.Their construction is central to the success of many vision algorithms but is generally not learned directly.
- 1 Introduction: SPN reformulates affinity learning as learning small, spatially varying row/column linear transformations that can be implemented differentiably in a deep network.A CNN predicts the transformation elements directly from the input RGB image.
- 1 Introduction: The three-way connection yields a dense affinity matrix from a sparse transformation matrix while requiring far fewer output channels than a fully connected design.This avoids hand-designed similarity features and kernel constructions.
- 1 Introduction: Data-driven affinity learning avoids fixed distance-space assumptions, task-specific kernel design, and iterative refinement requirements for satisfactory semantic segmentation.The method learns affinity entities under direct supervision from the ultimate task loss.
- 1 Introduction: The SPN architecture jointly trains a CNN affinity predictor and differentiable spatial propagation module, whose recurrent inference has linear time complexity.The module accepts images or general 2D matrices and propagates them under learned affinity guidance.
2 Related Work
Prior work designs or indirectly improves affinity structures for filtering, colorization, matting, and segmentation, but learning-based affinity construction remains difficult for complex visual analysis.
- 2 Related Work: Existing methods explicitly design affinity matrices for image filtering, colorization, matting, and segmentation according to each problem’s physical properties.Other approaches improve pairwise modeling through objectives or additional priors in diffusion PDEs.
- 2 Related Work: Graphical-model methods such as dense CRFs model densely connected semantic-label relationships, but rely on designed kernels.CRFs may be used as post-processing or integrated into jointly trained deep networks.
3 Proposed Approach
The spatial propagation network learns affinity through spatially varying row/column transformations, whose induced matrix is equivalent to a diffusion process. Sparse three-way connections enable globally and densely connected pairwise relationships while remaining computationally efficient.
- 3 Proposed Approach: The SPN transforms a 2D map through spatially varying propagation parameters and is theoretically equivalent to anisotropic diffusion.The propagation module is connected to a neural network through joint training.
- 3 Proposed Approach: A matrix is scanned independently in four fixed directions, with each column recursively combining the previous propagated column and the corresponding input column.The four directions include left-to-right and top-to-bottom scans, with reverse directions processed analogously.
- 3 Proposed Approach: The expanded propagation is a lower-triangular transformation matrix relating vectorized input and output maps, with block parameters formed from the local transformations.The vectorized maps concatenate the column vectors of the input and output matrices along one dimension.
- 3 Proposed Approach: The transformation matrix has unit row sums, and its off-diagonal elements form an affinity matrix while its diagonal terms form the degree matrix.This structure yields a Laplacian L = D − A and the discrete diffusion update U_T+1 − U_T = −LU_T.
- 3 Proposed Approach: Learning the affinity matrix is equivalent to learning the row/column transformation matrices, which can be predicted from input content by a differentiable deep CNN.The affinity is therefore conditioned on each input image rather than fixed independently of image content.
- 3 Proposed Approach: Three-way connections link each pixel to three pixels in the previous row or column, producing dense global affinities from sparse transformations.Unlike one-way connections, three-way propagation expands through triangular planes, and integrating four directions connects all pixels densely.
- 3 Proposed Approach: Propagation stability is ensured when the largest singular value of each transformation matrix satisfies λmax ≤1.This condition bounds the temporal Jacobian norm and provides a sufficient condition for stable propagation.
4 Implementation
The SPN implementation separates guidance from propagation: a deep CNN predicts transformation-matrix weights, while a linear module uses them to refine an input 2D map. The guidance network can accept task-specific 2D inputs, and the propagation module processes the resulting weights across four directions.
- 4 Implementation: The implementation uses two branches: a guidance CNN outputs transformation-matrix elements, while a linear propagation module produces the refined or transformed map.The guidance network can be any regular deep CNN designed for the task at hand.
- 4 Implementation: The guidance network accepts a 2D matrix such as an RGB image and outputs all weights constituting the transformation matrix w_t.
- 4 Implementation: Figure 2 separates the propagation module from the guidance network, with the latter producing entities for four affinity matrices.Each sub-matrix w_t is tridiagonal, and the modules are jointly learned through SGD.
- 4 Implementation: For an n × n × c input map, the guidance network outputs a weight map of size n × n × c × (3 × 4), assigning three scalar weights per pixel for each of four directions.
5 Experimental Results
Experiments evaluate SPN as a refinement model for high-resolution face parsing and generic object segmentation. Across HELEN and Pascal VOC, three-way SPN improves coarse or pretrained segmentation, including against dense CRF refinement.
- HELEN face parsing: The HELEN setup trains SPN on coarse segmentation maps to generate high-resolution outputs supervised by high-resolution ground truth.The baseline parser uses low-resolution inputs and outputs, while SPN performs refinement afterward.
- HELEN face parsing: On HELEN, both one-way and three-way SPNs outperform the baseline, CNN-Highres, and prior work by over 6 IoU points.The gains are especially pronounced for smaller facial components such as eyes and lips.
- Pascal VOC segmentation: On Pascal VOC, three-way SPN significantly improves segmentation accuracy over coarse results for all three baseline models.Table 2 compares FCN-8s, Deeplab VGG, and Deeplab ResNet-101 with the corresponding SPN variants.
- Pascal VOC segmentation: Three-way SPN generalizes across pretrained models, whereas one-way SPN fails on the more complex Pascal VOC data.The authors attribute the one-way failure to limited capacity for learning preferable affinities with a sparse form.
- Pascal VOC segmentation: On Pascal VOC, Deeplab ResNet-101 with SPN reaches mean IoU 80.22 versus 79.7 with dense CRF, without fine-tuning.SPN probability maps also show improved object details, boundaries, and semantic integrity.
- Pascal VOC segmentation: SPN also significantly improves dilated-convolution baselines, with the three-way variant yielding a larger margin than the context aggregation module.The comparison is reported for the “+3 way” and “+Context” variants in Table 3.
6 Conclusion
The spatial propagation network is presented as a general framework for learning affinity matrices and producing high-quality segmentation results. The paper demonstrates this effectiveness on HELEN face parsing and PASCAL VOC object semantic segmentation.
- SPNs provide a generic framework for learning affinity matrices in vision tasks.
- Experiments on HELEN face parsing and PASCAL VOC object semantic segmentation show that SPNs generate high-quality segmentation results.
- Figure 5 visualizes Pascal VOC segmentation results alongside object probability derived from 1 − Pb.