PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel-Adaptive Convolutional Neural Networks

Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, Jan Kautz

arXiv:1904.05373v1cs.CVcs.GRcs.LG

TL;DR

Standard convolutions use spatially shared filters that ignore image content, limiting adaptation across pixels and scenes. The paper proposes PAC, which modulates shared filters with feature-dependent kernels, and reports broad gains across vision tasks. PAC generalizes common filtering operations, supports guided upsampling, and serves as a faster alternative to fully-connected CRF inference while improving fine-tuned pre-trained networks.

  • Problem

    Standard convolution uses spatially shared, content-agnostic filters, although different pixels and image contents may require different filtering behavior.

  • Method

    PAC adapts a spatially invariant convolution filter at each pixel by multiplying it with a spatially varying kernel based on pre-defined or learned pixel features.

  • Results

    PAC demonstrates broad applicability across vision tasks, including state-of-the-art depth and optical-flow upsampling, competitive CRF inference, and improved fine-tuned pre-trained networks.

  • Takeaways & Limitations

    PAC generalizes several popular filtering operations and can replace standard convolution layers while retaining minimal computation overhead.

  • Takeaways & Limitations

    High-dimensional filtering incurs computational overhead and approximation error, while PAC’s reported scope covers the evaluated vision applications and settings.

Abstract

from arXiv · show

Convolutions are the fundamental building block of CNNs. The fact that their weights are spatially shared is one of the main reasons for their widespread use, but it also is a major limitation, as it makes convolutions content agnostic. We propose a pixel-adaptive convolution (PAC) operation, a simple yet effective modification of standard convolutions, in which the filter weights are multiplied with a spatially-varying kernel that depends on learnable, local pixel features. PAC is a generalization of several popular filtering techniques and thus can be used for a wide range of use cases. Specifically, we demonstrate state-of-the-art performance when PAC is used for deep joint image upsampling. PAC also offers an effective alternative to fully-connected CRF (Full-CRF), called PAC-CRF, which performs competitively, while being considerably faster. In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.

1. Introduction

Standard convolutions share spatially invariant, content-agnostic filters, which can be suboptimal across pixels and image content. PAC adapts these filters using pixel features and is evaluated across multiple vision problems.

  • Spatial Sharing: Standard convolution shares filter parameters across pixels, reducing parameters but potentially producing filters that are sub-optimal at specific locations.Spatially varying losses can require different optimal gradient directions at different pixels.
  • Content-Agnostic Filters: Standard CNN filters are applied regardless of image content, so one trained filter bank may not be optimal for different images or pixels.The paper contrasts content such as daylight versus night and sky versus pedestrian regions.
  • Pixel-Adaptive Convolution: PAC adapts a spatially invariant convolution filter W at each pixel by multiplying it with a spatially varying kernel K based on pre-defined or learned pixel features.The adapting kernel can use features such as position and color or features learned by a CNN.
  • Generality: PAC generalizes spatial convolution, bilateral filtering, and pooling operations, including average pooling and detail-preserving pooling.The paper presents PAC as a flexible modification of standard convolution rather than a wholly separate filtering framework.
  • Applications: PAC achieves state-of-the-art results for depth and optical-flow upsampling, improves a learnable CRF over fully-connected CRF, and improves fine-tuned pre-trained CNNs.The authors summarize these results as evidence of PAC’s versatility across computer vision tasks.

2. Related Work

Prior content-adaptive approaches either differentiate established image filters or predict position-specific convolution kernels. PAC instead reuses spatial filters while modifying them position by position.

  • Image-Adaptive Filtering: One class makes bilateral and guided image filters differentiable for use as content-adaptive CNN layers, generally enhancing rather than replacing standard convolutions.These methods include differentiable filtering layers embedded inside CNNs.
  • Dynamic Filter Networks: Dynamic filter networks predict filter weights at each pixel with a separate network branch, producing custom filters for different input data.Later extensions add attention and dynamic sampling mechanisms.
  • Position-Specific Adaptation: PAC differs from dynamic and deformable convolution by reusing spatial filters and modifying them position-specifically rather than predicting complete kernels or offsets.The related-work discussion also connects PAC to self-attention mechanisms.

3. Pixel-Adaptive Convolution

PAC modifies standard spatial convolution with a feature-dependent adapting kernel while retaining computation on the 2D grid. This formulation generalizes several filtering and pooling operations and supports transposed convolution variants.

  • Standard Spatial Convolution: Standard spatial convolution applies filter weights W over an s×s neighborhood Ω(i), producing a c′-channel output at each pixel.Pixel coordinates p_i and biases b define the spatial indexing and output offsets.
  • Motivation: Because standard-convolution weights depend only on pixel position, they are spatially shared and image-agnostic rather than content-adaptive.The paper identifies this as a central limitation motivating PAC.
  • High-Dimensional Filtering: High-dimensional filtering makes W depend on pixel features but incurs computational overhead, requires sparse lattice structures, restricts feature dimensionality, and loses spatial-sharing benefits.The discussion cites hand-specified feature spaces and sparsity in projected high-dimensional images as practical constraints.
  • PAC Formulation: PAC keeps filtering on the 2D grid by multiplying W with a fixed-form kernel K based on adapting features f, which may be hand-specified or learned end-to-end.A Gaussian kernel is given as the main example of K.
  • Generalizations: PAC includes standard convolution, bilateral filtering, average pooling, and detail-preserving pooling as special cases or modeled operations.Average pooling uses K(fi, fj) = 1 and uniform W, while detail-preserving pooling emphasizes distinct neighborhood pixels.
  • Implementation and Variants: The implementation supports back-propagation through adapting features and includes pixel-adaptive transposed convolution, PAC⊺, for guided upsampling.PAC is implemented as a GPU-accelerated PyTorch network layer.

4. Deep Joint Upsampling Networks

The deep joint upsampling network combines low-resolution signals with high-resolution guidance, using PAC⊺ layers to produce full-resolution depth and optical flow. It achieves strong quantitative and visual results across both tasks.

  • Network setup: Joint upsampling combines a low-resolution signal with a corresponding high-resolution guidance image to recover a full-resolution output.The network receives x ∈ R^c×h/m×w/m and g ∈ R^c_g×h×w, then outputs x↑ ∈ R^c×h×w.
  • Network architecture: The encoder processes the low-resolution signal, while the guidance branch generates adapting features for PAC⊺ decoder layers that progressively upsample features.Each PAC⊺ layer upsamples feature maps by a factor of 2; subsequent convolution layers generate the final output.
  • Visual results: Our method produces more detailed and guidance-edge-faithful results for 16× joint depth and optical flow upsampling.Figure 3 compares the methods on depth in the top row and optical flow in the bottom row.
  • Joint depth upsampling: The depth experiments use NYU Depth V2, with 1000 RGB-depth pairs for training and the remainder for testing across 4×, 8×, and 16× upsampling.Performance is evaluated using root mean square error (RMSE).
  • Joint depth upsampling: Our upsampling network consistently outperforms other techniques for different depth upsampling factors.A lighter version matching DJF’s parameter count also performs better, supporting the importance of PAC⊺ for upsampling.
  • Joint optical flow upsampling: For optical flow, the network uses RGB guidance and shows consistent advantage over bilinear interpolation and DJF, restoring fine details and smoothing uniform-motion regions.The Sintel clean-pass experiments evaluate two-channel flow using End-Point-Error (EPE).

5. Conditional Random Fields

PAC-CRF replaces dense CRF pairwise connections and high-dimensional filtering with learnable PAC operations over local, potentially dilated windows. It achieves stronger segmentation performance than the compared CRF baselines while offering favorable runtime, although dilation can introduce boundary noise.

  • Efficient, learnable CRF with PAC: PAC-CRF targets Full-CRF’s costly permutohedral-lattice computation, which has limited GPU suitability and introduces approximation error in inference and gradient computation.The proposed PAC-based inference is designed to be efficient and easy to integrate with CNN backbones.
  • Efficient, learnable CRF with PAC: PAC-CRF defines pairwise connections over fixed local windows and performs mean-field updates with PAC instead of sparse high-dimensional filtering.Its formulation supports learnable compatibility transforms, spatially varying offsets, and learnable adapting features.
  • Efficient, learnable CRF with PAC: Dilation gives a 5×5 PAC filter a 257×257 neighborhood at dilation 64, enabling long-range pairwise connections with manageable parameter counts.PAC-CRF uses dilation to enlarge receptive fields without increasing the number of PAC parameters.
  • Semantic segmentation with PAC-CRF: PAC-CRF uses scaled RGB features as guiding features and evaluates one 5×5 filter at dilation 32 or parallel filters at dilations 16 and 64.Five mean-field steps balance speed and accuracy; training first freezes the FCN backbone before subsequent training.
  • Semantic segmentation with PAC-CRF: The two-filter PAC-CRF variant achieves better mIoU than Full-CRF, BCL-CRF, and Conv-CRF while comparing favorably in runtime.The one-filter variant is slightly worse than Full-CRF and BCL-CRF but provides an even larger speed advantage.

6. Layer hot-swapping with PAC

Layer hot-swapping replaces selected trained convolution layers with PAC layers while retaining the original network configuration and weights. In semantic segmentation, PAC-FCN improves test mIoU with virtually no inference-time penalty, and combining it with PAC-CRF adds further improvement.

  • Layer hot-swapping with PAC: Layer hot-swapping replaces selected convolution layers with PAC layers, using earlier network features as adapting features while retaining trained weights.The illustrated modification replaces three CONV layers, and the original network weights are retained after the change.
  • Layer hot-swapping with PAC: The visual examples compare PAC-CRF and PAC-FCN against Full-CRF, BCL-CRF, and Conv-CRF on validation-set segmentation outputs.The figure caption highlights PAC-CRF’s recovery of finer details aligned with RGB boundaries.
  • Layer hot-swapping with PAC: PAC-FCN improves test mIoU from 67.20 to 69.18, about a 2-point gain, with virtually no inference-time runtime penalty.The hot-swapped model has the same number of parameters as the original FCN.
  • Layer hot-swapping with PAC: Combining PAC-FCN with the PAC-CRF 16-64 variant yields another 2-point mIoU improvement after joint training.The PAC-FCN improvement remains complementary to CRF post-processing.

7. Conclusion

The paper proposes PAC, a content-adaptive filtering operation that leverages guidance information across joint upsampling, semantic segmentation, and CRF inference. PAC generalizes standard spatial convolution and can replace pretrained convolution layers with minimal computation overhead.

  • PAC learns to leverage guidance information through a new content-adaptive filtering operation.
  • PAC applies across joint upsampling, semantic segmentation networks, and efficient CRF inference.
  • PAC generalizes standard spatial convolution and directly replaces standard convolution layers in pretrained networks.The replacement is reported to provide performance gain with minimal computation overhead.

Appendix

The appendix provides additional details and results for the paper’s deep joint upsampling experiments and PAC-CRF.

  • The appendix includes additional details for the deep joint upsampling experiments.
  • It also includes additional results for the deep joint upsampling experiments.
  • The appendix provides additional details and results for PAC-CRF.

A. Deep Joint Upsampling with PAC

The joint upsampling networks use separate encoder, guidance, and decoder branches, with PAC transposed layers selected by upsampling factor. Standard and lite variants trade architectural simplicity, memory use, and parameter count.

  • A. Deep Joint Upsampling with PAC: The joint depth upsampling networks contain Encoder, Guidance, and Decoder branches.The guidance branch supplies adapting features for the PAC layers.
  • A. Deep Joint Upsampling with PAC: 2×, 4×, 8×, and 16× upsampling require 1, 2, 3, and 4 PAC transposed layers, respectively.
  • A. Deep Joint Upsampling with PAC: The standard variant uses a simpler design but consumes more memory than DJF and changes parameter count across upsampling factors.
  • A. Deep Joint Upsampling with PAC: The lite variant reduces filters so its networks roughly match DJF’s parameter count.
  • A. Deep Joint Upsampling with PAC: Optical-flow upsampling uses similar architectures, modifying the first encoder and last decoder layers for two flow channels.
  • A. Deep Joint Upsampling with PAC: Table 5 defines C as regular convolution and P as PAC transposed convolution, with trailing numbers indicating output channels.

B. Conditional Random Fields

PAC-CRF reformulates pairwise potentials using learnable spatial filters and feature-dependent kernels, extending CRF label compatibility beyond fixed formulations. Dilation supports long-range connections, and experiments use five mean-field steps for other reported results.

  • B. Conditional Random Fields: Full-CRF pairwise potentials combine a label compatibility term with appearance and smoothness kernels.
  • B. Conditional Random Fields: PAC moves the smoothness kernel into a learnable spatial filter while allowing the appearance kernel to operate directly on learned features.These changes increase the pairwise potential’s learning capacity and support end-to-end learning for multi-channel inputs.
  • B. Conditional Random Fields: PAC-CRF uses a 4D filter tensor so label compatibility patterns can vary across pixel locations.The formulation in Eq. 10 is a special case that factorizes the tensor into two 2D matrices.
  • B. Conditional Random Fields: Five mean-field steps are used for all other experiments in the paper.Table 6 reports validation mIoU for different numbers of mean-field steps.
  • B. Conditional Random Fields: Dilation increases PAC’s receptive field without increasing parameter count and enables long-range PAC-CRF connections.
  • B. Conditional Random Fields: +2.13/+1.57 → +2.50/+1.91 after adding dilation to Conv-CRF validates dilation as an important ingredient.The remaining performance gap is attributed in the passage to the PAC formulation.
Loading 1904.05373v1…