Source-linked AI summary
Joint Image Filtering with Deep Convolutional Networks
Yijun Li, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang
TL;DR
Existing joint filters rely on explicit constructions or hand-designed objectives and may transfer structures inconsistent with the target. This paper learns a CNN-based joint filter from both target and guidance images, using feature fusion and residual learning. The resulting model generalizes across modalities and performs favorably across several joint-filtering applications.
Problem
Existing joint filters rely on explicit filter constructions or hand-crafted objectives and can transfer guidance structures that are inconsistent with the target.
Method
A three-subnetwork CNN extracts target and guidance features, concatenates them, and uses a skip connection to learn residuals for selective joint filtering.
Results
The model generalizes across modalities and performs favorably against state-of-the-art methods across denoising, upsampling, and texture-structure separation.
Takeaways & Limitations
Learning from both target and guidance images provides a generic joint-filtering framework that handles inconsistent structures and varied visual domains.
Takeaways & Limitations
The model can mistake small-scale guidance details for noise because its depth-based training data is mostly smooth and lacks many spatial details.
Abstract
from arXiv · showhide
Joint image filters leverage the guidance image as a prior and transfer the structural details from the guidance image to the target image for suppressing noise or enhancing spatial resolution. Existing methods either rely on various explicit filter constructions or hand-designed objective functions, thereby making it difficult to understand, improve, and accelerate these filters in a coherent framework. In this paper, we propose a learning-based approach for constructing joint filters based on Convolutional Neural Networks. In contrast to existing methods that consider only the guidance image, the proposed algorithm can selectively transfer salient structures that are consistent with both guidance and target images. We show that the model trained on a certain type of data, e.g., RGB and depth images, generalizes well to other modalities, e.g., flash/non-Flash and RGB/NIR images. We validate the effectiveness of the proposed joint filter through extensive experimental evaluations with state-of-the-art methods.
1 INTRODUCTION
Joint filtering transfers useful structures from a guidance image to improve a degraded target while avoiding inconsistent content. The paper proposes a data-driven CNN filter that uses both images, learns residuals, and generalizes across modalities.
- Motivation: Joint filtering enhances noisy or low-resolution targets by transferring important guidance-image structures while avoiding extraneous content.Applications include depth enhancement, joint upsampling, cross-modality noise reduction, and structure-texture separation.
- Motivation: Existing filters may transfer incorrect structures when local guidance and target content are inconsistent, while iterative methods rely on hand-crafted global objectives.These limitations motivate a learned consistency mechanism.
- Method: The proposed CNN extracts features separately from target and guidance images, concatenates them, and uses a skip connection to learn residuals toward the filtered output.The network is trained on large RGB/depth data to selectively transfer common structures.
- Method: The framework is purely data-driven and learns to selectively transfer structures using guidance-image priors rather than predicting specific output values.This design supports handling scenarios difficult to capture with hand-crafted objectives.
- Results: The model generalizes from training data such as RGB/depth to varied visual domains and achieves state-of-the-art depth upsampling on NYU v2 and SUN RGB-D, with competitive Middlebury performance.The paper also reports generalization across other modalities and applications.
2 RELATED WORK
Prior joint filters construct explicit weights or optimize hand-designed objectives, often using guidance structures without sufficient consistency constraints. The paper situates its CNN approach among learning-based low-level vision methods and emphasizes heterogeneous two-image processing with residual learning.
- Joint image filters: Joint filters primarily use explicit filter construction or global optimization of data fidelity and regularization terms.These classes differ in how they encode guidance and target relationships.
- Explicit joint filters: Bilateral and guided filters compute outputs using weights that depend solely on local guidance structure, allowing erroneous or extraneous structures to transfer.The proposed model instead considers both images through feature maps and learned consistency.
- Learning-based image filters: Learning-based filters replace predefined constructions with mappings learned from data, including CNNs that improve conventional filters or use multi-scale guidance for depth upsampling.These approaches form the immediate learning-based context for the proposed method.
- Deep models for low-level vision: Unlike many low-level vision models, the proposed network accepts two heterogeneous image streams such as RGB/NIR, flash/non-flash, and RGB/depth.Its feature responses are concatenated, rather than merged with the correlation operator used in related work.
- Related architectures: The method learns a new joint filter rather than approximating an existing edge-aware filter and uses residual learning through a skip connection.This distinguishes it from both single-image approximation networks and conventional one-input models.
3 LEARNING JOINT IMAGE FILTERS
The paper constructs a learning-based joint image filter from separate target and guidance streams, a fusion network, and a skip connection. Experiments show selective transfer of structures consistent across images, reduced texture copying, and generalization across training modalities.
- Network architecture design: The network uses CNNT and CNNG to extract target and guidance features separately before CNNF selectively transfers common structures.The concatenated feature responses feed CNNF, which generates the filtering residual.
- Network architecture design: A straightforward CNNF implementation poorly transfers guidance structures, motivating separate processing streams for target and guidance images.The generic network fails to transfer structures such as the bed corner, while edge-map guidance improves sharp-edge preservation.
- Skip connection: The skip connection adds the target input to the predicted residual, allowing the network to focus on differences rather than relearning shared low-frequency content.Quantitative results show notable improvement over the earlier method.
- Network training: Cross-dataset experiments find similar performance when models trained on RGB/depth data and RGB/flow data are applied across depth and flow upsampling tasks.The depth model produces competitive depth results when using the flow-trained model, with analogous observations for flow upsampling.
- What has the network learned?: CNNF suppresses guidance structures inconsistent with the target, including windows on walls and high-contrast mat textures that cause texture-copying artifacts.The learned guidance emphasizes salient boundaries while suppressing small-scale textures, and the residual model further alleviates texture copying.
4 EXPERIMENTAL RESULTS
The experiments evaluate the learned joint filter across depth upsampling, cross-domain upsampling, structure-texture separation, and noise reduction. Across these tasks, the method delivers competitive or favorable quality while remaining efficient and transferable across modalities.
- Experimental scope: The evaluation covers joint upsampling, texture-structure separation, and cross-modality image restoration across multiple datasets and image modalities.The same learned framework is tested on depth, colorization, saliency, inverse halftoning, RGB/NIR, and flash/non-flash tasks.
- Depth map upsampling: The proposed algorithm performs well against state-of-the-art methods across the NYU v2, SUN RGB-D, and Middlebury depth-upsampling datasets.The evaluation uses RMSE comparisons, with additional runtime measurements on NYU v2.
- Depth map upsampling: 526 KB versus 1,822 KB: the proposed model achieves improved results with fewer parameters than the re-trained DMSG comparison.The comparison uses the same absolute-depth training dataset and reports improved results for the proposed model with fewer parameters.
- Joint image upsampling: 2.82 seconds versus 8.20 seconds: the colorization pipeline is nearly three times faster than direct high-resolution colorization.The proposed pipeline combines a low-resolution colorization result with joint upsampling and achieves the lowest RMSE in the reported comparison.
- Cross-domain applications: The method produces sharper saliency edges, removes small-scale halftone dots more effectively than competing methods, and remains comparable for RGB/NIR and flash/non-flash denoising.For inverse halftoning, performance is worse on large-scale dots than methods specifically designed for that task, which require manual parameter selection.
5 DISCUSSIONS
The discussion studies how architecture choices affect training and performance, emphasizing the benefits of residual learning and structural support while identifying diminishing returns and a small-detail failure mode.
- Filter number: With a skip connection, increasing filter number progressively improves performance, whereas larger non-residual networks can become harder to train.The residual formulation benefits from increased capacity, although larger networks slow training.
- Feature channels: Multi-dimensional target and guidance feature maps slow training without clear performance gains in the tested settings.The authors therefore use single-channel output feature maps from the target and guidance subnetworks.
- Filter size: Larger filter sizes initially improve restoration by covering richer structural cues, but further enlargement does not continue the improvement trend.Small filters focus on local smooth regions that provide little information for restoration.
- Network depth: Residual CNNF performance improves with depth but reaches diminishing returns after d is larger than 4.Deeper three-subnetwork models generally perform better, while greater depth also induces additional computational cost.
- Limitations: The model can incorrectly treat small-scale guidance details as noise because the depth-based training data is mostly smooth and lacks many spatial details.This failure appears in textures such as porcelain spots and carpet strips in flash/non-flash filtering.
6 CONCLUSIONS
The paper concludes with a CNN-based joint filter that combines target and guidance features to transfer consistent structures while suppressing inconsistent ones. It generalizes from training data in one modality to diverse filtering applications and remains computationally efficient.
- 6 CONCLUSIONS: The CNN extracts features from both target and guidance images, then selectively transfers salient structures consistent across the pair.The architecture uses CNNT and CNNG feature extraction followed by concatenation into CNNF.
- 6 CONCLUSIONS: The learned filter's effectiveness depends on preserving fine guidance details, which remain vulnerable to over-smoothing.The reported failure cases involve small-scale textures being treated as noise.
- 6 CONCLUSIONS: The model generalizes across RGB/NIR and flash/non-flash pairs and performs favorably across denoising, joint upsampling, and texture-structure separation.The paper also reports computational efficiency across these applications.