Source-linked AI summary
A Unified Approach of Multi-scale Deep and Hand-crafted Features for Defocus Estimation
Jinsun Park, Yu-Wing Tai, Donghyeon Cho, In So Kweon
TL;DR
Defocus is difficult to estimate from a single image because its amount varies spatially and different defocus levels can look similar. The paper combines hand-crafted and CNN features, estimates defocus on sparse multi-scale edge patches, and propagates a sparse map with image guidance. It reports better results than state-of-the-art methods quantitatively and qualitatively, while noting occasional errors in large homogeneous areas.
Problem
Defocus estimation is challenging because defocus varies spatially and regions with different defocus amounts can have similar appearances.
Source: §1 · p. 1Method
The method concatenates hand-crafted and CNN features for classification, estimates defocus on sparse multi-scale edge patches, and propagates the resulting map using edge-preserving image guidance.
Source: §1 · p. 1, §4.2 · p. 5Results
The authors report better quantitative and qualitative results than state-of-the-art methods.
Source: §5.3 · p. 7Takeaways & Limitations
The paper demonstrates potential applications including segmentation, blur magnification, all-in-focus image generation, and 3-D estimation.
Source: §6 · p. 8The algorithm occasionally estimates incorrect defocus values in large homogeneous areas because those regions lack strong edges for estimation.
Source: §6 · p. 8
Abstract
from arXiv · showhide
In this paper, we introduce robust and synergetic hand-crafted features and a simple but efficient deep feature from a convolutional neural network (CNN) architecture for defocus estimation. This paper systematically analyzes the effectiveness of different features, and shows how each feature can compensate for the weaknesses of other features when they are concatenated. For a full defocus map estimation, we extract image patches on strong edges sparsely, after which we use them for deep and hand-crafted feature extraction. In order to reduce the degree of patch-scale dependency, we also propose a multi-scale patch extraction strategy. A sparse defocus map is generated using a neural network classifier followed by a probability-joint bilateral filter. The final defocus map is obtained from the sparse defocus map with guidance from an edge-preserving filtered input image. Experimental results show that our algorithm is superior to state-of-the-art algorithms in terms of defocus estimation. Our work can be used for applications such as segmentation, blur magnification, all-in-focus image generation, and 3-D estimation.
1. Introduction
Defocus estimation from a single image is challenging because defocus varies spatially and different blur levels can look similar, while relying only on strong edges can cause overconfidence and misestimation. The paper combines complementary hand-crafted and CNN features with sparse, multi-scale edge sampling and propagation to produce a full defocus map.
1. Introduction
- Defocus estimation is challenging because defocus varies spatially and regions with different defocus amounts can have similar appearances, while strong-edge-only estimates may cause overconfidence and misestimations.The paper motivates a more reliable and robust defocus descriptor.Source: §1 · p. 1
- Hand-crafted features capture frequency-domain power, gradient distributions, and image-patch singular values, while CNN features are learned from millions of in-focus and blurred patches.The features are concatenated and fed to a fully connected neural network classifier to estimate defocus.Source: §1 · p. 1
- The method estimates defocus on multi-scale patches from strong edges, then propagates those estimates into homogeneous regions using an edge-preserving filtered input image.The paper identifies homogeneous regions as ambiguous because they can look similar whether in focus or blurred.Source: §1 · p. 1
2. Related Work
Prior defocus-estimation methods use varied edge, statistical, and learned cues, but the cited work reports weaknesses including noise sensitivity, imprecise localization, and difficulty in homogeneous regions. This paper applies deep features alongside hand-crafted features, which capture local and global patch information, and reports improved quality and accuracy.
Defocus Estimation
- Prior defocus methods have reported noise sensitivity, imprecise blurry-region localization, weak performance in homogeneous regions, or limited robustness to large blur.These limitations are reported for the methods of Tai and Brown, Liu et al., Shi et al., and Shi et al., respectively.Source: §2 · p. 2
Neural Networks
- Earlier neural blur approaches include discretized motion-blur kernel estimation, blur identification with multivalued neurons, and two-stage blur-type and parameter classification.Yan and Shao rely only on frequency-domain features.Source: §2 · p. 2
Our Work
- The paper applies deep features learned from training data with different amounts of defocus blur instead of relying only on hand-crafted features.Its deep features are local, while the hand-crafted features capture both local and global patch information.Source: §2 · p. 2
- The paper reports that its approach significantly outperforms previous defocus-estimation work in quality and accuracy.It presents deep features as a complement to hand-crafted features, which capture both local and global patch information.Source: §2 · p. 2
3. Feature Extraction
The section combines hand-crafted frequency, gradient, and singular-value features with CNN features for defocus estimation. It uses edge-dependent patch scales to address ambiguity and reports improved classification when the feature types are combined.
3. Feature Extraction
- The method concatenates frequency, gradient, and singular-value hand-crafted features with a CNN feature to form the final defocus feature.The hand-crafted features describe grayscale patches, while the CNN processes color patches in RGB.Source: §3 · p. 2, §3.6 · p. 4
3.1. Multi-scale Patch Extraction
- Small patches from strong edges and large patches from weak edges reduce patch-scale ambiguity in defocus estimation.The strategy reflects that small weak-edge patches lack defocus information, while small strong-edge patches retain it.Source: §3.1 · pp. 2–3
3.2. DCT Feature
- The DCT feature measures frequency-power distribution; its sharp–blurry difference exceeds those of DFT and DST in the reported comparison.DCT energy compaction supports examining high-frequency components at higher resolution than DFT or DST.Source: §3.2 · p. 3
3.3. Gradient Feature
- The normalized gradient histogram is an effective discriminator between sharp and blurry patches.The feature is computed from Sobel-filtered patch gradients.Source: §3.3 · p. 3
3.4. SVD Feature
- Sharp patches show a longer tail in the SVD feature, indicating that more image details are preserved.The feature uses scaled singular values; larger-index nonzero singular values help indicate detail.Source: §3.4 · p. 4
3.5. Deep Feature
- The CNN feature is learned from training data and complements hand-crafted features with color and cross-channel information.Its activations are reported to be proportional to input-patch sharpness.Source: §3.5 · p. 4
3.6. Our Defocus Feature
- Combining all hand-crafted and deep features (fB) enhances classification; removing a hand-crafted feature drops accuracy by approximately 1–3%.The reported accuracies for [fD, fS, fC] and [fG, fS, fC] are 93.25% and 91.10%, respectively; single-scale extraction reduces fB accuracy to 91.00%.Source: §3.6 · p. 4, Table 1 · p. 4
4. Defocus Map Estimation
The method converts patch-level defocus classifications into a full map: confidence-aware filtering cleans sparse estimates, then guided matting-Laplacian propagation fills the map.
4.1. Neural Network Classifier
- Three fully connected layers (300-150-11 neurons) classify feature vectors; labels map to Gaussian-kernel σ values, while probabilities form the confidence map.The final layer uses softmax.Source: §4.1 · p. 5
4.2. Sparse Defocus Map
- Probability-joint bilateral filtering rejects outliers and noise and regularizes the sparse map using classification confidence; Figure 6 shows maps before and after filtering.The probability weight reduces unwanted effects from outliers in neighborhoods with low confidence.Source: §4.2 · p. 5, Figure 6 · p. 5
4.3. Full Defocus Map
- The matting Laplacian [18] uses guidance image IG to propagate the filtered sparse map into a full defocus map, with the solution formulated as a least-squares problem.A diagonal-matrix term depends on whether filtered sparse-map values are nonzero and on user parameter γ.Source: §4.3 · p. 5
5. Experiments
The experiments evaluate classifier training and scale encoding, then compare defocus estimation on a blur-detection dataset and demonstrate several applications. The results report stronger performance than state-of-the-art methods and show complementary contributions from deep and hand-crafted features.
5.1. Classifier Training
- Training data begins with approximately 1M multi-scale strong-edge patches treated as sharp patches, which are convolved with synthetic blur kernels to create blurry patches.The paper also reports selecting 300 sharp images from ILSVRC training data.Source: §5.1 · pp. 5–6
5.2. Scale Encoding
- Scale encoding assigns separate feature positions to small- and large-scale inputs, using zeros in the other scale’s positions; adding scale-specific constants lets the classifier converge.Training classifiers separately by scale was described as inefficient and potentially inconsistent.Source: §5.2 · p. 6, Figure 7 · p. 6
5.3. Blur Detection Dataset
- The evaluation uses a 704-image dataset with manually segmented binary blurry-region masks and compares the algorithm against,, and.Binary segmentation is obtained by thresholding the full defocus map.Source: §5.3 · p. 6
- 0.1572 on RelOrder exceeds ’s 0.13932, and ’s segmentation accuracy is 53.30%; the paper reports stronger quantitative and qualitative results overall.Figure 8 compares segmentation accuracy and precision-recall, while Figure 9 presents defocus maps and binary masks.Source: Figure 8 · p. 6, §5.3 · p. 7, Figure 9 · p. 7
- Deep and hand-crafted features each produce useful defocus maps, and their concatenation compensates for misclassifications in either feature alone.The paper attributes the combined feature’s gains to complementary roles of the deep and hand-crafted features.Source: §5.3 · p. 7, Figure 10 · p. 7
- Defocus magnification amplifies background blur to make foreground objects more prominent.Figure 11 illustrates this application.Source: §5.4 · p. 8, Figure 11 · p. 8
- For all-in-focus generation, per-pixel defocus values determine Gaussian kernels for deblurring, and the paper reports considerable restoration of blurry regions.The method uses a hyper-Laplacian prior for non-blind deconvolution.Source: §5.4 · p. 8, Figure 12 · p. 8
- Defocus values can serve as pseudo-depth when objects lie on the same side of the focal plane, and the resulting single-image depth map appears reasonable against methods using multiple-image correspondences.The comparison includes light-field depth methods and a single-image depth method.Source: §5.4 · p. 8, Figure 13 · p. 8
6. Conclusion
The conclusion presents a unified deep and hand-crafted feature approach with edge-strength-dependent multi-scale patches and edge-guided propagation. It notes occasional errors in large homogeneous regions and identifies fully convolutional networks as future work.
6. Conclusion
- The approach combines complementary hand-crafted and deep features with a neural-network classifier, multi-scale patches selected by edge strength, and propagation guided by an edge-preserving smoothed image.The paper reports that homogeneous regions are handled through a sparse defocus map and propagation.Source: §6 · p. 8
- Large homogeneous areas can still receive incorrect defocus values because they may contain no strong edge for estimation; random classification seeds can guide propagation there.The authors identify additional random seed points as a simple remedy.Source: §6 · p. 8
- The authors identify fully convolutional network architectures as a direction for future work.Source: §6 · p. 8