Source-linked AI summary
Examining the Impact of Blur on Recognition by Convolutional Networks
Igor Vasiljevic, Ayan Chakrabarti, Gregory Shakhnarovich
TL;DR
The paper examines why convolutional networks trained on artifact-free images become unreliable on blurred inputs. It evaluates blur effects and fine-tunes pretrained models with blurred examples, finding substantial recovery, cross-blur robustness, and blur-invariant internal representations, with a larger residual gap for semantic segmentation.
Problem
Convolutional networks are commonly trained and evaluated on high-quality images, leaving limited evidence about recognition reliability when natural captures contain blur.
Method
The paper systematically examines blur in image recognition and semantic segmentation, then fine-tunes pretrained networks using combinations of sharp and blurred images across blur types and scales.
Results
Fine-tuning with blurred examples significantly improves blurry-input performance, generalizes across diverse blur types, and produces internal representations invariant to blur.
Takeaways & Limitations
A single standard architecture can adapt to unknown blur through additional training data, making low-confidence predictions and supporting more reliable vision systems.
Takeaways & Limitations
For semantic segmentation, the sharp–blurred performance gap remains larger after fine-tuning, with greater cost to sharp-image performance because blur smears object boundaries.
Abstract
from arXiv · showhide
State-of-the-art algorithms for many semantic visual tasks are based on the use of convolutional neural networks. These networks are commonly trained, and evaluated, on large annotated datasets of artifact-free high-quality images. In this paper, we investigate the effect of one such artifact that is quite common in natural capture settings: optical blur. We show that standard network models, trained only on high-quality images, suffer a significant degradation in performance when applied to those degraded by blur due to defocus, or subject or camera motion. We investigate the extent to which this degradation is due to the mismatch between training and input image statistics. Specifically, we find that fine-tuning a pre-trained model with blurred images added to the training set allows it to regain much of the lost accuracy. We also show that there is a fair amount of generalization between different degrees and types of blur, which implies that a single network model can be used robustly for recognition when the nature of the blur in the input is unknown. We find that this robustness arises as a result of these models learning to generate blur invariant representations in their hidden layers. Our findings provide useful insights towards developing vision systems that can perform reliably on real world images affected by blur.
1. Introduction
Modern convolutional networks are trained and evaluated on artifact-free images, yet blur common in natural capture settings substantially degrades recognition. The paper shows that brief fine-tuning with diverse blurred examples restores much of the lost performance and supports blur-invariant recognition across tasks and blur types.
- Motivation: Large annotated datasets underpin modern convolutional-network methods, but their images are often selected to be high-quality and free of motion or defocus blur.This creates a disparity between training and evaluation image statistics and photographs encountered in practice.
- Motivation: Blur is common in natural capture, especially when users or devices prioritize automated vision over image quality, and it reduces classification accuracy.The paper focuses on blur from defocus, subject motion, and camera motion.
- Findings: Even moderate blur harms networks trained on sharp images, producing unreliable predictions accompanied by higher-entropy class distributions.Low confidence can help a vision system recognize unreliable inputs and request another image.
- Findings: Fine-tuning for only three epochs on mixed blurry and sharp images recovers most lost accuracy on blurry inputs without changing the network architecture.The result indicates that much of the degradation follows from training only on sharp images rather than an intrinsic lack of information or architectural deficiency.
- Findings: Fine-tuning with diverse blur types improves cross-blur performance, while related findings extend to semantic segmentation for identifying and localizing objects.The network can learn a level of blur invariance, and segmentation accuracy improves in blurred images.
- Implications: Convolutional networks can signal low confidence on out-of-distribution blur and adapt with additional training data, supporting recognition under unknown blur kernels.The paper frames this as a practical design consideration for reliable vision systems in non-ideal capture settings.
2. Background & Related Work
Blur commonly arises from defocus or motion and removes high-frequency detail, while deblurring remains difficult; this paper instead studies recognition on blurred inputs and their robustness to blur variability.
- Blur and recognition: Blur models sensor averaging caused by defocus or motion as a convolution of a sharp image with a blur kernel.Both defocus and motion kernels act as low-pass filters that attenuate high-frequency image detail.
- Blur and recognition: This work focuses on spatially uniform blur, although real images can contain spatially varying mixtures of kernels.Spatial variation may arise from depth differences, rotational motion, or out-of-plane motion.
- Prior work and paper focus: Deblurring unknown blur is an ill-posed, computationally expensive inverse problem, whereas this paper targets recognition directly from blurred images.The goal is to understand and mitigate recognition degradation rather than reconstruct sharp photographs for human viewing.
- Prior work and paper focus: Standard recognition benchmarks mainly characterize performance on similar high-quality images, while prior evaluations report significant accuracy drops under blur.The paper extends this concern to naturally prevalent optical blur rather than synthetic adversarial perturbations.
- Paper findings: The study finds that blur robustness can generalize across some degrees and types, but transfer from radially symmetric defocus to oriented motion kernels is limited.Training with moderate blur helps on severe blur, and defocus and camera-motion training improve performance on each other.
- Paper findings: Models trained with diverse blur kernels achieve nearly the robustness of blur-specific models and learn blur-invariant hidden representations.This supports recognition under unknown blur without requiring image restoration first.
3. Imagenet Classification on Blurred Images using Pre-trained Network Models
The experiments evaluate a sharp-image-trained VGG-16 on ImageNet validation images blurred with simulated defocus, motion, and camera-shake kernels. Accuracy declines as blur increases, and the decline is accompanied by lower-confidence predictions.
- Experimental setup: The study evaluates sharp-image-trained VGG-16 on blurred ImageNet 2012 validation images generated with multiple blur kernels.Images are resized before convolution to keep the effective blur consistent for a chosen kernel.
- Experimental setup: The experiments simulate defocus with disk kernels, uniform motion with horizontal or vertical box kernels, and camera shake with random trajectory kernels.The camera-shake set contains 100 random spline kernels with intensity variation along arbitrary motion trajectories.
- Results: Accuracy drops clearly as blur increases across all tested kernel types.Including the 512 scale makes the decline steeper for larger kernels, while scale 128 generally hurts performance because of training-scale mismatch.
- Results: Blurred inputs shift the correct class from first place to sixth in an example and make the predicted distribution more uniform.The corresponding class probabilities have higher entropy, indicating lower confidence in inaccurate predictions.
- Results: The evaluation compares original and blur-fine-tuned VGG-16 predictions using entropy and cross-entropy on sharp images and defocus-blurred images.Table 2 uses defocus radii 4 and 8 and evaluates the models at scale 256.
4. Robustness to Blur with Fine-tuning
Fine-tuning with diverse blurred and sharp training images substantially restores recognition on blurred inputs while preserving sharp-image performance. Benefits generalize across several blur types and arise alongside increasingly blur-invariant higher-layer features, although scale and blur-distribution choices matter.
- 4.1. Fine-tuning with Different Blur Distributions: Fine-tuning with a mix of sharp and blurred images significantly improves Top-5 accuracy on blurred inputs with negligible penalty on sharp images.At scale 256, radius-8 defocus accuracy rises from 61% to 88%, while sharp-image accuracy drops by less than one-third of a percentage point.
- 4.1. Fine-tuning with Different Blur Distributions: Training on diverse defocus and camera-shake kernels produces substantial cross-type generalization, but defocus training slightly harms uniform motion-blur performance.The differing statistics of oriented motion blur and radially uniform defocus blur help explain the limited generalization in this case.
- 4.1. Fine-tuning with Different Blur Distributions: Training only on a single severe blur can slightly improve that blur’s accuracy but severely degrades performance on sharp images and other blur types.The results therefore favor diverse blur training unless the input blur is accurately known.
- 4.1. Fine-tuning with Different Blur Distributions: Fine-tuning also increases confidence on blurred inputs, systematically lowering the uncertainty of correct predictions.The paper reports higher confidence through reduced average entropy of the predicted class distributions.
- 4.2. Fine-tuning for Different Scales: At scale 512, fine-tuning improves blurred-image accuracy but underperforms scale 256 and costs nearly three percentage points on sharp images.Separate networks at scales 256 and 512 eventually match the scale-256 blurred performance and exceed it by half a percentage point on sharp images.
- 4.3. Blur Invariance in Network Features: Fine-tuned models develop blur-invariant higher-layer representations, shown by low sharp-versus-blurred activation distances at the fifth pooling layer.The original model produces different activations at all layers, whereas the fine-tuned model achieves lower distances at higher pooling layers.
5. Semantic Segmentation on Blurred Images
Blur substantially degrades semantic segmentation, especially because localization depends on low-level detail near object boundaries. Fine-tuning with sharp and blurred images improves both recognition and localization, but leaves a larger sharp–blurred gap and greater sharp-image cost than classification.
- 5. Semantic Segmentation on Blurred Images: Blur harms semantic segmentation by attenuating low-level detail needed to localize objects precisely at pixel level.The evaluation separately measures mean IoU overall and within four pixels of class boundaries.
- 5. Semantic Segmentation on Blurred Images: Fine-tuning with a mix of sharp and blurred VOC training images improves blurred-image segmentation accuracy both overall and near class boundaries.Examples show improved boundary localization and correction of an incorrect class label on blurred inputs.
- 5. Semantic Segmentation on Blurred Images: After fine-tuning, segmentation retains a larger sharp–blurred performance gap and incurs greater sharp-image degradation than classification.The authors attribute this to blur smearing intensities across object boundaries, making object separation harder.
6. Conclusion
The paper concludes that blur-induced unreliability largely reflects poor generalization from sharp-only training rather than missing information or unsuitable architectures. Fine-tuning enables improved blurred-input performance and blur-invariant internal representations, informing deployment in real-world settings with unknown blur.
- 6. Conclusion: Fine-tuning sharp-trained networks with blurred examples significantly improves performance on blurry inputs without architectural modification.The conclusion frames the degradation primarily as a training-distribution generalization problem.
- 6. Conclusion: Standard architectures can handle diverse blurs by learning internal representations that are invariant to blur.The authors connect this representation-level invariance to robustness across blur conditions.
- 6. Conclusion: The findings provide insights for building and deploying vision systems in real-world settings where blur may be present.The authors also expect the analysis to inform work on other imaging non-idealities.
Supplementary Results on ResNet-51
ResNet-51 shows the same blur-related accuracy decline as VGG-16, indicating that this degradation is not specific to the VGG-16 architecture.
- ResNet-51 has slightly higher absolute accuracies than VGG-16 on both sharp and blurred images.
- The comparable blur-related decline confirms that performance degradation from blur is not peculiar to the VGG-16 architecture.
- Increasing defocus blur produces a similar drop in ResNet-51 accuracy to that observed for VGG-16.