Source-linked AI summary
Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions
Wenyu Liu, Gaofeng Ren, Runsheng Yu, Shi Guo, Jianke Zhu, Lei Zhang
TL;DR
Object detectors trained on high-quality images struggle with adverse-weather images, while existing methods have difficulty balancing enhancement and detection or preserving useful latent information. IA-YOLO jointly trains CNN-PP, differentiable image processing, and YOLOv3 so each image can be adaptively enhanced; experiments report effectiveness in foggy and low-light scenarios while retaining normal-condition handling.
Problem
High-quality-image-trained detectors struggle on adverse-weather images, and existing methods may not balance enhancement with detection or preserve latent information useful for detection.
Method
IA-YOLO uses CNN-PP to predict differentiable image-processing parameters and jointly trains the enhancement and YOLOv3 detection components end-to-end with weak supervision.
Results
IA-YOLO adaptively processes normal and adverse-weather images, with experiments demonstrating effectiveness in foggy and low-light scenarios.
Takeaways & Limitations
Adaptive image processing integrated with YOLOv3 supports object detection across normal, foggy, and low-light conditions within the reported experiments.
Takeaways & Limitations
The filters are learned from 256 × 256 downsampled images and then applied to original-resolution inputs, requiring resolution-independent filters.
Abstract
from arXiv · showhide
Though deep learning-based object detection methods have achieved promising results on the conventional datasets, it is still challenging to locate objects from the low-quality images captured in adverse weather conditions. The existing methods either have difficulties in balancing the tasks of image enhancement and object detection, or often ignore the latent information beneficial for detection. To alleviate this problem, we propose a novel Image-Adaptive YOLO (IA-YOLO) framework, where each image can be adaptively enhanced for better detection performance. Specifically, a differentiable image processing (DIP) module is presented to take into account the adverse weather conditions for YOLO detector, whose parameters are predicted by a small convolutional neural net-work (CNN-PP). We learn CNN-PP and YOLOv3 jointly in an end-to-end fashion, which ensures that CNN-PP can learn an appropriate DIP to enhance the image for detection in a weakly supervised manner. Our proposed IA-YOLO approach can adaptively process images in both normal and adverse weather conditions. The experimental results are very encouraging, demonstrating the effectiveness of our proposed IA-YOLO method in both foggy and low-light scenarios.
Introduction
Object detection models trained on high-quality images struggle in adverse weather because weather-specific interference degrades visibility and obscures object information. IA-YOLO addresses this by adaptively enhancing each image for detection using a jointly trained, weakly supervised framework.
- Adverse weather introduces domain shift and weather-specific interference that can cause general object detectors to perform poorly on foggy and dark images.
- Existing approaches either jointly learn enhancement and detection with difficult-to-tune task weighting or preprocess images using conventional enhancement methods.
- IA-YOLO combines a differentiable image processing module with a CNN-based parameter predictor that adaptively estimates filter hyperparameters from each input image.
- The framework jointly trains CNN-PP and YOLOv3 end-to-end, using bounding-box annotations to weakly supervise image enhancement for detection.
- IA-YOLO is designed to handle both normal and adverse weather images, with reported encouraging results on synthetic and real-world foggy and low-light scenarios.
Related Work
Prior work includes region-based and one-stage object detectors, adaptive image enhancement, and preprocessing methods for adverse weather. These approaches motivate an image-adaptive detector because improved visual quality does not necessarily improve detection.
- Object detection methods include region proposal-based systems and one-stage regression detectors such as YOLO and SSD.
- YOLOv3 is used as the baseline detector because it is a practical one-stage architecture with multi-scale prediction.
- Traditional adaptive enhancement methods calculate transformation parameters from image features such as illumination distribution.
- Other enhancement methods use small CNNs to learn image-transformation hyperparameters from global image context.
- Adverse-weather detection commonly preprocesses images with dehazing or enhancement methods, but image-quality improvements may not necessarily benefit detection.
Proposed Method
IA-YOLO uses CNN-PP to predict parameters for differentiable, resolution-independent image filters before YOLOv3 detection. The framework supports adaptive processing across weather conditions through end-to-end training and hybrid data augmentation.
- DIP Module: The pipeline resizes each image to 256 × 256 for CNN-PP, predicts DIP parameters, applies the filters to the original-resolution image, and sends the result to the detector.
- DIP Module: DIP filters must be differentiable for backpropagation and resolution-independent so parameters learned from low-resolution images can process high-resolution inputs.
- DIP Module: DIP contains six adjustable differentiable filters: Defog, White Balance, Gamma, Contrast, Tone, and Sharpen; Defog is specialized for foggy scenes.
- Pixel-wise Filters: Pixel-wise filters map each RGB input pixel Pi = (ri, gi, bi) to an output pixel Po = (ro, go, bo) through differentiable transformations.
- Pixel-wise Filters: Contrast filtering uses an input parameter to linearly interpolate between the original image and the fully enhanced image.
- Pixel-wise Filters: The tone filter is a monotonic piecewise-linear function whose differentiable parameters define its tone curve.
- Sharpen Filter: The sharpening filter adds a scaled difference between the image and its Gaussian-filtered version, with λ controlling sharpening strength.
- Defog Filter: The defog filter is derived from the atmospheric scattering model, where I(x) combines clean radiance J(x) with atmospheric light A and transmission t(x).
Experiments
IA-YOLO is evaluated on synthetic and real-world foggy and low-light datasets against YOLOv3 and other competing methods. Its adaptive processing improves detection across adverse conditions while preserving performance on normal images with modest computational overhead.
- Datasets: The experiments cover foggy and low-light scenarios using synthetic VOC_Foggy and VOC_Dark testbeds, plus real-world RTTS and ExDark datasets.The evaluation also includes normal-weather VOC test data.
- Baselines: IA-YOLO is compared with YOLOv3, preprocessing-based detectors, domain adaptation, and multi-task learning methods.Foggy comparisons include Defog+Detect, DAYOLO, and DSNet; low-light comparisons use Enhance+Detect, DAYOLO, and DSNet.
- Foggy-image results: IA-YOLO outperforms competing methods on the three foggy testing datasets, while avoiding the normal-weather deterioration observed for other methods.The reported metric is mean average precision (mAP).
- Low-light-image results: 0.89, 13.48, and 3.95 percent are IA-YOLO’s improvements over YOLO I on VOC_norm_test, VOC_Dark_test, and ExDark_test, respectively.Against YOLO II, the corresponding improvements are 4.69, 7.12, and 3.34 percent.
- Ablation study: Hybrid training, DIP preprocessing, and image adaptation each improve foggy-image detection, with IA-YOLO performing best when all three modules are used.The adaptive learning module also improves VOC_norm_test performance, unlike YOLOv3 II and fixed-DIP YOLOv3.
- Efficiency: IA-YOLO adds 165K trainable parameters and 13 ms over the YOLOv3 baseline while remaining faster than GridDehaze-YOLOv3 and MSBDN-YOLOv3.The reported overhead is measured on a single Tesla V100 GPU.
Conclusion
IA-YOLO adaptively enhances each input image with a differentiable image-processing module whose parameters are predicted by a small CNN and learned end-to-end with YOLOv3. Hybrid training and parameter prediction enable handling of normal and adverse weather conditions, with stronger performance than previous approaches in foggy and low-light scenarios.
- IA-YOLO adaptively enhances each input image to improve object detection in adverse weather conditions.
- Its differentiable image-processing module removes weather-specific information and restores latent content for YOLO detection.
- A small convolutional neural network predicts the image-processing hyperparameters, and detection loss weakly supervises this prediction network.
- Hybrid training and the parameter prediction network allow IA-YOLO to handle both normal and adverse weather conditions.
- IA-YOLO performs much better than previous approaches in both foggy and low-light scenarios.
Supplementary Material
The supplementary material describes IA-YOLO's learnable defogging mechanism, image-adaptive processing, evaluation settings, visual results, and efficiency. Across foggy and low-light examples, adaptive enhancement reveals details or gradients associated with better detection while adding limited computation.
- Defog Filter Design: The defog filter is motivated by the dark channel prior and introduces a learnable parameter ω controlling the degree of defogging.
- Defog Filter Design: Because the defogging operation is differentiable, ω can be optimized by backpropagation for foggy-image detection.
- Experiments on Foggy Images: The foggy-image experiments compare IA-YOLO with YOLOv3, defogging-and-detection methods, domain adaptation, and multi-task learning.
- Experiments on Foggy Images: IA-YOLO reduces haze and enhances local image gradients, which leads to better detection performance in visual foggy-image examples.
- Learned DIP Module: CNN-PP predicts DIP parameters from each image's brightness, color, tone, and weather-specific information, revealing details useful for subsequent detection.
- Experiments on Low-light Images: The low-light experiments evaluate IA-YOLO against YOLOv3, enhancement-and-detection, DAYOLO, and DSNet on three test datasets.
- Experiments on Low-light Images: IA-YOLO brightens low-light images and further increases contrast, which is described as essential to object detection.