Source-linked AI summary
End-to-End United Video Dehazing and Detection
Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, Dan Feng
TL;DR
CNN-based video dehazing has been little explored despite the temporal coherence between consecutive frames. The paper proposes EVD-Net and jointly trains it with video object detection in EVDD-Net, reporting more stable and accurate detection in hazy video.
Problem
CNN-based dehazing had largely focused on single images, leaving end-to-end video dehazing and its use of neighboring-frame temporal coherence underexplored.
Source: Introduction · p. 1Method
EVD-Net dehazes video using temporal fusion, and EVDD-Net concatenates and jointly trains it with a video object detection model.
Source: Introduction · p. 1, Fusion Strategy for Video Dehazing:… · p. 3, Unity Brings Power: Optimizing… · p. 4Results
EVDD-Net produces more stable and accurate detection in hazy video, with evaluations reporting superiority in dehazing quality and detection accuracy on synthetic and real-world datasets.
Source: Introduction · p. 1, Conclusion · p. 7Takeaways & Limitations
Joint optimization makes video dehazing a trainable front end for optimizing high-level task performance in hazy conditions.
Source: Introduction · p. 1, Unity Brings Power: Optimizing… · p. 3
Abstract
from arXiv · showhide
The recent development of CNN-based image dehazing has revealed the effectiveness of end-to-end modeling. However, extending the idea to end-to-end video dehazing has not been explored yet. In this paper, we propose an End-to-End Video Dehazing Network (EVD-Net), to exploit the temporal consistency between consecutive video frames. A thorough study has been conducted over a number of structure options, to identify the best temporal fusion strategy. Furthermore, we build an End-to-End United Video Dehazing and Detection Network(EVDD-Net), which concatenates and jointly trains EVD-Net with a video object detection model. The resulting augmented end-to-end pipeline has demonstrated much more stable and accurate detection results in hazy video.
Introduction
CNN-based single-image dehazing has advanced, but CNN-based video dehazing remains limited. The paper introduces EVD-Net to use temporal coherence and EVDD-Net to jointly train video dehazing with object detection.
Introduction
- CNN-based single-image dehazing has received substantial attention, while video dehazing has seen limited exploration despite its relevance to outdoor applications.Source: Introduction · p. 1
- EVD-Net directly regresses clean images from hazy inputs while using temporal coherence between neighboring frames; the AOD-Net architecture inspired its design.The paper identifies temporal fusion as a central design question and studies alternative strategies.Source: Introduction · p. 1, Figure 1 · p. 1
- EVDD-Net concatenates EVD-Net with a video object detector and jointly trains the pipeline, yielding more stable and accurate detection in hazy video.Source: Introduction · p. 1
Related Work
Prior dehazing methods estimate physical parameters or use end-to-end CNNs, but most single-image models omit video dynamics. Earlier video methods use temporal processing, while video CNN research has explored different structure configurations.
Related Work
- EVD-Net compares three fusion timings: concatenating input frames before the first layer, combining features during K estimation, or concatenating output images.Source: Figure 2 · p. 2
- Earlier dehazing methods estimate atmospheric light and transmission separately or regress transmission with a CNN; AOD-Net instead directly generates clean images end to end.AOD-Net uses a five-layer K-estimation module followed by clean-image generation.Source: Related Work · p. 2
- The scattering formulation relates hazy image I(x) to clean image J(x) through atmospheric light A and transmission t(x), which depends on scattering coefficient β and object-camera distance d(x).Source: Related Work · pp. 1–2
- AOD-Net and the other described single-image methods do not account for video temporal dynamics, while earlier video approaches often correct frame-wise results through post-processing.Source: Related Work · p. 2
- Earlier video CNN work explored structure configurations for super-resolution and classification, spatial alignment, frame accumulation for deblurring, and short- and long-term coherence for style transfer.Source: Related Work · p. 2
End-to-End Video Dehazing Network and Its Unity with Video Detection
EVD-Net extends AOD-Net to video by using temporal information from neighboring frames, and the paper studies where to fuse those frames. The authors then combine dehazing with video detection in a jointly optimized pipeline.
End-to-End Video Dehazing Network and Its Unity with Video Detection
- EVD-Net starts from AOD-Net and exploits neighboring-frame coherence, motivated by slowly changing depth and atmospheric light across consecutive frames.The method investigates three fusion structures, with any number of past and future frames accommodatable.Source: End-to-End Video Dehazing Network and… · pp. 2–3, Fusion Strategy for Video Dehazing:… · p. 3
Fusion Strategy for Video Dehazing: Three Structure Options
- I-level fusion concatenates the input frames before the first convolution, while K-level fusion combines their features during K estimation.J-level fusion keeps frame-wise K estimation separate until the clean-image generation module; K-level fusion assumes depth and atmospheric light change smoothly across neighboring frames.Source: Fusion Strategy for Video Dehazing:… · p. 3
Unity Brings Power: Optimizing Dehazing and Detection as An End-to-End Pipeline in Video
- MF-Faster R-CNN branches the previous, current, and next frames through the first two convolutional layers, then combines them to predict current-frame boxes.The video dehazing and detection pipeline is presented as a joint optimization approach; prior video validation of this unified approach was not known to the authors.Source: Unity Brings Power: Optimizing… · pp. 3–4
- EVDD-Net unites EVD-Net with a video-adapted detector in a tree-like pipeline that is jointly optimized.For the illustrated configuration, five-frame dehazing windows produce three dehazed frames that feed a three-frame detection window, spanning seven input frames overall.Source: Figure 3 · p. 3, Unity Brings Power: Optimizing… · p. 4
Experiment Results on Video Dehazing
EVD-Net’s five-frame, K-level fusion configuration is selected after fusion-structure comparisons, and it outperforms the compared dehazing methods on TestSet V1. Qualitative comparisons report detail-preserving, temporally consistent results on natural and synthetic videos.
Datasets and Implementation
- TestSet V1 uses 18 TUM RGB-D videos: five videos with 100,000 frames for training and 13 non-overlapping clips totaling 403 frames for testing.Depth is refined using the filling algorithm in.Source: Datasets and Implementation · p. 4
Fusion Structure Comparison
- Five input frames with K-level fusion at conv 2 produce the overall peak and are chosen as EVD-Net’s default configuration.Performance saturates and can decline with 7 or 9 frames as distant-frame relevance decays.Source: Fusion Structure Comparison · p. 4
Quantitative Comparison
- Over AOD-Net, EVD-Net gains over 0.3 dB in PSNR and 0.05 in SSIM; over STMRF, the gaps are 2 dB and 0.04.These comparisons are on TestSet V1.Source: Quantitative Comparison · p. 4
Qualitative Visual Quality Comparision
- EVD-Net gives the most visually pleasing, detail-preserving, and temporally consistent results among the compared methods on a challenging natural hazy video.The example is a city-road video recorded at PM 2.5 of 223.Source: Qualitative Visual Quality Comparision · pp. 4, 7
- On synthetic TestSet V1 frames, EVD-Net preserves details and color tones best against the ground truth.Source: Qualitative Visual Quality Comparision · p. 7
Experiment Results on Video Detection
EVDD-Net achieves a significantly boosted MAP over the detection baselines, and real-world examples show accurate, temporally consistent detections. The evaluation uses synthetic hazy videos for a two-category detection task and also includes real-world hazy videos.
Experiment Results on Video Detection
- EVDD-Net is tested on 1,634 frames from 10 hazy videos for car and bike detection, using a synthetic training set derived from ILSVRC2015 VID.Training uses 4,499 frames from 48 hazy videos; several real-world hazy videos are also evaluated.Source: Experiment Results on Video Detection · p. 7
- EVDD-Net is initialized with a trained EVD-Net and uses two-stage training: detection fully connected layers first, then the entire concatenated pipeline.The stages run for 90,000 and 10,000 iterations, respectively.Source: Experiment Results on Video Detection · p. 7
Results and Analysis
- EVDD-Net achieves a significantly boosted MAP over the other baselines on TestSet V2.The comparison includes original and retrained Faster R-CNN, a separately trained EVD-Net cascade, and JAOD-Faster R-CNN.Source: Results and Analysis · p. 7, Comparison Baselines · p. 7
- In four displayed real-world frames, EVDD-Net is the only approach that correctly detects all four cars throughout, including the rightmost car.The passage attributes this to temporal regularizations in both the low-level and high-level parts.Source: Results and Analysis · p. 7
Conclusion
The paper introduces EVD-Net, a fully end-to-end CNN video dehazing model, and EVDD-Net, which jointly trains dehazing with video object detection. It reports extensive evaluations on synthetic and real-world datasets and identifies strengthening video detection as future work.
Conclusion
- EVD-Net is presented as the first CNN-based, fully end-to-end video dehazing model, with fusion strategies thoroughly investigated.Source: Conclusion · p. 7
- EVDD-Net concatenates EVD-Net with a video object detection model and trains the pipeline jointly for detecting objects in hazy video.Source: Conclusion · p. 7
- The models are evaluated on synthetic and real-world datasets, and future work aims to strengthen EVDD-Net’s video detection part.Source: Conclusion · p. 7