Source-linked AI summary
Multi-Temporal Recurrent Neural Networks For Progressive Non-Uniform Single Image Deblurring With Incremental Temporal Training
Dongwon Park, Dong Un Kang, Jisoo Kim, Se Young Chun
TL;DR
Blind single-image deblurring is challenging for non-uniform blur, and multi-scale processing can lose high-frequency information through down-sampling. The paper constructs intermediate temporal blur levels, trains MT-RNNs incrementally, and progressively deblurs at the original scale. Its MT methods outperform state-of-the-art multi-scale methods on GoPro in PSNR with the smallest number of parameters.
Problem
Blind single-image deblurring is an ill-posed task of recovering a sharp image from blurred input, while typical multi-scale methods may lose high-frequency information during down-sampling.
Method
The paper constructs intermediate temporal blur levels from time-resolved data, trains recurrent networks incrementally from more blurred to less blurred images, and progressively deblurs at the original scale.
Results
MT methods outperform state-of-the-art multi-scale methods on the GoPro dataset in PSNR with the smallest number of parameters.
Takeaways & Limitations
Progressive temporal deblurring provides an alternative to spatial multi-scale recovery for non-uniform single-image deblurring.
Takeaways & Limitations
Ours-Z takes 2.08 seconds over six iterations versus 0.02 seconds for Zhang without iteration, with iteration time increasing substantially after the first iteration.
Abstract
from arXiv · showhide
Multi-scale (MS) approaches have been widely investigated for blind single image / video deblurring that sequentially recovers deblurred images in low spatial scale first and then in high spatial scale later with the output of lower scales. MS approaches have been effective especially for severe blurs induced by large motions in high spatial scale since those can be seen as small blurs in low spatial scale. In this work, we investigate alternative approach to MS, called multi-temporal (MT) approach, for non-uniform single image deblurring. We propose incremental temporal training with constructed MT level dataset from time-resolved dataset, develop novel MT-RNNs with recurrent feature maps, and investigate progressive single image deblurring over iterations. Our proposed MT methods outperform state-of-the-art MS methods on the GoPro dataset in PSNR with the smallest number of parameters.
1. Introduction
Blind single-image deblurring is an ill-posed problem, and conventional multi-scale methods can lose high-frequency information through down-sampling. The paper proposes multi-temporal deblurring with incremental training to progressively recover images while retaining full-resolution information.
- Blind single-image deblurring seeks to recover a sharp image from a blurred input, often without knowing non-uniform blur kernels.
- Deep-learning deblurring methods either estimate non-uniform blur explicitly or directly estimate the original sharp image.
- Multi-scale methods progressively restore images from large blur to small blur because blur becomes relatively smaller at lower spatial scales.
- Typical multi-scale approaches may lose high-frequency information during down-sampling, although strong edges support reliable deblurring.
- Multi-temporal deblurring uses high-speed-camera frame averaging to construct progressively milder blur levels and deblur little by little at the original image scale.
- The paper introduces incremental temporal training, MT-RNNs with recurrent feature maps, and state-of-the-art GoPro performance with the fewest parameters among recent methods.
2. Related Works
Prior deblurring work includes explicit blur estimation, direct sharp-image prediction, and recurrent or multi-scale architectures. The paper distinguishes its approach by incrementally training recurrent deblurring from more blurred to less blurred images.
- Earlier methods estimated uniform or non-uniform blur kernels using optimization, priors, motion models, segmentation, and other parameterized formulations.
- Recent deep networks either predict blur kernels explicitly or directly estimate sharp images from blurred inputs.
- Single-image deblurring research developed multi-scale networks, convolutional LSTM architectures, parameter sharing, skip connections, and hierarchical multi-patch designs.
- Recurrent networks use previous frames or features for sequential or iterative deblurring, including convolutional LSTM-based feature reuse.
- Unlike training directly from the most severe blur to ground truth, the proposed incremental temporal procedure trains from more blurred to less blurred images.
3. Multi-Temporal (MT) Approach
The MT approach replaces down-scaling with progressive deblurring across temporal levels, using intermediate blurred images and recurrent feature maps. MT-RNNs are trained incrementally and iterated to recover sharper images while retaining full-resolution information.
- 3.1. GoPro Dataset: The GoPro dataset averages 7–13 sharp frames to create blur–sharp pairs, with the middle frame selected as ground truth.Temporal level N denotes a blurred image formed from N frames.
- 3.1. GoPro Dataset: More averaged frames produce more severe blurs, motivating temporal-level progression from difficult inputs toward sharper targets.The dataset generation process averages multiple frames, and increasing the frame count increases blur severity.
- 3.2. Dataset For Incremental Temporal Training: Intermediate temporal-level data lets MT-RNN progressively estimate TL 7 to TL 5, TL 5 to TL 3, and TL 3 to TL 1 instead of solving TL 7 directly.For a TL 7 image, intermediate blurred images with TL 1–13 are generated for incremental temporal training.
- 3.2. Dataset For Incremental Temporal Training: As temporal level increases, U-Net PSNR decreases, supporting the conjecture that recovering sharper images through milder blurs is easier.The validation trains U-Net separately on one temporal level for inputs from TL 3–13.
- 3.4. Progressive Deblurring With MT Approach: Unlike multi-scale networks, MT-RNN uses temporal iterations with the original blurred image, the previous estimate, and recurrent feature maps.The model is based on U-Net encoders and decoders, and recurrent features move from the previous decoder into the next iteration’s feature-extraction layer.
- 3.5. Proposed MT-RNN With Feature Maps: Recurrent feature maps improve performance by 0.31dB, while residual learning with the input image improves GoPro PSNR by 0.15dB.The network concatenates recurrent decoder features with encoder features and uses a residual skip connection from the original blurred image.
- 3.4. Progressive Deblurring With MT Approach: PSNR increases through the 5th or 6th iteration and decreases afterward, leading the experiments to use iteration 6.This behavior was observed for networks trained with single temporal-level inputs across TL 7, 9, 11, and 13.
4. Experiments
Experiments evaluate MT-RNN through implementation details, ablations, temporal-step and parameter studies, transfer to other deblurring networks, and GoPro benchmarks. MT-based models achieve strong quantitative and qualitative results while supporting progressive deblurring.
- Implementation Details: The expanded GoPro dataset contains 5,500 training, 110 validation, and 1,200 test images with additional intermediate temporal-level images.The original dataset has 3,214 blurred images, divided into 2,103 training and 1,111 test images.
- Ablation Studies: The ablation study reports that MT improves performance over the conventional MS approach using the same DNN, while recurrent feature maps provide further improvement.Residual learning and a smaller kernel also improve performance, and MT alone outperforms combining MS and MT.
- Temporal Steps and Parameters: Two temporal steps were selected as the best step size after MT outperformed the one-stage approach and larger temporal-step settings performed less favorably.The comparison covered 0, 2, and 4 temporal steps.
- Temporal Steps and Parameters: Increasing parameter size generally improves performance but increases computation, while doubling parameters provides little additional performance over the selected model.Halving the parameter size substantially degrades performance while maintaining similar computation speed.
- MT Applied to Other Deblur DNNs: Applying MT with incremental temporal training to Kupyn and Zhang networks successfully increases performance over both baselines.The Table 4 comparison reports PSNR in dB and parameter counts in millions without recurrent feature maps.
- Benchmark Results: MT-RNN achieves the best GoPro test result among compared methods, reaching 31.15 dB PSNR, and produces visually better results in qualitative comparisons.The benchmark uses 1,111 test images; MT also improves the original Zhang network.
5. Discussion
The discussion reports that MT methods perform strongly with imperfect ground truth, while also identifying iteration-dependent artifacts, computational overhead, and unexplored parameter-sharing schemes.
- Imperfect Ground Truth: About 0.5 dB better PSNR was achieved by MT than MS under known perfect ground truth.With TL 3 ground truth, MT also outperformed conventional MS; the performance gap increased when TL 5 was used.
- Imperfect Ground Truth: MT approaches may be more robust to imperfect ground truth datasets than MS approaches.These are described as preliminary results across imperfect-ground-truth cases.
- Decreasing PSNR Beyond Trained Iterations: PSNR increased during early MT-RNN iterations, usually before iteration 6 or 7, then decreased in later iterations.The reported decrease was associated with tiny center artifacts that grew rapidly as iterations increased.
- Computation Time for Ours-Z: Ours-Z took 2.08 seconds for six iterations, compared with 0.02 seconds for Zhang without iteration.Measured per-image runtime increased from 0.015 seconds at the first iteration to 2.065 seconds by the sixth, and further investigation was noted as necessary.
- Weight Sharing: Partial parameter sharing was not investigated for MT structures, although related MS schemes reduce parameters without degraded performance.The paper identifies partial weight schemes for MT as an interesting direction for future work.
6. Conclusion
The paper presents multi-temporal deblurring as an alternative to multi-scale methods for non-uniform single image deblurring, using incremental temporal training and recurrent processing. On GoPro, the proposed methods outperform state-of-the-art MS methods in PSNR with the smallest number of parameters.
- MT methods use incremental temporal training, constructed MT-level data from time-resolved data, recurrent feature maps, and progressive deblurring over iterations.
- On the GoPro dataset, MT methods outperform state-of-the-art MS methods in PSNR with the smallest number of parameters.