PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Burst Super-Resolution

Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte

arXiv:2101.10997v2cs.CV

TL;DR

The paper addresses real-world multi-frame super-resolution, where SISR is limited to learned priors and MFSR must align and fuse shifted noisy frames. It introduces an optical-flow alignment and attention-fusion architecture, together with the BurstSR dataset, and reports strong performance on real-world bursts against single- and multi-frame alternatives.

  • Problem

    SISR is limited to adding high-frequency details through learned priors, while real-world MFSR requires accurate alignment, robust fusion, and suitable benchmark data.

  • Method

    The network aligns deep embeddings from noisy RAW burst frames with dense pixel-wise optical flow and adaptively fuses them to jointly denoise, demosaic, and super-resolve an RGB image.

  • Results

    The approach obtains promising real-world burst SR results and outperforms both single-frame and multi-frame alternatives.

  • Takeaways & Limitations

    BurstSR supplies real-world RAW bursts and high-resolution ground truths for evaluating and training burst super-resolution systems.

Abstract

from arXiv · show

While single-image super-resolution (SISR) has attracted substantial interest in recent years, the proposed approaches are limited to learning image priors in order to add high frequency details. In contrast, multi-frame super-resolution (MFSR) offers the possibility of reconstructing rich details by combining signal information from multiple shifted images. This key advantage, along with the increasing popularity of burst photography, have made MFSR an important problem for real-world applications. We propose a novel architecture for the burst super-resolution task. Our network takes multiple noisy RAW images as input, and generates a denoised, super-resolved RGB image as output. This is achieved by explicitly aligning deep embeddings of the input frames using pixel-wise optical flow. The information from all frames are then adaptively merged using an attention-based fusion module. In order to enable training and evaluation on real-world data, we additionally introduce the BurstSR dataset, consisting of smartphone bursts and high-resolution DSLR ground-truth. We perform comprehensive experimental analysis, demonstrating the effectiveness of the proposed architecture.

1. Introduction

Single-image super-resolution is limited to adding high-frequency details through learned priors, whereas multi-frame super-resolution can exploit shifted observations to reconstruct richer details. This work addresses real-world burst SR with a new architecture, dataset, and alignment-aware training approach.

  • Motivation: MFSR exploits sub-pixel shifts between multiple low-resolution images to obtain different scene samplings and reconstruct higher-quality images than SISR.The setting naturally arises in handheld mobile burst photography because natural hand tremors create small inter-frame offsets.
  • Challenges: Real-world MFSR requires sub-pixel alignment of noisy frames, robust fusion despite alignment errors, and benchmark datasets.These challenges have limited attention to MFSR compared with SISR.
  • Architecture: The proposed network processes noisy RAW bursts and jointly produces denoised, demosaicked, and super-resolved images.It uses dense pixel-wise optical flow to align deep feature representations before fusion.
  • Architecture: Attention-based fusion adaptively merges an arbitrary number of aligned input frames while discarding unreliable or misaligned regions.The module selects informative content from individual images rather than relying on fixed aggregation.
  • Dataset: BurstSR provides 200 real-world RAW bursts with corresponding high-resolution DSLR ground truth for training and evaluation.The dataset is intended as a benchmark and source of training data for MFSR.
  • Evaluation: The approach shows promising real-world burst SR performance and significantly outperforms alternative methods in a user study.Experiments cover synthetic data and the BurstSR test set, alongside an ablation study of key components.

2. Related Work

Prior work spans single-image methods built around diverse representations and losses, while multi-frame methods reconstruct high-resolution outputs by merging multiple observations. Real-world SR research also highlights the difficulty of transferring models trained on synthetic degradations to camera images.

  • Single Image Super-Resolution: SISR research includes frequency-domain, interpolation, sparse-representation, and patch-based methods, followed by deep CNNs with improved architectures and loss functions.The related work places the paper within a broad and mature single-image SR literature.
  • Multi-Frame Super-Resolution: MFSR methods merge multiple aliased images of the same scene rather than relying solely on image priors.Earlier approaches include frequency-domain reconstruction and iterative back-projection.
  • Learning Real-World Super-Resolution: Synthetic LR-image training often generalizes poorly to real-world images, motivating degradation modeling and camera-specific approaches.The paper extends this real-world focus to burst SR from handheld cameras.

3. Burst Super-Resolution Network

The network converts arbitrary-sized noisy RAW bursts into high-resolution RGB images by independently encoding frames, aligning their features with dense optical flow, and adaptively fusing them before decoding.

  • Network overview: The network accepts multiple noisy RAW LR images from a burst and jointly produces an HR RGB image, performing denoising, demosaicking, and super-resolution.The burst frames contain small inter-frame offsets that provide multiple aliased scene observations for super-resolution.
  • Encoder: Each burst image is independently encoded into a deep feature representation, with RAW Bayer pixels packed into four channels before convolutional processing.The encoder expands the representation to a high-dimensional embedding for more effective multi-frame fusion.
  • Alignment module: Dense pixel-wise optical flow aligns each feature embedding to the first burst image, accommodating both camera motion and scene object motion.The flow vectors warp feature maps to the base-frame coordinates before fusion.
  • Fusion module: An attention-based fusion module predicts element-wise weights from aligned features, flow information, and residuals, then combines the frames into one merged feature map.The design supports arbitrary burst sizes and adapts weighting to image content, noise levels, and alignment errors.
  • Decoder: The decoder transforms the merged feature map into the final HR RGB image using residual processing and sub-pixel convolution.Sub-pixel convolution decodes information distributed across feature channels while upsampling to the desired resolution.

4. BurstSR Dataset

BurstSR is a real-world burst super-resolution dataset pairing handheld smartphone RAW bursts with high-resolution DSLR ground truths. Its natural hand tremors provide the sub-pixel offsets needed for multi-frame reconstruction.

  • BurstSR contains 200 RAW burst sequences with corresponding high-resolution ground-truth images.Each sequence contains 14 RAW images captured with identical camera settings.
  • The bursts were captured using a handheld smartphone, while each ground-truth image was captured with a tripod-mounted DSLR.The smartphone captures introduce natural hand tremors and small random offsets between images.
  • The dataset was collected to support training and evaluation of burst super-resolution on real-world photography.The authors describe it as the first real-world burst super-resolution dataset and plan to release it as a training set and benchmark.

5. Training

Training combines synthetic pre-training with BurstSR fine-tuning to address the scarcity of paired real-world data and differences between synthetic and real camera image formation. Real-data training also accommodates spatial misalignment and sensor color mismatch between bursts and DSLR ground truth.

  • Training pipeline: The model is first pre-trained on synthetic bursts and then fine-tuned on BurstSR for real-world super-resolution.This pipeline addresses the impracticality of collecting large-scale paired real-world data for training from scratch.
  • Synthetic pre-training: Synthetic bursts are generated from sRGB training images by inverse camera processing followed by random translations and rotations.Translations range from [-24, 24] pixels and rotations from [-1, 1] degrees.
  • Real-data preparation: Real-data preparation extracts 160×160 crops with an 80-pixel stride and filters pairs whose normalized cross-correlation is below 0.9.Local homography alignment is estimated between each crop and its corresponding DSLR region before downsampling the ground truth.
  • Real-data training: The training loss corrects spatial misalignment by estimating optical flow and handles sensor color mismatch through global color mapping.The prediction is warped to ground-truth coordinates, mapped into the ground-truth color space, and compared using pixel-wise error.
  • Real-data training: The loss masks incorrectly aligned regions before computing the error between the processed prediction and ground truth.The error map is upsampled to the model-prediction resolution before constructing the mask.
  • Optimization: Training uses 100k synthetic iterations followed by 40k BurstSR fine-tuning iterations with ADAM and burst size 8.The complete training takes 30 hours on a single Nvidia V100 GPU.

6. Experiments

Experiments evaluate the burst super-resolution architecture through synthetic ablations and real-world BurstSR comparisons. The results support explicit alignment, adaptive attention-based fusion, and using multiple frames.

  • Experimental setup: All experiments use a 4× super-resolution factor and include synthetic and BurstSR validation evaluations.The synthetic test set contains 300 bursts and provides accurate high-resolution ground truth.
  • Impact of burst size: 0.76 dB in PSNR: using four input frames significantly outperforms the single-image baseline on the synthetic set.With 14 frames, the improvement reaches 2.67 dB in PSNR, and performance improves consistently with burst size.
  • Fusion analysis: Attention-based fusion achieves the best results on both the synthetic set and BurstSR, while MaxPool and AvgPool perform poorly.Concatenate and RecMerge obtain 37.80 dB and 37.55 dB in PSNR, respectively, on the synthetic set.
  • Weight predictor analysis: The best weight predictor uses feature residuals, base-frame features, and flow features together, indicating complementary information for fusion weights.Using residuals improves performance over using only the input feature, while adding base-frame features improves it further by 0.27 dB in PSNR on the synthetic set.
  • Real-world comparison: The proposed method obtains a 1.81 mean ranking in the user study and is ranked first in 53.6% of comparisons on BurstSR.It also achieves the best PSNR, LPIPS, and SSIM results, outperforming HighRes-net by 1.18 dB in PSNR.

7. Conclusions

The paper addresses real-world multi-frame super-resolution with a new RAW burst dataset and an attention-based network. The approach reports promising real-world results against single-frame and multi-frame alternatives.

  • Dataset: BurstSR contains handheld-camera RAW burst sequences paired with high-resolution ground truths captured using a zoom lens.The dataset is introduced for real-world multi-frame super-resolution.
  • Method: The proposed network adaptively combines multiple input images using attention-based fusion.The method is designed for burst super-resolution rather than single-frame input.
  • Results: The approach obtains promising results on real-world bursts, outperforming both single-frame and multi-frame alternatives.

Supplementary Material

The supplementary material provides additional architectural details and analyses of shifts, training data, training loss, and qualitative comparisons.

  • Supplementary analyses: Section A gives additional details about the network architecture.
  • Supplementary analyses: Section B analyzes the impact of sub-pixel shifts in the input images for multi-frame super-resolution.
  • Supplementary analyses: Section C analyzes the impact of the training dataset on real-world super-resolution.
  • Supplementary analyses: Section D provides qualitative analysis of the training loss used on the BurstSR dataset.

A. Network Architecture

The architecture encodes each RAW frame, incorporates flow information, and predicts adaptive fusion weights for aligned feature embeddings. Its effectiveness depends on exploiting sub-pixel shifts across burst frames.

  • Encoder: The encoder maps packed RAW images into feature embeddings through convolutional and residual-block processing.The supplied architecture description identifies the encoder as the module that produces feature embeddings from packed RAW input.
  • Weight Predictor: The weight predictor combines projected feature embeddings, flow features, and residual features to predict element-wise fusion weights.Flow features are derived from the flow vectors before the combined representation is processed to produce raw fusion weights.
  • Decoder: The decoder processes the merged feature map after fusion to produce the reconstructed output.The supplied passages identify the decoder as operating on the merged feature map, while Figure 6 presents its architecture.
  • Input Shifts: A 1.61 dB PSNR improvement over No Shifts shows that sub-pixel shifts contribute substantially to burst super-resolution performance.The comparison uses bursts with sub-pixel shifts against a baseline whose frames differ only in independent noise.

C. Impact of training dataset

Real-data fine-tuning is important for performance on BurstSR. The best results come from combining synthetic pre-training with subsequent fine-tuning on real data.

  • Real-World Fine-Tuning: Synthetic-only training obtains 44.52 dB PSNR on the BurstSR validation set, whereas training only on BurstSR obtains 47.14 dB.These results compare models trained exclusively on synthetic data or exclusively on the real-world BurstSR dataset.
  • Combined Training: Combining synthetic pre-training with fine-tuning on real data produces the best results.The paper evaluates synthetic-only, BurstSR-only, and combined training strategies on the BurstSR validation set.
  • Training Strategy: Fine-tuning adapts a model trained on synthetic data to differences in the real camera image-formation process.The training pipeline uses synthetic data for initialization and real BurstSR data for adaptation.

D. Impact of our training loss

The proposed training loss explicitly handles spatial misalignment between burst inputs and ground truth. Compared with direct pixel-wise loss, it produces sharper images with better details.

  • Loss Design: The proposed loss aligns network predictions with the ground-truth image to handle spatial misalignments between the input burst and ground truth.The comparison is against direct pixel-wise loss without explicit spatial alignment and color-space correction.
  • Validation Result: Training with the proposed loss generates sharper images with better details than direct pixel-wise loss.This qualitative comparison is reported on the BurstSR validation set.

E. Qualitative Examples

Qualitative comparisons show that the proposed approach reconstructs high-frequency image details more faithfully than the compared methods. The training-loss comparison also highlights sharper details, while color shifts require attention to image structure.

  • Method Comparison: Compared with Single Image, DeepJoint+RRDB, and HighRes-net, the proposed approach best reconstructs high-frequency details on BurstSR examples.The comparison uses visual examples from the BurstSR test set and evaluates fidelity to the high-resolution ground truth.
  • Training Loss: The proposed training loss produces sharper images with better details than direct pixel-wise loss in qualitative validation examples.The networks use different output color spaces, so the comparison emphasizes image details rather than color shifts.
Loading 2101.10997v2…