PaperlayerView exact revision on arXiv

Source-linked AI summary

FMA-Net++: Motion- and Exposure-Aware Joint Video Super-Resolution and Deblurring

Geunhyuk Youk, Jihyong Oh, Munchurl Kim

arXiv:2512.04390v2cs.CVcs.AI

TL;DR

VSRDB must restore sharp high-resolution video while accounting for blur that varies with motion and frame-wise exposure. FMA-Net++ combines parallel sequence-level temporal modeling with exposure-aware degradation estimation, and reports state-of-the-art results on exposure-varying benchmarks and strong performance on GoPro and real-world videos.

  • Problem

    VSRDB methods commonly assume fixed exposure, although exposure variation changes motion-blur severity across frames and complicates joint restoration.

    Source: §1 · p. 2

  • Method

    FMA-Net++ stacks HRBA blocks for parallel sequence-level modeling, conditions features on per-frame exposure with ETM, and separates degradation-prior prediction from restoration.

    Source: §1 · p. 3

  • Results

    Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on REDS-ME, REDS-RE, and GoPro, while outperforming recent methods in restoration quality and inference speed.

    Source: §1 · p. 3

  • Takeaways & Limitations

    REDS-ME and REDS-RE provide benchmarks for systematic VSRDB evaluation under controlled exposure-duration variation.

    Source: §1 · p. 3

    The formulation and REDS-ME/RE benchmarks focus on blur-extent variation from temporal integration, not other concurrent camera factors such as gain, white balance, clipping, or sensor noise.

    Source: S6 Limitations and Future Work · p. 28

Abstract

from arXiv · show

Joint video super-resolution and deblurring (VSRDB) requires both efficient long-range temporal modeling and robustness to frame-wise exposure-duration variation, which changes the extent of motion blur across video frames. We propose FMA-Net++, a non-recurrent, sequence-level framework built from Hierarchical Refinement with Bidirectional Aggregation (HRBA) blocks. By stacking HRBA blocks, FMA-Net++ processes video frames in parallel while hierarchically expanding the temporal receptive field, avoiding the limited temporal receptive field of sliding-window designs and the sequential bottleneck of recurrent ones. To handle exposure-duration-dependent blur, we introduce an Exposure Time-aware Modulation (ETM) layer that conditions HRBA features on exposure embeddings from an Exposure Time-aware Feature Extractor (ETE). The conditioned features guide an exposure-aware flow-guided dynamic filtering module to predict motion- and exposure-aware degradation kernels. FMA-Net++ decouples degradation learning from restoration: the former predicts degradation priors and the latter exploits them for efficient high-resolution restoration. To evaluate VSRDB under controlled exposure-duration variation, we introduce the REDS-ME (multi-exposure) and REDS-RE (random-exposure) benchmarks. Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on these benchmarks. It further shows strong out-of-distribution performance on GoPro and challenging real-world videos, while outperforming recent methods in both restoration quality and inference speed.

1 Introduction

FMA-Net++ addresses joint VSRDB under frame-wise exposure variation and temporal-modeling constraints with a sequence-level, exposure-aware architecture. It reports state-of-the-art restoration results on exposure-varying benchmarks and strong performance on GoPro and real-world videos.

1 Introduction

  • Motion determines blur's spatial pattern while exposure controls its temporal extent and intensity, and changing exposure makes blur severity vary within a sequence.Existing restoration methods often assume fixed exposure, while sliding-window and recurrent temporal models have limited context or sequential bottlenecks.Source: §1 · p. 2
  • Stacked HRBA blocks enable sequence-level parallel processing and expand temporal context; ETM conditions features on per-frame exposure embeddings to guide motion- and exposure-aware kernel estimation.Degradation learning predicts priors that the restoration network uses to restore sharp HR frames efficiently.Source: §1 · p. 3
  • Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on REDS-ME, REDS-RE, and GoPro, while outperforming recent methods in restoration quality and inference speed.Real-world robustness is also assessed with qualitative results and no-reference metrics.Source: §1 · p. 3

2.1 Joint Video Super-Resolution and Deblurring

  • Joint VSRDB remains sparsely addressed; FMA-Net++ replaces FMA-Net's sliding-window design and fixed-exposure assumption with sequence-level restoration and joint motion/exposure degradation modeling.The approach uses standard RGB inputs rather than requiring auxiliary event streams.Source: §2.1 · p. 4, §2.3 · p. 5

2.2 Temporal Modeling in Video Restoration

  • Sliding-window models constrain long-range context, recurrent models process sequentially, and transformer approaches can impose substantial compute and memory costs.These trade-offs motivate sequence-level backbones that hierarchically expand temporal receptive fields while retaining efficient parallel processing.Source: §2.2 · p. 4

2.3 Exposure Time-Aware Restoration

  • Prior exposure-aware restoration work does not explicitly model joint motion and exposure effects in VSRDB, while event-dependent designs limit practicality for standard RGB video.ETM injects per-frame exposure embeddings into temporal features, which guide FGDF to estimate motion- and exposure-aware degradation kernels.Source: §2.3 · pp. 4–5

3 Method

FMA-Net++ models exposure- and motion-dependent blur with a learnable degradation formulation, then separates degradation-prior estimation from HR restoration. Its HRBA backbone processes frames in parallel while hierarchically extending temporal context.

3.1 Problem Formulation

  • The blur kernel is modeled as a learnable approximation that jointly depends on exposure time and motion, capturing spatio-temporally varying degradation.The physical process integrates the moving sharp signal over each frame’s exposure interval before spatial downsampling.Source: §3.1 · p. 5

3.4 Degradation Learning and Restoration Networks

  • NetD estimates degradation priors and exposure-aware FGDF applies them, while NetR uses the priors to restore the sharp HR sequence.NetR refines features with HRBA blocks and predicts a high-frequency residual added to the upsampled blurry LR input.Source: §3.2 · p. 6, §3.4 · pp. 8–9

3.3 Hierarchical Refinement with Bidirectional Aggregation

  • Stacked HRBA blocks process the sequence in parallel and aggregate increasingly distant past and future frames to expand temporal context hierarchically.This avoids the limited temporal receptive fields of sliding-window methods and sequential dependencies in recurrent schemes.Source: §3.3 · pp. 6, 8, §3.2 · p. 6
  • Each HRBA refinement updates multiple flow-mask motion hypotheses, and exposure-conditioned features enable dynamic exposure adaptation across refinement stages.The flow-mask pairs combine optical flow with occlusion masks; multiple hypotheses can improve robustness under severe blur.Source: §3.3 · p. 7

3.5 Training Strategy

  • NetD uses reconstruction, warping, and RAFT-based flow supervision, then NetD and NetR are jointly fine-tuned end-to-end with a final-output restoration loss.RAFT provides training-time pseudo-supervision and is not required at inference.Source: §3.5 · pp. 9–10

4 Experiments

Experiments evaluate FMA-Net++ on controlled exposure variations, an unseen dataset, and real-world videos. The reported results show strong restoration performance, efficiency, and generalization.

4.1 Experimental Setup

  • REDS-ME tests five exposure levels, while REDS-RE mixes those levels across frames; evaluation includes challenging 5:4 and 5:5 exposures, GoPro, and real-world video.Restoration quality uses PSNR and SSIM, temporal consistency uses tOF, and real-world video uses no-reference metrics including NIQE and MUSIQ.Source: §4.1 · p. 10

4.2 Comparisons with State-of-the-Art Methods

  • For ×4 VSRDB on REDS4-ME at exposure levels 5:4 and 5:5, FMA-Net++ outperforms the second-best model, FMA-Net*, and is over 5.2× faster than RVRT*.For a 10-frame sequence at 180×320 resolution, FMA-Net++ requires 6.2 GB, compared with VRT’s 20.5 GB.Source: Table 1 · p. 11, §4.2 · p. 11
  • On REDS-RE’s dynamic exposure transitions and the unseen GoPro dataset, FMA-Net++ achieves the best performance across all reported metrics.The REDS-RE advantage widens compared with REDS-ME; GoPro has different scene and motion statistics from REDS-ME.Source: §4.2 · p. 11
  • Real-world comparisons report sharper details, cleaner edges, more legible text, fewer artifacts, and the best NIQE/MUSIQ perceptual quality.The comparisons use conventional RGB inputs and the model was trained solely on synthetic data.Source: §4.2 · p. 12

5 Ablation Study

The ablations examine temporal architecture, exposure guidance, and core restoration components. They report benefits from hierarchical modeling and exposure-aware conditioning, including on varying-exposure inputs.

5.1 Effectiveness of Hierarchical Architecture

  • On REDS4-ME-5:5, HRBA improves all metrics over the sliding-window variant and improves PSNR and tOF over the recurrent variant.The hierarchical variant also has more stable training and a modest speed advantage over the recurrent approach; the authors suggest its temporal-consistency gain may stem from mitigating gradient vanishing.Source: Table 3 · p. 13, §5.1 · p. 13

5.2 Effectiveness of Exposure-Aware Modeling

  • On REDS-RE, ETE raises PSNR from 29.88 to 30.13 and lowers tOF from 1.399 to 1.360 versus a capacity-matched baseline; the GoPro gain is +0.64 dB.Increasing the w/o-ETE baseline’s capacity absorbs most of the in-distribution REDS4-ME PSNR gain, but does not close the REDS-RE gap.Source: §5.2 · pp. 13–14, Table 4 · p. 14
  • The contrastive ETE outperforms frame-difference, classification-pretrained, and ordinal-regression exposure-conditioning alternatives on REDS-RE; its features cluster by exposure and its predicted kernel adapts spatial spread to guidance.The bidirectional test shows supplied guidance changes kernel spread for both severely blurred 5:5 and mildly blurred 5:1 inputs.Source: Figure 7 · p. 13, Table 5 · p. 14, §5.2 · p. 14
  • Frame-wise guidance corruption degrades both PSNR and tOF on REDS-RE, with larger drops on exposure-transition frames; random wrong-level guidance performs worse than removing guidance.The comparison keeps the input sequence fixed and corrupts only the guidance.Source: Table 6 · p. 14, §5.2 · p. 14
  • For fixed REDS4-ME-5:5 inputs, performance declines as exposure guidance moves from the correct level but remains stable even with the farthest 5:1 guidance.The authors attribute this stability to HRBA using strong spatio-temporal evidence rather than relying exclusively on ETE.Source: §5.2 · pp. 14–15, Table 7 · p. 15

5.3 Effectiveness of HRBA and Core Components

  • Increasing stacked HRBA blocks from one to two and then four progressively improves PSNR and tOF, while the full design’s features become sharper and more structurally aligned.This ablation tests hierarchical refinement by varying the number of blocks, M.Source: Table 8 · p. 15, §5.3 · p. 15
  • Replacing Degradation-Aware attention with standard SFT modulation significantly degrades performance, supporting the use of estimated degradation priors in targeted restoration.Source: §5.3 · p. 15

6 Conclusion

FMA-Net++ addresses joint VSRDB under unknown, dynamically varying exposure with parallel sequence-level temporal modeling and per-frame exposure conditioning. Experiments report state-of-the-art performance, efficiency, and robustness across the proposed benchmarks, GoPro, and real-world videos despite synthetic-only training.

6 Conclusion

  • FMA-Net++ uses HRBA blocks for sequence-level temporal modeling with efficient parallel processing.Source: §6 · p. 15
  • ETM injects per-frame exposure conditioning so exposure-aware FGDF can predict degradation kernels capturing coupled motion and exposure effects.Source: §6 · p. 15
  • Experiments on REDS-ME, REDS-RE, GoPro, and real-world videos report state-of-the-art results in performance, efficiency, and robustness after synthetic-only training.Source: §6 · p. 15

FMA-Net++: Supplementary Material

The supplementary material provides implementation and training details, benchmark synthesis procedures, further analyses, qualitative comparisons, and a discussion of limitations and future directions.

FMA-Net++: Supplementary Material

S1 Detailed Training Strategy

ETE is pretrained to represent exposure levels discriminatively, then frozen while NetD and NetR are optimized. Its contrastive objective treats features with matching exposure labels as positives.

S1 Detailed Training Strategy

  • ETE is pretrained with supervised contrastive learning, using discrete exposure levels as pseudo-labels to learn an exposure-discriminative representation space.The ETE is then frozen before subsequent optimization of NetD and NetR.Source: S1 Detailed Training Strategy · p. 20
  • Positive samples are mini-batch features sharing the anchor feature’s exposure label, and α is the temperature-scaling parameter.Source: S1 Detailed Training Strategy · p. 20
  • Freezing pretrained ETE establishes a stable exposure reference space and prevents representation drift during subsequent NetD and NetR optimization.Source: S1 Detailed Training Strategy · p. 20

S2 Implementation Details

FMA-Net++ uses a three-stage training schedule and is trained on 10-frame sequences before evaluation on full-length videos. The implementation uses four HRBA blocks in each network and nine multi-flow-mask pairs.

S2 Implementation Details

  • The three training stages use Adam on four NVIDIA A6000 GPUs: ETE pretraining, NetD training, then joint NetD and NetR training.ETE uses batch size 128, learning rate 0.01, and α = 0.5; the latter stages use batch size 8 and an initial learning rate of 2 × 10^-4, halved at 70%, 85%, and 95% of 280K iterations.Source: S2 Implementation Details · p. 20
  • FMA-Net++ is trained on 10-frame inputs with 64 × 64 spatial patches and evaluated on full-length videos, using an SR scale factor of 4.Source: S2 Implementation Details · pp. 20–21
  • Both NetD and NetR use four HRBA blocks, nine multi-flow-mask pairs, and degradation kernels of size 20; the loss coefficients are λ1 = λ2 = 10^-4 and λ3 = 0.1.The multi-attention block uses Restormer’s MDTA and GDFN modules.Source: S2 Implementation Details · p. 21

S3 Details of REDS-ME and REDS-RE Benchmarks

REDS-ME and REDS-RE are REDS-derived benchmarks for evaluating VSRDB under controlled exposure variation. REDS-ME varies exposure across synthesized sequences, while REDS-RE varies it over time within a sequence.

S3 Details of REDS-ME and REDS-RE Benchmarks

S3.1 Synthesis of Multi-Exposure Sequences (REDS-ME)

  • REDS-ME synthesizes blurry low-resolution frames by averaging consecutive interpolated high-frame-rate frames and then downsampling, isolating exposure duration as the controlled blur variable.Other camera factors, including gain, white balance, clipping, and sensor noise, are outside the benchmark’s scope.Source: S3.1 Synthesis of Multi-Exposure… · p. 21
  • REDS-ME varies five exposure levels; training uses all five, while evaluation uses the two most challenging levels, 5:4 and 5:5.The levels span from 5:1, the shortest exposure, to 5:5, the longest exposure.Source: S3.1 Synthesis of Multi-Exposure… · p. 22

S3.2 Synthesis of Random-Exposure Sequences (REDS-RE)

S4 Further Ablation Studies

The further ablations examine exposure embeddings, progressive HRBA refinement, multi-flow-mask pairs, exposure-aware filtering, and loss coefficients. They report representation separability and improvements associated with the tested design choices.

S4.1 Quantitative Evaluation of ETE Embeddings

S4.2 Progressive Feature Refinement in HRBA

S4.3 Effect of the Number of Multi-Flow-Mask Pairs

S4.4 Effect of Exposure-Aware FGDF

S4.5 Analysis of Loss Functions

  • Both warping and RAFT supervision losses are essential; λ1 = λ2 = 10^-4 gives the best trade-off and highest performance across all metrics.Removing RAFT supervision causes a notable performance drop, while too-large or too-small warping-loss weights impair restoration quality.Source: Table S3 · p. 26, S4.5 Analysis of Loss Functions · p. 26

S5 Additional Qualitative Results

The supplementary comparisons show FMA-Net++ restoring sharper, more temporally consistent structures than the sliding-window FMA-Net∗ baseline in challenging scenes. Additional examples cover REDS4-ME-5:5, GoPro, and real-world smartphone videos with exposure changes beyond the discrete training anchors.

S5.1 Qualitative Comparison with FMA-Net

  • FMA-Net∗’s narrow T = 3 window produces over-smoothed textures and distorted faces, while FMA-Net++ restores sharper, temporally consistent high-frequency details.The comparison focuses on severe motion blur and low spatial redundancy, where broader temporal context is critical.Source: S5.1 Qualitative Comparison with FMA-Net · p. 27

S5.2 Additional Visual Comparisons and Real-World Generalization

  • Additional visual comparisons cover challenging REDS4-ME-5:5 and GoPro test-set scenes.These examples complement the main-paper results.Source: S5.2 Additional Visual Comparisons and… · p. 27
  • Real-world smartphone examples show FMA-Net++ restoring videos with continuous exposure changes and non-uniform blur beyond its discrete training anchors.The learned exposure-aware feature space may interpolate between nearby exposure conditions, helping handle intermediate unseen states.Source: S5.2 Additional Visual Comparisons and… · p. 28

S6 Limitations and Future Work

The supplementary material adds visual comparisons on synthetic test sets and real-world smartphone videos. The authors delimit the current formulation and benchmarks to blur-extent variation from temporal integration, leaving other camera factors for future work.

S6 Limitations and Future Work

  • The formulation and REDS-ME/RE benchmarks address blur-extent variation from temporal integration, not other entangled camera factors.Unmodeled factors include ISO/gain, white balance, clipping, dynamic range, heavy sensor noise, and spatially varying lighting.Source: S6 Limitations and Future Work · p. 28
  • Additional qualitative comparisons include severe-motion-blur, complex-texture scenes from REDS4-ME-5:5 and GoPro.The figure reports sharper details, cleaner edges, and suppressed motion artifacts relative to existing methods.Source: Figure S6 · p. 29
  • Smartphone examples include continuous exposure changes and non-uniform blur beyond the discrete synthetic anchors used in training.FMA-Net++ restores these videos well and achieves favorable NIQE/MUSIQ scores in the examples.Source: Figure S7 · p. 30

Related papers

Loading 2512.04390v2…