PaperlayerView exact revision on arXiv

Source-linked AI summary

FilterFusion: Prior-Guided Adaptive Filter Fusion for Efficient Video Denoising

Wenlong Yang, Weihang Yuan, Canran Jin, Jiahao Li, Henrik Turbell, Lifeng Sun

arXiv:2610.21505v1cs.CV

TL;DR

Real-world video denoising must address mixed, spatially and temporally varying degradations, while classical filters and deep models each have practical trade-offs. FilterFusion uses low-resolution guidance to adaptively fuse full-resolution filter responses. On CRVD, it reaches 34.72 dB with 0.51 GFLOPs total, and compiled synthesis runs at 55.9 FPS on a single CPU thread at 1080p.

  • Problem

    Real-world video denoising must handle interacting sensor noise, gain amplification, compression artifacts, and motion, while classical methods can be hard to adapt and deep models costly and difficult to analyze or compress.

    Source: §I · p. 1

  • Method

    FilterFusion embeds classical filtering priors in a learnable multi-branch model, using low-resolution guidance and explicit temporal and spatial variance cues to predict fusion weights for full-resolution responses.

    Source: §I · pp. 1–2, §III · p. 2, B. Low-Resolution Weight Prediction · p. 3

  • Results

    On CRVD, FilterFusion reaches 34.72 dB with 0.51 total GFLOPs, and compiled synthesis achieves 17.9 ms per frame (55.9 FPS) at 1080p on a single CPU thread.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    The results support low-resolution control with explicit high-resolution filter synthesis as a practical and effective design principle for deployment-oriented video denoising.

    Source: §V · p. 6

Abstract

from arXiv · show

Real-world video denoising for communication systems must handle mixed degradations such as sensor noise, compression artifacts, and motion-induced inconsistency while remaining deployable under tight latency budgets. Purely deep models often achieve high accuracy but come with significant computational cost, whereas classical filters are robust and efficient but insufficiently adaptive. We present FilterFusion, a prior-guided multi-frame denoising framework that combines temporally-aware learned features with an interpretable bank of denoising branches. The model predicts a low-cost guidance signal and uses it to adaptively mix a compact set of denoising responses at the original resolution. The lightweight design uses only 109K parameters and explicitly injects temporal variance and spatial variance cues into the low-resolution controller. On CRVD (real sensor noise), it reaches 34.72 dB PSNR and 0.9330 SSIM with only 0.18 GFLOPs (network-only). Because the filter-bank operations are simple and regular (Gaussian blur, median, weighted sum), a straightforward compiled C implementation of the synthesis path achieves 17.9ms per frame (55.9 FPS) on a single CPU thread at 1080p---exceeding real-time and 122--252 times faster than lightweight CNN baselines that require over 2 seconds per frame. The results demonstrate a strong quality-efficiency-speed trade-off and a practical route to real-time denoising through low-resolution control and explicit filter-based synthesis.

I. INTRODUCTION

FilterFusion combines a temporal feature backbone with a prior-guided filter bank, using low-resolution control to adaptively mix denoising branches. On CRVD, it achieves 34.72 dB with 0.51 GFLOPs total and a favorable quality-efficiency trade-off.

I. INTRODUCTION

  • Real-world video denoising must handle interacting sensor noise, gain amplification, compression artifacts, and motion that vary across space and time.These mixed degradations make real-world denoising harder than synthetic benchmark restoration.Source: §I · p. 1
  • Classical denoisers are predictable but rely on hand-crafted rules, while deep models can lose performance under distribution shifts and remain difficult to analyze and compress.The paper frames FilterFusion as combining classical filtering robustness with learned content-aware adaptation.Source: §I · p. 1
  • FilterFusion uses a temporal feature backbone and prior-guided filter bank, with a low-resolution head predicting pixel-wise weights to mix complementary denoising responses.The branches include temporal averaging, multi-scale smoothing, median rejection, and reference-frame preservation.Source: §I · p. 1
  • Explicit temporal and spatial variance guidance supports compact, low-resolution weight prediction in the 109K-parameter multi-frame model.The framework embeds classical filtering priors in a learnable multi-branch architecture rather than a monolithic black-box mapping.Source: §I · pp. 1–2
  • On CRVD sensor noise, the model reaches 34.72 dB at 0.51 total GFLOPs (0.18 network-only), requiring 120×–240× less compute than matched-quality full-resolution CNN baselines.Source: §I · p. 2

II. RELATED WORK

Prior work spans robust classical denoisers, learning-based restorers, and hybrids that borrow filtering priors. FilterFusion emphasizes low-resolution adaptive decisions followed by a small bank of full-resolution filter responses.

II. RELATED WORK

  • Classical denoisers use patch redundancy and temporal aggregation, while learning-based methods improve restoration quality but remain computationally costly and less interpretable in mixed-noise scenarios.The cited learning-based examples include DVDnet, FastDVDnet, RViDeNet, and recurrent propagation models –.Source: §II · p. 2
  • Hybrid restoration work combines classical denoising modules with learned predictors; FilterFusion places most adaptive decisions in low-resolution latent space and limits full-resolution processing to interpretable filter responses.The passage situates this deployment-oriented design alongside earlier efficient models using classical filtering, residual learning, and multi-scale processing –.Source: §II · p. 2

III. APPROACH

  • FilterFusion reconstructs a clean reference frame from a temporal window by separating low-cost guidance extraction from full-resolution fusion of denoising responses.The reference frame is usually the last frame in the window.Source: §III · p. 2

A. Problem Formulation

FilterFusion restores the reference luma by mixing filter-bank responses, predicting pixel-wise weights at low resolution while retaining full-resolution synthesis. Its controller combines compact frame inputs with temporal and spatial variance cues to guide adaptation.

A. Problem Formulation

  • The denoised reference luma is formed by mixing full-resolution filter-bank responses using weights predicted at low resolution.The formulation identifies each filter response and its pixel-wise mixing weight, with low-resolution weights used to reduce estimation cost.Source: A. Problem Formulation · p. 2

B. Low-Resolution Weight Prediction

  • About 109K parameters power a lightweight encoder-decoder that predicts fusion weights from the reference frame and temporal mean, each downsampled by 8×.The predictor outputs five response logits, normalized and upsampled for full-resolution synthesis.Source: B. Low-Resolution Weight Prediction · p. 2
  • Temporal variance across the input window supplies a downsampled cue for motion, illumination, or flicker-related inconsistency.The compact predictor uses only two frames, but temporal variance is computed over all frames without explicit motion estimation; larger values favor reference-preserving responses.Source: B. Low-Resolution Weight Prediction · p. 3
  • After 8× downsampling, a low-resolution pixel covers an 8×8 region whose texture complexity is not conveyed by its bilinear mean alone.This leaves the controller unable to distinguish flat areas, where stronger smoothing helps, from textured areas, where over-smoothing should be avoided.Source: B. Low-Resolution Weight Prediction · p. 3
  • Spatial variance is estimated from nine uniformly sampled positions on a 3×3 sub-pixel grid within each low-resolution pixel’s coverage region.The sampling is computed directly in the low-resolution domain.Source: B. Low-Resolution Weight Prediction · p. 3
  • Low spatial variance marks flat regions where stronger smoothing is safe, while high variance marks structures to preserve.Source: B. Low-Resolution Weight Prediction · p. 3
  • The predictor concatenates downsampled reference and temporal-mean frames with spatial and temporal variance cues, addressing within-frame structure and cross-frame consistency separately.The two variance conditions jointly support fusion-map prediction under mixed degradations.Source: B. Low-Resolution Weight Prediction · p. 3

C. High-Resolution Filter-Bank Synthesis

FilterFusion synthesizes the output by mixing five heterogeneous full-resolution responses, using pixel-wise softmax weights that are also reused for chroma.

C. High-Resolution Filter-Bank Synthesis

  • Five full-resolution responses comprise temporal averaging, reference-frame preservation, two Gaussian-pyramid smoothing levels, and a spatial median.The branches provide distinct denoising regimes for pixel-wise selection.Source: C. High-Resolution Filter-Bank Synthesis · p. 3
  • The Gaussian pyramid smooths the temporally averaged signal, while the median branch targets motion and high-frequency regions where averaging ghosts or Gaussian smoothing erases detail.Temporal averaging reduces noise in motion-stable regions, allowing spatial smoothing to remain mild enough to preserve texture.Source: C. High-Resolution Filter-Bank Synthesis · p. 3
  • Softmax-normalized logits make the output a per-pixel convex combination of the candidate filter responses.Source: C. High-Resolution Filter-Bank Synthesis · p. 3
  • The luma-predicted weights are reused to combine a corresponding UV filter bank, adding negligible chroma-synthesis cost while preserving color consistency.For NV12, weights are upsampled to chroma resolution, one quarter of luma resolution.Source: C. High-Resolution Filter-Bank Synthesis · p. 3

D. Training Objective

FilterFusion is trained end-to-end with reconstruction, robust, gradient, and frequency-domain losses. The stated configuration uses Sobel regularization and spectral supervision alongside pixel and Charbonnier terms.

D. Training Objective

  • The training objective combines pixel reconstruction, Charbonnier, Sobel-based regularization, and frequency-domain supervision.The Sobel term suppresses residual high-frequency noise in smooth regions, while the frequency term aligns global spectra and helps preserve valid high-frequency components.Source: D. Training Objective · p. 3
  • The Sobel regularizer sums the L1 magnitudes of horizontal and vertical prediction gradients.The gradients are computed with Sobel operators.Source: D. Training Objective · p. 3
  • Training runs for 600k iterations with Adam, batch size 4 per GPU, and a multistep learning-rate schedule at 140k, 280k, and 360k iterations.A 2-pixel border crop is excluded from the loss to reduce boundary artifacts.Source: D. Training Objective · p. 3

A. Experimental Setup

Evaluation uses CRVD real sensor noise under its official two-stage training and test protocol. Runtime comparisons use a single CPU thread and include the complete FilterFusion NV12 pipeline.

A. Experimental Setup

  • CRVD evaluation uses 11 indoor scenes with motion across five ISO levels at 1920×1080, training on scenes 1–6 and testing on scenes 7–11 after synthetic pretraining.PSNR, SSIM, and CPU runtime are the main evaluation criteria.Source: A. Experimental Setup · p. 4
  • All methods and baselines are timed on one CPU thread, with FilterFusion timing covering RGB-to-NV12 conversion, luma inference and synthesis, UV synthesis, and NV12-to-RGB conversion.The Gaussian, median, and weighted-sum kernels use compiled C, with OpenCV resize for interpolation.Source: A. Experimental Setup · p. 4
  • Comparisons include classical denoisers and deep-learning baselines trained on the same data; exploratory runs lacking complete records are excluded.Source: A. Experimental Setup · p. 4

B. Comparison with State-of-the-Art Methods

On CRVD, FilterFusion combines competitive denoising quality with low compute and real-time CPU speed. Its learned temporal weighting avoids the severe ghosting of naive multi-frame fusion, while its filter-bank synthesis is much faster than full-resolution CNN baselines.

B. Comparison with State-of-the-Art Methods

  • 34.72 dB / 0.9330 on CRVD with 0.51 total GFLOPs; FilterFusion exceeds the best classical result by 1.32 dB over Bilateral at 33.40 dB.Table I reports the full CPU-side comparison; the two-stage baselines use SRVD pre-training followed by CRVD fine-tuning.Source: B. Comparison with State-of-the-Art… · p. 4, Table I · p. 4
  • Naive TemporalAvg, TemporalMedian, and TempAvg+Bilateral fall to ∼17 dB from severe ghosting, while FilterFusion’s Vt gating limits temporal averaging in motion regions.On static regions, the model uses multiple frames to suppress sensor noise without blurring thin power lines.Source: B. Comparison with State-of-the-Art… · p. 4
  • 17.9 ms per frame (55.9 FPS) makes FilterFusion real-time on a single CPU thread, 252× faster than UNetTiny despite a 0.65 dB lower PSNR.The 17.9 ms timing uses compiled C filter-bank synthesis; the stated comparison is 4,519 ms for UNetTiny.Source: B. Comparison with State-of-the-Art… · p. 4
  • Fig. 1 places FilterFusion above the classical quality ceiling while exceeding 30 FPS; UNetTiny reaches 35.37 dB but needs 4.5 s per frame and 122.6 GFLOPs.The figure compares seven representative methods across five axes.Source: B. Comparison with State-of-the-Art… · p. 5

C. Ablation Study

The ablations show that temporal and spatial guidance, multi-scale spatial filters, temporal fusion, and the combined loss contribute distinct benefits. The qualitative examples illustrate the different noise and ghosting failures when guidance or filtering choices are removed.

C. Ablation Study

  • At ISO 25600 and high Vt, Fig. 4 contrasts residual noise in (a), ghosting in (b), and clean, ghost-free output in (c), against ground truth in (d).Fig. 3 shows Vt gating preserves logos in high-Vt regions and suppresses noise without blurring power lines in low-Vt regions.Source: Figure 4 · p. 5, Figure 3 · p. 5
  • Removing Vs costs 0.15 dB PSNR, while removing Vt lowers SSIM by 0.0017 despite a 0.07 dB PSNR increase.Without Vt, the controller over-trusts temporal averaging: static noise falls, but motion ghosts violate local structure that SSIM captures and averaged PSNR can hide.Source: C. Ablation Study · pp. 5–6
  • Removing all spatial filters costs 2.23 dB PSNR and 0.117 SSIM; removing only the Gaussian pyramid costs 1.19 dB and 0.044.The all-spatial-filter removal is the largest ablation drop after variant (g).Source: C. Ablation Study · p. 6
  • The single-frame variant reaches 25.41 / 0.9043, a 9.35 dB PSNR drop versus the full model, while single L1 yields 34.83 / 0.9342.The combined Charbonnier, Sobel, and frequency-aware loss improves structural fidelity beyond pixel-domain supervision, despite single L1's slightly higher PSNR.Source: C. Ablation Study · p. 6

D. Discussion

FilterFusion is designed for deployment flexibility: selected branches can be dropped or the temporal backbone replaced while keeping the fusion formulation. Its temporal-variance guidance steers unreliable regions toward reference-preserving branches, trading less temporal denoising in dynamic regions for reduced ghosting.

D. Discussion

  • Selected branches can be dropped or the temporal backbone replaced with a lighter alternative without changing the overall fusion formulation.Explicit branch semantics make these deployment simplifications possible.Source: D. Discussion · p. 6
  • Temporal-variance guidance routes unreliable regions toward reference-preserving branches to suppress ghosting, so dynamic regions receive less temporal denoising.The paper frames this as a deliberate quality–latency trade-off and does not use explicit motion estimation.Source: D. Discussion · p. 6
  • At 1080p, the single-thread pipeline takes 17.9 ms (55.9 FPS): 2.2 ms for network inference, 11.5 ms for Y-channel post-processing, and 4.2 ms for UV post-processing.Fixed-kernel filters and element-wise weighted sums map naturally to efficient compiled code.Source: D. Discussion · p. 6

V. CONCLUSION

The conclusion presents FilterFusion as a low-resolution-control, filter-synthesis approach that exceeds a classical quality ceiling with much less compute and reaches real-time 1080p CPU speed.

  • 34.72 dB vs. 33.40 dB for Bilateral, with 240× less compute than UNetTiny, on CRVD (sensor noise).Source: §V · p. 6
  • 17.9 ms per frame (55.9 FPS) at 1080p on one CPU thread is 7.2× faster than EMVD’s multi-threaded result.Source: §V · p. 6
  • Compiled filter-bank synthesis makes low-resolution control with high-resolution filter synthesis a practical and effective design principle.The filter-bank operations are simple enough for efficient compiled implementations, unlike dense learned convolutions.Source: §V · p. 6

Related papers

Loading 2610.21505v1…