Source-linked AI summary
DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models
Jaewon Min, Jaeeun Lee, Yeji Choi, Paul Hyunbin Cho, Jin Hyeon Kim, Tae-Young Lee, Jongsik Ahn, Hwayeong Lee, Seonghyun Park, Seungryong Kim
TL;DR
Optical flow models trained on clean data struggle with real-world corruption, while degraded inputs can destroy the visual evidence needed for reliable matching. The paper introduces DA-Flow, which lifts image-restoration diffusion features with spatio-temporal attention and fuses them with CNN features in an iterative flow framework. Across degraded benchmarks, DA-Flow consistently improves flow estimation and substantially outperforms existing methods, while its training relies on pseudo-ground-truth flow because real-world degraded-video labels are unavailable.
Problem
Real-world degradations make optical flow correspondence ambiguous, and accurate flow estimation from corrupted inputs remains an open question.
Method
DA-Flow lifts a pretrained image-restoration diffusion model with cross-frame attention and fuses its features with CNN features while retaining RAFT’s correlation and iterative refinement.
Results
DA-Flow consistently improves flow estimation accuracy over existing methods on degraded optical flow benchmarks.
Takeaways & Limitations
Degradation-aware diffusion representations can provide correspondence-ready features for dense optical flow under severe corruption.
Takeaways & Limitations
Training uses pseudo-ground-truth flow because ground-truth optical flow for real-world degraded videos is unavailable.
Abstract
from arXiv · showhide
Optical flow models trained on high-quality data often degrade severely when confronted with real-world corruptions such as blur, noise, and compression artifacts. To overcome this limitation, we formulate Degradation-Aware Optical Flow, a new task targeting accurate dense correspondence estimation from real-world corrupted videos. Our key insight is that the intermediate representations of image restoration diffusion models are inherently corruption-aware but lack temporal awareness. To address this limitation, we lift the model to attend across adjacent frames via full spatio-temporal attention, and empirically demonstrate that the resulting features exhibit zero-shot correspondence capabilities. Based on this finding, we present DA-Flow, a hybrid architecture that fuses these diffusion features with convolutional features within an iterative refinement framework. DA-Flow substantially outperforms existing optical flow methods under severe degradation across multiple benchmarks.
1 Introduction
The paper introduces Degradation-Aware Optical Flow to estimate dense correspondences from severely corrupted videos. It lifts image-restoration diffusion features with inter-frame attention and combines them with convolutional features in DA-Flow.
- Motivation: Image-restoration diffusion representations provide degradation-aware structural cues, but they initially lack temporal awareness.The paper builds on evidence that diffusion features encode geometric and structural information useful for correspondence.
- Approach: Full cross-frame attention lifts a pretrained image-restoration diffusion model while preserving independent spatial latents for each frame.This avoids the early temporal entanglement of monolithic video diffusion backbones and supports explicit pairwise feature matching.
- Approach: DA-Flow fuses upsampled diffusion features with conventional CNN encoder features within RAFT’s correlation and iterative refinement framework.Training uses pseudo-ground-truth flow generated from high-quality video while degraded frames are provided as input.
- Results: DA-Flow substantially outperforms existing optical flow methods on degraded inputs, including severe degradations where competing methods fail.The qualitative comparison is reported on the Spring benchmark.
- Motivation: Degradation-Aware Optical Flow targets accurate dense correspondence estimation from severely corrupted videos.Real-world blur, noise, compression artifacts, and low resolution can make correspondence estimation ambiguous.
- Approach: The lifted diffusion features exhibit geometric correspondence cues even under severe corruption.The paper empirically verifies that these features can support correspondence estimation.
2 Related Work
Prior work has advanced optical flow, geometric correspondence, and image restoration, but degraded-video correspondence remains a distinct challenge requiring degradation-aware representations.
- Optical Flow Estimation: Modern optical flow methods use dense matching and iterative refinement, with RAFT serving as a strong baseline.RAFT combines dense all-pairs correlation with recurrent iterative refinement.
- Geometric Correspondence: Learned CNN and transformer models improve matching robustness, yet fine-grained dense correspondence remains difficult under large appearance variations.Classical pipelines instead rely on handcrafted local descriptors.
- Geometric Correspondence: Diffusion models provide spatially informative intermediate representations for correspondence and other vision tasks.Image-restoration diffusion models are proposed as a particularly suitable foundation for degraded inputs.
- Image Restoration Diffusion: Image-restoration diffusion models recover perceptually sharp details, but independent frame processing can cause temporal flickering and inconsistency.Video restoration models add cross-frame modeling, motivating different architectural choices for correspondence.
3 Preliminaries
Standard optical flow extracts features, builds a correlation volume, and iteratively refines flow, but degraded inputs corrupt features and distort matching. The paper also describes the latent diffusion restoration pipeline used to obtain restoration features.
- Optical Flow Estimation: Optical flow uses an encoder E, correlation operator C, and iterative update operator U: M = U ◦ C ◦ E.The encoder produces dense frame features, C constructs pairwise similarities, and U repeatedly refines the flow estimate.
- Optical Flow Estimation: Noise, compression, and blur corrupt extracted features and distort the correlation signal on low-quality videos.The feature encoder is therefore the pipeline stage most directly affected by degradation.
- DiT-based Image Restoration: The restoration model encodes low-quality and high-quality frames into latent representations with a pretrained VAE.The clean latent is the diffusion target, while the degraded latent conditions the model.
- DiT-based Image Restoration: Training constructs a noisy latent by linearly interpolating between Gaussian noise and the clean target at continuous noise level t.The denoising network predicts the velocity field conditioned on the degraded latent.
- DiT-based Image Restoration: The flow-matching objective minimizes the squared difference between predicted velocity and the clean-target-minus-noise velocity.At inference, iterative denoising starts from pure noise and the result is decoded by the VAE.
4 Method
DA-Flow formulates optical flow estimation for corrupted videos and builds a degradation-aware encoder by lifting an image-restoration diffusion model with full spatio-temporal attention. Its diffusion features are evaluated for correspondence and integrated with conventional flow estimation components.
- 4.1 Problem Formulation: Degradation-Aware Optical Flow targets accurate motion estimation from low-quality video inputs.
- 4.1 Problem Formulation: Because corrupted pixels create unreliable features whose errors propagate downstream, the method focuses on a robust, correspondence-ready feature encoder while keeping correlation and update stages unchanged.
- 4.2 Lifting Image Restoration Diffusion Models: The DiT-based restoration model is extended from independent frame processing to full spatio-temporal attention, preserving per-frame restoration priors while enabling temporal reasoning.
- 4.2 Lifting Image Restoration Diffusion Models: Folding frames into the batch makes MM-Attention independent per frame, so the lifted design reshapes tokens across frames to enable inter-frame reasoning.
- 4.3 Diffusion Feature Analysis: Lifting consistently achieves lower timestep-averaged EPE than the baseline across ranked layers and remains stable across denoising steps.
- 4.3 Diffusion Feature Analysis: The lifted model supplies the feature encoder, with attention features extracted from full spatio-temporal layers during iterative denoising for flow correspondence.
Evaluation protocol.
The evaluation protocol combines diffusion-feature correspondence analysis with a hybrid RAFT-based flow pipeline and pseudo-ground-truth training. Features are upsampled, concatenated with CNN features, and processed through correlation and recurrent refinement, while benchmark reporting uses EPE and outlier rates.
- Diffusion-feature evaluation: Finetuned full spatio-temporal attention features achieve lower EPE than an untrained baseline across layer ranks and remain stable across denoising steps.The comparison supports selecting diffusion layers for DA-Flow feature extraction.
- Hybrid flow pipeline: DA-Flow retains RAFT correlation and iterative update operators while combining lifted diffusion features with conventional CNN encoder features.The diffusion model supplies degradation-aware representations, while the CNN pathway preserves local spatial detail.
- Feature upsampling: Learnable upsampling maps coarse diffusion features to a resolution compatible with CNN features before correlation-volume construction and iterative flow updates.The diffusion features begin at 1/16 input resolution, and the upsampled features share H/8 × W/8 spatial resolution.
- Training and evaluation: Table 1 evaluates Sintel, Spring, and TartanAir using EPE and outlier rates at 1px, 3px, and 5px thresholds.The table caption specifies the metrics and highlights the best and second-best results.
- Feature upsampling: Query, key, and context features use separate upsampling heads for correspondence construction and spatial conditioning.The query and key heads preserve their distinct diffusion-attention representations, while the context head feeds the iterative update operator.
- Training and evaluation: Pseudo-ground-truth flow from high-quality frame pairs supervises multi-scale flow loss across recurrent refinement iterations.The reference flow is produced by a pretrained optical flow model applied to HQ frame pairs, while LQ frames are the DA-Flow inputs.
5 Experiment
DA-Flow is evaluated on degraded optical-flow benchmarks using pseudo-ground-truth training and comparisons with existing methods and feature-source variants. It performs strongly across benchmarks, while qualitative and ablation results examine flow quality and the contribution of lifted diffusion features.
- Evaluation setup: DA-Flow is evaluated on Spring, Sintel, and TartanAir using degraded inputs and EPE plus outlier-rate metrics.LQ inputs are generated with the same degradation pipeline used during training, and baseline models use 12 iterative updates.
- Quantitative results: On Sintel and Spring, DA-Flow achieves the best performance across all reported metrics and reduces EPE relative to the strongest baseline.On TartanAir, it achieves the best outlier rates at 1px, 3px, and 5px, but has higher EPE than FlowSeek.
- Qualitative results: Under degraded inputs, DA-Flow produces sharp, coherent flow fields that better localize motion boundaries and preserve structure than baseline methods.The comparisons cover Sintel, Spring, and TartanAir, including complex scenes and large displacements.
- Ablation: The feature-source ablation compares the same DA-Flow architecture using lifted features against features from an untrained full-attention model across denoising steps.On Sintel, DA-Flow consistently outperforms Baseline* at every step in EPE and 1px outlier rate.
6 Conclusion
The paper concludes that DA-Flow addresses degradation-aware optical flow by combining a lifted image-restoration diffusion model with flow estimation. Experiments on degraded benchmarks show consistent accuracy improvements over existing methods.
- Conclusion: DA-Flow estimates dense correspondences directly from corrupted inputs using a pretrained image-restoration diffusion model extended with spatio-temporal attention.Its intermediate representations provide degradation-aware priors and geometric correspondence cues while preserving spatial structure for dense matching.
- Conclusion: Extensive experiments on degraded optical-flow benchmarks demonstrate that DA-Flow consistently improves flow-estimation accuracy over existing methods.
A Implementation Details
The implementation uses DiT4SR as the image-restoration diffusion backbone and RAFT as the optical-flow network, with full cross-frame attention and pseudo-ground-truth supervision. Training and inference use specified frame chunks, prompts, denoising steps, and refinement iterations.
- Diffusion model: The lifting procedure reshapes frames into the spatial-token dimension so each attention layer computes full attention across frames.The diffusion backbone is DiT4SR with publicly available dit4sr_q weights.
- Flow network: The optical-flow network is RAFT-based and uses a DPT-style feature upsampler with pyramid scales of (1, 1, 2, 2).
- Training supervision: SEA-RAFT generates pseudo-ground-truth flow from high-quality videos, using 4 recurrent iterations for analysis and 20 for DA-Flow training.
- Prompting: The lifting model is conditioned on generated text prompts during training, while null prompts are used for feature analysis, DA-Flow training, and inference.Null prompts are used because low-quality frames make reliable captions difficult to generate.
- Inference: Inference uses 10 denoising steps, processes videos in 3-frame chunks, and estimates flow with 20 refinement iterations.
B.1 Full Feature Analysis Results
The feature analysis evaluates correspondence quality across diffusion layers, feature locations, denoising steps, and diffusion backbones. Lifting generally improves correspondence, while query/key features remain preferable to post-AdaNorm features for DA-Flow.
- Layer-wise analysis: Lifted diffusion features exhibit substantially lower timestep-averaged EPE across nearly all feature indices than baseline features.The analysis attributes this improvement to learning cross-frame information through lifting.
- Feature locations: Query and key features outperform post-AdaNorm features before lifting, while the two feature types become comparable after lifting except in the top three layers.
- Denoising analysis: Post-AdaNorm features show a noticeable EPE spike at the final denoising step, whereas query and key features lead to better flow estimation on Sintel after training.
- Video diffusion comparison: Video restoration diffusion models are analyzed by interpolating a single latent feature map into per-frame features, reflecting their temporal compression.The comparison uses query and key features from FlashVSR.
C.1 Metrics over Denoising Steps
DA-Flow is evaluated across denoising timesteps, against a finetuned RAFT baseline, and through architectural ablations. Results show broad benchmark strength and complementary benefits from CNN features and DPT-based aggregation.
- DA-Flow consistently outperforms existing methods across all timesteps on Sintel and Spring.On TartanAir, its timestep-averaged EPE is slightly higher than prior methods, but it performs best at steps 0 and 9.
- Finetuning RAFT: DA-Flow outperforms finetuned RAFT* in EPE, 1px, and 3px metrics on Sintel.Both models use the same training setup and pseudo-ground-truth flow generation.
- Architectural Ablations: Adding the CNN encoder consistently improves performance with both bilinear and DPT upsampling, especially with DPT.The encoder supplies fine-grained spatial information that diffusion features alone lack.
- Architectural Ablations: DPT upsampling yields clear gains across all metrics, including EPE, when the CNN encoder is incorporated.Without the CNN encoder, replacing bilinear interpolation with DPT does not improve all metrics consistently.
D Additional Qualitative Results
Additional qualitative results cover Sintel, Spring, and TartanAir, while a separate comparison addresses temporal consistency in video restoration. The paper also notes that DA-Flow's diffusion-based inference is slower than conventional flow networks.
- Additional qualitative results demonstrate DA-Flow's effectiveness across Sintel, Spring, and TartanAir.The examples span all benchmark datasets presented in Fig. 11, Fig. 12, and Fig. 13.
- Fig. 10 compares temporal consistency in video restoration.
- Multiple denoising steps make DA-Flow slower at inference than conventional flow estimation networks.One-step distillation is identified as a direction for reducing inference cost while preserving estimation quality.