Source-linked AI summary
The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, David J. Fleet
TL;DR
The paper asks whether optical flow and monocular depth can be estimated effectively with generic diffusion models rather than task-specific architectures and losses. It trains an image-to-image diffusion framework with self-supervised pre-training, mixed real and synthetic data, and procedures for incomplete noisy labels. The model captures uncertainty and multimodality and achieves strong benchmark results, including 3.26% Fl-all on KITTI, while remaining slower than RAFT and behind FlowFormer on fine-tuned Sintel.
Problem
Optical flow and monocular depth are typically treated as regression problems requiring specialized architectures and losses, while diffusion models’ effectiveness for these tasks is uncertain.
Method
The paper uses generic conditional diffusion for image-to-image prediction, with self-supervised pre-training, real and synthetic data, infilling, step-unrolled training, and coarse-to-fine refinement.
Results
3.26% Fl-all on KITTI is about 25% better than the best published method, while the model also captures multimodality and uncertainty through multiple samples.
Takeaways & Limitations
Diffusion models provide a generic framework for dense vision that can achieve state-of-the-art results while representing uncertainty, multimodality, and missing-value imputation.
Takeaways & Limitations
The method is slower than RAFT, and under fine-tuning it trails FlowFormer on Sintel Final, partly because of a difficult sequence and possible domain differences.
Abstract
from arXiv · showhide
Denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity. We show that they also excel in estimating optical flow and monocular depth, surprisingly, without task-specific architectures and loss functions that are predominant for these tasks. Compared to the point estimates of conventional regression-based methods, diffusion models also enable Monte Carlo inference, e.g., capturing uncertainty and ambiguity in flow and depth. With self-supervised pre-training, the combined use of synthetic and real data for supervised training, and technical innovations (infilling and step-unrolled denoising diffusion training) to handle noisy-incomplete training data, and a simple form of coarse-to-fine refinement, one can train state-of-the-art diffusion models for depth and optical flow estimation. Extensive experiments focus on quantitative performance against benchmarks, ablations, and the model's ability to capture uncertainty and multimodality, and impute missing values. Our model, DDVM (Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\% on the KITTI optical flow benchmark, about 25\% better than the best published method. For an overview see https://diffusion-vision.github.io.
1 Introduction
The paper asks whether generic diffusion models can handle optical flow and monocular depth without the specialized architectures and losses used by regression methods. It argues that diffusion models are effective when paired with suitable data and training procedures, while also representing uncertainty and multimodality.
- 1 Introduction: Diffusion inference captures uncertainty and multimodal predictions rather than producing only a single point estimate.The paper illustrates this benefit with multiple samples from the predictive distribution.
- 1 Introduction: Limited labelled data and noisy, incomplete annotations are central barriers to training diffusion models for dense vision tasks.
- 1 Introduction: The proposed pipeline uses multi-task self-supervised pre-training followed by supervised pre-training on combined real and synthetic data.
- 1 Introduction: Diffusion models can outperform state-of-the-art methods on optical flow and monocular depth without task-specific architectures or loss functions.The proposed training pipeline combines self-supervised pre-training, real and synthetic supervised data, infilling, step-unrolled denoising, and an L1 loss.
- 1 Introduction: 3.26% Fl-all on KITTI is about 25% better than the best published method.
2 Related work
Prior optical-flow and depth methods rely heavily on specialized architectures, losses, or in-domain pre-training. This work instead uses a generic denoising diffusion architecture that retains competitive task performance while modeling ambiguity.
- 2 Related work: Regression-based optical-flow methods use specialized components but provide only a single prediction and do not readily capture uncertainty or ambiguity.
- 2 Related work: The paper frames optical flow and depth estimation as applications of diffusion-based image-to-image translation rather than conventional regression.
- 2 Related work: The paper introduces a generic denoising diffusion model for optical flow that recovers fine motion-boundary details and captures multimodal motion distributions.
- 2 Related work: Monocular-depth research has emphasized specialized losses, multi-scale architectures, adaptive binning, and large-scale in-domain pre-training.
- 2 Related work: Self-supervised pre-training supports diffusion-based image-to-image translation for flow and depth while preserving multimodality, coarse-to-fine refinement, and imputation.
3 Model Framework
The model denoises a noisy depth or flow map conditioned on RGB images, using generic architectures and losses instead of task-specific inductive biases. Training addresses data bias, sparse annotations, train–inference mismatch, and resolution limits through data mixing, infilling, step unrolling, and coarse-to-fine refinement.
- 3 Model Framework: The denoiser predicts a target depth or flow map from its noisy version and the conditioning RGB image or image pair.The training loss penalizes residual error in the denoised target map.
- 3 Model Framework: AutoFlow-only pre-training biases diffusion models toward polygonal flow regions, while mixing FlyingThings3D, Kubric, and TartanAir improves fine details and motion boundaries.
- 3 Model Framework: Sparse and noisy ground truth creates a train–inference distribution shift because inference repeatedly refines model outputs rather than independently denoising ground truth maps.
- 3 Model Framework: Infilling missing labels while computing masked loss only on known pixels makes sparse training targets usable without treating imputed values as ground truth.
- 3 Model Framework: Step-unrolled training constructs noisy targets from model outputs without backpropagating through that forward pass, aligning training with iterative inference.
- 3 Model Framework: Coarse-to-fine refinement upsamples a low-resolution estimate, denoises overlapping high-resolution patches, and merges them with lower boundary weights.
4 Experiments
Experiments show that diffusion models achieve strong optical-flow and depth results across benchmark evaluations, while ablations identify the importance of data mixtures, infilling, step-unrolling, and coarse-to-fine refinement. The model also captures multimodality and supports missing-label imputation.
- Benchmark evaluation: State-of-the-art zero-shot optical-flow results on Sintel Final and KITTI are achieved against a stronger RAFT baseline trained on the proposed mixture.The comparison averages 8 coarse-resolution samples and applies full-resolution refinement.
- Benchmark evaluation: 30.34% lower KITTI error than FlowFormer highlights the method's strongest finetuned optical-flow comparison.On Sintel Final, the method is competitive but trails FlowFormer overall, with higher error concentrated on one test sequence.
- Ablation study: Infilling and step-unrolling are both important for training with sparse, noisy labels, with infilling providing most of the gains and both composing in the best models.Without either technique, performance deteriorates; without both, KITTI optical-flow models fail to train.
- Ablation study: Coarse-to-fine refinement improves zero-shot optical-flow metrics on Sintel and KITTI while adding fine-grained details to flow fields.Qualitative refinement can correct wrong flow and add detail to already correct flow.
- Ablation study: Mixing synthetic optical-flow datasets removes AutoFlow's polygonal-shape bias and improves fine details and zero-shot accuracy, while synthetic depth data boosts NYU fine-tuning.Reported improvements include shadows, trees, thin structures, motion boundaries, and fine-tuning performance on NYU.
- Interesting properties of diffusion models: Multiple samples represent uncertainty and ambiguity, and replacement guidance enables effective qualitative imputation of missing depth labels.The model produces plausible multimodal samples on NYU, KITTI, and Sintel, while depth completion and iterative 3D-scene generation remain less thoroughly evaluated.
5 Limitations
The main practical limitation is inference speed: the diffusion approach uses larger models and runs more slowly than RAFT. Fine-tuning also trails FlowFormer on Sintel, with the gap concentrated on a difficult sequence.
- 5 Limitations: Diffusion inference is slower than RAFT because standard image-generation practices produce larger models and longer running times.The authors identify progressive distillation and consistency models as possible ways to improve speed.
- 5 Limitations: Under fine-tuning, the method is state-of-the-art on KITTI but behind FlowFormer on Sintel Final.
- 5 Limitations: The authors suggest that fine-tuning procedures may favor KITTI over Sintel and that Sintel has a larger training–test domain gap.
- 5 Limitations: On Sintel, the Ambush 1 sequence has AEPE close to 30 for the method versus lower than 10 for FlowFormer and may explain much of the overall gap.
E Ablations
The ablations examine loss choice, self-supervised pretraining, refinement, architecture, and training data across depth and flow settings. L1 training and self-supervised pretraining improve monocular depth, while coarse-to-fine refinement adds visual detail without consistent quantitative gains.
- Loss and pretraining: L1 loss performs much better than L2 loss for monocular depth estimation on NYU and KITTI.Tables 11 and 12 report the loss-function ablations, with all results using a single sample.
- Loss and pretraining: Self-supervised Palette-style pretraining is effective for monocular depth estimation on NYU and KITTI.The comparison is reported in Tables 13 and 14.
- Coarse-to-fine refinement: Refinement improves fine-scale depth detail but does not provide significant quantitative improvements, so it is excluded from reported depth results.The authors identify the lack of robust quantitative gains as motivation for further work on coarse-to-fine depth refinement.
- Architecture: The model adapts Imagen’s Efficient UNet, using fewer self-attention layers, fewer parameters, and less high-resolution computation than prior diffusion U-Nets.The configuration drops text cross-attention while retaining selected self-attention layers, with additional attention enabled for supervised flow training.
- Coarse-to-fine refinement: The coarse-to-fine scheme marginally improves RAFT on Sintel Final but hurts performance on Sintel Clean and KITTI.Table 15 reports EPE on the Sintel and KITTI datasets.
- Training data: Training combines self-supervised ImageNet-1K and Places365 tasks with synthetic and real supervised datasets for flow and depth.The self-supervised tasks include colorization, inpainting, uncropping, and JPEG decompression; supervised training mixes task-specific datasets.
H.3 Step-unrolling and interpolation of missing depth and flow
The paper uses infilling and step-unrolled denoising to address missing training values and train–inference distribution shift. Ablations show that additional unroll steps can improve depth, while step-unrolling outperforms self-conditioning in a missing-data setting.
- Motivation: Missing depth and flow values create a train–inference distribution shift that infilling and step-unrolling are designed to mitigate.The missing values occur in training depth maps and flow fields.
- Interpolation of missing values: Nearest-neighbor infilling fills indoor depth gaps, handles outdoor depth except sky regions, and supports missing-flow supervision with a sequence-based procedure.Outdoor sky pixels are assigned the maximum modeled depth of 80m after sky segmentation.
- Step-unrolling: Depth performance improves through four step-unrolling steps and then plateaus, with larger benefits when missing depth is not infilled.Table 16 reports the number-of-unroll-steps ablation.
- Step-unrolling: Step-unrolled denoising is presented as hyper-parameter-free in its infilling component and potentially applicable to other sparse-data tasks.This comparison is made against synthetically adding holes with a chosen masking ratio.
- Step-unrolling: Self-conditioning does not bridge the noisy-latent train–inference shift for monocular depth, whereas step-unrolling substantially improves KITTI without infilling.The comparison is reported in Table 17.
I Limitations
The limitations concern diffusion inference speed, transfer from zero-shot to fine-tuned optical flow, and uncertainty or ambiguity in depth estimates. Reducing denoising steps can improve efficiency with only moderate performance reduction, but remains slower than DPT.
- Inference efficiency: Using 24 denoising steps yields performance comparable to DPT while remaining approximately 5× slower, modulo hardware differences.Table 18 evaluates the indoor depth model fine-tuned on NYU.
- Inference efficiency: The diffusion model is considerably slower than DPT because inference requires many denoising steps.The denoiser takes approximately 8.5 ms per step on a TPU v4, but total wall time remains substantially higher.
- Optical-flow transfer: The model’s superior zero-shot optical-flow performance over FlowFormer does not transfer to fine-tuning on Sintel.Figure 16 provides qualitative examples related to this transfer gap.
- Uncertainty and ambiguity: Depth uncertainty can be represented in the predictive posterior, including ambiguity between a farther play mat and a nearer table.Figure 17 illustrates multimodal estimation cases on NYU depth.