DepthFM: Fast Monocular Depth Estimation with Flow Matching

1 Introduction

• Discriminative models lack fine-grained high-frequency detail, while diffusion depth methods require long inference because their noise-to-depth ODE trajectories are highly curved.The introduction frames realistic geometry from a single image as a continuing challenge.Source: §1 · p. 1
• DepthFM directly models a flow-matching trajectory from image space to depth space, using data-dependent couplings rather than starting from noise.The authors propose this trajectory to improve training and inference efficiency.Source: §1 · p. 1
• Training from a pretrained image diffusion model provides visual cues, speeds training, and supports robust real-world generalization from a small amount of synthetic data.The method also leverages a pretrained discriminative depth model as an external knowledge source.Source: §1 · pp. 1–2
• The paper targets sampling, training, and data efficiency in generative depth estimation.Its contributions combine flow matching with external generative and discriminative knowledge.Source: §1 · p. 2
• DepthFM synthesizes depth maps in one inference step and achieves competitive or state-of-the-art benchmark performance, including state-of-the-art depth completion.The paper also reports competitive performance on natural images despite training solely on synthetic data.Source: §1 · p. 2

2 Related Works

• Earlier diffusion-based depth methods exploit vision-foundation-model knowledge for high-quality maps but face slow sampling and rely on a Gaussian source distribution.The paper notes that this source may not fully capture the natural relationship between images and their corresponding depth maps.Source: §2 · p. 2
• Flow-matching methods offer faster sampling, and optimal transport can use a non-Gaussian source distribution.DepthFM applies this paradigm to monocular depth estimation to reduce sampling costs through straighter trajectories.Source: §2 · p. 2
• The authors report no prior use of flow matching for image-to-depth distribution transfer, despite the modalities' intuitive proximity.They also seek training-efficiency gains by incorporating insights from diffusion models because the objectives are similar.Source: §2 · p. 2
• The training pipeline uses flow matching to regress a vector field between the image latent x0 and corresponding depth latent x1.Source: Figure 2 · p. 3

Background: Flow Matching

• Flow matching regresses vector fields along fixed conditional probability paths; its ODE transports density, and the common path linearly interpolates Gaussian source samples and targets.Conditional flow matching trains a network to regress conditional vector fields by sampling from the conditional path.Source: Background: Flow Matching · p. 3

Data Coupling in FM for Depth Estimation

• Paired image and depth latents define a direct transport with a shorter path than Gaussian-noise-to-depth transport, and image-depth pairing satisfies the optimal-transport condition.The method uses log-scaled depth in latent space; Table 1 reports paired coupling is better than random coupling.Source: Direct Transport between Image and Depth · p. 3, Flow in Latent Space · p. 3, Table 1 · p. 4
• Noise augmentation mixes the image feature with Gaussian noise at terminal time t = 0 while preserving variance; the authors hypothesize it smooths the density and supports stochastic confidence applications.The authors also report feasibility on image-depth pairs, a setting they say had not been explored before.Source: Noise Augmentation · p. 4

Dual Knowledge Transfer for better Training and Data Efficiency

• A pretrained diffusion image prior is fine-tuned with conditional flow matching because the diffusion and flow-matching objectives are similar, which speeds convergence.Source: Image Prior for Training Efficiency · p. 4
• A discriminative teacher predicts depths for unlabeled in-the-wild images; a random subset of those pairs joins ground-truth data to improve robustness and performance.The subset size is k·|DGT|.Source: Depth Prior for Data Efficiency · p. 4

Training and Evaluation Details

• Training uses 54K Hypersim and 20K Virtual KITTI samples plus Metric3D v2-generated image-depth pairs from Unsplash; evaluation covers five real-world benchmarks.Unless specified otherwise, evaluation uses an ensemble of 10 and four Euler steps, with predictions scaled and shifted to ground truth in log space.Source: Training and Evaluation Details · p. 4

Zero-shot Depth Estimation

• DepthFM achieves competitive zero-shot performance across indoor and outdoor datasets using 74K synthetic samples and 7.4K discriminative depth samples.Table 2 compares affine-invariant estimators across zero-shot benchmarks.Source: Zero-shot Depth Estimation · pp. 4–5, Table 2 · p. 5
• DepthFM outperforms Marigold at low NFE, including one NFE versus Marigold’s four, and produces realistic depth maps in two inference steps.Table 3 reports NYUv2 δ1 across NFE settings with ensemble size fixed at 10.Source: Zero-shot Depth Estimation · p. 5, Table 3 · p. 5, Figure 3 · p. 5
• On Middlebury-2014, DepthFM outperforms discriminative models in edge precision and recall, while high-uncertainty regions have 3.21 times higher average L1 loss.Ensemble standard deviation estimates uncalibrated uncertainty; Figure 6 shows the image, mean depth, and standard deviation.Source: Zero-shot Depth Estimation · p. 6, Table 4 · p. 6, Figure 6 · p. 6

Depth Completion

• With only 2% of ground-truth pixels available, fine-tuned DepthFM achieves state-of-the-art depth completion results on NYUv2.The evaluation uses root mean square error (RMSE).Source: Table 5 · p. 6, Depth Completion · p. 7

Ablation Studies

• Starting transport from the input image’s latent representation yields significantly higher NYUv2 accuracy than Gaussian-noise initialization, especially at low NFE.Both approaches condition on the image throughout the ODE trajectory.Source: Table 6 · p. 6, Image-depth Coupling · p. 7
• Image and depth priors improve performance; a mix with 10% discriminative samples is optimal, while greater use of those samples produces blurrier results.The study also reports better training-data efficiency than Depth Anything: 7.4K rather than 62M discriminative samples, without strong augmentation.Source: Image-depth Coupling · p. 7
• Noise augmentation is optimal at ts = 0.4.The level was selected through empirical analysis.Source: Image-depth Coupling · p. 7, Table 9 · p. 7

5 Conclusion

• DepthFM formulates monocular depth estimation as direct transport between image and depth, using flow matching for faster sampling than diffusion-based solutions.The method aims to avoid the common artifacts of discriminative depth estimation.Source: §5 · p. 7
• A pre-trained image diffusion model supplies an image prior that provides visual cues for depth inference and improves training efficiency.Source: §5 · p. 7
• Combining synthetic data with a discriminative depth prior improves data efficiency.Source: §5 · p. 7
• The paper reports that DepthFM produces high-fidelity depth maps without common discriminative-method artifacts.Source: §5 · p. 7

Appendix for DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching

• Figure 8 shows DepthFM generalizing to real-world data with varying resolutions and aspect ratios.Source: Figure 8 · p. 11

Additional Qualitative Comparison

• DepthFM produces higher-fidelity indoor and outdoor depth maps than other methods, particularly distinguishing distant objects from the sky where ZoeDepth and DMD fail.The comparisons cover the datasets listed for Figures 12 and 13.Source: Additional Qualitative Comparison · p. 11
• DepthFM generalizes from synthetic training at 384 × 512 pixels to real images with different resolutions and aspect ratios, producing realistic depth maps.Figure 8 illustrates this resolution generalization.Source: Generalization across Resolutions · p. 11
• DepthFM successfully inpaints depth from highly sparse partial input, including qualitative results on unseen NYUv2 data.The NYUv2 results are shown in Figure 21.Source: Depth Inpainting · p. 12
• DepthFM preserves fine patterns and straight edges that Depth-Anything models miss or distort, including bicycle spokes, a box pattern, and a piano-chair edge.The edge comparison uses precision and recall on Middlebury-2014; blur lowers precision, while fewer detected edges lower recall.Source: Edge Fidelity · p. 12, Figure 9 · p. 12
• Both the ensemble mean and individual ensemble members produce high-fidelity depth estimates with sharp, consistent edges.Figure 11 shows the qualitative comparisons.Source: High Fidelity in Ensemble Mean and… · p. 12

Further Applications

• For video, DepthFM estimates depth with five ensemble members and two ODE steps, then scales and shifts each frame using the previous frame’s depth for temporal consistency.The model predicts relative depth.Source: Video Depth Estimation · p. 12
• Depth-conditioned synthesis using DepthFM depth maps yields images that more closely resemble the source than synthesis based on ZoeDepth maps.The comparison uses a pre-trained depth-to-image ControlNet with Stable Diffusion 1.5 and a text prompt.Source: Conditional Synthesis · p. 12

Additional Ablations

• An optimal depth-prior sample ratio balances fidelity and metrics: too few samples fail to maximize performance, while excess samples reduce both.The ablation also reports that increasing the ratio can improve performance while making images increasingly blurry.Source: Additional Ablations · p. 12, Table 10 · p. 13
• Ensembling improves DepthFM performance, as visualized in Figure 16.Source: Additional Ablations · p. 12
• Log-scaled depth normalization allocates representational capacity more evenly between indoor and outdoor scenes than linear normalization, with an additional performance improvement.The comparison is shown in Figure 17 and Table 11.Source: Data Preprocessing · p. 13
• For depth maps with invalid values, the pipeline resizes and thresholds an invalid mask, then fills invalid pixels by nearest-neighbor interpolation before encoding.Source: Data Preprocessing · p. 13
• Adding virtual KITTI to Hypersim training improves zero-shot KITTI accuracy for outdoor scenes and yields realistic estimates of distant objects.Figure 19 compares a Hypersim-only model with one trained on both synthetic datasets.Source: Training Data · p. 13

Training Details

• DepthFM fine-tunes from the SD2.1 image prior, keeps the UNet fixed except for its first convolution, disables text conditioning, and uses null-text embeddings.The first convolution’s channel-wise weights are duplicated and scaled down; training uses global batch size 128, learning rate 3 × 10^-5, and EMA rate 0.999.Source: Training Details · p. 13
• Experiments use NVIDIA A100 40GB GPUs and PyTorch 2.1.0.Source: Training Details · p. 13
• Depth images are quantile-normalized per image for the latent encoder, with log or linear normalization; log normalization provides an additional performance improvement.The normalization uses dataset 2% and 98% quantiles.Source: Depth Normalization · pp. 13–14
• The depth-completion model is fine-tuned on partial maps with 2% of pixels available, after 10k Hypersim steps and 8k NYU-v2 completion steps.Sparse depth and its mask are upsampled using nearest-neighbor interpolation and an L2 distance function, respectively.Source: Training Details of the Depth… · p. 14

• Discriminative monocular depth methods regress metric or relative depth, while generative approaches use diffusion for metric estimation.Source: Monocular Depth Estimation · p. 14
• DDP encodes images and decodes depth, achieving state-of-the-art KITTI results; DepthGen handles noisy ground truth, and DDVM emphasizes synthetic and real-data pretraining.Source: Monocular Depth Estimation · p. 14
• Diffusion models have been applied to image generation, translation, text-to-image synthesis, editing, enhancement, segmentation, and depth estimation; flow matching spans image, video, motion, point-cloud, manifold, and text generation.Source: Diffusion and Flow Matching · p. 14
• Knowledge distillation transfers knowledge from larger task-trained models through their predicted labels, whereas transfer learning fine-tunes pretrained models, often aligning feature representations.The paper distinguishes its use of discriminative-model prior samples from these approaches.Source: Knowledge Distillation and Transfer… · p. 14
• On high-resolution Middlebury-2014, DepthFM predictions are described as sharper than discriminative counterparts, with consistent edge details across an ensemble of 8 members.Figure 11 visualizes 3 random ensemble members; some ground-truth pixels are invalid, while the model recovers details such as bicycle spokes.Source: Figure 10 · p. 15, Figure 11 · p. 16
• DepthFM’s depth fields support zero-shot ControlNet images that closely mimic their source, while ZoeDepth’s outputs show depth-field discrepancies.The comparison uses the same random seed; prompts appear at lower left and discrepancies are marked by red arrows.Source: Figure 14 · p. 18
• Supplementary figures show DepthFM video predictions and depth-completion examples on Hypersim and NYUv2.The Hypersim figure pairs partial depth, an estimate, and ground truth; the NYUv2 figure compares raw ground truth with predictions.Source: Figure 15 · p. 19, Figure 20 · p. 21, Figure 21 · p. 21
• A 0.1 ratio—74K depth-prior samples to 7.4K ground-truth samples—is described as balancing metric performance and prediction fidelity; synthetic outdoor data adds value on zero-shot KITTI.Source: Figure 18 · p. 20, Figure 19 · p. 20