Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

Jing He; Haodong Li; Wei Yin; Yixun Liang; Leheng Li; Kaiqiang Zhou; Hongbo Zhang; Bingbing Liu; Ying-Cong Chen

Source-linked AI summary

Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, Ying-Cong Chen

arXiv:2409.18124v5Sep 26, 2024cs.CV

TL;DR

Dense prediction methods increasingly use pretrained diffusion priors, but standard image-generation formulations can be poorly suited to this setting. Lotus adapts the formulation through direct annotation prediction, single-step diffusion, and a detail preserver, achieving strong zero-shot depth and normal estimation with limited data while improving efficiency.

Problem
Pretrained diffusion priors offer promising zero-shot dense prediction, but existing methods often inherit formulations designed for image generation without adapting them to dense prediction.
Method
Lotus directly predicts annotations, uses one-step diffusion, and adds a detail preserver that switches between image reconstruction and dense prediction.
Results
Lotus achieves promising or superior zero-shot depth and normal estimation across datasets, outperforming prior generative approaches in accuracy and efficiency with 59K training samples.
Takeaways & Limitations
Lotus provides a diffusion-based dense prediction model that supports practical applications including joint estimation and single- or multi-view 3D reconstruction.

Abstract

from arXiv · show

Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution to enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formulation, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis of the diffusion formulation for the dense prediction, focusing on both quality and efficiency. And we find that the original parameterization type for image generation, which learns to predict noise, is harmful for dense prediction; the multi-step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce Lotus, a diffusion-based visual foundation model with a simple yet effective adaptation protocol for dense prediction. Specifically, Lotus is trained to directly predict annotations instead of noise, thereby avoiding harmful variance. We also reformulate the diffusion process into a single-step procedure, simplifying optimization and significantly boosting inference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves SoTA performance in zero-shot depth and normal estimation across various datasets. It also enhances efficiency, being significantly faster than most existing diffusion-based methods. Lotus' superior quality and efficiency also enable a wide range of practical applications, such as joint estimation, single/multi-view 3D reconstruction, etc. Project page: https://lotus3d.github.io/.

LOTUS: DIFFUSION-BASED VISUAL FOUNDATION MODEL

Lotus is presented as a diffusion-based visual foundation model for dense geometry prediction, emphasizing zero-shot depth and normal estimation with minimal training data.

Lotus is a diffusion-based visual foundation model for dense geometry prediction.
The model targets zero-shot depth and normal estimation.
Figure 1 uses average rank across metrics, with lower values indicating better performance.

1 INTRODUCTION

Lotus addresses limitations in adapting diffusion models to dense prediction by tailoring the parameterization and diffusion process. Its protocol combines direct annotation prediction, single-step diffusion, and detail preservation, delivering strong zero-shot geometry results with limited data and improved efficiency.

Motivation: Diffusion priors can improve zero-shot dense prediction, but prior methods often inherit image-generation formulations without examining their suitability.
Analysis: Noise prediction is ill-suited to dense prediction because harmful variance creates errors that propagate through denoising.
Method: The detail preserver switches between annotation generation and image reconstruction to retain fine-grained input details.
Method: Lotus directly predicts annotations instead of noise and uses a single-step formulation to improve optimization and efficiency.
Results: Lotus achieves promising or superior zero-shot depth and normal estimation across datasets using only 59K training samples.
Applications: The approach supports joint estimation and single- or multi-view 3D reconstruction applications.

2 RELATED WORKS

Related work applies pretrained generative models and diffusion priors to dense perception, but efficiency and formulation remain central challenges. Lotus differs through systematic analysis, detail preservation, stochastic uncertainty, and fully single-step inference.

Generative models: Diffusion-based dense prediction methods leverage pretrained generative models but may inherit constrained performance and inefficient iterative denoising.
Generative models: GenPercept and StableNormal use single-step diffusion, but GenPercept is deterministic while StableNormal retains iterative refinement in its second stage.
Lotus: Lotus systematically analyzes diffusion formulation and adds a detail preserver for fine-grained predictions, while retaining stochastic uncertainty predictions.
Geometric dense prediction: Monocular depth and normal prediction require comprehensive scene understanding and are established geometric dense prediction tasks.

3 PRELIMINARIES

The formulation encodes images and dense annotations in a latent space, then adds scheduled noise and learns iterative denoising. It supports alternative parameterizations and DDIM-based fast sampling for inference.

Latent diffusion formulation: A pretrained auto-encoder maps images and dense annotations between RGB space and low-dimensional latent space.The encoder produces z_x and z_y, while the decoder approximately reconstructs the original inputs.
Forward diffusion: A noise schedule progressively degrades the annotation latent to pure Gaussian noise over T time-steps.The schedule is defined by β_1 through β_T, with the terminal step producing pure Gaussian noise.
Reverse diffusion: A U-Net iteratively removes noise from the corrupted annotation latent to predict the clean annotation latent.Training samples a random time-step t and minimizes the corresponding denoising loss.
Parameterization types: The two basic parameterizations train the network either to predict added noise or to directly predict the clean annotation latent.Noise prediction is the standard choice for image generation because it empirically produces detailed and realistic images.
Denoising process: DDIM accelerates multi-step sampling by denoising along an increasing subsequence of time-steps, optionally using deterministic updates.At inference, the sample is iteratively denoised from τ_S to τ_1 to obtain the clean output.

4 METHODOLOGY

Lotus adapts a pre-trained diffusion model for dense prediction by replacing noise prediction with direct annotation prediction, reducing diffusion to one step, and adding Detail Preserver regularization. The analysis motivates these choices through variance-induced instability, error propagation, optimization difficulty, and detail loss during adaptation.

Direct Adaptation: Direct Adaptation applies the original diffusion objective and multi-step DDIM sampling, but fails to achieve satisfactory performance.It serves as the baseline for analyzing parameterization, time-step count, and detail preservation.
4.1 Parameterization Types: Noise prediction introduces large variance at initial denoising steps, which iterative inference can preserve and amplify into unstable dense predictions.The variance is amplified as τ approaches T because the ϵ-prediction formulation contains a coefficient that grows substantially.
4.1 Parameterization Types: x0-prediction directly predicts the clean annotation without coefficients that amplify output variance, producing more stable and correct dense predictions than ϵ-prediction.The paper therefore replaces standard ϵ-prediction with x0-prediction for dense prediction.
4.2 Number of Time-Steps: Multi-step diffusion propagates prediction errors, is difficult to optimize with scarce high-quality data, and is slow and computation-intensive.These costs are less justified for simpler dense prediction tasks than for complex image generation.
4.2 Number of Time-Steps: Lotus reduces the training time-step sequence to one step, fixing t to T, which reduces the optimization space and improves computational efficiency while preventing harmful error propagation.The one-step formulation is selected for better adaptation under limited resources.
4.3 Detail Preserver: Detail Preserver adds an image-reconstruction capability through a task switcher, helping retain rich input-image details during dense-annotation adaptation.The denoiser switches between annotation prediction and input reconstruction without mutual interference.

5 EXPERIMENTS

Lotus is evaluated on zero-shot depth and surface-normal estimation using synthetic training data and multiple indoor and outdoor benchmarks. It achieves strong performance, with Lotus-G ranking best among generative depth baselines and both Lotus variants outperforming other methods for surface normals.

Training Datasets: Lotus is trained for depth and normal estimation on synthetic Hypersim and Virtual KITTI data covering indoor and outdoor scenes.Hypersim contributes approximately 39K filtered training samples, while batches sample Hypersim with probability 90% and Virtual KITTI with 10%.
Evaluation Datasets and Metrics: Zero-shot depth is evaluated on NYUv2, ScanNet, KITTI, ETH3D, and DIODE using AbsRel, δ1, and δ2.These benchmarks include indoor, outdoor, and mixed-scene settings.
Quantitative Comparisons: Lotus-G achieves the overall best rank among generative depth baselines across the evaluation datasets.Lotus-D remains comparable to the DepthAnything series while using substantially fewer training images.
Quantitative Comparisons: Both Lotus-G and Lotus-D outperform all other generative and discriminative methods on zero-shot surface-normal estimation.The comparison covers multiple evaluation datasets and reports mean angular error and threshold-based pixel percentages.
Efficiency: Lotus requires only single-step denoising, significantly increasing inference speed relative to multi-step diffusion methods.The speed improvement is reported alongside the depth comparison.

6 CONCLUSION AND FUTURE WORK

The paper concludes that Lotus adapts pre-trained diffusion priors to dense prediction through a tailored protocol. Experiments show promising zero-shot depth and normal-estimation performance with minimal training data, while the work points to practical applications and future extensions.

Conclusion: Lotus is introduced as a diffusion-based visual foundation model with a fine-tuning protocol tailored to dense prediction.The conclusion presents this adaptation as fitting rich visual priors from pre-trained diffusion models to dense prediction.
Conclusion: Lotus achieves promising zero-shot performance on depth and normal estimation with minimal training data.The conclusion states that extensive experiments support this result.
Applications and Future Work: The paper identifies practical applications including joint estimation and single- or multi-view 3D reconstruction.The conclusion refers readers to supplementary discussion of applications and future work.
Ablation Studies: Table 3 reports ablation studies on the step-by-step design of the adaptation protocol for monocular depth estimation.The table examines how the protocol components contribute to the dense-prediction adaptation.

A EXPERIMENTAL SETTINGS

The experiments implement Lotus from Stable Diffusion V2 without text conditioning and normalize annotation maps for the VAE input range. Training uses a fixed timestep, Adam optimization, and eight NVIDIA A800 GPUs.

Implementation Details: Lotus is implemented from Stable Diffusion V2 with text conditioning disabled.Both depth and normal maps are normalized to [−1, 1] to match the VAE’s designed input range.
Optimization: Training fixes the diffusion timestep at t = 1000 and uses Adam with learning rate 3 × 10^-5.The fixed timestep is used throughout training.
Compute: All experiments run on 8 NVIDIA A800 GPUs with a total batch size of 128.

A.2 EVALUATION DATASETS AND METRICS

The evaluation protocol combines multi-dataset metrics with diffusion-formulation analyses. It measures aligned depth accuracy, angular normal-estimation accuracy, average rank, and the effects of parameterization, single-step diffusion, and detail preservation.

Evaluation Metrics: Affine-invariant depth evaluation aligns predictions to ground truth using least-squares fitting before computing AbsRel, δ1, and δ2.AbsRel measures relative depth error, while δ1 and δ2 report proportions of pixels within specified ratio thresholds.
Evaluation Metrics: Surface-normal evaluation reports mean angular error and the percentages of pixels below 11.25° and 30° angular-error thresholds.
Evaluation Metrics: Average rank summarizes each method’s ranking across datasets and metrics, with lower values indicating better overall performance.
Direct Adaptation: Direct Adaptation encodes the image and annotation with a pretrained VAE, adds noise to the annotation, concatenates image and noisy-annotation latents, and applies a U-Net denoiser.Its input layer expands from 4 to 8 channels, and it uses the standard multi-step epsilon-prediction objective.
Ablation Analysis: Without AMRN, multi-step performance follows ε-pred. < v-pred. < x0-pred., validating x0-pred. + single step + detail preserver as the best protocol.With AMRN, differences between parameterizations become minimal because AMRN reduces variance in multi-step diffusion models.

C ANALYSIS OF “DIRECTION(zyτ )” IN DDIM PROCESS (EQ. 4)

The analysis compares diffusion parameterizations for dense prediction and finds that x0-prediction achieves the best performance with the lowest variance, while v-prediction is intermediate. In x0-prediction, the potentially variance-amplifying factor matters mainly near the final denoising steps, where network-output differences also weaken.

DDIM direction term: The direction(zyτ) term is weighted by wτ = √1 −ατ−1 and is calculated according to the chosen parameterization.The supplied derivation identifies d as the direction term and fθ as the model output for each parameterization.
Variance analysis: In x0-prediction, √1 −ατ approaches zero as τ approaches 1, which may amplify variance from the model output.The factor remains close to 1 for most of denoising and approaches zero mainly in the final steps.
Variance analysis: The factor’s influence is limited because x0-prediction’s network-output gap weakens in the final steps and gradually approaches zero.As √ατ approaches 1, the relevant network-output difference approaches zero as well.
Parameterization comparison: v-prediction combines added noise and the clean sample, balancing ϵ-prediction and x0-prediction.Its predicted variable is v = √ατϵ − √1 −ατzy.
Parameterization comparison: x0-prediction achieves the best performance with the lowest variance, whereas v-prediction falls between x0- and ϵ-prediction with moderate variance.Because minimizing variance is crucial for stable dense prediction, the method replaces standard ϵ-prediction with x0-prediction.

E EXPERIMENTS ON MORE DENSE PREDICTION TASKS: SEMANTIC SEGMENTATION AND DIFFUSE REFLECTANCE

Lotus is evaluated on semantic segmentation and diffuse reflectance prediction using Hypersim training and test sets. It outperforms Direct Adaption across all reported metrics, indicating applicability beyond geometric dense prediction without task-specific designs.

Experimental setup: The experiments train both tasks on Hypersim and evaluate them on their corresponding test sets.The evaluation reports mean intersection over union and mean accuracy for segmentation, and L1 and L2 distances for diffuse reflectance.
Results: Lotus outperforms the Direct Adaption baseline across all reported semantic segmentation and diffuse reflectance metrics.Semantic segmentation uses mIoU and mAcc; diffuse reflectance uses L1 and L2 distances to ground truth.
Qualitative results: Qualitative visualizations show accurate and high-quality results for semantic segmentation and diffuse reflectance prediction.The visual results are presented in Fig. 12.
Generalization: Without task-specific designs, Lotus can be applied to semantic dense prediction as well as geometric dense prediction tasks.The conclusion is supported by both quantitative comparisons and qualitative visualizations.

F FREQUENCY DOMAIN ANALYSIS OF THE DETAIL PRESERVER TAKE MONOCULAR DEPTH ESTIMATION AS AN EXAMPLE

The frequency-domain analysis shows that Detail Preserver makes depth estimates’ frequency characteristics more closely match input-image and ground-truth geometric details. Lotus also uses one-step diffusion and produces finer predictions in complex regions, supporting reconstruction and joint-estimation applications, while performance remains slightly behind a large-data baseline.

Frequency-domain analysis: Detail Preserver brings depth-estimation frequency energy closer to the input image, especially in middle and low frequencies.The affected frequency groups are 0, 1, 2, and 3; the analysis uses FFT on 100 randomly selected Hypersim pairs.
Frequency-domain analysis: With Detail Preserver, the input-to-estimation frequency-energy ratio is closer to the input-to-ground-truth ratio than without it.This comparison indicates enhanced geometric details in depth predictions.
One-step diffusion: The one-step diffusion formulation fixes the sole training timestep to T, while violating this rule causes performance degradation.The timestep study trains on Hypersim and evaluates on NYUv2 without Detail Preserver or mixture-dataset training.
Qualitative comparisons: Lotus produces finer and more accurate depth predictions in detailed or complex regions and handles surface-normal prediction across complex geometries and diverse environments.These qualitative comparisons highlight fine-grained prediction and robustness, including cases not fully reflected by metrics.
Applications: Lotus supports depth-to-point-cloud conversion, joint depth-and-normal estimation with 100% shared parameters, and single- and multi-view 3D reconstruction.Multi-view reconstruction can use Lotus depth and normal predictions without RGB supervision.
Scope and future work: Lotus has been applied to two geometric dense prediction tasks and performs slightly behind DepthAnything, which uses large-scale training data.Scaling training data is identified as having potential to further improve Lotus’s performance.