PaperlayerView exact revision on arXiv

Source-linked AI summary

Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model

Jing He, Haodong Li, Mingzhi Sheng, Ying-Cong Chen

arXiv:2512.01030v3cs.CV

TL;DR

Single-image geometric prediction is ill-posed, while discriminative models are limited by supervised-data coverage and diffusion models’ stochastic generation is mismatched to deterministic inference. Lotus-2 adapts diffusion world priors with a two-stage deterministic framework and, using 59K samples, achieves state-of-the-art monocular depth and highly competitive surface-normal prediction.

  • Problem

    Single-image geometric prediction is ill-posed, and discriminative models remain limited by the scale, quality, and diversity of geometric supervision.

  • Method

    Lotus-2 treats diffusion weights as structured world priors and separates single-step clean-data structure prediction from constrained multi-step rectified-flow detail refinement.

  • Results

    Using 59K training samples, Lotus-2 achieves new state-of-the-art monocular depth results and highly competitive surface-normal results.

  • Takeaways & Limitations

    Diffusion models can be repurposed as deterministic world priors for efficient, accurate, and fine-grained geometric reasoning.

Abstract

from arXiv · show

Recovering pixel-wise geometric properties from a single image is fundamentally ill-posed due to appearance ambiguity and non-injective mappings between 2D observations and 3D structures. While discriminative regression models achieve strong performance through large-scale supervision, their success is bounded by the scale, quality, and diversity of available data, as well as by limited physical reasoning. Recent diffusion models exhibit powerful world priors that encode geometry and semantics learned from massive image-text data, yet directly reusing their stochastic generative formulation is suboptimal for deterministic geometric inference: the former is optimized for diverse and high-fidelity image generation, whereas the latter requires stable and accurate predictions. In this work, we propose Lotus-2, a two-stage deterministic framework for stable, accurate and fine-grained geometric dense prediction, aiming to provide an optimal adaptation protocol to fully exploit the pre-trained generative priors. Specifically, in the first stage, the core predictor employs a single-step deterministic formulation with a clean-data objective and a lightweight local continuity module (LCM) to generate globally coherent structures without grid artifacts. In the second stage, the detail sharpener performs a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor, enhancing fine-grained geometry through noise-free deterministic flow matching. Using only 59K training samples, less than 1% of existing large-scale datasets, Lotus-2 establishes new state-of-the-art results in monocular depth estimation and highly competitive surface normal prediction. These results demonstrate that diffusion models can serve as deterministic world priors, enabling high-quality geometric reasoning beyond traditional discriminative and generative paradigms.

I. INTRODUCTION

Geometric dense prediction is ill-posed and conventional approaches remain limited by data coverage and physical reasoning. Lotus-2 repurposes diffusion world priors through a deterministic two-stage framework, achieving strong depth and normal estimation with little training data.

  • I. INTRODUCTION: The task is inherently ill-posed because one image can correspond to multiple plausible three-dimensional interpretations.Geometric dense prediction supports downstream applications including controllable image generation, 3D/4D reconstruction, and autonomous driving.
  • I. INTRODUCTION: Discriminative models remain constrained by the scale, quality, and diversity of geometric data, especially in rare scenes requiring reasoning beyond observable appearance.Examples include transparency, reflection, and low-texture conditions.
  • I. INTRODUCTION: Lotus-2 reframes diffusion models as structured world priors whose pretrained weights, rather than stochastic sampling trajectories, support deterministic geometric inference.This perspective motivates adapting generative models for stable dense prediction instead of directly reusing image-generation procedures.
  • I. INTRODUCTION: The optional detail sharpener applies constrained multi-step rectified-flow refinement within the core predictor’s manifold, adding fine-grained detail while preserving global structure.This separates structural accuracy from detail refinement and combines regression stability with generative expressiveness.
  • I. INTRODUCTION: The core predictor uses single-step clean-data regression and a local continuity module to produce stable, globally coherent geometry without grid artifacts.The local continuity module addresses artifacts from FLUX Pack–Unpack operations while preserving compatibility and efficiency.
  • I. INTRODUCTION: With 59K training samples, Lotus-2 achieves new state-of-the-art monocular depth results and highly competitive surface-normal results.The reported training set is 0.66% of MoGe’s and 0.09% of DepthAnything’s data usage.

II. RELATED WORK

Geometric prediction has progressed from assumption-heavy physical reasoning to data-driven regression, but both paradigms leave challenges in coverage and physical reasoning. Large diffusion models offer pretrained world priors, motivating Lotus-2’s deterministic adaptation of FLUX.

  • A. Traditional Paradigms for Geometric Dense Prediction: Physics-based methods rely on strong assumptions including multiple views, camera calibration, or Lambertian reflectance, limiting their applicability to complex single-view scenes.Examples include structure from motion, photometric stereo, and multi-view geometry.
  • A. Traditional Paradigms for Geometric Dense Prediction: Discriminative models struggle with limited geometric-data diversity and out-of-distribution scenes such as reflective, transparent, or rare compositions.Their limitations are attributed to dependence on distributional coverage rather than intrinsic physical laws governing scene structure.
  • B. World Priors from Generative Models: Modern diffusion models trained on billions of image–text pairs encode rich priors about geometry, semantics, and physical structure.These data volumes substantially exceed available dense geometric annotation, motivating their use for visual reasoning.
  • B. World Priors from Generative Models: FLUX provides Lotus-2’s architectural foundation because its rectified-flow DiT design delivers high visual quality and strong physical consistency.The paper identifies FLUX as the optimal world prior for geometric dense prediction.
  • C. Repurposing Generative Priors for Dense Prediction: Lotus-2 decouples structure prediction from detail refinement to address stochasticity, efficiency, and detail loss in generative dense-prediction adaptations.Its core predictor and detail sharpener use deterministic rectified-flow mappings for physically consistent, fine-grained reasoning.

A. Rectified-Flow Formulation

Rectified flow deterministically transports samples between source and target distributions along an ordinary differential equation. Training learns the velocity field, while inference uses discrete Euler steps to generate the target sample.

  • A. Rectified-Flow Formulation: Rectified flow models transport from source distribution p1 to target distribution p0 through the ODE d z_t = v(z_t,t)dt.The formulation maps z1 sampled from p1 to z0 sampled from p0 using a velocity field.
  • A. Rectified-Flow Formulation: Its target velocity follows a straight-line path, v = d z_t/dt = z1 − z0, reducing error accumulation relative to high-curvature denoising paths.A neural network f_θ is trained to approximate this target vector field.
  • A. Rectified-Flow Formulation: Training approximates the continuous-time expectation by randomly sampling discrete time-step values from a predefined set.The formulation uses a total of T training time-steps.
  • A. Rectified-Flow Formulation: During inference, a discrete Euler solver iteratively moves from the source state at t = 1 toward the target state at t = 0.The step size η satisfies 0 < η ≤ 1 and is determined by the number of inference steps.
  • A. Rectified-Flow Formulation: The stochastic adaptation estimates velocity from random noise latent ϵ to annotation latent z_y conditioned on image latent z_x, introducing output variance.Its target velocity is v = ϵ − z_y.

B. Architectural Foundation of FLUX

FLUX provides a VAE-latent rectified-flow architecture whose Pack-Unpack efficiency introduces grid artifacts, while Lotus-2 adapts its generative components into deterministic dense prediction.

  • FLUX latent architecture: FLUX operates in a VAE latent space, encoding images with E and reconstructing them with D around a diffusion-transformer backbone.Its standard image-generation flow starts from Gaussian noise and transports samples toward clean image latents.
  • Pack-Unpack operations: Pack groups nonoverlapping 2 × 2 latent patches into channels and Unpack restores spatial resolution, reducing computation but creating local discontinuities.These parameter-free operations can produce grid artifacts, especially under single-step prediction, degrading geometric fidelity.
  • Stochastic versus deterministic formulation: Stochastic adaptation begins from Gaussian noise, so different initializations produce inconsistent geometric structures and averaging can blend conflicting hypotheses.The deterministic formulation instead improves structural consistency and prediction accuracy relative to its stochastic counterpart.
  • Deterministic adaptation: Lotus-2 reframes diffusion adaptation as deterministic flow matching from image latent z_x to annotation latent z_y, replacing stochastic noise initialization with a noise-free transformation.Deterministic-DA predicts the velocity v = z_x − z_y, targeting stable geometric inference rather than diverse image generation.
  • Core predictor: The core predictor uses a single-step clean-data formulation with z_t = z_1 = z_x, while a local continuity module mitigates Unpack-induced artifacts.Experiments on NYUv2 report that T = 1 performs best across all evaluated data scales.

2) Analysis-2, Multi-Step Iterative Sampling:

The analysis finds that reducing rectified-flow training steps improves geometric dense prediction under limited data, with a single-step formulation offering the best efficiency and performance.

  • Multi-Step Iterative Sampling: Multi-step sampling is computationally intensive, accumulates prediction errors, and slows inference for dense geometric prediction.Its design is optimized for high-fidelity image synthesis and demands large-scale training data.
  • Multi-Step Iterative Sampling: Single-step training (T = 1) produces the best result as training time-steps are reduced, especially under limited data.Multi-step formulations are more sensitive to training-data scale, while the single-step setting constrains optimization and improves adaptation efficiency.
  • Multi-Step Iterative Sampling: Reducing the number of training time-steps constrains the optimization space and enables more effective adaptation for geometric dense prediction.

3) Analysis-3, Parameterization Types:

The parameterization analysis favors directly predicting clean geometric annotations over residuals, because residual prediction mixes geometry with image appearance and is harder to optimize.

  • Parameterization Types: Residual prediction requires learning image reconstruction and geometric estimation simultaneously, increasing optimization difficulty.
  • Parameterization Types: Residuals are dominated by high-frequency appearance signals, and imperfect removal causes appearance interference to leak into final predictions.
  • Parameterization Types: The model therefore adopts clean-data prediction as a simpler objective that directly predicts the clean annotation.
  • Parameterization Types: Clean-data prediction achieves significantly higher accuracy than residual prediction and avoids image-pattern interference in geometric outputs.The comparison shows residual prediction producing appearance artifacts, whereas direct clean-annotation prediction yields accurate results.

4) Analysis-4, Local Continuity:

The local continuity analysis combines a lightweight module with the deterministic predictor to remove Pack–Unpack grid artifacts while preserving efficiency, then adds constrained refinement for sharper detail.

  • Local Continuity: Pack–Unpack creates spatial discontinuities at 2×2 latent-patch boundaries that harm geometric fidelity.
  • Local Continuity: LCM eliminates grid artifacts while improving prediction accuracy and preserving model efficiency.The module addresses local discontinuities introduced by Pack–Unpack without removing those operations.
  • Local Continuity: Removing Pack–Unpack avoids grid artifacts but requires extra alignment layers, shifts features away from pretrained priors, and degrades accuracy.
  • Core Predictor: The final core predictor integrates single-step prediction, clean-data parameterization, and LCM into an efficient deterministic formulation.
  • Detail Sharpener: The detail sharpener refines a structurally correct coarse prediction through constrained multi-step rectified flow within the core predictor’s manifold.It learns a noise-free transformation from coarse to high-fidelity geometry using a velocity between the two geometric states.
  • Detail Sharpener: Detail sharpening improves fine-grained structure without the geometric hallucinations seen in unconstrained multi-step flow, while preserving core-predictor accuracy.Inference refinement uses up to 10 steps, adjustable to the desired sharpness.

C. Inference

Lotus-2 uses a two-stage deterministic inference pipeline: the core predictor establishes accurate coarse structure, and an optional sharpener refines high-frequency detail without stochastic noise.

  • Inference: Both stages are noise-free, separating structural correctness and efficiency from optional high-fidelity refinement.
  • Inference: The core predictor produces an accurate but coarse geometric result in one step, then the optional detail sharpener iteratively generates a sharper high-fidelity prediction.The final refined latent is decoded through the VAE decoder into the geometric prediction.
  • Inference: The detail sharpener improves fine-grained boundaries while avoiding the structural hallucinations observed in Deterministic-DA.

V. EXPERIMENTS

Lotus-2 is evaluated as a two-stage system trained on approximately 59K synthetic samples, with a core predictor followed by detail-sharpener refinement. The experiments assess its implementation and limited-data training setup.

  • Implementation: The system fine-tunes pretrained FLUX without text conditioning, using a single-step core predictor and a multi-step detail sharpener.The detail sharpener is trained from coarse predictions generated by the core predictor and ground-truth targets.
  • 2) Training Datasets:: Lotus-2 trains both depth and normal estimation solely on approximately 59K synthetic samples, far fewer than large-scale discriminative models.The training data comprise filtered Hypersim and Virtual KITTI samples.

3) Evaluation Datasets:

Lotus-2 is tested on unseen real-world datasets spanning indoor, outdoor, mixed-resolution, and dynamic scenes. Evaluation uses standard depth and surface-normal metrics plus average rank for aggregate comparison.

  • Evaluation Datasets:: Lotus-2 generalizes to five unseen real-world depth datasets and four unseen surface-normal datasets.The depth benchmarks cover indoor, outdoor, and mixed scenes, while normal prediction includes indoor and dynamic outdoor scenes.
  • Evaluation Metrics: Depth evaluation reports AbsRel and δ1 after least-squares alignment, while surface-normal evaluation reports mean angular error and the percentage below 11.25°.AbsRel and angular error are lower-is-better metrics; δ1 and the below-11.25° percentage are higher-is-better metrics.
  • Evaluation Metrics: Average rank aggregates performance across all datasets and metrics, with lower values indicating superior overall performance.The measure supports cross-dataset comparison of competing methods.

B. Comparison with State-of-the-Art

Lotus-2 achieves the strongest reported depth benchmark performance and highly competitive surface-normal results against discriminative and generative-prior methods. Ablations attribute gains to deterministic, single-step, clean-data prediction and local continuity design choices.

  • 1) Affine-Invariant Depth Estimation:: Lotus-2 achieves the best overall performance in zero-shot affine-invariant depth estimation across five real-world datasets despite training on only 59K samples.Table I reports the best overall performance among the compared methods and the best average rank.
  • 2) Surface Normal Prediction:: Lotus-2 delivers highly competitive surface-normal performance while preserving robust and fine-grained predictions in challenging scenes.The reported qualitative advantage is linked to deterministic world-prior adaptation and noise-free refinement.
  • 1) Ablation on the Core Predictor:: Core-predictor ablations show consistent gains across four datasets as deterministic, single-step, clean-data prediction and LCM are added.The deterministic shift improves stability, single-step prediction improves efficiency and accuracy, clean-data prediction improves structural accuracy, and LCM suppresses grid artifacts.

2) Ablation on the Detail Sharpener:

The detail sharpener refines high-frequency geometry through constrained multi-step flow while preserving the core predictor’s structural accuracy. Spectral analysis provides signal-level evidence that refinement restores high-frequency power.

  • 2) Ablation on the Detail Sharpener:: The detail sharpener maintains near-optimal core-predictor accuracy while improving local high-frequency fidelity.The final ablation result shows refinement enhances detail without compromising established structural accuracy.
  • 2) Ablation on the Detail Sharpener:: Lotus-2 recovers high-frequency power that decays in the coarse core-predictor output, confirming the detail sharpener’s refinement effect.Figure 12 compares average log-power across spatial frequencies on NYUv2; Deterministic-DA and Lotus-2 retain more high-frequency power than the core predictor.
  • Conclusion: The two-stage design uses deterministic structural prediction first, then constrained noise-free refinement within the established geometry manifold.The core predictor uses single-step clean-data prediction and LCM, while the detail sharpener selectively enhances fine-grained geometry.
  • Conclusion: Training on only 59K synthetic samples yields state-of-the-art monocular depth results and highly competitive surface-normal results.The conclusion links data efficiency, inference stability, and fine-grained fidelity to the deterministic adaptation protocol.
Loading 2512.01030v3…