PaperlayerView exact revision on arXiv

Source-linked AI summary

Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis

Bingxin Ke, Kevin Qu, Tianfu Wang, Nando Metzger, Shengyu Huang, Bo Li, Anton Obukhov, Konrad Schindler

arXiv:2505.09358v1cs.CVcs.LG

TL;DR

Dense image analysis needs strong pretrained models, but adapting generative vision models efficiently remains an open opportunity. Marigold fine-tunes pretrained latent diffusion models with synthetic data for depth, surface normals, and intrinsic decomposition, retaining a generative formulation. It achieves competitive or state-of-the-art zero-shot performance while training in under 3 GPU-days and supporting sub-100ms single-step inference.

  • Problem

    Dense image analysis in data-scarce settings depends on effective transfer from strong pretrained models, motivating reuse of visual knowledge in text-to-image latent diffusion models.

  • Method

    Marigold fine-tunes pretrained Stable Diffusion latent diffusion models with synthetic data for generative depth, surface-normal, and intrinsic-decomposition prediction.

  • Results

    Marigold provides competitive performance across depth, surface normals, and intrinsic decomposition, with training under 3 GPU-days and single-step inference under 100ms.

  • Takeaways & Limitations

    Pretrained latent diffusion visual priors can support affordable, zero-shot dense image analysis across multiple modalities.

Abstract

from arXiv · show

The success of deep learning in computer vision over the past decade has hinged on large labeled datasets and strong pretrained models. In data-scarce settings, the quality of these pretrained models becomes crucial for effective transfer learning. Image classification and self-supervised learning have traditionally been the primary methods for pretraining CNNs and transformer-based architectures. Recently, the rise of text-to-image generative models, particularly those using denoising diffusion in a latent space, has introduced a new class of foundational models trained on massive, captioned image datasets. These models' ability to generate realistic images of unseen content suggests they possess a deep understanding of the visual world. In this work, we present Marigold, a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models like Stable Diffusion and adapts them for dense image analysis tasks, including monocular depth estimation, surface normals prediction, and intrinsic decomposition. Marigold requires minimal modification of the pre-trained latent diffusion model's architecture, trains with small synthetic datasets on a single GPU over a few days, and demonstrates state-of-the-art zero-shot generalization. Project page: https://marigoldcomputervision.github.io

I. INTRODUCTION

Marigold repurposes pretrained latent diffusion image generators for zero-shot dense image analysis through an affordable fine-tuning protocol. Its design combines latent-space modality encoding, synthetic-data training, conditional generative prediction, and efficient inference.

  • Motivation and approach: Marigold adapts Stable Diffusion into a zero-shot generative image-analysis model for depth, surface normals, and intrinsic decomposition.The protocol leverages visual knowledge stored in pretrained generative image models.
  • Motivation and approach: Encoding both input images and output modalities in the pretrained VAE enables latent-space transfer while requiring noise-free, pixel-complete targets.Real depth ground truth often contains missing or invalid pixels, motivating synthetic data.
  • Motivation and approach: Synthetic data supports short fine-tuning by providing diverse, consistently labeled, noise-free supervision and dense, complete depth targets.These properties reduce noise in weight updates and permit VAE processing of the target modality.
  • Generative formulation: Conditional diffusion models represent multiple plausible outputs, with different noise samples producing predictions that can be ensembled to approximate a conditional mode.This formulation addresses ill-posed cases such as blur, over-exposure, and transparent-object ambiguity.
  • Efficiency and extensions: Single-step inference, scheduler correction, lightweight VAE use, and low-precision quantization reduce runtime to under 100ms on most commodity hardware.The study retains the generative formulation while extending it to additional dense analysis tasks and distillation protocols.
  • Efficiency and extensions: Training takes less than 3 GPU-days on commodity hardware, while the contributions include fast inference and high-resolution inference.The protocol is presented as a resource-efficient conversion of a foundational latent diffusion model.

II. RELATED WORK

Marigold builds on latent diffusion models and prior diffusion-based analysis work while retaining a generative formulation. Across surface normals and intrinsic decomposition, the paper reports competitive or leading performance with comparatively simple models and training choices.

  • Diffusion and foundation models: Stable Diffusion uses an LDM trained on 2.3B text-image pairs, whose latent-space denoising provides rich visual priors for image analysis transfer.LDMs run diffusion in the compressed latent space of a pretrained VAE, reducing mapping complexity.
  • Depth estimation: Marigold differs from prior diffusion-based depth methods by fine-tuning a pretrained LDM rather than training from scratch, using custom latent spaces, or requiring extensive training.The authors retain the generative formulation and incorporate a DDIM inference correction.
  • Surface normals: For surface normals, curated synthetic datasets and the vanilla Marigold protocol achieve the best performance on most evaluation datasets among recent diffusion-based methods.The comparison excludes additional foundational models, multi-task aggregation, privileged information, and refinement stages.
  • Intrinsic decomposition: Intrinsic image decomposition recovers albedo, shading, residual components, roughness, and metallicity as task-specific image outputs.Marigold-IID uses separate models for appearance-related material properties and lighting decomposition.
  • Intrinsic decomposition: The simpler Marigold-IID models achieve highly competitive quantitative and qualitative performance on InteriorVerse and HyperSim.They are compared with IID-Diffusion, RGB↔X, and Careaga et al. in their respective domains.

F. High-Resolution Estimation

Marigold addresses resolution bias by introducing a high-resolution inference strategy while retaining its conditional latent-diffusion formulation. The strategy is reported to achieve competitive or better performance than other high-resolution methods.

  • Resolution challenge: Stable Diffusion-derived models default to processing resolution 768, causing large inputs to lose detail during downsampling and upsampling.This resolution bias motivates the high-resolution inference strategy.
  • Resolution challenge: High-resolution depth methods include patch fusion, refinement, and tiled diffusion approaches, while Depth Pro uses a multi-scale transformer for HD images.These methods provide comparison points for Marigold-HR.
  • Results: Marigold-HR attains competitive or better performance compared to other high-resolution estimation methods.The comparison is reported in the paper's high-resolution estimation evaluation.
  • Generative formulation: Marigold models monocular depth estimation as conditional denoising diffusion over depth given an RGB image.The model progressively denoises a depth representation conditioned on the input image.
  • Generative formulation: The protocol encodes images and depth maps with a VAE, trains the denoiser in latent space, and decodes the estimated clean latent into depth.This formulation preserves the latent-space pipeline during inference.

B. Network Architecture

Marigold adapts Stable Diffusion into a conditional depth generator with minimal architectural changes. It uses the original VAE, modifies the U-Net input for image conditioning, normalizes depth affinely, and trains exclusively on synthetic data.

  • Network architecture: Marigold builds on pretrained Stable Diffusion and turns it into a conditional depth-map generator with minimal model changes.The adaptation targets the pretrained latent diffusion model rather than replacing its architecture.
  • Network architecture: The frozen VAE encodes both RGB images and depth maps, with single-channel depth replicated to three channels for the RGB-trained encoder.The encoded depth can be reconstructed with negligible error without modifying the VAE or latent-space structure.
  • Network architecture: Image and noisy-depth latents are concatenated, doubling the denoiser input channels and requiring a modified first U-Net layer.The duplicated input-layer weights are divided by two to preserve activation magnitudes and the pretrained structure.
  • Depth normalization: Affine depth normalization maps values primarily to [−1, 1] and uses each map's 2% and 98% depth percentiles.The representation is designed to focus on affine-invariant depth estimation independently of dataset statistics.
  • Training data: Training uses exclusively synthetic depth because it is dense, complete, and clean, reducing noise in gradient updates.The VAE cannot handle invalid pixels, making complete synthetic depth suitable for this protocol.

D. Inference

Marigold inference encodes the input image, initializes a depth latent with Gaussian noise, and progressively denoises it before VAE decoding. Multiple stochastic predictions can be aligned and merged to trade computation for prediction quality.

  • Inference pipeline: Inference encodes the image into latent space, initializes the depth latent with Gaussian noise, and progressively denoises it using DDIM.The final latent is decoded by the VAE and postprocessed by averaging channels.
  • Test-time ensembling: Different initial noise samples produce varying predictions for the same input, enabling test-time ensembling.The method runs inference N times and jointly estimates scale and shift for the predictions.
  • Test-time ensembling: The ensemble aligns affine-invariant predictions, takes their pixel-wise median, and regularizes the result to prevent collapse and enforce unit scale.The merged depth map is the final ensembled prediction.
  • Test-time ensembling: Choosing the number of inference passes N provides a flexible tradeoff between computation efficiency and prediction quality.The alignment requires no ground truth.

E. Implementation

Marigold-Depth uses Stable Diffusion with synthetic training data and test-time ensembling for affordable zero-shot depth estimation. It outperforms prior methods in low-data evaluation while supporting efficient few-step inference.

  • Implementation: Stable Diffusion v2 is fine-tuned for 18K iterations on synthetic depth data, using DDPM training and 1–50 DDIM inference steps.Training fits one GPU through a real batch size of 2 and gradient accumulation.
  • Implementation: Marigold-Depth trains on HyperSim and evaluates on five unseen real datasets spanning indoor, outdoor, and high-resolution scenes.The evaluation includes NYUv2, ScanNet, KITTI, ETH3D, and DIODE.
  • Evaluation: Marigold outperforms prior zero-shot depth methods in the low-data regime without real depth samples and remains competitive with methods trained on larger datasets.The comparison covers indoor and outdoor scenes.
  • Evaluation: Marigold captures scene layout, thin structures, flat surfaces, and detailed geometry in qualitative depth comparisons across datasets.Examples include chair legs, walls, and chairs in ETH3D and DIODE.
  • Ablations: DDIM trailing timesteps make one-step inference sufficient, whereas the original leading-timestep DDIM setting requires 4–10 function evaluations for peak performance.Marigold v1.1 uses trailing timesteps by default, with performance saturating at NFE=1.

IV. SURFACE NORMALS ESTIMATION MODEL

Marigold-Normals adapts the Marigold protocol to monocular surface-normal estimation with minimal task-specific changes. Its qualitative comparisons emphasize robustness to complex layouts, motion blur, and reflections.

  • Motivation: Surface normals encode 3D geometry as three-dimensional unit vectors, whereas depth assigns each pixel a single positive scalar.Real normal maps are difficult to collect outside simulation or controlled environments.
  • Evaluation: Compared with baseline methods, Marigold-Normals handles complex scene layouts and is more robust to motion blur and reflections.The qualitative comparison covers monocular surface-normal estimation across datasets.
  • Method: Marigold-Normals closely follows Marigold-Depth, streaming raw normal maps into the VAE and normalizing outputs along channels to produce unit-length predictions.No range normalization or channel replication is required.
  • Ablations: Increasing the ensemble size consistently improves Marigold-Normals performance, with diminishing returns beginning after 10 predictions per sample.The trend mirrors the depth model’s test-time ensembling behavior.
  • Method: Test-time ensembling averages multiple normal predictions, normalizes the mean, and selects per-pixel vectors with maximum cosine similarity to that mean.This aggregates stochastic predictions into a single unit-length normal map.

B. Implementation

Marigold extends its affordable diffusion-based adaptation to surface normals and intrinsic image decomposition using synthetic data. The models achieve strong zero-shot performance, while denoising steps trade efficiency against visible high-frequency detail.

  • Implementation: Marigold-Normals is fine-tuned for 26K iterations on three synthetic datasets covering indoor and outdoor scenes.The datasets are HyperSim, InteriorVerse, and Sintel.
  • Implementation: Evaluation uses five unseen normal-estimation benchmarks and reports mean angular error plus the percentage of pixels below 11.25° angular error.The benchmarks include indoor, outdoor, and in-the-wild imagery.
  • Ablations: Four denoising steps provide the best benchmark performance for Marigold-Normals, although one step is visually sufficient in most cases.Increasing steps can improve high-frequency detail without necessarily improving evaluation metrics.
  • Evaluation: Marigold-Normals consistently outperforms most baselines across datasets and metrics, including more complex diffusion-based strategies.The qualitative results remain accurate in challenging scene layouts and scenarios.
  • Ablations: Increasing denoising steps makes high-frequency details more pronounced, but benchmark metrics may not improve because ground truth often masks or over-smooths those regions.The figure illustrates this effect using 1, 4, and 20 steps.
  • Intrinsic Image Decomposition: Marigold-IID introduces Appearance and Lighting models for structured intrinsic image decomposition using physically meaningful reflectance and illumination components.The Lighting model represents the image as I = A · S + R.

A. Method

Marigold adapts diffusion models for intrinsic image decomposition by predicting multiple normalized component images, with variants for appearance and lighting. The models achieve quantitatively strong and qualitatively clean decompositions, including robustness to varying environmental lighting.

  • Intrinsic decomposition: IID-Appearance predicts albedo plus roughness and metallicity, while IID-Lighting predicts albedo, shading, and residual components.Appearance uses two output images; Lighting uses three, with intrinsic values normalized to the unit range.
  • Evaluation: Marigold outperforms two competing methods on InteriorVerse and achieves competitive performance on HyperSim.The comparisons use the InteriorVerse and HyperSim test sets for the respective intrinsic decomposition variants.
  • Architecture: The U-Net is modified for multiple predicted images by replicating input and output channels according to the number of outputs.The first convolution receives P + 1 replicated input channels, while the final layer produces P replicated output channels.
  • Training: 40K iterations on 45K InteriorVerse samples train IID-Appearance, while 36K iterations on 24K HyperSim samples train IID-Lighting.Both training pipelines resize samples to 480 × 640 and apply the stated color-space preprocessing.
  • Results: The method produces quantitatively more accurate and qualitatively cleaner decompositions, with consistent predictions across varying environmental lighting conditions.The qualitative comparisons report less baked-in shading and greater consistency with ground truth.

VI. LATENT CONSISTENCY MODEL (LCM)

Marigold-LCM distills the standard Marigold model into a latent consistency variant for one- or few-step inference. Training uses teacher, target, and student replicas, with consistency optimization and exponential moving-average target updates.

  • Motivation: Marigold-LCM is a Marigold variant designed to achieve similar prediction results in one or a few denoising steps.It adapts the latent consistency model approach from fast image generation to image analysis.
  • Model setup: Distillation initializes teacher, target, and student models from Marigold and updates only the U-Net while keeping the VAE frozen.The teacher is frozen; the student is optimized, and the target tracks the student through EMA.
  • Consistency training: The teacher denoises a randomly noised depth latent by a DDIM step, while student and target outputs are matched through a self-consistency objective.The DDIM step uses k = 200 in the reported implementation.
  • Optimization: Student weights are updated by gradient descent, while target weights follow a running average with decay µ = 0.95.The target update provides the moving reference used during consistency distillation.

B. Inference

Marigold inference extends from standard prediction to fast and high-resolution processing. Latent consistency distillation enables one-step inference, while Marigold-HR combines a global depth estimate with overlapping-tile diffusion refinement.

  • LCM results: Marigold-LCM generally does not outperform 50-step Marigold, but it surpasses prior art on most datasets and metrics.The comparison supports the viability of latent consistency distillation for Marigold and other modalities.
  • High-resolution challenges: High-resolution depth estimation is limited by fixed receptive fields and excessive memory use, while downsampling and patchwise processing can lose detail or global consistency.The native Stable Diffusion-derived processing resolution is 768 for Marigold models.
  • Marigold-HR: Marigold-HR first predicts a global depth map at native resolution, then uses it as conditioning for patch-based diffusion upsampling.The high-resolution pipeline performs the upsampling through overlapping tiles and a MultiDiffusion forward pass.

A. Method

Marigold-HR addresses high-resolution depth limitations by refining a globally consistent prediction with conditioned overlapping-tile diffusion. It achieves best or second-best performance across reported metrics and captures fine-grained details qualitatively.

  • Method: Marigold-HR initializes refinement from a global depth prediction that provides a coarse scene representation.The global prediction is produced at the model’s native processing resolution before refinement.
  • Method: A second diffusion model upsamples the global depth map by 2× using the resampled RGB image and the global prediction as conditioning.The refinement operates on high-resolution inputs while retaining the coarse global estimate.
  • Method: Overlapping tiles keep memory usage bounded, while MultiDiffusion synchronizes their latent representations on a shared canvas.Tile transforms map local latent variables back to the global canvas, and per-pixel blending weights combine tile predictions.
  • Training: The refiner resumes Marigold-Depth training for 12K iterations with additional conditioning on the lower-resolution inference.Training uses aligned global predictions and masks dissimilar patches so the refiner follows its conditioning.
  • Results: Marigold-HR achieves the best or second-best performance in all metrics, while Depth Pro leads global metrics and Marigold-HR is slightly better on Booster edge metrics.Evaluation uses affine-invariant depth metrics and edge-quality measures on Middlebury 2014 and Booster.
  • Results: Qualitative results show plausible high-resolution predictions that capture fine-grained details such as a cat’s whiskers.The reported examples include both benchmark and in-the-wild images.

D. Ablations

The high-resolution ablations examine global conditioning and MultiDiffusion, showing that their combination balances global context, fine edge detail, and moderate memory use. Marigold-HR improves the base model particularly on edge-quality metrics while remaining competitive with state-of-the-art estimators.

  • The ablation evaluates global conditioning and MultiDiffusion inference as Marigold’s two main high-resolution design choices.
  • Marigold-HR improves the base model particularly on edge-quality metrics, reaching performance competitive with current state-of-the-art depth estimators.
  • Marigold-HR produces fine-grained outputs while maintaining global context across aligned benchmark predictions and normalized in-the-wild examples.
  • Global conditioning improves edge metrics but slightly worsens global metrics, while MultiDiffusion improves high-resolution edges with bounded memory but loses global context.
  • Combining global conditioning with MultiDiffusion yields the best overall performance, balancing global and edge-focused metrics with memory costs below 15GB.
Loading 2505.09358v1…