PaperlayerView exact revision on arXiv

Source-linked AI summary

SViM3D: Stable Video Material Diffusion for Single Image 3D Generation

Andreas Engelhardt, Mark Boss, Vikram Voleti, Chun-Han Yao, Hendrik P. A. Lensch, Varun Jampani

arXiv:2510.08271v2cs.GRcs.CVcs.LG

TL;DR

Estimating shape and material from a single naturally lit image is ill-posed, and existing single-image 3D methods may not provide relightable PBR materials without extra steps. SViM3D adapts video diffusion to jointly generate camera-controlled views with materials and normals, and reports state-of-the-art multi-view consistency and improved real-world material reproduction.

  • Problem

    Estimating shape and materials from a single image under natural illumination is highly ill-posed, while reflectance in generated 3D models may be simplified or require additional estimation for relighting and appearance edits.

    Source: §1 · pp. 1–2

  • Method

    SViM3D adapts a latent video diffusion model to jointly generate camera-controlled multi-view RGB, spatially varying PBR parameters, and surface normals, with reconstruction techniques to address view inconsistencies.

    Source: §1 · p. 2

  • Results

    SViM3D achieves state-of-the-art multi-view consistency and improves material reproduction in real-world settings across evaluations of novel-view synthesis, material prediction, relighting, and 3D generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The generated views and material maps can be relit, edited, or used as a neural prior for generating full 3D assets.

    Source: §1 · p. 2

    SViM3D focuses on object-centric images rather than general video applications, and its PBR representation cannot represent more complex materials such as transparent objects.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present Stable Video Materials 3D (SViM3D), a framework to predict multi-view consistent physically based rendering (PBR) materials, given a single image. Recently, video diffusion models have been successfully used to reconstruct 3D objects from a single image efficiently. However, reflectance is still represented by simple material models or needs to be estimated in additional steps to enable relighting and controlled appearance edits. We extend a latent video diffusion model to output spatially varying PBR parameters and surface normals jointly with each generated view based on explicit camera control. This unique setup allows for relighting and generating a 3D asset using our model as neural prior. We introduce various mechanisms to this pipeline that improve quality in this ill-posed setting. We show state-of-the-art relighting and novel view synthesis performance on multiple object-centric datasets. Our method generalizes to diverse inputs, enabling the generation of relightable 3D assets useful in AR/VR, movies, games and other visual media.

1. Introduction

SViM3D addresses the ill-posed problem of estimating shape and material properties from a single naturally illuminated image. It adapts video diffusion to generate consistent multi-view appearance and material outputs, supporting relighting, 3D reconstruction, and improved evaluation results.

1. Introduction

  • Single-image inverse rendering under natural illumination is highly ill-posed, yet accurate material prediction is needed for assets to integrate into new lighting and environments.Without precise material prediction, relighting becomes nearly impossible and assets can feel out of place.Source: §1 · p. 1
  • SViM3D adapts a camera-controllable video diffusion model to jointly produce multi-view-consistent RGB, spatially varying PBR parameters, and surface normals.Its outputs can be used for relighting, material edits, or 3D asset generation; reconstruction adds view-dependent masking, homography correction, and differentiable environment-based lighting.Source: §1 · p. 2
  • SViM3D achieves state-of-the-art multi-view consistency and improves material reproduction in real-world settings across evaluations of novel-view synthesis, material prediction, relighting, and 3D generation.The authors attribute the improvement in material reproduction to exploiting multi-view appearance consistency.Source: §1 · p. 2

2. Related works

Prior work addresses inverse rendering through controlled capture, multi-view scene representations, and diffusion-based material or 3D generation. SViM3D distinguishes itself by jointly estimating all material parameters consistently across camera-controlled views.

2. Related works

  • Inverse rendering methods span controlled laboratory capture, casual single- or multi-shot setups, and neural-field or Gaussian representations, but the latter require multi-view input and per-object optimization.Recovering lighting under unknown passive illumination requires disentangling shape and materials from illumination.Source: §2 · p. 2
  • Diffusion-based material methods include intrinsic-image and PBR estimation, texture generation, and 3D reconstruction, while several approaches estimate only selected material or lighting maps.IID [59] fine-tunes Stable Diffusion on an interior dataset for PBR prediction and relighting; RGB↔X predicts selected maps conditioned on an image and text prompt.Source: §2 · pp. 2–3
  • Single-image 3D generation uses image or video diffusion with SDS or photometric losses, but prior methods may lack explicit camera control or require extra inputs or workflow steps for relightable PBR materials.SViM3D instead predicts RGB, PBR parameters, and normals simultaneously for explicit shape and illumination reconstruction from one image.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe video diffusion as generating an orbital sequence from a reference image and camera poses, and explain how PBR properties and surface normals determine outgoing radiance. SViM3D extends this setup to jointly generate those quantities for each view.

3. Preliminaries

  • A video diffusion model takes an object image and a sequence of camera poses, then generates an orbital video whose views can feed a multi-view 3D reconstruction pipeline.The model iteratively generates the K-frame sequence during inference [43] [91] [97] [98].Source: §3 · p. 3
  • Outgoing radiance integrates incoming illumination over the hemisphere, weighted by the BRDF and cosine shading; the Cook-Torrance model represents the specular BRDF with NDF, Fresnel, and geometric attenuation terms.The Disney BRDF parameterization uses basecolor and metallic to characterize Fresnel, and roughness to characterize the NDF and geometric attenuation [16, 78, 95].Source: §3 · p. 3
  • SViM3D adapts a 3D video diffusion framework to jointly generate each target view’s image, PBR materials, and surface normal.The material vector comprises basecolor, roughness, and metallic, alongside the normal [98].Source: §3 · p. 3

4. SViM3D: Multi-view PBR Generation

SViM3D takes an image and camera orbit to generate multi-view RGB, material, and normal outputs, then uses them for relighting and 3D reconstruction. Its design adapts video diffusion and adds reconstruction procedures for sparse views and imperfect generated views.

4. SViM3D: Multi-view PBR Generation

  • Given an image and a 21-view camera trajectory, SViM3D generates an augmented orbital video containing RGB, PBR parameters, and camera-space normals for each view.The outputs serve as pseudo-ground truth for a reconstruction pipeline that produces textured meshes with PBR materials.Source: §4 · pp. 3–4, Figure 3 · p. 4

4.1. SViM3D Architecture and Training

  • The architecture adapts SV3D by encoding material properties and normals as images with its pretrained VAE, extending the UNet’s latent channels, and training on multi-illumination, multi-view renders.Training proceeds in two phases: temporal attention is initially frozen, followed by fine-tuning the whole UNet on higher-quality PBR data.Source: §4.1 · pp. 4–5

4.2. Relighting

  • SViM3D relights predicted materials in image space using generated normals, an HDRI, and split-sum environment lighting, supporting interactive-rate relighting and 3D rendering.The method precomputes a roughness-and-cosine lookup and a filtered environment pyramid for lighting evaluation.Source: §4.2 · p. 5

4.3. 3D Reconstruction using SViM3D Outputs

  • The reconstruction pipeline uses SViM3D outputs as pseudo-ground truth, optimizing illumination, an implicit representation, and then a DMTet mesh before extracting and texturing the final asset.A quick-preview path can omit illumination pre-optimization and potentially later reconstruction phases.Source: §4.3 · p. 5
  • View-dependent masking emphasizes less-distorted regions, while learnable per-view homographies compensate for residual inconsistencies in generated views during reconstruction.The masking uses surface-normal alignment with the view direction as a trust proxy; the homographies are optimized during the latter part of Phase 1.Source: §4.3 · p. 6

5. Experiments and Results

SViM3D is evaluated on object-centric material prediction, multi-view synthesis, relighting, and 3D reconstruction, with results reported on Poly Haven and Stanford Orb. The reported comparisons favor SViM3D across most multi-view metrics and show benefits for reconstruction and relighting.

Datasets and ablation study

  • The evaluation introduces a 315-asset Poly Haven dataset and also tests relighting and 3D reconstruction on Stanford Orb.The ablation study reports that every ablated component contributes significantly to final reconstruction quality; Table 3 covers a Poly Haven subset.Source: §5 · pp. 6, 8, Table 3 · p. 7

Single image material prediction

  • Single-image material prediction results comfortably outperform the compared material-estimation baselines on the reference frame.The reference frame is included in the camera trajectory; the supplied passages do not provide the table’s metric values.Source: §5 · pp. 6–7

Multi-view novel view image and albedo synthesis (NVS)

  • SViM3D achieves state-of-the-art performance on almost all multi-view RGB and albedo metrics, and its PBR-based RGB prediction improves over the baseline.Table 2 evaluates RGB radiance and albedo generation on Poly Haven; results are averaged over three samples across all 21 frames.Source: §5 · p. 7, Table 2 · p. 7

3D Relighting and Novel View Synthesis

  • On Stanford Orb, reconstructed materials support rendering under original and novel illumination and reproduce the original illumination better than SF3D.The authors report that SViM3D’s roughness and metallic results are overall closer to ground truth than the compared methods.Source: §5 · pp. 7–8, Table 4 · p. 7

Runtime

  • SViM3D generates 21 views at 576×576 in 20 seconds, while generating full sequences with other methods takes minutes.3D reconstruction takes 15 minutes; the reported three-minute overhead versus SV3D is attributed to added PBR optimization.Source: §5 · p. 8

6. Conclusion

SViM3D jointly predicts multi-view RGB, spatially varying PBR materials, and normals from a conditioning image and user-defined camera path. Its outputs are used as pseudo-ground truth for 3D reconstruction, and the paper reports state-of-the-art performance across several view and material synthesis tasks.

Conclusion

  • SViM3D jointly predicts multi-view consistent RGB, spatially varying PBR parameters, and surface normals from a conditioning image and user-defined camera path.The paper adapts a video diffusion model and introduces architecture and training-data modifications for simultaneous material prediction.Source: §6 · p. 8
  • Using its outputs as pseudo-ground truth enables high-quality 3D reconstruction in this ill-posed setting, alongside state-of-the-art performance on several view and material synthesis tasks.The paper also introduces reconstruction innovations to correct multi-view inconsistencies and fast differentiable environment-based lighting.Source: §6 · p. 8

Overview

The supplementary overview covers video-diffusion denoising and coordinate-based NeRFs. The supplied passages describe how image and camera conditioning enter denoising and how NeRF represents and renders a camera ray.

A.1. Video Diffusion Denoising

  • The conditioning image is concatenated with the noisy latent, while its CLIP embedding is supplied to transformer cross-attention as key and value.The passage describes the conditioning pathway in the video-diffusion denoising process.Source: §A.1 · p. 14
  • Camera angles and noise timestep are sinusoidally embedded; transformed camera-pose embeddings are added to timestep embeddings and then to residual-block features.The feature dimension is matched through a linear layer, following SV3D.Source: §A.1 · p. 14

A.2. Coordinate-based MLPs and NeRF

  • A NeRF maps 3D location and view direction to view-dependent color and density, then approximates the ray’s final color by numerical quadrature.The ray is parameterized from its origin along its view direction, between near and far bounds.Source: §A.2 · p. 14

B.1. UNet training details

The supplementary implementation details describe diffusion training, geometric regularization, view masking, and homography correction. They also point to further evaluation and qualitative reconstruction comparisons.

B.1. UNet training details

  • Training precomputes latents and CLIP embeddings, uses the EDM diffusion loss, and fits a batch of two across 21 frames with Flash Attention v2.Triangular classifier-free guidance scales guidance according to distance from the reference view to reduce over-sharpening.Source: §B.1 · p. 14

B.2. Geometry regularization.

  • Geometry optimization supervises predicted normals, adds a coordinate-conditioned bump field, and applies depth and bilateral-smoothness regularization across two phases.LPIPS weight increases through Phase 1; homography correction begins after a 400-step warmup, while Phase 2 reduces LPIPS and increases smoothness regularization.Source: §B.2 · p. 14

B.3. View dependent masking

  • Masks are normalized by the maximum across views, then smoothstep and gamma correction steer contrast while clipping values to 0–1.Source: §B.3 · p. 14

B.4. Homography correction

  • To handle warped outlier views, Phase 2 masks views based on albedo-loss differences; consistently masked homographies are reinitialized and refined.The stated outliers include homogeneous image regions and complex edge features.Source: §B.4 · pp. 14–15

C. Further results

  • Further results include evaluation on additional datasets and qualitative comparisons related to the reconstruction pipeline.Source: C. Further results · p. 15

C.1. Overview of baseline methods

The section situates SViM3D among single-image material-generation baselines and describes the distinctive scope of its multi-view RGB and material synthesis. It contrasts spatially varying material parameters with the limitations of existing approaches and illustrates generated views under novel lighting.

C.1. Overview of baseline methods

  • IID predicts albedo, roughness, and metallic parameters for one frame, while MaterialFusion uses an object-centric 2D material diffusion prior and SDS-based 3D optimization.RGB↔X can generate selected PBR data, including albedo, roughness, metallic, or diffuse irradiance maps, from an image and text prompt.Source: §C.1 · p. 15
  • The baseline overview also includes a multi-view consistency evaluation on Poly Haven and a feature comparison of relighting methods.The supplied table captions establish evaluation scope, not comparative scores.Source: Table 5 · p. 16, Table 7 · p. 16
  • Figure 12 illustrates five generated orbit views with basecolor, roughness, metallic, and normal maps, alongside results under two new illumination settings.The examples use objects from the Poly Haven test dataset.Source: Figure 12 · p. 16
  • SViM3D is the only compared method offering RGB view synthesis and material synthesis together as a multi-view task.Table 6 summarizes features of the compared methods.Source: §C.1 · p. 16, Table 6 · p. 16
  • Spatially varying roughness and metallic values let SViM3D represent real-world objects more realistically than SF3D’s limited material model.The paper reports shading results closer to ground truth with its spatially varying parametrization.Source: Figure 10 · p. 15, §C.1 · p. 16

C.3. Quantitative evaluation across views

Across generated views, SViM3D achieves the best reported results on the section’s view-consistency comparisons. The paper also describes a single-image reconstruction pipeline that lifts generated views and their material predictions into a mesh.

C.3. Quantitative evaluation across views

  • SViM3D consistently yields the best SSIM results across evaluated views, and MeT3R also reflects improved multi-view consistency over SV3D baselines.Side views are the most challenging generations; the authors suggest extreme angles, grazing angles, and object boundaries may contribute to the difficulty.Source: §C.3 · p. 16

C.4. 3D reconstruction

  • The reconstruction pipeline lifts generated views, material parameters, and normals first to a NeRF and then to a polygon mesh, with material parameters well preserved through pseudo-GT supervision.The resulting mesh can be rendered under novel illumination; Figure 13 shows additional results sourced from Poly Haven and GSO.Source: §C.4 · p. 17, Figure 13 · p. 17

C.5. Multiple samples

Repeated denoising samples remain physically plausible while varying, with greater deviations as the camera moves farther from the conditioning view. Geometry evaluation notes occasional alignment failures alongside visually close reconstructions when optimization succeeds.

C.5. Multiple samples

  • Repeated denoising samples vary while remaining physically plausible, and their deviations increase as the camera moves farther from the conditioning frame.The comparison uses the same conditioning image for three denoising samples.Source: §C.5 · p. 17

C.6. 3D Geometry

  • On 80 real-world GSO objects, SViM3D can score slightly lower than SF3D because rare view-alignment failures occur, though successful reconstructions look close and sometimes retain finer detail with extra noise.The evaluation compares Chamfer distance and IoU against scanned ground-truth point clouds.Source: §C.6 · p. 17, Table 8 · p. 17
  • Using SV3D without PBR material prediction yields lower-quality RGB view synthesis than the reported material-aware approach.Table 9 describes a separate multi-view NVS evaluation on 100 BlenderVault objects, including RGB and albedo/basecolor generation.Source: §C.6 · p. 17, Table 9 · p. 17

C.7. Relighting

SViM3D’s generated material channels and normals support environment-based relighting, material edits, and integration of reconstructed assets into new scenes. The section compares relighting methods and illustrates the approach across generated and real-world examples.

C.7. Relighting

  • Generated material channels and normals provide dynamic direct illumination at real-time speed, while full 3D reconstruction supports indirect illumination.The 2.5D pipeline also supports material editing and relighting views along the camera trajectory with an environment map.Source: §C.7 · pp. 17–18, Figure 20 · p. 20
  • Examples include object-centric GSO items, generated-image inputs such as an out-of-distribution wizard raccoon, and casual smartphone captures of a shaker and strawberry.The paper describes plausible, view-consistent results for generated-image inputs and reports generalization to GSO when scenes are object-centric.Source: Figure 16 · p. 18, Figure 17 · p. 19, Figure 22 · p. 20
  • In the relighting comparison, SViM3D is the only model capable of joint novel-view synthesis and relighting, with better multi-view consistency and fewer artifacts than the compared methods.The comparison includes IC-Light, Neural-Gaffer, and DiLightNet against synthetic ground truth; IC-Light generally produces high-contrast output that is difficult to edit in real-world use cases.Source: §C.7 · p. 18, Figure 19 · p. 20
  • Reconstructed models with PBR materials can be integrated into new environments and rendered with global illumination using a path tracer.Figure 15 shows an example reconstruction pipeline from a single image through view generation and optimization to a mesh.Source: §C.7 · p. 18, Figure 15 · p. 18
  • The examples show roughness edits spanning almost zero to nearly one, with the original material value close to the second-from-left version.The output’s explicit material parameters are edited and visualized using the rendering framework.Source: Figure 18 · p. 19
  • On pathological glossy-plastic versus metallic cases, over 90% of predictions have low roughness and near-zero metalness; the authors attribute the pattern to dataset bias.The higher-value predictions often correspond to objects that would usually be metal.Source: §C.7 · p. 19, Figure 21 · p. 20
Loading 2510.08271v2…