Source-linked AI summary
Texture Space Material Diffusion
Jacob Munkberg, Peter Kocsis, Jon Hasselgren
TL;DR
Detailed material authoring is labor-intensive, while sparse 3D methods can require substantial memory and compute. This paper finetunes a video diffusion transformer to generate materials in texture space, reporting quality on par with or above the state of the art for material reconstruction and generation.
Problem
Detailed material authoring for complex 3D assets is labor-intensive, while sparse 3D approaches require substantial memory and compute and do not leverage large video-model priors.
Source: §1 · pp. 1–2Method
The method finetunes a diffusion transformer to generate complete PBR materials in texture space from posed views, text, or low-resolution maps, using world-space information for conditioning.
Source: §1 · p. 2Results
Takeaways & Limitations
The texture-space formulation avoids common view-consistency and data-sparsity problems and is described as simple and scalable.
Source: §6 · p. 9Inference is costly, particularly for the multi-view model, which takes 130 s for 17 views at 2K on a GB300 GPU; overlapping texture-atlas segments within patches can also be indistinguishable.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.
1 INTRODUCTION
Detailed material authoring for 3D objects is labor-intensive, while image- and object-space approaches face view-consistency or computational and architectural trade-offs. The paper generates complete materials in texture space, using pretrained image and video priors for robust reconstruction and scalable generation.
1 INTRODUCTION
- Authoring detailed materials on complex 3D assets is labor-intensive, despite texture maps being a standard way to store spatially varying material parameters.The paper focuses on generating realistic materials from text descriptions and/or photographs.Source: §1 · p. 1
- Image-space methods can predict the same surface inconsistently across views, while sparse 3D methods require substantial memory and compute and specialized architectures.The latter are trained from scratch on synthetic data rather than leveraging large video foundation model priors.Source: §1 · pp. 1–2
- The method formulates generation in a shared texture space, projecting views into a UV atlas to produce physically based material parameters.Its target setting is material estimation for a known object under uncontrolled lighting.Source: §1 · p. 2, Figure 2 · p. 2
- The framework generates complete PBR materials in a view-consistent 2D representation while reusing pretrained image and video priors.The paper presents this as the first diffusion framework to do so entirely in texture space.Source: §1 · p. 2
- Full texture-space generation enables robust material reconstruction under unknown lighting from single or multiple views.The method also targets robustness to inconsistencies in input views and geometry.Source: §1 · p. 2
- Training-free noise rolling and coverage-aware expert aggregation scale inference to 8K textures and over 100 input views.The upscaler and inference-time augmentation extend beyond the resolution and view counts used in training.Source: §1 · p. 2
2 RELATED WORK
Related approaches generate views or assets in image or 3D space, reconstruct materials through inverse rendering, or adapt diffusion models for material generation. This work instead formulates complete PBR material generation in texture space.
Image and Video Diffusion
- Diffusion transformers offer flexible finetuning, and this work builds on Wan 2.1 to process multi-view observations and produce multiple modalities.Figure 3 describes inputs including projected views, text prompts, and low-resolution material maps, with world positions and normals used as conditioning.Source: §2 · p. 2, Figure 3 · p. 3
Diffusion-based 3D Asset Generation
- Image-space asset methods can suffer from view inconsistency, while multi-view and video approaches improve consistency but have limited resolution and do not provide PBR materials.Image-space reconstruction often relies on optimization through differentiable rendering.Source: §2 · p. 3
- Object-space asset methods use volumetric primitives to generate shape and PBR materials, whereas this paper focuses on PBR materials rather than complete assets.Sparse 3D methods avoid view-consistency issues but can be resource-intensive and require specialized architectures.Source: §2 · p. 3
Material Reconstruction
- Material reconstruction methods range from differentiable rasterization and path tracing to NeRFs and Gaussian splatting, with trade-offs in optimization, noise, and disentangling shape, materials, and lighting.Diffusion-based intrinsic decomposition methods may also be combined with inverse-rendering optimization.Source: §2 · p. 3
Diffusion-based Material Generation
- Prior material-generation methods use retrieval or interpolation over material databases, hybrid image-diffusion refinement, or image-space models conditioned on geometry or lighting.Image-space methods can lead to texture seams and limited resolution; neural-material diffusion approaches also operate in image space.Source: §2 · pp. 3–4
- Texture-space generation extends the possibility demonstrated by TEXGen to complete PBR materials, rather than diffuse textures alone.TEXGen interleaves texture-space convolutions with point-cloud attention to outpaint diffuse textures.Source: §2 · p. 4
3 METHOD
The method predicts PBR material textures in texture space from posed views, text, or low-resolution maps, using known geometry and camera parameters. Its architecture supports multiple conditioning modes and inference-time scaling to higher resolutions and more input views.
3 METHOD
- Known geometry and camera parameters let the method project posed views into texture space, where a diffusion model predicts base-color and combined height, roughness, and metallicity maps.The method also accepts text prompts, single or multiple posed views, or low-resolution material maps.Source: §3 · p. 4
3.1 ARCHITECTURE
- The multi-view model merges sparse shaded texture-space observations, demodulates lighting, and fills uncovered regions; training adds noise and blur to improve robustness to imperfect views.Its input uses 17 texture-space views; random Gaussian noise and blur are applied during training.Source: §3.1 · p. 5
- For single-view generation, 3D-aware rotary positional embeddings combine frame IDs with world positions to improve attention between spatially adjacent surface regions distant in texture space.Text generation replaces the view input with a binary UV coverage mask and uses the same positional embedding.Source: §3.1 · p. 5
- A refinement model upscales full-coverage material maps, while neural material textures can replace PBR latents to represent complex appearances such as fuzzy materials.Upscaler training uses guides generated by downscaling and then upscaling references at factors of 2, 4, 8, or 16.Source: §3.1 · p. 5
3.2 TRAINING DATASET
- The training dataset contains 120k object-rendering videos from TexVerse, with 17 frames per object at 2048×2048 and path-traced rendering.Lighting uses one randomly selected probe per object from 697 Poly Haven probes; the dataset also stores texture-space positions, normals, and material maps.Source: §3.2 · pp. 5–6
3.3 INFERENCE SCALING
- Inference-time methods scale beyond training resolution and view counts: progressive noise rolling supports 8K crops, while overlapping view batches are fused using coverage-weighted top-2 predictions.Random spatial rolls offset crops between denoising steps to hide seams; top-2 fusion avoids the oversmoothing of averaging all experts.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments assess multi-view reconstruction, robustness to imperfect views, and generative material quality. Results show strong synthetic reconstruction and relighting performance, robust reconstruction from generated views, and semantically meaningful single-view generations.
4.1 MULTI-VIEW TO MATERIALS
- Our reference-geometry model leads all synthetic metrics, while DiffPT slightly leads real captures under original lighting; in relighting, DiffPT fails to disentangle illumination.With LSRM geometry, our method improves slightly over LSRM across metrics in the multi-illumination setting.Source: Table 1 · p. 7, §4.1 · p. 7
4 EXPERIMENTS
- Evaluation covers 32 synthetic models and eight DTC real captures; CRF fitting adjusts baseline tonemapping bias amid ambiguous material-lighting estimates.Synthetic evaluation includes eight input views and eight novel lighting conditions; real examples use the original lighting.Source: §4 · p. 6, §4.1 · p. 7
4.1 MULTI-VIEW TO MATERIALS
- Our method plausibly reconstructs materials from inconsistent views synthesized by a depth-conditioned video model, on a 17-object test set.The video model struggled to generate plausible orbits for rotationally symmetrical objects.Source: §4.1 · p. 7, Figure 5 · p. 8
- Inference augmentations scale material maps to 8K resolution while using more input views.Figure 8 illustrates an example of the augmented inference results.Source: §4.1 · p. 7
4.2 GENERATIVE MATERIALS
- Single-view generations are semantically meaningful across tested objects and lighting conditions, and the evaluation compares against TEXGen for diffuse texture generation.The text-guided evaluation compares against VideoMat and VideoMatGen; the single-view material comparison includes Trellis.2, Hunyuan3D 2.1, and VideoMatGen.Source: §4.2 · pp. 7–8
5 LIMITATIONS AND FUTURE WORK
The method has costly inference, challenges capturing specular appearance, and a patch-granularity ambiguity when atlas segments overlap. The authors identify higher resolution and careful data curation as possible ways to address some of these constraints.
5 LIMITATIONS AND FUTURE WORK
- Multi-view inference takes 130 s for the 17-view model at 2K on a GB300 GPU, and the implementation is not optimized.Video-model acceleration and distillation are suggested as possible efficiency improvements.Source: §5 · p. 8
- Specular appearance remains challenging despite improvement over previous work; careful data curation may help.Source: §5 · p. 9
- At Wan-2.1's 16 × 16 patch granularity, overlapping atlas segments cannot be distinguished; higher texture resolution mitigates the issue.Patch-aware unwrapping and pixel-space diffusion are proposed as future directions.Source: §5 · p. 9
6 CONCLUSION
Texture-space diffusion models are presented as tools for material reconstruction and generation, with quality on par with or above the state of the art.
6 CONCLUSION
- Texture-space diffusion achieves material reconstruction and generation quality on par with or above the state of the art, while avoiding common view-consistency and data-sparsity problems.Source: §6 · p. 9
A IMPLEMENTATION DETAILS
The appendix details the fitting, scaling, model-input, positional-encoding, and height-map procedures used in the method. It also describes how CRF adjustment addresses color shifts under unknown lighting.
A.1 CRF ADJUSTMENT
- CRF and exposure are alternately fitted against tonemapped renders using an L1 loss, with a monotonicity regularizer; the adjustment helps address lighting-related color shifts.The appendix reports 25 exposure steps, 200 CRF steps per cycle, 20 cycles, and regularizer weight 0.1; without CRF adjustment, some metrics are heavily biased by decomposition ambiguity.Source: §A.1 · p. 16, Figure S1 · p. 17, Table S1 · p. 17
A.2 INFERENCE SCALING
- Progressive noise rolling scales generations to 2048 × 2048 inputs after a 4× upscaler trained on 512 × 512 inputs and 2048 × 2048 outputs.The procedure generates at native 2048 × 2048 resolution, upsamples by nearest neighbor, then denoises with 50 steps at timestep 0.2; runtime and peak VRAM are reported in Table S2.Source: §A.2 · p. 17, Table S2 · p. 17
A.3 DIFFUSION TRANSFORMER MODEL
- The diffusion-transformer input combines texture-space views with world positions and normals, while the target latent encodes base color and height, roughness, and metallicity.Positions and normals are quantized to 8 bits per channel and normalized before encoding; the two target material images are encoded separately and concatenated along the temporal dimension.Source: §A.3 · pp. 17–18
A.4 3D-AWARE ROPE
- The 3D-aware RoPE replaces pixel coordinates with world-space position components while retaining frame identity, and is used to provide geometry guidance for text- and single-view material generation.World positions are downsampled 16× to match spatial tokens; the frequency allocation is (44,28,28,28) for (fid, px, py, pz), and standard and 3D-aware RoPE are blended over the first 2,000 training steps.Source: §A.4 · p. 18
A.5 MATERIAL HEIGHT MAP
- Height maps support bump-mapped surface detail and are converted to normals using neighboring height samples; converting normal maps back to height requires an approximate iterative Poisson solver.Exact reconstruction may not exist because normal maps are not necessarily integrable; the appendix uses 50 iterations and a height scale of α = 1 during evaluation.Source: §A.5 · pp. 18–19
B EVALUATION PROTOCOL
The evaluation protocol uses image-quality metrics for reconstruction and distribution-based metrics for generation, with task-specific rendering settings. It also documents differences from the Stanford-ORB protocol and checks LSRM scores against that pipeline.
B EVALUATION PROTOCOL
- Reconstruction is evaluated with PSNR, SSIM, and LPIPS, while text-to-material and single-view generation use CLIP-FID and CMMD.Metrics are computed on rendered images with a black background; reconstruction renders use 1024 × 1024 resolution, while distribution metrics use 512 × 512 renders.Source: B EVALUATION PROTOCOL · p. 19
- The black background makes scores dependent on scene scale, so views are selected to keep objects large while fitting them onscreen.Synthetic evaluations maximize the object bounding sphere within the frame; DTC evaluations use representative views of a revolution.Source: B EVALUATION PROTOCOL · p. 19
- The protocol differs from Stanford-ORB in resolution, mask handling, PSNR variants, normalization, and evaluation views; LSRM scores were comparable when checked with that pipeline.Stanford-ORB uses 512 × 512 resolution, erodes the mask by 5 × 5 pixels, and reports PSNR-L and PSNR-H with mean-color normalization.Source: B EVALUATION PROTOCOL · p. 19
C.1 MULTI-VIEW TO MATERIALS - EVALUATIONS ON MATERIAL VOMPONENTS
The section evaluates material components, reports reconstructions from casually captured views, and compares single-view generation against TEXGen. The TEXGen comparison favors this method across the reported metrics and in material coherence.
C.1 MULTI-VIEW TO MATERIALS - EVALUATIONS ON MATERIAL VOMPONENTS
- Material-component metrics use g-buffer renderings over 17 views of each of 32 scenes, with scale-invariant siPSNR additionally reported for base color.The scale adjustment normalizes each predicted image by the ratio of reference to predicted mean color.Source: §C.1 · p. 20
C.2 CASUAL CAPTURES
- The casual-capture examples use 17 randomly sampled frames from handheld video, paired with high-quality DTC geometry but less accurate camera poses.The sampling is temporally stratified; the examples are shown in Figure S3.Source: §C.2 · p. 20, Figure S3 · p. 21
C.3 COMPARISON AGAINST TEXGEN (ON DIFFUSE TEXTURES)
- Figure S4 shows single-view-guided base-color generations that are more consistent than TEXGen and demodulate lighting from the conditional view.The figure caption also notes that the method outputs full PBR materials, which are not shown.Source: Figure S4 · p. 21
- Our method produces more coherent materials and scores higher than TEXGen on all metrics across 2,048 relit views.The comparison uses single-image-to-material generation with diffuse-only renderings, although this method predicts full PBR materials from shaded views while TEXGen predicts albedo from demodulated views.Source: §C.3 · pp. 20–21
C.4 DTC MATERIAL RECONSTRUCTION RESULTS
On real-world DTC examples, the method produces sharper, more consistent texture details despite slightly lower view-reconstruction scores than differentiable rendering. The upscaler performs on par with prior work overall, while attention visualizations show stronger focus on nearby world-space locations.
C.4 DTC MATERIAL RECONSTRUCTION RESULTS
- Our method scores slightly lower than DiffPT on view reconstruction but produces sharper, more consistent texture details on DTC examples.Figure S5 shows comparisons using known geometry for DiffPT and this method, while LSRM jointly reconstructs geometry and materials.Source: §C.4 · p. 21, Figure S5 · p. 22
C.5 UPSCALER
- The texture upscaler performs on par with prior work overall, while PBR-SR performs better on roughness and metallicity parameters.The passage attributes PBR-SR's advantage to its expensive multi-view differentiable-rendering optimization; Table S4 compares 4× upscaling on 32 synthetic objects.Source: §C.5 · p. 22, Table S4 · p. 22
- The 3D-aware RoPE with G-buffer guidance has stronger attention activations at similar 3D locations than the compared Wan 2.1 variants.The visualizations compare standard RoPE, RoPE with G-buffer guidance, and the proposed combination; another visualization reports stronger attention to points closer in world space across layers.Source: Figure S6 · p. 22, Figure S7 · p. 23
C.6 IMPACT OF 3D-AWARE ROPE
The 3D-aware RoPE improves text-to-material results and focuses attention on regions close in world space, even when they are distant in texture space. Across random seeds, it produces more consistent materials and better captures global object structure.
C.6 IMPACT OF 3D-AWARE ROPE
- The 3D-aware RoPE shows a small but consistent quantitative improvement over standard Wan 2.1 RoPE variants on synthetic text-to-material generation.The ablation uses 512 × 512 textures, rather than the main experiments’ 2048 × 2048 resolution.Source: §C.6 · p. 23
- Its attention favors regions close in world space despite texture-space separation; Wan 2.1 RoPE+Gbuf shows this pattern only in middle layers.The 3D-aware RoPE combines world-space position and frame ID; the Gbuf variant supplies world positions and normals as VAE-encoded guide images.Source: §C.6 · p. 23
- Across four seeds, standard Wan 2.1 RoPE has higher variance and struggles to separate geometric components, while 3D-aware RoPE better captures global structure and material consistency.The examples show base color, roughness, and metallicity maps.Source: §C.6 · p. 23, Figure S8 · p. 24
- Text-only generations remain semantically meaningful in disjoint texture space, and texture discontinuities that align with material borders can sometimes benefit the model over image-space alternatives.Source: §C.6 · p. 24