PaperlayerView exact revision on arXiv

Source-linked AI summary

RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesis

Yifei Feng, Mingxin Yang, Shuhui Yang, Sheng Zhang, Jiaao Yu, Zibo Zhao, Yuhong Liu, Jie Jiang, Chunchao Guo

arXiv:2503.19011v1cs.CV

TL;DR

High-quality texture synthesis is challenging because existing methods face multi-view inconsistency and difficulty balancing image and geometry guidance. RomanTex combines geometry-aware multi-view diffusion with decoupled attention and geometry-related guidance. The authors report improved overall texture quality over existing methods and consistent, plausible results in challenging regions.

  • Problem

    High-quality texture generation remains challenging, with existing methods facing global and multi-view inconsistencies and conflicts between image and geometry guidance.

    Source: §1 · p. 2

  • Method

    RomanTex combines 3D-aware positional embedding, decoupled multi-attention, and geometry-related classifier-free guidance in a geometry-aware multi-view diffusion framework.

    Source: §1 · pp. 2–3

  • Results

    RomanTex shows significant overall texture-quality improvements over existing text-to-texture and image-to-texture methods, with coherent results in geometrically challenging regions.

    Source: §4.2 · pp. 6–7

  • Takeaways & Limitations

    The authors report that RomanTex synthesizes high-fidelity, high-diversity, seamless textures while maintaining cross-view consistency and image-following behavior.

    Source: §1 · p. 3

Abstract

from arXiv · show

Painting textures for existing geometries is a critical yet labor-intensive process in 3D asset generation. Recent advancements in text-to-image (T2I) models have led to significant progress in texture generation. Most existing research approaches this task by first generating images in 2D spaces using image diffusion models, followed by a texture baking process to achieve UV texture. However, these methods often struggle to produce high-quality textures due to inconsistencies among the generated multi-view images, resulting in seams and ghosting artifacts. In contrast, 3D-based texture synthesis methods aim to address these inconsistencies, but they often neglect 2D diffusion model priors, making them challenging to apply to real-world objects To overcome these limitations, we propose RomanTex, a multiview-based texture generation framework that integrates a multi-attention network with an underlying 3D representation, facilitated by our novel 3D-aware Rotary Positional Embedding. Additionally, we incorporate a decoupling characteristic in the multi-attention block to enhance the model's robustness in image-to-texture task, enabling semantically-correct back-view synthesis. Furthermore, we introduce a geometry-related Classifier-Free Guidance (CFG) mechanism to further improve the alignment with both geometries and images. Quantitative and qualitative evaluations, along with comprehensive user studies, demonstrate that our method achieves state-of-the-art results in texture quality and consistency.

1. Introduction

Texture synthesis supplies realistic material properties and artistic expression to 3D assets, but high-quality generation remains difficult. RomanTex addresses consistency and alignment challenges with geometry-aware multi-view attention, decoupled attention, and geometry-related guidance.

1. Introduction

  • Texture detail contributes realistic material properties and artistic expression to digital assets.The introduction frames surface detail as a core value of texture generation.Source: §1 · p. 2
  • High-quality texture generation remains challenging amid geometric complexity and scarce 3D-texture data.Mainstream methods draw on image-generation models while combining geometric conditions with text-to-image models.Source: §1 · p. 2
  • Pre-trained text-to-image methods can produce global style and color inconsistencies, including semantically incorrect backs, because they lack viewpoint awareness and 3D global perception.Multi-view methods can improve global semantic consistency, but cross-view attention alone does not guarantee consistent overlapping regions and can trade off generalization against view consistency.Source: §1 · p. 2
  • Image and geometry guidance can conflict when reference images and meshes are misaligned, risking weak image fidelity or textures that bleed across geometric edges.These competing constraints can make texture synthesis ill-posed.Source: §1 · p. 2
  • RomanTex combines 3D-aware positional embedding, decoupled multi-attention, and geometry-related classifier-free guidance to target consistency, image following, diversity, and geometry alignment.The authors describe the framework as producing high-fidelity, high-diversity, seamless textures for 3D assets.Source: §1 · pp. 2–3

2. Related Work

Prior texture-generation work draws on 2D diffusion priors, UV-space methods, and multi-view diffusion, but each direction has limitations. RomanTex adopts multi-view diffusion while integrating image priors with explicit geometry.

2.1. Diffusion-based Texture Generation.

  • Image-space diffusion methods distill 2D priors or generate multi-view images for texture baking, but inpainting can leave seams and synchronized denoising can yield plain textures.Earlier score-distillation methods can also produce oversaturated colors and require substantial time.Source: §2.1 · p. 3
  • Image-guided methods can lose image details, while UV-space and 3D-representation approaches may discard 2D priors or use representations incompatible with 3D engines.RomanTex adopts a reference branch and adds disentanglement and 3D-aware positional embedding to align reference images and geometry.Source: §2.1 · p. 3

2.2. Multi-view Diffusion.

  • Multi-view diffusion communicates across image latents to encourage cross-view coherence without sacrificing generative diversity.Methods vary from concatenating views into a large image to injecting view constraints through attention masks; RomanTex follows this general framework using object-centric rendered 3D data.Source: §2.2 · p. 3

3. Methodology

RomanTex extends image diffusion into a geometry-aware, image-guided multi-view framework. Its components encode 3D position in attention, decouple reference and multi-view branches, and balance image and geometry guidance.

3. Methodology

  • The framework extends Stable Diffusion with reference and multi-view attention and concatenates canonical-coordinate and normal maps with denoised latents as geometry conditions.The three proposed components are 3D-aware rotary positional embedding, a decoupled multi-attention module, and geometry-aligned inference.Source: §3 · pp. 3–4

3.1. Preliminary

  • Positional embedding supplies spatial prior information because full attention lacks inherent positional awareness, and RomanTex incorporates 3D geometry through 3D-aware RoPE.The method presents RoPE as a way to model positional relationships in attention.Source: §3.1 · p. 4

3.2. 3D-aware Rotary Positional Embedding

  • 3D-aware RoPE uses geometric correspondences between image features and voxels to provide attention with geometry cues and improve cross-view consistency.Canonical coordinate maps are downsampled to feature resolutions; voxel-index-based positional information links 2D feature grids with their 3D counterparts, with higher attention for nearby voxels and maximal similarity for identical 3D regions.Source: §3.2 · pp. 4–5

3.3. Decoupled Multi-attention Module

  • The attention structure combines frozen self-attention, 3D-aware multi-view attention, and reference attention, but functional entanglement between the latter branches can destabilize novel-view generation and increase CFG sensitivity.RomanTex drops out the multi-view attention branch during training, alongside the reference-image dropout strategy, to separate their functions.Source: §3.3 · p. 5

3.4. Geometry-related Classifier-Free Guidance

  • Geometry-related CFG guides generation using canonical-coordinate and normal-map conditions, while a CFG projection adjusts guidance when geometry and reference images are misaligned.The projection favors geometry details where it has sharp activations and the reference image where geometry is relatively plain.Source: §3.4 · pp. 5–6

3.5. Implementation Details

  • Training uses Stable Diffusion 2 initialized from a Zero-SNR checkpoint, with AdamW, a 5 × 10^-5 learning rate, and about 228 GPU days.The setup uses six training views and 6–12 inference views, and independently drops geometry conditions, reference images, and multi-view attention with probability 0.1 each.Source: §3.5 · p. 6

4. Experiment

The experiments compare RomanTex with texture-synthesis baselines, assess image-to-texture results through a user study, and ablate three core modules. The authors report improved texture quality and module-specific benefits.

4.1. Baselines and Metrics

  • Baselines span text-, image-, and text-and-image-to-texture methods; assessment covers distributional similarity and prompt fidelity.For fair comparison with text-guided methods, RomanTex uses a reference image generated with a depth-based ControlNet; the authors note limited multi-view diffusion baselines.Source: §4.1 · p. 6

4.2. Comparisons

  • RomanTex shows significant overall texture-quality improvements over existing text-to-texture and image-to-texture methods.Tables 1 and 2 report criteria for the respective tasks.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • For text-to-texture, RomanTex produces coherent results in challenging back and self-occluded regions, while baselines show inconsistency, oversmoothing, gradients, or noise.Text2Tex is described as cluttered, SyncMVD as oversmoothed with unintended gradient artifacts, and TexPainter as noise-corrupted.Source: §4.2 · p. 7
  • For image-to-texture, RomanTex is reported to achieve superior synthesis quality and best image fidelity against methods with detail loss, Janus effects, seam artifacts, ghosting, or color fragmentation.The comparison includes SyncMVD-IPA, TexGen, and HY3D-2.0.Source: §4.2 · p. 7

4.2.1. User Study

  • The user study evaluated image following, cross-view consistency, diversity, and overall texture quality using 50 hand-crafted and 100 generated geometries.Three technical artists and 25 non-experts each selected preferred results across 30 randomly assigned sets.Source: Table 3 · p. 7, §4.2.1 · pp. 7–8

4.3. Ablation Study

  • Ablations associate 3D-aware RoPE with multi-view consistency, the decoupled reference branch with meaningful back views, and geometry-related CFG with geometry alignment.The modules were disabled individually to examine their contributions.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Conclusion

RomanTex combines 3D-aware positional encoding, a decoupled reference branch, and geometry-related guidance for consistent texture synthesis. The authors report fewer seams and artifacts, while identifying baked-in illumination as an unresolved challenge.

5. Conclusion

  • RomanTex adapts multi-view image generation with 3D-aware Rotary Positional Embedding, a decoupled reference branch, and geometry-related Classifier-Free Guidance.The branch uses dropout training, and the guidance is applied during inference.Source: §5 · p. 8
  • Comparisons and ablations show fewer seams and artifacts in synthesized textures.Source: §5 · p. 8
  • Generated textures retain baked-in illumination; extending the pipeline to physically based rendering textures is proposed as future work.Source: §5 · p. 8

6. Supplementary on Baselines and Experimental Criteria

The section describes comparison baselines for image-to-texture evaluation and lists the metrics used to assess generated textures.

6. Supplementary on Baselines and Experimental Criteria

  • SyncMVD-IPA adapts SyncMVD for image-to-texture comparison by using SDXL-base and an additional Image IP-adapter.The authors note that most existing work focuses on text-to-texture.Source: §6 · p. 11
  • The comparison includes Text2Tex, SyncMVD, TexPainter, Paint3D, and TexGen, which use differing inpainting, synchronization, refinement, and UV-diffusion approaches.The authors initialize TexGen with a partial texture baked from the original reference image.Source: §6 · p. 11
  • FID_CLIP and CMMD assess distributional distance, while CLIP-I, CLIP-T, and LPIPS assess semantic alignment and consistency with reference images.FID_CLIP compares re-renderings with ground-truth renderings; CLIP-I and CLIP-T compare generated-texture renderings with the given image and text, respectively.Source: §6 · p. 11

7. Supplementary on Ablation Study

The supplementary ablation section uses local alignment distance to quantify multiview coherence and reports that 3D-aware RoPE outperforms naive self-attention without 3D geometry awareness.

7. Supplementary on Ablation Study

  • Local alignment distance (LAD) measures multiview coherence as average MSE over overlapping regions between adjacent views.The evaluated texture and mask are represented in UV space unwrapped from each view's image.Source: §7 · p. 11
  • 3D-aware RoPE significantly outperforms naive self-attention without 3D geometry awareness.The supplied ablation-table caption identifies Table 4 as an ablation study on local alignment distance, but gives no table values.Source: §7 · p. 11, Table 4 · p. 11

8. Supplementary on 3D-aware RoPE

The section introduces a RoPE-inspired positional encoding that rotates hidden-state vectors according to their corresponding voxels.

8. Supplementary on 3D-aware RoPE

  • The embedding injects positional information by rotating hidden-state vectors according to corresponding voxels.The approach is inspired by Rotary Position Embedding in language modeling.Source: §8 · p. 11

Related papers

Loading 2503.19011v1…