Source-linked AI summary
TAPESTRY: From Geometry to Appearance via Consistent Turntable Videos
Yan Zeng, Haoran Jiang, Kaixin Yao, Qixuan Zhang, Longwen Zhang, Lan Xu, Jingyi Yu
TL;DR
TAPESTRY addresses the difficulty of generating consistent appearances for untextured 3D models by conditioning video diffusion on explicit mesh geometry. Its geometry-guided turntable videos support complete texture synthesis and 3DGS reconstruction, with experiments showing improved consistency and reconstruction quality over existing approaches.
Problem
Generating high-quality, 3D-consistent appearances for untextured models remains difficult because existing video models struggle with geometric and appearance stability across views.
Method
TAPESTRY conditions video diffusion on rendered multi-modal geometric features and uses progressive context-aware 3D-aware inpainting for complete surface coverage.
Results
TAPESTRY produces consistent, detailed turntable videos that support seamless texture synthesis and directly drive high-quality 3DGS reconstruction, outperforming existing approaches in consistency and reconstruction quality.
Takeaways & Limitations
The generated turntable videos function as both dynamic previews and reliable 3D-aware intermediates for automated creation of complete textured and neural 3D assets.
Takeaways & Limitations
Performance depends on clean, well-topologized input meshes, and model-generated lighting is not controllable like an environment map, hindering relighting.
Abstract
from arXiv · showhide
Automatically generating photorealistic and self-consistent appearances for untextured 3D models is a critical challenge in digital content creation. The advancement of large-scale video generation models offers a natural approach: directly synthesizing 360-degree turntable videos (TTVs), which can serve not only as high-quality dynamic previews but also as an intermediate representation to drive texture synthesis and neural rendering. However, existing general-purpose video diffusion models struggle to maintain strict geometric consistency and appearance stability across the full range of views, making their outputs ill-suited for high-quality 3D reconstruction. To this end, we introduce TAPESTRY, a framework for generating high-fidelity TTVs conditioned on explicit 3D geometry. We reframe the 3D appearance generation task as a geometry-conditioned video diffusion problem: given a 3D mesh, we first render and encode multi-modal geometric features to constrain the video generation process with pixel-level precision, thereby enabling the creation of high-quality and consistent TTVs. Building upon this, we also design a method for downstream reconstruction tasks from the TTV input, featuring a multi-stage pipeline with 3D-Aware Inpainting. By rotating the model and performing a context-aware secondary generation, this pipeline effectively completes self-occluded regions to achieve full surface coverage. The videos generated by TAPESTRY are not only high-quality dynamic previews but also serve as a reliable, 3D-aware intermediate representation that can be seamlessly back-projected into UV textures or used to supervise neural rendering methods like 3DGS. This enables the automated creation of production-ready, complete 3D assets from untextured meshes. Experimental results demonstrate that our method outperforms existing approaches in both video consistency and final reconstruction quality.
1 Introduction
TAPESTRY addresses the difficulty of generating realistic, geometrically and photometrically consistent appearances for untextured 3D models. It uses geometry-conditioned turntable videos as an intermediate for complete texturing and downstream 3D reconstruction.
- Manual texture authoring is labor-intensive and difficult to scale, motivating automated appearance generation for untextured 3D models.
- Turntable videos can provide both dynamic previews and view information for downstream 3D reconstruction, but require strict geometric and photometric consistency.
- TAPESTRY conditions video diffusion on explicit mesh geometry, reframing appearance generation as precise visual texturing on a fixed geometric scaffold.
- A progressive 3D-aware inpainting pipeline rotates the model and uses generated content as context to complete self-occluded regions.
- The generated videos support complete texture creation and can directly drive 3D Gaussian Splatting reconstruction for interactive 3D visualization.
2 Related Work
Prior approaches use physical or digital turntable production, independent multi-view generation, or video synthesis, but struggle with cost, consistency, or long-range pixel-level stability. TAPESTRY uses dense, consistent turntable videos to support texture back-projection and neural reconstruction.
- Turntable Video Productions: Traditional turntable production requires either labor-intensive photography or costly digital scene setup, material authoring, lighting, and rendering.
- Multi-view and Video Generation: Independent multi-view image generation can produce plausible views, but photometric and geometric coherence across views remains difficult.
- Multi-view and Video Generation: Video models improve temporal coherence, yet existing controllable methods often fail to provide the long-range, pixel-perfect consistency required for full 360-degree turntable videos.
- Texturing from Generative Views: SDS-based reconstruction methods are limited by computational inefficiency and perceptual biases inherited from image models.
- Texturing from Generative Views: Dense turntable views provide continuous surface coverage, supporting texture back-projection with fewer seams and artifacts and training of 3D Gaussian Splatting.
3 Method
TAPESTRY generates geometry-consistent turntable videos from explicit 3D mesh features, then uses those videos in a progressive pipeline for complete texture reconstruction.
- Geometry-guided Video Generation: TAPESTRY reframes appearance generation as geometry-conditioned video synthesis from an untextured 3D mesh.The generated 360-degree turntable video serves as the central intermediate representation.
- Geometry-guided Video Generation: Multi-modal geometric conditioning couples generated video content to 3D structure through pixel-level guidance and latent-space injection.The framework prepares geometric conditions and injects their fused latents into the video diffusion model alongside reference information.
- Geometry-guided Video Generation: Normal and position-map videos jointly constrain local surface appearance and global spatial structure across the circular camera orbit.Normal maps encode surface orientation and local detail, while position maps provide absolute world-coordinate references that help prevent long-range drift.
- High-Fidelity Texturing from Video: The pipeline back-projects video frames into UV space using ray-traced visibility, angle and depth weighting, producing initial textures and confidence maps.The similarity mask combines λ_angle = cos θ^4 with a clipped depth-gradient penalty before weighted accumulation.
- High-Fidelity Texturing from Video: 3D-Aware Inpainting rotates the object to expose untextured regions while preserving the original camera trajectory for context-aware refinement.Repeated generation passes target self-occluded regions without introducing an independent, inconsistent turntable video.
- High-Fidelity Texturing from Video: Multi-stage texture fusion combines refined partial textures using confidence-derived weights, updating confidence maps for subsequent refinement iterations.More reliable viewpoints contribute more strongly to the final seamless texture.
4 Experiments
TAPESTRY is evaluated as both a turntable-video generator and a texture-reconstruction pipeline, with experiments covering quantitative, qualitative, ablation, and runtime comparisons. Across these evaluations, it maintains consistent appearances and produces seamless, high-fidelity textured assets.
- Experimental Setup: TAPESTRY is evaluated on generated turntable videos and final textured 3D assets using quantitative and qualitative comparisons.The experiments include video consistency, texture quality, progressive texturing, and geometric-condition ablations.
- Consistent Video Generation: 79.8% preference for alignment with the initial reference, 80.9% for geometric consistency, and 76.1% for overall video quality in the user study.Qualitative comparisons report that TAPESTRY avoids the appearance drift, Janus artifacts, flicker, and distortion seen in competing methods.
- Texture Generation: TAPESTRY achieves substantially better FID, KID(×104), and CLIP score than the compared texture-generation methods.The evaluation uses multi-view renderings, and runtime is slower than baseline texture-generation methods because the system produces both textures and geometry-aligned TTVs.
- Texture Generation: TAPESTRY generates consistent, seamless, and sharp textures across challenging self-occluded and high-frequency regions.Compared methods exhibit seams, color discontinuities, ghosting, or blurring in complex regions.
- Ablation Studies: Combining normal and position conditions provides both global geometric consistency and high-quality local detail, outperforming either modality alone across the ablation metrics.Position-only variants preserve global structure but weaken details, while normal-only variants can drift on complex geometries.
- Progressive Texturing: Context-aware progressive texturing removes the visible seams and abrupt color shifts produced by independently generated and naively blended partial textures.Each subsequent generation pass is conditioned on previously generated textures, making new content a consistent extension of existing appearance.
5 Discussion
TAPESTRY’s consistent turntable videos support both complete UV texturing and neural representations such as 3DGS. The framework’s context-aware inpainting addresses self-occlusion, while its main limitations are dependence on input mesh quality and non-controllable lighting.
- Consistent TTVs provide training signals for neural assets such as 3DGS, capturing volumetric details like fur more realistically than standard UV textures.This demonstrates that the generated videos support representations beyond mesh texturing.
- The progressive, context-aware inpainting pipeline produces complete and seamless UV textures from geometrically consistent turntable videos.The full pipeline resolves incomplete self-occluded regions and avoids the seams and color shifts seen when generating a second TTV independently.
- TAPESTRY’s high-consistency outputs can directly drive 3DGS for interactive web visualization.The framework is trained on a single DGX Spark, demonstrating feasibility in a low-budget setting.
- The method’s output quality is bounded by the quality and topology of the input 3D mesh.Global consistency can remain on suboptimal inputs, but high-fidelity results require clean, well-defined geometry.
- Lighting generated by the video model is not controllable like an environment map, which hinders relighting.