PaperlayerView exact revision on arXiv

Source-linked AI summary

SUPRA-TEX: 3D Generation with Super-Aligned Texture

Chen Yang, Jiemin Fang, Fanpeng Meng, Zihao Huang, Jichen Pang, Guanjun Wu, Zhikuan Bao, Qi Tian

arXiv:2610.22352v1cs.CVcs.GR

TL;DR

Single-image 3D texturing must preserve fine reference details while completing unobserved surfaces, but multi-view fusion can introduce artifacts and native generation struggles with detail alignment. SUPRA-Tex projects estimated visible materials into native observation latents and uses them as fixed anchors during 3D completion. It achieves state-of-the-art results on two texturing tasks and, with synthetic back-view conditioning, reports commercial-system-comparable or better performance in its comparisons.

  • Problem

    Single-image 3D texturing must preserve fine reference details while completing unseen surfaces, but multi-view fusion can introduce artifacts and native generation struggles to align details with surface locations.

    Source: §1 · p. 2

  • Method

    SUPRA-Tex estimates reference-view PBR maps, projects and encodes them as native observation latents, then uses fixed anchors to guide 3D material completion.

    Source: §1 · p. 2

  • Results

    SUPRA-Tex achieves state-of-the-art results in geometry-conditioned texturing and end-to-end asset generation, and with synthetic back-view conditioning matches or exceeds leading commercial systems in its comparisons.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The authors conclude that observation-guided completion preserves fine reference details while generating coherent, relightable materials across the full surface.

    Source: §6 · p. 12

    Large shape or pose discrepancies can misplace projected observations, and correspondence filtering cannot correct the fixed upstream geometry.

    Source: §5 · p. 11

Abstract

from arXiv · show

Single image 3D generation aims to dress a given 3D mesh with relightable materials while faithfully preserving the visual details of a reference image. Existing methods either introduce artifacts when fusing inconsistent multi-view predictions or struggle to preserve fine details through implicit image-feature conditioning. To resolve these challenges, we introduce SUPRA-Tex, built on the principle of decoupling visible-material estimation, surface alignment, and observation-guided 3D completion. This division of labor allows the 3D generator, trained on limited 3D data, to focus its capacity on preserving visible evidence and completing unobserved surfaces. Specifically, we first train an MLLM-based image editing model to predict lighting-separated PBR maps in the reference view. We then project these maps onto the visible surface and encode them with the pretrained material VAE encoder into native observation latents that share the generation target's latent space. An interleaved flow transformer holds these latents as fixed, read-only anchors while alternating unobserved-region and full-surface updates to generate complete material latents. This design achieves super-alignment with the reference, delivering What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces. Trained solely on public datasets, SUPRA-Tex achieves SOTA performance in both geometry-conditioned texturing and end-to-end asset generation, matching or even exceeding leading commercial systems on in-the-wild images. Project page: https://supra-3d.github.io/supratex/

1 Introduction

SUPRA-Tex addresses the challenge of preserving fine reference details while completing unobserved surfaces in relightable 3D assets. It separates visible-material estimation and surface alignment from 3D completion, using native material observations to guide generation.

1 Introduction

  • Multi-view synthesis can introduce seams and ghosting from cross-view discrepancies, while native 3D methods still struggle to place fine reference details at corresponding surface locations.Image cross-attention lacks explicit image-to-surface correspondence, and even projected image features remain visual features rather than material observations.Source: §1 · p. 2
  • SUPRA-Tex separates visible-material estimation and surface alignment from 3D completion so the generator can preserve visible details and complete unobserved surfaces.The authors motivate this division by the scarcity of 3D training data and the burden of asking one generator to perform these heterogeneous tasks jointly.Source: §1 · p. 2
  • An MLLM-based image editing model predicts lighting-separated PBR maps, which are projected onto visible surface locations and encoded as native observation latents in the target latent space.These directly decodable material observations provide aligned material content rather than abstract visual features.Source: §1 · p. 2
  • An interleaved flow transformer keeps observation latents as fixed, read-only anchors while alternating unobserved-cell and full-surface updates for coherent material completion.It allocates more updates to cells without observations while retaining full-surface interactions.Source: §1 · p. 2
  • Using only public datasets, SUPRA-Tex achieves state-of-the-art results in geometry-conditioned texturing and end-to-end asset generation, with synthetic back-view conditioning matching or exceeding leading commercial systems in the reported comparisons.The commercial comparison is for reference fidelity and visual quality on real photographs.Source: §1 · p. 2

2 Related Work

Prior work generates textures through multi-view synthesis and fusion or directly in native 3D representations. SUPRA-Tex uses projected material observations as conditioning anchors for native-space completion.

Multi-view texture generation

  • Multi-view texturing methods synthesize and fuse 2D views, while SUPRA-Tex completes materials in native 3D space using projected observations as anchors rather than fusing synthesized views into the final texture.Residual view inconsistencies can blur or ghost sharp reference details during projection and blending.Source: §2 · p. 3

Native texture generation

  • Native texture generation spans point, octree, and UV representations, latent texture generators, and 3D foundation models, with prior work also exploring partial observations and spatially aligned features.Even spatially aligned image features leave the generator to infer material properties, preserve reference details, and complete unobserved surfaces.Source: §2 · p. 3

3 Method

SUPRA-Tex estimates reference-view materials, aligns them to visible mesh surfaces, and uses those observations to guide complete, relightable material generation. Its pipeline separates visible-material estimation and surface alignment from 3D completion.

3.1 Preliminaries

  • SUPRA-Tex encodes geometry and surface materials as separate sparse VAE latents, then generates material latents conditioned on geometry for decoding onto the mesh.The material tuple includes albedo, metallic, roughness, and opacity.Source: §3.1 · p. 3

3.2 Image-to-Material Translation

  • An MLLM-based image editor predicts reference-view albedo and metallic-roughness maps, using a single forward pass to estimate material maps at 1024 × 1024 resolution.The model jointly predicts albedo, metallic-roughness, and auxiliary normal maps, and uses 2D image priors to disentangle illumination from intrinsic materials.Source: §3.2 · p. 4

3.3 Native Observation Encoding

  • Camera projection maps valid reference predictions to visible mesh surfaces, and the pretrained material encoder converts the partial PBR attributes into native observation latents.Unobserved voxels are omitted rather than assigned explicit material values; observation latents remain fixed during generation.Source: §3.3 · pp. 4–5, Figure 2 · p. 4
  • The observation representation includes spatial metadata describing coverage and proximity to unobserved regions, without padding unobserved surfaces with fictitious materials.The anchor latents and their spatial extent can also combine additional or edited views without retraining the generator.Source: §3.3 · p. 5

3.4 Observation-Guided Completion

  • An interleaved flow transformer keeps observations as fixed, read-only memory while alternating generation updates over unobserved cells and the full surface.It allocates more updates to unobserved cells while retaining full-surface interactions for coherent completion.Source: §3.4 · pp. 5–6

3.5 Training and Inference

  • Training conditions on visible-surface observations derived from public 3D assets and supervises complete material latents across the full surface; inference starts from noise and decodes the generated latents into PBR attributes.The observation representation remains fixed and unnoised during training; inference integrates the learned velocity field with Euler steps.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate SUPRA-Tex on DTC-Tex and Complex-Tex, then test geometry-conditioned, end-to-end, commercial, ablation, and runtime settings. The reported results show strong fidelity and quality alongside faster sampling, with memory and observation-encoding tradeoffs.

4.1 3D Generation with Given Geometry

  • SUPRA-Tex achieves the best mean in every reported metric across both view splits and datasets; reference-view shaded PSNRfg gains are 2.56 dB on DTC-Tex and 0.95 dB on Complex-Tex (TEXGen).Novel-view shaded PSNRfg gains are 1.76 dB and 1.09 dB, respectively, against Pixal3D⋆.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 End-to-End Single-Image-to-3D Generation

  • SUPRA-Tex leads all nine end-to-end metrics on both datasets; versus Pixal3D⋆, four-view PSNR improves by 1.77 dB on DTC-Tex and 2.49 dB on Complex-Tex.SUPRA-Tex and Pixal3D⋆ share predicted geometry but use separate appearance pipelines.Source: Table 3 · p. 8, §4.2 · p. 9

4.1 3D Generation with Given Geometry

  • SUPRA-Tex better preserves the mug’s lettering and plate’s patterns than multi-view and feature-conditioned baselines, while novel-view renders show coherent completed surfaces.Source: §4.1 · p. 8

4.3 Commercial Comparisons

  • On real and synthetic images, SUPRA-Tex receives 51.1%–84.8% human preference in visual-quality comparisons and ties for the highest VLM PBR score, 8.67.Source: §4.3 · pp. 9–10, Table 4 · p. 9

4.4 Ablation and Analysis

  • Read-only observations with alternating unobserved-cell and full-surface updates improve shaded fidelity over residual injection and match joint-update attention while improving albedo fidelity and sampling speed.The design samples 1.20× and 1.18× faster than joint-update attention on the two datasets.Source: Table 5 · p. 9, §4.4 · p. 10
  • Predicted metallic and roughness improve shaded PSNRfg over fixed or dataset-mean values; replacing predicted PBR with ground truth further improves quality, leaving room for better 2D estimates.Using Qwen-Image-2.1 narrows the oracle gap while keeping the same completion model.Source: §4.4 · p. 10

4.5 Runtime and Memory

  • SUPRA-Tex samples 1.20× faster on DTC-Tex and 1.18× faster on Complex-Tex; observation KV caching adds speed, while peak allocation rises by 1.94 and 1.81 GiB.The interleaved schedule reduces query-token evaluations by 17.54% and 16.91%, not total FLOPs.Source: Table 6 · p. 11, §4.5 · p. 11

4.4 Ablation and Analysis

  • Partial encoding changes representations even for unchanged materials; decoded examples retain main patterns but show additional detail and color errors, supporting material evidence for full-surface refinement.The three decoded cases are examples, not a cohort-wide reconstruction estimate.Source: §4.4 · p. 11

5 Limitations

SUPRA-Tex’s asset quality is limited by imperfect geometry, uncertain material estimates, and incomplete observations. These inputs can leave visible alignment errors or produce incoherent completion on hidden surfaces.

5 Limitations

  • Large image–geometry discrepancies can place observations on incorrect surfaces; filtering cannot fix the fixed geometry, and severe mismatches can prevent faithful texture alignment.Corrected-geometry controls are not evaluated.Source: §5 · p. 11
  • Implausible unseen structures in upstream geometry can hinder coherent texture completion and reduce final asset quality because SUPRA-Tex conditions on fixed geometry.Source: §5 · p. 11
  • When text or patterns extend beyond the visible region, the model may preserve the visible fragment yet fail to continue the layout coherently across the visibility boundary.Source: §5 · p. 12
  • The 2D material estimator can misinterpret properties such as metallicity, and the 3D generator may retain those errors, yielding implausible reflections under rendering or relighting.Accurate detail transfer does not guarantee correct material decomposition.Source: §5 · p. 12
  • Qualitative cases show motif distortion, spurious geometry, ambiguous reflectance estimates, and less precise hidden-side patterns; unseen-surface ground truth is not measured.The figure’s glossy painted wood example has no material ground truth, so its high metallic estimate illustrates ambiguity rather than verified error.Source: §5 · p. 12, Figure 6 · p. 12

6 Conclusion

SUPRA-Tex separates visible-material estimation and surface alignment from 3D completion, using fixed observation latents to guide full-surface generation. It achieves state-of-the-art results among evaluated methods and is competitive with commercial systems in human evaluations on a mixed real-and-synthetic comparison set.

6 Conclusion

  • Trained solely on public data, SUPRA-Tex achieves state-of-the-art results among evaluated methods in geometry-conditioned texturing and end-to-end asset generation.Source: §6 · p. 12
  • SUPRA-Tex separates visible-material estimation and surface alignment from 3D generation, using fixed, read-only native observation latents to preserve details while completing relightable materials across the surface.Source: §6 · p. 12
  • With synthetic back-view conditioning, SUPRA-Tex is competitive with commercial systems in human evaluations on the mixed real-and-synthetic comparison set.Source: §6 · p. 12

Supplementary Material

The supplement provides additional method and training details, evaluation protocols, results, and discussion. Its appendices cover geometry and observations, the generator, training, evaluation, analyses, visual comparisons, editing, and notation.

Supplementary Material

  • Appendix A details geometry and camera projection, visible-surface observations and filtering, the interleaved flow transformer, and training and inference procedures; Appendix B describes training data and optimization.Source: Supplementary Material · p. 18
  • Appendices C and D give evaluation protocols and 2D material prediction accuracy and runtime, with reference-view and novel-view fidelity and sampling efficiency reported in the main paper.Source: Supplementary Material · p. 18
  • Appendices E–G cover ablations and observation analyses, additional qualitative comparisons, synthetic back-view conditioning, texture editing, and limitations.Source: Supplementary Material · p. 18
  • Appendix H consolidates notation and definitions for material channels, surface and latent coordinates, observation metadata, and generator states.Source: Supplementary Material · p. 18

A Additional Method Details

The supplement specifies how image features and visible material observations are mapped to surface and latent coordinates. It also describes the interleaved generator and the data and optimization used to train its 2D material estimator.

A.1 Geometry and Surface Observations

  • Camera projection samples image features at latent-cell 3D positions; projected features and global image conditioning help align the predicted mesh and camera with the reference view.The active latent coordinates define the full surface, while valid visible surface voxels form the partial PBR input encoded before latent-level conditioning; observation coverage and boundary distance are defined on this support.Source: §A.1 · p. 19

A.2 Interleaved Flow Transformer

  • The generator uses separate observation and generation states at observed coordinates, with fixed observation tokens and alternating updates to unobserved cells and the full surface.Its 60 blocks read keys and values from both streams; unobserved states are updated in all 60 blocks and observed-coordinate generation states in 30, while output velocities cover all active cells. Table 7 summarizes representation dimensions, and Figure 7 depicts the streams and schedule.Source: §A.2 · p. 19, Table 7 · p. 19, Figure 7 · p. 20

B.1 Training Data

  • The 3D generator is trained on public 3D assets and human scans that provide geometry and material supervision for observation-guided full-surface completion.The 2D estimator is trained on paired images, material maps, and normal maps rendered from Objaverse assets; the datasets are used under their non-commercial academic research licenses.Source: §B.1 · p. 20

B.2 Training F2D

  • F2D is adapted and jointly fine-tuned on rendered aligned images and material and normal maps, with its VAE and text encoder frozen.It uses modality-axis attention adapters and a flow-matching objective; at inference, one prediction at the fixed maximum-noise endpoint estimates clean material latents decoded into 1024 × 1024 PBR maps.Source: §B.2 · p. 20

B.3 3D Generator Training

The generator is trained from visible ground-truth PBR observations encoded on visible surface voxels, while geometry remains conditioned throughout training. The evaluation datasets target detailed textures and exclude training-asset overlap.

B.3 3D Generator Training

  • Training observations come from public assets’ ground-truth PBR materials, with foreground validity and first-hit visibility selecting observed surface voxels; the full observation condition is dropped with probability 0.03.The geometry condition is retained during dropout.Source: §B.3 · p. 20
  • The generator is optimized for 30,000 steps with a global batch size of 576, using AdamW and a flow-matching objective.Training-time sampling uses σmin = 10−5 and t = sigmoid(ξ) with ξ ∼ N(1, 1).Source: §B.3 · pp. 20–21

C.1 Evaluation Datasets

  • DTC-Tex contains 100 selected Digital Twin Catalog objects, while Complex-Tex adds 110 Sketchfab meshes with denser inscriptions, finer patterns, and varied shapes.DTC-Tex emphasizes household products with printed labels or decorative motifs; Complex-Tex includes branded packaging, vehicles, buildings, and characters.Source: §C.1 · p. 21
  • Both evaluation benchmarks underwent duplicate checks to prevent overlap with training assets and are slated for release.Source: §C.1 · p. 21

C.2 Rendering and Metrics

The evaluation renders exported assets under shared cameras and lighting, using per-view and distribution metrics to assess appearance and reconstruction. The comparisons cover open-source texturing and generation methods as well as commercial systems, with commercial preferences and VLM ratings reported.

C.2 Rendering and Metrics

  • Assets are rendered under shared cameras and forward lighting; per-view PSNR, SSIM, VGG-LPIPS, and CLIP-I are supplemented by foreground-only PSNR and distribution or PBR-channel metrics.Full-image per-view metrics use a black background and therefore include silhouette errors; PSNRfg restricts scoring to the ground-truth object mask.Source: §C.2 · pp. 21–22

C.3 Open-Source Baselines

  • Given-geometry comparisons render 16 views for open-source texturing baselines, while end-to-end methods receive a single reference image and are evaluated through rendered asset appearance.Pixal3D⋆ and SUPRA-Tex share predicted geometry in the end-to-end comparison; scoring alignment does not affect generation.Source: §C.3 · p. 22
  • TEXTRIX, NaTex, LaFiTe, and MatMart are excluded from quantitative comparison because official implementations or pretrained weights were unavailable, and their reported scores use different datasets and protocols.Source: §C.3 · p. 23

C.4 Commercial Comparisons

  • Commercial comparisons use 45 separate images and two-view conditioning for SUPRA-Tex, while commercial systems receive the original reference image; 512 pairwise human evaluations and Qwen3.8-27B ratings are reported.The 45-image set includes real photographs and synthetic images; commercial systems’ training-data overlap with benchmark ground truth cannot be ruled out.Source: §C.4 · p. 23

D.1 2D Material Prediction

The section evaluates standalone F2D material predictions on 210 benchmark objects using reference-view images and specifies distinct metric domains for appearance and material channels. It also reports inference runtime and memory on an NVIDIA H800.

D.1 2D Material Prediction

  • Albedo PSNR and albedo, metallic, and roughness MAE use the ground-truth foreground, while full-image albedo PSNR, SSIM, and VGG-LPIPS include background pixels.Metallic and roughness MAE is measured in stored PNG encoding, not as physical-linear material error.Source: §D.1 · p. 23
  • Foreground PSNR is approximately 19–20 dB, while full-image PSNR is 28.25 dB on DTC-Tex and 28.19 dB on Complex-Tex.The full-image values include the shared background and should be interpreted separately from foreground PSNR.Source: §D.1 · pp. 23–24

D.2 F2D Runtime

  • On one NVIDIA H800, mean isolated DiT latency is 1.205 s per request and peak allocated memory is 66.25 GiB.Timing includes loading, conditioning, the DiT call, and VAE/PIL decoding, but excludes model initialization and PNG writing.Source: §D.2 · p. 24
  • Repeated DiT-only forward passes take 1.1990 s for one reference image and 1.1987 s when cycling through six images.These timings exclude loading, conditioning, VAE encoding and decoding, and file I/O.Source: §D.2 · p. 24

E.1 Ablation Configurations

The ablations vary how observed materials are encoded and used, and diagnostics compare observed-support encoding with zero filling and native PBR latents with image-feature conditioning. The supplied figures also identify comparisons of given-geometry, end-to-end, and commercial outputs.

E.1 Ablation Configurations

  • The architecture ablations compare conditioning designs, while material-content variants retain predicted albedo and replace metallic and roughness with constants or dataset means; the full-PBR variant uses all predicted channels.Oracle PBR is reported separately as a material-quality reference, and the variants share training and evaluation settings.Source: §E.1 · p. 24, Table 9 · p. 25

E.2 Partial and Complete Encoding: Diagnostic Protocol

  • The partial-versus-complete encoding diagnostic uses paired GT encodings, filtered reference-view observations, and object-bootstrap resampling to compare latent summaries and decoded material crops.Its coordinate coverage comes from each encoder’s spatial correspondence cache, so it is not evidence of independent geometry recovery.Source: §E.2 · p. 24

E.3 Observed-Support Encoding versus Zero Filling

  • Zero-filled observations darken unseen surfaces, whereas observed-support encoding retains a green calculator back and the dish’s red-and-white regions.The zero-filled examples also show spurious front-side characters on the calculator and dark mottling on the dish.Source: §E.3 · p. 24, Figure 9 · p. 25

E.4 Early Learning with Native PBR Latents

  • 29.46 dB shaded foreground PSNR is reached after 300 steps with GT PBR observations, versus 18.72 dB for Pixal3D⋆ with shaded RGB conditioning.At step 1500, shaded foreground PSNR is 30.86 versus 20.89 dB; the single-view oracle-material diagnostic does not isolate conditioning from architecture and optimization differences.Source: Figure 11 · p. 26, §E.4 · p. 26

F.1 Given-Geometry Texturing

  • Figure 12 supplies additional given-geometry texturing comparisons across twelve cases.Source: §F.1 · p. 26

F.2 End-to-End Generation

  • Figure 13 places each input beside reference- and novel-view renderings to compare detail placement at the reference view and appearance continuity across novel views.Source: §F.2 · p. 26

F.3 Commercial Comparisons

  • Figure 15 adds eight commercial-comparison cases, with front views showing reference detail reproduction and back views showing unseen-surface completion.Source: §F.3 · p. 26

G.1 Single-View Generation and Back-View Editing

SUPRA-Tex uses the same 3D generator for single-view completion, additional back-view material conditions, and edits to observed views. The examples show how synthetic back designs can guide unobserved surfaces without additional 3D training.

G.1 Single-View Generation and Back-View Editing

  • The same 3D generator accepts additional or edited view observations to put unseen surfaces under user control, without additional 3D training.Single-view results show its own completion; extensions show generation conditioned on supplied designs or edits.Source: §G.1 · p. 26
  • A geometry-consistent synthetic back view is converted with the reference into projected PBR observations and unified anchors for full-surface completion.The back view uses a minimally overlapping viewpoint, rendered surface normals, and MLLM-synthesized shading guided by the reference and optional prompts.Source: §G.1 · pp. 27–29
  • Supplementary visuals show twelve given-geometry comparisons, four end-to-end cases, and eight commercial-model cases, with novel views used to assess completed appearance.The four end-to-end cases share the reference image; SUPRA-Tex and Pixal3D⋆ share predicted geometry, while other baselines use their own geometry pipelines.Source: Figure 12 · p. 27, Figure 13 · p. 28, Figure 15 · p. 29
  • Editing the front or back observation transfers image edits to the corresponding 3D surface while keeping the edited region consistent with the rest of the asset.The workflow rebuilds the view’s observations and regenerates the complete material.Source: §G.1 · p. 29
  • On a porcelain plate, a geometry-guided back design supplies a second observation, and edits yield crane-and-peony, geometric, and koi variants without further 3D training.The synthetic back is a design hypothesis, not a second photograph of the real object.Source: §G.1 · p. 29

H Notation Reference

Related papers

Loading 2610.22352v1…