PaperlayerView exact revision on arXiv

Source-linked AI summary

GLOSS: Geometric Local Self-Similarity Learning for Faithful Reference-Guided Texture Fill

Chenyue Cai, Anita Hu, James Lucas, Szymon Rusinkiewicz, Masha Shugrina

arXiv:2608.25461v1cs.GR

TL;DR

GLOSS addresses the difficulty of generating faithful whole-object textures with fine geometric detail and flexible artist guidance. It trains a shape-specific local diffusion inpainting model from one 3D shape, using geometry-aware reference patches and self-similarity. The method matches or outperforms strong large-3D-data baselines while supporting interactive texture fill, though its small-data and local-correlation assumptions limit generalization and applicability.

  • Problem

    Existing generative methods struggle with faithful whole-mesh texturing from single-view references and offer limited interactive control over varied texture sources.

  • Method

    GLOSS learns local texturing priors from one shape and off-the-shelf image models, then uses geometry-aware reference patches with batch multi-attention for patchwise inpainting.

  • Results

    GLOSS outperforms or matches strong generative baselines, while enabling interactive and automatic texture generation with reference-guided local control.

  • Takeaways & Limitations

    Patch-based local texturing provides a promising route to controllable, geometry-consistent texture generation without additional textured 3D data.

  • Takeaways & Limitations

    Small-data training limits generalization and increases per-shape cost; weak geometry-texture correlation can make the method inapplicable, and PBR consistency is not guaranteed.

Abstract

from arXiv · show

Using conditional image generators, texture artists can explore many single-view looks for an existing 3D shape. Despite impressive progress, state-of-the-art generative methods still struggle to generate a full object texture while closely adhering to fine scale geometric detail and single view references, leaving little room for artists guidance. Furthermore, current automatic models lack the flexibility for artist to explore multiple textures from varied sources in an interactive and controllable manner. Unlike methods trained on large 3D datasets that generate full object textures from global guidance, our work explores a local and less data-hungry approach to texture with explicit artist control. We leverage the geometric self-similarity and geometry-texture correlation existing in many natural and man-made shapes; and train a shape-specific local texture generation and completion model. This model learns from existing image model priors and a single 3D shape, and is guided by attending to a set of geometry-aware reference patches. The trained shape-specific network can transfer any novel reference to the full target object texture through patchwise inpainting. We show improved or comparable quality to strong image-conditioned texture generation baselines, suggesting local texturing as a promising research direction. Our model also enables local geometry-conditioned texture inpainting, guided by artist-selected references, and generalizes to PBR materials and unseen meshes for texture transfer. We piloted our novel texture fill capability as a Blender addon with several 3D texturing professionals who reported positive feedback on the model's controllability, practical usefulness, and creative affordances.

1 INTRODUCTION

GLOSS addresses limited control, flexibility, and geometric fidelity in whole-mesh texture generation by learning local, geometry-aware texture priors from a single shape. It supports reference-guided patchwise synthesis and interactive artist editing without large-scale 3D training data.

  • Existing methods struggle to produce faithful whole-mesh textures from single-view references while preserving fine geometric detail and local artist control.
  • GLOSS learns shape-specific local texturing priors from one shape’s self-similarity and off-the-shelf image generative models.
  • Artists can combine references, fill selected regions, edit existing textures, and obtain seamless geometry-aware transitions through local inpainting.
  • Batch multi-attention lets the model generate texture patches while using geometry-aware references from other regions of the same rendering.
  • The paper contributes a no-textured-3D-data training pipeline, interactive texture fill, preliminary transfer to unseen meshes and PBR materials, and competitive patch-based generation.

2 RELATED WORK

Related work spans classical self-similarity synthesis, pretrained diffusion-based texture generation, and models learned from textured 3D datasets. These approaches establish useful priors but leave challenges in local control, multi-view consistency, or 3D awareness.

  • Synthesis via self-similarity: Classical 3D texture synthesis blends patches across mesh geometry, while shape-analysis methods exploit symmetries and self-similarity.
  • Texture generation via Pretrained Diffusion Models: Pretrained diffusion methods replace large 3D datasets with 2D image priors for texture optimization, improving fidelity, material fusion, and multi-view generation.
  • Texture generation via Pretrained Diffusion Models: View-conditioned inpainting progressively generates mesh textures across views, but often struggles with multi-view consistency and limited 3D awareness.
  • Generative Texture Models Learned from 3D Datasets: Models trained on textured meshes or renderings generate texture fields, latent surface codes, or UV textures directly from large-scale 3D data.

3 LOCAL TEXTURE INPAINTING MODEL

GLOSS trains a mesh-specific diffusion model to inpaint local albedo patches conditioned on geometry and reference appearance. Its data pipeline creates paired local geometry-texture examples, while batch multi-attention transfers self-similar cues across patches.

  • Preliminaries: The input mesh contains faces, vertices, UV mapping, and optional normals textures; the target is a missing albedo texture with an alpha map for known regions.
  • Preliminaries: Rendering functions produce geometric and material images, while inverse rendering back-projects camera renderings into partial textures.
  • Data Generation: Training pairs are generated by prompting off-the-shelf diffusion models with geometry-conditioned views, converting outputs to albedo, and sampling local patches from partial textures.
  • Model Architecture: GLOSS models texture completion in a mesh’s local render space, using local albedo patches, masks, normals, positions, and aligned geometry-texture references.
  • Model Architecture: Batch multi-attention flattens and combines patch features so each pixel can attend across the batch, capturing long-range structural repetition and appearance cues.
  • Model Architecture: Larger attention batches improve access to geometrically similar regions but impose quadratic computational complexity, higher memory use, and longer training time.
  • Training: Training uses masked diffusion denoising with additional masks applied to known texture regions and restricts supervision to masked latent-space areas.

4 TEXTURING APPLICATIONS

GLOSS supports automatic and interactive texture generation through patch-level references, enabling artists to fill selected regions, blend multiple textures, and extend the workflow to PBR materials.

  • Interactive Texture Fill: GLOSS provides interactive texture fill and editing guided by patch-level references for the source mesh.The system supports reference-guided texture fill and editing as applications of the trained shape-specific model.
  • Reference Brushes: The workflow jointly processes target and reference patches, retaining target predictions while using reference patches to control the filled texture.Different regions can use different references, such as separate references for fish fins and scales.
  • Reference Brushes: Artists can construct reference brushes from generated single-view looks or selected mesh regions, with automatic and user-guided sampling modes.References can also come from a textured square mesh or existing image-to-PBR conversion techniques.
  • Interactive Texture Fill: Artists can select target faces, while local camera sampling covers the target area and adds cameras where topology or occlusion prevents precomputed coverage.Generated patches are back-projected into the texture map and blended with existing texture.
  • Interactive Texture Fill: Synchronized diffusion first promotes global consistency, after which noisy guidance is denoised in render space; multiple references can texture from scratch or edit existing results.The synchronized strategy uses the first 8/20 diffusion steps, followed by noise addition for 18/20 steps.
  • PBR Material Fill and Transfer: Although trained only for albedo, GLOSS preliminarily generalizes to metallic and roughness channels, producing partial but not full consistency because materials and albedo are painted independently.This extends the workflow to PBR material generation and interactive editing.

5 RESULTS AND EVALUATION

GLOSS is evaluated through interactive texturing, transfer, user feedback, automatic generation, and ablations. It achieves competitive automatic-texturing performance while providing local reference and geometry control, though preprocessing, generalization, global consistency, speed, and inpainting boundaries remain constraints.

  • Interactive applications: GLOSS supports interactive artist-guided filling, inpainting, and blending from varied reference sources, with close adherence to reference texture and underlying geometry.The comparisons report better alignment than other automatic methods, which can fail to follow artist intent, references, or local geometric details.
  • Interactive applications: Reference brushes can come from images and PBR materials, enabling geometry-aware transfer such as distressed artifacts appearing on sea-urchin bumps.The transfer attends to normal-map similarities rather than naively applying the source material.
  • Transfer and generalization: GLOSS shows preliminary zero-shot transfer to similar unseen meshes, but the authors describe generalization across shapes as limited.A model trained on one mesh can texture other similar shapes, while broader cross-shape generalization remains future work.
  • Limitations: Although mesh-specific data generation and pretraining are automated, this preprocessing limits practical application.The model also lacks global context during generation and can struggle to enforce globally consistent texturing.
  • Automatic texturing: The data-limited model achieves competitive behavior against large-scale generative models on full-texture generation for the selected meshes.The evaluation covers from-scratch and fine-tuned settings across 10 base meshes and 9 transfer meshes.
  • Automatic texturing: GLOSS achieves the best overall score in 3/4 reported metrics and remains competitive on nine transferred meshes with shorter fine-tuning.Qualitative results likewise compare competitively with state-of-the-art methods.
  • User study: Five professional 3D artists provided positive early feedback: 4/5 expected improved efficiency, and all would integrate interactive reference- and geometry-guided filling.Participants also identified speed and PBR inference time as improvement areas.
  • Ablations: Ablations indicate that batch-wise attention and geometric conditioning have the greatest effect on local texture quality.The study compares removing batch attention, geometry conditioning, image loss, and fine-tuning under matched metrics.

6 CONCLUSION

GLOSS combines generative models with artist workflows while preserving control and flexibility, using no additional 3D data. Its small-data design limits generalization, increases per-shape cost, and depends on local geometry-texture correlation.

  • GLOSS integrates generative models into artist workflows while preserving control and flexibility.
  • Its data pipeline and training strategy require no additional 3D data while maintaining geometric consistency and high-quality outputs.
  • GLOSS is competitive with state-of-the-art automatic methods and advances interactive texturing.
  • Operating in the small-data regime limits generalization and increases cost per shape.
  • The method is not applicable when local geometry-texture correlation is weak, and PBR channel consistency is not guaranteed.

A Data Generation

The data-generation pipeline creates diverse single-view training inputs and filters or processes them for texture learning. It also uses mask down-sampling designed to prevent invalid regions from contributing to training loss.

  • OpenAI’s o5 model generates concise, diverse prompts for object appearances, including 500 additional prompts after an initial set.
  • CLIP-based aesthetic filtering removes low-quality ControlNet-generated views before training.
  • DiffusionRenderer provides intrinsic decomposition, whose predicted albedo generally has sufficient quality for training.
  • Min-pooling downsamples masks to latent resolution so any latent pixel overlapping a masked image region is masked.This prevents nearest-neighbor interpolation from leaking invalid untextured regions into the latent-space loss.

C Additional Training Details

GLOSS adopts Stable Diffusion 2.1’s diffusion loss formulation and trains shape-specific models using generated single-view data. The appendix specifies schedules, optimization settings, and comparison conditions.

  • The noisy input z_t is computed with the DDPM formulation adopted from Stable Diffusion 2.1.
  • The scaled linear DDPM schedule uses β_min = 8.5 × 10^-4 and β_max = 1.2 × 10^-2.
  • The model G_M is trained against a velocity-prediction target.
  • From-scratch training uses AdamW at learning rate 1e-4, batch size 32, and 80k steps, taking around 40 hours on 4 NVIDIA A100 GPUs per object.
  • Each mesh receives 550 generated 512×512 single-view renderings, with 50 views each assigned to evaluation and testing.
  • For fair comparison, all methods use the object name as text conditioning, while GLOSS uses an empty string and single-view conditioning.

D.2 Metrics Details

Patch-level experiments render 256×256 patches under controlled camera settings, while evaluation samples textured views for LPIPS and untextured views for FID and CMMD. Reported tables cover overall, patch-level, and transfer-mesh metrics.

  • Patch-level experiments render 256×256 patches with camera–mesh distance fixed at 0.25 and field of view varied from 0.4 to 0.8.
  • LPIPS evaluation samples camera views on faces with existing textures to compare against ground-truth textures.
  • FID and CMMD evaluation uses views that are entirely untextured.
  • Table 3 reports overall DreamSim and patch-level LPIPS, FID, and CMMD across 10 shapes with 50 views each.The caption describes competitive performance despite using no textured 3D training data.
  • Table 4 reports overall and per-example LPIPS, FID, DreamSim, and CMMD on nine transfer sub-meshes spanning three base shapes.

D.3 Model Ablations

The ablations show that multi-attention and geometry conditioning are essential for texture generation and geometric consistency, while image-space loss and fine-tuning mainly preserve subtle detail.

  • Removing multi-attention causes the model to completely fail at texture generation.
  • Without geometry conditioning, outputs become geometry-inconsistent, misaligning creases and extending fin textures onto the fish body.
  • Removing image-space loss or fine-tuning preserves competitive overall results but can lose fine-grained details or subtle texture variations.The urchin shell example illustrates failures from omitting fine-tuning.

D.4 Completion Ablations

Completion experiments examine pipeline settings, attention behavior, efficiency, failure cases, and the interactive prototype’s evaluation setup.

  • Completion settings: Setting SyncMVD latent guidance strength to 2 yields the highest quantitative texture quality, while strength 10 degrades performance.All ablated variants underperform the original completion setting, indicating that each pipeline component contributes to automatic texture completion.
  • Attention behavior: Batch multi-attention lets target patches use self-similar references from any mesh region or other runtime source.Visualizations show different attention emphasis for batches with the same references but different target images, including scales and fins.
  • Efficiency: Interactive completion typically uses seven references and one target patch, with total processing times of 3.4 seconds for 1K textures and 5.3 seconds for 4K textures.These timings exclude communication between the GLOSS server and Blender add-on.
  • Failure cases: The model struggles with semantic appearance, weak geometry-texture correlation, repetitive-pattern fidelity, and small stochastic strokes.Examples include insufficient detail in turtle and koi fish faces and inconsistent repetitive patterns.
  • User study: The pilot study recruited five participants with prior 3D texturing experience, who completed questionnaires, hands-on tasks, and post-task interviews.Sessions lasted approximately 90 minutes and included generating texture brushes and painting about 70% of the mesh.
  • User study: The evaluation queried texture quality, controllability, creative affordance, workflow integration, usefulness, and complementarity with existing tools.Participants also rated perceived control and workflow-related statements on five-point scales.

G.4 Suggestions for Improvement

The prototype received overall positive feedback, but participants identified concrete interaction and fidelity issues requiring improvement.

  • Interactive control: Participants requested finer inpainting-strength control, real-time texture updates during brush strokes, and painting with full material channels.The fixed patch-based update scheme made small localized edits difficult.
  • Texture fidelity: The inpainting model can overly smooth transitions and lose sharp details, limiting localized editing fidelity.
  • View dependence: Certain camera angles can produce lower-fidelity results, reducing perceived controllability and visual quality.
  • Additional results: Additional results demonstrate seamless blending, further texturing applications, finetuning completions, and baseline comparisons across global and zoomed-in views.Automatic completion examples also include views opposite the conditional view and an additional model view.
Loading 2608.25461v1…