PaperlayerView exact revision on arXiv

Source-linked AI summary

GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer

Sayan Deb Sarkar, Sinisa Stekovic, Vincent Lepetit, Iro Armeni

arXiv:2510.16136v1cs.CVcs.AIcs.GR

TL;DR

3D appearance transfer must handle substantial geometric differences while carrying texture and fine detail to the input object. GuideFlow3D uses inference-time guidance with a pretrained rectified-flow model and reports better results than baselines. Its framework supports mesh, image, and text inputs, though its optimization-based approach is not intended for real-time use.

  • Problem

    3D appearance transfer must handle varied appearance representations and substantial geometric differences, while preserving the input object's global form and transferring finer appearance details.

    Source: §1 · pp. 1–2

  • Method

    GuideFlow3D interleaves inference-time rectified-flow sampling with differentiable guidance, using part-aware appearance or self-similarity losses without retraining the pretrained model.

    Source: §1 · pp. 2–3

  • Results

    GuideFlow3D consistently outperforms baselines, producing visually appealing results that reflect intended style and capture fine-grained geometry.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports mesh, image, or text appearance inputs, with mesh guidance affecting geometry and texture and image or text guidance targeting texture alone.

    Source: §1 · p. 2

    Because the method is optimization-based, it is not meant for real-time use.

    Source: §6 · p. 10

Abstract

from arXiv · show

Transferring appearance to 3D assets using different representations of the appearance object - such as images or text - has garnered interest due to its wide range of applications in industries like gaming, augmented reality, and digital content creation. However, state-of-the-art methods still fail when the geometry between the input and appearance objects is significantly different. A straightforward approach is to directly apply a 3D generative model, but we show that this ultimately fails to produce appealing results. Instead, we propose a principled approach inspired by universal guidance. Given a pretrained rectified flow model conditioned on image or text, our training-free method interacts with the sampling process by periodically adding guidance. This guidance can be modeled as a differentiable loss function, and we experiment with two different types of guidance including part-aware losses for appearance and self-similarity. Our experiments show that our approach successfully transfers texture and geometric details to the input 3D asset, outperforming baselines both qualitatively and quantitatively. We also show that traditional metrics are not suitable for evaluating the task due to their inability of focusing on local details and comparing dissimilar inputs, in absence of ground truth data. We thus evaluate appearance transfer quality with a GPT-based system objectively ranking outputs, ensuring robust and human-like assessment, as further confirmed by our user study. Beyond showcased scenarios, our method is general and could be extended to different types of diffusion models and guidance functions.

1 Introduction

3D appearance transfer must carry texture and fine geometric detail across objects with substantial shape differences, while the input object determines global geometry. GuideFlow3D steers a pretrained rectified-flow model at inference time and reports improved transfer results with a GPT-based evaluation validated by a user study.

1 Introduction

  • Appearance transfer must handle varied 3D representations and style cues from shapes, images, or language, including substantial geometric differences between objects.In this setting, the input dictates global geometry while the appearance object supplies texture and finer geometric detail.Source: §1 · pp. 1–2
  • Prior 2D-view-based 3D methods can be geometrically inconsistent across views, while direct use of 3D generative models generalizes poorly to appearance transfer across differing geometries.The authors identify limited control as an additional issue for direct application of these models.Source: §1 · pp. 1–2
  • GuideFlow3D adaptively steers a pretrained generative model at inference time through guided rectified-flow sampling, without retraining.The framework interleaves flow updates with latent-space optimization and supports objectives for which the base model was not trained.Source: §1 · pp. 2–3
  • Part-aware appearance and self-similarity losses provide guidance for localized appearance correspondence and structure-preserving transfer.The part-aware loss co-segments shapes into semantic parts; self-similarity preserves intrinsic structure within input regions.Source: §1 · pp. 2–3
  • The framework accepts mesh, image, or text appearance representations, with mesh guidance able to affect geometry and texture and image or text guidance targeting texture alone.When only an image is available, the authors note that a mesh can be generated using existing methods.Source: §1 · p. 2
  • GuideFlow3D consistently outperforms baselines, producing visually appealing results that reflect intended style and capture fine-grained geometry.Because ground-truth shapes are unavailable and geometries differ, the authors use GPT-based pairwise rankings and validate their consistency with a user study.Source: §1 · p. 2

2 Related Work

Prior work explores 3D generation and stylization across representations and conditioning approaches, but the surveyed methods have limits in geometry control, output format, or generality. GuideFlow3D builds on rectified flow and diffusion guidance to perform geometry-aware appearance transfer without task-specific fine-tuning.

2 Related Work

  • 3D generation spans autoencoders, GANs, diffusion, 2D generative-model-based approaches, and autoregressive mesh models; Trellis [72] generates structured 3D latents that support multiple decoded representations.GuideFlow3D builds on Trellis [72] for its ability to capture fine geometry and texture detail.Source: §2 · p. 3
  • 2D style-transfer methods evolved from optimization over pretrained CNN features to feed-forward, multimodal, diffusion, and instance-specific approaches, with generalization across diverse geometries remaining challenging.The surveyed instance-specific formulations derive structure and appearance constraints from pretrained vision-transformer features.Source: §2 · p. 3
  • 3D stylization covers point clouds, meshes, implicit fields, and Gaussian splatting, but surveyed methods may lack explicit mesh control, keep geometry fixed, or depend on specific training setups and modalities.GuideFlow3D combines universal guidance with Trellis [72] and geometric priors from PartField [41] for geometry-aware transfer without task-specific fine-tuning.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe SLAT as a sparse, structured representation of 3D assets and introduce universal guidance as a way to condition diffusion with differentiable objectives.

3 Preliminaries

  • SLAT encodes a 3D object's geometry and appearance as local latent codes anchored on active voxels of a grid.Source: §3 · p. 4
  • Sparse active voxels reduce computational demand and allow higher resolution; their positions outline coarse structure while their codes capture fine geometric and visual features.Source: §3 · p. 4
  • GuideFlow3D keeps voxel positions fixed to preserve coarse input geometry and steers only the associated latent codes during generation.Source: §3 · p. 4
  • A diffusion model reverses noise addition over time, with a trained network predicting the added noise during denoising.Source: §3 · p. 4
  • Universal guidance conditions diffusion with any differentiable guidance function in the reverse update, without altering the diffusion training process.Source: §3 · p. 4

4 GuideFlow3D

GuideFlow3D steers pretrained rectified-flow sampling with objectives that transfer appearance while respecting the input object's geometry. It uses structured latents and condition-specific guidance, interleaving optimization with flow updates to preserve realism and enable fine-grained control.

4.1 Structured Latents for Appearance Transfer

  • Input voxel positions remain fixed while their initialized structured latents are learnable, enforcing global geometric consistency with the input object.The latents are initialized by sampling from a normal distribution.Source: §4.1 · p. 5
  • With an appearance mesh, structured appearance latents support an appearance objective, while PartField features clustered by k-means provide approximate co-segmented correspondences.The latent features are not part-aware, so the method uses approximate rather than one-to-one matching.Source: §4.1 · p. 5
  • Without an appearance mesh, geometric clustering supplies same-cluster positives and other-voxel negatives for a self-similarity loss, promoting local consistency without globally homogenizing appearance.The loss promotes similarity within parts and disparity between parts.Source: §4.1 · pp. 5–6

4.2 Guiding Structured Latents for Appearance Transfer

  • Interleaving differentiable latent-space optimization with conditioned rectified-flow steps enables fine-grained appearance and structure control without retraining while preserving global realism.The optimized latent is used in the next flow step, and outputs are decoded into a 3D representation.Source: §4.2 · pp. 6–7
  • Image conditioning supports appearance guidance when an appearance mesh is available or structure guidance without one; text conditioning uses Lstructure with semantic clustering.Source: §4.2 · p. 7

5 Experiments

The experiments assess appearance transfer across intra- and inter-category settings, then extend it to diverse in-the-wild assets. GuideFlow3D is reported to preserve structure while adapting appearance across these settings.

5 Experiments

  • The benchmark uses procedurally generated input meshes and an ABO subset, with 250 pairs per setup across four intra- and inter-category combinations.The five categories are bed, cabinet, chair, table, and sofa.Source: §5 · p. 7
  • Evaluation uses GPT-5 rankings across six transfer criteria because ground-truth transferred textures are unavailable, and a user study checks alignment with human preferences.The criteria include Style Fidelity, Structure Clarity, Style Integration, Detail Quality, Shape Adaptation, and Overall Quality.Source: §5 · p. 7

5.1 Appearance Transfer

  • For intra-category chair transfers, GuideFlow3D improves appearance transfer and structural clarity in both image and text settings, with smooth material transitions and consistent fabric textures.The passage describes qualitative results; Table 1 covers GPT-based rankings for the simple-complex intra-category set.Source: §5.1 · p. 8, Table 1 · p. 8
  • GuideFlow3D maintains global coherence and fine structural fidelity under cross-category transfer, while several baselines degrade with large shape discrepancies.Trellis also closes a side hole in the bunk-bed example, changing the input geometry.Source: §5.1 · p. 8

5.2 Application: In-the-wild Appearance Transfer

  • Across diverse in-the-wild categories, GuideFlow3D transfers texture and fine geometric detail while preserving the target structure, without additional assumptions or adaptations.The examples include animals, vehicles, and furniture; the comparison reports more coherent, part-aware transfer than Trellis [72].Source: §5.2 · p. 9, Figure 6 · p. 10

6 Conclusion

The paper concludes that GuideFlow3D offers a flexible route to 3D asset stylization, while noting runtime and input-mesh constraints. Appendix results report text-conditioned transfer and a rendered-mesh image-conditioning variant.

6 Conclusion

  • The conclusion presents realistic asset generation from simple CAD designs as a possible route toward more accessible 3D content creation.The authors connect this possibility to rapid development, prototyping, and creativity in XR and gaming.Source: §6 · p. 10
  • GuideFlow3D is optimization-based and not intended for real-time use; it assumes noiseless meshes and depends on the performance of Trellis and PartField.The reported runtime is 96 seconds on an NVIDIA 4090 GPU, compared with 78 seconds for baseline [72].Source: §6 · p. 10

Appendix

  • The appendix includes additional qualitative results, scene editing, ablations, and human-evaluation and GPT-evaluation analyses.It also lists setup details and an analysis of perceptual-similarity metrics.Source: Appendix · p. 16

A Text Conditioning with Lstructure

  • With text conditioning, GuideFlow3D ranks above the compared methods across intra- and inter-category settings, including when structure and appearance belong to different object types.The passage attributes meaningful texture placement to self-similarity guidance, which encourages intra-part consistency and inter-part contrast.Source: A Text Conditioning with Lstructure · p. 16, Table A.1 · p. 16

B Rendered Image Conditioning with Lappearance

  • When no reference image is available, rendering the appearance mesh from a fixed viewpoint provides image conditioning, and this variant consistently outperforms Trellis [72].Its ranking scores are not directly comparable to Table 2 because this experiment includes only the top-performing baseline.Source: B Rendered Image Conditioning with… · pp. 16–18, Table B.1 · p. 18

C Ablation Study

The ablations indicate that guidance choices and semantically informed segmentation matter for transfer quality. A separate scene-editing application stylizes multiple objects while retaining their geometric structure.

C Ablation Study

  • The full method outperforms alternatives using global pooled features, nearest-neighbor matching, or K-means co-segmentation, supporting semantically informed part segmentation for correspondences.Global features were insufficient for rich semantic correspondence, while unstructured similarity lacked robust integration across different shapes.Source: C Ablation Study · p. 18, Table C.1 · p. 18
  • Appearance guidance yields stronger detail fidelity, while structure guidance gives better alignment and adaptability, supporting condition-specific loss selection.Both guidance objectives improve transfer quality in their respective settings.Source: C Ablation Study · p. 18

D Application: Scene Editing

  • In ScanNet scenes, GuideFlow3D stylizes multiple objects while preserving their geometric structure; objects with inaccurate pose annotations or high Trellis-processing demands are excluded.The application uses per-object CAD annotations and selects an appearance object for each semantic category.Source: D Application: Scene Editing · pp. 18–19, Figure 9 · p. 19

E Experimental Details

The experiments render meshes from standardized viewpoints and average metrics across objects. GuideFlow3D uses pretrained Trellis and PartField configurations with interleaved optimization and rectified-flow sampling.

E Experimental Details

  • Evaluation renders each asset from four viewpoints at a fixed radius and pitch, with yaw angles spaced 90 degrees apart, then averages metrics across objects.Meshes are placed in canonical poses to align the input, appearance reference, and output across views.Source: E Experimental Details · p. 19
  • GuideFlow3D uses Trellis image- or text-conditioning configurations and PartField features queried at mesh voxel coordinates for part-aware guidance.The implementation follows the pretrained models’ default settings.Source: E Experimental Details · p. 19
  • Each instance is optimized with AdamW at a 5 × 10^-4 learning rate for 300 steps interleaved with rectified-flow sampling on one NVIDIA RTX 4090 GPU.The same optimization settings are used across conditioning types.Source: E Experimental Details · p. 19

F Issues with Perceptual Similarity Evaluation

Appearance-transfer outputs lack a definitive target, and input and appearance geometries may be intentionally dissimilar. Perceptual similarity metrics can therefore miss local alignment and semantic details that matter to evaluation.

F Issues with Perceptual Similarity Evaluation

  • Without ground-truth outputs, reconstruction-based metrics are inapplicable, while perceptual scores provide only a coarse proxy for appearance transfer across dissimilar geometries.The appearance is adapted rather than reconstructed, and the setting lacks a definitive target output.Source: F Issues with Perceptual Similarity… · pp. 19–20
  • CLIP scores may remain close or favor inferior outputs despite visibly better texture alignment and part-aware fidelity; the gap grows on a cropped chair region.The example contrasts GuideFlow3D with Trellis and illustrates how scores differ between global and local geometry.Source: F Issues with Perceptual Similarity… · p. 20, Figure 10 · p. 20
  • Encoder-based metrics could be more useful with ground-truth part-level correspondences, which would allow comparison on cropped regions.Without such references, these metrics can overlook structural misalignment, texture stretching, and poor integration.Source: F Issues with Perceptual Similarity… · p. 20

G Human Evaluation

A user study compared GuideFlow3D with baselines on image-conditioned appearance transfer. Participants preferred GuideFlow3D most often, and the relative method ranking aligned with the LLM-based evaluation.

G Human Evaluation

  • Across approximately 1,000 human evaluations of simple-complex intra- and inter-category outputs, GuideFlow3D received the highest preference at 27.2%, and method rankings remained consistent with LLM-based evaluation.Source: G Human Evaluation · p. 21, Figure 11 · p. 21
  • Participants judged overall quality using the appearance image and front and back views of each structure mesh and method output.They were unaware of the methods and received the same prompts as the LLM evaluator.Source: G Human Evaluation · pp. 20–21
  • UV Nearest Neighbor ranked third despite lower quantitative performance, likely reflecting visually appealing but less semantically grounded results.The passage presents this explanation as a likelihood, not a demonstrated cause.Source: G Human Evaluation · p. 21

H GPT-based Evaluation

The GPT-based evaluator ranks textured outputs using a structured rubric and consistent rendered views. The authors note that multimodal-model biases and vulnerabilities may distort its rankings.

H GPT-based Evaluation

  • The evaluator ranks all methods’ outputs together from one consistent rendered view, judging each as a complete 3D object.The task is to assess appearance transfer from an image or text style source to a structure mesh.Source: H GPT-based Evaluation · pp. 21–22
  • Six criteria structure the GPT rubric: Style Fidelity, Structure Clarity, Style Integration, Detail Quality, Shape Adaptation, and Overall Quality.The prompt gives detailed explanations, fixed output formatting, and optionally staged in-context examples.Source: H GPT-based Evaluation · p. 22
  • GPT-based rankings may be distorted by hallucinations, position sensitivity, high-contrast texture preferences, or adversarial artifacts that exploit visual priors.Keeping these metrics ungamable remains an open challenge.Source: H GPT-based Evaluation · p. 22

I Qualitative Results

Qualitative comparisons show GuideFlow3D producing semantically and structurally consistent transfers under both text and image conditioning. Abstract prompts, strong geometry-appearance mismatches, and large shape variations remain challenging.

I Qualitative Results

  • Across text and image conditions, GuideFlow3D produces coherent, part-aware textures that preserve structural alignment and material cues.Examples include upholstery and wood patterns with sharp boundaries and material coherence.Source: I Qualitative Results · p. 22, Figure 12 · p. 23
  • For text prompts, GuideFlow3D grounds material and structural descriptions in geometry, while abstract descriptions such as “metal” or “mirror” remain challenging for all methods.Trellis improves texture plausibility but can miss color cues and local part correspondence.Source: I Qualitative Results · p. 22
  • Strong geometry-appearance mismatches and texture flattening under large shape variations remain difficult edge cases.The reported example identifies a failure case for all methods, where UV Nearest Neighbor is the most visually coherent.Source: I Qualitative Results · p. 22, Figure 12 · p. 23
  • For image conditions, UV Nearest Neighbor can misplace textures, while MambaST, Cross Image Attention, and EasiTex tend to blend or oversmooth appearance cues.Trellis improves realism but lacks geometric grounding; GuideFlow3D’s interleaved guidance cannot recover from its black-output failure case.Source: I Qualitative Results · p. 22, Figure 12 · p. 23
Loading 2510.16136v1…