PaperlayerView exact revision on arXiv

Source-linked AI summary

Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning

Ido Sobol, Kihyuk Sohn, Yoav Blum, Egor Zakharov, Max Bluvstein, Andrea Vedaldi, Or Litany

arXiv:2605.13852v1cs.GRcs.CVcs.LG

TL;DR

Precise 3D controls remain difficult to add to diffusion image generators without sacrificing photorealism. Realiz3D separates visual domain from control signals and uses layer-aware training and sampling, achieving controllable, photorealistic generation across multiview texturing and related tasks.

  • Problem

    Diffusion image generators lack precise geometry, material, and viewpoint controls needed for 3D-consistent generation, while synthetic-data fine-tuning can compromise realism through domain leakage.

  • Method

    Realiz3D decouples visual domain from 3D controls using Domain Shifters, layer-aware training, domain reassignment, and domain-aware sampling.

  • Results

    Realiz3D achieves strong performance in both control and realism across text-to-multiview generation and 3D-input texturing, with improved realism but slightly lower control adherence than synthetic baselines.

  • Takeaways & Limitations

    Realiz3D enables effective transfer of synthetic-data control to the real domain while preserving photorealistic generation.

  • Takeaways & Limitations

    A small control-adherence gap remains, and Realiz3D may not directly support image-based conditions because it targets largely domain-agnostic signals.

Abstract

from arXiv · show

We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.

1. Introduction

Realiz3D addresses the domain gap caused by fine-tuning real-image generators on synthetic renders for 3D control. It decouples visual domain from control signals and uses domain-shifting adapters plus layer- and denoising-aware strategies to preserve photorealism while transferring control to real images.

  • Motivation: Synthetic renders provide 3D annotations but differ substantially from real images, causing domain-gap degradation when used to fine-tune image generators.Real images lack geometry, material, and camera annotations, motivating supervision from rendered synthetic assets.
  • Realiz3D framework: Realiz3D decouples visual domain from 3D controls by learning a binary real-versus-synthetic signal through lightweight residual Domain Shifters.The framework first learns domain identity, then introduces desired 3D controls using synthetic supervision.
  • Realiz3D framework: At inference, operating in real mode while supplying 3D controls produces images that combine real-domain appearance with controllability.Domain Shifters reduce domain leakage by treating visual domain as a separate covariate before control learning.
  • Domain transfer: Realiz3D uses layer- and denoising-aware training and sampling, giving synthetic data stronger influence on structure and real data stronger influence on detailed appearance.Early layers and denoising steps predominantly determine structure, whereas later ones determine detailed appearance.
  • Contributions: Together, the framework’s domain-shifting adapters and progressive feature-space unification enable realistic, controllable generation while maintaining the base model’s realistic prior.The approach is presented as a flexible recipe for tuning diffusion models on controllable, domain-shifted datasets.

2. Related Work

Prior work augments pretrained image generators with control signals for conditional image and 3D generation, while related adapter methods address domain gaps through learned domain representations. Realiz3D differs by avoiding modifications to existing conditioning and paired data, while addressing potential controllable-versus-synthetic mode collapse.

  • Control in Image and 3D Generation: Control methods inject depth, normal maps, semantic masks, camera viewpoints, or human poses into pretrained generators and fine-tune them for conditional generation.Learning 3D controls is presented as an important application of controllable image generation.
  • Training Adapters To Mitigate domain Gaps: Wonder3D jointly generates multiview RGB images and normal maps using a learned 1D domain vector concatenated with the timestep embedding.Its domain switcher modifies the model’s existing conditioning mechanism, but it does not explicitly enforce consistency between generated images and normal maps and relies on synthetic data.
  • Training Adapters To Mitigate domain Gaps: Realiz3D learns domain embeddings without modifying existing conditioning or relying on paired data, while recognizing that joint adapter training can cause controllable-synthetic and realistic-uncontrollable modes.The related-work discussion identifies this mode-collapse risk because realistic and synthetic domains are both represented in text-to-image models.

3. Diffusion Models and Domain Gaps

Diffusion generation evolves from low-frequency structure to high-frequency detail across denoising timesteps and network layers, creating a basis for separating 3D control from visual realism. Realiz3D applies this principle by enforcing controls in earlier transformer layers while preserving realism in deeper layers.

  • Timesteps and Domain Gaps: Early denoising timesteps establish low-frequency structure, whereas later timesteps determine high-frequency details.Sampling proceeds from t = T to t = 0, starting from Gaussian noise and iteratively denoising the sample.
  • Timesteps and Domain Gaps: Noised real and synthetic distributions converge to the same Gaussian distribution as t approaches T, reducing their domain difference at early timesteps.SDEdit similarly adds noise before denoising to produce realistic images while preserving structure.
  • Layers and Domain Gaps: Low-resolution UNet features capture rough shapes and low-frequency patterns, while high-resolution features encode textures and fine details.Feature maps capture progressively finer details as denoising advances, with similar patterns reported for diffusion transformers and vision transformers.
  • Layers and Domain Gaps: Realiz3D enforces 3D controls in earlier diffusion-transformer layers while allowing deeper layers to maintain realism.This design leverages the observed relationship between layer depth, denoising progression, and the level of generated detail.

4. Method

Realiz3D separates visual domain identity from spatial control signals using lightweight Domain Shifters, then trains the shared diffusion backbone to transfer controllability across synthetic and real domains. Its staged training and Representation Binding strategy preserve realistic appearance while enabling realistic, control-faithful generation.

  • Motivation: Mixed-domain training alone does not fully prevent the model from associating non-null controls with synthetic appearance and forgetting realism.Synthetic-only fine-tuning learns controls but can catastrophically forget realistic image appearance.
  • Domain Shifters: Domain Shifters encode real or synthetic identity as low-rank residual adapters, separating domain modulation from control conditioning.Each module uses domain embeddings and a shared rank-r transformation with r ≪ d; Stage 1 trains only these adapters on real and synthetic images with null control.
  • Inference: At inference, setting the domain adapter to real mode while providing a non-null control produces outputs that are realistic and faithful to the specified control.This control transfer is established during fine-tuning and can be further strengthened without sacrificing realism.
  • Backbone Fine-Tuning: Stage 2 freezes Domain Shifters and fine-tunes the shared backbone with real and synthetic data to learn controllability without tying it to synthetic-domain statistics.Synthetic-only backbone fine-tuning can cause realism forgetting and incomplete transfer when real-domain adapters are activated.
  • Representation Binding: Representation Binding uses early domain-agnostic layers as a bridge while later layers retain real-domain appearance, promoting control transfer from synthetic to real images.During real-data iterations, randomly selected early DiT blocks are frozen, integrating real samples into the synthetic feature space while preserving realism.

5. Experiments

Realiz3D is evaluated on multiview texturing and text-to-multiview generation using synthetic and photorealistic data under a shared experimental setup. Across both tasks, it improves realism while maintaining strong 3D consistency and geometric or viewpoint control.

  • Experimental Scope: Experiments cover multiview texturing and text-to-multiview generation.The study evaluates Realiz3D on both tasks.
  • Datasets: The dataset contains 120K synthetic 3D assets rendered from V = 4 viewpoints with normal and position maps, plus photorealistic images generated from matched prompts.Synthetic and real data use the same textual descriptions to ensure fairness.
  • Implementation Details: For all tasks, the model generates a 2 × 2 grid of V = 4 orthogonal views at 512 × 512 resolution.The experiments use an internal pretrained text-to-image diffusion transformer and fine-tune it under a shared setup.
  • Multiview Texturing: Realiz3D achieves strong performance in both control and realism, whereas these objectives typically trade off.Evaluation includes 3D consistency, prior preservation, and text-image alignment, with realism measured using FIDI and KIDI.
  • Text-to-Multiview Generation: In text-to-multiview generation, Realiz3D significantly improves realism while maintaining 3D consistency comparable to the synthetic-only baseline.Qualitative results further show notable photorealism improvements with strong 3D consistency.
  • Multiview Texturing: In multiview texturing, Realiz3D significantly improves realism while maintaining 3D consistency comparable to the synthetic-only baseline.The quantitative comparison is summarized in Table 1, and qualitative results show photorealistic, geometrically faithful, cross-view-consistent outputs.

6. Conclusions, Limitations and Future Work … A.1. Motivation

Realiz3D enables controllability from synthetic data while preserving the base model’s photorealism through domain separation, layer-aware training, domain reassignment, and domain-aware sampling. The paper identifies remaining control-adherence gaps and motivates its strategies by showing that diffusion features shift from domain-agnostic structure to fine-grained detail across denoising steps and layers.

  • 6. Conclusions, Limitations and Future Work: Realiz3D enables controllability from synthetic data while preserving the photorealism of the base diffusion model.The framework is built around Domain Shifters, layer-aware training, domain reassignment, and domain-aware sampling.
  • 6. Conclusions, Limitations and Future Work: Domain Shifters learn separable visual domains, while layer-aware training maintains realism and controllability and domain reassignment improves transfer to the real domain.A domain-aware sampling process further boosts performance at test time.
  • 6. Conclusions, Limitations and Future Work: Realiz3D retains a small control-adherence gap because fine-grained details affect 3D consistency, domain gaps can alter geometry, and lighting bias can cause inconsistent appearance.The paper points to relighting advances as a direction for addressing these limitations, with failure cases provided in the Appendix.
  • A.1. Motivation: The appendix analyzes diffusion-transformer representations across denoising timesteps and network layers to understand how visual details evolve.The analysis builds on prior studies of diffusion features.
  • A.1. Motivation: The analysis uses LLM-generated prompts spanning photorealistic, cartoon, watercolor, anime, and comic styles, with images synthesized by the base DiT using 20 DDIM steps.The prompts describe random objects in random styles.
  • A.1. Motivation: Intermediate feature maps are extracted at multiple timesteps and layers, reduced with independently computed PCA, and visualized as RGB images.Features are extracted at timesteps 800, 700, 500, and 200 with T = 1000, across three layers representing the beginning, middle, and end of the network.
  • A.1. Motivation: At high noise levels, diffusion features primarily capture coarse, structural, and largely domain-agnostic information, while lower noise reveals high-frequency, fine-grained patterns.This trend supports distinguishing structural information from appearance-related detail during denoising.
  • A.1. Motivation: Across network depth, early layers encode coarse structural patterns shared by synthetic and real data, whereas later layers capture fine details.The layer-wise trend parallels the progression observed across denoising noise levels.

A.2. Case Study: Layer-Selective Training in 2D Image Generation

The case study tests whether layer-selective fine-tuning can learn controllability from synthetic data while preserving realism from real data. Assigning synthetic training to early DiT blocks and realistic training to later blocks motivates the layer-aware training approach.

  • Layer-Aware Training: Early diffusion-transformer layers capture coarse structure, whereas later layers capture fine details; layer-aware training therefore gives synthetic data stronger influence early and real data stronger influence late.This layer-role observation underlies the proposed Layer-Aware Training strategy.
  • Proof-of-Concept Experiment: The proof-of-concept used two equal-sized datasets: rendered synthetic images and realistic T2I-generated images prompted with similar objects, “highly realistic,” and “white background.”The experiment was designed to test controllability from synthetic data alongside realism from real data.
  • Proof-of-Concept Experiment: Layer-selective fine-tuning trained the first 50% of DiT blocks on synthetic data and the remaining 50% on realistic data, freezing the opposite block groups.The synthetic-only alternative fine-tuned the model solely on synthetic data; both approaches shifted generation toward object-centric images with white backgrounds.
  • Results: The layer-selective strategy consistently preserves realism while learning controllable properties from synthetic data, as illustrated in Figure 6.The figure describes early-block synthetic training and later-block real training as enabling controllability while maintaining photorealism.
  • Results: A corresponding timestep-selective experiment found the layer-selective approach more stable and robust.This proof-of-concept supplied the core motivation and inspiration for the layer-aware training approach in the main paper.

B. Method … C.1. Data

Realiz3D separates visual-domain adaptation from controllability through domain shifters, staged fine-tuning, and inference-time domain reassignment. Its data pipeline combines synthetic and photorealistic views with ImageNet-based real-world evaluation data.

  • B. Method: Realiz3D provides implementation details for its component modules and training stages.
  • B.1. Domain Shifters (Stage 1): Domain Shifters rebalance existing visual modes in the pretrained model rather than introducing new modalities.The design contrasts with adapters handling larger shifts, such as temporal conditioning or normal-map streams.
  • B.2. Fine-tuning with Representation Binding (Stage 2): Real samples mainly update later diffusion blocks for appearance refinement, while early blocks remain frozen to preserve synthetic-data control learning.This stage uses real images without explicit control supervision, avoiding disruption of controls learned from synthetic data.
  • B.2. Fine-tuning with Representation Binding (Stage 2): During real-data training, stochastic freezing selects early DiT blocks B ∈ [0, Bi] with i sampled from [0, τB].Setting τB ∈ [0.4, 0.5] of the total blocks provides stable and robust performance.
  • C. Implementation Details: The method uses the traditional diffusion loss throughout training, optimizing Domain Shifters at stage 1 and the DiT backbone at stage 2.
  • B.3. Inference-time Domain Shifting: Domain Reassignment probabilistically switches early DiT blocks to synthetic mode for real samples by substituting edomain ← esyn in their Domain Shifters.
  • B.3. Inference-time Domain Shifting: Time-based shifting has a stronger realism effect, whereas layer-based shifting is milder and more stable; hyperparameters are selected by validation over Bmax and tmax candidates.Bmax candidates span 0%–40% of blocks in 10% increments, while tmax values span 800–1000 in steps of 50.

C.2. Training and Sampling · D. Evaluation · D.1. Implementation Details

The evaluated models use stable reconstruction-loss training on 64 NVIDIA H100 GPUs, while evaluation compares Realiz3D with SDEdit and TRELLIS-based multiview generation. SDEdit performs worse than Realiz3D in both 3D consistency and realism, whereas TRELLIS generates assets rendered into four orthogonal views.

  • C.2. Training and Sampling: All evaluated models train for 10 epochs on 64 NVIDIA H100 GPUs with batch size 8 and learning rate 5e−5.Separately trained adapters use 3 epochs.
  • C.2. Training and Sampling: The models and adapters converged without overfitting during training.
  • C.2. Training and Sampling: Training relies only on the standard reconstruction diffusion loss, described as well-established and empirically stable.
  • D.1. Implementation Details: SDEdit applies noise to synthetic full-fine-tuning outputs and independently denoises the images with the pretrained T2I model.The same noise realization is used across images in each grid, with noise injection timestep t = 500.
  • D. Evaluation: SDEdit yields inferior performance to Realiz3D in both 3D consistency and realism, even at t = 500.
  • D.1. Implementation Details: For text-to-multiview generation, TRELLIS is evaluated using its official “TRELLIS-text-large” model with original hyperparameters.
  • D. Evaluation: TRELLIS directly generates a 3D asset from text, which is rendered into four orthogonal views: front, back, and side views.

D.2. Ablation Study: Qualitative Results · D.3. Additional Qualitative Results

The qualitative ablation examples show that Representation Binding and Inference-Time Domain Shifting improve control adherence while preserving strong realism. Additional results evaluate the method on main-paper data and held-out internal objects across multiview texturing, including comparisons with all baselines.

  • D.2. Ablation Study: Qualitative Results: Representation Binding improves control adherence in the model’s final outputs.This effect is illustrated qualitatively in Fig. 7.
  • D.2. Ablation Study: Qualitative Results: Inference-Time Domain Shifting improves control adherence in the model’s final outputs.The qualitative effect is illustrated in Fig. 7.
  • D.2. Ablation Study: Qualitative Results: Both ablation techniques preserve strong realism while improving control adherence.The passage attributes these two benefits jointly to Representation Binding and Inference-Time Domain Shifting.
  • D.3. Additional Qualitative Results: The additional qualitative evaluation uses the main paper’s evaluation data.These results are presented in the additional qualitative-results section.
  • D.3. Additional Qualitative Results: The evaluation also includes additional internal-dataset test objects held out from training.These objects were reserved for evaluation and were not used during training.
  • D.3. Additional Qualitative Results: Additional multiview-texturing results are presented in Fig. 8, Fig. 9, and Fig. 10.The figures show all baselines described in the main paper, with the corresponding prompt appearing in each caption.
  • D.3. Additional Qualitative Results: The method uses both normal and position maps as geometric conditions for multiview texturing.The passage notes that only the normal map is further discussed in the provided excerpt.

D.4. Text-to-3D Results

Realiz3D generates text-to-3D assets by backprojecting textures from four orthogonal views onto corresponding meshes. It achieves 3D consistency comparable to fully synthetic and full fine-tuning baselines while producing coherent, realistic assets.

  • Method: Text-to-3D generation backprojects generated textures onto their corresponding original meshes using only four orthogonal views.The four views may not fully cover the entire surface.
  • Results: Realiz3D achieves comparable 3D-consistency to fully synthetic and full fine-tuning baselines, producing coherent and realistic 3D assets.The full-tuning baseline was trained on both real and synthetic data for fairness.

E. Limitations and Future Work

Realiz3D improves realism but retains a small control-adherence gap, partly because smooth synthetic textures make fine-grained 3D consistency easier. Future work includes addressing lighting bias and extending the approach to video diffusion models trained with 3D conditions.

  • Limitations: Realiz3D significantly improves realism, but a small gap in control adherence remains.The paper attributes this limitation to several factors, including the sensitivity of 3D consistency to fine-grained details.
  • Limitations: Smooth textures in synthetic data make 3D consistency easier, allowing synthetic baselines to produce relatively smooth outputs without perfect pixel-level adherence.The passage states that synthetic baselines therefore face less need to learn perfect pixel-level 3D consistency.
  • Limitations: Multiview texturing can exhibit inconsistent lighting caused by the base text-to-image model’s lighting bias.
  • Future Work: A natural extension is to apply Realiz3D’s techniques to video diffusion models, which have recently demonstrated remarkable capabilities.Video diffusion models trained to incorporate 3D conditions are often fine-tuned on synthetic data, introducing a similar domain gap.
Loading 2605.13852v1…