PaperlayerView exact revision on arXiv

Source-linked AI summary

Gate3DGen: A Pre-processing Model that Selectively Purifies Real-World Images for 3D Generation

Xiaolin Li, Zihao Huang, Fanpeng Meng, Chen Yang, Guanjun Wu, Kai Zhang, Jiemin Fang, Qi Tian

arXiv:2610.31649v1cs.CV

TL;DR

Real-world blur, uneven illumination, and occlusions challenge single-image 3D generation, while purification must preserve useful visual cues and leave clean inputs unchanged. Gate3DGen routes images through a one-step purifier before frozen generators; on synthetic inputs, it reduces TRELLIS.2 mesh distance by 61.5% relative to degraded inputs.

  • Problem

    Real-world blur, uneven illumination, and occlusions violate the assumptions of centered, sharp, fully visible objects under uniform lighting, corrupting cues used for 3D generation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Gate3DGen selectively bypasses inputs classified as clean and uses degradation-aware router features to guide a one-step purifier before frozen 3D generators.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    61.5% reduction in TRELLIS.2 mesh distance relative to degraded inputs; Gate3DGen also closes up to 83.1% of the mesh-distance gap to clean-reference inputs.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The findings support evaluating purification through downstream geometry, rendered appearance, and object fidelity, not image similarity or visual quality alone.

    Source: §5 · p. 8

    FD-DINOv2 compares input and render feature distributions but does not directly measure degradation removal or per-object fidelity, so its rankings need other evidence.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Single-image 3D generation has achieved high fidelity on curated benchmarks, yet degrades sharply on real-world images with uneven illumination, blur, and occlusions. Purifying such inputs requires deciding when to intervene and preserving the visual cues needed for 3D generation, while leaving clean inputs unchanged. We introduce Gate3DGen, a selective pre-processing model that gates image inputs to existing 3D generators without retraining them. A degradation-aware router bypasses inputs classified as clean and extracts features to guide purification of the remaining inputs. Conditioned on these features, a 3D-oriented one-step purifier purifies degraded inputs with spatial losses that encourage preservation of object identity, contours, and textural cues. To provide paired supervision, we construct a synthetic data engine that renders clean views of 3D assets and synthesises composite degradations. On the synthetic benchmark, the purifier improves image fidelity and downstream 3D geometry in only 2.3s on average, reducing TRELLIS.2 mesh distance by 61.5% relative to degraded inputs. VLM evaluation and human preferences on real photographs further support identity preservation and degradation correction. More demos and code are available at https://supra-3d.github.io/gate3dgen.

1 Introduction

Real-world blur, uneven illumination, and occlusions challenge assumptions made by single-image 3D generators, while generic editing can alter cues needed for reconstruction. Gate3DGen selectively purifies inputs and improves image fidelity and downstream geometry without retraining the generators.

1 Introduction

  • Real-world blur, uneven illumination, and occlusions violate the centered, sharp, fully visible, uniformly lit image assumptions of current 3D generators.Common preprocessing does not directly address photometric corruption or foreground occlusion, while generic editing can alter appearance or contours and harm downstream geometry.Source: §1 · p. 2
  • Gate3DGen bypasses inputs classified as clean and uses router features to guide a one-step purifier that preserves identity and geometric and textural cues.Spatial losses encourage cue preservation, and the downstream generators remain frozen.Source: §1 · p. 2
  • 61.5% reduction in TRELLIS.2 mesh distance and 39.1% reduction in foreground LPIPS relative to degraded inputs; inference averages 2.3 s per image.Purified inputs also improve all geometry and appearance metrics across TRELLIS.2 and Pixal3D.Source: §1 · p. 2
  • The synthetic data engine renders 3D assets and synthesizes composite corruptions to provide paired supervision and a benchmark with 3D ground truth.The benchmark supports evaluation of both image fidelity and downstream 3D geometry.Source: §1 · p. 2

2 Related Work

Prior work addresses input robustness through preprocessing, specialized restoration, occlusion recovery, and image editing, but these approaches leave gaps for composite degradations and downstream 3D fidelity. Gate3DGen instead focuses on selective purification before frozen 3D generators.

Robustness in Single-Image 3D Generation

  • Common foreground segmentation and recentering do not directly correct illumination, blur, or foreground occlusions; specialized methods target illumination, while other work recovers occluded geometry.The cited recovery approaches include volumetric reconstruction and multi-view synthesis.Source: §2 · pp. 2–3

2D Image Editing and Degradation Routing

  • Gate3DGen targets selective purification of composite illumination, blur, and occlusion degradations before frozen generators, with clean references and spatial losses used only in training.Its focus is input purification rather than modifying the downstream generator.Source: §2 · p. 3
  • Instruction-guided editing and all-in-one restoration optimize compelling images without explicitly measuring downstream 3D fidelity, and edits can alter appearance or contours.Editing clean photographs can also introduce unwanted changes, motivating selective restoration.Source: §2 · p. 3

3 Method

Gate3DGen selectively purifies degraded images before frozen 3D generators while leaving inputs classified as clean unchanged. Its synthetic data engine supplies paired supervision for purification and routing.

3.1 Problem Formulation

  • The formulation maps an observed image toward its matched clean reference, while a router selects purification or an unchanged KEEP path to avoid unnecessary edits to clean inputs.Uneven illumination, blur, and occlusions can violate the assumptions of centered, sharp, fully visible subjects under uniform lighting.Source: §3.1 · p. 3

3.2 Synthetic Data Engine

  • The engine renders eight views per asset, selects four informative anchors, synthesizes composite degradations, and produces about 44k purification pairs and 100k routing pairs using disjoint training and evaluation assets.The assets come from about 4k high-fidelity 3D assets.Source: §3.2 · p. 4

3.3 Degradation-Aware Router

  • The router derives a purification probability from PURIFY and KEEP logits, thresholds it into a routing decision, bypasses predicted-clean inputs, and extracts continuous degradation-aware features for conditioning.The router returns both the probability and features used by the purifier.Source: §3.3 · p. 4

3.4 3D-Oriented One-Step Purifier

  • The purifier conditions on the observed-image latent and router features, predicts at the noise endpoint t = 1, and uses one network evaluation followed by decoding instead of iterative denoising.The decoded image is supervised alongside the clean latent.Source: §3.4 · p. 4
  • Region-weighted reconstruction supervises the full subject and degradation-affected regions, while edge losses encourage preservation of contours without gradients through the frozen 3D generator.The two region terms are normalized independently; the edge term matches horizontal and vertical finite differences weighted by adjacent subject-mask values.Source: §3.4 · p. 5

4 Experiments

Experiments assess routing, 2D purification, downstream 3D generation, real-photo outputs, and purifier ablations. Gate3DGen improves synthetic 3D metrics and receives favorable real-photo identity and purification evaluations, with routing limitations on in-the-wild proxy labels.

4.1 Experimental Setup

  • The benchmark uses held-out DTC cases with matched clean references and ground-truth meshes, comparing seven editors through Pixal3D and TRELLIS.2.The reported 3D metrics include mesh distance, F1, Chamfer distance, normal-map PSNR, and rendered shaded and albedo LPIPS.Source: §4.1 · p. 5

4.2 Degradation-Aware Routing Results

  • 88.45% recall and 90.77% accuracy yield 89.59% H on synthetic inputs, 5.55 points above MANIQA; on real-photo proxy labels, it does not outperform always-PURIFY.The real-image router reaches 70.56% H, while the always-PURIFY strategy has matching 58.33% accuracy.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 2D Purification Results

  • Gate3DGen alone improves all five clean-reference metrics, lowering LPIPS 39.1% and gradient MAE 16.4% while raising silhouette IoU to 0.9758.It takes 2.3 s per image, versus 15.6–100.8 s for the six baselines.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Downstream 3D Generation Results

  • On TRELLIS.2, MD and CD fall 61.5%, while F1 scores rise 3.8% and 6.4%; shaded and albedo LPIPS fall 10.9% and 10.6%.The method closes 83.1% of the MD gap to the clean-reference oracle.Source: §4.4 · p. 6, Table 3 · p. 7

4.6 Ablation Studies

  • Adding router features h lowers MD from 0.820 to 0.755 (7.9%) and rendered LPIPS from 0.1794 to 0.1658 (7.6%), while spatial losses yield lowest shaded/albedo LPIPS and more faithful contours.Source: §4.6 · p. 8

4.5 In-the-Wild Results

  • Gate3DGen leads VLM Identity (8.62) and Purification (8.00); human preferences exceed 50% against every baseline, with lowest rates versus GPT Image-2.5 Flare.FD-DINOv2 ranks it first on Pixal3D and second on TRELLIS.2 among editors, but none improves over unedited inputs; the metric does not directly measure degradation removal or per-object fidelity.Source: §4.5 · p. 8, Table 4 · p. 8

5 Conclusion

Gate3DGen uses paired images synthesized from 3D assets to purify inputs before frozen 3D generators. Its reported results support gains in image fidelity and geometry, with evaluation extending beyond image similarity and visual quality.

5 Conclusion

  • On 1,082 cases, Gate3DGen outperforms seven image editing methods in image fidelity and surface geometry for TRELLIS.2 and Pixal3D.The purifier closes up to 83.1% of the mesh-distance gap to clean-reference inputs.Source: §5 · p. 8
  • Gate3DGen combines a degradation-aware router and a 3D-oriented one-step purifier trained on image pairs synthesized from 3D assets, before frozen 3D generators.Source: §5 · p. 8
  • VLM scores and human preferences on real photographs support identity preservation and degradation correction.The authors motivate evaluating purification through downstream surface geometry, rendered appearance, and object fidelity beyond image similarity or visual quality alone.Source: §5 · p. 8

A Method Details

The method trains a degradation-aware router to choose between KEEP and PURIFY, then trains a one-step purifier using paired images and spatially weighted losses. The router and purifier use distinct training configurations, while inference uses a single forward pass.

A Method Details

  • The router learns PURIFY/KEEP decisions with a frozen development-set threshold, updating language-transformer layers 24–27 and final language normalization while freezing the language-model head and visual backbone except vision-block LoRA adapters.Token-level supervision weights decision and end-of-sequence tokens more heavily; rare degradation families and subtle degradations receive higher sample weights.Source: A Method Details · p. 12
  • The purifier updates image-transformer blocks 29–49 and cross-attention projections receiving router features h, while blocks 1–28 remain frozen.Source: A Method Details · p. 12
  • Purifier training uses a soft alpha subject mask and a pooled RGB difference degradation map to weight spatial losses, with adjacent mask values weighting horizontal and vertical edge terms.The mask comes from RMBG-2.0; it and the paired images receive the same resize and random crop.Source: A Method Details · p. 12
  • The purifier trains at 1024 × 1024 for 15,000 updates on 32 GPUs and uses one inference forward pass with guidance scale 1.0 and a fixed random seed.Training uses flow-matching endpoint t = 1; loss weights are λlat = 0.1, λ1 = 1, λlpips = 0.5, λregion = 1, and λedge = 1.Source: A Method Details · p. 12

B Router Ablation

Router ablations compare the full model with an MLP on VAE latents and a language-only variant without vision LoRA. The full router balances recall and accuracy better than either reported variant.

B Router Ablation

  • The full router achieves 88.45% recall and 90.77% accuracy, compared with 97.04% recall and 75.30% accuracy for the language-only variant.Removing vision LoRA therefore corresponds to higher recall but lower accuracy, indicating a stronger bias toward PURIFY.Source: B Router Ablation · p. 12, Table 6 · p. 12
  • The MLP on VAE latents achieves 53.42% recall, below both the full router and language-only variant.Source: B Router Ablation · p. 12, Table 6 · p. 12
  • Table 6 reports recall and accuracy as percentages for synthetic evaluation inputs, with boldface marking each column maximum.Source: Table 6 · p. 12

C Relative Performance across 2D and 3D Metrics

Figure 4 aligns 2D purification and downstream TRELLIS.2 metrics against degraded-input and clean-input references. Gate3DGen moves toward the clean oracle across all shown metrics, while several editing baselines trade degradation removal against reconstruction-cue preservation.

C Relative Performance across 2D and 3D Metrics

  • Among the representative editing baselines in Figure 4, Gate3DGen makes positive progress toward the clean oracle on every metric.FireRed, GPT Image-2.5 Flare, and Mage-Flow show negative changes on several 2D fidelity or fine-grained 3D geometry measures.Source: C Relative Performance across 2D and… · p. 12
  • Figure 4 uses degraded inputs as the zero reference and matched clean inputs as the oracle reference at one, reversing lower-is-better metrics so larger radial values indicate better performance.Source: C Relative Performance across 2D and… · p. 12
  • Several editing baselines show a trade-off between visible degradation removal and preserving reconstruction cues, with negative changes on some fidelity or fine-grained geometry measures.Source: C Relative Performance across 2D and… · p. 12

D Additional Pixal3D Results

On Pixal3D, Gate3DGen leads the compared image editors on several geometry metrics and albedo LPIPS, but does not lead on every rendering metric.

D Additional Pixal3D Results

  • Figure 4 reports oracle-normalized progress from degraded input toward a clean-reference oracle, with farther outward always indicating better performance.Blue labels are 2D image metrics, green labels are 3D geometry and normal metrics, and orange labels are rendered-appearance metrics; negative values indicate deterioration from the input.Source: Figure 4 · p. 13
  • Gate3DGen obtains the best F_all1, CD, F_outer1, and albedo LPIPS among image editing methods on Pixal3D.Table 7 evaluates the same 1,082 cases with the same metric definitions as the TRELLIS.2 evaluation.Source: D Additional Pixal3D Results · pp. 12–13, Table 7 · p. 13
  • FireRed has higher normal-map PSNR and lower shaded LPIPS, while Gate3DGen's albedo LPIPS is 0.2773 versus 0.2775.Rendered-appearance improvements are smaller on Pixal3D than on TRELLIS.2.Source: D Additional Pixal3D Results · p. 13, Table 7 · p. 13

Related papers

Loading 2610.31649v1…