Source-linked AI summary
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, Alexander C. Berg
TL;DR
Single-image novel 3D view synthesis must infer unseen object regions despite ambiguous geometry and missing correspondences. The paper uses transformation-grounded generation: it relocates visible pixels with a flow and visibility map, then completes and refines the remaining view. Experiments report realistic, accurate results that outperform existing techniques on standard synthetic datasets, with successful results on real photographs and sufficient accuracy for multi-view 3D reconstruction.
Problem
Single-view novel 3D synthesis requires inferring unobserved object appearances while handling ambiguous 3D shape and absent pixel-to-pixel correspondences.
Method
TVSN predicts transformed visible pixels and a visibility map, then conditions an image-completion network on that result to hallucinate and refine disoccluded regions.
Results
The method generates realistic images, outperforms existing techniques on standard CG-rendering datasets, supports multi-view 3D reconstruction, and produces successful results on real web photographs.
Takeaways & Limitations
Grounding image generation in viewpoint transformation preserves input-view cues while enabling synthesis of parts revealed only in the target view.
Takeaways & Limitations
The deterministic encoder-decoder can make unseen structures resemble common base shapes, motivating diverse or probabilistic generation methods.
Abstract
from arXiv · showhide
We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Instead of taking a 'blank slate' approach, we first explicitly infer the parts of the geometry visible both in the input and novel views and then re-cast the remaining synthesis problem as image completion. Specifically, we both predict a flow to move the pixels from the input to the novel view along with a novel visibility map that helps deal with occulsion/disocculsion. Next, conditioned on those intermediate results, we hallucinate (infer) parts of the object invisible in the input image. In addition to the new network structure, training with a combination of adversarial and perceptual loss results in a reduction in common artifacts of novel view synthesis such as distortions and holes, while successfully generating high frequency details and preserving visual aspects of the input image. We evaluate our approach on a wide range of synthetic and real examples. Both qualitative and quantitative results show our method achieves significantly better results compared to existing methods.
1. Introduction
Novel 3D view synthesis from one image must handle missing geometry and weak pixel correspondences caused by occlusion and viewpoint changes. TVSN grounds generation in predicted transformations and visibility, then completes disoccluded regions with losses designed to preserve detail and realism.
- Problem: Single-view synthesis is challenging because unseen object appearances, ambiguous 3D shape, and missing pixel-to-pixel correspondences must be inferred.These issues distinguish the task from ordinary dense prediction and can produce distorted outputs.
- Approach: A second encoder-decoder image-completion network conditions on the transformed result to hallucinate missing regions and refine distortions or unrealistic details.The architecture consists of two consecutive convolutional encoder-decoder networks.
- Training and outcome: Training combines adversarial and feature-reconstruction losses with pixel-level reconstruction to improve synthesized image quality and detail.The contribution specifically links recognition-style feature reconstruction and L1 image loss to improved results.
- Approach: TVSN predicts how existing input pixels transform and where disocclusion occurs, converting the remaining synthesis task into image completion and repair.The pipeline uses transformation where strong cues exist and reserves hallucination for missing or damaged regions.
- Approach: Its disocclusion-aware appearance flow network relocates visible input pixels while predicting a visibility map for regions that should be removed or completed.This explicitly separates reliable transformed content from newly revealed content.
2. Related Work
Prior approaches either transform available geometry without hallucinating disocclusions or generate views without reliable correspondence. TVSN combines learned pixel transformation, visibility reasoning, completion, and complementary perceptual and adversarial objectives.
- Geometry-based view synthesis: Geometry-based methods can transfer colors, textures, and local features accurately but cannot hallucinate newly revealed parts of an object.Single-view methods may also fail when disocclusion exposes content absent from the input.
- Geometry-based view synthesis: Database-driven geometry methods depend on sufficiently similar 3D models and are limited by the variation represented in those collections.Their quality is high when a close model exists, but coverage constrains applicability.
- Image generation networks: Encoder-decoder generation methods address single-image novel views but struggle with disentangling factors and globally smooth pixel-wise losses.These limitations contribute to lower visual quality than geometry-based methods that exploit strong correspondence.
- Transformation-based methods: Equivariant and flow-based methods facilitate transformations, but direct transformation is bounded by the input and cannot generate disoccluded pixels.TVSN grounds a generation network on learned transformations to hallucinate those missing pixels.
- Losses and completion: Perceptual and adversarial losses provide complementary image-generation objectives, while image-completion methods must also repair imperfect intermediate transformations in this setting.Unlike standard completion assumptions, the transformed input is not guaranteed to be correct.
3. Transformation-Grounded View Synthesis
The method decomposes novel-view synthesis into transforming visible pixels, identifying occluded or disoccluded regions, and completing the remaining image with learned context. It uses visibility and background masks to condition completion while preserving transformed appearance and refining artifacts.
- Novel view synthesis is modeled as moving persistent pixels, removing occluded pixels, and hallucinating pixels newly revealed by disocclusion.
- Disocclusion-aware Appearance Flow Network: DOAFN predicts a dense appearance flow and visibility map, masking transformed regions that should not be reused because of occlusion.Unlike AFN, DOAFN explicitly encodes which transformed pixels remain valid before image generation.
- Disocclusion-aware Appearance Flow Network: Visibility is derived from transformed 3D coordinates and surface normals, marking target locations visible when the viewing vector has a positive normal dot product.The mapping applies the desired transformation and perspective projection to source-image object coordinates.
- Disocclusion-aware Appearance Flow Network: Symmetry-aware visibility maps extend visibility reasoning to reflectionally symmetric objects, including unseen opposite sides in large viewpoint changes.The paper illustrates this difference for rotations from 160 to 340 degrees.
- View Completion Network: A second hourglass encoder-decoder completes missing content from the transformed view, preserves low-level information through skip connections, and repairs distortions or unrealistic details.It is motivated by cases where more than half of the novel view may be absent from the input.
- Training Losses: Training combines perceptual, adversarial, L1 reconstruction, and total-variation losses to improve complementary aspects of generated-image quality.VGG16 supplies perceptual feature losses, while feature matching is used with adversarial training for stability.
4. Experiments
Experiments compare TVSN against loss-function and visibility-map baselines on synthetic and real images, including 360-degree view generation and reconstruction. The results show improved handling of disocclusions, distortions, thin structures, and textured 3D reconstruction.
- Training Setup: Table 1 compares TVSN(DOAFN) with single-stage networks using L1, VGG16, adversarial, or combined VGG16+Adv losses, plus TVSN(AFN) without visibility masking.The training procedure first fits DOAFN, then the completion network, followed by end-to-end fine-tuning.
- Results: Compared with AFN, the method completes disoccluded regions consistently, corrects geometric distortions, better captures lighting, and restores thin chair structures such as legs.Perceptual and adversarial losses help missing legs contribute more strongly to the generated shape.
- Results: TVSN(AFN) and TVSN(DOAFN) perform similarly on L1 and SSIM, but visibility masking prevents severe AFN artifacts from being integrated into completed disoccluded regions.TVSN(DOAFN) instead leaves those regions for the view completion network to hallucinate consistently.
- Results: Combining VGG16 perceptual and adversarial losses produces the most effective configuration, whereas adversarial-only results can contain inconsistent colors and details and VGG16-only results remain blurry.The comparison evaluates each loss individually against their combination.
- 360 degree rotations and 3D reconstruction: Generating views around a full 360-degree rotation yields geometry- and detail-consistent images that produce significantly better-quality textured meshes than AFN.The method is evaluated by reconstructing textured 3D models from generated multi-view images.
- Real Images: Training with synthetic renderings composited onto random SUN397 backgrounds enables experiments on real internet images, although the simple compositing introduces inconsistent lighting and viewpoint.The background images are randomly cropped to 256×256×3.
- Real Images: The authors state that further investigation and more advanced physically based rendering are needed to model complex light interactions.This limitation concerns the training-data rendering process.
5. Conclusion and Future Work
The method outperforms existing techniques on standard CG-rendering datasets and produces synthesized views accurate enough for multi-view 3D reconstruction. Real web photographs also show successful, robust results, while unseen structures may resemble common base shapes because of the deterministic architecture.
- The transformation-grounded image generation network outperforms existing techniques on standard datasets of CG renderings.
- Synthesized images are accurate enough to perform multi-view 3D reconstruction.
- The method produces successful results for real photographs collected from the web, demonstrating robustness.
- Generated structures such as car headlights and wheels can resemble common base shapes, especially when absent from the input view.The authors attribute this to the deterministic encoder-decoder architecture and suggest diverse or probabilistic modeling as possible alleviation.
- The pipeline may extend to video prediction by transforming pixels from previous frames and completing missing regions.
A. Detailed Network Architectures
The detailed network architecture is provided in Figure 10.
- Figure 10 provides the detailed network architecture of the approach.
B. More examples
Additional car and chair examples show the method's novel views alongside intermediate outputs and results from other approaches.
- Figures 11 and 12 provide additional visual examples for car and chair categories.
- The examples include synthesized views, visibility maps, and DOAFN outputs.
- Views synthesized by other approaches are also provided for comparison.
C. Test results on random backgrounds
Figure 13 reports test results on synthesized images with random backgrounds, showing intermediate stages and comparisons with L1 and AFN baselines.
- Figure 13 presents test results on synthesized images with random backgrounds.
- The figure shows visibility maps, background masks, and DOAFN outputs as intermediate stages.
- The method is compared against L1 and AFN baselines.L1 and AFN could perform better on background areas if backgrounds were treated separately as in TVSN.
D. Arbitrary transformations with linear interpolations of one-hot vectors
The network generalizes beyond its 17 trained discrete transformations by synthesizing arbitrary viewpoints through linear interpolation of one-hot transformation vectors. Figure 14 demonstrates views generated between consecutive discrete viewpoints.
- The network was trained with 17 discrete transformations spanning 20° to 340° in 20° increments.
- Linear interpolation between adjacent one-hot transformation vectors represents intermediate viewpoints, such as 50° between 40° and 60°.
- Figure 14 synthesizes intermediate views using interpolated one-hot vectors between consecutive discrete views from the original transformation set.
E. More categories
The paper evaluates its approach across cars, chairs, motorcycles, and flowerpots, using categories selected for varied appearance, geometry, and generalization challenges. The reported figures present qualitative results across these settings and transformation experiments.
- Cars provide challenges from varied reflectance, textures, shapes, and many object instances.
- Chairs test thin structures such as legs and non-convex geometry, while also enabling comparison with prior work evaluated on cars or chairs.
- Motorcycle and flowerpot experiments assess generalization to additional categories using the same experimental setup.The motorcycle category contains 337 models and the flowerpot category contains 602 models.
- Figures 10–15 show the network architecture and qualitative results for cars, synthetic backgrounds, interpolated transformations, motorcycles, and flowerpots.