Source-linked AI summary
SynSin: End-to-end View Synthesis from a Single Image
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson
TL;DR
Single-image view synthesis remains difficult because one image must support both 3D scene understanding and plausible semantic completion. SynSin predicts a learned feature point cloud, renders it differentiably into a target view, and refines the result end to end without ground-truth 3D supervision. It generates high-quality views across three real-world datasets and generalises to higher resolutions and novel scene types.
Problem
Single-image view synthesis requires understanding 3D structure and scene semantics, while existing approaches commonly rely on multiple views, ground-truth depth, or synthetic scenes.
Method
SynSin predicts learned features and depth from one image, forms a 3D feature point cloud, renders it differentiably from a target viewpoint, and refines the rendered features into an image.
Results
SynSin generates high-quality images and outperforms voxel-based baselines on Matterport, RealEstate10K, and Replica, while generalising to higher resolutions and novel scene types.
Takeaways & Limitations
The learned 3D component supports view synthesis from a single image, including high-resolution outputs and reasonable videos along specified trajectories.
Takeaways & Limitations
Ground-truth-depth variants are upper bounds restricted to Matterport3D and cannot be trained on RealEstate10K because they use true depth information.
Abstract
from arXiv · showhide
Single image view synthesis allows for the generation of new views of a scene given a single input image. This is challenging, as it requires comprehensively understanding the 3D scene from a single image. As a result, current methods typically use multiple images, train on ground-truth depth, or are limited to synthetic data. We propose a novel end-to-end model for this task; it is trained on real images without any ground-truth 3D information. To this end, we introduce a novel differentiable point cloud renderer that is used to transform a latent 3D point cloud of features into the target view. The projected features are decoded by our refinement network to inpaint missing regions and generate a realistic output image. The 3D component inside of our generative model allows for interpretable manipulation of the latent feature space at test time, e.g. we can animate trajectories from a single image. Unlike prior work, we can generate high resolution images and generalise to other input resolutions. We outperform baselines and prior work on the Matterport, Replica, and RealEstate10K datasets.
1. Introduction
Single-image view synthesis aims to generate new viewpoints in complex real-world scenes, but requires understanding both 3D structure and scene semantics. SynSin addresses this with an end-to-end, 3D-aware model trained from image pairs without ground-truth 3D supervision.
- Single-image view synthesis generates images from new viewpoints using only one input image, supporting applications such as editing, animation, and 3D viewing.
- Successful synthesis requires 3D structure for viewpoint-dependent motion and semantics for plausible completion of partially visible objects.
- Multiple-view methods improve reconstruction but limit applicability because most images lack views from other angles.
- Depth-based single-image methods require ground-truth depth and may fail to generalise beyond the scene types represented in training data.
- Prior 3D-aware generative methods trained without 3D information showed promise mainly on synthetic single-object scenes and struggled with complex real-world scenes.
- SynSin predicts a high-resolution point cloud of learned features, renders it from a target viewpoint with a differentiable renderer, and refines the result into an image.
- SynSin is evaluated on Matterport, RealEstate10K, and Replica, where it generates high-quality images, outperforms voxel-based baselines, and generalises to higher resolutions and novel scene types.
2. Related work
Related work spans multi-image reconstruction, supervised single-image synthesis, and generative 3D representations. SynSin instead uses a single test image and an end-to-end pipeline that predicts and renders feature point clouds.
- View synthesis from multiple images: Multi-image view synthesis reconstructs geometry from depth maps or multi-view geometry before generating new views.
- View synthesis from multiple images: Learning-based multi-image methods synthesise views within narrow baselines, while object-focused methods learn implicit voxel representations from many training views.
- Single-image synthesis with supervision: Supervised single-image approaches rely on large datasets with ground-truth 3D or semantic information, whose acquisition is time-consuming and difficult, especially outdoors.
- Generative models: SynSin uses generative-model advances to produce high-quality images from its 3D-aware intermediate representation.
- SynSin pipeline: SynSin takes one image and a relative pose, embeds the image into features and depth, projects them into a feature point cloud, and renders the target view.
3. Method
SynSin combines learned semantic features and predicted depth into a 3D point cloud, renders it differentiably from a target view, and refines the result with generative inpainting.
- 3.1. Spatial feature and depth networks: SynSin projects the input image into semantic feature and depth maps, combines them into a 3D feature point cloud, and transforms it using a differentiable renderer.The spatial feature network preserves image resolution for semantic features, while the depth network estimates 3D structure without requiring perfect accuracy.
- 3.2. Neural point cloud renderer: Because the renderer softens spatial and depth-ordering decisions, gradients can propagate to all points in the z-buffer, supporting end-to-end learning of depths and features.This differentiability allows training from losses on the final rendered image rather than ground-truth 3D supervision.
- 3.2. Neural point cloud renderer: The neural point cloud renderer splats projected points across regions of radius r and accumulates the K nearest points with alpha compositing instead of selecting only the nearest point.The spread and fall-off are controlled by r and M, while γ controls blending; γ = 0 recovers hard z-buffering.
- 3.2. Neural point cloud renderer: Custom CUDA kernels accelerate tiled point projection and per-pixel depth sorting, rendering six 512^2 point clouds in 36ms forward and 5ms backward on one V100.The implementation uses shared-memory priority queues to reduce global memory traffic and is substantially faster than converting the same point cloud to a voxel grid.
- 3.3. Refinement module and discriminator: The refinement module inpaints regions lacking projected features and corrects local projection errors while aiming to preserve semantic texture and geometric structure.It uses ResNet blocks conditioned on the input view, with discriminators operating at lower and higher resolutions.
4. Experiments
SynSin is evaluated on realistic indoor and outdoor datasets against ablations, voxel baselines, depth-based systems, and a two-view method. It outperforms baselines, generalises to higher resolutions and novel domains, and produces preferred trajectory videos.
- Experimental setup: SynSin is evaluated on Matterport3D, RealEstate10K, and Replica, using realistic indoor and outdoor scenes with substantial viewpoint changes.The experiments include 2000 test image pairs and report PSNR, SSIM, and perceptual similarity.
- Comparisons with other methods: SynSin performs better than 3DView despite its ground-truth-depth training and approaches StereoMag, which uses two input views at test time.The comparison highlights SynSin’s single-image setting against systems with additional depth supervision or input views.
- Main results: SynSin outperforms baselines on all reported metrics and conditions on Matterport3D and RealEstate10K.The results support the utility of its 3D representation and inpainting module.
- Generalisation to higher resolution images: SynSin can apply models trained at 256 × 256 directly to 512 × 512 images without further training, while the voxel baseline suffers a dramatic performance drop.Fully convolutional networks and point-cloud sampling support resolution changes; coarse voxel grids lose precision through downsampling.
- Depth predictions: SynSin’s predicted depth is higher resolution and more realistic than baseline predictions, although small objects and fine details are not accurately recreated.The differentiable point cloud renderer appears to improve depth quality over hard z-buffering or a smaller footprint.
- User study: Animating still images: Users prefer SynSin’s generated trajectory videos over the Vox w/ ours baseline in both user-study protocols.The authors attribute this preference to smoother motion and fewer artefacts.
5. Conclusion
The conclusion presents SynSin as an end-to-end single-image view-synthesis model built around differentiable neural point-cloud rendering and generative refinement. It reports learning on realistic datasets, generalisation to new settings and resolutions, and trajectory video synthesis.
- 5. Conclusion: SynSin combines a differentiable neural point cloud renderer with a generative refinement module for single-image view synthesis.These are identified as the system’s two key components.
- 5. Conclusion: The approach can be learned end-to-end on multiple realistic datasets and generalises to unseen scenes and higher image resolutions.The conclusion also reports trajectory synthesis from an initial image.
- 5. Conclusion: SynSin can generate reasonable videos along a specified trajectory, and neural point-cloud rendering may apply to other generative tasks.The latter is stated as a potential application beyond view synthesis.
SynSin: Appendix
The appendix provides additional results, architectural details, baseline information, dataset information, and discussion of successful and unsuccessful design choices.
- SynSin: Appendix: The appendix contains supplementary results, architecture details, baseline information, dataset information, and discussion of design choices.These materials are organised in Sections A–E.
A. Additional experimental results
Additional experiments report SynSin’s KITTI comparison, qualitative results across several datasets, and depth and trajectory visualisations. The results also identify failure cases under large viewpoint changes and show smoother motion than a voxel baseline.
- KITTI results: SynSin achieves similar or better results than [8] on all KITTI metrics.The comparison uses a held-out KITTI set and compares SynSin with [8].
- KITTI results: [8] produces severe artefacts for backward motion because it resamples the input image and cannot generate unseen pixels.Both methods also show failure cases when viewpoint changes substantially exceed the average test-time change.
- Qualitative results: Additional qualitative results are provided for RealEstate10K, Replica, and Matterport3D.The supplementary video also shows generated images along a specified trajectory.
- Qualitative results: SynSin generates trajectory videos with smoother motion and fewer artefacts than the Vox w/ ours baseline.The comparison is reported in the supplementary video.
- Depth and point-cloud visualisations: Additional RealEstate10K visualisations show depth predictions, feature point clouds, and point clouds rotated by −45° and 0°.The visualised point clouds contain learned features rather than RGB values.
B. Additional architectural details
The supplementary material specifies the ResNet-based spatial feature and refinement networks, the UNet depth regressor, and the perceptual-loss choice used in SynSin.
- Network components: The spatial feature and refinement networks are composed of ResNet blocks.The blocks follow the design used in [3], with configurations for changing feature resolution.
- Network components: The spatial feature network uses the setup in Fig. 16(a), while the refinement network uses Fig. 16(b).These networks are formed by stacking the specified ResNet blocks.
- Network components: The depth regressor uses a UNet architecture.Its architecture is illustrated in Fig. 17.
- Training objective: The perceptual loss follows the formulation used in [44].The supplementary text provides additional details about this loss.
C. Additional details on baselines
The supplementary material describes baseline architectures, viewpoint conditioning, voxel occupancy processing, and training choices used for comparisons with SynSin.
- Baseline architectures: The Im to im baseline is modified to accept 12 viewpoint values because the datasets include three-axis rotations and translational motion.The viewpoint change T is passed to the angle encoder instead of using only the four values in [75].
- Voxel baselines: Vox w/ unet represents 3D shape with voxels, predicts voxel occupancy, reweights projected features, and decodes the new-view scene.Its original setup trains one model per instance, while the reimplementation trains from a single image of a scene.
- Voxel baselines: The Vox w/ unet reimplementation uses UNet components and SynSin’s discriminators and photometric losses for a fair discriminator comparison.Its occupancy network is a 3D UNet with softmax normalisation over depth.
- Voxel baselines: Vox w/ ours replaces the UNet spatial feature and refinement networks with ResNet-block sequences and uses larger capacity.The spatial feature network downsamples features, while the refinement network upsamples projected features.
- Training sensitivity: The Vox w/ ours results are highly dependent on the learning rate.The baseline also involved experiments with alternative ResNet-block sequences and feature resolutions.
- Depth baseline: 3DView is a depth-prediction baseline whose predicted depth has a scale ambiguity.Baseline depth ranges are specified separately for the evaluated datasets.
D. Additional information about datasets
The supplementary material specifies dataset-specific minimum and maximum depth values for Matterport3D, RealEstate10K, and KITTI.
- Depth ranges: Matterport3D uses a minimum depth of 0.1 and a maximum depth of 10.
- Depth ranges: RealEstate10K uses a minimum depth of 1 and a maximum depth of 100.
- Depth ranges: KITTI uses a minimum depth of 1 and a maximum depth of 50.
E. A description of other setups we tried
The authors tested alternative network and differentiable-renderer configurations, finding that some degraded performance or training while accumulation variants produced similar results.
- Replacing the ResNet blocks with a UNet in the spatial feature and refinement networks led to much worse results and harder training.
- Using a larger renderer radius of r = 8 increased training time and produced worse results.
- Weighted-sum accumulation with or without normalisation gave similar results, but removing normalisation made training noisier.