PixelSynth: Generating a 3D-Consistent Experience from a Single Image

1. Introduction

• Large view changes require extrapolating beyond input pixels, while multiple outputs must stay consistent and capture perspective changes.The paper frames these as central challenges for single-image scene synthesis.Source: §1 · p. 1
• Prior 3D-aware methods interpolate among multiple views but do not extrapolate, while single-image methods handle only small rotations and translations [29] [28] [43] [44] [48] [50] [15] [47].Autoregressive completion extrapolates beyond image boundaries but does not explicitly model a consistent 3D world [30] [35] [49].Source: §1 · p. 1
• PixelSynth autoregressively outpaints extremal support views, lifts them into a shared 3D representation, and re-renders intermediate views for consistency.The approach adapts VQVAE2 [35] and Locally Masked Convolutions [18] for image-specific outpainting orderings, and uses techniques from SynSin to obtain 3D.Source: §1 · pp. 1–2
• Users select PixelSynth 73% of the time versus the best multiple-SynSin-ablation variant, and support-set re-rendering is more consistent than alternate accumulation on an average of 72% of images.The evaluation covers Matterport 3D+Habitat [3] [41] and RealEstate10K, with view changes 6× larger than.Source: §1 · p. 2

2. Related Work

• Multi-view methods infer 3D from several images and represent scenes as depth images, multi-plane images, point clouds, voxels, meshes, or radiance fields; PixelSynth uses point clouds [1] [28] [37] [47] [25] [44] [15] [20] [26] [29] [50] [60].Source: §2 · p. 2
• Single-image view-synthesis methods have been limited to small angle changes [45] [47], and SynSin treats outpainting like inpainting; PixelSynth instead uses completion-based large-region outpainting.Source: §2 · p. 2
• Concurrent approaches differ in scene focus or representation: PixelSynth targets indoor scenes and explicit outpainting, unlike nature-scene trajectories [23], mesh-based interpolation [15], and image generation without a learned 3D representation [38].Source: §2 · p. 2
• Inpainting struggles with the large missing regions of large-angle views, while existing outpainting methods are not flexible to arbitrary missing regions [24] [46].Source: §2 · p. 2
• Autoregressive models extrapolate pixels sequentially but do not by themselves produce 3D-consistent outpainting; PixelSynth adapts masked convolutions for per-image regions and selects among completions for realism [18] [35] [33].Source: §2 · p. 2

3. Approach

• The approach addresses high-quality extrapolation under large transformations and 3D consistency by combining deep autoregressive modeling with 3D modeling.Source: §3 · pp. 2–3

3.1. 3D and Synthesis Modules

• The Depth Module lifts images to colored point clouds, the Projector renders point clouds at a pose, and the Outpainter fills image-specific missing regions in a VQ-VAE latent space.The Refinement Module corrects local errors and inconsistencies associated with imperfect depth; autoregressive ordering fills pixels from near the visible region outward.Source: §3.1 · p. 3

3.2. Inference

• Inference first projects the input point cloud into an extremal support view, outpaints and refines it, then lifts the result into a point cloud; intermediate poses are rendered from the combined input and support clouds and refined.This support-view-first process can be reused for multiple viewing directions.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. Training and Implementation Details

• Training is staged because the Outpainter requires ground-truth input and output, breaking gradient flow; the Depth and Refinement Modules are trained separately from the Outpainter components.Source: §3.3 · p. 3
• The Depth and Refinement Modules use curriculum training from small to larger view changes, while diverse outpainting samples are selected using classifier entropy and discriminator losses.The paper also reduces autoregressive-model depth by 60% and width by 50%, using 32 × 32 completions with refinement for detail.Source: §3.3 · p. 4

4.1. Experimental setup

• The evaluation uses Matterport3D and RealEstate10K, with Matterport rotations extended to 120° and RealEstate10K pairs averaging about 30° apart.RealEstate10K pairs are selected to be 20°–60° apart and no more than 1 m away.Source: §4.1 · p. 5

4.2. Evaluating Quality

• On Matterport, explicit outpainting outperforms baselines including SynSin across metrics; larger-rotation training and sequential generation narrow but do not close the gap.The table reports A/B preference and FID; the paper also reports a larger human-judgment gap on RealEstate10K.Source: Table 1 · p. 6, §4.2 · p. 6
• On RealEstate10K, SynSin-6X can score well on FID despite repeated or mean-colored outputs, illustrating that distribution-level scores can differ from human judgments.The paper attributes this partly to frequent doorway views whose neighboring walls have bland, repeated colors.Source: §4.2 · p. 6
• Human A/B judgments are the primary quality measure because automated metrics, especially PSNR, are poor measures of extrapolation.Appearance Flow often produces gray RealEstate10K images and loses to the proposed method 98.1% of the time in A/B testing, despite competitive PSNR.Source: §4.1 · p. 5, §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Evaluating Consistency

• For consistency, the full method improves on sequential accumulation and independent generation, reaching Perc Sim/PSNR of 0.419/18.6 versus 0.456/17.9 and 0.606/13.6, respectively.The controlled comparison uses warped RealEstate10K views; Table 3 reports human pairwise consistency judgments.Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4. Ablations

• VQ-VAE-space outpainting is superior to RGB autoregression, while a single completion can suffice on Matterport's more homogeneously lit scanned environments.The paper notes that multiple completions are less helpful empirically for RGB models and that one completion is more efficient but often less effective overall.Source: §4.4 · p. 8, Table 4 · p. 8
• Combining classifier entropy with discriminator selection yields sensible completions with realistic texture, while either criterion alone lacks one of those qualities.The combined selection performs best on RealEstate10K and is also helpful on Matterport.Source: §4.4 · p. 8, Table 4 · p. 8
• The point-cloud-based method can generalize to synthesis from two input images.Source: Figure 9 · p. 8

• The authors describe synthesizing a rich, full world from one image as a steep new challenge.Source: §5 · p. 8
• They argue that 3D awareness is important for good results and generality, even when the system uses a single image.Source: §5 · p. 8
• The system can be applied to two views by ingesting two point clouds.Source: §5 · p. 8

• Videos show the proposed method producing high-quality, consistent scenes, unlike SynSin – 6X and No 3D Accumulation, respectively.The authors say these comparisons are best seen in video.Source: A. Appendix · p. 9
• The supplemental PDF contains detailed model architectures, implementation details, and experimental setup.Source: A. Appendix · p. 9
• The supplemental PDF also includes additional results and A/B testing details.Source: A. Appendix · p. 9

• A depth module predicts a depthmap, and a projector renders the resulting colored point cloud at a novel pose to identify pixels needing outpainting.The depth module maps a 256 × 256 × 3 image to a 256 × 256 × 1 depthmap; projection produces a 256 × 256 × 3 image.Source: §A.1 · p. 9
• Reprojected border pixels at the visible-region edge are treated as background to be outpainted, preventing a mean-color border from guiding completion.Rendered pixels just outside the point-cloud silhouette can be nonzero because of interpolation.Source: §A.1 · p. 9
• The outpainter completes missing regions autoregressively in an autoencoder’s latent space, using an image-specific ordering because missing regions vary by image and pose.The lightweight VQVAE2 adaptation maps a 256×256×3 image to a 32×32×1 quantized embedding space.Source: §A.1 · p. 9
• The completion order starts with background pixels nearest the foreground’s center of mass, then adds adjacent pixels, breaking distance ties with an outward spiral.Adjacent-pixel ordering preserves as much known data as possible during prediction.Source: §A.1 · p. 10
• A refinement module takes the full outpainted 256 × 256 × 3 image and produces a refined output of the same size using adversarial training.The generator follows BigGAN [2] and SynSin.Source: §A.1 · p. 10

• For each image, the system generates 50 completions and selects the best using average ranks from discriminator loss and MIT Places 365 classifier entropy.Higher discriminator loss and lower entropy receive the preferred ranks.Source: §A.2 · p. 10
• Sampling temperature is 0.5 for Matterport and RealEstate10K’s 1-completion model, and 0.7 for RealEstate10K’s 50-completion model.The authors report that 0.5 balances realism and detail, while the 50-completion model can select sensible outputs from higher-temperature samples.Source: §A.2 · p. 10
• Eight support views cover up, down, left, right, and the four diagonals; each new view reprojects existing 3D information and outpaints only as needed.Source: §A.2 · p. 10

A.3. Experimental Setup

• Matterport pairs are sampled using embodied-agent navigation, with angle changes drawn up to 120° in each direction from a dataset of 3.6k pairs.Source: §A.3 · p. 10
• RealEstate10K pairs must have at least 20° of angle change, with translations limited to 1.0 meters and angles to 60° to reduce extreme cross-room changes.Source: §A.3 · pp. 10–11
• The filtered RealEstate10K test set contains 3,600 pairs selected from over 3.5 million possibilities across over 2.4k video clips, and all evaluation pairs are cached.Source: §A.3 · p. 11

A.4. Additional Results

• Video results provide the best simultaneous display of quality and consistency; the proposed method tends to produce high-quality, consistent scenes, unlike SynSin - 6X and No 3D Accumulation.Additional frame-level images appear in Figures 10 and 11.Source: §A.4 · p. 11
• Table 5 expands image-quality results using PSNR and Perceptual Similarity, which the authors describe as poor measures for extrapolation; ground-truth depth is available for Matterport but not RealEstate10K.Source: §A.4 · p. 11
• Appearance Flow is competitive with other methods on PSNR but loses to the proposed method 98% of the time in A/B testing.The authors caution that PSNR correlates poorly with perceived quality when multiple completions are possible.Source: §A.4 · p. 11
• The primary limitation is producing both consistent and detailed content during large-view-change outpainting, with a trade-off between sample diversity and inaccurate content.On Matterport, lower sampling temperature can reduce inconsistent completions but also reduce detail.Source: §A.4 · p. 11

A.5. Additional and A/B Testing Details

• Human A/B testing is the primary measure of success, using worker instructions, a comprehension qualifier, and gold-standard sentinel labels.Source: §A.5 · p. 11
• Table 5 reports full PSNR and Perceptual Similarity results for reference, while Figures 10 and 11 are labeled as additional results.Source: Table 5 · p. 11, Figure 10 · p. 13, Figure 11 · p. 14
• For quality comparisons, workers see the input image reprojected into a novel view and choose the more sensible final image; they do not compare against ground truth.At least two workers must agree, and ties are disallowed; Figure 12 shares the instructions.Source: §A.5 · p. 11, Figure 12 · p. 15
• For consistency judgments, methods generate two images at successive half-sized rotation and translation changes, which workers compare without input or ground-truth images.Large fixed rotations and restricted directions help make consistency differences easier to judge; Figure 13 gives the instructions.Source: §A.5 · pp. 11–12, Figure 13 · p. 16
• Homography validation uses pure rotations and evaluates overlapping regions with PSNR and warped images with non-overlapping regions masked for Perceptual Similarity.Scores are averaged across both directions between intermediate and extreme images.Source: §A.5 · p. 12