PaperlayerView exact revision on arXiv

Source-linked AI summary

Splatter Image: Ultra-Fast Single-View 3D Reconstruction

Stanislaw Szymanowicz, Christian Rupprecht, Andrea Vedaldi

arXiv:2312.13150v2cs.CV

TL;DR

Monocular 3D reconstruction needs to infer complete object geometry and appearance from limited visual input efficiently. Splatter Image uses a feed-forward image-to-image network that predicts one 3D Gaussian per pixel, achieving fast reconstruction and rendering with strong benchmark performance. It also extends to multiple views through cross-view attention.

  • Problem

    Monocular reconstruction must produce a complete 3D object representation from a single image, despite the input showing only one side.

  • Method

    Splatter Image predicts one colored 3D Gaussian per input pixel with a 2D image-to-image network and can fuse multiple views using cross-view attention.

  • Results

    Splatter Image achieves state-of-the-art reconstruction performance across synthetic and real benchmarks while training and evaluating much faster than many alternatives.

  • Takeaways & Limitations

    Fast inference and Gaussian Splatting rendering allow rapid training and evaluation without requiring absolute or canonical camera poses at test time.

  • Takeaways & Limitations

    Efficiency comparisons with NeRFDiff and FE-NVS are only indicative because hardware and experimental conditions could not be perfectly controlled.

Abstract

from arXiv · show

We introduce the \method, an ultra-efficient approach for monocular 3D object reconstruction. Splatter Image is based on Gaussian Splatting, which allows fast and high-quality reconstruction of 3D scenes from multiple images. We apply Gaussian Splatting to monocular reconstruction by learning a neural network that, at test time, performs reconstruction in a feed-forward manner, at 38 FPS. Our main innovation is the surprisingly straightforward design of this network, which, using 2D operators, maps the input image to one 3D Gaussian per pixel. The resulting set of Gaussians thus has the form an image, the Splatter Image. We further extend the method take several images as input via cross-view attention. Owning to the speed of the renderer (588 FPS), we use a single GPU for training while generating entire images at each iteration to optimize perceptual metrics like LPIPS. On several synthetic, real, multi-category and large-scale benchmark datasets, we achieve better results in terms of PSNR, LPIPS, and other metrics while training and evaluating much faster than prior works. Code, models, demo and more results are available at https://szymanowiczs.github.io/splatter-image.

1. Introduction

Splatter Image enables fast monocular 3D reconstruction by predicting one 3D Gaussian per input pixel with an image-to-image network. Its efficient representation supports 360° reconstruction, multi-view extension, and strong quality-speed trade-offs across benchmarks.

  • Method: Splatter Image predicts one 3D Gaussian per input pixel, storing opacity, position, shape, and colour in a 2D image-like representation.The Gaussians mostly lie along camera rays but may also be placed off-ray, enabling 360° object representation.
  • Method: An image-to-image network with efficient 2D operators replaces more expensive 3D processing, while a U-Net captures small image details.This design reduces reconstruction to learning an image-to-image mapping and supports higher-quality reconstructions.
  • Efficiency: The Gaussian representation accelerates rendering and training enough for complete-view perceptual optimization and single-GPU training on standard benchmarks.Large datasets such as Objaverse require two GPUs, while alternative methods may require dozens or hundreds.
  • Multi-view extension: The method extends to several input views by registering per-view Gaussian mixtures to a common frame and connecting views with cross-view attention.This extension combines the predicted mixtures through their union.
  • Reconstruction capability: 360° reconstructions emerge from training priors even though the network observes only one side of the object.Different Gaussians within a 2D neighbourhood can encode different parts of the 3D object.
  • Results: 38 FPS inference and state-of-the-art reconstruction performance are reported across synthetic, real, multi-category, and large-scale benchmarks.The method is described as operating at 38 FPS on a standard GPU while achieving reconstruction quality and speed advantages across multiple benchmarks.

2. Related work

Prior approaches trade rendering speed, resolution scalability, training cost, or geometric supervision against reconstruction quality. Splatter Image instead predicts 3D Gaussians feed-forward from as few as one image, producing fast reconstruction and rendering without inpainting networks.

  • Implicit representations: Implicit MLP-based representations can render a single 128 × 128 image in up to 2s.NeRF-based methods have dominated learning-based few-view reconstruction but are described as notoriously slow to render.
  • Explicit representations: Voxel-grid methods provide significant speed-ups but scale poorly with resolution and assume known absolute viewpoints for object images.These limitations arise from their voxel-based representation and camera-pose requirement.
  • Triplane representations: Triplane reconstructors scale to datasets such as Objaverse, but training can require hundreds of GPUs for multiple days.They offer a compromise between rendering speed and memory consumption while remaining slower than explicit representations.
  • Splatter Image: Splatter Image predicts a mixture of 3D Gaussians feed-forward, trains with 1-2 GPUs, renders in real time, and achieves state-of-the-art image quality across multiple metrics.The comparison spans multiple standard single-view reconstruction benchmarks, including single- and multi-category ShapeNet.
  • Scope: The method primarily targets object-centric reconstruction and extends to fuse multiple views, rather than generalizing to unseen scenes.It is presented as complementary to generative few-view methods because it supports few-view 3D reconstruction.
  • Point-cloud methods: Point-cloud approaches can require ground-truth 3D point clouds, limiting their applicability.Point clouds have also been used as intermediate representations for reconstruction.
  • Gaussian representations: Unlike prior Gaussian Splatting systems requiring many images per scene, Splatter Image predicts 3D Gaussians from as few as one image.It outputs a full 360° reconstruction without 2D or 3D inpainting networks.

3. Method

Splatter Image represents monocularly reconstructed objects as one predicted 3D Gaussian per input pixel, enabling efficient feed-forward rendering and training. The method extends to multiple views by warping and fusing per-view Gaussian mixtures, with view-dependent colour supported through spherical harmonics.

  • Gaussian Splatting: Gaussian Splatting represents radiance fields with a mixture of colored 3D Gaussians rendered differentiably from a viewpoint.Each Gaussian has opacity, center, covariance, and colour parameters.
  • The Splatter Image: Splatter Image learns an inverse renderer that maps an input image to a Gaussian mixture, storing one Gaussian’s parameters per pixel.The image-to-image network outputs opacity, position, shape, and colour for each pixel-associated Gaussian.
  • The Splatter Image: Each Gaussian location is formed from a pixel ray depth d and a predicted 3D offset, while the network also predicts opacity, shape, and colour.The predicted depth and offset determine the Gaussian mean; opacity can be set to zero to remove unnecessary points.
  • Learning formulation: The network uses efficient 2D image operators and can train from source-target-viewpoint triplets using rendered target-view reconstruction losses.Rendering complete images enables image-level losses such as LPIPS in addition to decomposable losses.
  • Extension to multiple input viewpoints: For multiple input views, Splatter Image predicts one Gaussian mixture per view, warps each mixture into a reference frame, and takes their union.Gaussian means and covariances are transformed by the relative camera rotation and translation.
  • View-dependent colour: View-dependent colours are represented with spherical harmonics, whose coefficients transform under camera rotations; the implementation considers orders L = 0 and L = 1.Order 0 gives Lambertian colour, while order 1 adds three directional components.
  • Neural network architecture: The architecture outputs per-pixel opacity, offset, depth, scale, rotation, and colour parameters, which are transformed into valid Gaussian parameters.A final 1 × 1 convolution produces the parameter channels, followed by nonlinear activations and covariance construction.

4. Experiments

Splatter Image is evaluated across synthetic, real, multi-category, and large-scale benchmarks for reconstruction quality, generalization, multi-view performance, and efficiency. It consistently matches or outperforms slower baselines while requiring substantially less inference and training time.

  • Experimental setup: The evaluation covers six standard single-view benchmarks, multi-category datasets, Objaverse-trained generalization to Google Scanned Objects, two-view reconstruction, and reconstruction speed.Experiments use ShapeNet-SRN, CO3D, multi-category ShapeNet, Objaverse-LVIS, Google Scanned Objects, and ShapeNet-SRN Cars for two-view evaluation.
  • Single-view reconstruction: Splatter Image achieves state-of-the-art quality on all ShapeNet-SRN Car metrics and two Chair metrics while operating in camera view-space.The method uses relative camera poses rather than requiring absolute or canonical poses.
  • Single-view reconstruction: On CO3D, Splatter Image outperforms PixelNeRF across all metrics, produces sharper reconstructions, and is 1,000× faster.The comparison concerns Hydrants and Teddybears.
  • Large-scale and open-world evaluation: On Google Scanned Objects, Splatter Image outperforms OpenLRM quantitatively and remains qualitatively comparable despite training for only 7 GPU days.OpenLRM is trained using hundreds of GPUs for several days, while Splatter Image also performs well on Internet images after preprocessing.
  • Two-view reconstruction: The method supports two-view reconstruction with relative camera poses and improves over PixelNeRF, SRN, CodeNeRF, and FE-NVS on real and synthetic data.Other listed methods rely on absolute camera poses, whereas Splatter Image and PixelNeRF can use relative poses.
  • Efficiency: At training time, Splatter Image is 246× faster than implicit methods and 1.5× faster than Viewset Diffusion under the reported four-view forward-time protocol.Models reach state-of-the-art quality on one A6000 GPU in 7 days; Objaverse models train on two A6000 GPUs in 3.5 days.

5. Conclusion

Splatter Image is a simple single- or few-view 3D reconstruction method that combines an image-to-image network with Gaussian Splatting. It achieves fast training and evaluation with state-of-the-art reconstruction performance across synthetic and real benchmarks.

  • Splatter Image predicts a pseudo-image containing one colored 3D Gaussian per input pixel.It uses an off-the-shelf 2D image-to-image network as its reconstructor.
  • Fast inference and Gaussian Splatting rendering allow rapid training and evaluation on synthetic and real benchmarks.
  • Splatter Image achieves state-of-the-art reconstruction performance without requiring absolute or canonical camera poses at test time.
  • The method is simple to implement and can be trained and tested much faster than many alternatives.

A. Additional results

Additional experiments compare Splatter Image qualitatively with PixelNeRF, VisionNeRF, and OpenLRM, while an ablation evaluates components of the multi-view model. The ablation finds that all examined components contribute to final performance.

  • Additional qualitative results: Figures compare Splatter Image with PixelNeRF and VisionNeRF on ShapeNet-SRN Cars and Chairs, and with PixelNeRF on CO3D Hydrants and Teddybears.
  • Multi-view model ablation: Table 8 removes multi-view attention blocks, camera embedding, and warping individually to assess their importance.
  • Multi-view model ablation: Removing any of the multi-view attention blocks, camera embedding, or warping component reduces the final multi-view model performance.

B. Data details

The data details describe dataset-specific views, preprocessing, evaluation frames, and camera handling for ShapeNet-SRN, CO3D, NMR, and related reconstruction experiments.

  • ShapeNet-SRN: ShapeNet-SRN uses 128 × 128 images, provided intrinsics and poses, relative camera poses, view 64 for single-view conditioning, and views 64 and 128 for two-view conditioning.All other available views are used as target views.
  • CO3D: CO3D evaluation uses the first frame as input and subsequent frames as targets, retaining Hydrant and Teddybear sequences with valid foreground masks at probability p > 0.8.This yields 49 Hydrant and 93 Teddy- sequences in practice.
  • CO3D preprocessing: CO3D preprocessing takes the largest center crop around the principal point, resizes to 128 × 128, adjusts focal length, and removes backgrounds.This is the only preprocessing applied; CO3D point clouds are already normalized to zero mean and unit variance.
  • Camera-distance handling: Estimating object-camera distance is difficult because focal lengths, cropping, camera trajectories, and viewing angles vary across CO3D sequences.
  • Camera-distance handling: For Splatter Image, znear and zfar are set around the ground-truth object distance using znear = zgt − w and zfar = zgt + w, with w = 2.0.
  • NMR: NMR uses 64 × 64 renders from fixed-elevation cameras with the same source and target views as prior work.

B.4. Objaverse and GSO data details.

Objaverse and Google Scanned Objects experiments use rendered, resized data with camera-distance and focal-length adjustments. Qualitative comparisons report faster rendering alongside reconstruction quality relative to established methods.

  • Objaverse data: Objaverse-derived data uses 512 × 512 renders, focal length 560px, randomly sampled camera distances, and resizing to 128 × 128 with focal-length adjustment.
  • Camera scaling: Ground-truth camera positions are rescaled during training and testing to account for object-camera distance normalization.
  • ShapeNet-SRN comparison: On ShapeNet-SRN, Splatter Image reconstructions are better than PixelNeRF and more or equally accurate than VisionNeRF while rendering 3 orders of magnitude faster.Rendering speed is reported in frames per second beneath each method name.
  • CO3D comparison: On CO3D, Splatter Image reconstructions are sharper than PixelNeRF while rendering 3 orders of magnitude faster.Rendering speed is reported in frames per second beneath each method name.
  • Google Scanned Objects comparison: On Google Scanned Objects, Splatter Image reconstructions are comparable in quality to OpenLRM while requiring ×50 less training resources.

C.1. Splatter Image training.

Training uses reconstruction losses and dataset-specific settings to stabilize Gaussian prediction across standard and large-scale benchmarks. Large-scale Objaverse training uses mixed precision on two GPUs for around 3.5 days.

  • Standard training: 800,000 iterations train the model on three unseen views plus the conditioning view using L2 reconstruction loss.Single-class models use Adam with learning rate 5 × 10^-5 and batch size 8; multi-class ShapeNet uses batch size 32.
  • Large dataset training: 350,000 iterations with α = 0 precede 40,000 iterations with α = 0.338 for Objaverse training.The training uses mixed precision and effective batch size 32.
  • Regularizers: CO3D training regularizes excessively large and small Gaussians for numerical stability.Large Gaussians are constrained when their activated scale exceeds sbig = 20, while small Gaussians use a threshold ˆssmall = −5.
  • Ablations: Ablation models use 100k iterations with L2 followed by 25k iterations combining L2 and LLPIPS with α = 0.1.The shorter schedule is used because of computational costs.
  • Baseline setup: The CO3D baseline changes opacity activation from ReLU to Softplus with β = 3.0 for improved training stability.The baseline uses the official PixelNeRF implementation and the same preprocessed data as Splatter Image.
  • Resource reporting: Table 9 reports GPU type, GPU memory capacity, GPU count, and training duration for each method.These fields define the resource comparison presented in the training-resources table.

D. Training resource estimate

The resource estimate compares training hardware, memory, GPU count, and duration across methods, showing that Splatter Image has substantially lower training cost than several alternatives.

  • Large-scale comparison: ×50 cheaper training than LRM is reported for Splatter Image on large datasets such as Objaverse.The comparison uses GPU type, memory capacity, GPU count, and training days; estimates are marked where applicable.
  • Method comparison: Splatter Image is significantly cheaper than VisionNeRF and NeRFDiff, with costs similar to Viewset Diffusion and PixelNeRF.The method also achieves better performance than those similarly costly methods and does not require absolute camera poses.

E. Covariance warping implementation

Covariance warping transforms 3D Gaussians between reference frames by rotating their covariance orientation. The implementation performs this rotation in quaternion space using relative-rotation composition.

  • Covariance transformation: The covariance is warped between views as ˜Σ = RΣR⊤, where R is the relative rotation matrix.This applies the reference-frame transformation to the Gaussian covariance.
  • Covariance parameterization: Each covariance is parameterized by a three-dimensional scale and quaternion rotation as Σ = RqSRq⊤, with S = diag(exp(ˆs))2.The scale and orientation determine the Gaussian covariance before view warping.
  • Quaternion implementation: Quaternion composition implements the rotation as ˜q = pq, where p is the quaternion representation of the relative rotation.The predicted quaternion q is composed with p instead of explicitly rotating the covariance matrix.
Loading 2312.13150v2…