PaperlayerView exact revision on arXiv

Source-linked AI summary

EscherNet: A Generative Model for Scalable View Synthesis

Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, Andrew J. Davison

arXiv:2402.03908v2cs.CV

TL;DR

View synthesis needs a general 3D representation that can scale beyond scene-specific optimization and flexibly use sparse reference views. EscherNet uses conditional diffusion with camera positional encoding, and the authors report stronger generation quality than 3D diffusion models alongside plausible synthesis from very limited views.

  • Problem

    The paper asks how to learn a general 3D representation for scalable view synthesis without scene-specific coordinate dependence or a fixed single-reference formulation.

    Source: §1 · p. 2

  • Method

    EscherNet adapts a 2D diffusion model with camera positional encoding so attention models relative camera transformations across reference and target views.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    EscherNet outperforms 3D diffusion models in generation quality and produces plausible views from very limited inputs, unlike scene-specific methods that often struggle.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its coordinate-independent design avoids ground-truth geometry and expensive 3D operations, making it easier to scale with everyday posed 2D image data.

    Source: §1 · p. 2

    EscherNet's current capability is constrained to a 3 DoF setting by its training dataset, which may not match real-world views spanning SE(3).

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce EscherNet, a multi-view conditioned diffusion model for view synthesis. EscherNet learns implicit and generative 3D representations coupled with a specialised camera positional encoding, allowing precise and continuous relative control of the camera transformation between an arbitrary number of reference and target views. EscherNet offers exceptional generality, flexibility, and scalability in view synthesis -- it can generate more than 100 consistent target views simultaneously on a single consumer-grade GPU, despite being trained with a fixed number of 3 reference views to 3 target views. As a result, EscherNet not only addresses zero-shot novel view synthesis, but also naturally unifies single- and multi-image 3D reconstruction, combining these diverse tasks into a single, cohesive framework. Our extensive experiments demonstrate that EscherNet achieves state-of-the-art performance in multiple benchmarks, even when compared to methods specifically tailored for each individual problem. This remarkable versatility opens up new directions for designing scalable neural architectures for 3D vision. Project page: https://kxhit.github.io/EscherNet.

1. Introduction

EscherNet addresses the challenge of scalable view synthesis by learning coordinate-independent implicit 3D representations and flexibly conditioning generation on multiple views. The authors report strong generation quality and plausible synthesis from very limited views.

1. Introduction

  • The paper asks how to learn a general 3D representation that supports scalable view synthesis.It frames the problem around re-rendering scenes from arbitrary viewpoints using reference views.Source: §1 · pp. 1–2
  • Unlike scene-specific methods, EscherNet needs neither ground-truth 3D geometry nor a specific coordinate system, and avoids expensive 3D operations.The authors say this makes it easier to scale with everyday posed 2D image data.Source: §1 · p. 2
  • EscherNet uses an image-to-image conditional diffusion model and camera positional encoding to represent relative camera transformations for view-consistent generation.Its attention captures reference-to-target and target-to-target relationships.Source: §1 · p. 2
  • Despite training on a fixed three-reference-to-three-target setup, EscherNet can generate any number of targets at arbitrary poses from any number of references.Generation quality improves with more reference views.Source: §1 · p. 2
  • EscherNet outperforms 3D diffusion models in generation quality and generates plausible views from very limited inputs, unlike scene-specific methods that often struggle.The comparison names InstantNGP and Gaussian Splatting [18] as examples of scene-specific methods.Source: §1 · p. 2

2. Related Work

Earlier neural 3D representations often depend on ground-truth geometry or coordinate-based scene optimization, while 3D diffusion approaches can be costly or limited in view flexibility. EscherNet instead learns from posed images and supports arbitrary target views and poses.

2. Related Work

  • Explicit voxel and point-cloud representations and implicit signed-distance or occupancy methods rely heavily on ground-truth 3D geometry, limiting them to small-scale synthetic data.The cited approaches include voxels, point clouds [40] [41], signed distance functions [35], and occupancies [28] [37].Source: §2 · p. 2
  • Unlike NeRF's coordinate-based radiance fields, EscherNet learns interactions among multi-view posed images without a coordinate-system-specific representation.NeRF [29] optimizes MLPs to encode 5D radiance fields.Source: §2 · p. 2
  • PixelNeRF [59] learns scene priors across multiple scenes but remains computationally demanding because it uses volumetric rendering.EscherNet encodes scenes through image space to learn more general scene priors from large-scale datasets.Source: §2 · p. 3
  • Score-distillation methods such as DreamFusion [39] and Magic3D [19] can require up to an hour of iterative optimization, while direct 3D generative models use costly 3D operations.Image-to-3D methods can also produce unrealistic results, including the multi-face Janus problem.Source: §2 · p. 3
  • EscherNet generates an unrestricted number of target views at arbitrary poses, unlike prior multi-view diffusion models limited to fixed target counts and camera poses.Zero-1-to-3 [21] generates only one target view from one reference view.Source: §2 · p. 3

3. EscherNet

EscherNet formulates view synthesis as conditional generation from reference images and camera poses to target images at requested poses. Its architecture reuses a 2D diffusion model, while camera positional encoding lets attention model relative camera transformations.

3. EscherNet

  • Target views are conditionally generated from reference views, their poses, and requested target poses; the numbers of reference and target views can vary arbitrarily.Each target view depends on its relative camera transformation to the reference views.Source: §3 · p. 3

3.1. Architecture Design

  • EscherNet adapts Stable Diffusion's U-Net self-attention across target views and cross-attention between reference and target views to support both consistency paths.A lightweight vision encoder represents reference views with high- and low-level signals; ConvNeXtv2-Tiny compresses them into conditioning tokens.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2. Camera Positional Encoding (CaPE)

  • Camera Positional Encoding (CaPE) applies pose information to tokens so attention depends on relative camera transformations, not a particular coordinate system.The design includes 4 DoF object-centric and 6 DoF generic pose encodings.Source: §3.2 · pp. 4–5
  • Camera positions differ from language-token positions: rotations are cyclic and bounded, translations are continuous and unbounded, and view relationships depend on relative transformations.EscherNet's encoding is designed to accommodate these camera-position properties.Source: §3.2 · pp. 4–5

4 DoF CaPE

The 4 DoF CaPE encodes object-centric camera pose components so dot-product attention can represent relative rotations and view-radius differences. Its radius encoding uses a logarithmic normalization before mapping distance into a rotation range.

4 DoF CaPE

  • The 4 DoF pose uses azimuth, elevation, look-at orientation, and camera distance as disentangled components.The pose is represented in a spherical coordinate system.Source: §4 · p. 5
  • The encoding conditions dot-product similarities on relative angle differences and relative view-radius ratios.The stated conditions express relative rotation and scale differences through the encoded feature dot product.Source: §4 · p. 5
  • A block-diagonal rotation matrix applies the pose encoding to token features, combining the angle and radius encodings in a RoPE-inspired formulation.The formulation uses log-radius differences, which are unchanged by multiplying both radii by the same positive scalar.Source: §4 · p. 5
  • Log-normalized camera radius is mapped into [0, π] for its rotation encoding.The passage says this constrains the radius encoding to the rotation range.Source: §4 · p. 5

6 DoF CaPE

For 6 DoF camera poses, the position components are entangled, so they cannot be reformulated as a multidimensional position as in 4 DoF CaPE. EscherNet extends the encoding with a block-diagonal pose transformation and a key/query equivalence.

6 DoF CaPE

  • Entangled components in 6 DoF camera poses prevent reformulating position as a multidimensional position, unlike 4 DoF camera poses.The section denotes 6 DoF poses as P = [R t] ∈ SE(3).Source: §6 · p. 5
  • 6 DoF CaPE lifts P ∈ R^4×4 to ϕ(P) ∈ R^d×d with repeated diagonal blocks, then encodes π(v, P) = ϕ(P)v.The construction uses a key/query equivalence for the encoding.Source: §6 · p. 5
  • The encoding dimension d must be divisible by dim(P) = 4, and translation t is rescaled within a unit range for efficient training.The implementation reshapes v to match Ψ; the stated dimension is 4 for 6 DoF CaPE.Source: §6 · p. 5

4. Experiments

Across novel view synthesis and 3D reconstruction, EscherNet performs well with sparse references and can generate multiple consistent views. The experiments also use it to turn text-generated images into 3D-consistent views.

4. Experiments

  • Experiments use Objaverse-1.0's 800K objects and 12 randomized rendered views per object; the main paper reports 4-DoF CaPE, which was visually more consistent on real-world images.Both 4-DoF and 6-DoF CaPE were evaluated, and baselines were re-evaluated from official checkpoints on the same reference views.Source: §4 · p. 5, §4.1 · p. 6

4.1. Results on Novel View Synthesis

  • EscherNet significantly outperforms 3D diffusion baselines on GSO and RTMV, despite Zero-1-to-3-XL using 10× more training data and RealFusion requiring iterative score distillation.EscherNet can jointly generate multiple consistent target views with precise camera control.Source: §4.1 · p. 6, Table 1 · p. 6
  • With fewer than five references, EscherNet outperforms InstantNGP and Gaussian Splatting without scene-specific optimization, while both methods improve as reference views increase.Figure 4 reports mean PSNR averaged over drum-scene test views; scene-specific methods improve substantially with more than 10 references.Source: Table 2 · p. 6, §4.1 · p. 6, Figure 4 · p. 6

4.2. Results on 3D Generation

  • For NeuS reconstruction, EscherNet supplies 36 fixed views spanning azimuths from 0° to 360° at elevations of −30°, 0°, and 30°.These views are rendered every 30° in azimuth.Source: §4.2 · p. 7
  • EscherNet improves Chamfer distance over SyncDreamer by approximately 25% with one reference view and 60% with 10 reference views.The paper attributes its reconstruction quality to flexible view counts and comprehensive geometry constraints; SyncDreamer's fixed 30° elevation can weaken lower geometry.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Results on Text-to-3D Generation

  • The two-stage text-to-3D pipeline pairs an off-the-shelf text-to-image model with EscherNet, producing consistent views with appealing textures even for out-of-distribution and counterfactual content.The experiments use MVDream and SDXL as image-generating models.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

5. Conclusions

EscherNet is a multi-view-conditioned diffusion model that learns implicit 3D representations with CaPE for consistent novel-view synthesis from varying numbers of reference views. Its flexibility supports autoregressive generation, but that mode degrades quality, and current capability is constrained to 3 DoF.

5. Conclusions

  • EscherNet combines multi-view-conditioned diffusion with CaPE to learn implicit 3D representations and produce consistent novel views from varying numbers of reference views.Source: §5 · p. 8
  • Autoregressive generation reduces inference time but degrades generation quality, while current capability is constrained to 3 DoF by the training dataset.The authors identify real-world SE(3) view variation as a setting that may not align with this constraint.Source: §5 · p. 8

A. Python Implementation of CaPE

  • The CaPE implementations transform feature vectors using camera poses: the 4 DoF version uses an SE3 matrix, while the 6 DoF version applies pose-dependent rotations and radius scaling.Source: A. Python Implementation of CaPE · p. 12

B. Additional Training Details and Experimental Settings

  • Training uses batches of 3 reference and 3 target views sampled with replacement from 12 views per object, and takes one week on 6 NVIDIA A100 GPUs.Images are 256 × 256, and training runs for 100,000 steps.Source: B. Additional Training Details and… · p. 13

B.1. Evaluation Details

The evaluation covers novel-view synthesis and 3D reconstruction across NeRF Synthetic, GSO, and RTMV settings. EscherNet with 6 DoF CaPE performs comparably to, and sometimes slightly better than, its 4 DoF design.

B.1. Evaluation Details

  • The NeRF Synthetic evaluation covers all 8 scenes and 200 test views per scene, while GSO uses 30 objects with 10 reference and 15 target views each.GSO views use random camera poses and randomly generated environment lighting.Source: §B.1 · p. 13
  • For 3D generation evaluation, methods are compared using 4096 sampled points, with generated meshes aligned to ground truth using reference-view camera poses.Source: §B.1 · p. 13
  • RTMV evaluation uses 10 complex scenes and assigns the first 10 images as references and the next 10 as targets.Source: §B.1 · p. 13

C. Additional Results on 6 DoF CaPE

D. Additional Results on NeRF Synthetic Dataset

The appendix provides additional NeRF Synthetic visualizations and reports novel-view synthesis performance by scene and number of reference views.

D. Additional Results on NeRF Synthetic Dataset

  • Additional visualizations show NeRF Synthetic results for EscherNet trained with 4 DoF CaPE.Source: D. Additional Results on NeRF… · p. 15
  • Table 5 reports average PSNR per scene for NeRF Synthetic novel-view synthesis, conditioned on the number of reference views.Source: Table 5 · p. 15
  • The reported metric is averaged at the scene level rather than presented as a single overall PSNR value.Source: Table 5 · p. 15

E. Additional Results on Text-to-3D

The appendix includes text-to-image-to-3D visualizations and a text-to-3D generation table comparing results for SDXL and MVDream.

E. Additional Results on Text-to-3D

F. Additional Discussions, Limitations and Future Work

The discussion weighs direct and autoregressive view generation, examines how additional views affect synthesis quality, and outlines training-data choices and possible video sources.

Direct v.s. Autoregressive Generation

  • Autoregressive generation is more than 20× faster than direct generation for 200 views, but quality may drift as each view depends on earlier imperfect outputs.Its inference cost changes from quadratic to linear in each self-attention block, and the authors identify SLAM as a suitable scenario.Source: F. Additional Discussions, Limitations… · p. 17

Stochasticity and Consistency in Multi-View Generation

  • On GSO, view quality improves as reference and target counts rise; the tested target views duplicate one pre-selected view, and gains beyond 15 targets are marginal.The authors report that at least 15 target views is the optimal configuration; additional views yield only marginal improvements.Source: Figure 8 · p. 17, F. Additional Discussions, Limitations… · p. 17

Training Data Sampling Strategy

  • Training uses 3 reference and 3 target views per object to balance GPU memory and speed against learning multi-view correspondences; sampling with replacement slightly improves performance.Sampling with replacement allows repeated images among the six training views.Source: F. Additional Discussions, Limitations… · p. 17

Scaling with Multi-view Video

Related papers

Loading 2402.03908v2…