PaperlayerView exact revision on arXiv

Source-linked AI summary

Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention

Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, Wenping Wang, Qifeng Liu, Yike Guo

arXiv:2405.11616v3cs.CV

TL;DR

Multiview diffusion for single-view 3D reconstruction faces camera-prior distortion, high attention costs, and resolution limits. Era3D predicts input focal length and elevation and applies row-wise attention to canonical-camera outputs. It reports superior reconstruction geometry versus baseline methods, while sparse six-view generation remains a boundary for intricate shapes.

  • Problem

    Multiview diffusion methods assume predefined input cameras and use dense attention, creating distortion risks and computation and resolution constraints.

    Source: §1 · p. 2

  • Method

    Era3D predicts input elevation and focal length to condition diffusion and uses row-wise attention for cross-view information exchange in canonical-camera outputs.

    Source: §1 · p. 3

  • Results

    Era3D achieves superior geometry quality in single-view 3D reconstruction compared with baseline methods.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Row-wise attention facilitates training for high-resolution multiview generation by restricting attention to corresponding image rows.

    Source: §3.3 · p. 6

    Generating only six multiview images makes intricate geometries difficult to model.

    Source: §5 · p. 10

Abstract

from arXiv · show

In this paper, we introduce Era3D, a novel multiview diffusion method that generates high-resolution multiview images from a single-view image. Despite significant advancements in multiview generation, existing methods still suffer from camera prior mismatch, inefficacy, and low resolution, resulting in poor-quality multiview images. Specifically, these methods assume that the input images should comply with a predefined camera type, e.g. a perspective camera with a fixed focal length, leading to distorted shapes when the assumption fails. Moreover, the full-image or dense multiview attention they employ leads to an exponential explosion of computational complexity as image resolution increases, resulting in prohibitively expensive training costs. To bridge the gap between assumption and reality, Era3D first proposes a diffusion-based camera prediction module to estimate the focal length and elevation of the input image, which allows our method to generate images without shape distortions. Furthermore, a simple but efficient attention layer, named row-wise attention, is used to enforce epipolar priors in the multiview diffusion, facilitating efficient cross-view information fusion. Consequently, compared with state-of-the-art methods, Era3D generates high-quality multiview images with up to a 512*512 resolution while reducing computation complexity by 12x times. Comprehensive experiments demonstrate that Era3D can reconstruct high-quality and detailed 3D meshes from diverse single-view input images, significantly outperforming baseline multiview diffusion methods. Project page: https://penghtyx.github.io/Era3D/.

1 Introduction

Era3D addresses camera mismatch, computational cost, and resolution limits in multiview diffusion for single-view 3D reconstruction. It predicts input-camera parameters and uses row-wise attention to generate high-resolution multiview images.

1 Introduction

  • Single-view 3D reconstruction is motivated by applications including game design, virtual reality, and robotics.Diffusion models offer a way to generate novel views from an input image for reconstruction.Source: §1 · p. 1
  • Existing methods assume predefined input cameras, risking distortion, while dense attention raises computation and memory costs and often limits resolution to 256×256.A small-focal-length perspective input can violate an orthogonal-camera assumption.Source: §1 · pp. 2–3
  • Era3D uses arbitrary-focal-length and elevation inputs but generates orthogonal views at fixed 0° elevation, predicting input-camera information during denoising.Its regression-and-condition scheme uses UNet features to predict camera information and condition diffusion.Source: §1 · p. 3
  • Row-wise attention uses the canonical camera setting to align cross-view epipolar lines with image rows, reducing the cost of multiview information exchange.This design supports efficient high-resolution generation.Source: §1 · p. 3
  • Era3D achieves state-of-the-art single-view 3D generation performance and scales row-wise attention to 512×512 multiview images.The paper links this resolution to reconstructing more detailed 3D meshes.Source: §1 · p. 4

2 Related Works

Related work covers image-to-3D generation, multiview diffusion, and camera-pose handling. Era3D positions its approach as feed-forward generation with attention designed to reduce computation.

2 Related Works

  • The paper centers feed-forward image-to-3D generation, unlike early per-scene optimization based on Score Distillation Sampling [6] [49] [62] [71] [29].Image-to-3D work also directly predicts representations or generates multiview images for later reconstruction.Source: §2 · p. 4
  • Multiview methods exchange information through correspondence-aware or epipolar attention, tiled views, multiview self-attention, 3D-volume fusion, or video priors.These approaches model 3D generation with a joint multiview probability distribution.Source: §2 · p. 4
  • Earlier camera-pose approaches use fixed focal lengths [32] [55], fixed poses [34] [54], elevation prediction [30], or pixel-wise similarity [20].The paper identifies camera-induced distortion as an unresolved issue for real-world data.Source: §2 · p. 4

3 Methods

Era3D generates canonical-camera multiview images from single-view inputs, estimates input camera parameters to guide denoising, and uses row-wise attention to reduce cross-view computation.

3 Methods

  • Given an input image, Era3D generates canonical-camera multiview images and normal maps, then reconstructs a mesh with neural reconstruction methods such as NeuS [68].The method overview identifies camera canonicalization, camera regression and conditioning, and row-wise attention as its key components.Source: §3 · pp. 4–5, Figure 4 · p. 5

3.1 Camera Canonicalization

  • Era3D generates orthogonal output images at 0° elevation despite varying input focal lengths and viewpoints, using views spaced around the input azimuth.The method calls this output arrangement the Canonical camera setting; it addresses distortion from mismatched input and assumed camera intrinsics.Source: §3.1 · p. 5

3.2 Regression and Condition Scheme

  • The Elevation and Focal length Regression module predicts camera parameters from UNet features during diffusion and conditions denoising on those estimates.It applies global average pooling to a middle-level transformer feature map, regresses elevation and focal length, and feeds their positional encodings to UNet upsampling layers.Source: §3.2 · pp. 5–6

3.3 Row-wise Multiview Attention (RMA)

  • With orthogonal cameras at 0° elevation, epipolar lines align with image rows, so row-wise attention exchanges features along corresponding rows without sampling points.This special epipolar attention avoids dense all-pixel attention and facilitates training at resolutions such as 512×512.Source: §3.3 · p. 6

4 Experiments

Era3D is evaluated on GSO and in-the-wild images, with results covering novel-view synthesis, reconstruction, pose estimation, and attention ablations. The reported comparisons show stronger reconstruction and novel-view results, reduced distortion, and more efficient high-resolution attention.

4.1 Experimental Results

  • Era3D outperforms prior single-view reconstruction methods in novel-view-synthesis quality by a significant margin (Table 1).Source: §4.1 · p. 8, Table 1 · p. 9
  • On GSO, Era3D produces more complete and detailed meshes than baselines, while some alternatives are incomplete, over-smoothed, or distorted.Source: Figure 5 · p. 7, §4.1 · p. 8
  • Era3D outperforms all other methods on reconstruction, with lower Chamfer Distance and higher Volume IoU (Table 1).Source: §4.1 · p. 8, Table 1 · p. 9
  • Compared with Unique3D, Era3D greatly alleviates the severe perspective distortion shown in the Figure 7 comparisons.Source: §4.1 · p. 8, Figure 7 · p. 9

4.2 Accuracy of Estimated Elevations and Focal Lengths

  • On GSO pose tests spanning elevations and focal lengths, Era3D achieves superior error and variance performance against the baselines (Table 2).Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Ablations and Discussions

  • Removing EFReg distorts shapes and yields unreasonable canonical-view outputs; conditioning on predicted elevation and focal length guides undistorted cross-domain generation.Source: §4.3 · p. 9, Figure 8 · p. 9
  • At 512 resolution, row-wise attention uses 30-fold less memory and nearly 100-fold less running time than epipolar attention; with xFormers, it takes 1.86 ms versus 22.9 ms.Source: §4.3 · p. 9, Table 3 · p. 10
  • At 256 resolution, row-wise attention performs comparably to dense attention and surpasses the baseline on Chamfer distance, LPIPS, and PSNR.Source: §4.3 · p. 10, Table 4 · p. 10

5 Limitation and Conclusion

Era3D predicts input-camera focal length and elevation, conditions diffusion on them, and generates views in a canonical camera setting. Row-wise attention replaces dense attention, while the method’s reconstruction has stated geometry and surface limitations.

5 Limitation and Conclusion

  • Generating only six views makes intricate geometries such as thin structures difficult to model, and NeuS cannot reconstruct open-surface meshes.The authors suggest integrating other 3D representations, such as Gaussian splatting, in future work.Source: §5 · p. 10
  • Era3D predicts input focal length and elevation and conditions diffusion on them while generating images in a canonical camera setting.Inputs may have arbitrary camera intrinsics and viewpoints.Source: §5 · p. 10
  • Era3D uses row-wise multiview attention instead of dense attention, significantly reducing computational workloads and facilitating high-resolution cross-view generation; compared with baseline methods, it achieves superior geometry quality in single-view 3D reconstruction.Source: §5 · p. 10
  • Era3D achieves superior geometry quality in single-view 3D reconstruction compared with baseline methods.Source: §5 · p. 10

6 Ethics Statement

The authors identify possible deception by generated 3D models as an ethical risk and call for continued attention to it in 3D generative modeling.

6 Ethics Statement

  • Era3D can generate 3D objects from a single image, with a potential risk that the resulting models could deceive viewers.Source: §6 · p. 10
  • The authors state that this misuse risk is not unique to Era3D but is prevalent in other generative-model methodologies.Source: §6 · p. 10
  • They call for current and future 3D generative-modeling research to address and reassess these considerations consistently.Source: §6 · p. 10

A.1 Implementation Details

The implementation renders perspective input images and orthogonal generated views for training, and uses cross-domain attention, texture refinement, and a linear noise scheduler. The appendix also identifies a table evaluating cross-domain-block position.

A.1 Implementation Details

  • Training data pairs perspective input renderings with orthogonal-camera generated renderings, adjusting camera distance to make their rendered sizes similar.For orthogonal scale s and focal length f, the distance is d = f/s.Source: §A.1 · p. 15
  • The experiments use focal lengths of 24, 35, 50, 85, 105, and 135 mm, with orthogonal views assigned a normalized focal length of 0.Focal lengths are regressed in normalized rather than actual form.Source: §A.1 · p. 15
  • Combining cross-domain and multiview fusion in one row-wise block produced overly smoothed normal maps in earlier experiments.The authors hypothesize that row-wise information alone is insufficient for fine-grained features across domains.Source: §A.1 · p. 15
  • The attention block uses self-cross-domain, row-wise multiview, and cross-domain attention, enabling single-stage joint training to align images from both domains.The authors state that placing cross-domain processing in the self-attention block helps handle domain disparities and improve generalization.Source: §A.1 · p. 15
  • Differential rendering refines reconstructed-mesh textures by optimizing vertex colors against generated images while preserving NeuS geometry.The refinement takes less than 1 second; it uses 200 iterations at 1024 resolution with a learning rate of 1e-3.Source: §A.1 · p. 15
  • Table 5 is an ablation study of the position of the cross-domain block.Source: Table 5 · p. 15
  • The experiments use a linear noise scheduler, which the authors report enhances generation quality and accelerates background convergence at larger image sizes.They describe this adjustment as critical for supporting model performance and efficiency at 512 resolution.Source: §A.1 · p. 16

A.2 Proof of Proposition 1

The proof relates corresponding points in two camera views through epipolar geometry, then uses the canonical camera setting to show that relevant projections share image rows. This motivates row-wise attention as a more efficient epipolar-attention form.

A.2 Proof of Proposition 1

  • The proof frames the epipolar constraint using corresponding image-plane points from two cameras with relative rotation and translation.It then represents the epipolar line in the second camera’s image plane.Source: §A.2 · p. 16
  • Epipolar attention requires querying dense points along epipolar lines, which is computationally inefficient because their directions and lengths vary.Source: §A.2 · p. 16
  • In the canonical camera setting, row-wise multiview attention is a special epipolar-attention form for the cameras’ relative pose.The passage specifies the gravity, right-hand, and object-depth axis directions.Source: §A.2 · p. 16
  • With identical orthographic scale, scene points from different views that share a y-coordinate project to the same image row.Source: §A.2 · p. 16

A.3 Pose Estimation

Era3D averages pose predictions across denoising steps and reports improved elevation and focal-length estimation over baselines. Reconstructed meshes also remain similar to the orthogonal reference across tested camera distortions.

A.3 Pose Estimation

  • Averaging classifier-free-guidance predictions across denoising steps gives the final pose; the authors report that this average achieves the highest accuracy.They attribute the result to random dropping of condition images during training.Source: §A.3 · p. 16
  • Elevation estimates are closer to ground truth, with lower variance in most cases than Dino and One-2-3-45.The authors attribute baseline errors to difficulty predicting absolute elevation from a single image and ambiguity in generated multiview images.Source: §A.3 · p. 16, Table 6 · p. 17
  • Visual checks show consistent generations across elevation inputs and compare color, normal, and mesh outputs for distorted inputs.The elevation figure uses zero-degree elevation reconstructions as its reference.Source: Figure 10 · p. 17, Figure 11 · p. 17
  • Era3D predicts large focal distortions (focal=35, 50) and has smaller errors than the baseline at large focal lengths.Table 7 reports focal-length accuracy on the GSO dataset.Source: Table 7 · p. 17, §A.3 · p. 18
  • Across elevations −10 to 40 degrees and focal lengths {35, 50, 85, 105, 135, ∞}, the 0-degree orthogonal reference achieves the best CD, with other settings on the same bar.CD denotes Chamfer Distance.Source: §A.3 · p. 18, Table 8 · p. 18

NeurIPS Paper Checklist

The checklist records the authors’ claims about experimental support, reproducibility, and limitations. It also reports training resources and evaluation details.

NeurIPS Paper Checklist

Loading 2405.11616v3…