PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, Ravi Ramamoorthi

arXiv:2207.05736v2cs.CVcs.GR

TL;DR

Image-conditioned novel view synthesis from a single unposed image can degrade when target regions are occluded in the input. The method combines ViT global features and CNN local features in a 3D representation for NeRF rendering. It reports favorable comparisons with existing approaches across object categories and real input images.

  • Problem

    Image-conditioned models can degrade at viewpoints with occlusion because features from the input view may correspond to the wrong surface.

    Source: §1 · p. 1

  • Method

    The method combines global ViT features and local CNN appearance features in a 3D representation used for volumetric rendering of novel views.

    Source: §1 · p. 1, §3.4 · p. 5

  • Results

    The method performs favorably against existing approaches and generalizes across multiple categories and real input images.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The hybrid representation shows promising performance in recovering occluded shapes and appearance without requiring a canonical coordinate system.

    Source: §5 · p. 8

    The method does not use geometry priors such as symmetry and does not fully utilize high-level object semantics.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Although neural radiance fields (NeRF) have shown impressive advances for novel view synthesis, most methods typically require multiple input images of the same scene with accurate camera poses. In this work, we seek to substantially reduce the inputs to a single unposed image. Existing approaches condition on local image features to reconstruct a 3D object, but often render blurry predictions at viewpoints that are far away from the source view. To address this issue, we propose to leverage both the global and local features to form an expressive 3D representation. The global features are learned from a vision transformer, while the local features are extracted from a 2D convolutional network. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. This novel 3D representation allows the network to reconstruct unseen regions without enforcing constraints like symmetry or canonical coordinate systems. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model. Quantitative and qualitative evaluations demonstrate that the proposed method achieves state-of-the-art performance and renders richer details than existing approaches.

1. Introduction

Single-image novel view synthesis degrades when target regions are occluded in the input view. The method combines global ViT features and local CNN features in a 3D representation to render novel views, including unseen regions.

1. Introduction

  • When target-view regions are invisible in the input, image-conditioned models can use features from the wrong surface, degrading rendering quality.The paper illustrates this with a target-view car wheel associated with input-view window features.Source: §1 · p. 1, Figure 2 · p. 1
  • ViT features encode global information while CNN features capture local appearance, and volumetric rendering uses this hybrid representation to synthesize novel viewpoints.The approach is designed to render unseen regions with more accurate structure and finer details.Source: §1 · p. 2
  • The method performs favorably against existing approaches and generalizes across 13 ShapeNet categories, category-agnostic data, and real input images.The reported comparisons include SRN, PixelNeRF, FE-NVS, SRT, and FWD.Source: §1 · p. 2

2. Related work

Prior view-synthesis work uses varied scene representations, while this work combines global information with local image features. Its ViT-based approach differs from transformer methods that aggregate CNN features or represent scenes with latent codes.

2.1. Novel View Synthesis

  • The proposed 3D representation combines global information and local image features rather than relying on pure 1D, 2D, or 3D representations.The related-work discussion situates it among image-based rendering, proxy geometry, layered representations, learning-based methods, and NeRF approaches.Source: §2.1 · p. 2

2.2. Transformer

  • Unlike SRT’s latent-code scene representation, this method uses a radiance field and per-pixel feature maps with explicit input-image-to-query mapping.The paper also distinguishes its focus on learning global image information with ViT from transformer approaches that aggregate CNN-extracted source-view features.Source: §2.2 · p. 2

3. Novel View Synthesis From a Single Image

The method infers a 3D representation from one image by combining ViT global features with 2D-CNN local appearance features, then uses a NeRF module for novel-view synthesis.

3.1. Synthesizing Occluded Regions

  • 1D methods encode holistic shape but have limited inductive bias; 2D methods improve visible-area rendering yet blur unseen parts, while voxels use 3D priors for unseen geometry but limit resolution and receptive fields.The proposed method combines local image features with global information.Source: §3.1 · p. 3
  • The hybrid pairs ViT-learned global information with local 2D-CNN features, retaining high-resolution image features while expanding receptive fields for unseen-region reconstruction.Unlike CodeNeRF and DISN, it does not require a canonical coordinate system for global features.Source: §3.1 · pp. 3–4

3.2. Global Features from Vision Transformer

  • The ViT tokenizes projected image patches, uses self-attention for long-range dependencies, then decodes coarse-to-fine global feature maps.The decoder produces multi-level maps for concatenation with local appearance features at rendering.Source: §3.2 · pp. 4–5

3.3. Local Features from Convolutional Networks

  • Because ViT-only features compromise near-view color and appearance consistency, a 2D CNN supplies local features that are fused with global maps into hybrid feature W.The local features improve consistency in visible regions.Source: §3.3 · p. 5

3.4. Volumetric Rendering with NeRF

  • For each target ray, the method samples points between near and far bounds, projects them into the source image, and uses sampled features, 3D position, and view direction to predict color and density for volume rendering.The MLP receives the per-pixel feature, camera-coordinate sample location, and viewing direction.Source: §3.4 · p. 5

4. Experimental Results

Experiments evaluate single-image view synthesis on category-specific and category-agnostic ShapeNet tasks, real car images, component and backbone ablations, and method limitations. The results report strong benchmark performance and sharper, more detailed renderings, while identifying missing geometry priors and semantic understanding as limitations.

4.1. Category-specific View Synthesis

  • On chairs, the method improves all three reported metrics; on cars, it leads in LPIPS and SSIM while PixelNeRF has higher PSNR.The authors report clearer chair structures and finer car details than competing methods.Source: Table 2 · p. 6, §4.1 · p. 6

4.2. Category-agnostic View Synthesis

  • Across 13 categories, the method achieves state-of-the-art PSNR and SSIM and competitive LPIPS, using a single model.The authors describe the hybrid representation as more expressive than locally conditioned models or 3D voxel methods.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. View Synthesis on Real Images

  • On real car images, the ShapeNet-trained model reconstructs occluded shape and details such as headlights and side mirrors without camera-pose input.The comparison is against PixelNeRF, whose predictions are blurry in occluded regions.Source: §4.3 · pp. 7–8, Figure 8 · p. 7

4.4. Ablation Studies

  • The ablations show that local CNN features improve input-view color fidelity, viewing direction sharpens structures, and ResBlocks further refine geometry.The ViT-only baseline can render realistic images but may not preserve local appearance and color from the input view.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Global Features from Different Backbones

  • Replacing ViT with EfficientNet or ConvNeXt yields worse LPIPS on both car and chair categories.The authors attribute ViT's advantage to architecture design rather than parameter count.Source: §4.5 · p. 8, Table 5 · p. 8

4.6. Limitations and Future Work

  • The method does not use geometry priors such as symmetry or fully exploit high-level object semantics; semantic completion and generative texture for occluded regions remain future directions.The authors note that symmetry could help reuse unseen car details, while choosing a symmetry plane or canonical space remains an open question.Source: §4.6 · p. 8

5. Conclusions

The conclusion presents a NeRF-based method for novel-view synthesis from one unposed image. It highlights the hybrid global-local representation and its recovery of occluded shape and appearance.

5. Conclusions

  • The proposed NeRF-based algorithm synthesizes novel views from a single unposed image.The conclusion frames this as the paper's central task.Source: §5 · p. 8
  • Vision transformer and convolutional features form a hybrid 3D representation that shows promising performance in recovering occluded shapes and appearance.Source: §5 · p. 8
  • The vision transformer supplies global information without enforcing a canonical coordinate system, which would require camera-pose estimation.Source: §5 · p. 8

A. Video Results

The appendix provides video results comparing the input, several methods, and ground truth. It notes that SRN and PixelNeRF renderings were obtained from PixelNeRF's authors on request.

A. Video Results

B. Network Architecture

The architecture uses a pretrained ViT encoder, a convolutional decoder and local-feature network, and a NeRF MLP for hierarchical rendering. Global and local feature representations are concatenated before rendering.

B. Network Architecture

  • The transformer encoder uses pretrained ViT-b16 with 12 layers, LayerNorm, GELU, and positional embeddings resized to the input image.Source: B. Network Architecture · p. 10
  • The convolutional decoder uses tokens from layers 3, 6, 9, and 12 to generate multi-level feature maps, which are resized and processed by a two-layer CNN.Source: B. Network Architecture · p. 10
  • A 2D CNN with ResBlocks generates local features, which are concatenated with the global feature representation as the hybrid feature.Source: B. Network Architecture · p. 10
  • The NeRF MLP uses hierarchical rendering and six width-512 ResNet blocks at both coarse and fine stages.Source: B. Network Architecture · p. 10

C. Implementation Details

The MLP uses a warm-up and step-based decay schedule, while ViT and the 2D CNN use a lower learning rate.

C. Implementation Details

D. Experiment Configurations

The appendix gives configuration references and comparison protocols, including evaluation on real images and an unseen mug category.

D. Experiment Configurations

D.1. Category-specific View Synthesis.

  • CodeNeRF lacks available pre-generated results and unposed-inference support for LPIPS evaluation, and its reported input view differs from the other methods.The comparison uses results from prior tables for SRN, PixelNeRF, CodeNeRF, and FE-NVS; FE-NVS qualitative images are cropped from its paper because source code is unavailable.Source: §D.1 · pp. 10–11
  • Supplementary architecture tables show the convolutional decoder and the final two layers that generate the global feature representation W′ G.Source: Table 6 · p. 11, Table 7 · p. 11

D.2. Category-agnostic View Synthesis

  • For category-agnostic results, comparison numbers and qualitative images are drawn from prior tables, project materials, and author-provided results; SRT lacks full dataset renderings.Source: §D.2 · p. 11

D.3. View Synthesis on Real Images

  • Real-car evaluation removes backgrounds, resizes images to 128 × 128, assumes an identity input camera pose, and simulates 360° views with rotations.PixelNeRF results use its source code and pretrained model at the same camera poses.Source: §D.3 · p. 11

E.1. Generalization on Unseen objects.

  • On an unseen real mug absent from training, the category-agnostic model predicts reasonable novel views.Source: §E.1 · p. 11

E.2. Results on Category-specific and Category-agnostic View Synthesis

The supplement adds category-specific and category-agnostic renderings, selecting target views that emphasize regions occluded in the input.

E.2. Results on Category-specific and Category-agnostic View Synthesis

  • Supplementary results include additional category-specific and category-agnostic view-synthesis examples.Source: §E.2 · p. 11
  • Target views are selected where most pixels are not visible in the input view, focusing comparison on occluded-region rendering quality.Source: §E.2 · p. 11
  • Video results are also provided on the project website.Source: §E.2 · p. 11
Loading 2207.05736v2…