PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Attention All That NeRF Needs?

Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang

arXiv:2207.13298v3cs.CV

TL;DR

NeRF-based view synthesis commonly relies on scene-specific fitting and handcrafted volume rendering, motivating a more transferable learned alternative. GNT uses view and ray transformers to aggregate epipolar-aligned features and render target pixels from sampled ray points, with experiments showing improved single-scene and cross-scene results and attention-based depth and occlusion cues.

  • Problem

    Prior NeRF methods rely on scene-specific optimization and handcrafted rendering, while generalizable methods still use classical volume rendering after feature aggregation.

  • Method

    GNT uses a view transformer for epipolar-geometry-constrained feature aggregation and a ray transformer for learned ray-based rendering.

  • Results

    GNT improves single-scene and cross-scene training results, and attention visualizations show that depth and occlusion can be inferred from attention maps.

  • Takeaways & Limitations

    The results support transformers as a potential universal modeling tool for physically grounded rendering and as replacements for some conventional view-synthesis inductive biases.

  • Takeaways & Limitations

    The epipolar-constrained view transformer may miss non-epipolar scenes and complex light transport, and missing boundary correspondences can cause minor artifacts.

Abstract

from arXiv · show

We present Generalizable NeRF Transformer (GNT), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views. While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages. (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinate-aligned features by aggregating information from epipolar lines on the neighboring views. (2) The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching. Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer. When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~10% on average. Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physically-grounded rendering. Our results show the promise of transformers as a universal modeling tool for graphics. Please refer to our project page for video results: https://vita-group.github.io/GNT/.

1 INTRODUCTION

GNT replaces conventional NeRF scene fitting and handcrafted rendering with transformers for transferable novel-view synthesis. Its view and ray transformers aggregate multi-view and ray-sampled information, while analyses indicate depth- and occlusion-aware attention.

  • Motivation: Prior generalizable NeRF methods aggregate source-view features but still decode them into radiance fields and use classical volume rendering.The paper identifies this rendering dependence as a limitation of prior approaches.
  • Approach: GNT treats transferable novel-view synthesis as multi-view feature fusion followed by sampling-based rendering integration.Both stages are implemented with transformer-based attention aggregation.
  • Approach: The view transformer aggregates epipolar-geometry-compliant multi-view features to infer coordinate-aligned features.This provides the scene-representation stage of GNT.
  • Approach: The ray transformer learns ray-based rendering to predict target colors from aggregated features.It replaces the conventional explicit rendering component with learned attention-based rendering.
  • Results: GNT achieves state-of-the-art results on complex scenes and cross-scene generalization, while attention analysis finds depth- and occlusion-aware behavior.The experiments cover both the architecture’s performance and the properties of its learned attention.
  • Implications: The paper argues that attention mechanisms can replace inductive biases such as persistent 3D models and hard-coded rendering equations.This is presented as the broader implication of the combined architecture.

2 RELATED WORK

Related work extends NeRF toward transformer-based and generalizable novel-view synthesis. GNT differs by learning direct target-pixel rendering rather than relying on classical volume rendering after feature aggregation.

  • Transformers: Transformers have been applied broadly as learning backbones for long-range correlations in language, vision, speech, and protein structure.The related-work discussion situates GNT within this wider transformer literature.
  • NeRF: NeRF synthesizes novel views by fitting each scene as a continuous 5D radiance field parameterized by an MLP.Subsequent work improves scale handling, view-dependent effects, surface representation, and dynamic-scene modeling.
  • Transformer-based radiance fields: Generalizable methods such as IBRNet and NeRFormer use transformers or attention to aggregate source views under epipolar geometry constraints.Their approaches construct feature representations across views for novel-view synthesis.
  • Transformer-based radiance fields: Unlike these related methods, GNT’s ray transformer learns to render the target pixel directly instead of using classic volume rendering.This is the paper’s principal methodological distinction from the cited transformer-based radiance-field methods.
  • Alternative formulations: Other transformer approaches generate views from a single image or map camera-pose embeddings directly to images without physical constraints.The related work also contrasts these methods with light-field formulations of view synthesis.

3 METHOD: MAKE ATTENTION ALL THAT NERF NEEDS

GNT constructs a feed-forward feature-field representation from source views and renders target pixels with attention over sampled ray points. Its architecture uses epipolar geometry for view aggregation and a learned transformer in place of explicit volume rendering.

  • Overview: GNT takes input views with known camera parameters and synthesizes novel views while generalizing to new scenes.The pipeline has two stages: on-the-fly feature-space representation construction and feature-field re-rendering at a target angle.
  • Scene representation: The method models a 3D scene as a coordinate-aligned latent feature field rather than an explicit color-density field.The field maps spatial coordinates and viewing directions to latent features.
  • 3.1 EPIPOLAR GEOMETRY CONSTRAINED SCENE REPRESENTATION: The view transformer combines source-view features sampled at projected image locations while restricting attention according to epipolar geometry.This design supplies multi-view geometric structure and reduces unrestricted global attention.
  • 3.1 EPIPOLAR GEOMETRY CONSTRAINED SCENE REPRESENTATION: The view transformer uses point coordinates, viewing direction, and relative source-view directions as positional information for feature aggregation.Projected features are obtained through bilinear interpolation on encoded source-view feature grids.
  • 3.2 ATTENTION DRIVEN VOLUMETRIC RENDERING: The ray transformer is intended to learn attention-based aggregation for occlusion, surface sharpness, lighting, and complex light transport.The paper also reports that physical properties such as depth can be inferred from its attention.
  • 3.2 ATTENTION DRIVEN VOLUMETRIC RENDERING: For each ray, GNT samples points between near and far planes and feeds their feature sequence to a ray transformer that predicts the target color.Mean pooling and an MLP map the transformer outputs to RGB.

4 EXPERIMENTS

GNT is evaluated under per-scene optimization and cross-scene generalization, including challenging scenes with reflections and refractions. Its transformer-based design performs strongly while learned attention supports view selection, depth inference, and data-driven rendering.

  • Experimental setup: Experiments compare GNT under both per-scene optimization and cross-scene generalization.Training uses posed multi-view images and evaluates standard image-quality metrics across scenes and datasets.
  • Single-scene results: On single-scene datasets, GNT outperforms most methods and matches NLF despite using fewer rays per batch and only coarse sampling.It gains 2.49 dB on Drums, 0.82 dB on Ship, and 0.10 dB on Leaves, while achieving approximately 27% lower LPIPS across LLFF scenes.
  • Experimental setup: GNT uses multi-view datasets for training and evaluates transfer on Synthetic, LLFF, and complex-optics Shiny-9 scenes.Training combines synthetic object renderings with real-world scene collections; validation LLFF scenes are excluded from handheld cellphone training captures.
  • Cross-scene generalization: GNT outperforms state-of-the-art methods by ~17% on LLFF and ~9% on Synthetic average scores during cross-scene generalization.It also reports 3% higher SSIM and 27% lower LPIPS across both datasets, with improved visual quality on unseen scenes.
  • Complex optics: GNT outperforms many per-scene methods and approaches NLF on Shiny without extra training, while exceeding GPNR by 3 dB PSNR and ~40% in average metric.Qualitatively, it reconstructs complex refractions through a test tube and interference patterns on a disk more accurately than NeX.
  • Ablation studies: The two-stage transformer outperforms one-stage and alternative two-stage aggregation, while the ray transformer outperforms classic volumetric rendering.Subtraction-based attention is slightly better overall than dot-product attention, but performance is not heavily dependent on the attention operation.
  • Interpretation on attention maps: View attention favors source views that are least occluded in corresponding target regions, while ray attention produces depth maps from point-wise correspondences.The visualizations connect learned attention with source-view correspondence, occlusion handling, and depth inference.

5 CONCLUSION

GNT reconstructs and renders NeRFs on the fly using transformers, combining epipolar-geometry-aware view aggregation with attention-based ray rendering. Experiments show improvements in single-scene and cross-scene settings, including refraction and reflection scenes, while attention maps reveal depth and occlusion.

  • 5 CONCLUSION: GNT is a pure transformer architecture whose view transformer uses epipolar geometry and whose ray transformer decodes sampled point features during ray marching.The architecture reconstructs NeRFs on the fly without requiring a persistent per-scene representation.
  • 5 CONCLUSION: GNT improves single-scene and cross-scene rendering results and shows promise on scenes containing refraction and reflection.The conclusion frames these results as evidence for attention-based modeling of physically grounded rendering.
  • 5 CONCLUSION: Attention maps can infer depth and occlusion, supporting attention as a universal modeling tool for physically grounded rendering.The paper identifies relaxing epipolar constraints for more complicated light transport as a future direction.

A PRELIMINARIES

NeRF represents a scene with a coordinate-based network and renders pixels by integrating color and density along camera rays. Transformer self-attention instead aggregates token features according to pairwise relationships, with standard blocks adding feed-forward transformations, residual connections, and normalization.

  • Self-Attention and Transformer: Self-attention computes pairwise token relations, normalizes them row-wise, and aggregates value features across the input sequence.The attention matrix weights how each token incorporates information from every other token.
  • Self-Attention and Transformer: A transformer block combines multi-head attention with a point-wise feed-forward network, skip connections, and layer normalization.The residual structure first updates attention features and then applies the feed-forward transformation.
  • Neural Radiance Field: NeRF maps spatial coordinates and viewing directions to color and density, then renders images by composing samples along each ray.The radiance field is parameterized by an MLP and queried from new viewpoints.
  • Neural Radiance Field: Training NeRF minimizes squared differences between rendered ray colors and ground-truth pixel colors.The training set consists of sampled rays paired with observed pixel colors.

B IMPLEMENTATION DETAILS

The view transformer makes cross-view attention efficient and geometry-aware by using one read-out token and relative view directions. Its implementation combines epipolar features, projected positional information, and MLP-parameterized attention operations.

  • Memory-Efficient Cross-View Attention: A single read-out token summarizes features from all sampled view inputs, reducing each attention layer’s memory complexity from O(N^2) to O(N).The token is initialized by element-wise max-pooling and iteratively aggregates the other data points.
  • Geometry-Aware View Attention: The view transformer incorporates relative source-to-target directions as positional information in its modified attention computation.A learned linear projection lifts relative directions into the hidden dimension before they modify attention features.
  • Geometry-Aware View Attention: The modified view attention reads out the updated query token after aggregating geometry-enhanced values.The diagonal operation selects the updated read-out token from the attention output.
  • Network Architecture: The network extracts source-view features with a U-Net-like ResNet34 encoder-decoder, using single-headed cross-attention for view blocks and four-headed self-attention for ray blocks.Both attention outputs feed ReLU-activated feed-forward blocks with hidden dimension 256 and residual connections.
  • Implementation: The implementation reuses epipolar view features to derive keys and values across view-transformer blocks, enabling further efficiency gains.The shared features can be computed once rather than repeatedly for each block.

C.1 AUTO-REGRESSIVE DECODING

GNT’s ray transformer renders a ray by processing sampled point features with attention and pooling the predicted tokens into an RGB color. Its autoregressive variant follows far-to-near decoding to better reflect volumetric composition, while caching reduces redundant computation.

  • Ray Transformer: GNT mean-pools ray-transformer outputs and maps the pooled feature vector to RGB for the final rendered color.This learned renderer replaces explicit point-wise color-and-density decoding followed by classical volume rendering.
  • Ray Transformer: The ray transformer maps coordinate-aligned point features, point coordinates, and target-view directions into per-ray attention inputs.The implementation position-encodes the point and direction inputs before forming query, key, and value features.
  • Auto-Regressive Decoding: The autoregressive decoder queries sampled points from farthest to nearest, concatenating each new point with the previous output token.The final pass also incorporates the encoded viewing direction before producing the ray output.
  • Auto-Regressive Decoding: Autoregressive decoding is required during training because ground-truth colors for individual sampled points are unavailable.Unlike language decoding, neural rendering cannot rely on per-step ground-truth output tokens.
  • Auto-Regressive Decoding: Caching stores previous per-layer token outputs so each new decoding step avoids recomputing earlier attention.The cache does not remove the iterative loop but improves decoding speed as redundant computation decreases.

C.2 ATTENTION GUIDED COARSE-TO-FINE SAMPLING

GNT uses ray attention to estimate point-wise importance and guide coarse-to-fine sampling. Fine attention-derived depth maps better distinguish nearby pixels, although gains are not significant across every scene without matching training exposure.

  • Attention-Guided Sampling: Ray-transformer attention learns point-to-point correspondences that the authors use to model visibility, occlusion, and point-wise density.The learned point-wise importance values are treated as equivalent to density σ for depth analysis.
  • Attention-Guided Sampling: Fine attention-derived depth maps better estimate differences between nearby pixels, producing higher-resolution outputs than coarse maps.Figure 9 visualizes the coarse and fine attention stages, with red denoting farther and blue nearer points.
  • Experimental Comparison: Performance improvements are not significant across all scenes because the model was not trained with the coarse-to-fine sampling strategy.The authors expect results to improve with training that matches the evaluation strategy.

D ADDITIONAL RESULTS AND ANALYSIS

Additional experiments show that GNT generalizes strongly across scenes and datasets, while its simpler two-stage design remains competitive with or superior to transformer and NeRF-based alternatives. Qualitative results also indicate that the learned renderer can model reflections, though boundary correspondence failures produce minor artifacts.

  • Additional quantitative results: GNT quantitatively surpasses original NeRF and achieves on-par results with state-of-the-art methods in per-scene breakdowns.The authors note slight underperformance relative to NLF on some scenes and question the fairness of that comparison because NLF uses larger batches and longer training.
  • Comparison with SRT: GNT largely outperforms LFN, PixelNeRF, and SRT on all metrics in cross-scene evaluation on NMR.The evaluation uses cross-scene-trained GNT without further tuning; SRT had substantially more pre-training data with a narrower domain gap.
  • Single-scene rendering: GNT’s learnable renderer models physical phenomena such as reflections in single-scene rendering.Qualitative examples are shown for the Trex scene from LLFF and the Materials scene from Blender.
  • Comparison with GPNR: ~3dB: GNT outperforms GPNR on the Shiny dataset with a simpler architecture and lighter feature engineering.The comparison reports that GNT outperforms GPNR on all tested datasets.
  • Qualitative limitations: Missing epipolar correspondences for boundary pixels cause minor stripe artifacts in qualitative comparisons with ground truth.The limitation is illustrated in Figure 11.

E DEFERRED DISCUSSION

GNT’s attention mechanisms are interpreted as learned correspondence and visibility estimators. The view transformer uses epipolar-constrained matching for occlusion awareness, while the ray transformer’s attention weights support depth inference analogous to volumetric rendering weights.

  • Discussion on Occlusion Awareness: The view transformer’s attention acts as a matching algorithm that selects neighboring-view pixels maximizing view consistency.Its attention can be interpreted as the likelihood that a source pixel images the same 3D point without intervening geometry.
  • Discussion on Occlusion Awareness: Epipolar geometry-constrained attention can learn occlusion without explicitly regressing visibility.The view transformer’s U-Net supplies multi-scale features before attention performs correspondence selection.
  • Discussion on Depth Cuing: Ray-transformer attention weights represent each sampled point’s importance in forming the image and reflect visibility and occlusion.Averaging attention scores across points allows depth maps to be inferred from attention maps, analogous to accumulated volume-rendering weights.

F LIMITATIONS

GNT’s view transformer depends on valid epipolar constraints, limiting what it can capture in non-epipolar scenes and under complex light transport. Missing boundary correspondences can also create minor artifacts.

  • Scope limitations: The view transformer can aggregate information only from valid epipolar lines, limiting non-epipolar scene coverage.The authors state that non-epipolar scenes may not be captured by the view transformer.
  • Scope limitations: Complex light transport may not be captured reliably, although the image encoder’s contribution to global light transport remains unclear.The method uses a large-receptive-field feature extractor and empirically works well on complex lighting effects, but the captured information is unresolved.
  • Artifact limitation: Missing epipolar correspondence at boundary pixels causes some minor artifacts.The paper identifies these artifacts in its qualitative limitation analysis.
Loading 2207.13298v3…