PaperlayerView exact revision on arXiv

Source-linked AI summary

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

Yu Wu, Minsik Jeon, Jen-Hao Rick Chang, Oncel Tuzel, Shubham Tulsiani

arXiv:2601.15275v3cs.CVcs.LG

TL;DR

Multi-view attention needs positional encodings for patches from different posed images, but prior approaches do not meet the desired combination of properties. RayRoPE uses ray-based positions projected into query-camera frames, with estimated depth; it improves performance across 3D vision tasks.

  • Problem

    Prior multi-view positional encodings do not jointly provide cross-view uniqueness, geometry adaptiveness, and multi-frequency similarity alongside SE(3) invariance.

    Source: §1 · p. 2, §3.3 · pp. 5–6

  • Method

    RayRoPE encodes patch-associated rays in each query token’s camera frame and predicts per-token depth to place positions along those rays.

    Source: §1 · p. 2, §4.1 · p. 7, §4.2 · p. 7

  • Results

    Across novel-view synthesis, stereo depth estimation, and feed-forward 3D Gaussian Splatting, RayRoPE outperforms prior positional encodings and generalizes to unseen view counts and datasets.

    Source: §1 · p. 3, §5.2 · p. 13

  • Takeaways & Limitations

    The authors conclude that satisfying the identified positional-encoding properties yields more effective multi-view attention across tasks and models in 3D vision.

    Source: §6 · p. 15

    The work focuses on posed input images; modeling uncertainty in camera matrices and positional encodings for unposed or mixed images remain open challenges.

    Source: §6 · p. 15

Abstract

from arXiv · show

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).

1 Introduction

Multi-view transformers need positional encodings for patches from different posed images, but prior schemes miss key desiderata. RayRoPE uses ray-based positions in query-camera frames and improves performance across 3D vision tasks.

1 Introduction

  • Multi-view positional encodings should provide SE(3) invariance, cross-view uniqueness, geometry adaptiveness, and multi-frequency similarity, which prior schemes do not jointly provide.Prior methods use image coordinates and camera matrices; they lack uniqueness, geometry adaptiveness, or multi-frequency use of camera information.Source: §1 · p. 2
  • RayRoPE encodes patch-associated rays in query-camera projective coordinates, with per-token depth predictions that let positions adapt to scene geometry.Depth is predicted without direct supervision, and uncertainty is modeled to compute expected positional encodings across frequencies.Source: §1 · pp. 2–3
  • RayRoPE outperforms prior relative and absolute positional encodings across novel-view synthesis, stereo depth estimation, and feed-forward 3D Gaussian Splatting reconstruction.The paper also reports strong generalization beyond training views and datasets.Source: §1 · p. 3
  • RayRoPE is presented as an efficient, general positional encoding that can be integrated into generic multi-view models.Its runtime is comparable to existing positional encoding methods.Source: §1 · p. 3

2 Related Work

Transformer positional encodings restore structural information to permutation-invariant attention, progressing from absolute embeddings to relative schemes such as RoPE. In multi-view vision, camera-based relative encodings provide coordinate-frame invariance but do not adapt to scene geometry or support multi-frequency camera similarity.

2 Related Work

  • Transformers use positional encodings because attention treats tokens as an unordered set; approaches progressed from absolute embeddings to relative methods, including RoPE.RoPE rotates query and key features according to position, making attention scores depend on relative position.Source: §2 · p. 3
  • Multi-view transformers process posed images and must encode spatial relationships among patches across views; existing approaches often concatenate rays or camera matrices as absolute encodings.Such encodings depend on an arbitrary global coordinate system.Source: §2 · p. 4
  • Camera-pose relative encodings provide SE(3) invariance but lack geometry adaptiveness and multi-frequency similarity; adding 2D RoPE can break cross-view uniqueness.Concurrent work augments standard RoPE with depth, but its generalizability beyond two views is not explored.Source: §2 · p. 4

3 Preliminaries: Positional Encodings

This section reviews positional encodings for multi-view attention and shows that existing approaches do not combine SE(3) invariance, uniqueness, geometry-adaptiveness, and multi-frequency similarity.

3 Preliminaries: Positional Encodings

  • The attention formulation computes each query token’s output from its query-key scores and the corresponding value features.The pairwise structure of attention supports relative positional encodings.Source: §3 · p. 4

3.1 Rotary Positional Encoding

  • RoPE applies rotations at multiple frequencies so attention depends on relative position, providing translation invariance and multi-scale positional information.Its direct use on 3D ray maps remains dependent on the global coordinate frame.Source: §3.1 · pp. 4–5

3.2 Camera-Based Relative Positional Encoding

  • CaPE encodes camera extrinsics and yields attention based on relative poses, while GTA and PRoPE add patch-index RoPE but do not encode camera frequencies.PRoPE uses the full projection matrix to incorporate camera intrinsics.Source: §3.2 · p. 5

3.3 Limitations of Existing Methods

  • Prior camera-based encodings are SE(3) invariant but lack multi-frequency camera encoding; adding patch-index RoPE still breaks uniqueness and omits scene geometry.Standard RoPE is multi-frequency but not SE(3) invariant.Source: §3.3 · pp. 5–6

4 RayRoPE

RayRoPE defines multi-view token positions with rays, then projects them into each query camera frame for relative, SE(3)-invariant multi-frequency encodings. Predicted depth and uncertainty adapt the encodings to scene geometry while keeping runtime comparable to existing methods.

4.4 Applying RayRoPE Efficiently

  • 4% training and 13% inference overhead versus PRoPE at N=3; grouping attention by query view keeps the added computation negligible.RayRoPE remains compatible with KV-caching and FlashAttention.Source: Figure 3 · p. 7, §4.4 · p. 9

4.1 Image Patch as Ray Segments

  • A ray segment represents each patch using its camera center and the point where its ray intersects the scene at depth d, predicted per layer without supervision.A linear layer estimates depth from token features; supplied depth maps can also be incorporated.Source: §4.1 · p. 7

4.2 Relative Encodings in Query Frame

  • Projecting global rays into each query camera frame makes the RoPE encoding depend on relative positions, ensuring SE(3) invariance while retaining multi-frequency similarity.The projected ray is represented by query-frame camera-center coordinates, projected pixel coordinates, and disparity.Source: §4.2 · pp. 7–8

4.3 Modeling Uncertainty via Expected RoPE

  • Expected RoPE analytically averages positions over predicted uncertainty, reducing to regular RoPE at zero width and smoothing rotations as uncertainty grows.The projected position is assumed uniform between the ray segment’s bounds; this yields stable, geometry-aware encodings despite approximate depths.Source: §4.3 · p. 8
  • RayRoPE outperforms all baseline positional encoding methods across three novel-view synthesis datasets when incorporated in LVSM.The bottom two rows report models trained at increased scale.Source: Table 1 · p. 9

5 Experiments

Experiments compare RayRoPE with positional-encoding baselines for novel-view synthesis, analyze its design and behavior, and evaluate it on broader 3D vision tasks. The reported findings cover benchmark performance, depth and uncertainty predictions, robustness, and transfer to other models.

5.1 Novel-View Synthesis

  • RayRoPE consistently outperforms all positional-encoding baselines across CO3D, Objaverse, and RE10K at both small and large scales.Relative encodings tend to outperform the absolute Plucker raymap; naive RoPE-on-rays struggles on challenging CO3D.Source: §5.1 · p. 10
  • For highly overlapping target views, RayRoPE generates superior high-frequency details; with little overlap, it produces more coherent views as all methods lose sharpness.The qualitative examples compare RayRoPE with baselines across evaluation datasets.Source: §5.1 · p. 10, Figure 4 · p. 11

5.2 Ablations and Analysis

  • Ablations show performance drops when uncertainty modeling, depth prediction, or multi-frequency encoding is removed; encoding value and output features is also helpful.The uncertainty-modeling drop is especially notable with larger pose variations on CO3D and Objaverse.Source: Table 2 · p. 10, §5.2 · p. 11
  • By Layer 5, plausible depth maps emerge without depth supervision, while deeper-layer uncertainty decreases and correlates positively with depth error.The figure also shows predicted depths aligning with ground-truth depth at deeper layers.Source: §5.2 · p. 12, Figure 5 · p. 12
  • RayRoPE maintains superior performance with varying reference-view counts and on unseen datasets and categories; incorporating known reference depths yields larger gains than the RGB-only counterpart.Models trained on CO3D are tested zero-shot, and known depths replace predicted depths for reference views.Source: §5.2 · pp. 12–13, Figure 6 · p. 12, Table 3 · p. 13

5.3 Broader Applications

  • RayRoPE improves stereo depth estimates over UniMatch and PRoPE, outperforms both baselines in feed-forward 3DGS reconstruction, and remains advantageous in multi-view diffusion.The diffusion performance gap widens as the number of reference views increases; depth predictions also remain robust on noisy diffusion-model images.Source: §5.3 · pp. 13–15, Table 4 · p. 14, Table 5 · p. 15

6 Discussion

The discussion identifies four desirable properties for multi-view positional encodings and presents RayRoPE as a ray-based encoding for posed images. It reports improvements across 3D vision tasks and identifies camera-matrix uncertainty and unposed or mixed images as open directions.

6 Discussion

  • The paper identifies four desirable properties for positional encodings in multi-view attention.Source: §6 · p. 15
  • RayRoPE is a ray-based positional encoding for posed input images, and the paper reports improved performance across various 3D vision tasks and models.Source: §6 · p. 15
  • Modeling uncertainty in camera matrices and designing encodings for unposed or mixed images remain open challenges.Source: §6 · p. 15

A.1 Details on Applying RayRoPE

RayRoPE predicts depth and uncertainty from token features, constructs ray representations using camera poses, then computes attention in each query camera’s coordinates. The processed outputs are concatenated across views.

A.1 Details on Applying RayRoPE

  • Token features are linearly projected into queries, keys, values, depth, and uncertainty, which together with camera poses determine global-frame ray representations.Source: §A.1 · p. 19
  • For each query camera, all ray segments are projected into its local coordinates and used to compute expected-RoPE encodings.Attention is executed iteratively for each query camera to account for relative geometric transformations.Source: §A.1 · p. 19
  • The encoding is applied to the query subset, keys, values, and attention output, after which outputs are concatenated across views.Source: §A.1 · p. 19

A.2 Details on Expected RoPE

Expected RoPE averages positional encodings over a uniform distribution of projected positions. The appendix gives an endpoint-based closed form, notes its efficient implementation, and proves preservation of relative-position dependence under an independence assumption.

A.2 Details on Expected RoPE

  • The implementation assumes the projected position is uniformly distributed between its minimum and maximum values.Source: §A.2 · p. 19
  • Expected RoPE computes the expected positional encoding across frequencies, with each complex exponential expectation given by a closed form using interval endpoints.The resulting expression uses standard trigonometric functions and can be computed efficiently.Source: §A.2 · p. 19
  • Under the assumption that projected positions are independent, the expected relative encoding factors into expected encodings and depends on relative ray position, not absolute position.Source: §A.2 · pp. 20–21

B Experimental Details

The experiments adapt LVSM and other multi-view models to evaluate RayRoPE across datasets and tasks. They also specify training configurations and an attention-similarity analysis setup.

B Experimental Details

  • The smaller LVSM uses 6 transformer layers, a 1024-dimensional feedforward layer, 8 attention heads, and about 47M parameters.It is trained on 2 GPUs with batch size 8, 2 reference views, 1 supervised view, identity-normalized first-view extrinsics, and 256 × 256 resolution.Source: B Experimental Details · p. 21
  • Experiments use RE10K and Objaverse splits and camera sampling, plus CO3D training on 41 categories with held-out-scene evaluation.CO3D evaluation also includes 10 held-out categories in Sec. D.Source: B Experimental Details · p. 21
  • The larger LVSM has 12 layers and about 150M parameters; Unimatch and TokenGS follow their respective established training configurations.TokenGS trains on DL3DV-10K and uses test-time tuning for evaluation.Source: B Experimental Details · p. 21
  • For TokenGS cross-attention, RayRoPE encodes image features in global coordinates and skips query-frame projection for positionless 3DGS query tokens.The same scheme is applied to multi-view self-attention; global coordinates are used because TokenGS predicts 3DGS in that coordinate system.Source: B Experimental Details · p. 21
  • The attention-similarity setup varies camera translation, camera rotation, or predicted depth and uncertainty, with the first patch depth fixed at 1.The figure depicts the three pose settings and the varying second-patch depth and uncertainty.Source: Figure 8 · p. 22
  • EscherNet is trained and evaluated on DL3DV-10K with 3 input and 3 target views for 70K iterations.Training uses an effective batch size of 32.Source: B Experimental Details · p. 22

C Analysis of Attention Similarity

The analysis varies relative camera pose, ray depth, and uncertainty to examine RayRoPE attention similarity. Similarity responds to pose changes and to whether rays encode the same 3D point.

C Analysis of Attention Similarity

  • Attention similarity is highest for identical patches, then decreases at long range and oscillates at short range as relative translation or rotation increases.The oscillations arise from higher-frequency channels in the multi-frequency similarities.Source: C Analysis of Attention Similarity · p. 22
  • When the two ray segments encode the same 3D point, similarity is maximized at d2 = 1 and decreases with oscillations as d2 deviates from ground-truth depth; increasing uncertainty σ smooths high-frequency oscillations.Source: C Analysis of Attention Similarity · p. 23
  • Increasing uncertainty smooths high-frequency oscillations, avoiding unstable high-frequency encodings when uncertainty is high.The analysis uses constant query and key features to isolate positional encoding effects; higher similarity indicates stronger mutual attention.Source: C Analysis of Attention Similarity · pp. 22–23

D Additional Results

Additional experiments test RayRoPE across varying reference-view counts, out-of-domain scenes, pose-variation types, and qualitative reconstruction settings. The results report robustness across several of these settings and stronger visual quality in two model applications.

D Additional Results

  • RayRoPE produces Gaussian splats with better visual quality in TokenGS and target views with better visual quality and greater 3D consistency in EscherNet.TokenGS takes four input views; EscherNet conditions on three reference views and generates target views for specified cameras.Source: Figure 9 · p. 24, Figure 10 · p. 25
  • RayRoPE performs best across varying reference-view counts and outperforms baselines in most settings across three datasets.Its advantage over camera-based encodings widens as the number of reference views increases, including when evaluation uses more views than training.Source: D Additional Results · p. 23, Figure 11 · p. 26
  • On out-of-domain scenes, RayRoPE consistently outperforms baseline methods, retaining an advantage on Objaverse and RE10K despite performance degradation across methods.The evaluation uses an LVSM model trained on CO3D and tests zero-shot on Objaverse, RE10K, and held-out CO3D categories.Source: D Additional Results · p. 23, Table 7 · p. 27
  • For radial pose variations, ray-based encodings significantly outperform camera-based baselines, with multi-frequency encodings supporting fine-detail transfer from reference features.Radial targets share a reference view's elevation and azimuth but differ in intrinsics and radius; compound variations also change viewing angles.Source: D Additional Results · p. 23, Table 8 · p. 27
  • Across four additional CO3D scenes, Layer 5 depth predictions match ground truth most closely, while predicted uncertainty is higher in earlier layers and lower in later layers.The uncertainty pattern agrees with the analysis in Fig. 5.Source: Figure 12 · p. 28
Loading 2601.15275v3…