Source-linked AI summary
Cameras as Relative Positional Encoding
Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, Angjoo Kanazawa
TL;DR
Multiview transformers need camera geometry to ground visual tokens in 3D, but existing absolute and pose-based encodings represent that geometry incompletely or depend on global framing. The paper compares these approaches and proposes PRoPE, a relative encoding of complete camera frustums; relative conditioning improves novel view synthesis and generalizes across tasks and model sizes. The paper also identifies numerical stability, distorted cameras, and multi-frequency encoding as future challenges.
Problem
Multiview transformers must bind viewpoint information to image tokens, while naive or absolute camera encodings can be suboptimal and sensitive to reference-frame choices.
Method
The paper surveys raymaps and relative SE(3) encodings, then proposes PRoPE, which injects camera intrinsics and extrinsics into self-attention as a relative positional encoding.
Results
Relative camera conditioning, particularly PRoPE, improves novel view synthesis across camera settings and generalizes to stereo depth, discriminative spatial cognition, and larger models.
Takeaways & Limitations
Representing cameras as relative positional encodings that capture intrinsics and extrinsics benefits multiview transformers across settings and tasks.
Takeaways & Limitations
Future work must address numerical stability, distorted camera models, and the difficulty of applying multi-frequency encoding to non-commutative projective transforms.
Abstract
from arXiv · showhide
Transformers are increasingly prevalent for multi-view computer vision tasks, where geometric relationships between viewpoints are critical for 3D perception. To leverage these relationships, multi-view transformers must use camera geometry to ground visual tokens in 3D space. In this work, we compare techniques for conditioning transformers on cameras: token-level raymap encodings, attention-level relative pose encodings, and a new relative encoding we propose -- Projective Positional Encoding (PRoPE) -- that captures complete camera frustums, both intrinsics and extrinsics, as a relative positional encoding. Our experiments begin by showing how relative camera conditioning improves performance in feedforward novel view synthesis, with further gains from PRoPE. This holds across settings: scenes with both shared and varying intrinsics, when combining token- and attention-level conditioning, and for generalization to inputs with out-of-distribution sequence lengths and camera intrinsics. We then verify that these benefits persist for different tasks, stereo depth estimation and discriminative spatial cognition, as well as larger model sizes.
1 Introduction
Multiview transformers need camera-aware positional information to bind image tokens to 3D geometry. The paper surveys absolute and relative conditioning, introduces PRoPE, and reports benefits across camera settings, tasks, and model sizes.
- Camera geometry gives pixel coordinates physical meaning and grounds visual observations in 3D space.
- Naive camera encodings may be suboptimal, motivating a shift from absolute to relative conditioning for multiview transformers.
- The study compares pixel-aligned raymaps, attention-level SE(3) pose encodings, and PRoPE, which captures complete frustum geometry.
- PRoPE injects both camera intrinsics and extrinsics into transformer self-attention blocks as a relative positional encoding.
- Relative encodings, particularly PRoPE, improve novel view synthesis across shared and varying intrinsics, hybrid conditioning, and out-of-distribution inputs.
- These camera-conditioning benefits extend to stereo depth estimation, discriminative spatial cognition, and larger model sizes.
2 Related Work
Related work frames camera conditioning as an adaptation of the broader shift from absolute to relative positional encodings. Existing multiview systems commonly use raymaps, while relative encodings model camera relationships differently.
- Transformer position encoding has increasingly shifted from absolute methods toward relative methods such as RoPE.
- Multiview transformers use multiple images together with known camera geometry for tasks including reconstruction, view synthesis, depth prediction, and robotics.
- Raymaps are the dominant camera-conditioning approach and concatenate per-pixel ray or Plücker embeddings with image pixels.
- Raymaps encode camera intrinsics and extrinsics at token level but require a reference frame because they are expressed in global terms.
3 Conditioning Transformers on Cameras
The paper conditions multiview transformers on camera geometry using token-level raymaps, attention-level relative pose encodings, and PRoPE, which represents complete camera frustums. PRoPE combines frustum relationships with within-camera patch positions while retaining relative-encoding invariance to the world frame.
- Preliminaries: Camera intrinsics define frustum shape and field of view, while extrinsics define camera position and orientation.Together, they are encapsulated in each camera’s world-to-image projection matrix.
- Pixel-aligned camera encoding: Raymaps concatenate images with per-pixel geometric embeddings, encoding camera intrinsics and extrinsics at the token level.Naive raymaps use ray origins and directions, whereas Plücker raymaps replace origins with moment terms invariant to the ray’s origin choice.
- Relative SE(3) encoding: Absolute raymaps depend on a global reference frame, whose arbitrary choice can hinder generalization.Relative pose encodings avoid this dependence by relating camera poses directly and making the network invariant to the world-frame definition.
- Relative SE(3) encoding: CaPE transforms query and key matrices using relative SE(3) pose, while GTA additionally transforms values during attention aggregation.GTA experiments include value transformation together with SE(3) camera pose and RoPE for 2D patch position.
- Projective Positional Encoding (PRoPE): PRoPE uses projective relationships between complete camera frustums rather than relative poses alone, incorporating intrinsics and extrinsics into attention.Its attention matrices encode both inter-camera frustum relationships and relative patch positions using GTA-style attention and RoPE terms.
- Projective Positional Encoding (PRoPE): PRoPE reduces to relative SE(3) attention for identity intrinsics and to ordinary RoPE for tokens from the same image.These reductions connect PRoPE to existing relative camera conditioning and single-image vision-transformer positional encoding.
4 Experiments
The experiments compare camera-conditioning strategies for novel view synthesis and test their robustness across intrinsics, hybrid encodings, distribution shifts, tasks, and model scales. Relative encodings generally outperform absolute raymaps, with PRoPE providing broad gains.
- The study compares camera-conditioning strategies in LVSM-based novel view synthesis under matched model and training settings.The experiments use consistent code, hyperparameters, training steps, and approximately 25M-parameter models in the main comparisons.
- 4.2 Relative vs Absolute Positional Encodings: Relative encodings consistently outperform absolute raymap encodings, with PRoPE followed closely by GTA producing the best results.The comparison includes Naive and Plücker raymaps, CAPE, GTA, and PRoPE.
- 4.2 Relative vs Absolute Positional Encodings: PRoPE consistently outperforms other methods on RealEstate10K and matches GTA on Objaverse when constant intrinsics make intrinsic information unimportant.This supports the benefit of encoding complete camera information while reducing to GTA-like behavior when intrinsics are unnecessary.
- 4.3 Attention-Level Intrinsics Conditioning: On intrinsics-augmented RealEstate10K and Objaverse, PRoPE outperforms all alternative conditioning techniques.The augmented datasets vary zoom factors or fields of view to test camera-intrinsics conditioning.
- 4.3 Attention-Level Intrinsics Conditioning: Hybrid conditioning combines attention-level relative encodings with token-level CamRay, whose local camera-frame rays encode intrinsics without an absolute coordinate system.CamRay can be combined with relative methods while preserving global frame invariance.
- 4.3 Attention-Level Intrinsics Conditioning: PRoPE is comparable to or better than GTA+CamRay despite using only attention-level conditioning, while both PRoPE and GTA benefit from additional CamRay input.GTA benefits more from CamRay, consistent with standard SE(3)-based GTA lacking intrinsic information.
- 4.5 Out-of-distribution Robustness: PRoPE improves robustness to longer test sequences and unseen focal lengths, whereas adding CamRay to PRoPE hurts intrinsics extrapolation.The out-of-distribution evaluations vary input-view count and focal length relative to training.
- 4.6 Task Generalization: PRoPE significantly improves multiview understanding for stereo depth estimation and spatial cognition, with spatial-cognition performance continuing to improve as test-time views increase.Plücker raymaps do not show the same increasing-view trend, while replacing them with CamRay also improves results.
5 Conclusion and Future Work
The paper concludes that relative camera positional encodings, especially PRoPE, improve multiview transformers by capturing both camera intrinsics and extrinsics. It identifies numerical stability, distorted cameras, and multi-frequency encoding as directions for future work.
- Relative camera encodings that capture both intrinsics and extrinsics improve multiview transformers across settings and tasks.
- Directly multiplying projective matrices with Q/K/V vectors may create numerical-stability issues, including ill-conditioned matrices from telephoto focal lengths.
- Future work could extend PRoPE to distorted camera models through per-patch projective approximations.
- Multi-frequency encoding of camera parameters remains challenging because projective transforms are non-commutative, as with relative SE(3) attention methods.
A.1.1 LVSM-based Model Details
The LVSM-based experiments follow the original implementation where possible, but use smaller training configurations for most experiments and a larger setup for scaling studies.
- The experiments adhere to the original LVSM implementation specifications across most settings.Complete configuration details are delegated to the original LVSM paper.
- All standard experiments train at 256×256 resolution without additional higher-resolution fine-tuning.
- Most experiments use six transformer blocks, an MLP channel dimension of 1024, two GPUs, and a total batch size of 4.These settings differ from the original paper’s batch size of 512.
- Scaling experiments use 12 transformer blocks, retain the 3072-dimensional MLP, and train on eight GPUs with a total batch size of 64.
A.1.2 UniMatch Modification Details
UniMatch receives camera information in both its cross-attention and self-attention transformers using the paper’s formulation, which reduces to RoPE for single-view inputs.
- Camera information is injected into UniMatch’s cross-attention and self-attention transformers on the Q/K/V/O vectors.
- The formulation exactly falls back to RoPE in the single-view scenario, allowing use in both UniMatch transformer networks.
A.1.3 Spatial Cognition Model Details
The spatial cognition model classifies which image-camera pair is geometrically inconsistent, using token-level outputs aggregated into pair probabilities.
- The task is formulated as classification, with one class for each input image-camera pair and the target class identifying the inconsistent pair.
- The model modifies the final linear layer to output one scalar per token, averages outputs within each pair, and applies softmax to obtain bad-pair probabilities.
- Training optimizes identification of the inconsistent pair with cross-entropy loss.
A.2.1 Ablating PRoPE
The ablations identify camera projective relationships as PRoPE’s crucial component, while patch-coordinate modeling adds further understanding of vision-transformer patch relationships. Additional experiments show PRoPE’s benefits across model integrations, intrinsic settings, and related tasks.
- PRoPE ablation: Projective camera relationships alone already yield strong performance, establishing DProj_t as a crucial PRoPE component.PRoPE also allocates half of its feature channels to each of its two terms.
- PRoPE ablation: Adding RoPE further improves patch-relationship understanding, which operates at the minimal-unit level of vision-transformer patches.The ablation compares the projective relationship term with the relative patch-coordinate term.
- Model integration: Adding PRoPE to CAT3D improves the original multiview diffusion model with zero additional parameters and negligible computational overhead.The result is reported in the CAT3D integration study.
- Generalization and tasks: Additional experiments evaluate PRoPE under out-of-distribution intrinsics for novel view synthesis and stereo depth estimation, alongside qualitative results for both tasks.The novel-view-synthesis study uses LVSM with augmented 1–3× zoom-in intrinsics, while the stereo-depth study uses UniMatch with reduced training resources.
- Comparison with GTA: PRoPE consistently outperforms GTA, and both PRoPE variants outperform GTA+CamRay despite encoding the same information and sharing global-frame invariance.These results support the importance of attention-level intrinsic conditioning in PRoPE.