PaperlayerView exact revision on arXiv

Source-linked AI summary

HRoPE: Homography Relative Positional Encoding

Huachen Gao, Yang Yang, Xiaoqian Liang, Xingyu Chen, Youjia Zhang, Qi Zhang, Baorui Ma, Anpei Chen, Wei Yang

arXiv:2610.28284v1cs.CV

TL;DR

Existing geometric positional encodings describe camera or ray geometry but leave local surface orientation unmodeled, despite its role in cross-view neighborhood appearance. HRoPE represents each patch with a plane-induced transform and uses relative homographies in attention. It consistently improves novel view synthesis over evaluated positional encodings across multiple benchmarks.

  • Problem

    Existing positional encodings anchor geometry to sensors and omit local surface orientation, which determines how patch neighborhoods deform across views.

    Source: §1 · pp. 1–2

  • Method

    HRoPE assigns each patch a local plane and uses pairwise plane-induced homographies to modulate attention, with transformation parameters predicted from patch features.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5

  • Results

    HRoPE improves novel view synthesis over evaluated positional encodings on RealEstate10K, CO3Dv2, Objaverse, and DL3DV.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The findings support local surface geometry as a useful inductive bias for multi-view attention.

    Source: §5 · p. 11

    Per-patch, per-view plane predictions do not formally guarantee that the same physical surface is encoded identically across views.

    Source: §5 · p. 11

Abstract

from arXiv · show

Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry. Such sensor-centric formulations, however, describe only where a token is observed from, leaving the rich surface geometry of the scene region depicted by the token. In this paper, we advocate a shift toward surface-aware positional encoding that explicitly grounds patch tokens in the geometry of physical surfaces. We introduce HRoPE, which represents the local geometry of each patch as a slanted 3D plane, following the slanted support-window assumption of classical multi-view stereo. We inject token-level, plane-induced homographies into attention as relative transformations. This equips the transformer with a geometric prior for rectifying perspective distortions and aligning patch features across viewpoints. Our encoding requires no ground-truth geometry: the plane parameters can be predicted end-to-end under a view-synthesis objective alone, or derived from foundation models. Extensive experiments on RealEstate10K, CO3Dv2, and DL3DV show that our best variant of HRoPE consistently outperforms the evaluated geometric positional encodings for novel view synthesis. Extensive ablations and analysis show the effectiveness of the proposed encoding. Please refer to our project page at: https://gaohchen.github.io/HRoPE

1 INTRODUCTION

Existing geometric positional encodings anchor token geometry to sensors, leaving local surface orientation unmodeled. HRoPE represents each patch with a tangent plane and uses relative homographies in attention; it consistently outperforms evaluated geometric encodings on multiple novel-view-synthesis benchmarks.

1 INTRODUCTION

  • Existing positional encodings describe sensor or ray geometry but leave local surface structure and orientation unmodeled.Multi-view matching compares neighborhoods whose appearances deform with the surface, so position or depth alone imposes a fronto-parallel assumption on attention.Source: §1 · pp. 1–2
  • The authors frame local surface orientation as the missing degree of freedom: existing encodings capture patch position but not its first-order cross-view appearance transformation.They argue that learning to compensate for this bias requires sufficient training data and compromises efficiency and generalization.Source: §1 · p. 2
  • HRoPE assigns each patch a tangent plane and uses plane-induced homographies to make attention comparisons depend on relative cross-view transformations.Per-patch transformation parameters can be learned jointly with view synthesis using that objective alone, or predicted using pretrained features.Source: §1 · p. 2
  • HRoPE consistently outperforms evaluated view-, projection-, and ray-based encodings on RealEstate10K, CO3Dv2, and DL3DV novel view synthesis.It also improves generalization to unseen zooms and wider baselines.Source: §1 · p. 3
  • Ablations show that geometry must condition token interactions multiplicatively; injecting the same information as additive features sharply degrades performance.The paper also reports transfer to a multi-view diffusion model and a spatial-reasoning MLLM, improving both tasks.Source: §1 · p. 3

2 RELATED WORK

Prior geometric positional encodings range from camera-relative transforms to token-level ray or point geometry, while HRoPE adds an explicit local-plane parameterization for cross-view neighborhood deformation.

Geometric Positional Encoding

  • Camera-relative encodings use relative extrinsics or projection matrices, while token-level ray methods add depth or ray-based transformations.These approaches describe camera relationships or point- and ray-level geometry; HRoPE adds local-plane orientation to model neighborhood deformation.Source: §2 · p. 3

Multi-view Transformers

  • Multi-view transformers associate image tokens across views and support applications including novel view synthesis, which is HRoPE’s primary evaluation setting.The paper evaluates HRoPE with the LVSM backbone and plane-induced positional encoding.Source: §2 · p. 3

Slanted Support Windows

  • Slanted support windows address fronto-parallel bias by estimating a local disparity plane and evaluating matching costs over slanted neighborhoods.Local planar geometry induces a homography between corresponding image neighborhoods across views.Source: §2 · p. 3
  • HRoPE brings the slanted-window matching principle into positional encoding by incorporating per-patch, plane-induced transformations into multi-view attention.The overview depicts per-patch parameter prediction and token-specific transformations in cross-view attention.Source: §2 · p. 4, Figure 2 · p. 4

3 METHOD

HRoPE replaces view-level or ray-based geometric cues with patch-specific plane transforms that modulate attention through pairwise relative homographies. The method supports predicted and calibrated variants, with parameters learned or derived from patch features.

3.1 PRELIMINARIES: GEOMETRIC POSITIONAL ENCODINGS

  • HRoPE represents local geometry with a tangent plane, allowing positional encoding to account for how a patch neighborhood deforms across views.Prior camera-level projection transforms are shared by all tokens in a view and cannot encode local surface variation.Source: §3.1 · pp. 4–5

3.2 HOMOGRAPHY RELATIVE POSITIONAL ENCODING

  • For a patch, the plane-induced homography maps points on its assumed plane into the image using the plane and camera geometry.The same homography describes cross-view neighborhood deformation when a local neighborhood is approximately planar.Source: §3.2 · p. 5
  • HRoPE uses per-patch plane-induced transformations whose pairwise composition modulates query-key comparisons; the composition is not guaranteed to represent a physical warp for patches on different planes.When both patches lie on the same physical plane, the product recovers the plane-induced homography.Source: §3.2 · p. 5
  • Attention uses token-specific relative transforms, while standard 2D RoPE encodes relative image positions in the remaining feature channels.The relative factor is the identity when two patches in the same view have equal predicted transforms.Source: §3.2 · p. 5
  • The predicted transformation’s inverse is computed analytically from its rotation-plus-rank-one-update structure, avoiding explicit matrix-inversion routines.The method clamps parameters and the determinant-related denominator to keep the transformation numerically well-conditioned.Source: §3.2 · p. 6

3.3 INSTANTIATION

  • The predicted instance regresses patch transformations from features using a lightweight head, with parameters predicted only for observed input views.Features may come from a scratch-trained predictor, DINO, or VGGT; the primary implementation uses VGGT features.Source: §3.3 · p. 6
  • With camera calibration, the calibrated instance retains PRoPE’s projection and predicts a local plane frame for each patch.Its relative factor combines local-frame geometry with camera projections and reduces to PRoPE when all patches share a common frame.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments compare HRoPE with geometric positional encodings for novel view synthesis and examine its parameterization, local geometry, and robustness to shifts in view sampling and camera intrinsics.

4 EXPERIMENTS

  • The evaluation applies positional encodings to LVSM models trained separately on CO3Dv2, RealEstate10K, and Objaverse, with a larger model tested on DL3DV using 2, 4, or 8 reference views.The models take two reference views to synthesize a target view in the main setup.Source: §4 · p. 6, §4.1 · p. 7

4.1 NOVEL VIEW SYNTHESIS

  • HRoPE gains 0.53, 1.27, and 1.03 dB in PSNR over PRoPE on CO3Dv2, RealEstate10K, and Objaverse, and also outperforms it on DL3DV with 2, 4, and 8 views.The reported gains on the first three datasets accompany higher SSIM and lower LPIPS.Source: §4.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.2.1 PARAMETERIZATION AND FEATURE SOURCES

  • With frozen VGGT features or a predictor trained from scratch, HRoPE has the highest PSNR among evaluated transformations and outperforms an unconstrained 4×4 transform.The comparison also finds the homography form performs better than a 768-dimensional additive encoding using the same VGGT features.Source: §4.2.1 · pp. 7–8
  • HRoPE gains do not depend on one pretrained backbone, while stronger pretrained features improve performance and the geometry-aware VGGT backbone achieves the highest PSNR.Fine-tuned DINOv2† features acquire implicit 3D awareness in VGGT’s 3D pretraining pipeline, helping explain their stronger performance than off-the-shelf DINOv2.Source: §4.2.1 · p. 9

4.2.2 ANALYSIS OF LOCAL SURFACE GEOMETRY

  • On RealEstate10K, adding the tangent-frame transform F to PRoPE raises PSNR from 26.142 to 27.115 and SSIM from 0.826 to 0.840, while lowering LPIPS from 0.111 to 0.104.HRoPE with predicted transformations further improves all three metrics over calibrated P·F.Source: §4.2.2 · p. 9, Table 3 · p. 9
  • Predicted plane orientations are locally coherent in approximately planar regions and qualitatively consistent across reference views, without direct plane supervision.Source: §4.2.2 · pp. 9–10, Figure 5 · p. 10
  • On ASE, ground-truth geometry improves HRoPE PSNR from 28.555 to 28.751 and also improves SSIM and LPIPS over VGGT-predicted parameters.This comparison uses the ASE oracle protocol with ground-truth depth, surface normals, and camera parameters.Source: Table 3 · p. 9, §4.2.2 · p. 10

4.2.3 OUT-OF-DISTRIBUTION GENERALIZATION

  • As target focal length increases from 1× to 5×, HRoPE achieves the lowest LPIPS across evaluated focal lengths, while calibrated P·F degrades more at large zoom.Source: §4.2.3 · p. 10

5 CONCLUSION

HRoPE uses local surface geometry to modulate multi-view attention, and experiments report improved novel view synthesis over evaluated positional encodings. The authors identify the lack of a formal guarantee of cross-view surface consistency as a limitation.

5 CONCLUSION

  • HRoPE represents each patch with a local plane and uses plane-induced homographies to modulate attention through relative transformations.Its transformation parameters are predicted from patch features and learned jointly with the view-synthesis model.Source: §5 · pp. 10–11
  • Figure 6 evaluates RealEstate10K generalization as target-camera zoom increases from 1× to 5× and under soft extrapolation with an enlarged input-view baseline.The caption identifies PSNR and LPIPS as the metrics but does not state the comparison outcome.Source: Figure 6 · p. 11
  • HRoPE improves novel view synthesis over evaluated positional encodings on RealEstate10K, CO3Dv2, Objaverse, and DL3DV.A calibrated extension also improves PRoPE by incorporating local planar frames while retaining its camera projections.Source: §5 · p. 11
  • Per-patch, per-view plane prediction does not formally guarantee that the same physical surface is encoded identically across views.The authors report learned planes are cross-view consistent in practice, while a hard guarantee remains future work.Source: §5 · p. 11

A APPENDIX

The appendix expands the geometric positional-encoding notation, derives the plane-induced homography, and details how it is applied across attention heads.

A APPENDIX

  • Appendix A.1 expands notation and derivations for the geometric positional encodings summarized in Section 3.1.Source: A APPENDIX · p. 17
  • Appendix A.2 derives a plane-induced homography that maps a pixel in view j directly to a pixel in view i and explains why a purely projective encoding cannot express this correspondence.Source: A APPENDIX · p. 17
  • Appendix A.3 writes out the block-diagonal operator that applies the homography across the head dimension.Source: A APPENDIX · p. 17

A.1 BACKGROUND ON GEOMETRIC POSITIONAL ENCODINGS

This section sets out notation for patch-token features and calibrated cameras, then reviews how positional encodings represent relative image positions and camera geometry.

A.1 BACKGROUND ON GEOMETRIC POSITIONAL ENCODINGS

  • For a patch token u in view i, q_i,u, k_i,u, and v_i,u denote its query, key, and value vectors; calibrated camera quantities include intrinsics and pose.The camera pose is represented by R_i and t_i, distinct from HRoPE's patch-specific predicted parameters.Source: §A.1 · p. 17
  • RoPE applies position-dependent rotations so attention depends on relative displacement, with its usual 2D extension applied separately along the image axes.Source: §A.1 · p. 17
  • GTA and CaPE encode relative camera extrinsics, while PRoPE also incorporates intrinsics through projection matrices and its relative projective factor is world-frame invariant.The invariance follows because a shared rigid transform of the world frame leaves the relative projective factor unchanged.Source: §A.1 · p. 17

A.2 DERIVATION OF THE PLANE-INDUCED HOMOGRAPHY

The derivation uses a local plane to resolve the depth ambiguity of a source viewing ray and obtain a pixel-to-pixel homography. It also distinguishes conditions under which geometry-derived relative transforms are world-frame invariant.

A.2 DERIVATION OF THE PLANE-INDUCED HOMOGRAPHY

  • Substituting the ray–plane intersection depth into projection yields a homography that maps points on the plane between the two views.The derivation combines the projection and camera-transform equations and eliminates the depth-dependent scale.Source: §A.2 · pp. 17–18
  • A source pixel identifies a viewing ray but leaves depth unknown, so camera-only relative projection generally constrains its counterpart to an epipolar line rather than a unique pixel.A plane hypothesis determines ray–plane depth when the intersection is finite and in front of the source camera.Source: §A.2 · p. 18
  • Unlike PRoPE’s camera-only relative projection, a local plane resolves depth ambiguity along a viewing ray and induces a pixel-to-pixel mapping on that plane.Source: §A.2 · p. 18
  • Image-predicted HRoPE transforms and relative factors remain unchanged under a world-frame change, but full attention invariance also requires world-frame-independent input features.For geometry-derived transforms, invariance is not generally guaranteed for different patch planes; the GT instantiation uses a common reference-camera frame and scale.Source: §A.2 · p. 18

A.3 THE FULL BLOCK-DIAGONAL ENCODING MATRIX

The full HRoPE operator combines repeated per-patch geometric transforms with standard 2D rotary blocks. Input-view tokens use their predicted geometry, while target-view tokens use an intrinsics-only geometric block.

A.3 THE FULL BLOCK-DIAGONAL ENCODING MATRIX

  • Each D-dimensional query, key, or value is split into D/2 geometric coordinates and two D/4 rotary blocks for patch x and y positions.The rotary blocks use D/8 frequencies per axis.Source: §A.3 · p. 18
  • Input-view tokens use Hi,u = KiAi,u, whereas target-view tokens have no predicted plane and reduce to Ki alone.For target-view tokens, Ai,u = I; their 2D rotary blocks remain active.Source: §A.3 · p. 19
  • The geometric block repeats the 3 × 3 per-patch homography D/6 times, so it acts on the D/2 geometric coordinates.Each copy operates on a 3-vector; the illustrated setting uses D/6 = 8.Source: §A.3 · p. 19
  • The geometric blocks contract to a relative homography, while the rotary blocks recover the standard 2D-RoPE relative property.The x and y rotary blocks encode in-image patch position alongside cross-view geometry.Source: §A.3 · p. 19

A.4 INTEGRATING HROPE INTO MULTI-VIEW ATTENTION

HRoPE changes multi-view self-attention using token-specific linear operators while leaving the other transformer components unchanged. Its attention logits use relative homographies, and the aggregated output is mapped into the query token’s coordinate frame.

A.4 INTEGRATING HROPE INTO MULTI-VIEW ATTENTION

  • For N views with P patches each, the layer receives NP tokens, per-patch transforms {Ai,u}, and camera intrinsics {Ki}.It constructs Hi,u from these quantities and applies token-specific operators to queries, keys, and values.Source: §A.4 · p. 19
  • The attention logit for tokens (i, u) and (j, v) contains the relative operator Hi,uH−1j,v.The per-patch transformations are applied to keys and values as part of the attention computation.Source: §A.4 · p. 19
  • After attention aggregation, the output is transformed by Hi,u into the query token’s coordinate system.The inverse of Ai,u is computed in closed form.Source: §A.4 · p. 19

A.5 ADDITIONAL EXPERIMENT DETAILS

The appendix details the datasets, training configuration, and stability measures used in the experiments. It also compares parameter sources for the RealEstate10K plane predictions.

A.5 ADDITIONAL EXPERIMENT DETAILS

  • Experiments use RealEstate10K, CO3Dv2, Objaverse, DL3DV, and Aria Synthetic Environments, with dataset-specific splits, view sampling, and preprocessing.The CO3Dv2 subset has 31k training and 1k test scenes across 52 categories; Objaverse retains 96k objects and renders 30 views per object.Source: §A.5 · p. 20
  • Aria Synthetic Environments supplies per-frame depth for deriving ground-truth planes, fitted by least squares over inverse-depth samples within each 14 × 14 patch.Patches with more than half invalid samples or median relative residual above 5% fall back to a fronto-parallel plane at median depth.Source: §A.5 · p. 20
  • Scene models train for 80k iterations at 336 × 336 resolution on four RTX 4090 GPUs, using a photometric objective plus perceptual loss.The perceptual weight is 0.5 for scene datasets and 0.1 for the object dataset.Source: §A.5 · p. 21
  • Training stability measures clamp predicted translation and inverse plane distance, normalize the plane normal, and construct an orthonormal rotation.The Sherman–Morrison denominator is clamped to 0.1, and entries of A and A−1 are clamped to [−5, 5].Source: §A.5 · p. 21
  • At the same cost of 33 GPU-hours, the learned predictor improves RealEstate10K PSNR from 25.282 to 26.419; DA3 reaches 27.137 PSNR in 48 GPU-hours and VGGT reaches 27.410 PSNR in 76 GPU-hours.All parameter-source variants use the same 80k-iteration schedule.Source: §A.5 · p. 21, Table 4 · p. 21

A.6 GENERALIZATION BEYOND ARCHITECTURES

The appendix evaluates HRoPE in a multi-view diffusion model and a spatial multimodal language model, and provides further qualitative novel-view comparisons. The reported results support HRoPE’s use beyond feed-forward decoding, with gains on spatial video reasoning alongside a temporal-task regression.

A.6 GENERALIZATION BEYOND ARCHITECTURES

  • HRoPE improves EscherNet on every reported metric at every input-view count, extending the evaluation beyond LVSM and feed-forward decoding.The experiment replaces EscherNet’s native CaPE with HRoPE while keeping the model and training recipe otherwise unchanged.Source: §A.6 · p. 21, Table 5 · p. 22
  • The MLLM experiment adds per-patch surface geometry to visual tokens while leaving text tokens and frame temporal ordering unchanged.HRoPE is applied to visual tokens in layers L−10 through L−2 of the 28-layer LLaVA-Video-7B backbone.Source: §A.6 · p. 22
  • On VSI-Bench, HRoPE reaches an average of 59.5, 1.8 points above the same backbone fine-tuned on identical data without it.Gains include absolute distance (+8.0), relative direction (+7.1), and route planning (+5.7), while appearance order falls 6.9 points.Source: §A.6 · pp. 22–23, Table 6 · p. 22

A.7 ADDITIONAL QUALITATIVE RESULTS

  • On RealEstate10K and CO3Dv2, HRoPE qualitatively recovers sharper structure on oblique surfaces; on Objaverse, it produces cleaner geometry and more faithful appearance than CaPE.Figures 8–10 compare methods using the same inputs and backbone within each setting.Source: §A.7 · p. 23, Figure 8 · p. 24, Figure 9 · p. 25

Related papers

Loading 2610.28284v1…