PaperlayerView exact revision on arXiv

Source-linked AI summary

MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation

Zhijian Qiao, Xinjiang Wang, Jiajie Chen, Haoming Huang, Meng Li, Chih-Chung Chou, Jing Wang, Shaojie Shen

arXiv:2609.01252v1cs.CVcs.RO

TL;DR

Existing geometry-aware positional encodings struggle to represent real-world metric translation without destabilizing attention or discarding physical scale. MeRoPE introduces a norm-preserving relative camera encoding with metric-pose and cross-view geometric components, achieving stronger camera control across nuScenes and PanShot.

  • Problem

    Existing encodings struggle to represent real-world metric translation because raw displacements can destabilize attention while normalization discards physical scale.

  • Method

    MeRoPE combines norm-preserving metric relative-pose encoding, orthogonal ray-orientation rotations, metric translation phases, and disparity-anchored cross-view rotations.

  • Results

    MeRoPE achieves the best joint rotation–translation consistency on nuScenes and PanShot, including the lowest CamMC across both benchmarks.

  • Takeaways & Limitations

    Norm-preserving metric camera encoding provides a stable basis for camera control across physical baselines and camera optics.

  • Takeaways & Limitations

    Homogeneous projective encodings exhibit baseline-dependent attention-logit and feature-norm growth in large-baseline settings.

Abstract

from arXiv · show

In camera-controlled video generation, geometry-aware positional encodings condition tokens on camera extrinsics and per-token viewing rays. Existing schemes, however, have a scale-dependent failure mode on real-world metric camera trajectories: homogeneous projective encodings cause attention logits and feature norms to grow unbounded with physical translation baselines. We propose MeRoPE (Metric Rotary Position Embedding), a norm-preserving relative camera encoding for attention. MeRoPE encodes relative orientations between calibrated viewing rays with orthogonal rotation blocks, maps raw metric displacements into multi-frequency rotary phases, and adds a disparity-anchored correspondence prior along the epipolar arc. This design strictly preserves feature norms, bounds pre-softmax attention logits regardless of the physical translation scale, and maintains exact invariance to global rigid coordinate changes. Across nuScenes and PanShot, which cover large-baseline trajectories and diverse camera optics, respectively, MeRoPE achieves stronger camera control than prior encodings, with the best consistency between generated camera motion and conditioning poses in both rotation and translation. Code will be made publicly available.

1 INTRODUCTION

MeRoPE addresses the instability of metric camera encodings by preserving full metric relative-pose information while controlling scale-dependent attention growth. It combines orthogonal ray and translation operators with disparity-anchored spherical rotations, achieving improved camera control across two demanding benchmarks.

  • Homogeneous projective encodings make attention logits and feature norms grow unbounded with physical translation baselines, allowing translation to suppress visual content.
  • The design goals are full depth-independent metric relative-pose dependence, strict per-token factorization, and norm-preserving attention actions.Theorem 1 shows that all three cannot be satisfied simultaneously by a continuous finite-dimensional encoding sensitive to metric translation.
  • MeRoPE introduces a norm-preserving encoding with full metric relative-pose dependence for camera-controlled video generation.
  • MeRoPE uses ray-local minimum-rotation frames and orthogonal blocks for relative ray orientation, while mapping metric displacements into multi-frequency rotary phases.
  • Disparity-anchored rotations along spherical epipolar arcs encode cross-view correspondence hypotheses without restricting token interactions or relying on dataset-specific metric-depth intervals.
  • MeRoPE achieves the best joint rotation–translation consistency on large-baseline nuScenes and camera-diverse PanShot.

2 RELATED WORK

Related work develops increasingly geometry-aware camera positional encodings, progressing from camera-level poses and projective frustums toward ray-based and depth-aware representations. The paper organizes these approaches by metric dependence, per-token factorization, and norm preservation.

  • Camera conditioning pathways include feature fusion and adapters that inject camera extrinsics, multi-scale features, or per-pixel Plücker rays into generative backbones.
  • Early geometry-aware positional encodings operate on camera-level extrinsic transformations or projective viewing frustums, with later methods incorporating camera intrinsics.
  • The taxonomy evaluates encodings by depth-independent full metric relative pose, strict per-token factorization, and norm-preserving attention action.
  • Homogeneous and projective encodings preserve metric dependence and factorization but violate norm preservation because translation blocks are non-orthogonal.

3 METHOD

MeRoPE formulates camera-conditioned attention around orthogonal, metric-aware pose operators and adds bounded spherical disparity anchors for cross-view relations. The construction preserves feature norms while representing ray orientation, translation, and correspondence geometry.

  • 3.1 GEOMETRY-AWARE ATTENTION AND METRIC-SCALE LIMITATION: Geometry-aware attention uses a pairwise operator U_ab to modulate query–key comparisons and value aggregation for patches from query and key cameras.
  • 3.1 GEOMETRY-AWARE ATTENTION AND METRIC-SCALE LIMITATION: Homogeneous relative-pose attention contains an unnormalized translation inner product, causing logits and aligned value features to scale unbounded with baseline length.
  • 3.1 GEOMETRY-AWARE ATTENTION AND METRIC-SCALE LIMITATION: As baselines grow, UCPE shifts attention toward distant early frames, whereas MeRoPE avoids this baseline-driven temporal-attention shift.
  • 3.2 ORTHOGONAL METRIC POSE ENCODING: MeRoPE decomposes relative pose into orthogonal ray-local rotation and query-frame metric translation operators.
  • 3.2 ORTHOGONAL METRIC POSE ENCODING: Calibrated viewing rays receive minimum-rotation local frames, whose relative rotation captures 3D orientation between query and key rays while preserving feature norms.
  • 3.2 ORTHOGONAL METRIC POSE ENCODING: Each translation coordinate becomes multi-frequency rotary phases, and the resulting direct-sum operator preserves feature norms and prevents baseline-scaled logits.
  • 3.3 DISPARITY-ANCHORED SPHERICAL ENCODING: Disparity anchors sample a bounded great-circle arc from the transformed key-ray direction to the epipole, representing possible scene-depth correspondences without mutually exclusive predictions.
  • 3.3 DISPARITY-ANCHORED SPHERICAL ENCODING: Degenerate epipolar configurations collapse all anchor directions to the transformed key-ray direction to avoid division by zero.

4 EXPERIMENTS

Experiments evaluate MeRoPE on large-baseline driving and camera-diverse settings, showing strong camera controllability and complementary contributions from its encoding components.

  • Experimental setup: MeRoPE is evaluated on large-baseline nuScenes driving videos and multi-lens PanShot camera control.The experiments cover trajectory diversity, matched-backbone comparisons, and optics ranging from pinhole to fisheye cameras.
  • nuScenes pose control: 2.32 CamMC is MeRoPE’s nuScenes result, versus 2.45 for UCPE, the strongest baseline.MeRoPE also achieves the best rotation accuracy and lowest FID and FVD point estimates.
  • Ablation: Removing metric translation substantially worsens tr% and CamMC, while ray-local rotation reduces rotation error from 1.66° to 1.37°.Disparity anchors improve both AUC metrics and lower tr% and CamMC without materially changing rotation.
  • Qualitative results: Qualitative nuScenes comparisons show commanded original, left-turn, and right-turn trajectories alongside paths recovered from generated videos.Frames are sampled at t ∈ {0, 16, 32, 48}.
  • PanShot camera diversity: On PanShot, MeRoPE has the lowest TransErr and CamMC across both conditioning-pose settings, while UCPE and MeRoPE remain close on RotErr.Raw conditioning displacements increase TransErr and CamMC for all methods, but MeRoPE has the smallest increase.

5 CONCLUSION

The conclusion presents MeRoPE as a norm-preserving metric camera encoding that resolves a formal trade-off through query-camera grouping and geometric cross-view rotations. Across nuScenes and PanShot, it reports the strongest joint camera-control consistency, while identifying attention overhead as a future-work limitation.

  • Contribution: MeRoPE relaxes strict per-token factorization through query-camera grouping to preserve metric pose information and feature norms.The design combines orthogonal relative-ray and raw-translation blocks with disparity-anchored spherical rotations.
  • Conclusion: MeRoPE achieves the lowest CamMC across nuScenes and PanShot, with the lowest nuScenes FID/FVD point estimates and lowest PanShot TransErr under both conditioning settings.These results support norm-preserving metric camera encoding across physical baselines and camera optics.
  • Limitation and future work: Query-camera grouping introduces additional local camera-attention cost, motivating future work to reduce overhead while retaining metric sensitivity.
  • Formal trade-off: No strictly per-token unitary or orthogonal positional operator can completely capture metric relative translation.This result formalizes the three-way trade-off motivating MeRoPE’s architecture.

A.2 THE IMPOSSIBILITY THEOREM

The theorem proves that continuous, finite-dimensional, unitary per-token encodings cannot retain sensitivity to nonzero metric translations while satisfying the stated representation assumptions.

  • Theorem statement: Theorem 1 establishes translation blindness for any continuous unitary relative per-token encoding satisfying hypotheses (H1)–(H4).For every pure metric translation, the operator becomes trivial.
  • Proof structure: The homomorphism assumptions make the encoding a group representation of SE(3), while pure translation operators form a commuting unitary family.The proof first derives representation structure, then applies commutativity of the translation subgroup.
  • Spectral reduction: Simultaneous unitary diagonalization reduces each translation operator to continuous unit-modulus characters of R3, yielding a finite set of spatial frequencies.The continuous characters have the form of spatial plane waves, and finite feature dimension makes the active spectrum finite.
  • Spherical-orbit contradiction: Rotation conjugation maps every frequency ω to Rω, forcing the finite frequency spectrum to be invariant under all continuous 3D rotations.The semi-direct product relation between rotations and translations produces this orbit constraint.
  • Conclusion: Because a finite SO(3)-invariant frequency set can contain only zero, the positional operator cannot distinguish any nonzero relative translation.The arbitrary pose then reduces to its pure-rotation operator, completing the contradiction.

B.1 ARCHITECTURE AND TRAINING CONFIGURATIONS

The implementation uses a four-way orthogonal head partition and evaluates commanded-versus-recovered camera motion with gauge-adjusted pose metrics under dataset-specific scale treatments.

  • Architecture: The primary Wan2.2 TI2V-5B implementation partitions 128 head channels into disparity, rotation, translation, and native sub-blocks.The allocation is ddisp + drot + dtrans + dnative = 36 + 36 + 24 + 32.
  • Training configuration: Training uses AdamW with learning rate 1 × 10^-5, bfloat16 precision, and 64 NVIDIA H20 GPUs for 20k iterations at global batch size 64.PanShot models instead use 8 H20 GPUs for 10k iterations.
  • Pose recovery: Camera poses are recovered from generated frames, then commanded and estimated trajectories are expressed relative to the first frame to remove the absolute SE(3) gauge.This relative formulation supports rotation and translation consistency evaluation.
  • Evaluation protocol: Driving evaluation averages scores over 384 test clips, while PanShot averages over 272 jitter-filtered clips; pairwise AUC is micro-averaged over within-clip frame pairs.The reported metrics include rotation, translation, joint-pose AUC, and camera motion consistency measures.
  • Scale handling: Driving translation estimates receive one least-squares global scale alignment, whereas CamMC and TransErr independently normalize trajectories by maximum translation norms.Rotation metrics are invariant to translation scaling, and the two translation treatments are not interchangeable.

C.1 ARCHITECTURAL DESIGN AND LAYER-WISE MODULATION

Ablations examine compression, translation frequency, injection schedule, and integration style, showing that multi-frequency metric encoding and parallel camera attention provide the strongest overall balance.

  • Ablation design: Table 5 varies adapter compression, translation frequency, injection schedule, and integration style around the default MeRoPE configuration.Design rows change one factor at a time with compression fixed at C = 4.
  • Adapter compression: C = 4 retains similar CamMC to C = 2 while using half the adapter capacity, although C = 2 achieves the strongest pose scores.The compression ratio changes camera-branch width and head count while preserving 128-dimensional head size.
  • Translation frequency: Single-frequency translation leaves directional AUC nearly unchanged but degrades tr% and CamMC, indicating unreliable translation control across evaluated trajectories.The variant changes only translation frequency while keeping the 128-dimensional channel allocation fixed.
  • Injection schedule: Peak 11,13 preserves rotation accuracy and strong directional AUC while increasing CamMC from 2.32 to 2.43 and reducing camera-branch executions by 87%.Blocks 11 and 13 contribute over half of the residual amplitude; the default alternating schedule uses 15 blocks.
  • Integration style: The parallel CamSA branch provides a better overall balance than ReRoPE, which improves tr% but weakens rotation and CamMC.CamSA uses a dedicated parallel camera self-attention branch, whereas ReRoPE inserts low-frequency camera rotary channels into native self-attention.

C.2 COST OF RELAXING PER-TOKEN FACTORIZATION

Relaxing strict per-token factorization through query-camera grouping raises isolated camera-attention costs, but much of this overhead is amortized in the full backbone. The section also examines retrieved-history pathways, finding stronger reported results for History SA than History CA.

  • Computational design: MeRoPE constructs query-dependent key/value representations by expressing metric translation and disparity anchors in each query-camera frame.This relaxes strict per-token factorization, where each token is transformed only once.
  • Computational cost: Relative to UCPE, MeRoPE requires about 2.5× isolated-block latency and 3× peak memory, while full-model overhead is substantially smaller.The benchmark measures both a complete isolated CamSA block and the full production model.
  • Computational cost: Full-model throughput decreases by 3.3% for training and 5.9% for denoising, with peak-memory increases of 2.7% and 0.3%, respectively.GTA and UCPE have nearly identical system costs under the matched schedule.
  • Computational cost: The extra local cost comes from separate transformed K/V tensors for each query-camera group rather than additional attention dot products.Smaller group batches can reduce peak memory, while fused grouped attention could reduce expanded K/V storage, memory, and latency.
  • Retrieved-history pathways: History SA concatenates retrieved reference tokens with current video tokens for joint self-attention, whereas History CA supplies retrieved latents as cross-attention K/V.Both pathways use temporal tagging, and training includes current-image dropout and lateral-conflict signals to address retrieval and pose mismatch.
  • Retrieved-history pathways: Both history variants provide modest pose-control gains, while History SA achieves the best CamMC, FID, and FVD in Table 7.The qualitative comparison reports that History SA preserves retrieval-supported late-frame geometry, whereas History CA follows the no-history layout.

D ADDITIONAL QUALITATIVE RESULTS

Additional qualitative comparisons evaluate commanded and VGGT-Ω-recovered camera paths alongside generated frames across multiple nuScenes scenes. In the reported fixed clips, MeRoPE preserves more coherent late-frame geometry.

  • Matched-backbone comparison: Figure 9 presents three fixed clips separately, with each row showing commanded and VGGT-Ω-recovered paths plus frames at t ∈{0, 16, 32, 48}.The three clips are separated for readability, while Table 2 reports aggregate evaluation.
  • Matched-backbone comparison: MeRoPE preserves more coherent late-frame geometry in the three fixed clips shown in Figure 9.The figure presents the same matched-backbone protocol across the three scenes.
  • Additional nuScenes scenes: Figure 10 extends the qualitative comparison to three additional nuScenes scenes, pairing commanded and VGGT-Ω-recovered paths with frames at t ∈{0, 16, 32, 48}.The left plot compares commanded yellow solid paths with VGGT-Ω-recovered cyan dashed paths.
Loading 2609.01252v1…