Source-linked AI summary
ReRoPE: Repurposing RoPE for Relative Camera Control
Chunyang Li, Yuanbo Yang, Jiahao Shao, Hongyu Zhou, Katja Schwarz, Yiyi Liao
TL;DR
Pretrained video models need camera control that generalizes beyond fixed reference frames without costly architectural redesign. ReRoPE repurposes underused low-frequency temporal RoPE bands for relative camera geometry, and evaluations report superior camera accuracy and 3D consistency with high visual fidelity across V2V and I2V. Its I2V responsiveness can be constrained when pretrained subject-centering biases compete with user-defined trajectories.
Problem
Fixed-reference camera pose conditioning limits shift-invariant generalization, while integrating arbitrary-pair relative poses into pretrained video diffusion models remains costly and architecturally difficult.
Method
ReRoPE injects relative camera geometry into underutilized low-frequency temporal RoPE bands without modifying the transformer backbone or adding auxiliary camera encoders or regression heads.
Results
ReRoPE achieves superior camera accuracy and 3D consistency while maintaining high visual fidelity across V2V and I2V tasks.
Takeaways & Limitations
The framework provides a plug-and-play, training-efficient route to adapting pretrained video models for precise camera control while preserving their generative priors.
Takeaways & Limitations
In I2V, pretrained subject-centering biases can constrain trajectories that move a subject away from the center or out of frame.
Abstract
from arXiv · showhide
Video generation with controllable camera viewpoints is essential for applications such as interactive content creation, gaming, and simulation. Existing methods typically adapt pre-trained video models using camera poses relative to a fixed reference, e.g., the first frame. However, these encodings lack shift-invariance, often leading to poor generalization and accumulated drift. While relative camera pose embeddings defined between arbitrary view pairs offer a more robust alternative, integrating them into pre-trained video diffusion models without prohibitive training costs or architectural changes remains challenging. We introduce ReRoPE, a plug-and-play framework that incorporates relative camera information into pre-trained video diffusion models without compromising their generation capability. Our approach is based on the insight that Rotary Positional Embeddings (RoPE) in existing models underutilize their full spectral bandwidth, particularly in the low-frequency components. By seamlessly injecting relative camera pose information into these underutilized bands, ReRoPE achieves precise control while preserving strong pre-trained generative priors. We evaluate our method on both image-to-video (I2V) and video-to-video (V2V) tasks in terms of camera control accuracy and visual fidelity. Our results demonstrate that ReRoPE offers a training-efficient path toward controllable, high-fidelity video generation. See project page for more results: https://sisyphe-lee.github.io/ReRoPE/
1 Introduction
Camera control enables explicit viewpoint manipulation while preserving scene identity and temporal consistency, but fixed-reference pose conditioning limits generalization. ReRoPE addresses this gap by injecting relative camera information into underused low-frequency RoPE bands without changing the pretrained backbone.
- Motivation: Camera control is essential for directing viewpoints while preserving scene identity and temporal consistency in video generation.The paper highlights interactive content creation, gaming, and robotics simulation as applications requiring this capability.
- Limitations of Existing Methods: Fixed-reference 6-DoF pose conditioning ties trajectories to a global reference, limiting generalization to unseen trajectories or shifted reference frames.Existing methods commonly define camera rotation and translation relative to the first frame.
- Relative Camera Representation: Relative pose changes between arbitrary viewpoint pairs provide a local-motion representation aligned with RoPE’s relative positional design.RoPE makes attention depend on relative offsets through position-dependent rotations of hidden features.
- Integration Challenge: Existing camera-relative RoPE methods typically require training from scratch because they fundamentally repartition RoPE channels.This conflicts with the cost of retraining large-scale video diffusion models while preserving pretrained architectural priors.
- ReRoPE: ReRoPE injects relative camera information into low-frequency RoPE components that are redundant for positional differences in pretrained video diffusion models.The framework is designed as a plug-and-play adaptation rather than a new backbone architecture.
- Contribution: ReRoPE targets precise camera control while preserving pretrained generative priors across video-to-video and image-to-video tasks.The paper identifies low-frequency redundancy and validates the mechanism across three state-of-the-art backbones.
2 Related Work
Prior camera-control methods use latent manipulation, geometric guidance, or direct camera-parameter injection, each with distinct constraints. RoPE supplies a relative-positioning foundation for modeling camera transformations in video generation.
- Camera Control Generation: Latent-based camera-control methods manipulate model weights or learn patterns but often lack explicit geometric constraints for precise trajectory following.These methods include approaches based on LoRA and pattern learning.
- Camera Control Generation: Geometry-guided methods enforce consistency through noise warping or point-cloud projection but depend on depth estimation, which is difficult for dynamic scenes.Their effectiveness is therefore constrained by the quality of structural scene estimation.
- Camera Control Generation: Direct camera-parameter methods inject geometric conditioning through ray-based or parameter-based representations.Ray-based approaches use Plücker ray maps, while parameter-based approaches fuse encoded extrinsic information.
- Relative Positional Encoding: RoPE encodes relative position by rotating query and key vectors so attention depends on relative token distance rather than absolute indices.This property has supported RoPE’s adoption in language and vision transformers.
3 Preliminary
RoPE represents relative positions through rotations, while video transformers factorize those rotations across temporal and spatial axes. Camera-aware variants allocate channels to relative transformations, but their channel structure is difficult to merge with pretrained video RoPE.
- RoPE: RoPE rotates query and key vectors across independent 2D frequency planes to make attention depend on relative position.The rotation frequency is defined as ω_f = θ^-2f/d with θ = 10^4.
- RoPE: The rotation operator is a block-diagonal matrix whose diagonal blocks encode separate feature transformations.The encoded query and key vectors preserve an inner-product relation governed by relative offsets.
- RoPE for Videos: Video RoPE partitions the head dimension into disjoint temporal, height, and width bands and concatenates their rotations block-diagonally.A common choice assigns equal dimensions to the three axes, making attention depend on relative spatio-temporal offsets.
- Cameras as Relative Positional Encoding: Camera-aware positional encoding allocates a separate head-dimension band to relative camera transformations while retaining spatial location channels.Camera projection matrices are lifted to homogeneous form before being used in the transformation.
- Integration Challenge: The camera-pair transformation encodes relative geometry, but its channel-wise structure differs from spatial-temporal RoPE.This mismatch makes direct integration into pretrained video models difficult without disrupting generative priors.
4 ReRoPE
ReRoPE repurposes redundant low-frequency RoPE bands to encode relative camera information while preserving the remaining positional structure and pretrained generative behavior. The framework supports V2V and I2V through camera-conditioned temporal embeddings and targeted fine-tuning.
- Overview and setting: ReRoPE targets relative camera control in V2V and I2V generation settings.V2V adapts an input video to a target trajectory, while I2V synthesizes video from a reference frame under a target trajectory.
- Low-Frequency Redundancy of RoPE: The toy analysis shows high-frequency bands provide fine-grained temporal discrimination, whereas low-frequency bands remain nearly constant across standard video lengths.The low-frequency attention values approach the constant-vector dot product of 2, indicating underutilization for positional encoding.
- Low-Frequency Redundancy of RoPE: Across Wan 2.1, Wan 2.1-I2V, and CogVideoX, masking low-frequency bands preserves generation quality, while masking high-frequency bands significantly degrades performance.The masking experiment applies identity rotations to the selected low-frequency subspace across spatial and temporal RoPE components.
- ReRoPE: Repurposing RoPE for Camera Control: ReRoPE replaces only temporal low-frequency bands with relative camera information while retaining high-frequency temporal and spatial RoPE bands.The temporal dimension is targeted because latent frame counts are typically smaller than spatial resolutions, creating greater temporal redundancy.
- ReRoPE: Repurposing RoPE for Camera Control: The camera projection encodes relative transformations between camera pairs, with translation normalization used because this operation is non-norm-preserving.The relative transformation is defined between camera poses, and translation components are normalized to stabilize training.
- Training: Training fine-tunes the pretrained transformer for V2V and I2V while freezing the backbone except for self-attention layers.Optimization uses flow matching with MSE loss, AdamW, a 1e−5 learning rate, and gradient clipping of 0.05.
5 Experiments
ReRoPE is evaluated on V2V and I2V camera control, with comparisons covering baselines, qualitative behavior, and ablations of its design choices. Across these experiments, it combines accurate camera control with visual fidelity, faster convergence, and dynamic-subject synthesis.
- Experimental Setup: ReRoPE is evaluated on V2V and I2V camera-controlled video generation, including baseline comparisons and ablations of key design choices.The experiments cover both tasks and assess camera accuracy, visual quality, convergence, and component choices.
- V2V Comparison: ReRoPE maintains tighter V2V trajectory alignment than baselines, supporting precise geometric control and consistent visual quality.In Fig. 5, baseline trajectories deviate from ground truth while ReRoPE remains closely aligned.
- V2V Comparison: ReRoPE achieves higher V2V camera controllability than TrajectoryCrafter and ReCamMaster while maintaining competitive perceptual quality.TrajectoryCrafter is sensitive to depth-estimation errors, whereas ReRoPE encodes relative camera relationships within the RoPE subspace without explicit geometric conditioning.
- I2V Comparison: ReRoPE achieves the highest I2V camera accuracy among the compared methods while maintaining comparable video fidelity on static DL3DV scenes.Quantitative comparison is restricted to static scenes because camera and scene motion are difficult to disentangle reliably in dynamic videos.
- I2V Comparison: On dynamic I2V scenes, ReRoPE generates natural object motion that follows the specified camera trajectory, unlike baselines producing frozen rigid subjects.The qualitative evaluation uses SDG-1.5M because quantitative metrics are unreliable for comparing rigid and non-rigid outputs.
- Ablations: ReRoPE converges faster and reaches lower final FID than Full-Temporal Replacement and Double RoPE, while translation normalization improves camera fidelity and FVD.The ablations attribute the alternatives’ degradation to disrupting temporal priors or causing signal interference, while normalization stabilizes projective conditioning.
6 Conclusion
The conclusion presents ReRoPE as a unified plug-and-play approach that reframes camera control as relative positional encoding. It reports precise trajectory control and strong camera accuracy, 3D consistency, and visual fidelity across V2V and I2V.
- Conclusion: ReRoPE injects camera geometry into underutilized low-frequency temporal RoPE bands without architectural changes.The framework is designed for both V2V and I2V while preserving pre-trained generative priors.
- Conclusion: Across diverse datasets, ReRoPE achieves superior camera accuracy and 3D consistency while maintaining high visual fidelity.The reported scope includes both video-to-video and image-to-video camera-controlled generation.
7 Analysis of Low-Frequency Redundancy
The analysis explains that the lowest temporal RoPE frequencies barely rotate over video-relevant distances because video models use short temporal contexts. ReRoPE therefore repurposes this effectively identity-like subspace for camera signals.
- Low-Frequency Redundancy: For temporal head dimension d_τ=32, the lowest frequency rotates approximately 10^-4 radians per token, accumulating only O(10^-2) radians across 100 frames.The resulting rotation matrix is effectively the identity over the stated temporal distance.
- Low-Frequency Redundancy: Video models’ short temporal contexts leave the lowest-frequency RoPE components underutilized compared with their original LLM motivation.The base frequency θ=10^4 was inherited from LLM settings designed for dependencies across tens of thousands of tokens, whereas video training uses approximately 10 to 128 temporal tokens.
- Low-Frequency Redundancy: ReRoPE uses this underutilized low-frequency subspace as a carrier for camera signals.The design follows directly from the negligible phase accumulation of the lowest temporal band.
8 Data Annotation and Captioning Strategy
The annotation strategy is designed to keep captions aligned with scene content, object dynamics, and viewpoint consistency while preventing text from encoding camera motion. It also adapts captioning prompts to static versus dynamic data.
- Caption Generation: Dense video captions are generated with Qwen3-VL Plus and processed through the Wan2.2 caption_extend script.This preprocessing targets alignment between textual descriptions and visual content.
- Camera-Motion Decoupling: Camera-motion descriptors are filtered from prompts so ReRoPE embeddings, rather than textual correlations, drive camera control.The protocol removes phrases such as “pan left” and “zoom in” to prevent shortcut learning.
- Multi-View Consistency: For synchronized MultiCam views, one caption is propagated from the primary view to preserve semantic consistency across viewpoints.This encourages the model to interpret view changes as geometric transformations of the same content.
- Static Scenes: Static-dataset prompts exclude dynamic-motion directives to avoid hallucinating motion in rigid scenes.The resulting captions are intended to reflect the static nature of the environment.
9 Pose Estimation and Scale Unification Protocol
The V2V pipeline extracts source camera poses and unifies source and target trajectories through staged normalization to preserve relative motion across coordinate systems.
- VIPE extracts the camera pose sequence from the source video for V2V inference.
- A scale discrepancy arises because estimated source trajectories have arbitrary monocular scale while target trajectories are user-defined.
- Individual pre-normalization rescales trajectories whose maximum translation magnitude exceeds one, reducing numerical instability from extreme scales.
- Joint normalization scales both pre-normalized trajectories using a global factor derived from their union.
- The unified scaling preserves relative source-target motion within the model-friendly range [0, 1], bridging distinct coordinate systems.
10 Progressive Training Strategy
The V2V model uses progressive sequence-length training: it learns camera-control injection on short clips before adapting the checkpoint to longer contexts.
- Training begins on short video clips with T = 10 to efficiently learn camera-control injection.
- The short-clip checkpoint is then loaded and training resumes on longer sequences with T = 42.
- This progressive scaling reduces total training duration compared with training on long contexts from scratch.
11 Limitations
ReRoPE’s I2V control can be constrained by subject-centering priors learned from web-scale data, although static scenes without clear subjects retain precise 6-DoF responsiveness.
- I2V performance is occasionally constrained by compositional bias in web-scale pre-training data.
- Subject-centering priors can counteract trajectories that move subjects away from the center or out of frame by synthesizing compensatory subject motion.
- On static scenes without clear subjects, ReRoPE responds precisely to all 6-DoF target poses.