PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encoding

Byeongjun Park, Byung-Hoon Kim, Hyungjin Chung, Jong Chul Ye

arXiv:2511.19827v1cs.CV

TL;DR

Video-retake methods face challenges with dynamic camera motion and variable-length inputs, while external geometry estimation can introduce artifacts. ReDirector aligns input and target positions using shared RoPE and adds camera-conditioned phase shifts. It improves camera control, geometric consistency, and visual quality across varied trajectories and video lengths, with a stated limitation under extreme object and camera motion.

  • Problem

    Existing retake methods assume fixed-length inputs with minimal camera motion, and external geometry estimators can degrade under dynamic motion and complex structures.

    Source: §1 · p. 2

  • Method

    ReDirector applies shared RoPE to input and target videos and integrates camera pose information through camera-conditioned RoPE phase shifts.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    ReDirector consistently outperforms previous methods in camera control, geometric consistency, and visual quality across diverse trajectories and video lengths.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method improves dynamic-object localization and static-background preservation, including under dynamic camera motion.

    Source: §6 · p. 8

    ReDirector still struggles when extreme object and camera motion coincide, particularly when large dynamic objects dominate the frame.

    Source: C. Limitations and Discussions · p. 13

Abstract

from arXiv · show

We present ReDirector, a novel camera-controlled video retake generation method for dynamically captured variable-length videos. In particular, we rectify a common misuse of RoPE in previous works by aligning the spatiotemporal positions of the input video and the target retake. Moreover, we introduce Rotary Camera Encoding (RoCE), a camera-conditioned RoPE phase shift that captures and integrates multi-view relationships within and across the input and target videos. By integrating camera conditions into RoPE, our method generalizes to out-of-distribution camera trajectories and video lengths, yielding improved dynamic object localization and static background preservation. Extensive experiments further demonstrate significant improvements in camera controllability, geometric consistency, and video quality across various trajectories and lengths.

1. Introduction

Prior retake methods struggle with dynamic camera motion and variable-length videos, while external geometry estimation can introduce artifacts. ReDirector aligns input and target positions with shared RoPE, adds camera-conditioned phase shifts, and outperforms previous methods across varied trajectories and video lengths.

1. Introduction

  • External depth and tracking estimators can degrade under dynamic camera motion and complex structures, letting warping artifacts pass into the generator inadequately addressed.These methods warp frames before generative refinement and inpainting.Source: §1 · pp. 1–2
  • Existing methods, including warping-based approaches, assume fixed-length inputs with minimal camera motion and degrade outside those regimes.The open challenge is integrating input-video and target-camera signals across variable lengths and camera trajectories.Source: §1 · p. 2
  • ReDirector applies the same RoPE to input and target videos, aligning spatiotemporal positions while camera conditions encode multi-view geometry within and across them.RoCE adds camera-conditioned phase shifts initialized at zero and learned during fine-tuning.Source: §1 · p. 2
  • RoCE phase shifts replace explicit geometric transforms in attention, remain compatible with fine-tuning video generators, and improve dynamic-object/background disentanglement.The paper reports that this yields physically plausible retakes.Source: §1 · p. 2
  • ReDirector consistently outperforms previous methods in camera control, geometric consistency, and visual quality across diverse target trajectories and video lengths.Evaluations on iPhone videos further validate generalization to out-of-distribution trajectories, video lengths, and resolutions.Source: §1 · p. 2

2. Related Works

Video-retake methods either warp frames using estimated geometry or condition generators directly on video and camera inputs; both approaches have limitations. Camera encodings likewise trade pixel-level detail against frame-level generality, motivating an encoding tailored to fine-tuning video generators on dynamic scenes.

2.1. Video Retake Generation

  • Warping-based retake pipelines reproject estimated 3D geometry to target cameras before generative refinement, but depend on accurate depth and heuristic scale alignment.Dynamic motion and complex scenes can propagate warping artifacts, while per-frame warping undermines dynamic-object/background disentanglement and aligned scene representations.Source: §2.1 · p. 2
  • Implicit retake methods condition generative models on camera extrinsics and input-video latents, avoiding explicit warping but remaining sensitive to training data and out-of-distribution trajectories and lengths.ReDirector extends these methods by integrating the control signals to support more robust generalization.Source: §2.1 · p. 3

2.2. Camera encoding in vision transformers

  • Pixel-aligned camera encodings attach per-pixel geometry for fine-grained multi-view relationships, but require a reference frame and global world coordinates that can hinder generalization.These encodings commonly concatenate camera information with tokens.Source: §2.2 · p. 3
  • Frame-aligned encodings generalize across sequence lengths without a consistent global frame, but lack pixel-level detail for fine correspondence and visibility reasoning.PRoPE combines both encoding types but trains a multi-view transformer from scratch on static scenes; ReDirector instead integrates camera encoding with RoPE for video-model fine-tuning on dynamic scenes.Source: §2.2 · p. 3

3. Preliminary

RoPE encodes relative token positions through rotations applied to attention vectors, with multiple frequencies covering a range of offsets. For video, 3D RoPE combines rotations for frame, height, and width positions.

Rotary Position Embedding

  • RoPE represents query and key vectors as complex-valued channel pairs and applies rotations in attention layers.The token sequence contains f h w tokens, with f, h, and w denoting compressed frame, height, and width dimensions.Source: §3 · p. 3
  • RoPE rotations use exponentially decaying per-channel frequencies to encode relative token positions as phase offsets.Multiple frequencies let the attention matrix cover a wide range of relative positions.Source: §3 · p. 3
  • The RoPE-adjusted query and key vectors modulate attention scores according to relative token positions.The real part is extracted and the key is complex-conjugated in the stated attention expression.Source: §3 · p. 3

3D RoPE

  • 3D RoPE combines complex rotations for the frame, height, and width axes into a channel-wise concatenated positional encoding.Each axis contributes a rotation matrix before the matrices are reshaped and concatenated.Source: §3 · p. 3

3. Preliminary

  • The overview shows ReDirector conditioned on target trajectories, an input video, and its poses; training updates self-attention layers, where RoCE shifts query/key and value paths.The value path applies a negative phase before attention weighting and the inverse phase after value aggregation.Source: Figure 2 · p. 4

4. Methods

ReDirector adapts a pretrained camera-controlled video model to retake dynamically captured, variable-length videos. It aligns source and target video positions with shared 3D RoPE and adds camera-conditioned phase shifts to encode multi-view geometry.

4.1. Video-to-Video Generative Model

  • Shared 3D RoPE aligns source and target spatiotemporal positions, while Plücker rays encode their camera trajectories for token-level multi-view conditioning.Source trajectories come from training data or are estimated at inference with ViPE; the shared encoding supports longer input sequences, including lengths absent from training.Source: §4.1 · p. 4
  • Training uses conditional flow matching with rectified flow, updates only self-attention weights, and generates retakes by solving the learned ODE from t = 1 to t = 0.Identity-retake pairs encourage alignment between tokens with matching 3D RoPE and RoCE, while time-reversed videos broaden the trajectories represented in training.Source: §4.1 · pp. 4–5

4.2. Rotary Camera Encoding

  • RoCE couples token-wise camera encoding to RoPE phase shifts, making attention more sensitive to relative pose and preserving static-region consistency while moving-object tokens diverge.The SO(2) phase shifts modify values before attention weighting and after value aggregation, enabling geometry-aware attention compatible with fine-tuning video generative models.Source: §4.2 · p. 5

5. Experimental Results

ReDirector is evaluated on DAVIS and iPhone retakes across varying trajectories and lengths, with results emphasizing camera control, geometric consistency, and robustness to longer videos. Ablations indicate that aligned positional encoding, RoCE, and geometry-aware attention contribute to the final system’s performance.

5.1. Experimental Setup

  • ReDirector is fine-tuned on MultiCamVideo for 20K steps using 81-frame training videos, then compared with implicit and explicit retake-generation baselines.The listed baselines include GCD, ReCamMaster, TrajectoryCrafter, and CogNVS.Source: §5.1 · p. 5
  • The DAVIS evaluation uses 50 videos and 10 target trajectories for 500 test cases, with video lengths ranging from dozens to about 100 frames.The evaluation reports visual quality, geometric consistency, input-video consistency, and relative translation and rotation errors.Source: §5.1 · pp. 5–6

5.2. Main Results

  • ReDirector leads the DAVIS comparison in geometric consistency and camera control, while some visual-quality metrics favor methods using milder camera motion.The caption attributes the visual-quality trade-off to ReDirector exploring larger scene scales under the same camera motion.Source: Table 1 · p. 6, §5.2 · p. 7
  • On DAVIS, ReDirector remains stable under dynamic camera motion, localizes dynamic objects over time, and preserves static backgrounds across distant viewpoints.The qualitative comparison contrasts input video and generated retakes; previous methods degrade when video length departs from their assumed setting.Source: §5.2 · pp. 6–7, Figure 4 · p. 6
  • The full combination of aligned 3D RoPE, RoCE, and geometry-aware attention delivers the best ablation performance across geometric and visual metrics.RoCE alone improves visual quality and camera accuracy but reduces geometric-consistency metrics; the full stack performs best across both metric groups.Source: §5.2 · p. 7, Table 2 · p. 7
  • On iPhone novel-view synthesis, ReDirector achieves the best performance, with consistent improvements as input video length increases.It remains robust on long sequences without ground-truth metric depth or external geometry models.Source: §5.2 · p. 8, Table 3 · p. 8
  • Additional training iterations produce steady gains in camera accuracy and geometric consistency.Table 4 reports the training-iteration ablation on DAVIS.Source: §5.2 · p. 8, Table 4 · p. 8

6. Conclusion

ReDirector combines shared positional encoding for the input and target videos with camera-conditioned phase shifts and geometry-aware attention. The paper reports improved camera control, geometric consistency, and visual quality, including generalization to out-of-distribution trajectories, lengths, and resolutions.

6. Conclusion

  • The method applies the same RoPE to input and target videos and uses Plücker rays to encode token-level multi-view relationships.The shared encoding aligns spatiotemporal positions and supports length-agnostic conditioning.Source: §6 · p. 8
  • ReDirector introduces RoCE, a camera-conditioned RoPE phase shift combined with geometry-aware attention.The conclusion associates these components with improved geometric consistency, dynamic-object localization, and static-background preservation under dynamic camera motion.Source: §6 · p. 8
  • Experiments report improvements in camera control, geometric consistency, and visual quality, with generalization to out-of-distribution trajectories, lengths, and resolutions.Source: §6 · p. 8

A. Evaluation Details

The appendix describes camera-trajectory setup and iPhone evaluation conditions, alongside qualitative figure references.

A. Evaluation Details

  • Inference uses 50 steps and a CFG scale of 5, with BLIP used for text captioning.Source: A. Evaluation Details · p. 12
  • For DAVIS, the 81-frame target trajectories are reparameterized as constant per-frame updates for input videos of arbitrary length.Camera intrinsics estimated by ViPE are used to convert target trajectories into Plücker rays.Source: A. Evaluation Details · p. 12
  • On the iPhone evaluation, ReDirector uses the provided camera trajectories and test viewpoints without video-pose-estimation methods or other external models, and achieves the best dynamic novel-view-synthesis performance among the compared retake methods.Source: A. Evaluation Details · p. 12
  • Figures 7 and 8 are identified as qualitative ablations of input trajectories and training iterations, respectively.Source: Figure 7 · p. 12, Figure 8 · p. 12

B.1. Ablative Results

The ablations examine the input camera trajectories and the effect of additional training on retake quality.

B.1. Ablative Results

  • Metrically aligned ViPE poses yield better performance than more accurate but scale-ambiguous MegaSaM poses because target trajectories are also defined in metric scale.The accompanying qualitative comparison reports that ViPE poses produce geometrically accurate retakes, while MegaSaM poses can fail to localize the person properly.Source: §B.1 · p. 12
  • Additional training improves both geometric consistency and dynamic-object localization in the qualitative results.Source: §B.1 · p. 12
  • The appendix also includes additional qualitative results on DAVIS.Source: Figure 9 · p. 13

B.2. More Results and Applications

ReDirector’s additional results show faithful camera control and preservation of dynamic objects and backgrounds across challenging scenes and varied video lengths. Time-reversed training supports retakes with non-overlapping first frames, and metrically scaled trajectories can be stretched.

B.2. More Results and Applications

  • ReDirector preserves dynamic objects and backgrounds while following target camera trajectories, including in examples with textureless backgrounds.The comparisons contrast this with prior methods’ unrealistic retakes, inconsistent rendering, or imprecise camera control.Source: §B.2 · p. 13
  • Retakes are reported as high-quality, geometrically accurate, and temporally coherent across varied trajectories and lengths.The training videos have fully overlapped first frames; time-reversed versions enable retakes with non-overlapping first frames.Source: §B.2 · p. 13
  • Time-reversed training videos enable ReDirector to generate retakes with non-overlapping first frames despite training videos having fully overlapped first frames.The non-overlapping examples are shown in Fig. 11-(b).Source: §B.2 · p. 13
  • ReDirector can generate retakes with target camera trajectories scaled by a factor of two.The paper links this capability to alignment with metrically scaled camera trajectories; the examples are shown in Fig. 11-(c).Source: §B.2 · p. 13

C. Limitations and Discussions

ReDirector struggles with extreme object and camera motion, especially when large dynamic objects dominate the frame. The authors identify integration with next-frame video generation or external geometry models as possible directions for improving geometric robustness and fidelity.

C. Limitations and Discussions

  • ReDirector struggles under extreme object and camera motion, particularly when large dynamic objects dominate the frame and multi-view relationships are difficult to infer.This is the limitation illustrated in Fig. 10.Source: C. Limitations and Discussions · p. 13
  • One proposed direction is integrating ReDirector with next-frame video generation models in the context of world models.The authors present this as a future direction.Source: C. Limitations and Discussions · p. 13
  • Combining ReDirector with external geometry models is another proposed direction for improving geometric robustness and fidelity.The authors identify geometry models as a potential complement to the framework.Source: C. Limitations and Discussions · p. 13

Related papers

Loading 2511.19827v1…