Source-linked AI summary
VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, Dahua Lin
TL;DR
Adapting 1D RoPE to video remains challenging because existing variants do not fully address video’s spatiotemporal structure and temporal frequency allocation. The paper analyzes four criteria, introduces VideoRoPE and V-NIAH-D, and reports consistent gains over prior variants across video retrieval, understanding, and hallucination tasks.
Problem
Existing RoPE-based approaches for video do not fully satisfy key requirements for complex spatiotemporal inputs, while temporal frequency allocation remains insufficiently studied.
Method
The paper analyzes four criteria for video positional encoding and proposes VideoRoPE with 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable temporal spacing, alongside the V-NIAH-D evaluation task.
Results
VideoRoPE consistently outperforms previous RoPE variants across long video retrieval, video understanding, and video hallucination benchmarks, including +12.4 on V-NIAH and V-NIAH-D over M-RoPE.
Takeaways & Limitations
The results support VideoRoPE as a robust video positional-embedding design across diverse long-context video tasks.
Abstract
from arXiv · showhide
While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce \textbf{VideoRoPE}, with a \textit{3D structure} designed to preserve spatio-temporal relationships. VideoRoPE features \textit{low-frequency temporal allocation} to mitigate periodic oscillations, a \textit{diagonal layout} to maintain spatial symmetry, and \textit{adjustable temporal spacing} to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \href{https://github.com/Wiselnn570/VideoRoPE}{https://github.com/Wiselnn570/VideoRoPE}.
1. Introduction
Video RoPE adaptation must address video’s spatiotemporal structure, frequency allocation, spatial symmetry, and temporal index scaling. VideoRoPE combines these requirements and achieves stronger performance across retrieval, understanding, and hallucination benchmarks.
- Motivation: Video LLMs require positional encodings that represent temporal, horizontal, and vertical structure rather than flattening video into a 1D sequence.Flattening discards neighborhood information and hinders explicit spatial and temporal representation.
- Motivation: High-frequency temporal allocation can cause periodic oscillations, making distant positions resemble one another and allowing distractors to mislead retrieval.V-NIAH-D inserts similar images around the target needle to expose this vulnerability.
- VideoRoPE: VideoRoPE uses a 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable temporal spacing to satisfy the four identified properties.These components target spatiotemporal modeling, oscillation reduction, spatial symmetry, and decoupled temporal indexing.
- Evaluation: V-NIAH-D exposes frequency-allocation weaknesses in existing position embeddings by adding similar-image distractors to the long-video retrieval setting.The task is designed to test whether retrieval remains robust when distractors create periodic confusion.
- Results: VideoRoPE outperforms M-RoPE by +12.4 on V-NIAH and V-NIAH-D, +2.9 on LongVideoBench, +4.5 on MLVU, +1.7 on Video-MME, and +11.9 on VideoHallucer.The reported gains span long-video retrieval, video understanding, and hallucination benchmarks.
2. Related Work
Prior work extends RoPE to multimodal and video inputs either by flattening visual tokens into a 1D sequence or by incorporating higher-dimensional structure. These approaches still leave spatiotemporal structure and cross-modal positional differences incompletely addressed.
- RoPE: RoPE encodes positions through sinusoidal functions with different frequencies across feature dimensions and is widely used for long-context modeling.Lower dimensions correspond to higher frequencies under the described design.
- Video and multimodal extensions: One multimodal approach applies standard 1D RoPE to flattened visual and text tokens, but this overlooks video’s spatiotemporal structure and inter-modal differences.Variants such as TAD-RoPE add indexing or attention changes while retaining the 1D formulation.
3. Analysis
The analysis identifies limitations in adapting RoPE to video, especially in preserving 3D structure, allocating frequencies, and maintaining spatial symmetry. M-RoPE’s temporal encoding is vulnerable to oscillation and distractors, while its attention patterns emphasize spatial rather than temporal information for retrieval.
- 3D Structure: Flattening video into 1D RoPE discards neighborhood information and hinders explicit spatial and temporal representation.
- Frequency Allocation: M-RoPE divides 128 dimensions into 32 temporal, 48 horizontal, and 48 vertical dimensions, assigning higher frequencies to temporal positions.The temporal dimensions use lower-indexed rotary angles.
- V-NIAH-D: M-RoPE shows a significant performance decline from V-NIAH to V-NIAH-D after similar distractor images are inserted around the needle.
- Attention Analysis: M-RoPE may locate the needle but fail to answer the question because attention relies primarily on vertical positional information rather than temporal features.
- Spatial Symmetry: Existing 3D RoPE variants do not satisfy spatial symmetry, which equalizes the distances between visual input and preceding or subsequent text.
4. VideoRoPE
VideoRoPE combines a 3D positional structure with low-frequency temporal allocation, diagonal spatial layout, and adjustable temporal spacing. These components allocate long-range modeling to temporal features, preserve spatial balance, and flexibly align video and text indices.
- Overview: VideoRoPE combines 3D structure, Low-frequency Temporal Allocation, Diagonal Layout, and Adjustable Temporal Spacing to satisfy four video-RoPE requirements.
- Low-frequency Temporal Allocation: Using the last 16 rotary angles gives VideoRoPE wider, more monotonic temporal intervals than M-RoPE’s first 16 angles, reducing distractor influence.
- Low-frequency Temporal Allocation: VideoRoPE assigns higher dimensions and lower frequencies to temporal features, while interleaving lower dimensions for spatial features.The design keeps 48 dimensions for horizontal positions, 48 for vertical positions, and 32 for temporal positions.
- Diagonal Layout: VideoRoPE’s diagonal layout places each frame’s central patch at (t, t, t), preserving visual-token relative positions and approximate corner-to-center equidistance.
- Diagonal Layout: The diagonal layout also preserves vanilla RoPE’s indexing pattern while incorporating spatial modeling, unlike M-RoPE’s discontinuities across textual and visual tokens.
- Adjustable Temporal Spacing: Adjustable Temporal Spacing introduces δ to scale temporal indices and align temporal information between visual and textual tokens.
- Adjustable Temporal Spacing: For video tokens, the temporal index is scaled from the video start, while starting and ending text use linear index progressions.
5. Experiment
Experiments evaluate VideoRoPE across long-video understanding, retrieval, and hallucination benchmarks, with comparisons against existing RoPE variants. VideoRoPE consistently outperforms prior methods across these tasks.
- Long Video Understanding: VideoRoPE outperforms vanilla RoPE, TAD-RoPE, and M-RoPE across LongVideoBench, MLVU, and Video-MME.The evaluation covers context lengths from 8k to 64k.
- Long Video Retrieval: VideoRoPE improves retrieval performance over M-RoPE by 12.44% on both V-NIAH and V-NIAH-D.V-NIAH-D is more challenging because it includes distractor frames.
- Video Hallucination: VideoRoPE achieves a 29.5% improvement on VideoHallucer’s Temporal Hallucination task over existing methods.It also improves the Object-Relation Hallucination subtask by 18.0%.
- Ablation Studies: Ablation experiments progressively evaluate the DL, LTA, and ATS modules against the M-RoPE baseline.M-RoPE scores 54.35 on LongVideoBench and 61.10 on MLVU at 64k context length.
6. Conclusion
The paper identifies four criteria for adapting positional encoding to video and proposes VideoRoPE to address them. Experiments show that VideoRoPE outperforms previous RoPE variants across retrieval, understanding, and hallucination tasks.
- The paper identifies 2D/3D structure, frequency allocation, spatial symmetry, and temporal index scaling as four criteria for video positional encoding.
- VideoRoPE combines 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable temporal spacing.
- VideoRoPE outperforms previous RoPE variants on long video retrieval, video understanding, and video hallucination tasks.
A.1. Supplementary Ablation Experiments
Supplementary ablations examine temporal scaling, spatial allocation, diagonal layout, and frequency allocation across video benchmarks. The results favor interleaved spatial allocation and low-frequency temporal allocation.
- ATS Scaling: The supplementary experiments evaluate how the temporal scaling factor δ affects alignment between video and text representations.Evaluation covers LongVideoBench, MLVU, and VideoMME.
- x, y Allocation: Interleaving x and y allocations achieves superior performance compared with sequential allocation.The paper hypothesizes that interleaving preserves similarity between the x and y dimensions.
- Diagonal Layout: Additional ablations test the Diagonal Layout across MLVU, VideoHallucer, V-NIAH, and V-NIAH-D.
- Frequency Allocation: Low-frequency temporal allocation consistently outperforms the interleaved [t t x y x y x y] pattern on average.The comparison uses LongVideoBench across varying context lengths and diverse video dynamics.
A.2. Extrapolation to 128k Experiments
The paper evaluates VideoRoPE under a 128k visual context to assess extrapolation beyond the standard setting. Although all methods degrade, VideoRoPE experiences the smallest performance drop.
- At 128k context length, all four methods exhibit performance degradation on LongVideoBench.
- VideoRoPE experiences the least performance drop at 128k, demonstrating robustness under extreme extrapolation settings.Inference uses the vLLM framework in Server-API processing mode.
B. Additional Details on Evaluation Benchmarks
The paper evaluates long-video retrieval, understanding, and hallucination with benchmarks targeting long-range reasoning and specific-moment retrieval. Its V-NIAH-D benchmark extends V-NIAH with periodic, semantically similar distractors to test discriminability under temporal dependencies.
- LongVideoBench tests reasoning over frame sequences lasting 8 seconds to 1 hour, excluding questions with subtitles.
- V-NIAH inserts a question-specific needle image at random positions within a 3,000-frame haystack for long-video retrieval.Each frame is encoded with 144 tokens, and validation checks positions from 100 through 3,000 frames.
- V-NIAH-D periodically inserts semantically similar but question-irrelevant distractors 200 frames from the needle.The 200-frame period approximates 2 · π · 100000032/128 ≈198.7, based on rotary bases.
- VideoHallucer evaluates intrinsic and extrinsic hallucinations across object-relation, temporal, semantic-detail, and factual subcategories.
- V-NIAH-D requires human annotation only for its visual questions and answers, supporting evaluation of long-context reasoning in multimodal language models.
E. Supplementary Attention Analysis
Supplementary attention analysis compares M-RoPE and VideoRoPE on 8k-context inputs aggregated by frame. The analysis attributes M-RoPE’s local attention to high-frequency temporal modeling, whereas VideoRoPE captures long-range dependencies and identifies the needle.
- The attention comparison uses 8k-context inputs, aggregates tokens from each frame by average pooling, and represents one axis tick as one inference frame.
- M-RoPE’s high-frequency temporal modeling limits attention to local information and hinders needle identification for question answering.
- VideoRoPE’s low-frequency temporal modeling captures long-range dependencies and successfully identifies the needle for accurate responses.
F. Supplementary Explanation on Frequency Allocation
The supplementary explanation contrasts M-RoPE’s high-frequency temporal allocation with VideoRoPE’s low-frequency allocation. The latter uses wider monotonic intervals, reducing periodic distortion and distractor effects.
- Frequency allocation: M-RoPE models temporal dependency with the first 16 rotary angles, whose higher frequencies produce greater oscillation.
- Frequency allocation: Periodic oscillations distort M-RoPE’s temporal position embeddings and can make distant positions share the same embedding.Lower dimensions have even shorter monotonic intervals, increasing susceptibility to distortion.
- Frequency allocation: VideoRoPE models temporal dependency with the last 16 rotary angles, which have much wider monotonic intervals and avoid oscillation in the illustrated dimensions.
- Frequency allocation: VideoRoPE’s frequency allocation significantly suppresses misleading distractor effects in temporal position modeling.