Source-linked AI summary
Towards Understanding Camera Motions in Any Video
Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, Deva Ramanan
TL;DR
Camera motion understanding lacks reliable, broad evaluation across geometric and semantic primitives in diverse videos. The paper introduces CameraBench with an expert-developed taxonomy and quality-controlled annotations, then evaluates and fine-tunes models. SfM/SLAM and VLMs show complementary strengths, while fine-tuning a generative VLM substantially improves performance across tasks.
Problem
Camera motion is needed for scene reconstruction and video-language reasoning, but existing approaches and annotations have limited coverage of dynamic, semantic, and geometric motion.
Method
The paper builds CameraBench using an expert-developed taxonomy, robust annotation framework, training program, and diverse benchmark for SfM/SLAM and VLM evaluation.
Results
SfM/SLAM methods struggle with semantic and low-parallax motion, VLMs show complementary strengths, and supervised fine-tuning achieves ∼2x performance, matching MegaSAM.
Takeaways & Limitations
CameraBench supports human-like camera-motion understanding by combining precise motion definitions, expert oversight, and model evaluation across multiple video tasks.
Takeaways & Limitations
The paper leaves camera-motion control in video generation and post-training techniques beyond SFT for future work.
Abstract
from arXiv · showhide
We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video.
1 Introduction
Camera motion is central to visual perception and video understanding, yet existing computational methods and datasets struggle with dynamic scenes, semantic intent, and reliable annotation. CameraBench addresses these gaps with expert-designed taxonomy, quality-controlled annotations, and broad model evaluation.
- Motivation: Camera motion supports depth perception, 4D scene reconstruction, and video-language reasoning about dynamics.SfM and SLAM estimate pose trajectories, while VLMs require camera-motion understanding to reason about and generate video dynamics.
- Dataset gaps: Existing datasets contain inaccurate specifications, contradictory labels, and major annotation errors.Reported problems include conflating translation with rotation or zoom, labeling clips as both static and moving, and reversing motion direction.
- CameraBench: CameraBench provides labels and captions for diverse camera-motion tasks and evaluates 20 SfM/SLAM and VLM models under a fair protocol.The benchmark covers motion-primitive classification, retrieval, captioning, and video question answering.
- Computational gaps: Classic SfM/SLAM methods struggle with dynamic or low-parallax scenes and basic motion classification.Stationary or rotation-only cameras can make it difficult to determine whether the camera moves or moves upward.
- Findings: Learning-based SfM/SLAM methods outperform classic COLMAP by 1-2x, while fine-tuned VLMs achieve 1-2x improvements across discriminative and generative tasks.Generative VLMs are especially promising for semantic motions such as tracking shots.
- Contributions: The paper contributes an expert-developed taxonomy, robust annotation framework, diverse benchmark, and analysis of model strengths and limitations.These resources target improved understanding of camera motions in any video.
2 Related Work
Prior camera-motion datasets represent trajectories, labels, or descriptions, but often rely on static scenes, ambiguous categories, limited coverage, or undocumented annotations. CameraBench’s taxonomy is designed to broaden and clarify this representation space.
- Existing representations: Trajectory-based datasets provide geometric camera descriptions but often rely on estimated pseudo-ground truth and mostly static scenes.Ground-truth trajectories are difficult to obtain for real-world dynamic videos.
- CameraBench: CameraBench’s taxonomy spans object-, ground-, and camera-centric reference frames and covers steadiness, translation, rotation, intrinsic changes, and object-centric movements.It was developed with cinematographers and refined iteratively over three months using real-world videos and expert feedback.
- Existing representations: Prior labels conflate translation with rotation or zoom, include contradictory annotations, and may force co-occurring motions into a single label.Examples include grouping pan with truck, dolly with zoom, and labeling videos as both static and pan.
- Existing representations: Motion-description datasets often lack open-source taxonomies or documented annotation guidelines and cover narrow video domains.Existing coverage may focus on static scenes, movies, or unedited footage.
- Generative models: Generative-video studies commonly use synthetic camera movements, static scenes, or SfM-estimated trajectories, while their datasets are often not open-sourced.The accuracy of SfM trajectories for real-world or synthetic videos remains unclear in this setting.
3 Camera Motion Requires Clear Specification and Expert Oversight
Camera-motion annotation is difficult because terminology depends on reference frames, similar motions are easily confused, and real videos can contain complex temporal patterns. The paper therefore combines precise definitions, expert collaboration, robust annotation, and training.
- Challenges: Analysis of seven prior datasets identifies inaccurate motion specifications, inconsistent labels, and insufficient expert verification.These issues limit their usefulness for camera-motion understanding.
- Challenges: Camera-motion descriptions can change with the reference frame, such as calling optical-axis movement downward when the camera descends toward the ground.Context can also make a camera appear to follow a subject even when it moves backward ahead of that subject.
- Challenges: Amateurs may confuse intrinsic zoom changes with extrinsic camera translation, while videos may contain compound or sequential motions that resist one-label classification.A drone can move forward and then abruptly reverse direction within one clip.
- Our solution: The authors collaborate with cinematographers and experienced film-school participants to develop precise terminology, a comprehensive taxonomy, annotation framework, and training program.The approach is motivated by professional cinematography practice and the need for careful oversight.
4 Taxonomy Design, Annotation Framework, and Training Program
CameraBench’s taxonomy and annotation framework combine cinematographer-informed motion primitives with iterative quality control and structured annotator training. The resulting process supports diverse labels, captions, and reliable large-scale annotation.
- Taxonomy design: Iterative annotation with cinematographers resolves disagreements and refines label definitions by adding missing terms and clarifying edge cases.
- Taxonomy design: The taxonomy finalizes over 50 motion primitives after consensus on approximately 800 videos, compared with 4–5 in prior work.
- Taxonomy design: Labels cover motion type, steadiness, translation, rotation, tracking, zoom-related size changes, speed, cinematic effects, and scene movement.
- Annotation framework: The framework separates ground-centric and camera-centric translation because human interpretations can differ, especially in bird’s-eye-view footage.
- Annotation framework: Annotations combine binary primitive labels with natural-language descriptions, while explicitly accommodating single, compound, ambiguous, and conflicting motions.
- Training program: Experts outperform non-experts by more than 15% in accuracy, while multimodal tutorials and extended practice with feedback improve performance across participants.
- Training program: The completed dataset contains over 150K binary labels across 3,381 fully annotated videos.
5 CameraBench for Motion Understanding
CameraBench evaluates camera-motion understanding across classification, VQA, captioning, and retrieval, exposing complementary weaknesses in geometric and semantic models. Supervised fine-tuning substantially improves a generative VLM across these tasks.
- Benchmark tasks: Motion primitive labels and captions support classification, retrieval, VQA, and captioning evaluations across 20 SfM/SLAM and VLM models.
- Benchmark tasks: Classification evaluates camera-centric primitives using estimated translation, rotation, and focal change from camera extrinsics and intrinsics.
- Classification results: Approximately 50% AP is the best overall classification performance, leaving all methods far from solving motion-primitive recognition.
- Classification results: Learning-based SfM/SLAM methods outperform COLMAP, while generative VLMs outperform discriminative VLMs but remain challenged by motions such as roll.
- Classification results: Fine-tuning Qwen2.5-VL on approximately 1,400 non-overlapping training videos yields approximately 2× performance and matches MegaSAM.
- VQA results: The SFT model achieves state-of-the-art VQA results across all skills, especially Tracking Shot and Only Motion, which require object-centric and logical reasoning.
- Other tasks: The SFT model generates more accurate camera-motion captions than state-of-the-art VLMs, measured qualitatively and with SPICE and LLM-as-a-Judge.
6 Conclusion
The paper concludes that CameraBench provides a taxonomy, annotation framework, and benchmark for human-like camera-motion understanding. It also shows that fine-tuning VLMs on the dataset improves understanding while leaving integration and generation-control directions open.
- Limitations and future work: Future work may explore post-training methods beyond supervised fine-tuning, including optimized preset prompts.
- Limitations and future work: Camera-motion control in video generation remains future work, and integrating SfM/SLAM with VLMs is identified as promising.
- Conclusion: CameraBench combines a cinematographer-developed motion taxonomy, robust annotation framework, and benchmark for analyzing existing models.
- Conclusion: The dataset supports fine-tuning VLMs for improved camera-motion understanding.
Outline
The supplement covers dataset analysis, annotation and training details, taxonomy definitions, experimental results, and CameraBench’s VQA task structure.
- Supplement outline: Section A analyzes errors in prior datasets, while Section B provides additional CameraBench statistics and examples.
- Supplement outline: Sections C and D detail the annotation framework, guidelines, training program, and quality-control pipeline.
- Supplement outline: Sections E and F present experimental setup, additional results, and the detailed label taxonomy.
- Supplement outline: Section G details CameraBench’s 9 top-level skills and 81 VQA sub-tasks.
A Error Analysis of Prior Datasets
The paper identifies widespread problems in prior camera-motion datasets, including incorrect specifications, inconsistent labels, annotation errors, missing motion types, and narrow coverage. These issues motivate rigorous definitions, expert oversight, and broader video coverage.
- Implications: These findings support interactive inspection of videos and expert annotations to understand annotation challenges and the need for rigorous oversight.The paper documents issues in seven datasets and encourages readers to inspect linked visual reports.
- Specification: Prior datasets conflate translation, rotation, and zoom, often without providing taxonomies or definitions for their labels.MovieNet and MovieShot define dolly-in as zoom, while AVE groups pan and truck together; other benchmarks provide no label definitions.
- Annotation consistency: Over 500 AVE clips are labeled both static and pan, while prior datasets lack guidance for conflicting or compound motions.The paper also notes that AVE labels over 1,000 clips as both static and motion types such as pan or tilt.
- Verification: Recent benchmarks contain substantial motion-description errors, including wrong motion types, incorrect directions, omitted motion, and hallucinated camera movement.The paper reports these problems across VidComposition, DREAM-1K, VDC, and AVE audits.
- Coverage and scope: Existing datasets omit common motions, use unclear reference frames, mishandle shot transitions, and often cover only narrow domains or static scenes.Examples include missing arc and tracking shots, ambiguous “move down” directions, multi-clip videos treated as single shots, and film-only coverage.
D Training Program and Quality Control
CameraBench combines explicit captioning guidance with staged annotator training and layered review. The process targets clear, concise, accurate descriptions and resolves disagreements before author-level verification.
- Training: Tutorials provide textual definitions, video examples, clear guidelines, and complex edge cases before participants annotate the dataset.The tutorial is designed to familiarize participants with camera movements and align them with the labeling policy.
- Caption guidelines: Captions should state movement purpose when clear, major motions and directions, and relevant scene or subject context.Highly intricate or fragmented motion is summarized at a higher level.
- Caption quality: Annotators assess descriptions for clearness, conciseness, grammar, and fluency, with LLMs permitted for polishing without changing meaning.The stated polishing goal is improved readability while preserving intended details and tone.
- Training: Five training rounds of 30 videos each provide detailed feedback reports, and performance typically improves by 15–20%.Participants receive accuracy comparisons with ground truth after each round and may receive direct author guidance.
- Quality control: Quality control uses independently assigned labelers, reviewers who resolve disagreements, and an additional author review of every video.Only annotators who complete all training are hired.
E Experimental Setup and Results
The experiments evaluate SfM/SLAM and VLM systems across camera-motion understanding tasks, retrieval, and generation-related applications. The setup includes model-specific scoring and fine-tuning comparisons.
- Motion control: Fine-tuning CogVideoX-1.5 improves control of dolly, zoom, and arc motions in image-to-video generation.The preliminary experiment compares the original and LoRA fine-tuned models on randomly selected test samples.
- VLM evaluation: VLM evaluations use CLIPScore, ITMScore, and VQAScore for video-text matching, with model-specific frame-sampling procedures.Most models use uniform sampling, while Gemini processes video files directly and Qwen2.5-VL uses FPS sampling.
- Fine-tuning: Qwen2.5-VL is fine-tuned on approximately 1,400 non-overlapping training videos using supervised full fine-tuning for 5 epochs.The vision tower and multimodal projector are frozen, with model-specific learning rates and cosine scheduling.
- SfM/SLAM: The study benchmarks six classic and learning-based SfM/SLAM methods, including COLMAP, VGGSfM, DUST3R, MAST3R, and CUT3R.The methods use different matching, initialization, and pointmap-prediction strategies.
- Retrieval: Video-text retrieval reports Text Score, Video Score, and Group Score for skill-based and caption-based tasks.The skill-based setting covers eight skills except Complex Description, while caption-based retrieval evaluates Complex Description.
G Skills and Tasks in CameraBench
CameraBench organizes camera-motion understanding into nine top-level skills and 81 subtasks, spanning motion properties, scene dynamics, directions, tracking, and descriptions. Its taxonomy distinguishes geometric, intrinsic, object-centric, and contextual motion attributes.
- Task structure: CameraBench defines nine top-level skills and 81 subtasks with textual definitions used to construct VLM prompts.The task inventory is listed in Table 18 and related task tables.
- Motion type: Motion type distinguishes nonexistent, simple, minor, and complex camera motion.Simple motion is clear and consistent, while complex motion is ambiguous or conflicting.
- Translation: Translation covers physical camera movement forward, backward, up, down, right, and left, with directions defined in camera-centric or ground-centric reference frames.For bird’s-eye views, ground-centric forward is north and backward is south; worm’s-eye views reverse these directions.
- Rotation and zoom: Rotation and intrinsic change are separated: pan, tilt, and roll rotate the camera, whereas zoom changes focal length.Pure rotation and pure zoom preserve relative object positions without parallax, unlike translation.
- Object-centric and contextual motion: Object-centric categories include arcs and multiple tracking directions, while additional labels capture speed, motion effects, and scene movement.Tracking categories include tail-, lead-, side-, aerial-, tilt-, pan-, and arc-tracking.