PaperlayerView exact revision on arXiv

Source-linked AI summary

When Left Is Right? Camera-Motion Understanding beyond Screen

Chong Cheng, Ran Zhang, Xinyuan Li, Alan Zhao, Hao Wang, Xi Wang

arXiv:2610.17242v1cs.CV

TL;DR

Most VLMs remain near chance on left/right camera-motion questions, and adding camera-motion data or geometric representations separately has not closed the gap. CameraVLM combines camera and flow states with language alignment; with 9B parameters, it outperforms open-source and camera-motion-specific models and matches or surpasses GPT-6 Astra and human viewers on average.

  • Problem

    Most open-source and commercial VLMs answer left/right camera-motion questions about half the time, while humans reach 86%, and data-focused or geometry-focused approaches tried separately have not closed the gap.

    Source: §1 · p. 2

  • Method

    CameraVLM jointly represents off-screen camera motion with pose-grounded camera tokens and on-screen subject and background motion with flow tokens, then aligns these states with language through two-stage training.

    Source: §1 · pp. 2–3

  • Results

    With 9B parameters, CameraVLM outperforms open-source and camera-motion-specific models by a wide margin and matches or surpasses GPT-6 Astra and human viewers on average.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Keeping off-screen camera motion and on-screen image motion together yields consistent benchmark gains, particularly for spatial direction and compound camera motion.

    Source: §5 · p. 10

    Some state learning may inherit errors from offline supervision, rare compound motions and unusual imaging conditions are underrepresented, and the current setting covers shot-level motion rather than relations across cuts or longer sequences.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/

1 Introduction

Camera-motion understanding remains difficult because image motion alone can obscure camera motion and spatial direction depends on the query’s reference. CameraVLM combines camera and flow states with language alignment, and its 9B-parameter model performs strongly across benchmarks and against human viewers.

1 Introduction

  • Most open-source and commercial VLMs answer left/right camera-motion questions about half the time, while humans reach 86%.Prior camera-motion data and geometric representations were tried separately, but neither alone closes the gap.Source: §1 · p. 2
  • Camera motion can resemble subject motion, and spatial descriptions change with the reference entity, so the representation must couple off-screen geometry with on-screen motion.For example, a pan and a truck can be hard to distinguish without near and far objects, and a camera following a person moves relative to the street but not the person.Source: §1 · p. 2
  • CameraVLM pairs pose-grounded camera tokens for off-screen motion with flow tokens for cross-frame subject and background motion and visibility.The joint states help distinguish camera motion from subject motion and camera moves that look alike on screen.Source: §1 · p. 2
  • Its two-stage alignment first trains new tokens with the language model frozen, then jointly trains language, pose, and flow objectives with the model unfrozen.The first-stage QA gradient aligns token states with the existing representation space; general video QA is mixed into the second stage to preserve language abilities.Source: §1 · p. 2
  • With 9B parameters, CameraVLM outperforms open-source and camera-motion-specific models by a wide margin and matches or surpasses GPT-6 Astra and human viewers on average.The comparison covers three camera-motion benchmarks and a 48-participant human study.Source: §1 · pp. 2–3

2 Related Work

Prior work adds camera-motion labels, geometric or spatial states, or dynamic states, but these approaches leave gaps in transfer, cross-frame motion, or reference-aware interpretation. CameraVLM instead retains camera and flow states within the VLM and aligns them with language.

Camera-motion understanding

  • Camera-motion benchmarks and explicit camera information improve recognition, but prior methods mainly use predefined motion semantics without states that can be reinterpreted as the target, reference, or temporal range changes.The related work describes CameraBench, CineTechBench, ShotBench, ACaM, and methods using camera pose, camera tokens, or external motion estimates.Source: §2 · p. 3

Spatial representations for VLMs

  • Depth and spatial-representation methods model depth, pose, or scene structure, but have limited explicit cross-frame motion correspondence and continuous spatio-temporal states.The passage names DepthLM, DepthVLM, Cambrian-P, VLM-3R, SpaceMind, and Q-GeoMem.Source: §2 · p. 3

Embodied and world models

  • Richer embodied and world-model dynamics alone do not resolve reference ambiguity in camera motion, because camera, subject, and background motion vary with reference and frame.These models learn egocentric perception, action, interaction, or broader physical states and transitions.Source: §2 · pp. 3–4

Camera and motion state estimation

  • Camera geometry and tracking methods estimate camera or motion states, but typically separate state estimation from language interpretation; CameraVLM retains both states in the VLM and aligns them with camera-motion semantics.Its shared representation supports state prediction and language reasoning together.Source: §2 · p. 4, Figure 3 · p. 4

3 Method

CameraVLM represents video with complementary camera-pose and cross-frame-flow states, interpreted alongside the question in a shared vision-language model. Its training aligns these states with language while preserving continuous motion information.

3.1 Problem Formulation

  • Camera-motion understanding is formulated as interpreting a continuous dynamic state under a query, because image motion mixes camera and scene motion and depends on the reference entities, frame, and interval.Translation, rotation, and intrinsic changes can produce similar image motion, while parallax across depths can distinguish some cases.Source: §3.1 · pp. 4–5

3.2 Architecture

  • The architecture integrates pose-grounded camera states and pooled flow states with video and question tokens in one shared VLM, whose heads predict camera parameters and answer the query.The shared backbone contextualizes the token types together.Source: §3.2 · p. 5

3.3 Pose-Grounded Camera States

  • One learnable camera token per source frame yields states refined across frames to predict translation, rotation, and field of view under pose supervision.The language model reads these same geometry-supervised states, retaining continuous camera motion rather than reducing it to discrete labels.Source: §3.3 · p. 6

3.4 Trajectory-Grounded Flow States

  • The flow pathway tracks points initialized on a uniform image grid, pools their motion states into fixed tokens, and leaves camera, subject, and background roles to joint interpretation.Relative motion and parallax across subjects and foreground and background provide cues for separating camera motion from scene motion.Source: §3.4 · p. 6

3.5 Dynamic-State Learning and Language Alignment

  • Training first adapts camera and flow pathways with the language backbone frozen, then unfreezes it for joint state–language optimization.QA gradients in the first stage align new states with the pretrained model’s representation space, while camera and flow losses supervise geometry and motion.Source: §3.5 · p. 6
  • Video-only, pose-only, and video-plus-pose samples tie inferred or embedded camera geometry to the same camera-motion language; inference uses only video and the question.The mixed samples are designed to connect states from different sources to shared language descriptions.Source: §3.5 · p. 6

3.6 Training Data

  • CameraVLM (9B) surpasses all open-source and camera-motion-specific baselines and exceeds human average accuracy; it leads on most of eight camera-motion primitives.The primitive-level figure reports GPT-6 Astra as the only model ahead of CameraVLM in the remaining categories.Source: Table 1 · p. 7, Figure 4 · p. 7
  • Training pairs camera-motion language with pose and flow labels on the same clips, drawing language data from ACaM and CameraBench and additional pose and trajectory supervision from other datasets.The passage reports 27K ACaM QA examples and 68K CameraBench QA examples and captions; training and evaluation videos are disjoint.Source: §3.6 · p. 7

4 Experiments

CameraVLM is evaluated on camera-motion benchmarks, motion-state tasks, and ablations, with limitations including imperfect offline supervision and a shot-level scope. Its results cover benchmark QA, retained camera and flow states, and the effects of removing components or masking inference states.

4.1 QA evaluation

  • CameraVLM consistently outperforms its backbone and open-source baselines across three benchmarks, exceeds average human performance, and remains competitive with GPT-6.Its gains persist across benchmarks with different motion distributions, unlike single-benchmark fine-tuning.Source: §4.1 · p. 9
  • CameraVLM performs well across eight camera-motion categories and achieves the best overall accuracy, with background flow helping disambiguate subject motion in a truck-left example.Most models mistake the shot for a pull-out; CameraVLM correctly recovers the camera motion.Source: §4.1 · p. 9

4.2 State Evaluation

  • On Sintel, CameraVLM achieves the best ATE and RPEt among streaming baselines, while its camera states retain transferable camera-motion information.The study also evaluates pose on TUM RGB-D and ScanNet; main benchmark evaluations use raw video and questions without external geometric information.Source: Implementation details · p. 8, §4.2 · p. 9
  • Integrated flow states retain competitive occlusion accuracy and cross-frame tracking information on TAP-Vid DAVIS, despite not being optimized as a standalone tracker.Table 3 reports AJ, δavg, and OA, with higher values better.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 Ablation Study

  • Removing flow states or camera-and-flow supervision degrades performance, and the full two-stage strategy improves on one-stage training.The ablations indicate that cross-frame motion complements camera states and that state learning and state–language alignment are complementary.Source: §4.3 · pp. 9–10, Table 4 · p. 9
  • Inference-time masking causes only limited degradation, suggesting dynamic information is distributed across the shared representation.The model is also robust to most flow configurations.Source: §4.3 · p. 10

4.4 Limitations

  • Training may inherit errors from offline VGGT-Ω and CoWTracker supervision; rare compound motions and unusual imaging conditions are less represented, and the model covers shot-level motion rather than relations across cuts or longer sequences.Implementation uses at most 128 sampled frames and two training stages of 20K and 30K iterations.Source: Implementation details · p. 8, §4.4 · p. 10

5 Conclusion

The conclusion emphasizes continuous camera and flow states aligned with camera-motion semantics, while the appendix explains how motion descriptions depend on reference frames and how different camera actions can produce similar image motion. It also distinguishes answer equivalence under training queries from equivalence of the underlying states.

5 Conclusion

  • Representing off-screen camera motion together with on-screen image motion yields consistent benchmark gains, particularly for spatial direction and compound camera motion.The authors describe this representation as a basis for interpreting shot composition and subject–camera relations, and for controllable video generation and editing.Source: §5 · p. 10

A.1 Relative Motion under a Moving Reference Frame

  • A static world relation can have different coordinate representations across reference frames, and opposite horizontal axes reverse the displacement’s horizontal sign.Relative velocity is expressed in the moving frame F.Source: §A.1 · p. 16

A.2 Camera-Frame Motion Decomposition

  • Camera-frame motion terms correspond to scene or subject translation relative to camera translation, and camera rotation.Source: §A.2 · p. 16

A.3 Pan–Truck Ambiguity

  • Pan and lateral camera translation can produce similar first-order image motion; depth-dependent parallax, higher-order perspective terms, or camera-pose information can resolve the ambiguity.Source: §A.3 · p. 17

A.4 Zoom–Dolly Ambiguity

  • Zoom and forward axial translation can produce the same instantaneous radial flow at a single depth, but depth variation in flow separates axial translation from focal-length change.Source: §A.4 · p. 17

A.5 Reflection

  • Reflection reverses a motion vector’s normal component while leaving tangential components unchanged; a left–right reversal depends on mirror orientation and the queried reference frame.Source: §A.5 · p. 18

A.6 Query-Induced State Equivalence

  • States that produce the same answers for all observed training queries may share an internal representation without additional QA loss, even though answer equivalence does not imply state equivalence.Without metric depth supervision, world-frame object translation remains ambiguous in scale.Source: §A.6 · p. 18

B Implementation Details

CameraVLM adds camera and flow tokens to a Qwen3.5-VL-9B model, placing the state tokens alongside video and question inputs. Separate camera-pose and flow pathways generate and supervise those states, which are trained in two stages.

Architecture

  • Each source frame receives a camera token whose hidden state is decoded into translation, rotation, and viewpoint change; a separate flow pathway predicts trajectories, visibility, and confidence before pooling them into flow tokens.Camera tokens are interleaved with source frames, while flow tokens are inserted before the question.Source: B Implementation Details · p. 18

Supervision

  • Camera states receive multi-stage pose regression supervision, while flow states receive trajectory, visibility, and confidence losses; CoWTracker supplies offline pseudo labels and is not used at inference.Source: B Implementation Details · p. 19

Optimization

  • The two training stages run for 20K and 30K iterations, respectively, using AdamW with a cosine schedule; total training takes about 7.5 days.Videos are sampled to at most 128 frames, and the language backbone has 32 layers with hidden dimension 4096.Source: B Implementation Details · pp. 18–19

Training data.

The training-data description tracks language, camera, and flow supervision across sources, along with each source’s sampling ratio.

Training data

  • Training sources are listed with their language, camera, and flow supervision and sampling ratios; camera labels may be ground-truth poses or offline pseudo poses, and flow labels may be ground-truth trajectories or teacher labels.Source: Training data · p. 19, Table 5 · p. 19

C Additional Experiments

Inference-time masking tests whether the fully trained model depends on camera or flow state tokens. Limited performance changes suggest dynamic information is distributed across the shared multimodal representation after state–language alignment.

C.1 Inference-time State Token Masking

  • Table 6 compares fully trained model variants that differ only in which state tokens are masked at inference.Source: Table 6 · p. 19
  • Masking camera, flow, or both state-token groups at inference produces only limited performance changes.Source: §C.1 · p. 19
  • The limited changes indicate that learned dynamic information is distributed across the shared multimodal representation after state–language alignment.Source: §C.1 · p. 19

C.2 Hyperparameter Sensitivity

The sensitivity analysis varies flow-pathway settings and input-frame counts, with other settings held fixed. CineTech accuracy is relatively insensitive to flow-pathway configuration but falls when temporal coverage is reduced.

C.2 Hyperparameter Sensitivity

  • Flow-token count, spatial grid, temporal window, and refinement iterations each change CineTech accuracy by at most 1.69 points.Source: §C.2 · p. 19, Table 7 · p. 20
  • Reducing the input to 64 and 32 frames decreases CineTech accuracy by 4.35 and 7.01 points, respectively.Source: §C.2 · p. 19, Table 7 · p. 20
  • Table 7 changes only the specified configuration while holding all other settings fixed.Source: Table 7 · p. 20

D.1 Full Per-Category Results

CameraVLM achieves the highest pooled accuracy across the three benchmarks and leads in most camera-motion categories, with particular gains on geometrically challenging primitives. Qualitative examples show it handling varied motions that baselines often confuse.

D.1 Full Per-Category Results

  • 85.48% overall accuracy makes CameraVLM the top model on the pooled items from CameraBench, CineTechBench, and ShotBench.Source: §D.1 · p. 20, Table 8 · p. 20
  • CameraVLM leads in most categories, with particularly strong gains on Crane, Dolly, and Truck, where many baselines fall below 55%.Source: §D.1 · p. 20, Table 8 · p. 20

D.2 Qualitative Examples

  • On pull-out and compound pan-left-push-in clips, only CameraVLM correctly identifies both motions among the compared models.Source: Figure 6 · p. 21
  • Tilt-up examples expose baseline failures on the binary tilt question, while pan-right examples reveal direction confusions or hallucinated crane motion.Source: Figure 7 · p. 21
  • Dolly-zoom, zoom-in description matching, rolling, and boom-up examples test compound-motion parsing, smoothness, rotation direction, and vertical-translation awareness.Source: §D.2 · p. 20, Figure 8 · p. 22, Figure 9 · p. 22

Related papers

Loading 2610.17242v1…