PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling the Value of Motion for Cinematic Camera Trajectories

Ziqi Zhou, Yujian Yuan, Laura Sevilla-Lara

arXiv:2609.38683v1cs.CVcs.AI

TL;DR

Pose-centric camera trajectories entangle movement direction, speed, and location, despite cinematic language describing how a camera moves. The paper introduces DIRSPEED, CINESCRIPT, a contrastive alignment protocol, and CINEGEN; it reports improved alignment and generation performance and preservation of broader cinematic patterns.

  • Problem

    Pose-centric representations combine direction, speed, and spatial location, although human descriptions emphasize how the camera moves, complicating text-to-trajectory generation and cross-modal alignment.

    Source: §1 · p. 1

  • Method

    The paper decomposes poses into normalized direction and log-speed with DIRSPEED, introduces CINESCRIPT and a contrastive evaluation protocol, and builds the masked autoregressive generator CINEGEN.

    Source: §1 · pp. 1–2

  • Results

    CINEGEN achieves superior trajectory quality and text alignment, while generated trajectories preserve broader patterns including era, genre, and directorial style.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that direction-and-speed representations can capture higher-level cinematographic information that traditional absolute poses obscure.

    Source: §8 · p. 9

    The metadata-linked CINESCRIPT subset is relatively small and severely long-tailed, so the authors use attributes to evaluate latent cinematic correlations rather than train explicit era-, genre-, or director-controlled generation.

    Source: K Limitations and Broader Impacts · p. 24

Abstract

from arXiv · show

Cinematic camera motion is a fundamental storytelling tool, defined not only by where the camera is positioned in the scene, but also by how it moves in terms of direction and speed. Recent work on camera trajectory generation and alignment to text relies on pose-centric representations. While in principle a network could derive direction of movement and speed, we find that in practice this might not happen. In fact, in this paper we discover that decomposing the camera trajectory representation from the traditional per-frame poses to direction and speed has surprising benefits across multiple tasks, including trajectory-to-text alignment as well as text-to-trajectory generation. To accurately evaluate the former, we introduce a simple and reliable protocol that overcomes the limitations of prior evaluation baselines. For the latter, building on this representational insight, we propose a novel generative model for camera trajectories, CineGEN, that achieves superior performance across a variety of metrics. We also propose a novel dataset, CineScript, containing movie clips that are enriched with scene descriptions as well as higher-level metadata. This novel data allows us to test models' ability to capture high-level cinematographic information. We show that, despite its simplicity, representing camera trajectories through direction and speed not only helps numerically to achieve better alignment and generation, but also inherently encodes complex directorial intent.

1 Introduction

The paper argues that pose-centric trajectories mismatch language about camera movement, complicating text-to-trajectory generation and cross-modal alignment. It introduces DIRSPEED, CINESCRIPT, a contrastive evaluation protocol, and CINEGEN, reporting benefits for alignment, generation, and exploration of cinematic attributes.

1 Introduction

  • DIRSPEED converts absolute poses into normalized direction and log-speed, aligning trajectory features with descriptions of how a camera moves.It separates movement axis from magnitude rather than requiring a model to infer those factors from global coordinates.Source: §1 · pp. 1–2
  • CINESCRIPT combines movie clips with motion captions, screenplay-style loglines, and linked movie-level attributes such as era, genre, and director.The benchmark contains approximately 28K clips and 10M frames drawn from diverse public sources.Source: §1 · p. 2
  • The paper introduces a lightweight, purely contrastive protocol to measure motion–text alignment without trajectory-reconstruction objectives that can obscure instance-level correspondence.The introduction reports improved retrieval for DIRSPEED over pose-based alternatives regardless of evaluator.Source: §1 · p. 2
  • CINEGEN achieves superior performance in trajectory quality and text alignment, while generated trajectories preserve broader patterns such as era, genre, and directorial style.The paper also reports that its experiments support the value of motion-centric representations and a more reliable contrastive evaluation protocol.Source: §1 · p. 2

2 Related Work

Prior camera-trajectory work expanded generation architectures and task scope, but largely retained pose-centric parameterizations. This paper positions motion-centric representation as a way to bridge physical geometry and language.

Camera trajectory generation

  • Earlier work advanced text-conditioned diffusion, real-movie evaluation, causal autoregression, joint scene generation, actor-camera coherence, and multi-shot planning, while largely inheriting pose-centric representations.The cited examples include CCD [8], E.T., GenDoP [10], Director3D [21], PulpMotion [11], and ShotVerse [22].Source: §2 · p. 2
  • Figure 1 depicts CINESCRIPT construction alongside two trajectory representations, situating the paper's representation comparison within its dataset overview.The overview shows movie clips reprocessed into trajectories, motion captions, screenplay-style loglines, and linked movie attributes.Source: Figure 1 · p. 3

Masked autoregressive models

  • Masked autoregressive models combine bidirectional masked-modeling context with progressive autoregressive commitment, and this work extends continuous-token MAR to camera trajectories.Unlike image and video generation, the approach applies masked autoregression directly in compact continuous trajectory features rather than relying on lossy visual autoencoders.Source: §2 · p. 3

3 DIRSPEED: A Motion-Centric Trajectory Representation

DIRSPEED replaces absolute per-frame poses with a frame-to-frame representation that separates motion direction from speed. Its normalized directions and log-transformed speeds provide a language-aligned, numerically stable feature representation.

3 DIRSPEED: A Motion-Centric Trajectory Representation

  • POSE9D concatenates each frame's continuous 6D rotation representation with its 3D translation, leaving relative motion to be inferred from absolute coordinates.The paper identifies this geometric entanglement as a difficulty for text–trajectory alignment.Source: §3 · p. 3
  • DIRSPEED computes translational and rotational velocities from the pose sequence to represent frame-to-frame camera behavior.The first time step uses zero placeholders for translation and rotation increments to align sequence lengths.Source: §3 · p. 3
  • DIRSPEED decomposes velocity vectors into normalized directions and scalar speeds, separating movement orientation from magnitude.Logarithmic transformation compresses the dynamic range of cinematic movement speeds.Source: §3 · pp. 3–4
  • The resulting 8D per-step feature concatenates bounded directions and log-compressed speeds, which the paper says eases neural-network optimization.The authors describe the representation as both natively language-aligned and numerically stable.Source: §3 · p. 4

4 The CINESCRIPT Dataset

CINESCRIPT extends motion-level camera data with scene loglines and linked movie attributes. Its construction and statistics support analysis of scene context and cinematic patterns beyond local motion descriptions.

4 The CINESCRIPT Dataset

  • CINESCRIPT extends datasets that pair trajectories with motion descriptions by adding screenplay-style loglines and linked real-movie attributes.The linked attributes support exploration of camera behavior and macro-level cinematic style.Source: §4 · p. 4
  • The dataset pipeline reprocesses heterogeneous public movie clips into trajectories, motion captions, screenplay-style loglines, and, when identifiable, movie metadata.Quality filtering and spatial preprocessing precede extraction and annotation.Source: §4 · p. 4
  • CINESCRIPT contains approximately 28K clips and 10M frames, and its metadata-linked subset contains 3,163 clips from roughly 1,400 films.The dataset's average clip duration is 12.0 seconds.Source: §4 · pp. 4–5
  • The dataset is the first described here to jointly include motion captions, scene loglines, and real-movie attributes for a subset of clips.The attributes include release year, genre, and director.Source: §4 · pp. 4–5
  • Extracted motion primitives are dominated by static and simple single-axis motions, while the dataset also contains substantial multi-axis and composite motion.The authors characterize this distribution as reflecting real cinematic camera behavior while retaining motion diversity.Source: §4 · p. 5, Figure 2 · p. 5
  • Clips with the same coarse motion pattern, such as a dolly-in, can have markedly different trajectory styles depending on movie attributes.This observation motivates using metadata to explore cinematic structure beyond basic motion alignment.Source: §4 · p. 5, Figure 2 · p. 5

5 A Closer Look at Evaluation

The evaluation section introduces a contrastive protocol for isolating motion–text alignment and uses real-trajectory probes to test whether camera paths carry higher-level cinematic signals.

5.1 Alignment Evaluation

  • The encoder-only protocol removes CLaTr’s reconstruction objective and trains solely with contrastive loss to isolate instance-level alignment.The paper says CLaTr’s multi-objective design compromises the discriminative margins needed for strict instance-level alignment.Source: §5.1 · pp. 5–6
  • DIRSPEED lifts retrieval R@1 from 17.8 to 25.2 across evaluators, with Fig. 3 showing tighter matched text–trajectory clusters than POSE9D.The comparison is on motion captions; Table 1 evaluates CLaTr and the purely contrastive protocol under both trajectory representations.Source: Table 1 · p. 6, §5.1 · p. 6, Figure 3 · p. 6
  • Alignment uses AlignScore, retrieval recall, and MedR; generated trajectory quality uses motion-primitive F1, FCD, and Manifold Coverage.The section defines these as functional metrics for alignment and for the physical realism and diversity of generated outputs.Source: §5.1 · p. 6

5.2 Attribute-based Evaluation

  • Three real-trajectory probes test Era, grouped Genre, and Director because local motion captions do not capture macro-level stylistic nuances.The genre probe groups categories into three buckets, and the director probe covers Christopher Nolan, Wes Anderson, and Steven Spielberg.Source: §5.2 · pp. 6–7
  • DIRSPEED provides a more reliable signal than POSE9D across Era, Genre, and Director probes, with directorial style the most distinctive attribute.Table 2 reports macro-F1 (×100) on real trajectories.Source: §5.2 · p. 7, Table 2 · p. 7

6 Generation

CINEGEN generates continuous camera trajectories directly in trajectory feature space. It combines masked sequence modeling, conditional denoising, multimodal conditioning, and variance-guided unmasking.

6 Generation

  • CINEGEN generates directly in trajectory feature space, using a Transformer-based masked autoregressive sequencer and a conditional diffusion denoiser for missing values.The design avoids a lossy autoencoding bottleneck and models dependencies among partially visible trajectory tokens.Source: §6 · p. 7
  • During training, masked positions are replaced with a learned embedding; the sequencer uses visible tokens and conditioning signals to compute context vectors.The denoiser models continuous values at masked positions.Source: §6 · p. 7
  • Conditioning combines the motion caption, screenplay-style logline, and initial camera pose, with the signal injected into both sequencer and denoiser.The text components use a frozen CLIP encoder, while the first-pose feature grounds motion to the starting viewpoint.Source: §6 · p. 7
  • At inference, CINEGEN reveals positions with the smallest context-vector variance first, treating lower variance as higher contextual confidence.The process repeats until the trajectory is complete and is converted back to per-frame poses.Source: §6 · p. 8

7 Experiments

On CINESCRIPT, CINEGEN outperforms prior methods across trajectory quality, text alignment, and cinematic attribute preservation. The experiments also examine its qualitative paths, representation variants, and conditioning choices.

7 Experiments

  • The evaluation uses CINESCRIPT and a contrastive protocol with a DIRSPEED encoder, comparing CCD, E.T., and GENDOP in released-checkpoint and CINESCRIPT-retrained forms.Attribute classifiers use a separate movie-level split to avoid sharing clips from the same film across training and validation.Source: §7 · p. 8
  • CINEGEN raises retrieval R@1 from 0.93 to 3.35 versus GENDOP and reduces FCD by nearly 70%, while achieving the highest macro-F1 across attribute probes.The comparison covers trajectory quality, text alignment, and attribute preservation.Source: §7 · p. 8, Table 3 · p. 9
  • Ablations show that reverting to absolute poses causes the most severe degradation, while removing the first-pose anchor, changing the unmasking schedule, or removing loglines also reduces performance.The reported effects include lower trajectory quality, realism, or movie-attribute preservation.Source: §7 · p. 9
  • CINEGEN produces smoother, more coherent paths that better reflect compound motion instructions.The qualitative comparison includes prior generation methods.Source: Figure 5 · p. 9, §7 · p. 9

8 Conclusion

The conclusion presents direction and speed as central to cinematic camera motion and reports benefits for trajectory alignment and generation. The appendix details supplementary experiments and how scene loglines are generated and used alongside motion captions.

8 Conclusion

  • The conclusion frames DIRSPEED as beneficial for trajectory alignment and generation, and says motion-centric representation can capture higher-level cinematographic information obscured by absolute poses.The paper also identifies CINESCRIPT, a contrastive alignment protocol, and CINEGEN as supporting contributions.Source: §8 · p. 9

Appendix

  • The appendix provides supplementary details, extended experimental results, and ablations, covering logline generation, dataset statistics, evaluation protocols, and text choice for alignment evaluation.Source: Appendix · p. 13

A Logline Generation Details

  • Loglines are generated from clips as screenplay-style scene summaries without camera-motion descriptions, then manually reviewed; they provide narrative context complementary to motion captions.Motion captions give more direct text–trajectory correspondence, while loglines are auxiliary conditioning for generation.Source: A Logline Generation Details · pp. 13–14

B Additional Dataset Statistics

The appendix describes CINESCRIPT’s linked movie attributes, evaluation metrics, attribute probes, and extended baseline comparisons. It notes that dataset imbalance shapes attribute evaluation and that model and conditioning choices affect the comparison.

B Additional Dataset Statistics

  • CINESCRIPT’s linked metadata is imbalanced across eras and directors, so attribute probes use frequent-class subsets; its combination of motion captions, loglines, and real movie attributes supports motion- and scene-conditioned generation and high-level evaluation.Figure 6 shows the distributions, including the director long tail.Source: B Additional Dataset Statistics · p. 14, Figure 6 · p. 14, Table 4 · p. 14

C Alignment Evaluation Metrics

C.1 Text–Trajectory Alignment Metrics

  • Trajectory-to-text retrieval is the primary alignment metric because it tests whether text conditioning is recoverable from generated motion; AlignScore measures matched-pair similarity but neither distractor discrimination nor comparable absolute values across evaluator embedding spaces.Retrieval metrics are used for cross-evaluator comparisons.Source: §C.1 · p. 15

C.2 Trajectory-Quality Metrics

  • Trajectory-quality measures include motion-primitive F1, which scores instance-level structural fidelity using quantized translation and rotation labels, and distributional and coverage measures in a learned feature space.The described metrics include Fréchet Camera Distance and Manifold Coverage.Source: §C.2 · pp. 15–16

D Details of Attribute-based Evaluation

  • Era and Director are single-label probes, while Genre is multi-label; evaluation probes measure preserved attribute correlations, not explicit controllable style, and exclude clips outside target classes.The classifiers use a separate movie-level split, with no film shared between training and validation.Source: D Details of Attribute-based Evaluation · pp. 16–17, Table 5 · p. 16

E Extended Baseline Comparison

  • Diagnostic DIRSPEED variants generally improve motion F1 and language alignment, but distributional gains are not uniform; changing GENDOP’s discrete pose-token output to continuous motion alters the modeling demands.The variants retain the baseline generative framework while replacing its native trajectory output.Source: E Extended Baseline Comparison · p. 17, Table 6 · p. 17
  • With conditioning matched to captions only, CINEGEN retains its advantage over GENDOP across reported metrics, while full and caption-only CINEGEN achieve comparable trajectory quality.Caption-only CINEGEN scores higher on caption-based alignment.Source: E Extended Baseline Comparison · p. 17, Table 7 · p. 18

F Human Evaluation

A blinded study compared anonymized camera trajectories on fixed scenes, with participants judging whether each motion was smooth, natural, and followed the reference direction. CINEGEN was selected more often than the strongest baseline.

Protocol

  • In a blinded study of 30 randomly sampled CameraBench clips, participants judged seven anonymized trajectories against each fixed reference scene.Only the generated trajectory changed; trajectories were re-anchored to the source pose and their translation extent rescaled to match the reference.Source: F Human Evaluation · p. 18
  • Twenty-four participants completed all clips, producing 720 method-level judgements; excluding the author-participant changed the selection rate by only 0.2 percentage points.Selection rate is the fraction of judgements selecting a method; sole-selection rate is the fraction selecting it alone.Source: F Human Evaluation · p. 18

Results

  • CINEGEN was selected in 70.1% of judgements and was the sole selection in 30.1%, versus 28.9% and 6.2% for retrained GENDOP.Every participant selected CINEGEN more often than GENDOP individually (two-sided sign test, p < 10−6).Source: F Human Evaluation · p. 18

G Evaluator Robustness and Retrieval Context

CINEGEN remains best under an independent frozen-CLaTr evaluation, while retrieval provides context for interpreting alignment scores. Ablations show that DIRSPEED’s decomposition and CINEGEN’s conditioning and unmasking choices matter to performance.

G Evaluator Robustness and Retrieval Context

  • Under the independent frozen official CLaTr encoder, CINEGEN remains best on all metrics.Table 8 reports this evaluation; unmarked baselines were retrained on CINESCRIPT, while starred entries are released pretrained checkpoints.Source: G Evaluator Robustness and Retrieval… · p. 18, Table 8 · p. 19
  • Absolute cosine values from separately trained evaluators are not directly comparable: CLaTr-DIRSPEED has higher, more dispersed mismatched similarities as well as a larger matched cosine.The section recommends retrieval metrics for cross-evaluator comparison and AlignScore only within a fixed evaluator.Source: G Evaluator Robustness and Retrieval… · p. 19, Table 9 · p. 19
  • CINEGEN reaches R@1 of 3.35%, about 84× random, and R@5/R@10 of 12.67%/19.79%, versus 4.03%/7.33% for retrained GENDOP.The validation pool has approximately 2.5K candidates, and retrieval counts only the designated text–trajectory pair as correct, so retrieval is an alignment diagnostic rather than evidence of near-perfect instance matching.Source: G Evaluator Robustness and Retrieval… · p. 19

H Ablation Studies and Extended Analysis

  • Replacing DIRSPEED with standard poses causes the most severe degradation across quality, alignment, and cinematic metrics; removing the initial geometric anchor or logline also harms performance.Random unmasking slightly raises manifold coverage but harms stability, particularly alignment and directorial-style preservation; Table 10 presents the ablations.Source: H Ablation Studies and Extended Analysis · p. 19, Table 10 · p. 20
  • Normalized direction supplies most of the alignment signal, speed alone is insufficient, and the full DIRSPEED representation achieves the highest R@K and MedR.Raw velocity performs poorly; Table 11 compares direction-only, speed-only, raw-velocity, and full-representation variants using the same contrastive architecture trained from scratch.Source: H Ablation Studies and Extended Analysis · pp. 19–20, Table 11 · p. 20
  • Table 12 compares alignment retrieval using motion captions, captions plus loglines, or loglines alone across 2,578 validation clips.The caption identifies the comparison scope and notes that random motion-to-text R@1 is approximately 0.04%; no outcome values are supplied here.Source: Table 12 · p. 20

H.1 Representation Robustness and Matched Controls

Matched controls indicate that DIRSPEED’s advantage is not explained solely by coordinate or scale conventions. The section also reports data and sequence-length robustness, distinguishes alignment from generation uses of text, and states limitations on cinematic-style control.

H.1 Representation Robustness and Matched Controls

  • Canonicalizing POSE9D and normalizing its trajectory scale improve it only modestly, leaving a substantial gap to DIRSPEED.This indicates that the representation advantage is not explained solely by coordinate or scale conventions; Table 13 presents the matched controls.Source: §H.1 · p. 20, Table 13 · p. 21
  • DIRSPEED remains stable as sequence horizons grow, while POSE9D degrades substantially on long sequences; combining canonicalized poses with DIRSPEED adds only a small gain.DIRSPEED alone captures most of the improvement with less than half the input dimensionality; a pose-to-DIRSPEED-to-pose round trip on trajectories longer than 200 frames has median endpoint errors of 3.84 × 10−9 in translation and 0.130° in rotation.Source: §H.1 · p. 21, Table 15 · p. 21, Table 16 · p. 22
  • DIRSPEED benefits more consistently from larger training sets and exceeds full-data POSE9D with only 50% of the data; larger encoders add little improvement for either representation.With normalized direction fixed, the default logarithmic speed parameterization performs best overall and compresses the long-tailed motion-magnitude distribution.Source: §H.1 · pp. 20–21, Table 14 · p. 21, Table 17 · p. 22

I Experimental Settings

  • The contrastive evaluator typically converges within 30 epochs and about 10 minutes, while CINEGEN training typically takes 150–200 epochs and approximately 12 hours.Experiments use a single NVIDIA A100 GPU.Source: I Experimental Settings · p. 21

I.1 Contrastive Alignment Evaluator Configurations

  • The contrastive evaluator uses symmetric InfoNCE with temperature 0.1, filters highly similar text negatives, and avoids trajectory reconstruction, KL divergence, and decoder overhead.Off-diagonal pairs with text-to-text cosine similarity above 0.99 are masked.Source: §I.1 · p. 22

I.2 CINEGEN Generative Model Configurations

  • CINEGEN predicts denoised trajectory tokens from separately encoded motion captions and loglines, conditioned on multimodal text and initial geometric anchors.It has approximately 28.4M trainable parameters, excluding the frozen text encoder and EMA copies.Source: §I.2 · p. 22

J The Complementary Roles of Motion Captions and Loglines

  • Motion captions yield the strongest retrieval for both DIRSPEED and POSE9D; adding loglines degrades retrieval, and loglines alone nearly collapse recall.For generation, loglines can affect trajectory realism and preservation of movie-attribute correlations; with a fixed motion caption, they can modulate scale, pace, and subtle dynamics, but do not provide explicit era-, genre-, or director-style control.Source: J The Complementary Roles of Motion… · pp. 22–23, Figure 8 · p. 24

K Limitations and Broader Impacts

  • The linked movie-metadata subset is small (approximately 3.2K clips) and severely long-tailed, limiting reliable supervised style control; extracted camera poses may retain estimation noise and VLM loglines may hallucinate or miss narrative cues.The study uses attributes to evaluate latent correlations rather than train an era-, genre-, or director-controlled generator.Source: K Limitations and Broader Impacts · p. 24

Related papers

Loading 2609.38683v1…