PaperlayerView exact revision on arXiv

Source-linked AI summary

TKCAM: Text and Keyframe to Camera Trajectory Generation

Haozhe Yang, Zhiyang Dou, Zekai Gu, Cheng Lin, Wenping Wang, Yuan Liu, Taku Komura

arXiv:2610.11105v1cs.CV

TL;DR

Generating camera motion from text is difficult because language underspecifies geometry and timing, while existing data and evaluation tools have limitations. TKCAM uses hierarchical motion tokens and masked transformers with text and sparse-keyframe conditioning. It reports the best FID, matching, and retrieval scores among compared methods on a mixed-domain benchmark.

  • Problem

    Text underspecifies camera-motion geometry and timing, and existing datasets and tools rarely provide reliable camera annotations with temporally grounded captions or standardized text-motion evaluation.

    Source: §1 · pp. 1–2

  • Method

    TKCAM quantizes 12-dimensional camera trajectories into hierarchical RVQ tokens and uses two masked transformers for multimodal trajectory generation and refinement.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    TKCAM achieves the best FID, text-motion matching, and retrieval scores among compared methods on the mixed-domain benchmark.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sparse RGB keyframes provide temporally localized visual guidance for generating coherent in-between trajectories alongside text prompts.

    Source: §1 · p. 2

    Training data is mainly indoor real-estate footage, leaving highly dynamic or stylized cinematography underexplored.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generating high-quality and controllable camera motion is essential for AI-assisted cinematography, video synthesis, and 3D scene understanding. We introduce TKCAM, a Text- and Keyframe-conditioned CAMera-motion synthesis framework based on generative masked modeling. We represent camera dynamics using a 12-dimensional kinematic feature comprising position, velocity, and a continuous rotation representation and discretize them into hierarchical motion tokens via a Residual Vector Quantizer (RVQ). A two-stage masked transformer architecture then learns to reconstruct and refine these tokens, utilizing explicit self- and cross-attention modules for multimodal conditioning. A central feature of our framework is sparse visual keyframe conditioning: users can provide free-form text prompts together with RGB observations at selected timestamps, which provide temporally localized visual guidance for generating coherent in-between trajectories. Furthermore, to advance evaluation standards, we curate RealEstate10K-Cap, a large-scale text-camera dataset, and establish a cross-domain benchmark with a Universal CLaTr Evaluator. Extensive experiments demonstrate that TKCAM surpasses recent state-of-the-art baselines on Fr\'echet distance (FID), text-motion matching scores, and retrieval metrics (R@K), while additional analyses evaluate temporal smoothness and cross-domain generalization. Code is available at https://github.com/linearalgebrayhz/TKCAM.

1 Introduction

TKCAM addresses the challenge of generating metrically consistent, semantically aligned camera motion from underspecified text, with sparse keyframes as visual guidance. It introduces a captioned trajectory dataset and cross-domain evaluator, and reports top scores on its mixed-domain benchmark.

1 Introduction

  • Text-to-trajectory generation remains difficult because language underspecifies geometry and timing, while existing datasets and evaluation tools offer limited caption grounding and alignment assessment.The introduction also notes that direct regression can drift semantically and miss fine-grained visual semantics or temporal emphasis.Source: §1 · pp. 1–2
  • Sparse RGB keyframes at selected timestamps provide visual guidance alongside free-form prompts for generating coherent in-between trajectories.The framework also supports text-only generation.Source: §1 · p. 2
  • TKCAM uses hierarchical RVQ motion tokens and masked transformers with decoupled motion self-attention and multimodal cross-attention.The residual transformer refines higher quantization levels in a coarse-to-fine manner.Source: §1 · p. 2
  • RealEstate10K-Cap pairs camera trajectories with temporally grounded directorial captions, and the Universal CLaTr Evaluator supports cross-domain evaluation.The evaluator is retrained on a balanced mixture of indoor, cinematic, and character-centric trajectory data.Source: §1 · p. 2
  • TKCAM achieves the best FID, text–motion matching, and retrieval scores among compared methods on the mixed-domain benchmark.The experiments also report semantically faithful trajectories with smooth translational motion under text-only and text-keyframe settings.Source: §1 · p. 2

2 Related Work

Prior camera-trajectory work spans optimization, regression, diffusion, and discrete sequence modeling, alongside datasets that trade geometric reliability against cinematographic expressiveness. TKCAM uses masked discrete generation to support parallel decoding and bidirectional keyframe conditioning.

2 Related Work

  • Earlier trajectory-generation approaches include optimization-based planning, continuous regression, VLM-based planning, interactive sketch authoring, diffusion, and discrete sequence models.Direct regression is described as prone to high-frequency spatial jitter and weak semantic alignment.Source: §2 · p. 2
  • Diffusion-based camera generators require dozens of iterative denoising steps, while autoregressive models can accumulate errors and cannot naturally condition on future observations.The latter makes filling the gap between start and end keyframes more difficult.Source: §2 · p. 3
  • Existing datasets trade calibrated trajectories in structured or synthetic scenes against richly described film trajectories that can contain SfM-disrupting artifacts.RealEstate10K-Cap is presented as stable indoor captures with scene-grounded captions rather than a new captioning strategy.Source: §2 · p. 3
  • TKCAM combines a 12-dimensional kinematic representation, RVQ, and masked generation to avoid diffusion's iterative sampling and autoregressive unidirectionality.Its holistic trajectory prediction supports parallel decoding and bidirectional keyframe conditioning through multimodal cross-attention.Source: §2 · p. 3

3 Dataset

RealEstate10K-Cap is built from filtered indoor camera trajectories and paired with captions that combine geometric motion descriptions with visual scene context. Its motion distribution and scale are reported alongside comparisons with film-derived datasets.

3 Dataset

  • Preprocessing excludes videos shorter than 4 seconds, truncates motions longer than 10 seconds, and filters near-static segments, leaving approximately 25K high-fidelity trajectories.The source is RealEstate10K, with SfM-reconstructed trajectories across indoor scenes.Source: §3 · p. 3
  • Captioning derives a rule-based motion prior from camera poses, then uses a VLM to ground it in visual context while treating the motion prior as an immutable constraint.The VLM samples up to 32 frames; without visual frames, captions fall back to the rule-derived description.Source: §3 · pp. 3–4
  • Static segments comprise 2.6% of RealEstate10K-Cap versus 53% of DataDoP and 38% of E.T.; compound motion comprises 71% versus 9% and 14%, respectively.The top-13 patterns leave 35.1% of sequences across 50+ other patterns.Source: Figure 1 · p. 4, §3 · p. 4
  • The dataset contains approximately 25K sequences, 5 million frames, and 50 hours of footage, averaging 201.7 frames per trajectory.Sequences average approximately 6.72 seconds at 30 FPS.Source: §3 · p. 4

4 Method

TKCAM represents trajectories as hierarchical RVQ tokens and predicts them with two masked transformers conditioned on text and sparse visual keyframes. The base transformer establishes motion structure, while the residual transformer refines higher-level details before decoding.

4.1 Motion Representation and Quantization

  • The 12-dimensional camera representation combines normalized position, per-frame linear velocity, and a continuous 6D rotation representation.Velocity acts as implicit temporal regularization, and the rotation representation avoids singularities.Source: §4.1 · p. 5
  • RVQ maps encoded trajectories to summed codewords across quantization levels, with lower levels capturing coarse patterns and higher levels refining high-frequency details.Training randomly truncates active quantizers with probability 0.2.Source: §4.1 · p. 5
  • The RVQ autoencoder is trained with reconstruction, explicit kinematic, and commitment losses, including an orthogonality term for predicted 6D rotations.The explicit kinematic loss separately supervises translation-velocity and rotation components.Source: §4.1 · p. 6

4.2 Masked Generative Modeling

  • The base masked transformer predicts primary motion tokens with motion self-attention followed by cross-attention to text and keyframe features.Timestamp-aligned visual positional embeddings distinguish keyframes, which guide generation without clamping metric camera poses.Source: §4.2 · p. 6
  • A residual masked transformer predicts higher-level tokens conditioned on prior quantization levels and the same multimodal context, refining motion coarse-to-fine.At inference, the base transformer reconstructs masked tokens in parallel, the residual transformer refines them, and the RVQ decoder returns a continuous trajectory.Source: §4.2 · pp. 6–7

5 Experiment

On a mixed-domain benchmark, TKCAM is evaluated under text-only and keyframe-conditioned settings against camera-motion baselines. It outperforms baselines on reported generation metrics, while ablations test its representation, fusion, and RVQ design.

5 Experiment

  • TKCAM trains only on RealEstate10K-Cap and is tested zero-shot on E.T. and DataDoP, using ground-truth horizons and Universal CLaTr metrics validated against 90 human-rated pairs.Universal CLaTr correlates positively with mean human judgments (Spearman ρ = 0.312, 95% CI [0.089, 0.492]).Source: §5 · p. 7

5.1 Main Results

  • With sparse keyframes, TKCAM reaches FID 0.503 and R@10 12.67%; first-frame conditioning yields FID 0.518 and R@10 12.27%, exceeding depth-conditioned GenDoP's FID 0.616 and R@10 11.07%.RGB keyframes also provide soft visual guidance rather than explicit metric camera-pose constraints.Source: §5.1 · p. 7, Table 2 · p. 8, §5.2 · p. 8
  • Text-only TKCAM leads baselines with FID 0.529, Matching Score 0.100, and R@10 9.60%, while qualitative paths capture complex compound motions.The figure comparison notes baselines struggle with motion magnitude, directional alignment, or trajectory quality.Source: §5.1 · p. 7, Table 2 · p. 8, Figure 3 · p. 9

5.2 Ablation Studies

  • Removing velocity (9D) worsens FID 0.503→0.593, Matching 0.129→0.106, R@10 12.67%→10.66%, and ∆Div 0.070→0.074.The authors interpret explicit velocity as an implicit temporal regularizer.Source: §5.2 · p. 8
  • Decoupled cross-attention beats Prefix Concatenation on FID 0.503 vs. 0.580, R@10 12.67% vs. 12.07%, and Matching 0.129 vs. 0.119.The authors associate this with improved fine-grained temporal grounding.Source: §5.2 · p. 8
  • Across RVQ depths, L1 has the highest R@1 but largest diversity gap; L1–L4 achieves best R@10 with competitive FID and diversity, trading exact-match retrieval.Table 3 reports the hierarchical RVQ-level ablation.Source: §5.2 · p. 8, Table 3 · p. 8

6 Conclusion

TKCAM combines RVQ-discretized camera motion with a two-stage masked transformer for text- and keyframe-conditioned trajectory generation. RealEstate10K-Cap spans a long-tailed distribution of motion patterns, while the paper identifies several remaining limits in TKCAM's scope and smoothness.

6 Conclusion

  • TKCAM uses 12-dimensional kinematic sequences, RVQ-VAE, and a two-stage masked transformer for multimodal fusion, refinement, and animator-style keyframe conditioning.Experiments with the Universal CLaTr Evaluator demonstrate strong distributional realism and cross-modal alignment.Source: §6 · p. 9
  • Rotational smoothness remains weaker than translational smoothness, and the model requires a predefined generation horizon.Training is mainly on indoor real-estate footage, leaving highly dynamic or stylized cinematography underexplored; the framework focuses on single-camera extrinsic motion.Source: §6 · p. 10

A.1 Full Motion Distribution

  • Arc+forward accounts for ∼35% of sequences, while the remaining 65% spans 50+ diverse motion patterns.Figure 5 extends the main-text Top-13 summary to all identified patterns.Source: Figure 5 · p. 13, §A.1 · p. 13

A.2 Data Captioning Pipeline

RealEstate10K-Cap captions are built through two stages that keep descriptions consistent with camera trajectories while adding scene context. Geometric motion is first classified from camera poses, then visually grounded using sampled frames and a prompt that preserves the motion description.

A.2 Data Captioning Pipeline

  • The two-stage captioning pipeline is designed to keep motion descriptions consistent with trajectories while adding scene context.It first derives a geometric motion prior, then enriches it with visual context.Source: §A.2 · p. 14
  • Stage 1 integrates local camera velocities and applies rule-based kinematic thresholds to classify translational and rotational movements from relative camera poses.Simultaneous lateral tracking and yaw are merged into compound descriptions such as “arc left”.Source: §A.2 · p. 14
  • Stage 2 samples up to 32 frames per sequence and feeds them with the geometric prior to Qwen3-VL-4B-Instruct for visual grounding.Generation uses temperature 0.7, top-p 0.9, and at most 128 new tokens.Source: §A.2 · p. 14
  • The prompt requires the generated description to preserve the provided motion sequence and include movement type, direction, and pace for each stage.Descriptions use “then” or “finally” to separate stages and merge motion with key scene elements.Source: §A.2 · p. 14

B Detailed Training Objectives

The appendix details TKCAM's training objectives and evaluation protocols, including a controlled comparison of RealEstate10K-Cap and DataDoP as training sources. It also specifies trajectory-length handling and the predefined horizon assumption.

B Detailed Training Objectives

B.1 RVQ Autoencoder Objective

  • The RVQ objective combines global reconstruction, explicit kinematic, and commitment losses.The explicit kinematic loss separately supervises translation-velocity and 6D rotation components, including rotation-vector orthogonality.Source: §B.1 · pp. 14–15

B.2 Masked Transformer Objective

  • The masked transformers predict masked motion tokens; the residual transformer minimizes conditional masked prediction loss over time for a sampled active layer.The base model's cross-entropy loss applies only to masked non-padding positions.Source: §B.2 · p. 15

C.1 Full List of Hyperparameters

C.2 Baseline Training and Evaluation Protocol

  • Baselines are evaluated on the same mixed-domain benchmark, trajectory-length protocol, and Universal CLaTr evaluator, with matched retraining where available.CCD and E.T. are retrained on RealEstate10K-Cap; GenDoP is evaluated in original and fine-tuned configurations, while Director3D uses its released pretrained model.Source: §C.2 · p. 15

C.4 Controlled Dataset Comparison

  • The RealEstate10K-Cap-trained model achieves lower FID and a smaller diversity gap across all four test domains than the equally sized DataDoP-trained model.The comparison includes the DataDoP test subset and E.T., which is out-of-domain for both training sources.Source: Table 5 · p. 16, §C.4 · p. 16

C.3 Trajectory-Length Handling

  • TKCAM uses each sample's ground-truth trajectory length at evaluation, while generation substantially beyond the training-length distribution has not been systematically evaluated.Its tokenizer decodes to a multiple of four frames before resampling to the target length; all methods use the same target duration.Source: §C.3 · p. 16

C.5 Temporal Smoothness Analysis

The analysis compares generated derivative distributions with ground truth to assess temporal smoothness. TKCAM closely matches translational dynamics under sparse-keyframe conditioning, while rotational acceleration and jerk remain less well matched than GenDoP-RGBD.

C.5 Temporal Smoothness Analysis

  • The analysis measures translational acceleration and jerk plus angular velocity, acceleration, and jerk, using Wasserstein distance to compare derivative distributions with ground truth.Raw derivative magnitudes alone may favor over-smoothed trajectories.Source: §C.5 · p. 16
  • Sparse-keyframe TKCAM achieves the smallest Wasserstein distances for translational acceleration and jerk, while rotational acceleration and jerk are less well matched than GenDoP-RGBD.Table 6 reports mean derivative magnitudes and Wasserstein distances to ground-truth distributions.Source: §C.5 · pp. 16–17, Table 6 · p. 17

C.6 Human Validation of Universal CLaTr

A pilot blinded study found that human ratings ranked TKCAM highest, while Universal CLaTr broadly tracked human judgments but disagreed on TKCAM versus GenDoP. Additional qualitative examples show mostly correct directional semantics across varied motion types and scene contexts.

C.6 Human Validation of Universal CLaTr

  • Inter-rater agreement was moderate (Krippendorff’s α = 0.49), while the three-rater averaged score had higher reliability (ICC(2, k) = 0.76).The analysis uses mean ratings across raters.Source: §C.6 · p. 17
  • Universal CLaTr correlated positively with mean human judgments across 90 caption–trajectory pairs (Spearman ρ = 0.312; generated trajectories: ρ = 0.395).The overall caption-bootstrap 95% CI was [0.089, 0.492].Source: §C.6 · p. 17
  • Human raters scored TKCAM highest at 3.30, and Universal CLaTr preserved five of six pairwise orderings among generated methods, with TKCAM versus GenDoP indistinguishable.The study used 18 captions, five trajectory sources, and three raters.Source: §C.6 · p. 17

D.1 More Qualitative Comparison

  • Across additional examples of translational and compound motions, TKCAM usually produced correct directional semantics.The examples span dolly-forward, lateral tracking, and motions combining arcing, dollying, and tilting.Source: §D.1 · pp. 18–19, Figure 6 · p. 18

D.2 Failure Cases of TKCAM and Analysis

TKCAM’s identified failures are rare and concentrated near training-distribution boundaries, including underrepresented vertical motion and low-magnitude secondary actions. The discussion also describes potential applications and misuse risks, alongside a trajectory-only constraint and planned usage policy.

D.2 Failure Cases of TKCAM and Analysis

  • When backward dollying co-occurs with tilting down, TKCAM may capture translation but neglect rotation, consistent with horizontal-navigation bias in RealEstate10K-Cap.The authors attribute this to underrepresented tilt motions in indoor real-estate tours.Source: §D.2 · p. 19
  • In two-stage prompts, a dominant dolly may be generated while a later, lower-magnitude pan is omitted, especially when it is brief and subtle.The authors attribute this occasional omission to under-sampling during masked token prediction.Source: §D.2 · p. 19
  • The reported failures are rare edge cases tied mainly to out-of-distribution vertical motions and low-magnitude secondary actions, not typical generation behavior.The authors state that Table 2’s quantitative performance indicates these misalignments are not typical.Source: §D.2 · p. 19

E Ethical and Societal Impact Discussion

  • The authors describe potential benefits for non-expert camera-path authoring in filmmaking, pre-visualization, accessibility tools, and cinematography education.They also identify downstream research applications including video generation, 3D scene understanding, and embodied AI.Source: E Ethical and Societal Impact Discussion · p. 20
  • The authors note potential misuse in deceptive video synthesis or synthetic surveillance footage and plan a usage policy; TKCAM itself generates trajectories, not visual content.Visual output requires a separate video-synthesis component.Source: E Ethical and Societal Impact Discussion · p. 20
Loading 2610.11105v1…