PaperlayerView exact revision on arXiv

Source-linked AI summary

Go-with-the-Track: Video Compositing and Motion Control with Point Tracking

Koichi Namekata, Yash Kant, Zhizheng Liu, Ryan D Burgert, Yuancheng Xu, Kuan Heng Lin, Emmett Steven, Julien Philip, Li Ma, Andrea Vedaldi, Paul Debevec, Ning Yu

arXiv:2606.20891v1cs.CVcs.LG

TL;DR

Existing video-generation methods separate precise motion control from reference-image compositing, limiting unified control over appearance and spatiotemporal placement. Go-with-the-Track jointly conditions on multiple reference images and reference-anchored point-tracks, and outperforms all baselines while supporting compositing and camera control across static and dynamic scenes.

  • Problem

    Existing video-generation methods lack unified, editable control over multiple references and their precise spatiotemporal placement.

  • Method

    Go-with-the-Track jointly conditions video generation on multiple reference images and reference-anchored point-tracks using spatially-aware embeddings and a lightweight adapter.

  • Results

    Go-with-the-Track outperforms all baselines and supports point-track-driven compositing and camera control for static and dynamic scenes.

  • Takeaways & Limitations

    The framework provides a single model for controlling when and where reference elements appear and how they move in generated videos.

  • Takeaways & Limitations

    Performance depends on the quality of the supplied point-tracks.

Abstract

from arXiv · show

Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/

1 Introduction

Go-with-the-Track unifies multi-reference compositing and spatiotemporally precise motion control through reference-anchored point-tracks. It introduces spatially aware embeddings, a lightweight adapter, and mixed-data training to support controllable video generation and filmmaking applications.

  • Motivation and limitations: Filmmaking requires unified control over scene appearance and motion, but existing methods either anchor point-tracks to the first frame or lack editable motion representations for multiple references.First-frame anchoring assumes controllable elements are initially visible, while other approaches make spatial-temporal control over multiple reference elements difficult.
  • Applications and training: The framework supports motion-preserved restylization, mesh- or keypoint-driven compositing, and camera control for static and dynamic scenes using multiple references and associated point tracks or point clouds.Training uses a mixture of synthetic datasets to address the lack of accurate ground-truth point-track annotations and the noise introduced by off-the-shelf trackers.
  • Core contribution: Go-with-the-Track jointly controls multiple reference images and their spatiotemporal placement using reference-anchored point-tracks.The point-tracks establish correspondences between generated frames and reference content, enabling a unified conditioning paradigm for motion control and compositing.
  • Spatially-aware conditioning: Spatially-aware point-track embeddings encode coordinates with a coordinate-wise MLP and temporal pooling, producing identifiers that also represent each track’s spatial characteristics.This design addresses the challenge of associating many conditioning point-tracks between generated frames and multiple references, where tracks may not be spatiotemporally continuous.
  • Point-track adapter: A lightweight adapter aggregates pixel-space point-tracks within each 4 × 16 × 16 spatiotemporal block into compressed conditioning vectors for the patchified latent diffusion transformer.The adapter bridges high-resolution point-track coordinates and latent representations while avoiding the motion-detail loss of naive point-track subsampling.

2 Related work

Related work spans increasingly realistic and temporally coherent video diffusion, including transformer-based frameworks, while prior editing methods preserve source-video correspondences. Go-with-the-Track instead separates motion from appearance and jointly uses temporally coherent point-tracks with temporally agnostic reference images for reference insertion and video editing.

  • Video diffusion: Video diffusion models have improved the realism and temporal coherence of generated videos, with transformer-based frameworks such as CogVideoX and Wan advancing scalable synthesis.
  • Video editing: Prior video-editing methods maintain temporal correspondences with the source video, whereas Go-with-the-Track disentangles motion from appearance.
  • Video editing: The model conditions on temporally coherent point-tracks and temporally agnostic reference images, enabling reference insertion beyond standard video-to-video editing.
  • Point-track-to-video generation: Point-tracks extracted from a source video also allow the model to support conventional video editing.

3 Method

Go-with-the-Track is designed as a multi-reference, point-track-conditioned video generator that makes reference-image elements appear and evolve consistently according to specified spatiotemporal conditions. It anchors 2D point-tracks across reference images and generated frames, and combines spatially aware embedding, patch-space adaptation, and mixed-dataset training to support this control.

  • Task and point-track conditioning: The method synthesizes videos from multiple reference images and user-specified 2D point-tracks, with reference elements evolving consistently under their spatiotemporal conditions.
  • Point-track representation: Each point-track’s coordinate sequence is encoded by a shared coordinate-wise MLP and temporal max-pooling into a spatially aware embedding distributed across frames.
  • Task and point-track conditioning: Point-tracks are anchored across input reference images and generated video frames to establish explicit correspondence beyond conventional within-video point flow.
  • Training data: Because off-the-shelf trackers yield noisy or incomplete correspondences under strong camera motion or occlusions, training mixes static-scene, synthetic dynamic-scene, and generic video datasets.DL3DV and TartanAir provide ground-truth depth and camera pose for accurate point-track computation.
  • Reference-image encoding: Reference images are VAE-encoded and concatenated with noisy video tokens, while distinct RoPE indices and timestep embeddings distinguish reference tokens from denoised tokens.

4 Experiments

Experiments evaluate Go-with-the-Track against state-of-the-art baselines on unseen datasets using visual, reconstruction, and motion-fidelity metrics. Results show stronger motion and reference control across arbitrary-frame, mid-dense, sparse, and multi-reference settings, while ablations support the model’s key design choices.

  • Baseline comparisons: Go-with-the-Track outperforms the baselines and supports point-tracks starting and ending at arbitrary frames, unlike ATI, Wan-Move, and DAS.The compared methods include DAS, GWTF, Tora, ATI, and Wan-Move; several baselines cannot use tracks absent from the first frame.
  • Evaluation setup: Evaluation uses unseen DAVIS 2017 and TAPVid3D-ADT videos, covering diverse camera and object motion, occlusions, and ground-truth point-tracks.Metrics span visual fidelity (FID, FVD), reconstruction accuracy (LPIPS, PSNR, SSIM), and motion fidelity via endpoint error (EPE).
  • Baseline comparisons: Go-with-the-Track also outperforms baselines for mid-dense and sparse tracks, including fair comparisons where all tracks are visible in the first frame.The reported advantage is attributed to differences in motion conditioning, particularly prior methods’ VAE-based or naïve track downsampling.
  • Baseline comparisons: The performance gap increases on challenging TAPVid3D-ADT dense-track evaluations, supporting the architecture and real-synthetic data-mixing strategy.The paper hypothesizes that prior methods’ exclusive training on real videos with noisy point-track labels weakens precise motion following.
  • Ablation studies: Replacing spatially-aware point-track embeddings with random vectors causes clear performance drops in motion-related results.Coordinate-based embeddings provide spatial priors that help distinguish and locate tracks across video frames and reference images, improving correspondence, coherence, and controllability.
  • Qualitative and user evaluations: Qualitative comparisons show better source-motion preservation, spatial-structure retention, and subject-identity preservation for reconstruction and restylization than the baselines.A user study with 45 participants further evaluates preferences across methods.

5 Conclusion

Go-with-the-Track unifies point-track-driven motion control and multi-reference compositing for precise appearance and motion control in video generation. Dense point-tracks represent spatio-temporal relationships, controlling when and where reference elements appear and how they move.

  • Unified framework: Go-with-the-Track combines point-track-driven motion control with multi-reference compositing in a unified video-generation framework.The framework targets precise control of both appearance and motion.
  • Spatio-temporal representation: Dense point-tracks represent spatio-temporal relationships between generated video content and reference elements.This representation supports compositing and motion control through explicit track information.
  • Control capabilities: The method controls when and where reference elements appear and how they move throughout generated videos.Its stated goal is reliable control over both the timing, location, and motion of reference elements.

A Dataset preprocessing · B Iterative densification of point tracks · C Details of point-track augmentation.

The preprocessing pipeline filters real videos, enriches point-track coverage through iterative resampling, and augments tracks with complementary missing-coordinate patterns and varied densities. These procedures target dataset quality, sparse-region coverage, robustness to incomplete tracks, and controllable training complexity.

  • A Dataset preprocessing: Real-video preprocessing begins by collecting diverse public datasets and filtering videos with scene cuts, subtitles, minimal motion, or other quality issues that could harm training.The collected sources include OpenVidHD, MiraData, and OpenHumanVid.
  • A Dataset preprocessing: Videos are captioned in a single paragraph under 200 words, describing visual appearance, motion, interactions, and detailed scene or background context.The captioning instructions distinguish descriptions for videos with humans from those featuring animals, vehicles, or other active elements.
  • A Dataset preprocessing: Point-track preprocessing iteratively resamples query points by tracking an initial set, enumerating empty patches on a C×C grid, and adding points at uncovered patch centers.The algorithm specifies a maximum query count N, T iterations, patch size C, and final query set Q.
  • B Iterative densification of point tracks: Uniform query sampling creates sparse regions, especially for small or briefly appearing objects and background elements, motivating iterative point-track densification.Sampling 2,500 queries uniformly over an 81-frame video is cited as an example of this sparsity problem.
  • B Iterative densification of point tracks: Iteratively resampled tracks provide better spatial-temporal coverage and reduced sparsity, particularly for small objects undergoing rapid motion, while remaining practical for large-scale preparation.The method is reported to run on a Tesla T4 GPU with 15 GB memory.
  • C Details of point-track augmentation.: To model sparse or incomplete real-world tracks caused by occlusion or tracking failures, augmentation randomly marks subsets of track coordinates as undefined using three complementary strategies.The strategies include visibility-based dropout, spatial dropout, and a third complementary strategy described in the supplied passage.
  • C Details of point-track augmentation.: Visibility-based dropout discards tracks absent from reference images, requiring the model to infer motion and appearance for content appearing only in generated frames.This simulates motion specification limited to elements present in reference images.

D Model architecture details

The architecture encodes variable-length spatial point-tracks and injects them into the compressed video-latent space through a positional-aware adapter. Relative positional encoding preserves within-block location information and is crucial for precise motion controllability.

  • Spatially-aware point-track embedder: Point-tracks are embedded from valid 2D coordinates spanning generated frames and reference images, while undefined coordinates are excluded, allowing variable-length tracks.Each valid coordinate represents pixel location and frame position before aggregation into a unified track embedding.
  • Point-track adapter: The adapter maps pixel-space point-track embeddings to compressed patchified latent tokens by partitioning videos into non-overlapping 4 × 16 × 16 spatiotemporal blocks.Embeddings from tracks entering each block are gathered and aggregated to match the diffusion model’s latent-token resolution.
  • Point-track adapter: Relative positional encoding is crucial for precise motion controllability because naive blockwise max pooling discards positional information within each block.The method concatenates each point’s relative block coordinate with its track embedding before applying an MLP and pooling.
  • Computational efficiency: The adapter operates on latent tokens and point-track embeddings rather than raw pixels, providing substantial computational savings.This latent-space design is contrasted with prior coordinate downscaling and rounding, which discards fine-grained spatial information and degrades motion controllability.
  • Reference frame handling: Reference images are encoded independently without temporal compression and replicated four times to form static four-frame sequences compatible with the adapter’s temporal pooling.The same handling is applied to the first frame and each reference image.

E Experiment details

The experiments specify how baseline inputs are configured and report the compute used to train the 480p and 720p models. Baseline evaluation includes strict ground-truth motion following for Go-With-The-Flow and first-frame visibility restrictions for DiffusionAsShader.

  • Baseline settings: Go-With-The-Flow receives dense optical flow with zero noise degradation to strictly follow the video’s ground-truth motion.This configuration is used for baseline evaluation.
  • Baseline settings: DiffusionAsShader excludes point tracks not visible in the first frame, matching its original problem formulation.The authors also experimented with modifying input layers to support tracks originating outside the first frame.
  • Training time and resources: The 14B 480p model trains for 7 days on 8×A100 GPUs, while the 720p model trains for 7 additional days on 8×H200 GPUs.The 720p model is initialized from the 480p checkpoint.
  • Training time and resources: Training uses FSDP to improve memory utilization and scalability.FSDP denotes Fully Sharded Data Parallel training.

F Quantitative results on TapVid3D-ADT · G Additional ablation study

On the challenging TapVid3D-ADT dataset, Go-with-the-Track consistently outperforms all baseline approaches, with a larger performance margin than on DAVIS 2017. This further demonstrates the method’s effectiveness and robustness.

  • F Quantitative results on TapVid3D-ADT: Go-with-the-Track consistently outperforms all baseline approaches on the challenging TapVid3D-ADT dataset.The comparison is reported in table 5, alongside the DAVIS 2017 evaluation in table 1.
  • F Quantitative results on TapVid3D-ADT: The performance margin over baselines is larger on TapVid3D-ADT than on DAVIS 2017.This larger margin is presented as evidence of the approach’s effectiveness and robustness.

G.1 Classifier-free guidance

The study varies classifier-free guidance over text prompts and reference images during inference. It selects cfg = 3.0 as the best trade-off, avoiding saturation at higher scales and reduced quality at lower scales.

  • Classifier-free guidance: Classifier-free guidance is evaluated by varying the guidance scale over text prompts and reference images during inference.The tested scales are cfg ∈ {1.5, 3.0, 5.0, 7.0}.
  • Classifier-free guidance: cfg = 5.0 is overly aggressive and occasionally causes image saturation, while cfg = 1.5 slightly reduces video quality.These effects motivate selecting an intermediate guidance scale.
  • Classifier-free guidance: The experiments adopt cfg = 3.0 for all experiments as a trade-off between the higher-scale saturation and lower-scale quality loss.Applying CFG only to text prompts is suboptimal; CFG is instead applied jointly to reference images and text prompts.

G.2 Qualitative ablation analysis

Qualitative ablations show that the proposed design choices are necessary for accurately following point-tracks during reference-conditioned zoom-in generation. Removing or weakening these components causes substantially different zoom behavior, imprecise track adherence, and undesirable pattern-driven overfitting.

  • Qualitative ablation analysis: Random point-track embeddings fail to follow the provided tracks, producing substantially different zoom-in behavior.The qualitative comparison shows that video models struggle with this task without the proposed design choices.
  • Qualitative ablation analysis: Removing the point-track adapter or relative position injection causes imprecise track adherence and a tilted TV.These variants correspond to Random embeddings and Max pool only in Table 2.
  • Qualitative ablation analysis: Models trained exclusively on noisy annotations treat input tracks as unreliable, correcting them instead of faithfully following motion cues and overfitting to specific patterns.This behavior leads to undesirable overfitting rather than faithful adherence to the provided point-track conditions.

G.3 PCA analysis of point-track embeddings

The paper analyzes its spatially-aware point-track embeddings through PCA, finding that they preserve clear spatial correlations unlike random embeddings. This supports their design choice and accompanies a substantial performance improvement over random embeddings.

  • Embedding design: Spatially-aware point-track embeddings substantially improve performance over random embeddings in the comparison reported in table 2.The passage identifies this as a key design choice and reports a substantial improvement, without providing numerical values.
  • PCA visualization: PCA visualization shows that the proposed embeddings exhibit clear spatial correlations, whereas random embeddings do not preserve meaningful spatial structure.Figure 11 provides the qualitative comparison between the two embedding types.

H Details of Qualitative Comparison · I Details of Applications

Go-with-the-Track qualitatively improves reconstruction and restylization by preserving source motion, spatial structure, object identity, and stylized appearance. Its applications extend to sparse-track generation, compositing, camera retargeting, and temporally stabilized intrinsic decomposition.

  • H Details of Qualitative Comparison: H Details of Qualitative Comparison: In reconstruction, Go-with-the-Track more closely aligns with ground truth while better preserving spatial structure and object identity.
  • H Details of Qualitative Comparison: H Details of Qualitative Comparison: In restylization, the model faithfully preserves source-video motion while adhering to the stylized first frame’s appearance.
  • H Details of Qualitative Comparison: H Details of Qualitative Comparison: Even with extremely sparse point tracks, the model generates videos that faithfully follow the specified motion conditions despite not being explicitly trained for such inputs.
  • I Details of Applications: I Details of Applications: Video restylization uses tracked source videos with full-frame or segmented cropped-object reference images edited by NanoBanana and SAM 3.
  • I Details of Applications: I Details of Applications: Using reprojected point tracks and four uniformly sampled reference frames, Go-with-the-Track retargets camera motion in static and dynamic scenes along new trajectories.Invisible reprojected points are discarded to prevent noisy reprojections.
  • I Details of Applications: I Details of Applications: Propagating albedo and shading estimates from the first and last frames produces temporally consistent intrinsic-decomposition videos with reduced flicker versus frame-by-frame estimation.
  • I Details of Applications: I Details of Applications: The model supports mesh-driven compositing and stylization, transferring rendered static or dynamic mesh scenes and projected-vertex point tracks into stylized animated videos.
  • I Details of Applications: I Details of Applications: Keypoint-derived reference-anchored point tracks enable facial and full-body appearance transfer from a reference image while preserving the source video’s motion.
Loading 2606.20891v1…