Go-with-the-Track: Video Compositing and Motion Control with Point Tracking

1 Introduction

• Filmmaking needs consistent reference appearances and precise spatiotemporal control, but existing motion-conditioned and reference-conditioned approaches address these requirements separately.First-frame-anchored point-track models [Chu et al. 2025; Geng et al. 2025; Gu et al. 2025b; Hou et al. 2024; Ji et al. 2025; Jin et al. 2025; Lei et al. 2025; Li et al. 2025a; Ma et al. 2023; Namekata et al. 2024; Qiu et al. 2024; Wang et al. 2023] struggle with controllable elements entering later. Multi-reference and personalized models [Bian et al. 2025; Blattmann et al. 2023; Chefer et al. 2024; Chen et al. 2023, 2025a; Deng et al. 2025; Jiang et al. 2023, 2025; Liu et al. 2025; Sang et al. 2025; Wang et al. 2024a] typically lack explicit, editable motion representations.Source: §1 · pp. 1, 3
• Go-with-the-Track jointly controls reference appearance and motion by anchoring point-tracks to corresponding coordinates in multiple reference images and generated frames.Each track specifies both which reference element is inserted and how it moves, supporting elements entering, interacting, and exiting at chosen locations and times.Source: §1 · p. 3
• Spatially-aware point-track embeddings use a coordinate-wise MLP and temporal pooling to create track identifiers whose similarity correlates with spatial proximity.These spatial cues help associate points across reference images and generated frames despite disjoint coordinates, addressing the challenge of matching up to 15,000 conditioning tracks.Source: §1 · p. 3
• A lightweight adapter aggregates pixel-space tracks within each 4 × 16 × 16 spatiotemporal block into one conditioning vector for the compressed diffusion transformer.Built on the video latent diffusion transformer [Wan 2025], the adapter bridges pixel-to-patch resolution differences without substantial motion-detail loss and with minimal computational overhead.Source: §1 · p. 3
• Hybrid training combines synthetic, static-scene, and dynamic-scene datasets to learn precise motion control while preserving diverse, photorealistic video priors.Synthetic and static-scene data supply ground-truth tracks from mesh vertices or depth and camera annotations, reducing reliance on noisy tracker-generated supervision. Dynamic-scene videos provide diversity and photorealism.Source: §1 · p. 3
• Go-with-the-Track reports outperforming baselines in quantitative metrics and user studies while enabling restylization, mesh- or keypoint-driven compositing, and camera control.Restylization preserves tracked motion while changing appearance through multiple references; mesh vertices or detected keypoints drive compositing and stylization. Camera control covers static and dynamic scenes using references from different viewpoints and time steps with associated point clouds.Source: §1 · pp. 3–4

Reference-to-video generation

• Go-with-the-Track extends reference-to-video conditioning with point-tracks to enable precise, spatially controlled reference insertion.It follows the existing paradigm of temporally concatenating VAE-encoded reference images and fine-tuning video diffusion transformer self-attention layers.Source: §2 · p. 4

Video diffusion

• Video diffusion has improved realism and temporal coherence, but precise control over generated content and motion remains an open challenge.This motivates reference-image and point-track conditioning beyond text prompts.Source: §2 · p. 4

Point-track-to-video generation

• Learnable spatially-aware point-track embeddings encode spatial characteristics as unique identifiers, helping the model associate and distinguish tracks.Random RGB identities [Burgert et al. 2025a] and sinusoidal embeddings [Burgert et al. 2025b; Shin et al. 2025; Wang et al. 2025a] struggle with spatiotemporally discontiguous correspondences. MotionV2V [Burgert et al. 2025a] reports handling only 25 point-tracks optimally.Source: §2 · p. 4

Video-to-video generation

• Go-with-the-Track disentangles motion from appearance by combining temporally coherent point-tracks with temporally agnostic reference images.This enables reference insertion beyond standard video-to-video editing, while tracks extracted from source videos also support conventional video editing.Source: §2 · p. 4

Point-track-to-video generation

• Most point-track-conditioned methods assume objects appear in the first frame or are fully text-specified, limiting their handling of new content from other frames.First-frame coordinate encoding [Gu et al. 2025b; Zhang et al. 2025b] preserves spatial smoothness but restricts reference conditioning to the first frame.Source: §2 · p. 4

3.1 Task formulation and inputs

• Reference-anchored point-tracks connect multiple input images to generated frames, enabling reference-content placement and motion control with sparse or dense guidance.Sparse tracks provide soft motion guidance, whereas dense tracks enforce strong spatial alignment; invisible or filtered coordinates remain undefined. Generated-only tracks constrain motion, while reference-only tracks are discarded.Source: §3.1 · pp. 4–5

3.2 Training datasets and preprocessing

• Hybrid training combines static scenes, synthetic dynamic scenes, and general videos to address unreliable tracked correspondences and improve real-world generalization.Synthetic-to-real mismatch makes tracker annotations noisy or incomplete under strong camera motion or occlusion. DL3DV [Ling et al. 2024] and TartanAir [Wang et al. 2020] provide depth/pose-derived tracks; PointOdyssey [Zheng et al. 2023] provides vertex tracks to disentangle camera and object motion; DELTA [Ngo et al. 2024] annotates general videos.Source: §3.2 · p. 5
• Iterative track densification detects and fills sparsely tracked regions, maintaining scene coverage when occlusions undermine simpler sampling strategies.Training examples pair 49 consecutive target frames with 1–4 reference frames selected from inside and outside the target sequence, together with their tracks. All videos receive Gemini 2.5 Flash [Google DeepMind 2025] captions for joint text-conditioned training.Source: §3.2 · p. 5
• Reference-image transformations prevent copy-paste shortcuts, while track augmentations improve robustness to incomplete tracking and varying track densities.Random zoom, crop, resize, and translation break spatial alignment between references and target frames, forcing track-based spatial reasoning. Visibility-based dropout, spatial dropout, moving foreground overlays, and varying track counts address missing coordinates and prevent missing tracks from being interpreted automatically as occlusions.Source: §3.2 · pp. 5–6

3.3 Go-with-the-Track modeling and training

• VAE-encoded reference images are concatenated with noisy video tokens, using distinct positional and timestep encodings to separate references from denoising targets.Reference RoPE indices start at 100, following [Cao et al. 2025]. A learnable embedding initialized from timestep 0 replaces reference timestep embeddings, following [He et al. 2025a], preserving pretrained weights.Source: §3.3 · p. 6
• Spatially aware track embeddings use sinusoidally encoded coordinates and frame indices, a shared coordinate-wise MLP, and temporal max-pooling.Only generated-frame coordinates are encoded: their continuous, distinct sequences identify tracks, while nearby tracks produce similar embeddings. This addresses spatially uncorrelated random embeddings that hinder reference insertion, convergence, and generalization.Source: §3.3 · p. 6
• The point-track adapter aggregates pixel-space embeddings into latent tokens while preserving fine-grained positions within each spatiotemporal block.Each 4 × 16 × 16 block collects track embeddings, concatenates their relative positions, and applies MLPs before max-pooling. Adapter outputs are concatenated channel-wise with patchified video tokens, then projected back to the original channel dimension.Source: §3.3 · p. 6
• 480 × 832 resolution and 49 frames are used for training, with new components trained from scratch and transformer layers selectively fine-tuned.Wan 2.1-T2V-1.3B/14B and Wan-2.2 T2V-14B [Wan 2025] backbones use AdamW [Loshchilov 2017], lr 1e-5, weight decay 1e-3, 1000-step warmup, and up to 24K iterations. Self-attention and feed-forward layers are fine-tuned while cross-attention remains frozen; prompts and references are independently dropped with probability 0.15 for joint classifier-free guidance.Source: §3.3 · p. 6

4 Experiments

• 3K dense, 512 mid-dense, and 32 sparse tracks test control using first-frame references on held-out DAVIS 2017 and TAPVid3D-ADT videos.Evaluation uses 77 DAVIS videos and 50 TAPVid3D-ADT videos; only dense tracks may start after the first frame, while Tora and ATI inputs are capped at 10 and 40 tracks respectively. Metrics cover visual fidelity through FID/FVD, reconstruction through LPIPS/PSNR/SSIM, and motion through EPE, the L2 distance between visible input tracks and CoTracker3 [Karaev et al. 2024a] estimates.Source: §4 · pp. 6–7

4.1 Quantitative comparisons

• Go-with-the-Track outperforms baselines across dense, mid-dense, and sparse tracks, with a larger dense-track advantage on TAPVid3D-ADT.It uses tracks starting and ending at arbitrary frames, unlike ATI, Wan-Move, and DAS, while avoiding motion-detail loss attributed to baseline downsampling. GWTF’s optical flow struggles with long-range movements under occlusion.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 Ablation study

• Random embeddings reduce motion controllability and visual fidelity, while real-video-only training degrades EPE and reconstruction accuracy on both evaluation datasets.Coordinate-based embeddings supply spatial priors for associating tracks with reference-image regions. Removing static-scene and synthetic training data causes the real-video-only degradation.Source: Table 2 · p. 8, §4.2 · pp. 8–10
• Max pooling with relative-position concatenation yields the best adapter results, outperforming random sampling, position-free pooling, and attention pooling.The authors hypothesize that max pooling preserves salient track identities rather than smoothing them through weighted averaging. Qualitative ablations show geometric distortions and severe motion-following failures when embeddings, the adapter, relative-position injection, or hybrid training are removed.Source: Table 2 · p. 8, Figure 4 · p. 8, §4.2 · pp. 9–10

4.1 Quantitative comparisons

• Additional references consistently improve results, while performance remains stable across reference positions and robust to cropped, resized references outside the generation interval.Tests cover individual first, middle, or last frames; first-and-last references; and four references. This comparison uses Wan2.1 14B trained for 18,000 iterations.Source: §4.1 · p. 7, Table 3 · p. 9

4.3 Qualitative comparisons

• More than twice as many preferences favor Go-with-the-Track over the second-best method in a user study assessing motion following, subject preservation, and overall quality.Forty-five participants evaluated 30 randomly selected videos. Reconstruction and restylization comparisons also show closer source-motion alignment and better preservation of spatial structure and first-frame subject identity than baselines.Source: Figure 5 · p. 9, Table 4 · p. 10, §4.3 · p. 10

4.4 Applications

• Video restylization and mesh-driven compositing preserve source motion or animation while adopting stylized reference appearances.Restylization uses tracked source videos and segmented, cropped, stylized subject references; mesh compositing uses vertex trajectories under a specified camera path. Demonstrated mesh animations include human motion and scene interactions.Source: §4.4 · p. 10
• Keypoint-driven appearance transfer and reprojected 3D tracks enable human compositing and camera control in both static and dynamic scenes.Facial/body keypoint correspondences transfer a reference person’s appearance; camera control projects static point clouds or densified dynamic 3D tracks onto target paths using estimated cameras, excluding invisible dynamic points. Camera retargeting uses four source-frame references and supports spiral paths, multi-view interpolation, and substantially changed viewpoints.Source: §4.4 · p. 10

• Go-with-the-Track combines point-track-driven motion control with multi-reference compositing, using dense point-tracks to control when and where reference elements appear and how they move.Source: §5 · p. 10
• Experiments demonstrate superior controllability and video quality, while qualitative examples showcase mesh stylization, motion transfer, camera control, and track-level editing.Source: §5 · p. 10
• A spatially-aware point tracking ID embedding uses a lightweight encoder to map track identities into the patchified latent space.Source: §5 · p. 10
• Fast motion or incorrect visibility estimates can cause temporal artifacts, while track spatial resolution limits precision for detailed contact and subtle motions.Output quality also depends on the underlying video diffusion model.Source: §5 · p. 10

A Dataset preprocessing

• Approximately 500K curated real videos remain after filtering prominent text overlays, abrupt scene cuts, and insufficient motion.Filtering uses OCR [Cui et al. 2025a,b], scene-cut detection [Souček and Lokoč 2020], and optical flow.Source: A Dataset preprocessing · p. 14
• DELTA [Ngo et al. 2024] tracks real videos at 384 × 512 using depth aligned from disparity and metric estimates.VideoDepthAnything [Chen et al. 2025b] estimates disparity while preserving fine foreground geometry, and UniDepthV2 [Piccinelli and et al. 2024] estimates metric depth. Least-squares alignment resolves disparity scale ambiguity.Source: A Dataset preprocessing · p. 14
• Gemini 2.5 Flash [Google DeepMind 2025] supplies captions after manual comparison found greater accuracy and faithfulness than Qwen2.5 [Yang et al. 2024b].Captions complement references and tracks with contextual guidance. The prompt requests a single paragraph under 200 words describing appearance, motion, interactions, scene, lighting, and camera work.Source: A Dataset preprocessing · p. 14
• 11:3:3:3 sampling combines real videos, DL3DV, PointOdyssey, and TartanAir, respectively, mixing real/synthetic and static/dynamic training sources.This produces approximate synthetic-to-real and static-to-dynamic ratios of 3:7, alongside a 9:11 ratio of ground-truth to tracker-estimated point tracks.Source: A Dataset preprocessing · p. 14

B Iterative densification of point tracks

• 3000 queries across five iterations effectively eliminate sparse regions while keeping iterative point-track densification computationally feasible.Each iteration divides frames into patches with C = 20, randomly selects up to N/T empty patches, adds one query per selected patch, and reruns the tracker. Coverage improves over uniform sampling, particularly for small, rapidly moving objects; preprocessing runs on a Tesla T4 with 15 GB memory.Source: B Iterative densification of point… · pp. 14–15
• Sparse tracks caused by missing depth in static-scene datasets are retained as implicit augmentation for robustness to incomplete supervision.This applies to DL3DV [Ling et al. 2024] and TartanAir [Wang et al. 2020], rather than correcting their naturally sparse regions.Source: B Iterative densification of point… · p. 15

C Details of point-track augmentation.

• Visibility-based and spatial dropout remove reference-invisible tracks or tracks crossing a random masked frame region, respectively.Each strategy is applied with 7% probability; spatial dropout removes affected tracks across the entire sequence and supports sparse/dense conditioning. Visibility dropout requires inferring appearance and motion for content appearing only in generated frames.Source: C Details of point-track augmentation · p. 15
• Foreground overlays teach that missing tracks need not imply occlusion, countering hallucinated foreground content [Geng et al. 2025].Moving YouTube-VOS [Xu et al. 2018] masks are overlaid without associated tracks with 15% probability. Track counts are randomly subsampled between 256 and 16,384; more aggressive augmentation destabilizes training and degrades motion controllability.Source: C Details of point-track augmentation · p. 15

Spatially-aware point-track embedder

• The spatially-aware embedder uses spatial and frame-index sinusoidal encodings and max pooling to represent variable-length tracks spanning generated frames and reference images.Undefined coordinates, including those caused by occlusion or filtering, are excluded before pooling.Source: D Model architecture details · p. 15

Point-track adapter

• 4 × 16 × 16 spatiotemporal blocks map pixel-space track embeddings to individual patchified latent tokens through max pooling.The first frame and each reference image are encoded independently without temporal compression. Each reference frame is replicated four times to match the adapter’s temporal pooling.Source: D Model architecture details · pp. 15–16
• Relative within-block coordinates are concatenated with track embeddings and processed by an MLP before pooling, addressing positional information loss from naive max pooling.The reported ablation identifies relative positional encoding as crucial for precise motion controllability.Source: D Model architecture details · p. 15

Computational efficiency

• The adapter reduces computation versus pixel-space alternatives by operating on compressed latent tokens and point-track embeddings.Prior methods without dedicated adapters [Chu et al. 2025; Wang et al. 2025a] instead downscale coordinates by 16 with rounding, discarding fine-grained spatial information and degrading motion controllability.Source: D Model architecture details · p. 15

Baselines

• DiffusionAsShader [Gu et al. 2025a], ATI [Wang et al. 2025a], and Wan-Move [Chu et al. 2025] receive only first-frame-visible tracks.Modifying DiffusionAsShader’s input layers to accommodate tracks originating after the first frame did not improve performance. ATI is capped at 40 tracks because visual quality degrades with increasing track counts; Wan-Move follows ATI’s evaluation protocol.Source: E Experiment details · p. 16
• Go-With-The-Flow [Burgert et al. 2025b] receives dense optical flow, whereas Tora [Zhang et al. 2024] is evaluated only under Task-3.Go-With-The-Flow uses zero noise degradation to enforce ground-truth motion. Tora supports only sparse point-track conditioning, so evaluation randomly samples up to 10 tracks because performance deteriorates with more tracks.Source: E Experiment details · p. 16

Training time and resources

• The experimental setup combines method-specific baseline inputs with staged training of a 14B 480p model and its 720p continuation.The 480p model trains for seven days on one node with 8×A100 GPUs; the 720p model starts from that checkpoint and trains another seven days on one node with 8×H200 GPUs. Training uses Fully Sharded Data Parallel for memory efficiency and scalability.Source: E Experiment details · p. 16

F Quantitative results on TapVid3D-ADT

• Go-with-the-Track consistently outperforms all evaluated baselines on TapVid3D-ADT.The supplied passage reports this comparison from Table 5 without providing individual metric values.Source: F Quantitative results on TapVid3D-ADT · p. 16
• The performance margin over baselines is larger on TapVid3D-ADT than on DAVIS 2017.The authors interpret this larger margin as further evidence of the approach's effectiveness and robustness.Source: F Quantitative results on TapVid3D-ADT · p. 16
• The TapVid3D-ADT comparison supplements the main paper's DAVIS 2017 evaluation with a more challenging dataset.No passages were supplied for G Additional ablation study, so its findings cannot be summarized.Source: F Quantitative results on TapVid3D-ADT · p. 16

• Joint CFG on reference images and text prompts produces more stable and robust results than applying CFG only to text prompts.Source: §G.1 · p. 18
• cfg = 3.0 is adopted for all experiments as the preferred guidance trade-off.The evaluated guidance scales are 1.5, 3.0, 5.0, and 7.0.Source: §G.1 · p. 18
• cfg = 5.0 occasionally causes image saturation, whereas cfg = 1.5 slightly reduces video quality.Source: §G.1 · p. 18

G.2 Qualitative ablation analysis

• Training solely on real videos with noisy point-track labels catastrophically fails to generate the desired zoom-in, highlighting the importance of ground-truth annotations.The authors hypothesize that noisy-only training encourages models to treat tracks as unreliable and “correct” them, overfitting to specific patterns rather than faithfully following motion conditions.Source: §G.2 · pp. 18–19
• Random point-track embeddings fail to follow the supplied tracks, producing substantially different zoom-in behavior.The task generates a zoomed-in view of a TV from a reference image of a zoomed-out conference room.Source: §G.2 · p. 18
• Removing the point-track adapter or relative position injection loses fine-grained motion details, producing a tilted TV rather than precisely following the supplied tracks.The passage associates these ablations with “Random embeddings” and “Max pool only” in table 2, respectively.Source: §G.2 · p. 18

G.3 PCA analysis of point-track embeddings

• Spatially-aware point-track embeddings substantially improve performance over random embeddings.The passage reports this comparison in Table 2 but supplies no numerical values.Source: §G.3 · p. 19
• PCA visualizations show clear spatial correlations in the proposed embeddings, whereas random embeddings preserve no meaningful spatial structure.The passage identifies Figure 11 as the visualization supporting this distinction.Source: §G.3 · p. 19
• The analysis suggests that spatially correlated embeddings are crucial for effectively incorporating reference-anchored point-track conditioning into the video model.Source: §G.3 · p. 19

H Details of Qualitative Comparison

• Extremely sparse point-track inputs still produce videos that follow the supplied motion, despite the model not being explicitly trained for such sparse conditions.The authors note that this observation is consistent with [Geng et al. 2025].Source: H Details of Qualitative Comparison · p. 20, Figure 14 · p. 22
• 45 participants each evaluated 30 samples in an anonymous user study assessing motion adherence, subject identity preservation, and overall visual quality.Participants saw input tracks, ground-truth video, and randomly shuffled baseline and Go-with-the-Track outputs; the interface supported synchronized side-by-side playback.Source: H Details of Qualitative Comparison · p. 20, Figure 15 · p. 23

I Details of Applications

• Video restylization combines source-video point tracks with edited full-frame or cropped-object references to change appearance while preserving underlying motion.Tracks come from an off-the-shelf tracker [Ngo et al. 2024], and selected frames are edited with NanoBanana [Google DeepMind 2025]. Object references use SAM 3 [Carion et al. 2025] segmentation and cropping, with pixels outside the selected region grayed out.Source: I Details of Applications · p. 20, Figure 16 · p. 24
• Mesh-driven compositing generates stylized mesh-animated videos from stylized renders and point tracks obtained by projecting mesh vertices along predefined camera trajectories.Static and dynamic mesh scenes are rendered from arbitrary viewpoints in Blender, then stylized with NanoBanana [Google DeepMind 2025] to provide reference images.Source: I Details of Applications · p. 20, Figure 17 · p. 25
• Keypoint-driven compositing transfers a reference subject's appearance onto a human-centric source video while preserving its original motion.Facial and full-body detectors [Giebenhain et al. 2025; Yang et al. 2026] extract landmarks from the video and reference image; matching landmark indices form reference-anchored point tracks.Source: I Details of Applications · p. 20, Figure 18 · p. 26

H Details of Qualitative Comparison

• Go-with-the-Track outperforms baselines qualitatively in reconstruction structure and identity preservation, and in preserving source motion while matching stylized first-frame appearance.Reconstruction uses the source video's first frame and extracted point tracks; restylization instead supplies a stylized first frame with the source tracks.Source: H Details of Qualitative Comparison · p. 20, Figure 12 · p. 21, Figure 13 · p. 22

I Details of Applications

• Camera retargeting supports static and dynamic scenes through reprojected 3D points, custom camera trajectories, and four uniformly sampled reference frames.Static scenes use point clouds and camera poses from 𝜋3 [Wang et al. 2025c], with z-buffering to remove occlusions. Dynamic scenes use densified DELTA [Ngo et al. 2024] tracks and 𝜋3 camera estimates, discarding reprojections of points invisible in the original frame.Source: I Details of Applications · p. 21
• First- and last-frame albedo and shading estimates enable temporally stable sequence generation with less flicker than frame-by-frame estimation.The method propagates reference estimates from a re-implementation of, demonstrating generalization to these unseen modalities.Source: I Details of Applications · p. 21, Figure 21 · p. 29