PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Video Motion Editing with 3D Point Tracks

Yao-Chih Lee, Zhoutong Zhang, Jiahui Huang, Jui-Hsien Wang, Joon-Young Lee, Jia-Bin Huang, Eli Shechtman, Zhengqi Li

arXiv:2512.02015v1cs.CV

TL;DR

Precisely editing both camera and fine-grained object motion remains difficult because existing methods lack either joint control or full-scene context. Edit-by-Track addresses this with a track-conditioned V2V model using source videos and paired 3D tracks, and supports diverse motion edits with reported improvements over prior approaches.

  • Problem

    Precisely editing camera and complex object motion jointly remains challenging, while existing approaches often lack full-scene context or fine-grained object-motion control.

  • Method

    Edit-by-Track conditions a V2V video-generation model on the full source video and paired source-target 3D point tracks, trained through synthetic and real-data fine-tuning.

  • Results

    The method substantially outperforms state-of-the-art approaches in editing camera motion and object dynamics while preserving input-context coherence.

  • Takeaways & Limitations

    The framework supports joint camera and object manipulation, depth-aware motion editing, occlusion handling, motion transfer, and shape deformation.

  • Takeaways & Limitations

    The method may struggle with densely clustered tracks, especially for small objects, and may fail to synthesize complex physical phenomena from edited motions.

Abstract

from arXiv · show

Camera and object motions are central to a video's narrative. However, precisely editing these captured motions remains a significant challenge, especially under complex object movements. Current motion-controlled image-to-video (I2V) approaches often lack full-scene context for consistent video editing, while video-to-video (V2V) methods provide viewpoint changes or basic object translation, but offer limited control over fine-grained object motion. We present a track-conditioned V2V framework that enables joint editing of camera and object motion. We achieve this by conditioning a video generation model on a source video and paired 3D point tracks representing source and target motions. These 3D tracks establish sparse correspondences that transfer rich context from the source video to new motions while preserving spatiotemporal coherence. Crucially, compared to 2D tracks, 3D tracks provide explicit depth cues, allowing the model to resolve depth order and handle occlusions for precise motion editing. Trained in two stages on synthetic and real data, our model supports diverse motion edits, including joint camera/object manipulation, motion transfer, and non-rigid deformation, unlocking new creative potential in video editing.

1. Introduction

Edit-by-Track is a unified V2V framework for jointly editing camera and object motion while preserving the full input-video context. It uses paired 3D tracks, a 3D-aware conditioner, and two-stage synthetic-to-real training to support precise motion control.

  • Existing methods typically edit either object motion without camera control or camera motion while preserving object motion, leaving joint editing challenging.
  • Edit-by-Track unifies camera and object motion editing in video through point trajectories as a general motion representation.
  • The V2V model conditions on the full source video and paired source-target 3D tracks, preserving scene context while enabling precise motion editing.Track pairs establish sparse correspondences between source and target motions.
  • A 3D-track conditioner uses cross-attention for adaptive sampling and splatting, projecting visual context into target-frame space with 3D awareness and noise robustness.
  • Two-stage fine-tuning bootstraps control on synthetic animated-human pairs with ground-truth tracks, then adapts the model using diverse real-world video pairs.The strategy addresses the difficulty of obtaining real paired videos with 3D-track annotations.
  • The system supports versatile motion edits, including joint camera and object manipulation, depth changes, occlusion handling, motion transfer, and shape deformation.The paper reports substantial improvements over state-of-the-art methods while preserving input-context coherence.

2. Related Work

Prior video-editing approaches separately address appearance, object motion, camera viewpoints, or simple shape changes, often lacking full-scene context and joint fine-grained control. Edit-by-Track instead combines full-video conditioning with 3D track-based motion control for joint camera and object editing.

  • Figure 2 contrasts methods for joint camera and object edits: prior camera-controlled V2V misses secondary effects, while track-conditioned I2V loses scene context from using only the first frame.Edit-by-Track preserves input context and maintains coherent causal effects in the illustrated example.
  • Point tracks can represent both camera and object motion, with background movements corresponding to camera motion and foreground movements to object motion.
  • Track-conditioned I2V methods are poorly suited to video editing because they omit subsequent input frames and lose the dynamic scene’s full context.The proposed V2V framework conditions on the entire input video instead.
  • Edit-by-Track framework: The Edit-by-Track framework estimates camera poses and 3D tracks, lets users edit them, and feeds source-target motion pairs with source-video tokens into a DiT denoising pipeline.
  • Camera-controlled video-to-video synthesis: Camera-controlled V2V and reconstruction methods primarily target viewpoint changes; reconstruction-based methods additionally lack generative priors for hallucinating unseen regions.
  • Video editing: Existing video-editing studies mainly alter appearance or composition while preserving original motion, whereas motion and shape methods address only subsets of the broader problem.
  • Video pose, depth, and 3D track estimation: 3D pose and track estimation enables controllable editing of camera and 3D object motion, extending 2D tracking into a depth-aware representation.

3. Method

Edit-by-Track generates target videos from source videos and edited camera/object motions represented by paired 3D point tracks. Its conditioner transfers source visual context into target spaces with 3D-aware correspondences, supporting joint and diverse motion edits.

  • Motion-controlled V2V framework: The framework takes a source video plus source and target camera parameters and 3D point tracks to synthesize a target video reflecting user-specified motion edits.Source camera and point tracks are estimated first, then edited to define target motion before V2V generation.
  • Motion-controlled V2V framework: The V2V model concatenates encoded source-video tokens with noisy target-video tokens, preserving source-scene context during target generation.The resulting denoised latent is decoded into the target RGB video.
  • 3D track conditioning: Cross-attention samples per-track context from source tokens, aggregates temporal information, and splats it into source and target frame spaces to establish correspondence.Adaptive sampling and splatting retrieve values from corresponding 3D coordinates, improving robustness to noisy point tracks.
  • 3D track conditioning: Projected 3D tracks combine screen coordinates with normalized disparity and are converted into aligned source-target track tokens for 3D-aware conditioning.The tracks are projected using the respective camera parameters and temporally downsampled before conditioning.
  • Inference limitations: Background-track jitter can distort some stationary-viewpoint edits, so background points may optionally be fixed as static using segmentation labels.This constraint specifically affects certain viewpoint-editing cases with estimated background tracks.
  • Applications: The method supports independent camera or object-motion edits, SMPL-X human-motion transfer, non-rigid shape deformation, object removal, and object duplication.These applications use edited 3D tracks, including transformations of selected point groups and off-screen or repeated tracks.

4. Experiments

Experiments evaluate joint camera/object motion editing on DyCheck and in-the-wild videos, alongside visual comparisons and ablations. Edit-by-Track outperforms prior approaches while preserving scene context, and its 3D conditioning and two-stage training are validated.

  • 4.1. Evaluation on the DyCheck iPhone Dataset: Edit-by-Track outperforms prior approaches on all metrics for joint camera and viewpoint changes while preserving original scene context.Table 1 evaluates the joint motion manipulation task on DyCheck; competing methods suffer from hallucinations, distortions, or inadequate context.
  • 4.2. Quantitative Evaluation on In-the-Wild Videos: The 1.3B model achieves superior overall results and best track control on 100 MiraData videos, despite ATI using a larger 14B model.FVD measures visual quality, while EPE measures the L2 distance between generated and ground-truth estimated 2D tracks.
  • 4.3. Comparisons on Video Editing: On 10 real-world examples, I2V methods lose visual context, while GEN3C leaves shadows at edited objects’ original locations.The comparison covers camera and/or object motion edits, with the first frame left unedited for I2V baselines but not required by Edit-by-Track.
  • 4.4. Ablation Study: Adaptive cross-attention sampling and splatting are less sensitive to noisy tracks than fixed Gaussian sampling, while 3D depth cues improve large-viewpoint control.Injecting z embeddings into sampled tokens further reduces track control error (EPE).
  • 4.4. Ablation Study: Two-stage training outperforms single-stage alternatives: synthetic data establishes motion control, while real-data fine-tuning improves generalization.Synthetic-only training generalizes weakly, whereas real-only training produces poor track control.

5. Discussion and Conclusion

The framework jointly manipulates 3D viewpoints, object motion, and shape deformation, but remains limited by clustered tracks and insufficient physical grounding. The authors point to physically grounded generative models and data scaling as potential remedies.

  • 5. Discussion and Conclusion: Edit-by-Track enables joint manipulation of 3D viewpoints, object motion, and shape deformation.These tasks are described as challenging for existing methods.
  • 5. Discussion and Conclusion: Dense, clustered point tracks can hinder visual-context extraction and motion control, especially for small objects.This is identified as a limitation of the approach.
  • 5. Discussion and Conclusion: The method may fail to synthesize complex physical phenomena caused by edited motions, reflecting limited physical grounding in current generative priors.The authors reference failure cases in the supplementary material.

A.1. Model and Training Details

The model is fine-tuned with LoRA and a newly initialized 3D track conditioner, using synthetic paired clips to bootstrap motion control. Its conditioner samples, aggregates, and splats track-conditioned visual tokens across frames.

  • A.1. Model and Training Details: LoRA fine-tunes the pretrained DiT blocks, while the 3D track conditioner is initialized from scratch.LoRA is applied to MLPs in attention modules, including query, key, value, and feed-forward projection layers.
  • A.1. Model and Training Details: The conditioner uses cross-attention for sampling, temporal self-attention within each track, and shared-weight cross-attention for splatting.Track-coordinate positional embeddings are supplied to the sampling and splatting cross-attention modules.
  • A.1. Model and Training Details: Training takes 60 hours on 16 A100-80GB GPUs: 4,000 synthetic-data iterations followed by 8,000 real-data iterations.The conditioner has 45M parameters and the DiT LoRA adapters have 87M parameters.
  • A.1. Model and Training Details: Synthetic training pairs share objects and backgrounds but vary object actions and camera movements, isolating motion changes from appearance.The dataset uses textured dome-shaped backgrounds and randomly selected human characters.
  • A.1. Model and Training Details: The synthetic dataset contains 500 base scenes and 2,000 video clips, with training formed by randomly pairing clips from each scene.Each base scene produces four clips with different camera-motion and human-action combinations.

A.3. Real Data Curation

Stage 2 uses curated real monocular videos with estimated camera poses and 3D tracks, supplemented by object-removal pairs. The final mixture emphasizes dynamic monocular videos.

  • A.3. Real Data Curation: The primary real-data component contains 24K filtered monocular videos, excluding scenes with dynamic clutter such as crowds, birds, and fast traffic.Camera pose and 3D tracks are estimated during preprocessing.
  • A.3. Real Data Curation: The training mixture uses 85% dynamic monocular videos, 10% DL3DV, and 5% Gen-Omnimatte pairs.The 46 Gen-Omnimatte real-video pairs improve object removal while preserving the remaining scene.

A.4. Track Perturbation and Data Augmentation

The model augments noisy real-video tracks and clips during training to improve robustness and capture detailed foreground motion. It also supports object removal and efficient track-based editing at inference.

  • Track perturbation: Source 3D tracks are augmented because monocular depth errors can propagate into target tracks and create epipolar-line misalignments.Additional random homography perturbations address cases where epipolar jitter is insufficient under small viewpoint changes.
  • Video augmentation: Randomly masking up to 50% of source frames discourages reliance on only the first and last frames and promotes use of full-video context.
  • Object removal: An existence label set to 0 for target tracks specifies object removal, while non-removal tracks use label 1.Inference combines the zero label with moving the object tracks off-screen.
  • Occlusion handling: The model processes all tracks regardless of visibility, using 3D motion to reason about depth order and occlusion instead of visibility labels.
  • Track construction: Foreground-biased sampling densely tracks dynamic objects, with masks generated automatically in training and supplied interactively at test time.Foreground masks associate tracks with objects and capture detailed motions such as arms and legs.
  • Editing workflow: Basic track and viewpoint edits typically take less than 1 second for an 81-frame video, while rendering a preview takes about 30–60 seconds.The full model does not consume the preview video; inference uses a random sample of 1,000 tracks and a fixed seed of 0.

B.1. Robustness to Sparse Point Tracks

The final model remains effective with moderately sparse point-track inputs but fails when correspondences become extremely sparse. It is trained with 500–1000 tracks and evaluated using EPE on 100 in-the-wild videos.

  • Robustness to sparse point tracks: The model achieves reasonable performance with approximately 256 tracks but fails with extremely sparse inputs such as N = 32 because too few correspondences remain.Training uses a random number of tracks between 500 and 1000, and evaluation measures End-Point Error on 100 MiraData videos.

B.2. Robustness to Noisy Point Tracks

Track perturbation training improves robustness to noisy inputs, while text prompts help generate unseen regions and random seeds mainly affect newly revealed areas. Remaining limitations concern small, rapidly moving objects and complex physical effects.

  • Robustness to noisy point tracks: Approximately 4 pixels of Gaussian track noise increases error by only 1.26 pixels after perturbation training, whereas the ablated model degrades faster.Both models are evaluated with EPE on 100 MiraData videos.
  • Effects of text prompts: Text prompts provide supplementary context for generating unseen regions revealed by novel viewpoints.
  • Effects of random seeds: Random seeds introduce variations primarily in areas newly revealed by the target motion, so other evaluations use fixed seed 0.
  • Limitations: Small objects undergoing large motion changes can distort when their tracks are densely clustered and noisy.A cited example is a 270° front-flip.
  • Limitations: The model may fail to synthesize complex motion-dependent physical phenomena such as coffee and milk mixing.Simpler secondary effects, including water splashes and shadows, can remain plausible.

C.1. Evaluation on DyCheck

DyCheck evaluates joint, camera-only, and object-only motion editing under standardized video conditions. The method performs comparably to camera-controlled V2V methods, improves object-motion scores through input-video context, and receives higher human preference than representative baselines.

  • Evaluation scenarios: The joint-motion evaluation uses non-overlapping clips from moving-camera videos, while separate scenarios isolate camera or object motion.DyCheck supplies synchronized multi-view videos with depth and camera-pose annotations.
  • Evaluation protocol: Evaluation uses landscape-cropped videos at 672 × 384, with temporal subsampling to accommodate baselines producing fewer than 81 frames.
  • Camera motion only: The method performs comparably to dedicated camera-controlled V2V methods on the camera-motion-only DyCheck task.These baselines rely on dense depth warping, and the comparison is reported in Table 6.
  • Object motion only: On object-motion-only editing, the method achieves overall better scores than track-conditioned I2V baselines despite not using their privileged ground-truth first frame.The authors attribute this to effective use of context from the input video.
  • Human evaluation: Human evaluation with 42 subjects over 10 real-world cases shows higher preference for the method than representative track-conditioned I2V and inpainting-based V2V baselines.Subjects judged desired-motion alignment, input-context preservation, and perceived visual quality.
Loading 2512.02015v1…