Point Prompting: Counterfactual Tracking with Video Diffusion Models

1 INTRODUCTION

• Pretrained video diffusion models can produce zero-shot point tracks by propagating a distinctively colored query-point marker through regenerated frames.The method uses SDEdit and estimates the marker’s position with basic image processing.Source: §1 · p. 1
• Unedited-frame negative prompting guides generation to retain markers that strong generative priors might otherwise ignore as unnatural.The original initial frame is used as a negative prompt alongside the manipulated frame.Source: §1 · p. 1
• Earlier counterfactual motion-estimation methods train a purpose-built future-prediction model, whereas this approach uses off-the-shelf video diffusion models.The paper distinguishes its visual trajectory prompting from methods that extract model features to match image pairs.Source: §1 · p. 2
• The paper reports that video diffusion models track through occlusions, outperform prior zero-shot methods, and can yield trajectories distilled into a fast tracker.It also reports iterative refinement using inpainting and performance often competitive with specialized self-supervised models.Source: §1 · p. 2

2 RELATED WORK

• Point trackers improve long-range accuracy, but motion-estimation methods often rely on synthetic data that may limit real-world generalization.The cited work also describes limitations inherited by self-supervised optical-flow methods.Source: §2 · p. 2
• Image-diffusion features such as DIFT support image matching, while a concurrent video-feature method does not handle occlusion.The paper contrasts feature extraction with prompting a video diffusion model, which it describes as architecture-agnostic.Source: §2 · p. 3
• Earlier counterfactual motion methods perturb inputs to future predictors; this paper instead uses an off-the-shelf video diffusion model for long-range point tracking.Stojanov et al. (2025) learn RGB perturbations and use a jointly trained sparse optical-flow module, unlike this paper’s no-additional-training approach.Source: §2 · p. 3
• Image diffusion models capture semantic correspondences, but the cited work notes they lack the temporal reasoning needed for motion-centric tasks.Video diffusion models address temporal consistency, though many still prioritize appearance over motion.Source: §2 · p. 3
• The paper introduces a colored dot at the pixel to track as a visual prompt for point tracking with video diffusion models.It presents this as the first use of image prompting for point tracking in video diffusion models, to the authors’ knowledge.Source: §2 · p. 3
• SDEdit uses noise perturbation and iterative denoising for training-free guided synthesis, while other approaches offer spatial control through inpainting or fine-tuning.Related video methods extend structured editing through architectural design and hierarchical sampling.Source: §2 · pp. 3–4

3.1 PRELIMINARIES: VIDEO DIFFUSION MODELS

• The method uses video diffusion’s reverse denoising process to regenerate existing videos from an intermediate noise level, and can constrain inpainting updates to masked regions.Regeneration retains coarse video structure while allowing fine details to change; inpainting limits changes to the specified mask.Source: §3.1 · pp. 4–5

3.2 POINT PROMPTING FOR COUNTERFACTUAL TRACKING

• A red dot marks the query point, and intermediate-noise regeneration conditioned on the edited first frame propagates it through subsequent frames.The trajectory is extracted from the generated frames using color-based tracking.Source: §3.2 · p. 5
• Conditioning on the marked frame while steering away from the unedited frame helps keep the marker from disappearing under the model’s natural-video priors.The method contrasts noise estimates for the original and edited first-frame conditioning.Source: §3.2 · pp. 5–6
• A local color-based tracker follows the marker frame by frame, expanding its search radius when the marker is absent and resetting it when the marker reappears.The tracker selects nearby red pixels and averages their positions; the adaptive search handles temporary occlusions and large displacements.Source: §3.2 · p. 6

3.3 EXTENSIONS

• Color rebalancing suppresses natural red hues, while mask-constrained refinement revises coarse tracks near the tracked point to address false detections and regeneration misalignment.The refinement uses inpainting; the figure describes it as correcting temporal artifacts such as object shifts.Source: Figure 3 · p. 5, §3.3 · p. 6

4.1 VIDEO MODELS

• The study considers Wan2.1, Wan2.2, and CogVideoX image-conditioned video diffusion models, using Wan2.1’s 14B variant unless otherwise noted.Sampling uses 50 denoising steps at noise strength 0.5; generating a 50-frame video takes about 30 minutes with Wan2.1-14B and 20 minutes with CogVideoX.Source: §4.1 · p. 7

4.2 TAP-VID BENCHMARK

• Evaluation uses the TAP-Vid DAVIS split of 30 videos and Kinetics split of 30 sampled 250-frame videos, with one query point per video.The query is marked in the first frame and the trajectory is extracted from the generated video; these natural videos match the models’ training distribution.Source: §4.2 · p. 7, §4 · p. 7
• The reported TAP-Vid measures are average fraction of visible points within distance thresholds, Occlusion Accuracy, and Average Jaccard.Occlusion Accuracy measures visibility prediction accuracy; Average Jaccard measures overlap between predicted and ground-truth visible points across thresholds.Source: §4.2 · p. 7

Quantitative Results

• On TAP-Vid DAVIS, the method achieves an AJ score of 42.21, outperforming zero-shot baselines and GMRW while exceeding zero-shot and self-supervised methods in occlusion accuracy.The benchmark table reports that it matches self-supervised methods in AJ on DAVIS-First and is competitive with supervised trackers.Source: §5 · p. 7, Table 1 · p. 8
• On TAP-Vid Kubric’s 30-video subset, the method outperforms zero-shot baselines, although performance is comparatively lower on synthetic Kubric than on real-video benchmarks.The paper attributes the lower Kubric performance as likely related to its synthetic nature and video models primarily being trained on real videos.Source: §5 · p. 7

Different Video Models

• Wan2.1 and Wan2.2 achieve the strongest results among tested video models, and the method tracks points with all four tested backbones.Wan2.1-14B outperforms the 1.3B variant; the authors suggest higher video-generation quality may improve tracking.Source: §5 · p. 8, Table 2 · p. 8

Generation Resolution

• On high-resolution DAVIS frames, Wan2.1 reaches an AJ score of 48.6, and upsampling benchmark inputs also improves tracking.The authors connect higher-resolution inputs with improved video-generation quality and better alignment with the model’s training distribution.Source: §5 · p. 8, Table 3 · p. 8

Point Propagation Ablations

• Removing inpainting refinement reduces positional accuracy, while removing counterfactual enhancement guidance disrupts point propagation.The reported positional-accuracy reduction is attributed to spatial shifts during denoising.Source: §5 · p. 8

Ablations

• A noise strength of 0.5 and query-point radius of 2 pixels yield the best reported results; disabling color rebalancing also degrades performance.Without color rebalancing, red background tones can trigger false positives, particularly when the query point is occluded.Source: Figure 4 · p. 8, §5 · p. 9

Distilling the Generator into a Tracker

• A CoTracker trained from scratch on trajectories from 1,000 unlabeled videos performs close to the Wan2.1 teacher while running orders of magnitude faster at inference.The teacher trajectories are generated by the marker-propagation method.Source: §5 · p. 9, Table 5 · p. 9

Qualitative Results

• Generated red markers remain consistent through long occlusions, and extracted tracks maintain accuracy over long temporal ranges.The figures show marker recovery after occlusion and trajectories displayed across frames.Source: §5 · p. 9, Figure 5 · p. 9

Efficiency

• Generating a separate video for every tracked point limits efficiency; the authors did not optimize the approach because their goal was to demonstrate tracking by video generators.They identify distillation, more efficient generation, and tracking multiple points at once as possible ways to address the cost.Source: §6 · pp. 9–10

Generation failures

• Video generators sometimes fail to interpret the red dot as attached to the object surface, especially in likely out-of-distribution computer-generated videos.Illustrated failures include stationary markers, symmetry confusion, vanished markers, and shifts near image boundaries.Source: §6 · p. 10, Figure 7 · p. 10

• Careful prompting lets a video diffusion model mark a point’s location as it moves through a scene over time.Source: §7 · p. 10
• The resulting simple point tracker obtains effective tracking results and outperforms previous zero-shot approaches.Source: §7 · p. 10
• The work points toward adapting large pretrained video diffusion models to new tasks and using video generative models as a source of tracking pretraining.Source: §7 · p. 10

• Overall performance on Kubric is comparatively lower, likely due to the dataset’s synthetic nature.Simple textures and non-natural, erratic motion challenge faithful video regeneration, affecting point propagation and tracking accuracy.Source: A QUANTITATIVE RESULTS ON TAP-VID · p. 18
• On TAP-Vid Kubric, the Wan2.1-14B method outperforms zero-shot baselines.Table 6 reports results on a subset of 30 videos, consistent with the main paper’s Table 1.Source: Table 6 · p. 18, A QUANTITATIVE RESULTS ON TAP-VID · p. 18
• Figure 8 contrasts successful point propagation with a failure on a very low-texture surface.Source: Figure 8 · p. 18

A.1 ABLATIONS

• Local search, an expanding search radius under occlusion, and averaging nearby red-pixel coordinates progressively improve tracking, with position refinement performing best overall.Using LAB instead of HSV causes a slight accuracy drop in this setup.Source: §A.1 · p. 18, Table 7 · p. 19
• On a subset of TAP-Vid DAVIS videos, λ = 8 performs best, and red markers provide a slight performance gain over other tested colors.The λ ablation varies the edited-image noise estimate’s weight in counterfactual enhancement guidance.Source: §A.1 · p. 19, Table 8 · p. 19, Table 9 · p. 19

A.2 V-BENCH SCORES

• Tracking and generation quality improve progressively from CogVideoX to Wan2.1-1.3B and then Wan2.1-14B.The authors attribute the pattern to higher video-generation quality, reflected in VBench scores; Table 10 presents VBench results for the evaluated models.Source: §A.2 · p. 19, Table 10 · p. 19

B.1 VIDEO PREPROCESSING

• The preprocessing removes red pixels from the original frame to avoid false positives, using HSV thresholds and saturation clipping to desaturate detected pixels.Source: §B.1 · p. 19
• Wan and CogVideoX inputs are padded by repeating the last frame to reach 4T + 1 frames, then the added frames are removed after regeneration.Source: §B.1 · p. 20
• Upscaling input video improves point propagation by reducing generation artifacts and drift; the pipeline upscales TAP-Vid’s 256 × 256 videos to 1024 × 1024, then downscales to 480 × 832.Outputs are resized back to 256 × 256 for tracking evaluation.Source: §B.1 · p. 20

B.2 POINT PROPAGATION

• The diffusion timestep is calculated from denoising strength γ = 0.5 and the total number of diffusion steps T.The method uses SDEdit to control the signal-to-noise ratio.Source: §B.2 · p. 20
• Counterfactual enhancement guidance computes the noise estimate using Eq. 5, with classifier-free guidance weight λ set to 8.The guidance is intended to enhance the effect of the edited image containing the colored dot.Source: §B.2 · p. 20

B.3.1 MARKER DETECTION

• Marker detection thresholds pixels in HSV space, using hue (H-10, H+5), saturation (150, 255), and value (150, 255).Pixels within all three intervals are treated as potential marker pixels.Source: §B.3.1 · p. 20

B.3.2 LOCAL SEARCH AND OCCLUSION HANDLING

• Local search uses a circular region of default radius 90 around the previous detection, expanding after occlusion up to radius 150 before resetting.The expanded search accommodates greater positional uncertainty after an occlusion.Source: §B.3.2 · p. 20

B.3.3 CENTER ESTIMATION

• Center estimation averages red-pixel detections within 20 pixels of the candidate nearest the previous detection.This produces the predicted tracking point for the current frame.Source: §B.3.3 · p. 20