PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking

• PointOdyssey targets tracking a world surface point from any pixel coordinate in any video frame for as long as possible.Source: §1 · p. 1
• Unlike datasets for short-range optical flow and coarse-grained long-range tracking, few works address fine-grained long-range tracking; prior synthetic training data uses random objects and motion [39] [19] [31] [44] [24] [15].The paper argues that such simplistic data limits learning long-range context and scene-level semantic awareness.Source: §1 · p. 1
• The dataset uses motions, scene layouts, and camera trajectories mined from real-world sources, with domain randomization across scene attributes to promote naturalism and diversity.Human and animal motion captures [37] [33] drive actors; real-video camera trajectories [34] and replicated indoor capture environments preserve scene-aware motion and interactions.Source: §1 · p. 2
• PointOdyssey includes 42 humanoid shapes, 7 animals, 1K+ textures, 1K+ objects, 20 scenes, and 50 environment maps, with fog and smoke adding partial occlusion.Source: §1 · p. 2
• The modified PIPs method widens its 8-frame temporal window and adds template updates, achieving higher tracking accuracy than all existing methods on PointOdyssey and real-world benchmarks [24].Source: §1 · p. 2

Motion Datasets

• Middlebury offered high-quality annotations but fewer than 100 frames, while Sintel added visual diversity; PointOdyssey is orders of magnitude larger and more realistic.Sintel contains 1,064 frames from a Blender-animated movie, including lighting variation, shadows, reflections, complex materials, and atmospheric effects.Source: §2 · pp. 2–3
• Unlike random-object datasets [16, 39, 24, 49, 23] and approximate KITTI labels, PointOdyssey targets thousands-of-frame tracking, human-scene interactions, and fine-grained ground truth.Only FlyingThings++ [24] and Kubric-MOVi-E [23] among those synthetic datasets provide multi-frame trajectories; KITTI estimates motion using LiDAR, egomotion, and fitted car models.Source: §2 · p. 3
• Training on PointOdyssey’s richer synthetic data improves performance on TAP-Vid, whose test set uses manually annotated tracks from Kinetics [29] and DAVIS [45].PointOdyssey’s test split also annotates points during occlusions, unlike TAP-Vid, which annotates only visible points.Source: §2 · p. 3
• Unlabelled-video methods [7] avoid the sim-to-real gap by training on real footage, while improved rendering may make synthetic supervision more viable [17].The cited works use auxiliary objectives to encourage tracking to emerge from unlabelled data.Source: §2 · p. 3

Motion Understanding

• Motion research has shifted from handcrafted-feature optimization to neural feedforward or iterative methods [27], alongside optical flow and multi-frame tracking.Structure-from-motion [42] and SLAM continue to use handcrafted approaches.Source: §2 · p. 3
• Replacing PIPs’ fixed 8-frame MLP-Mixer with a deep 1D convolutional network enables wider temporal context; trained on PointOdyssey, it outperforms PIPs and TAP-Net while retaining reasonable estimates during occlusions.The convolutional network applies fixed-length kernels across arbitrary temporal spans, and jointly tracks multiple timesteps.Source: §2 · p. 3

3.1. Long-Term Motion Data

• Humans and robots are animated by retargeting real-world SMPL-X motion to target rigs, while animal motion remains bound to its meshes; indoor scenes match mocap environments.The dataset uses 42 artist-designed human and robot assets and seven animals.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. 3D Environment Context

• Outdoor scenes combine character-object interactions and randomized forces for collision-rich motion, while 20 manually built indoor scenes recreate mocap environments for natural, collision-free motion.The indoor motions reflect the true affordances of their environments.Source: §3.2 · pp. 4–5

3.3. Camera Motion

• Outdoor cameras follow trajectories mined from YouTube videos with structure-from-motion, while indoor videos use cinematic paths and subject-mounted egocentric views.The synthetic egocentric views produce particularly challenging motion trajectories.Source: §3.3 · p. 5

3.4. Scene Randomization

• Scene diversity comes from randomized indoor surfaces and lighting, outdoor object textures, animal fur and HDR backgrounds, plus procedural fog and smoke.The materials include 80 indoor surface materials, 1,000 outdoor texture maps, 24 fur materials, and 50 HDR images.Source: §3.4 · p. 5

3.5. Annotation Generation

• The dataset exports 2D and 3D point trajectories, visibility, depth, normals, instance segmentation, and camera calibration to support tracking and other applications.Visibility is computed by comparing tracked-point depth with rendered depth at projected coordinates.Source: §3.5 · p. 5, Figure 5 · p. 5

3.6. Statistics

• 216K images at 540 × 960 and 30 FPS span 43 outdoor and 61 indoor scenes, with 166K, 24K, and 26K frames assigned to training, validation, and testing.Rendering used 2,600 GPU hours with Blender Cycles.Source: §3.6 · p. 5

4.1. Preliminaries (PIPs)

• PIPs tracks queried points in chained 8-frame windows using first-frame features and iterative local correlations, but chaining is drift-prone under long occlusions and computationally expensive.PIPs outputs a trajectory across each 8-frame segment; its feature-update operator cannot act as a template update because it cannot access the input frames.Source: §4.1 · p. 6

4.2. Expanding the temporal field of view (PIPs+)

• Replacing PIPs’ fixed-width MLP-Mixer with an 8-block temporal 1D Resnet yields a 35-timestep receptive field that grows further through iterative inference.Each residual block uses two kernel-size-3 convolution layers with instance normalization and ReLU.Source: §4.2 · p. 6
• PIPs+ improves long-range tracking accuracy and raises average inference speed from 4 FPS to 55 FPS on 720×1080 video using an Nvidia V100 GPU.The convolutional design supports training and testing on different sequence lengths, although similar sequence lengths remain important in practice.Source: §4.2 · p. 6

4.3. Extending to multiple templates (PIPs++)

• PIPs++ concatenates correlations from trajectory-sampled recent-appearance templates with the initial template, retaining it to help prevent forgetting.Recent templates can temporarily capture an occluder, which may be the appropriate entity to track during self-occlusion; the full model omits residual feature updates and visibility estimation.Source: Figure 6 · p. 6, §4.3 · p. 7

5.1. Experimental setup

• The comparison includes PIPs [24], TAPNet [15], PIPs+, PIPs++, RAFT with chained optical flow, and DINO [13] as a feature-matching method.Source: §5.1 · p. 7
• PIPs and TAP-Net are trained and tested on PointOdyssey using 4–8 A5000 GPUs; pretrained RAFT and DINO are evaluated, alongside tests on TAP-Vid-DAVIS [15] and CroHD.TAP-Vid-Kinetics [15] is omitted because it contains hard cuts, while the experiments focus on continuous video.Source: §5.1 · p. 7

5.2. Evaluation

• Evaluation reports TAP-Vid's [15] average position accuracy δavg, median trajectory error (MTE), and survival rate, measured as time until tracking failure relative to video length.δavg averages the percentage of tracks within thresholds {1, 2, 4, 8, 16} at normalized 256 × 256 resolution; failure is an L2 error above 50 pixels.Source: §5.2 · p. 7
• PointOdyssey appears harder than TAP-Vid-DAVIS [15] and CroHD, yet method rankings are consistent across datasets; PIPs models degrade more slowly, though all methods lose tracks over time.The authors say consistent rankings suggest progress on PointOdyssey correlates with progress on real-world videos.Source: §5.2 · p. 7
• PIPs+ and PIPs++ achieve the best results among tested methods, supporting wide temporal awareness; their narrow gap suggests the multi-template strategy helps only modestly on average.The authors attribute the improvement to wider temporal awareness and describe the multi-template strategy as helpful on average.Source: §5.2 · p. 7, Table 2 · p. 8
• Qualitatively, the proposed method produces more stable trajectories than PIPs [24] and other baselines, but all methods struggle near boundaries, especially on thin objects.Source: §5.2 · pp. 7–8, Figure 7 · p. 8

6. Limitations

• PointOdyssey lacks large outdoor scenes with long-distance camera travel, a frequent scenario in driving data [17], leaving long-range agent-scene and agent-agent interactions unexplored.The authors suggest recent generative models [46] could help lift limits on human and animal motion profiles imposed by the base datasets.Source: §6 · p. 8
• The dataset’s human and animal motion profiles are limited by their base datasets, a constraint the authors suggest recent generative models could help lift [46].This limitation concerns the available motion profiles, not the dataset’s other randomized scene and appearance factors.Source: §6 · p. 8
• PointOdyssey’s potential applications beyond point tracking remain unexplored, including 3D scene flow, dynamic-scene novel view synthesis, human and animal pose estimation, and ego-centric vision.Dynamic-scene novel view synthesis would be especially challenging with PointOdyssey’s atmospheric effects.Source: §6 · p. 8
• PIPs++ remains a low-level tracker relying entirely on appearance-matching cues and a temporal prior, leaving scene-level and semantic cues for tracking as open challenges.The authors expect PointOdyssey’s training data to be especially valuable for exploring those cues.Source: §6 · p. 8

7. Conclusion

• PointOdyssey is a diverse, naturalistic synthetic dataset and generator for long-term point tracking and a resource for training general-purpose fine-grained trackers.Source: §7 · p. 8
• PIPs++ leverages long-term temporal context and outperforms the state of the art, demonstrating PointOdyssey’s usefulness.Source: §7 · p. 8
• PointOdyssey could support trackers using scene-level and semantic cues, an avenue the authors have not explored, as well as work in 3D and 4D scene analysis and higher-level video understanding.Source: §7 · p. 8

A. Implementation Details

• PIPs+ and PIPs++ are trained with the weighted L1 distance between estimated and ground-truth trajectories across iterative updates, following Harley et al. [24].The full loss averages the per-point loss across all query points.Source: A. Implementation Details · p. 11
• γ is set to 0.8 in the experiments, and the full training loss averages per-point loss across the N query points.Source: A. Implementation Details · p. 11
• The trajectory loss is applied during occlusions, requiring the model to estimate tracks through visibility gaps.Source: A. Implementation Details · p. 11
• Visibility-classification and score losses used by PIPs [24] can be omitted without harming tracking performance, so training uses only the L loss in Eq. (4).Source: A. Implementation Details · p. 11
• PIPs++ tracks beyond its GPU memory limit with non-overlapping 36-timestep sliding windows, a size found best in a small grid search.The authors suggest 36 may work well because it matches the training sequence length.Source: A. Implementation Details · p. 11
• At 720 × 1080 on an Nvidia V100, PIPs++ reaches 55.2 FPS with windowing, versus 3.6 FPS for PIPs chaining 8-frame predictions with visibility-based reinitialization.PIPs++ uses no visibility check; PIPs reinitializes per point at the last timestep whose estimated visibility exceeds a threshold.Source: A. Implementation Details · p. 11

• Figures 9 and 10 show animated dataset characters retargeted with motion data to generate varied interacting sequences.Source: B. Additional Visualizations of… · p. 11
• Figure 11 shows samples of the rebuilt motion-capture environments.Source: B. Additional Visualizations of… · p. 11
• Figure 12 shows sample pixel trajectories from the dataset.Source: B. Additional Visualizations of… · p. 11

C. Additional Results

• On Tap-Vid-DAVIS [15], PIPs++ trajectories are on average less sensitive to occlusions than PIPs [24], which becomes unreliable when occlusions occur.The figure suggests PIPs++ may withstand occlusions more often because of its wider temporal receptive field.Source: C. Additional Results · p. 11, Figure 13 · p. 15