PaperlayerView exact revision on arXiv

Source-linked AI summary

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking

Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, Leonidas J. Guibas

arXiv:2307.15055v1cs.CV

TL;DR

Fine-grained long-term tracking lacks large-scale annotated data that captures the naturalistic motion and complexity of real videos. PointOdyssey provides such synthetic data and extends PIPs to use wider temporal context, with the resulting tracker outperforming state-of-the-art methods on PointOdyssey and real-world benchmarks. The paper also identifies missing large outdoor camera-motion scenarios and unexplored scene-level and semantic cues as scope boundaries.

  • Problem

    Fine-grained long-range tracking needs large-scale training data that captures naturalistic motion and scene complexity beyond short-video optical-flow settings.

  • Method

    PointOdyssey combines motion-capture-driven actors, realistic scene and camera generation, broad domain randomization, and PIPs modifications with wider temporal context and template updates.

  • Results

    PIPs++ trained on PointOdyssey outperforms state-of-the-art tracking methods on PointOdyssey and real-world benchmarks.

  • Takeaways & Limitations

    PointOdyssey provides a diverse, naturalistic resource for training general-purpose fine-grained trackers and supports tracking with long-term temporal context.

  • Takeaways & Limitations

    PointOdyssey lacks large outdoor scenes involving cameras traveling long distances, and its human and animal motion profiles remain limited by the base motion datasets.

Abstract

from arXiv · show

We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks. Our data and code are publicly available at: https://pointodyssey.com

1. Introduction

PointOdyssey targets fine-grained long-range tracking with synthetic videos designed to preserve real-world complexity, diversity, and naturalism. It combines realistic motion and scene generation with widened temporal modeling for improved tracking.

  • PointOdyssey models tracking as a long-range problem involving camera motion, object motion, deformation, and interactions rather than only optical flow.
  • PointOdyssey is a large-scale synthetic dataset for training and evaluating long-term fine-grained tracking.
  • Its videos include trajectories reflecting object-scene interactions, while synthetic rendering provides pixel-perfect annotations.
  • The dataset uses real-world motion captures, scene layouts, and camera trajectories, alongside broad randomization of scene attributes.Randomized attributes include environment maps, lighting, bodies, camera trajectories, and materials.
  • Widening PIPs beyond its 8-frame temporal window and adding template updates improves tracking accuracy on PointOdyssey and real-world benchmarks.

2. Related Work

Prior motion datasets trade off realism, annotation precision, scale, or temporal span, while recent work increasingly targets multi-frame point correspondences. PointOdyssey addresses these limits with richer synthetic data and a tracker that uses wider temporal context.

  • Earlier datasets provide useful annotations or visual diversity but remain small, short, or limited in realism and temporal extent.Middlebury has fewer than 100 frames, while Sintel contains 1064 extracted frames and PointOdyssey is described as substantially larger and longer.
  • Real-world datasets such as KITTI require approximations for motion annotation, whereas synthetic generation can provide perfect fine-grained ground truth.
  • Synthetic motion datasets offer diversity but often use random objects, motions, and backgrounds that produce unrealistic data and mostly short-range supervision.
  • TAP-Vid supports synthetic training and real-world testing, and the paper reports improved TAP-Vid performance after training on richer synthetic data.
  • The proposed PIPs extension replaces the hard 8-frame constraint with a deep 1D convolutional network that accepts wider temporal spans.The paper reports that the resulting model, trained from scratch in PointOdyssey, outperforms PIPs and TAP-Net.

3. PointOdyssey Dataset

PointOdyssey generates long, naturalistic synthetic tracking videos by combining motion-captured characters and environments with randomized scene content, camera motion, and atmospheric effects. It exports dense point-tracking and scene annotations across 216K images split into training, validation, and test sets.

  • Long-Term Motion Data: PointOdyssey renders complex long-range motion using humanoids, robots, and animals driven by motion-capture data in randomized scenes.The pipeline includes deformable characters, physical interactions, materials, lighting, and atmospheric effects.
  • Long-Term Motion Data: Indoor scenes reproduce motion-capture environments, yielding collision-free, naturalistic motion that reflects the environments’ true affordances.Twenty indoor scenes are manually built to match captured layouts and furniture, then populated with motion-captured characters.
  • Camera Trajectories: Outdoor cameras use structure-from-motion trajectories mined from YouTube, while indoor cameras use cinematic and egocentric trajectories.The resulting egocentric views produce particularly challenging motion trajectories.
  • Annotation Generation: The pipeline exports 2D and 3D point trajectories, visibility, depth, normals, instance segmentation, and camera calibration annotations.Visibility is computed by comparing tracked-point depth with rendered depth at projected coordinates.
  • Dataset Statistics: The dataset contains 43 outdoor scenes and 61 indoor scenes, totaling 216K 540 × 960 images at 30 FPS.The split is 166K training frames, 24K validation frames, and 26K test frames.

4. Long-Term Tracking with PIPs++

PIPs++ extends PIPs for long-term fine-grained tracking by widening temporal context and updating appearance templates. Its PIPs+ variant uses temporal convolutions to improve long-range accuracy and substantially speed inference.

  • 4.1. Preliminaries (PIPs): PIPs processes 8-frame RGB inputs and iteratively updates a target trajectory from local feature correlations, positions, and appearance features.The model initializes framewise positions and features, then produces additive updates and visibility estimates.
  • 4.1. Preliminaries (PIPs): PIPs is limited by its fixed 8-frame temporal field of view, making chained long tracks vulnerable to drift after extended occlusions.Its visibility-aware chaining is also difficult to parallelize and computationally expensive for long-range multi-particle tracking.
  • 4.2. Expanding the temporal field of view (PIPs+): PIPs+ replaces the fixed-width MLP-Mixer with an 8-block 1D Resnet whose final receptive field spans 35 timesteps.Because the module is iterated during inference, its effective receptive field is substantially larger than 35 timesteps.
  • 4.2. Expanding the temporal field of view (PIPs+): PIPs+ improves long-range tracking accuracy and increases average inference speed from 4 FPS to 55 FPS on 720×1080 videos using an Nvidia V100 GPU.The convolutional design supports training and testing with different video lengths, although roughly similar sequence lengths remain important in practice.
  • 4.3. Template updating (PIPs++): PIPs++ adds recent-appearance templates sampled along estimated trajectories to complement the first-frame template during appearance changes and occlusions.The full model omits residual feature updates and visibility estimation after adding the multi-template strategy.

5. Experiments

The experiments compare PIPs variants with established point trackers on PointOdyssey and two real-video benchmarks using accuracy, trajectory error, and survival metrics. PIPs+ and PIPs++ perform best among the evaluated methods, while qualitative results show more stable trajectories but persistent difficulty near boundaries.

  • 5.1. Experimental setup: The evaluation compares PIPs, TAPNet, PIPs+, PIPs++, RAFT, and DINO on PointOdyssey, TAP-Vid-DAVIS, and CroHD.PIPs and TAPNet are trained on PointOdyssey, while pretrained RAFT and DINO weights are used for evaluation.
  • 5.1. Experimental setup: The study reports average position accuracy, Median Trajectory Error, and Survival, which measures the fraction of video frames tracked before failure.Failure is defined as an L2 distance exceeding 50 pixels at normalized 256 × 256 resolution.
  • 5.2. Evaluation: PIPs+ and PIPs++ achieve the best results among all evaluated methods, demonstrating the effectiveness of wider temporal awareness.The narrow performance gap between the two variants indicates that multi-template updating has a modest but average-positive effect.
  • 5.2. Evaluation: Method rankings are consistent across PointOdyssey, TAP-Vid-DAVIS, and CroHD, while PointOdyssey appears more challenging than the two real-world datasets.The authors interpret this ranking consistency as suggesting a correlation between progress on PointOdyssey and progress on videos in the wild.
  • 5.2. Evaluation: Qualitative results show more-stable point trajectories for the proposed method than for PIPs and other baselines, but all methods struggle with targets near boundaries.Thin objects are identified as especially difficult targets.

6. Limitations

PointOdyssey has scope limitations in its current outdoor-scene coverage and motion-profile diversity, while leaving scene-level and semantic tracking cues unexplored.

  • PointOdyssey lacks large outdoor scenes where the camera travels substantial distances, a scenario frequent in driving data.
  • Human and animal motion profiles are constrained by the base motion-capture datasets used to generate them.
  • Scene-level and semantic cues remain an open challenge for tracking, although the training data may support future work in this direction.

7. Conclusion

The paper concludes that PointOdyssey is a diverse, naturalistic synthetic resource for long-term point tracking, demonstrated through PIPs++, while identifying broader applications and future tracking directions.

  • PointOdyssey is a large-scale synthetic dataset and data generator for long-term point tracking.
  • PIPs++ demonstrates the dataset’s usefulness by leveraging long-term temporal context and outperforming state-of-the-art.
  • The dataset may support trackers using scene-level and semantic cues, although the paper does not explore this direction.
  • The authors identify potential uses beyond point tracking, including 3D and 4D scene analysis and higher-level video understanding.

A. Implementation Details

The implementation trains PIPs+ and PIPs++ with an iteratively weighted L1 trajectory loss, including occluded targets, while contrasting PIPs++ with prior trajectory chaining.

  • PIPs+ and PIPs++ use a weighted L1 distance between estimated and ground-truth trajectories across iterative updates.
  • The full training loss averages per-point losses across the N query points.
  • The trajectory loss remains active during occlusion, requiring the model to estimate tracks through visibility gaps.
  • PIPs++ avoids the prior method’s per-point 8-frame trajectory-chaining strategy, which ran at 3.6 FPS on the stated hardware and resolution.

B. Additional Visualizations of PointOdyssey

Additional visualizations show animated characters, reconstructed motion-capture environments, and pixel trajectories from PointOdyssey, with supplementary video recommended for further examples.

  • Retargeted motion data animates dataset characters into interacting sequences.
  • Figure 11 presents samples of the motion-capture environments rebuilt in the simulator.
  • Figure 12 shows sample pixel trajectories from the dataset.
  • The authors recommend the supplementary video for additional visualizations.

C. Additional Results

PointOdyssey combines diverse synthetic scenes, characters, motions, and trajectories, while PIPs++ is reported to handle occlusions more reliably than PIPs on real-world data.

  • PIPs++ trajectories are on average less sensitive to occlusions than PIPs on real-world data.PIPs tracks visible points effectively but struggles when occlusions occur.
  • Sample characters illustrate the range of entities represented in PointOdyssey.
  • The dataset includes sample motions and rebuilt 3D scenes featuring humans placed at random trajectory timesteps.
  • Sample videos span robots, crystal rabbits, dancing humans, and egocentric human interactions with point trajectories.
  • Figure 13 qualitatively contrasts PIPs and PIPs++ across horse-riding and bike-riding sequences with different visibility conditions.PIPs performs well during high visibility, whereas PIPs++ withstands occlusions more frequently.
Loading 2307.15055v1…