Source-linked AI summary
Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories
Adam W. Harley, Zhaoyuan Fang, Katerina Fragkiadaki
TL;DR
Pixel tracking is limited by the lack of long-range temporal context and by flow failures during occlusion. The paper revisits particle videos with PIPs, combining learned temporal priors, cost maps, iterative updates, and visibility estimation. Across trajectory and keypoint propagation evaluations, PIPs is reported to handle occlusions and provide strong correspondence performance, while remaining limited by independent trajectories and a finite temporal window.
Problem
Feature matching lacks temporal context, and optical-flow tracks fail when targets are occluded, motivating long-range pixel trajectories.
Method
PIPs takes an RGB video and an initial point, then combines multi-frame cost maps, iterative position-and-appearance updates, learned temporal priors, and visibility estimation.
Results
PIPs recovers targets after occlusions, produces smoother and finer-grained correspondences than feature matching, and performs best on average in the BADJA keypoint propagation evaluation.
Takeaways & Limitations
Long-range point trajectories can preserve amodal correspondences through multi-frame occlusions while extending moderate-length estimates using visibility cues.
Takeaways & Limitations
Independent trajectory estimation discards information shared between trajectories, and the model loses targets that remain occluded beyond its temporal window.
Abstract
from arXiv · showhide
Tracking pixels in videos is typically studied as an optical flow estimation problem, where every pixel is described with a displacement vector that locates it in the next frame. Even though wider temporal context is freely available, prior efforts to take this into account have yielded only small gains over 2-frame methods. In this paper, we revisit Sand and Teller's "particle video" approach, and study pixel tracking as a long-range motion estimation problem, where every pixel is described with a trajectory that locates it in multiple future frames. We re-build this classic approach using components that drive the current state-of-the-art in flow and object tracking, such as dense cost maps, iterative optimization, and learned appearance updates. We train our models using long-range amodal point trajectories mined from existing optical flow data that we synthetically augment with multi-frame occlusions. We test our approach in trajectory estimation benchmarks and in keypoint label propagation tasks, and compare favorably against state-of-the-art optical flow and feature tracking methods.
1 Introduction
The paper revisits particle videos as a long-range pixel-tracking representation that combines temporal context with persistence through occlusions. Persistent Independent Particles (PIPs) estimate independent multi-frame trajectories and are reported as robust to occlusions while producing fine-grained correspondences.
- Particle videos represent videos with particles moving across multiple frames, bridging feature tracking and optical flow through long-range temporal priors.
- Feature matching lacks temporal context, while optical-flow tracks disappear during occlusions because occluded targets are absent from the flow field.
- Particle videos target persistence through occlusions and multi-frame temporal context, addressing limitations of feature matching and mostly pairwise flow methods.
- PIPs take a T-frame RGB video and an initial target coordinate, then independently produce a T × 2 trajectory for each queried particle.
- Experiments report that PIPs recover entities after occlusion, provide smoother feature correspondences, and support arbitrarily-long trajectories using visibility cues.
2 Related Work
Related work spans optical flow, feature matching, multi-frame flow, and object-centric tracking. PIPs differs by explicitly updating trajectories across multiple frames and targeting amodal point tracks that reconnect after occlusions.
- Modern optical-flow methods estimate displacement fields between consecutive frames, with RAFT using iterative optimization-like refinement.
- Ren et al. fuse candidate flows from previous frames, whereas PIPs explicitly reasons over multiple frames and iteratively updates estimates across them.
- Earlier multi-frame approaches use short context or constant-velocity priors, while PIPs learns a temporal prior to search for re-emergence after occlusion.
- Feature-matching methods use temporal consistency, reconstruction, or supervised correspondence objectives to connect features across frames.
- The paper transfers occlusion handling and temporal priors from object-centric tracking to point-level correspondence.
3 Persistent Independent Particles (PIPs)
PIPs track queried points through a video by combining feature extraction, trajectory initialization, local multi-scale similarity, and iterative position-and-appearance updates. Visibility estimation and synthetic amodal supervision support tracking through occlusions.
- 3.1 Setup and Overview: PIPs take an RGB video and an initial pixel coordinate, output per-timestep coordinates and visibility estimates, and process multiple target points without sharing trajectory information.
- 3.2–3.3 Initialization: The model extracts independent per-frame convolutional features, then initializes target features and positions across time using appearance-constancy and zero-velocity priors.
- 3.4 Measuring Local Appearance Similarity: Local appearance similarity uses correlation maps, bilinear crops around current positions, and a spatial pyramid to measure matches across timesteps and scales.
- 3.5 Iterative Updates: An MLP-Mixer consumes features, correlations, and displacements to update positions and features iteratively, recomputing correlation pyramids after each update.
- 3.5–3.6 Supervision: The final features produce per-timestep visibility scores, while trajectory supervision applies an exponentially weighted L1 loss even during occlusion or out-of-bounds periods.
4 Implementation Details
The implementation uses RAFT-inspired convolutional features and correlation pyramids with an MLP-Mixer update module. Its speed advantage applies for manageable target counts, while memory scales with sequence length and particle count.
- CNN: The CNN uses RAFT’s BasicEncoder, producing 256-dimensional features after strided and residual convolutions.
- Local correlation pyramids: Correlation pyramids use four levels with radius 3, producing four 7 × 7 patches per timestep.
- MLP-Mixer: The MLP-Mixer flattens per-timestep displacements, features, and correlation pyramids into token-like vectors before mapping outputs to feature and coordinate updates.
- Training: Training samples 128 trajectories per batch element at 368 × 512 resolution for 100,000 steps using AdamW and a 3e-4 learning rate.
- Complexity: 200ms versus 2000ms at 480 × 1024: PIPs is faster than RAFT when target counts fit on a 12G GPU, but may be slower when batching is required.
5 Experiments
Experiments evaluate one shared PIPs model across synthetic and real trajectory benchmarks, keypoint propagation, and qualitative occlusion cases. PIPs is generally robust to occlusions and competitive across domains, while its main limitations concern spatial information sharing and temporal-window length.
- Experimental setup: The same PIPs model is evaluated on FlyingThings++, KITTI, CroHD, BADJA, and DAVIS visualizations without domain-specific parameter customization.The benchmarks cover synthetic and real videos, including object, crowd-head, and animal keypoint tracking.
- Experimental setup: FlyingThings++ training mines accurate multiframe trajectories, then adds synthetic occlusions to create challenging training data.The source trajectories are filtered using consistency, image-bound, and instance-identity checks before occlusion augmentation.
- Trajectory estimation in FlyingThings++: On FlyingThings++, DINO struggles with rapid motion, while RAFT errors increase sharply for heavily occluded trajectories and often drift onto occluders.PIPs can locate targets after re-emergence and inpaint missing trajectory portions, producing better overall performance.
- Trajectory estimation in KITTI: On KITTI, RAFT is slightly better for continuously visible targets, whereas PIPs is slightly better under occlusion; DINO has much higher error.KITTI motion is relatively slow, but DINO’s low-resolution features only coarsely track targets.
- Trajectory estimation in CroHD: On CroHD, PIPs outperforms RAFT and DINO by a wide margin under both visibility conditions.The dataset contains high-resolution crowd videos with people’s heads annotated across 8-frame sequences.
- Keypoint propagation in BADJA: In BADJA, PIPs is best on four of seven videos and leads on average by 9 points, while DINO wins the remaining videos by at most 3 points.Qualitative occlusion examples show PIPs staying on target where RAFT loses track after occlusion.
- Limitations: PIPs trades spatial awareness for temporal awareness, preventing information sharing between trajectories despite strong single-particle tracking performance.The authors identify concurrent estimation of multiple point trajectories as a direct future-work direction.
- Limitations: Because the MLP-Mixer is not recurrent across time, PIPs loses the target when occlusion lasts beyond its temporal window.Longer trajectories can be generated by reinitializing inference, but longer or variable temporal context remains an open direction.
6 Conclusion
The paper presents PIPs for multi-frame point trajectory estimation through occlusions, combining cost volumes, iterative inference, and temporal reasoning. It positions longer-range trajectories as a complement to optical flow and feature matching.
- PIPs estimates point trajectories across multiple frames while jointly reasoning about location and appearance.The method combines cost volumes, iterative inference, and a deep temporal network.
- Particle videos add temporal context absent from matching-based methods and persistence through occlusions missing from flow-based methods.
- The authors propose longer-range fine-grained correspondences as a basis for future architectures and datasets for video understanding.
- The acknowledgements identify support from Toyota Research Institute, U.S. government agencies, and multiple research awards.