PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Track with Object Permanence

Pavel Tokmakov, Jie Li, Wolfram Burgard, Adrien Gaidon

arXiv:2103.14258v2cs.CV

TL;DR

Online tracking-by-detection can lose objects when instantaneous observations are degraded by occlusion. The paper introduces a recurrent joint detection-and-tracking model trained with synthetic invisible-object annotations, and reports state-of-the-art results on KITTI and MOT17.

  • Problem

    Tracking-by-detection depends on frame-local observations, creating a gap in reasoning about object locations and identities during occlusion.

  • Method

    PermaTrack extends CenterTrack to arbitrary-length videos with recurrent spatio-temporal memory and synthetic supervision for invisible objects.

  • Results

    State-of-the-art results are obtained on the KITTI and MOT17 multi-object tracking benchmarks.

  • Takeaways & Limitations

    The model tracks fully occluded objects and is more robust to partial occlusions and other low-visibility conditions such as motion blur.

  • Takeaways & Limitations

    No real dataset provides large-scale, consistent annotations for occluded objects because collecting them is costly and complex.

Abstract

from arXiv · show

Tracking by detection, the dominant approach for online multi-object tracking, alternates between localization and association steps. As a result, it strongly depends on the quality of instantaneous observations, often failing when objects are not fully visible. In contrast, tracking in humans is underlined by the notion of object permanence: once an object is recognized, we are aware of its physical existence and can approximately localize it even under full occlusions. In this work, we introduce an end-to-end trainable approach for joint object detection and tracking that is capable of such reasoning. We build on top of the recent CenterTrack architecture, which takes pairs of frames as input, and extend it to videos of arbitrary length. To this end, we augment the model with a spatio-temporal, recurrent memory module, allowing it to reason about object locations and identities in the current frame using all the previous history. It is, however, not obvious how to train such an approach. We study this question on a new, large-scale, synthetic dataset for multi-object tracking, which provides ground truth annotations for invisible objects, and propose several approaches for supervising tracking behind occlusions. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI and MOT17 datasets thanks to its robustness to occlusions.

1. Introduction

The paper targets online multi-object tracking methods whose frame-local localization can fail under occlusion. It introduces a recurrent, end-to-end approach trained with synthetic visibility annotations to reason about hidden object trajectories.

  • Motivation: Object permanence lets the model infer that fully occluded objects continue to exist and approximately localize them.The motivation draws on human reasoning about a person disappearing behind a moving car.
  • Motivation: Tracking-by-detection localizes objects independently in each frame before associating them with existing trajectories, limiting robustness under occlusion.The localization step does not use previous history.
  • Approach: The proposed approach extends CenterTrack from frame pairs to arbitrary-length video sequences using a convolutional gated recurrent unit.The recurrent module encodes spatio-temporal object evolution across the entire history.
  • Training: A synthetic dataset supplies accurate labels for objects regardless of visibility, enabling supervision of tracking behind occlusions.The paper analyzes both ground-truth and pseudo-ground-truth labels for invisible instances.
  • Results: Joint training on synthetic and real data overcomes the synthetic-to-real domain gap and supports trajectory hallucination for fully occluded objects.The model is validated on KITTI and MOT17 and is reported to outperform the state of the art.

2. Related Work

Related work largely separates detection and association, with online methods using current detections and offline methods constructing whole-video graphs. The paper differs by jointly learning detection and tracking with long-term memory and synthetic supervision for visibility.

  • Multi-object tracking: Multi-object tracking localizes objects from predefined categories and associates them over time by identity.Existing approaches commonly separate these tasks.
  • Tracking paradigms: Online trackers associate new frame detections with active trajectories, whereas offline methods optimize associations over spatio-temporal graphs.Offline formulations include maximum-flow and minimum-cost problems.
  • End-to-end tracking: Prior end-to-end detection-and-tracking methods capture only short-term correspondences from frame pairs or adjacent frames.The paper positions its video-level memory as addressing this limitation.
  • Object permanence: Classical object-permanence methods rely on inflexible heuristics, while recent learning-based work had promising results mainly on toy synthetic examples.The paper claims capability for full occlusions in real-world settings.
  • Video object detection: Video object detection methods use temporal context or memory to improve detection robustness, but do not tackle multi-object tracking.Their short-term associations primarily smooth detection outputs or produce tubelets.
  • Synthetic data: Synthetic data has been used to avoid manual labeling, often with unsupervised domain adaptation when real-world labels are unavailable.The paper uses synthetic supervision to address invisible-object annotations.

3. Methodology

The method extends CenterTrack from frame pairs to arbitrary-length videos by maintaining recurrent spatio-temporal memory. It also introduces visibility-aware and deterministic supervision for tracking objects through occlusions.

  • 3.2. A video-level model for tracking: CenterTrack is extended from consecutive frame pairs to arbitrary-length video sequences processed through a recurrent model.The original architecture localizes centers, regresses box sizes, predicts displacements, and greedily associates detections across frames.
  • 3.2. A video-level model for tracking: A ConvGRU updates a memory state with each frame’s feature map, representing the history of previously seen objects.The memory is recurrently decoded into object boxes and tracks, allowing the model to use prior observations rather than only instantaneous features.
  • 3.2. A video-level model for tracking: A visibility head distinguishes visible from fully occluded centers, while invisible predictions recover identities but are excluded from output bounding boxes.Occluded locations are supervised as localization positives and visibility negatives, preventing invisible predictions from being counted as false positives.
  • 3.3.1. Supervising occluded objects: Occluded objects are supervised only after being visible for at least two frames, providing enough information to estimate their location and velocity.The procedure uses visibility thresholds and treats objects visible for two consecutive prior frames as positives even when currently occluded.
  • 3.3.2. Supervising the invisible: Because future motion under full occlusion is ambiguous, the method replaces unavailable trajectory labels with constant-velocity pseudo-ground-truth propagated in 3D and projected into the camera frame.This deterministic supervision uses synthetic data with full ground-truth information, camera intrinsics, and extrinsic matrices.
  • 3.4. Bridging the sim-to-real domain gap: Real datasets lack large-scale, consistent annotations for invisible objects, motivating synthetic data for studying supervision behind occlusions.The annotation gap includes visibility scores, 3D coordinates, and camera matrices.

4. Experiments

Experiments evaluate the model architecture, occlusion-supervision strategies, dataset-size requirements, domain adaptation, and benchmark performance using Track AP/mAP and standard tracking metrics.

  • Evaluation protocol: Track AP is used for ablations on PD and domain adaptation on KITTI, while standard dataset metrics are reported for state-of-the-art comparisons.Track IoU-based metrics emphasize association accuracy more than MOTA.
  • Model variants: Longer video training unlocks object-permanence behavior, and the resulting variant significantly outperforms the baseline even when trained only on visible objects.The recurrent model is evaluated against frame-pair and architectural variants.
  • Tracking behind occlusions: 3.5 mAP improvement results from filtering ambiguous invisible-object annotations during occlusion supervision.Naively supervising all invisible objects slightly decreases performance, while filtered annotations improve it.
  • Tracking behind occlusions: 3D constant-velocity propagation produces deterministic labels consistent with observations and further improves performance over 2D propagation.The 2D variant is less robust to camera changes than ground-truth locations.
  • Effect of dataset size: The gap between full and visible-only training consistently increases with dataset size, while fewer than 75 videos produce nearly no gap.KITTI would not provide enough videos to train tracking behind occlusions, even with the required annotations.
  • Domain adaptation: Joint training on synthetic and real data achieves the best overall domain-adaptation results, whereas KITTI fine-tuning improves Person performance but lowers Car AP.Synthetic-only training can outperform CenterTrack without KITTI frames, but the Person-category gap remains large because of higher visual variability.
  • Comparison to the state of the art: PermaTrack outperforms the state of the art on all KITTI metrics and maintains a car trajectory through full occlusion in a qualitative example.The only exception is ML on Car, where the method is 0.4 points behind.

5. Conclusion

PermaTrack combines recurrent memory with synthetic-data supervision to reason about objects through full occlusion. It achieves state-of-the-art results on KITTI and MOT17, while history also improves robustness in low-visibility conditions.

  • 5. Conclusion: PermaTrack jointly detects and tracks objects using recurrent memory over the entire previous history.This enables reasoning about object locations beyond the current observation.
  • 5. Conclusion: Synthetic-data supervision trains PermaTrack to track objects when they are fully occluded through object permanence.
  • 5. Conclusion: PermaTrack obtains state-of-the-art results on the KITTI and MOT17 multi-object tracking benchmarks.
  • 5. Conclusion: Hallucinating trajectories of invisible objects is a crucial factor in final performance.Using the full history also increases robustness to partial occlusions and motion blur.

Appendices

The appendices provide supplementary visualizations, experimental results, implementation details, and expanded dataset descriptions, including the synthetic PD dataset.

  • Appendices: The appendix adds visualizations, experimental results, implementation details, and descriptions of the datasets used in the work.It also further elaborates on the ParallelDomain dataset.
  • Appendices: The supplementary video contains qualitative examples of the algorithm’s output.
  • Appendices: The appendix introduces additional material on the real-world nuScenes dataset for 3D tracking.

A. Qualitative analysis

Qualitative examples show PermaTrack maintaining trajectories through full, repeated, multi-target, and cross-view occlusions, while also exposing detector and confidence-related failure modes.

  • A. Qualitative analysis: PermaTrack successfully tracks a moving car through full occlusion while both cars and the ego vehicle are moving.
  • A. Qualitative analysis: PermaTrack maintains a grey car’s trajectory through repeated occlusions by three other vehicles.
  • A. Qualitative analysis: The method preserves trajectories through repeated occlusions involving pedestrians, groups, and cars.
  • A. Qualitative analysis: PermaTrack handles a complex multi-target occlusion scenario in MOT17.
  • A. Qualitative analysis: The method generalizes to a top-down street view in which a group of people is occluded by a pole.
  • A. Qualitative analysis: When two car centers overlap, the CenterPoint-based detector predicts one center, causing a missed wagon detection and an identity switch.Short-term constant-velocity post-processing can often fix such mistakes.
  • A. Qualitative analysis: A partially occluded person can eventually be lost because low initial confidence limits the approach.

B. Datasets

The evaluation uses KITTI and MOT17 real-world tracking benchmarks alongside ParallelDomain, a synthetic dataset for learning to track behind occlusions.

  • B. Datasets: KITTI contains city-driving sequences with 2D bounding-box annotations for cars, pedestrians, and cyclists at 10 FPS.The evaluation uses the car and pedestrian categories.
  • B. Datasets: MOT17 is a people-tracking benchmark whose videos mostly use static cameras and contain crowded indoor and outdoor areas.Its videos are annotated with 2D bounding boxes at 25–30 FPS.
  • B. Datasets: ParallelDomain is the synthetic dataset used to learn tracking behind occlusions.
  • B. Datasets: KITTI and MOT17 training videos are split to create validation sets, while KITTI test data is reserved for state-of-the-art comparison.

C. Statistics for the Parallel Domain Dataset

The Parallel Domain dataset provides large-scale synthetic urban-driving videos with amodal annotations for tracking visible and occluded objects. Its occlusion statistics supply substantial examples for learning object permanence.

  • The dataset contains 630 independent videos from 210 ten-second crowded urban-driving snippets, captured at 20FPS across three camera views.The split includes 582 training videos and 48 validation videos.
  • Amodal bounding boxes, instance identities, and visibility scores annotate nine object classes, with experiments focusing on Pedestrians and Cars.Both visible and occluded boxes are labeled across video frames.
  • 64.9% of Pedestrian and 58.1% of Car trajectories are fully occluded for at least 10% of their duration.These statistics provide training examples for learning to track with object permanence.
  • Photo-realistic imagery and amodal annotations support model analysis, prototyping, and pre-training for transfer to real-world datasets.The dataset is used with a sim-to-real adaptation strategy described in the experimental analysis.

D. Evaluation on nuScenes

The paper evaluates whether its approach can generalize to 3D tracking on nuScenes. It reports improved validation performance over CenterTrack while limiting the 3D study to an initial validation of generalization.

  • The proposed method improves nuScenes validation performance over the CenterTrack baseline on the main 3D tracking metrics.The comparison is conducted on the large-scale nuScenes benchmark.
  • 4.1 AMOTA points, or a 60% relative improvement, separate the proposed method from CenterTrack on nuScenes validation.This is the specific AMOTA improvement reported by the authors.
  • The 3D evaluation is intended to validate that the method can in principle generalize to the 3D scenario, not to provide a thorough 3D tracking investigation.The paper identifies low frame rate and substantial label noise in nuScenes as caveats.

E. Full Tables for KITTI and MOT17

The paper reports comprehensive KITTI and MOT17 evaluations using the benchmark metrics for each dataset. KITTI results cover HOTA, CLEAR MOT, and MT/PT/ML metrics, while MOT17 uses CLEAR MOT and MT/PT/ML metrics.

  • KITTI evaluation reports HOTA-based, CLEAR MOT, and MT/PT/ML metrics in Tables 8, 9, and 10.Full results are also available on the challenge website.
  • MOT17 evaluation reports CLEAR MOT and MT/PT/ML metrics on both validation and test sets.The test-set evaluation uses private detections, while the validation table includes private and public detections.

F. Further Implementation Details

Further details describe occlusion-focused loss weighting, dataset-specific fine-tuning, pseudo-ground-truth supervision, and the reported benchmark table organization.

  • Training for occlusions: Fully occluded instances receive a localization-loss weight multiplied by 20, and sequences are sampled in proportion to occlusion length.These choices discourage the model from ignoring invisible objects.
  • Domain adaptation: KITTI and MOT17 adaptation first pre-trains on Parallel Domain, then jointly fine-tunes on synthetic and corresponding real data with equally probable batches.Training uses batch size 16 for all datasets and five epochs in the described setup.
  • Domain adaptation: MOT17 occluded person locations are supervised with trajectory-interpolation pseudo-ground truth using sequences of length 13.This uses constant velocity as a reasonable approximation for mostly short-term occlusions from static cameras.
  • nuScenes training: nuScenes training uses pseudo-ground-truth labels and progressively increases sequence lengths from 2 to 6 and then 17 frames.The final stage freezes the backbone to capture longer-term occlusions.
  • Reported tables: The KITTI tables are organized by HOTA, CLEAR MOT, and MT/PT/ML metrics, while MOT17 tables report validation and test-set benchmark metrics.The cited tables include Tables 8–12 and the metric headings include identity, accuracy, tracking, and fragmentation measures.
Loading 2103.14258v2…