PaperlayerView exact revision on arXiv

Source-linked AI summary

Track to Reconstruct and Reconstruct to Track

Jonathon Luiten, Tobias Fischer, Bastian Leibe

arXiv:1910.00130v3cs.CV

TL;DR

Long-term tracking through occlusion and disappearance remains a challenge for methods focused on continuously visible objects. MOTSFusion builds short 2D tracklets, uses dynamic 3D reconstructions to merge them and recover missing detections, and outperforms prior methods for bounding-box and segmentation tracking.

  • Problem

    Many tracking approaches handle continuously visible objects but fail to maintain correct IDs through disappearance and occlusion, despite the importance of long-term tracking for autonomous systems.

    Source: §I · p. 1

  • Method

    MOTSFusion forms short tracklets from segmentation-mask consistency under optical-flow warps, then uses depth and ego-motion to reconstruct objects and merge tracklets using estimated 3D motion.

    Source: §III · p. 3

  • Results

    MOTSFusion outperforms previous methods for both bounding-box and segmentation tracking, and tracks through complete occlusion or missed detections.

    Source: §V · p. 8

  • Takeaways & Limitations

    The evaluations show a benefit from using 3D reconstruction to improve tracking, including long-term tracking through occlusions.

    Source: §V · p. 8

    The 3-DoF motion simplification is described as valid for KITTI because its ground plane is approximately flat.

    Source: §III · p. 4

Abstract

from arXiv · show

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstruction-based tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.

I. INTRODUCTION

Long-term tracking through occlusion remains difficult for methods focused on continuously visible objects. MOTSFusion closes the tracking–reconstruction loop by using 3D motion from reconstructions to improve tracking.

I. INTRODUCTION

  • MOTSFusion addresses long-term tracking failures by using reconstructed 3D object motion to track through occlusion and recover missing detections.The introduction identifies persistent IDs through disappearance and occlusion as important for applications including autonomous vehicles and mobile robots.Source: §I · p. 1
  • Figure 1 pairs a world-coordinate scene reconstruction with object reconstructions for three tracks, while Figure 2 contrasts tracklet points in world space and object-centric space.Figure 2 states that each point from the tracklet masks appears in both visualizations.Source: Figure 1 · p. 1, Figure 2 · p. 2
  • The pipeline builds short tracklets from segmentation-mask consistency under optical-flow warps, then uses depth and camera ego-motion to form dynamic 3D reconstructions and estimate object motion.Estimated trajectories merge tracklets through occlusion and missing detections; they are also used to fill missing detections and masks.Source: §I · p. 1
  • MOTSFusion reduces ID switches by around 50% versus initial 2D tracking and has 60–70% fewer than competing methods and using the same detections.The paper reports that the resulting tracks remain consistent through occlusion.Source: §I · p. 2

II. RELATED WORK

Prior tracking work explores varied association strategies and motion or appearance cues. MOTSFusion separates 2D tracklet creation from 3D tracklet merging and uses per-pixel depth to build object reconstructions over time.

II. RELATED WORK

  • MOTSFusion separates hierarchical association into 2D-motion tracklet creation and 3D-motion tracklet merging, unlike prior approaches,, and.It relies on motion consistency rather than visual similarity for long-term tracking.Source: §II · p. 2
  • Unlike prior 3D tracking methods that often use bounding boxes and average scene-flow vectors [27],, MOTSFusion aligns point correspondences given by optical flow.It also extends segmentation-tracking work using optical or scene flow, by using per-pixel depth to reconstruct objects over time.Source: §II · p. 2

III. OUR APPROACH

MOTSFusion first forms short image-space tracklets, then uses dynamic 3D object reconstructions and motion estimates to merge them into longer tracks and recover missing detections. Its pipeline also supports online tracking by restricting trajectory extrapolation to the future.

III. OUR APPROACH

  • MOTSFusion builds short tracklets from 2D mask motion under optical-flow warps, then uses depth and ego-motion to reconstruct objects and merge tracklets by 3D motion consistency.The resulting extrapolated positions are also used to fill detections where they are missing.Source: §III · pp. 2–3
  • Short tracklets are formed by warping segmentation-mask pixels with optical flow and associating masks across frames, while depth and camera pose place scene points in a shared world frame.The 4D scene reconstruction supports object-motion estimates independent of camera motion.Source: §III · p. 3
  • Before fitting motion, MOTSFusion removes high-LOF points, keeps corresponding points across timesteps, and samples at most 200 correspondences over each mask.The local outlier factor is computed using local reachability density and reachability distance.Source: §III · p. 4
  • For each tracklet, neighboring-frame point clouds are aligned with accumulated rigid-body transformations to form a consistent object-centric reconstruction and estimate 3D motion.The motion is restricted to X and Z translation and rotation around the ground-plane normal; the text states this simplification is valid for KITTI’s approximately flat ground plane.Source: §III · p. 4
  • Tracklets are merged when extrapolated 3D motion is consistent, using trusted motion regions of at least two contiguous low-residual transformations and position uncertainty in the consistency calculation.Motion is extrapolated toward the intervening timesteps; when robust estimates are unavailable, one-sided extrapolation or stationary assumptions are used.Source: §III · pp. 4–5
  • Between merged tracklets and at track boundaries, estimated 3D boxes are projected into image space and segmentation masks are checked against depth to fill visible missed detections while rejecting occlusions.The illustrated example shows interpolated 3D boxes across frames without detections and filled 2D boxes, masks, and a merged track ID.Source: Figure 4 · p. 4, §III · p. 5
  • The online version extrapolates trajectories only forward, matches current detections to those trajectories, and fills missing detections in the current frame.Initial 2D tracklet building remains online.Source: §III · p. 5

IV. EXPERIMENTS

On KITTI tracking benchmarks, MOTSFusion improves mask and bounding-box tracking, with experiments also examining reconstruction-based merging, missing-detection recovery, depth inputs, and runtime.

Segmentation Tracking: State-of-the-art comparison

  • On KITTI MOTS validation, MOTSFusion beats TrackR-CNN in sMOTSA: 78.2 vs. 76.2 for cars and 50.1 vs. 46.8 for pedestrians, with 61% and 56% fewer IDS, respectively.Source: Table I · p. 6, §IV · p. 6

Segmentation Tracking: Method ablation

  • Tracklet merging reduces IDS from 61 to 31 for cars and 55 to 35 for pedestrians without changing FPs or FNs; detection recovery further reduces FNs with marginally more FPs.Recovery lowers FNs from 386 to 364 for cars and 814 to 784 for pedestrians, while FPs rise from 37 to 44 and 94 to 99, respectively.Source: §IV · p. 6

Segmentation Tracking: Online version

  • The online version has negligibly more IDS than the offline version across five setups, but cannot use the same priors for missing-detection recovery.Source: §IV · p. 6

Segmentation Tracking: Oracle comparison

  • The best car method’s sMOTSA is only 1.2 percentage points below the theoretical maximum of a perfect tracker.The oracle still has false positives because the evaluation script prevents masks in a frame from overlapping, which can leave matched masks unmatched.Source: §IV · p. 6

Segmentation Tracking: Depth input ablation

  • MOTSFusion performs similarly with LiDAR and stereo depth; less accurate monocular depth lowers accuracy, but still improves over the 2D version.Source: §IV · p. 7, Table III · p. 7

Segmentation Tracking: Tracklet merging ablation

  • Merging tracklets up to 15 frames apart quickly improves MOTSA and IDS; beyond 25 frames recovery becomes less accurate, and beyond 60 frames merging also introduces errors.The main experiments use a 20-frame threshold; the figure contrasts reconstruction-based tracking with 2D tracklets and shows MOTSA without missing-detection recovery.Source: §IV · p. 7, Figure 5 · p. 7

Bounding Box Tracking: State-of-the-art comparison

  • On KITTI MOT validation/test, MOTSFusion cuts IDS versus BeyondPixels from 31 to 9 and 468 to 275; validation IDS* is 31 versus 91.MOTA is 0.3% and 0.6% higher on validation and test, respectively; validation MOTA* is 1.6% higher.Source: Table IV · p. 7, Table V · p. 7, §IV · p. 7

Runtime

  • MOTSFusion runs at 0.44 seconds per frame including preprocessing; reconstruction-based merging takes 0.17 seconds, mostly for fitting 3D transformations.The reported runtime excludes detector time; TrackR-CNN takes 0.50 seconds per frame and BeyondPixels 0.30 seconds, plus 0.06 seconds for MOTS mask generation.Source: §IV · p. 8

V. CONCLUSION

MOTSFusion combines tracking and 3D object reconstruction so each can benefit from the other. It improves bounding-box and segmentation tracking and can track objects through complete occlusion or missed detections.

V. CONCLUSION

  • MOTSFusion outperforms previous tracking methods in both bounding-box and segmentation tracking.The conclusion reports this across its evaluations.Source: §V · p. 8
  • The approach tracks objects long-term through complete occlusion or missed detections.The conclusion identifies these as settings where reconstruction-based tracking is useful.Source: §V · p. 8
  • The framework combines tracking and 3D object reconstruction, with each enabling the other.Source: §V · p. 8
Loading 1910.00130v3…