PaperlayerView exact revision on arXiv

Source-linked AI summary

DepthTrack : Unveiling the Power of RGBD Tracking

Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, Joni-Kristian Kämäräinen

arXiv:2108.13962v1cs.CV

TL;DR

The paper addresses the shortage of RGBD tracking data for training depth-based trackers and testing when depth cues matter. It introduces DepthTrack and the DeT baseline, and reports that DeT wins both evaluation protocols by 12% in F-score over the second-best tracker.

  • Problem

    Available RGBD datasets provide too little annotated data to train depth trackers or challenge RGB trackers with sequences where depth cues are essential.

    Source: §1 · p. 2

  • Method

    The authors introduce DepthTrack and DeT, pre-training DeT with generated RGBD sequences and fine-tuning it on DepthTrack training data.

    Source: §1 · p. 2, §4.2 · p. 5, §6 · p. 8

  • Results

    DeT wins both evaluation protocols and obtains +12% better F-score than the second-best tracker.

    Source: §5.1 · p. 5

  • Takeaways & Limitations

    The authors expect their experiments, including findings on RGB-depth feature fusion, to facilitate better RGBD tracking datasets and trackers.

    Source: §1 · p. 2

    DeT has no long-term tracking capabilities.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

RGBD (RGB plus depth) object tracking is gaining momentum as RGBD sensors have become popular in many application fields such as robotics.However, the best RGBD trackers are extensions of the state-of-the-art deep RGB trackers. They are trained with RGB data and the depth channel is used as a sidekick for subtleties such as occlusion detection. This can be explained by the fact that there are no sufficiently large RGBD datasets to 1) train deep depth trackers and to 2) challenge RGB trackers with sequences for which the depth cue is essential. This work introduces a new RGBD tracking dataset - Depth-Track - that has twice as many sequences (200) and scene types (40) than in the largest existing dataset, and three times more objects (90). In addition, the average length of the sequences (1473), the number of deformable objects (16) and the number of annotated tracking attributes (15) have been increased. Furthermore, by running the SotA RGB and RGBD trackers on DepthTrack, we propose a new RGBD tracking baseline, namely DeT, which reveals that deep RGBD tracking indeed benefits from genuine training data. The code and dataset is available at https://github.com/xiaozai/DeT

1. Introduction

RGBD trackers have largely extended deep RGB trackers, using depth for special cases rather than as a trained tracking cue. DepthTrack addresses the shortage of annotated RGBD data with a new dataset, experiments, and the DeT baseline.

1. Introduction

  • The paper frames RGBD tracking as a setting where trackers localize an initially identified object in video, while evaluation distinguishes short-term and long-term protocols.The protocols differ in re-initialization and in whether trackers need target-presence detection and re-detection.Source: §1 · p. 1
  • The leading RGBD challenge trackers use RGB features from deep trackers, with depth mainly supporting occlusion, disappearance, or re-detection handling.The authors say this leaves the potential of depth features unclear.Source: §1 · pp. 1–2
  • Only 116 RGBD tracking sequences had target and attribute annotations, versus tens of thousands in major RGB datasets, motivating larger and more diverse RGBD data.The existing PTB, STC, and CDTB datasets contain 100, 36, and 80 sequences, respectively; PTB annotations are not public.Source: §1 · p. 2
  • DepthTrack is presented as a new annotated RGBD dataset, accompanied by experiments with state-of-the-art RGB and RGBD trackers and the DeT baseline.The sequences, metadata, and evaluation code are compatible with the VOT 2020 Python Toolkit.Source: §1 · p. 2

2. Related Work

Earlier RGBD benchmarks vary in size, annotation, and recording scope, while leading challenge trackers extend deep RGB models and use depth selectively. The paper positions its work against both these dataset constraints and prior tracking approaches.

2. Related Work

  • PTB has 100 indoor Kinect sequences but limited scene and attribute diversity, with reported RGB-depth synchronization errors in about 14% and misalignment in 8%.STC was designed to address PTB's drawbacks, though it contains 36 sequences and focuses on short-term tracking.Source: §2 · p. 2
  • STC contains 36 sequences with thirteen attributes, while CDTB has 80 sequences and 101,956 frames recorded indoors and outdoors.CDTB is used in the VOT-RGBD 2019 and 2020 challenges.Source: §2 · p. 2
  • Earlier RGBD trackers used engineered features or ad hoc RGB-depth fusion, while leading VOT-RGBD 2020 trackers extend ATOM, DiMP, and MDNet and rely mainly on RGB.The cited winning trackers use depth for long-term tracking tasks.Source: §2 · p. 2

3. The DepthTrack Dataset

DepthTrack combines diverse RGBD sequences and target annotations with a long-term tracking evaluation protocol. Its dataset design includes a scene-disjoint train/test split, per-frame attributes, and both sequence- and frame-based metrics.

3. The DepthTrack Dataset

  • DepthTrack uses one Intel RealSense 415 sensor, recording synchronized 640 × 360 RGB and depth frames at 30 fps.RGB is stored as 24-bit JPEG and depth as 16-bit PNG.Source: §3 · p. 2
  • DepthTrack emphasizes varied target types and difficult RGB-only cases, including camouflage, deformable objects, similar colors, dark scenes, and multiple similar objects.The test set has a more varied target distribution than CDTB, whose sequences are dominated by boxes, humans, and robots.Source: §3 · p. 2, Figure 2 · p. 3, Figure 3 · p. 3
  • The dataset has 150 training and 50 test sequences with no common scenes or objects, comprising 218,201 training frames and 76,390 test frames.Only basketball, soccer, and pigeon targets recur in the test set.Source: §3 · p. 3
  • Each frame has an axis-aligned target box and 15 scene attributes, including new background-clutter and camera-motion attributes.The boxes tightly fit the target; background clutter marks shared target/background color or texture, while camera motion marks movement linked to substantial target-depth change.Source: §3 · p. 3
  • Evaluation follows a long-term protocol in which trackers report visibility confidence and box coordinates, with precision, recall, and their harmonic-mean F-score used for ranking.The protocol reports both sequence-based and frame-based averages to address variable sequence lengths, and stores the highest F-score across confidence thresholds.Source: §3 · p. 4

4. Baseline RGBD Trackers

The section establishes existing RGB and RGBD trackers as baselines and introduces DeT, whose depth features are learned from depth tracking data. DeT’s feature-extraction output can be paired with an ATOM, DiMP, or similar tracking head.

4.1. Existing Baselines

  • 23 existing trackers were selected, including VOT-RGBD challenge winners, handcrafted RGBD trackers, and deep RGB trackers used by leading RGBD methods.The trackers use their authors’ code and default settings; parameters had been tuned on existing datasets, with VOT RGBD trackers optimized for CDTB.Source: §4.1 · p. 4, §4 · p. 4

4.2. A New Baseline - DeT

  • DeT obtains the best results using deep depth features learned from depth tracking data, with a tracking head that can be paired with ATOM or DiMP.The depth feature extractor and pooling layer form the baseline; the tracking head performs the actual tracking.Source: §4.2 · pp. 4–5
  • DenseDepth-estimated depth from 646 LaSOT sequences and 82K COCO images is used to pre-train DeT from scratch.The authors manually selected the sequences and images for generating the depth training data.Source: §4.2 · p. 5
  • After 50 training epochs, separately extracted RGB and depth features are merged by element-wise maximum pooling for the tracker’s reference and test branches.The pooling layer provides the single feature image required by the ATOM or DiMP tracking head; the head can also use another similar online tracking part.Source: Figure 4 · p. 4, §4.2 · p. 5

5. Experiments

The experiments compare DeT with RGB and RGBD trackers on DepthTrack and test its depth-trained features, pooling choices, and transfer to CDTB. DeT leads the main evaluation, and the ablations largely support the value of depth training.

5.1. Performance Evaluation

  • DepthTrack results are substantially lower than CDTB results for state-of-the-art trackers; ATCAIS scores 0.702 on CDTB and 0.476/0.483 on DepthTrack.Source: §5.1 · p. 5
  • With similar target and background colors or multiple similar objects, RGBD cues help tracking; depth-trained DeT succeeds on all illustrated examples.Depth-based trackers have difficulty with targets rotating or moving quickly in the depth direction.Source: §5.1 · p. 5, Figure 5 · p. 7
  • DeT wins both evaluation protocols on DepthTrack, with a +12% better F-score than the second-best tracker.Table 1 reports the DepthTrack test-set results for sequence- and frame-based evaluation.Source: §5.1 · p. 5, Table 1 · p. 6
  • DeT wins on 11 of 15 visual attributes, while DDiMP handles target loss such as partial occlusion and out-of-frame cases.DeT is strongest on depth-related attributes including dark scene and depth change.Source: §5.1 · p. 6, Figure 7 · p. 8
  • DeT runs in real time, while the second-best RGBD trackers DDiMP and ATCAIS reach 4.77 fps and 1.32 fps, respectively.Source: §5.1 · p. 6

5.2. DeT Ablation Study

  • DeT variants improve F-score over the original RGB trackers by 7.7% for ATOM and 10.5% for DiMP50.The comparison uses ATOM and DiMP50 as master trackers and adds a depth-trained depth pathway.Source: §5.2 · pp. 6–7, Table 2 · p. 6
  • Depth-only DeT-DiMP50(D) outperforms RGB-trained DiMP50(D) and is almost on par with RGB-input DiMP50.Table 3 defines the compared depth-only variants and the RGB-input master tracker.Source: Table 3 · p. 6, §5.2 · p. 7
  • Convolutional, max, and mean pooling all perform well against using RGB features for the depth channel; max and mean differ only slightly.Max pooling was selected because DiMP performed better than ATOM and was used in the method comparison.Source: §5.2 · p. 7, Table 4 · p. 8

6. Conclusion

DepthTrack is presented as the largest and most diverse RGBD benchmark and the first to provide a separate training set for deep RGBD trackers. The authors train DeT with RGBD data and report the best F-score in all experiments.

6. Conclusion

  • DeT obtained the best F-score in all experiments, with its depth pathway trained from depth maps.It was pre-trained with generated RGBD sequences and fine-tuned on the DepthTrack training set.Source: §6 · p. 8
  • DepthTrack is described as the largest and most diverse RGBD benchmark and the first with a separate training set for deep RGBD trackers.Source: §6 · p. 8
  • The authors identify a lack of public RGBD tracking datasets despite consumer RGBD sensors being available for years and popular in robotics.They say the power of the depth cue for tracking had remained unknown.Source: §6 · p. 8
  • DeT was pre-trained with generated RGBD sequences and then fine-tuned using the DepthTrack training set.Source: §6 · p. 8
Loading 2108.13962v1…