Source-linked AI summary
D3S -- A Discriminative Single Shot Segmentation Tracker
Alan Lukežič, Jiří Matas, Matej Kristan
TL;DR
Bounding-box trackers are robust but limited for general transformations and poorly shaped targets, motivating segmentation-based tracking. D3S combines adaptive Euclidean and deformation-invariant models in a single-shot segmentation pipeline. It performs strongly across tracking and segmentation benchmarks without dataset-specific retraining, while operating close to real time.
Problem
Bounding-box tracking is limited for general transformations and becomes unreliable when a box poorly approximates deformable or articulated targets.
Method
D3S uses complementary GEM and GIM discriminative models and produces a segmentation map in one end-to-end network pass trained for segmentation.
Results
D3S outperforms state-of-the-art trackers on major tracking benchmarks, competes with top segmentation methods, and runs up to 200× faster without per-dataset retraining.
Takeaways & Limitations
D3S narrows the gap between visual object tracking and video object segmentation by combining robust discrimination with accurate online segmentation.
Abstract
from arXiv · showhide
Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker - D3S, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid object to simultaneously achieve high robustness and online target segmentation. Without per-dataset finetuning and trained only for segmentation as the primary output, D3S outperforms all trackers on VOT2016, VOT2018 and GOT-10k benchmarks and performs close to the state-of-the-art trackers on the TrackingNet. D3S outperforms the leading segmentation tracker SiamMask on video object segmentation benchmark and performs on par with top video object segmentation algorithms, while running an order of magnitude faster, close to real-time.
1. Introduction
D3S addresses limitations of bounding-box tracking by combining complementary geometric models in a single-shot segmentation tracker. It achieves robust localization and segmentation while generalizing across tracking and segmentation benchmarks without per-dataset retraining.
- Template-based trackers efficiently handle translation and scale but become inefficient for aspect-ratio changes and rotation, while refinement approaches remain restricted to axis-aligned rectangles.
- High-dimensional transformation estimation becomes unreliable when bounding boxes poorly approximate elongated, rotating, deformable, or articulated targets.Per-pixel segmentation is presented as a more accurate target-location model in these cases.
- D3S encodes targets with GEM, an adaptive Euclidean model, and GIM, a deformation-invariant model covering broader transformations.The two models are designed to address the limitations of existing tracking approaches.
- D3S produces segmentation as its primary output in a single network pass, with GIM supporting deformation tolerance and GEM improving discrimination and target selection.The network is trained end-to-end for segmentation only.
- The tracker converts segmentation masks into rotated rectangles, avoiding greedy search while handling location, scale, aspect ratio, and rotation changes.
- D3S outperforms state-of-the-art trackers on most major tracking benchmarks and outperforms the leading segmentation tracker while matching top video object segmentation algorithms at much higher speed.A single pretrained version is used without retraining for different benchmarks.
2. Related Work
Related tracking and video object segmentation methods optimize different operating conditions, motivating D3S's unified architecture. D3S processes GEM and GIM pathways in parallel and refines their outputs into a segmentation map.
- Discriminative correlation filters emphasize target-background discrimination through template learning, while Siamese trackers use fixed templates and backbones for real-time performance.
- Video object segmentation methods achieve strong results but often require large networks and finetuning, and their short-video assumptions make them less effective for small, fast-moving tracking targets.
- D3S's architecture processes backbone features through parallel GEM and GIM pathways, producing location, foreground similarity, and target posterior channels for refinement.
3. Discriminative segmentation network
D3S combines geometrically invariant and geometrically constrained target models in parallel, then fuses their complementary channels into an upscaled segmentation map. GIM supports deformation-tolerant target-background separation, while GEM adds discriminative localization and target selection.
- Architecture: D3S processes GIM and GEM in parallel, producing coarse target-presence channels that a refinement pathway fuses into a detailed segmentation map.The refinement pathway combines the foreground similarity and posterior channels from GIM with GEM’s target location channel, then progressively increases resolution.
- Geometrically invariant model: GIM uses target and background feature sets to compare search-region pixels and produce foreground, background, and posterior channels.The posterior channel is obtained by applying softmax to the foreground and background similarity channels.
- Geometrically invariant model: GIM relaxes spatial constraints to localize targets under significant deformation but cannot reliably distinguish targets from similar instances.Its broad transformation tolerance improves target-background separation, while similar-instance discrimination remains limited.
- Pathway complementarity: GEM provides robust but coarse localization and discriminative selection, complementing GIM’s detailed but less discriminative segmentation information.The complementary outputs address the weaknesses of relying on either pathway alone.
- Geometrically constrained Euclidean model: GEM reduces backbone features to 64 channels, correlates them with a DCF, and converts the maximum response into a per-pixel target-location confidence channel.A Euclidean distance transform spreads confidence from the most likely target position across the search region.
4. Discriminative Segmentation Tracker
D3S applies its single-shot segmentation network online from a first-frame supervised example, producing masks for subsequent frames and fitting bounding boxes when required. Initialization and tracking update the GIM and GEM models using target-location information.
- Online tracking: Given one supervised first-frame example, D3S produces target segmentation masks in all remaining frames and derives bounding boxes when evaluation requires them.Axis-aligned boxes tightly fit the segmentation mask, while rotated boxes use a separate fitting procedure.
- Bounding box fitting: Rotated-box fitting thresholds the segmentation map at 0.5, keeps the largest connected component, fits an ellipse, and optimizes rectangle overlap.The procedure uses a modified IoU cost and coordinate descent to reduce oversized initial rectangles.
- Bounding box fitting: 2ms is the average runtime of the rotated bounding-box fitting method.This timing refers to the fitting procedure rather than the full tracking pipeline.
- Initialization: D3S initializes with the ground-truth target location, training GEM from a bounding box or extracting GIM foreground and background samples from a segmentation mask or approximation.The background neighborhood extends to four times the target size, and GEM initialization uses first-frame backpropagation when a box is available.
- Tracking: During tracking, D3S processes a region four times the target size, outputs a segmentation mask, optionally fits a rotated box, and updates GEM’s DCF.The DCF update follows the estimated target location after each new frame is processed.
5. Experiments
D3S is evaluated across short-term tracking and video object segmentation benchmarks, with strong accuracy, robustness, generalization, and speed. Ablations show that its GIM, GEM, position estimation, and rotated-box fitting choices materially affect performance.
- Tracking benchmarks: D3S outperforms all tested trackers on VOT2016, including SPM by 14% in EAO and ATOM by 25% in robustness.It also exceeds SiamMask by over 3% in accuracy and approximately 50% in robustness.
- Segmentation benchmarks: D3S achieves 0.66 average IoU on VOT2016 segmentation, compared with 0.63 for SiamMask.This corresponds to a nearly 5% improvement on small, challenging targets during successful tracks.
- Tracking benchmarks: On VOT2018, D3S exceeds SiamRPN++ by 18% in EAO, SiamMask by over 5% in accuracy, and LADCF by over 6% in robustness.It outperforms SiamMask by over 45% in robustness.
- Ablation study: Removing the target posterior channel causes a 13.5% performance drop, compared with 4.5% after removing foreground similarity.Both channels contribute individually to robust target localization.
- Ablation study: Removing the GIM module reduces overall tracking performance by 27%, while removing GEM reduces it by nearly 50%.GIM removal increases failures by 56%, whereas GEM removal increases failures by over 270%, demonstrating complementary robustness and target-selection roles.
- Ablation study: Updating GEM from its own estimated position reduces overall performance by 7.5% and increases failures by over 15%.The result links accurate final segmentation-based position estimation to effective DCF learning in GEM.
- Segmentation benchmarks: D3S performs on par with most top DAVIS video object segmentation methods while running 200 times faster than the top performer on DAVIS2016.It also improves over SiamMask by over 5% in both Jaccard index and contour-based F-measure.
6. Conclusion
D3S combines geometrically invariant and Euclidean models in a single-shot, end-to-end segmentation tracker. It achieves strong tracking and segmentation performance while remaining substantially faster than competing methods.
- D3S combines geometrically invariant and geometrically restricted Euclidean models in parallel pathways to localize deformable targets and discriminate distractors.The end-to-end architecture connects discriminative tracking with accurate segmentation.
- D3S outperforms state-of-the-art trackers on VOT2016, VOT2018, and GOT-10k, and performs on par with top trackers on TrackingNet.The same model was trained once on Youtube-VOS for segmentation only and used across all benchmarks.
- D3S significantly outperforms SiamMask across all video object segmentation benchmarks and metrics.
- 200× faster than recent top segmentation methods, D3S runs close to real-time while achieving performance close to top segmentation methods.
Qualitative examples
Qualitative examples show D3S maintaining accurate tracking and segmentation under distractors, deformation, clutter, low contrast, small targets, and abrupt motion. The tracker outputs masks and derives benchmark-specific bounding boxes from them.
- D3S outputs segmentation masks, fitting rotated or axis-aligned bounding boxes as required by VOT and GOT-10k protocols.DAVIS evaluation uses the binary segmentation mask directly.
- Video object segmentation: D3S produces near-real-time video object segmentation results on DAVIS16 and DAVIS17 datasets.For multiple-object sequences, the tracker runs independently on each target.
- Distractors: D3S segments the correct target despite nearby or overlapping visually similar distractors through its discriminative GIM and GEM models.
- Deformation: D3S segments deformable targets under rotation, articulation, and substantial shape changes using its geometrically invariant model.
- Challenging sequences: D3S maintains accurate segmentation in low-contrast underwater sequences and tracks small targets among similar leaves with a large search range.The leaf example combines a search range four times the target size with the discriminative architecture.