PaperlayerView exact revision on arXiv

Source-linked AI summary

MAST: A Memory-Augmented Self-supervised Tracker

Zihang Lai, Erika Lu, Weidi Xie

arXiv:2002.07793v2cs.CVcs.LG

TL;DR

Self-supervised dense tracking has progressed but still trails supervised methods, motivating a model that learns without manual annotations. MAST revises training signals, adds dynamic memory, and evaluates generalizability, achieving competitive benchmark performance and stronger transfer to unseen categories.

  • Problem

    Self-supervised dense tracking avoids labeled data but has remained far behind supervised methods, despite dense tracking’s importance for applications such as autonomous driving and video surveillance.

  • Method

    MAST learns pixel-wise frame correspondences self-supervised, revisits training and reconstruction choices, and adds a coarse-to-fine memory component for tracking.

  • Results

    15% on DAVIS-2017 and 17% on YouTube-VOS: MAST surpasses existing approaches by significant margins, becomes competitive with supervised methods, and outperforms most supervised methods on unseen categories.

  • Takeaways & Limitations

    Self-supervised tracking can learn strong representations without semantic annotations and may offer greater generalization capability than supervised approaches.

  • Takeaways & Limitations

    The decorrelated-color-space improvement is presented as a hypothesis that dropout forces more robust representations when missing channels cannot be predicted from observed channels.

Abstract

from arXiv · show

Recent interest in self-supervised dense tracking has yielded rapid progress, but performance still remains far from supervised methods. We propose a dense tracking model trained on videos without any annotations that surpasses previous self-supervised methods on existing benchmarks by a significant margin (+15%), and achieves performance comparable to supervised methods. In this paper, we first reassess the traditional choices used for self-supervised training and reconstruction loss by conducting thorough experiments that finally elucidate the optimal choices. Second, we further improve on existing methods by augmenting our architecture with a crucial memory component. Third, we benchmark on large-scale semi-supervised video object segmentation(aka. dense tracking), and propose a new metric: generalizability. Our first two contributions yield a self-supervised network that for the first time is competitive with supervised methods on standard evaluation metrics of dense tracking. When measuring generalizability, we show self-supervised approaches are actually superior to the majority of supervised methods. We believe this new generalizability metric can better capture the real-world use-cases for dense tracking, and will spur new interest in this research direction.

1. Introduction

Dense tracking assigns pixel-level masks to objects throughout a video, but self-supervised methods have lagged behind heavily supervised systems. MAST addresses this gap with revised training choices, memory augmentation, and a generalizability evaluation.

  • Dense tracking relocates objects throughout videos using pixel-level segmentation masks, unlike bounding-box visual object tracking.
  • Supervised dense tracking commonly fine-tunes pretrained encoders on extensive pixelwise annotations from datasets such as COCO, Pascal, DAVIS, and YouTube-VOS.
  • Self-supervised methods avoid labeled data but have remained far behind supervised methods in performance.
  • MAST revisits self-supervised training and reconstruction losses, then adds a memory component with coarse-to-fine attention to reduce tracker drift.
  • MAST introduces generalizability to measure performance gaps between seen and unseen categories, reporting stronger generalization for self-supervised approaches.
  • 15% on DAVIS-2017 and 17% on YouTube-VOS: MAST surpasses existing approaches by significant margins and becomes competitive with supervised methods.

2. Related Work

Related dense-tracking methods propagate masks or detect and segment objects, while video self-supervision learns representations from spatiotemporal structure. MAST extends memory-based tracking with scalable coarse-to-fine processing for high-resolution, multi-frame memories.

  • Dense tracking methods are typically propagation-based or detection/segmentation-based.
  • Propagation-based methods transfer masks from the first frame and often establish temporal correspondences using optical flow or metric learning.
  • Video self-supervised learning exploits spatiotemporal information, including temporal ordering and contrastive objectives, to learn representations.
  • Memory-augmented models read prediction-relevant information from internal or explicit memory repositories, commonly using recurrent updates or attention.
  • MAST uses coarse-to-fine attention to localize salient memory regions before fine-grained processing, addressing scaling limits with multiple high-resolution frames.

3. Method

MAST is a self-supervised dense tracker that learns frame correspondences through reconstruction and improves mask propagation with memory over multiple reference frames. Its method combines revised training choices with coarse-to-fine attention for efficient matching across temporally distant frames.

  • MAST trains without manual annotations and requires an object mask only for the first frame during inference.
  • Self-supervised tracking learns feature correspondences by reconstructing a target frame from pixels copied from previous frames through attention.Queries and keys are computed by a shared-weight feature extractor, while values are raw reference frames during training and masks during inference.
  • 3.2.1 Decorrelated Color Space: MAST replaces color-channel dropout in correlated RGB space with dropout in decorrelated representations such as Lab to encourage more robust tracking features.RGB channels are highly correlated, allowing a dropped channel to be inferred from the others; decorrelated dropout is intended to prevent reliance on local color information.
  • 3.2.2 Classification vs. Regression: MAST uses a Huber regression loss because precise pixel relocation requires preserving information that color quantization can lose.The authors conjecture that directly optimizing reconstructed versus real frames provides more discriminative correspondence signals.
  • 3.3.1 Multi-frame tracker: Memory augmentation addresses error accumulation from recursive propagation by attending to multiple reference frames instead of relying on one previous frame.The current frame supplies queries, while keys and values are retrieved from a memory bank; values are raw frames during training and object masks during testing.
  • 3.3.2 ROI Localization: MAST uses two-step attention for distant frames: coarse matching localizes likely regions of interest, followed by fine-grained pixel matching within those regions.The algorithm aggregates ROI-pixel labels using affinity-weighted similarities, while restricted attention controls the computational cost of memory access.

4. Implementation Details

The implementation uses a ResNet18 encoder trained from random initialization solely on raw video sequences, producing feature embeddings at one-quarter of the input image resolution.

  • The model uses the ResNet18 feature encoder architecture and produces embeddings at 1/4 of the original image resolution.
  • Training is completely self-supervised: the network starts with random weights and uses no information beyond raw video sequences.

5. Experiments

MAST is evaluated for dense tracking on DAVIS-2017 and YouTube-VOS, including standard accuracy, ablations, and generalizability across seen and unseen categories. It outperforms prior self-supervised methods and approaches supervised performance, while memory, training choices, and feature alignment materially affect results.

  • 5.2 Generalizability: The generalization gap measures average performance differences between seen and unseen object classes, penalizing large seen-category advantages and rewarding higher unseen-category performance.YouTube-VOS provides separate measures for seen and unseen categories, enabling category-agnostic evaluation.
  • 5.1.1 Main results: MAST surpasses prior self-supervised methods on DAVIS-2017 and can exceed supervised approaches despite using a ResNet18 encoder.The reported DAVIS comparison gives 65.5 versus 50.3 on J & F against CorrFlow.
  • 5.1.2 Ablation Studies: Lab color space, real-valued photometric regression, and improved image-feature alignment each outperform their compared alternatives.Cross-color-space matching performs worse, while feature alignment improves performance by about 2.2%.
  • 5.1.2 Ablation Studies: The combined short- and long-term memory model performs best, with qualitative results linking the improvement to reduced tracker drift during severe occlusion.The memory retrieves high-resolution information from temporally distant frames.
  • 5.2 Generalizability: MAST reaches 64.2 versus 46.6 against other self-supervised approaches on YouTube-VOS and achieves comparable performance to many heavily supervised methods.The evaluation uses the YouTube-VOS validation split with 474 videos and 91 object categories.
  • 5.2 Generalizability: On unseen object categories, MAST has a smaller generalization gap than other self-supervised methods and is comparable in generalizability to OSVOS.The reported gaps are 2.7 for OSVOS and 0.4 for MAST.

6. Conclusion

The paper concludes that MAST provides accurate and generalizable pixel-level tracking without semantic annotations. Its results narrow the gap with supervised methods and show strong performance on unseen categories.

  • MAST enables accurate and generalizable pixel-level tracking while training without semantic annotations.
  • MAST surpasses previous self-supervised methods on existing benchmarks and narrows the gap with supervised methods.
  • On unseen object categories, MAST outperforms all but one existing method trained with heavy supervision.

Appendix A. Network architecture

The network architecture uses a modified ResNet-18, with architectural details provided in the appendix table.

  • MAST uses a modified ResNet-18 architecture, as in CorrFlow.
  • Table 9 presents the network architecture and identifies residual blocks as residually connected operation sequences.

Appendix B. Optimal memory size

The memory-size analysis varies short- and long-term memory and finds that performance improves with additional reference frames before plateauing at five frames.

  • Performance initially increases as the memory grows and eventually plateaus at 5 frames.
  • The tested memory size consists of n short-term and m long-term frames, with n and m growing alternately.

Appendix C. Analysis by attributes

MAST is robust across DAVIS video attributes, with especially strong gains on occlusion-related sequences; background clutter remains a relative bottleneck.

  • MAST outperforms all other self-supervised and unsupervised models by a large margin across all DAVIS video attributes.The attributes represent different difficulties associated with the validation sequences.
  • MAST obtains significant gains on occlusion-related sequences such as OOC and OV.The authors suggest memory augmentation enables retrieving occluded objects from previous frames.
  • Background clutter produces a relatively small improvement over previous methods.The authors conjecture that the shared photometric loss causes this bottleneck and suggest texture-consistency losses as a possible alternative.
  • Table 10 lists DAVIS video attributes and reports validation accuracy broken down by attribute.

Appendix D. YouTube-VOS 2019 dataset

MAST is also evaluated on the larger YouTube-VOS 2019 validation set, which contains more videos and object instances than the 2018 version, with similar reported trends.

  • MAST and two other self-supervised methods are evaluated on the YouTube-VOS 2019 validation dataset.The numerical results are reported in Table 11.
  • The YouTube-VOS 2019 version contains more videos and object instances than the 2018 version.
  • The evaluation shows a similar trend of significant improvement and a lower generalization gap.
  • Table 11 reports video segmentation results on YouTube-VOS 2019, where higher values are better and daggers mark reimplemented results.

Appendix E. More qualitative results

Qualitative results show MAST handling difficult dense-tracking conditions, while attribute-wise results report a significant margin over previous self-supervised methods across all attributes.

  • MAST handles multiple similar objects, large camera shake, unseen object poses, and related tracking difficulties well.The examples include risks of conflating similar objects and motion blur from camera shake.
  • MAST outperforms previous self-supervised methods by a significant margin on all attributes.The attribute breakdown is presented in Figure 9.
  • Figure 10 shows qualitative tracking examples with frame numbers displayed at the top left.Rows cover similar-object tracking with scale change, occlusions and out-of-scene objects, camera shake, fine-detail small objects, and unseen deer poses.
Loading 2002.07793v2…