PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Jiarui Xu, Xiaolong Wang

arXiv:2103.17263v5cs.CV

TL;DR

Learning space-time correspondence is important for tracking and video segmentation, while existing self-supervised approaches often use explicit object- or patch-tracking pretexts. VFS learns correspondence by comparing embeddings of frames from the same video, and the authors report that it surpasses state-of-the-art self-supervised approaches on tracking and segmentation benchmarks.

  • Problem

    Space-time correspondence is important for tracking and segmentation, but many approaches rely on simulated training supervision or limited human annotations.

    Source: §1 · p. 1

  • Method

    VFS learns space-time correspondence by comparing fully connected embeddings of frames from the same video without an explicit tracking-based pretext task.

    Source: §1 · p. 2

  • Results

    VFS surpasses state-of-the-art self-supervised correspondence approaches on OTB object tracking and DAVIS video object segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VFS demonstrates that frame-level similarity learning can yield both fine-grained and object-level correspondence in different convolutional layers.

    Source: §5 · p. 8

Abstract

from arXiv · show

Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired by the recent success in image-level contrastive learning and similarity learning for visual recognition. Our hypothesis is that if the representation is good for recognition, it requires the convolutional features to find correspondence between similar objects or parts. Our experiments show surprising results that VFS surpasses state-of-the-art self-supervised approaches for both OTB visual object tracking and DAVIS video object segmentation. We perform detailed analysis on what matters in VFS and reveals new properties on image and frame level similarity learning. Project page with code is available at https://jerryxu.net/VFS

1. Introduction

Space-time correspondence matters for tracking and segmentation, but many existing approaches rely on simulated data or limited human annotations. VFS instead learns from comparing video-frame embeddings, and the paper reports strong correspondence results with frame gaps, augmentation, negative-pair, and network-depth effects.

1. Introduction

  • Unlike prior correspondence approaches relying on simulation or limited human annotations, VFS asks whether frame-level similarity can learn correspondence without explicit tracking tasks.The paper frames the question against methods using explicit object- or patch-tracking pretexts.Source: §1 · p. 1
  • VFS surpasses state-of-the-art self-supervised correspondence approaches, using res5 for object-level tracking and res4 for fine-grained segmentation tasks.The paper evaluates object-level correspondence on OTB tracking and fine-grained correspondence on DAVIS segmentation.Source: §1 · p. 2
  • VFS compares fully connected embeddings of frames from the same video, with the hypothesis that semantic recognition features require correspondence between similar objects and parts.The authors propose that minimizing frame-level feature distance lets correspondence emerge in convolutional features.Source: Figure 1 · p. 1, §1 · p. 2
  • Larger frame gaps improve fine-grained DAVIS correspondence by ∼3% and object-level OTB correspondence by > 10%, with multiple frame pairs adding further improvement.Source: §1 · p. 2
  • Training with color augmentation decreases DAVIS performance by ∼3% but improves OTB performance by ∼10%, benefiting object-level while harming fine-grained correspondence.The authors interpret the OTB gain as better object invariance.Source: §1 · p. 2
  • Training without negative pairs improves both fine-grained and object-level correspondence.The paper presents this as a property of VFS similarity learning.Source: §1 · p. 2
  • Deeper networks improve correspondence with VFS, unlike prior tracking-pretext approaches that show little improvement or worse results with ResNet-50 than ResNet-18.The cited prior approaches are [76] [35] [45].Source: §1 · p. 2
  • The authors position VFS as a strong baseline and report that similarity learning without negative pairs improves both object-level and fine-grained correspondence.Source: §1 · p. 2

2. Related Work

Prior work studies fine-grained and object-level temporal correspondence, often using synthetic data or human annotations. This paper situates VFS as an implicit alternative based on image-level similarity learning.

Temporal Correspondence

  • Fine-grained correspondence is studied in optical flow and motion estimation, while object-level correspondence supports long-range visual tracking.The section notes synthetic-data training as a constraint on optical-flow generalization and annotation reliance in tracking approaches.Source: §2 · p. 2

Self-supervised Learning from Videos

Image-level Similarity Learning

  • Image-level similarity learning attracts positive image pairs and repels negative pairs; contrastive methods use this framework for visual representation learning [55] [30] [3] [68] [28] [13] [53] [12] [10] [60].Prior work also reports that negatives are not necessary for good representations [26].Source: §2 · p. 3
  • VFS extends image-level similarity learning to video frames for correspondence, and reports that learning without negatives benefits space-time correspondence.Source: §2 · p. 3

3. Method

The method unifies image-level similarity learning with and without negative examples under Video Frame-level Similarity (VFS), which compares frames from different timestamps. It uses temporal sampling and image augmentations to learn frame representations and can extend training to multiple frame pairs.

3.1. Background: Image-level Similarity Learning

  • Image-level similarity learning brings augmented views of an image together, either contrasting positive pairs against negatives or minimizing feature distance without negatives.For the negative-pair approach, InfoNCE uses a momentum-updated queue as the negative bank; the negative-free approach uses a predictor and stops gradients to the target network.Source: §3.1 · pp. 3–4

3.2. Video Frame-level Similarity Learning

  • VFS treats frames from different timestamps as views for similarity learning, with each frame pulled toward a global video feature.The authors propose that temporal views encourage convolutional features to represent object structure and correspondence despite deformation and viewpoint changes.Source: §3.2 · p. 4
  • VFS supports both negative-pair and negative-free objectives, with multi-pair training comparing features across two splits of sampled video frames.With n sampled frames, the affinity matrix is n/2 × n/2 without negatives and n/2 × (n/2 + K) with K negatives from other videos.Source: §3.2 · p. 4
  • Distant sampling divides a video into segments and randomly selects one frame per segment, while continuous sampling uses a fixed frame interval.Distant sampling covers more of the video and provides a more aggressive augmentation effect; continuous sampling is used in some 3D-convolution architectures for locally consistent feature maps.Source: §3.2 · pp. 4–5, Table 1 · p. 4, Figure 3 · p. 4
  • VFS applies spatial and color augmentations alongside temporal views, and the authors report that color augmentation matters for correspondence learning.Examples include random cropping and flipping, as well as grayscale and color jitter.Source: §3.2 · p. 5

4. Experiments

The experiments evaluate VFS on fine-grained and object-level correspondence, with ablations of training choices and comparisons against prior methods. VFS performs strongly across both task types, while frame sampling, augmentation, negative pairs, and network depth affect results.

4.1. Self-Supervised Pre-Training

  • Pre-training uses ResNet backbones on about 240,000 Kinetics videos, with 100 epochs for ResNet-18 and 500 for ResNet-50.The paper describes architectures for training with and without negative pairs.Source: §4.1 · p. 5

4.2. Evaluation

  • Evaluation covers fine-grained correspondence on DAVIS-2017, JHMDB, and VIP, and object-level tracking on OTB-100.Fine-grained evaluation uses pre-trained features without fine-tuning; object-level comparison also reports a fine-tuning setting.Source: §4.2 · p. 5

4.3. Results and Ablative Analysis

  • Color augmentation decreases DAVIS performance by more than 3% but improves OTB precision by over 10%.The authors conjecture that appearance invariance helps object-level correspondence, while per-pixel distortion confuses lower-level features.Source: §4.3 · p. 6
  • ResNet-50 improves correspondence over ResNet-18, while res4 features improve for fine-grained matching and res5 scores remain near their early-training levels.Without negative pairs, ResNet-50 improves DAVIS J &Fm by 3.9% and OTB precision by 2.2% over ResNet-18.Source: Table 6 · p. 6, §4.3 · p. 7, Figure 4 · p. 7
  • Increasing frame intervals consistently improves both correspondence types, distant sampling gives the best results, and training with more frame pairs generally improves representations.These trends are reported with and without color augmentation for multiple-frame training.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 6
  • Adding negative pairs hurts both correspondence types, with a slight DAVIS drop and a significant OTB tracking decline.Without color augmentation, OTB precision drops by about 9%; negative-pair training also yields about 2% better ImageNet-1k linear classification, which the authors interpret as supporting their hypothesis about the trade-off.Source: §4.3 · pp. 6–7, Table 5 · p. 6

4.4. Comparison with State-Of-The-Art

  • VFS with ResNet-50 achieves state-of-the-art fine-grained results and surpasses ImageNet pre-training and SeCo on object-level tracking.ResNet-50 improves DAVIS by 2.2% and VIP by 3.3% over ResNet-18; on OTB, it gains 5% precision over ResNet-18, reaching 73.9% precision and 52.5% success.Source: §4.4 · pp. 7–8, Table 7 · p. 8, Table 8 · p. 8

5. Discussion and Conclusion

The conclusion presents VFS as a strong baseline and argues that tracking-based pretext tasks may not be necessary. It highlights trade-offs from color augmentation and negative pairs, as well as the benefits of larger frame gaps and multiple frame pairs.

5. Discussion and Conclusion

  • Tracking-based pretext tasks may be unnecessary, but the authors say they need to scale across model sizes and architectures to be useful.The paper states that these tasks are limited by small backbone models and are surpassed by frame-level similarity learning.Source: §5 · p. 8
  • Color augmentation benefits object-level correspondence but jeopardizes fine-grained correspondence, while large frame gaps and multiple pairs improve correspondence.The authors attribute the color trade-off to appearance invariance and confusion of lower-layer convolution features.Source: §5 · p. 8
  • Training without negative pairs improves both correspondence types; the paper describes this as a new property of similarity learning without negatives.Source: §5 · p. 8
  • VFS is proposed as a strong baseline, with frame-level similarity learning improving both object-level and fine-grained correspondence.The paper also presents intermediate representations as a way to understand what image-level similarity learning has learned.Source: §5 · p. 8

A. Implementation Details

The appendix gives inference and fine-tuning implementation details for fine-grained and object-level correspondence.

A. Implementation Details

  • Fine-grained inference propagates labels from the first-frame ground truth and recent predictions using top-k similar pixels within a local radius.The appendix points to a table of dataset-specific inference hyperparameters.Source: A. Implementation Details · p. 12, Table A.1 · p. 12
  • Object-level fine-tuning learns only an added 1×1 convolution over a frozen representation, using GOT-10K.The stated comparison protocol is described as linear evaluation; fine-tuning runs for 50 epochs.Source: A. Implementation Details · p. 12

B. Visualization

The visualizations show VFS correspondence across fine-grained segmentation and tracking tasks, including qualitative advantages over [35] and robustness in challenging OTB-100 tracking examples.

B. Visualization

Loading 2103.17263v5…