Source-linked AI summary
Self-supervised Learning for Video Correspondence Flow
Zihang Lai, Weidi Xie
TL;DR
The paper addresses self-supervised learning of feature embeddings for pixelwise video correspondence matching without manual correspondence labels. It trains a pointer to reconstruct target frames from reference frames, using an information bottleneck and recursive long-window training with scheduled sampling and cycle consistency. The resulting representation achieves state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking, with further video-segmentation gains from more diverse training data.
Problem
Manual correspondence annotations are expensive or impossible under occlusions and complex object deformations, motivating self-supervised pixelwise matching in videos.
Method
The method reconstructs target frames from reference frames while using an information bottleneck, recursive long-window training, scheduled sampling, and forward-backward cycle consistency.
Results
The approach achieves state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking, outperforming previous self-supervised methods by a significant margin.
Takeaways & Limitations
Additional and more diverse video data further improves video segmentation, indicating that self-supervised correspondence flow remains far from saturated.
Takeaways & Limitations
Training on Kinetics is not ideal because it is human-centric and does not cover many DAVIS classes, such as cars and animals.
Abstract
from arXiv · showhide
The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal coherence in videos, we propose to train a ``pointer'' that reconstructs a target frame by copying pixels from a reference frame. We make the following contributions: First, we introduce a simple information bottleneck that forces the model to learn robust features for correspondence matching, and prevent it from learning trivial solutions, \eg matching based on low-level colour information. Second, to tackle the challenges from tracker drifting, due to complex object deformations, illumination changes and occlusions, we propose to train a recursive model over long temporal windows with scheduled sampling and cycle consistency. Third, we achieve state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking tasks, outperforming all previous self-supervised learning approaches by a significant margin. Fourth, in order to shed light on the potential of self-supervised learning on the task of video correspondence flow, we probe the upper bound by training on additional data, \ie more diverse videos, further demonstrating significant improvements on video segmentation.
1 Introduction
The paper targets self-supervised pixelwise correspondence matching by exploiting videos’ spatial-temporal coherence, avoiding expensive manual correspondence annotations. It introduces robustness mechanisms for trivial matching and tracker drift, then evaluates transfer to segmentation and keypoint tracking.
- Correspondence matching supports depth estimation, optical flow, segmentation, tracking, and 3D reconstruction, but manual annotations are expensive or impossible under occlusions and complex deformations.
- Without fine-tuning, the learned representation transfers to video segmentation and keypoint tracking, while additional diverse videos further improve video segmentation.
- Videos provide abundant data and intrinsic spatio-temporal coherence that can supply proxy supervision for learning correspondence representations.
- The method uses channel-wise dropout and colour jittering during frame reconstruction so features cannot rely on low-level colour information.
- Recursive training over long temporal windows with scheduled sampling and forward-backward consistency improves robustness against tracker drifting.
2 Related Work
Prior correspondence methods use synthetic image transformations, while optical-flow approaches commonly rely on brightness constancy and spatial smoothness. This paper instead exploits temporal coherence in videos for self-supervised correspondence learning.
- Correspondence Matching: Rocco et al. [35] [36] learn semantic correspondences from artificial transformations and use a differentiable soft inlier score to assess spatial-feature alignment.
- Optical Flow: Conventional and learned optical-flow methods establish or refine matches using brightness constancy, spatial smoothness, and feature embeddings.
- Correspondence Matching: The paper differs by learning correspondence matching from temporal coherence in videos rather than relying on synthetic transformations.
3 Approach
The approach learns pixelwise correspondence embeddings by reconstructing target frames from reference-frame colours, while information bottlenecks, restricted attention, and recursive training improve robustness over long temporal windows.
- 3 Approach: A shared embedding network matches pixels by computing affinities between frame features and softly copying reference-frame colours to reconstruct the target.The affinity matrix is formed from feature embeddings, and the soft-copy operation reconstructs the target by borrowing colours from the reference frame.
- 3.2 Feature Embedding with Information Bottleneck: Random channel dropout and brightness, contrast, and saturation perturbations form an information bottleneck that discourages reliance on low-level colour information.The model randomly zeros 0, 1, or 2 input channels and perturbs image appearance by up to 10%; full-colour images are used directly downstream.
- 3.3 Restricted Attention: Restricted attention limits each target pixel’s search to a local (2M+1)×(2M+1) reference patch, reducing the quadratic cost of full attention and enabling higher resolution.The resulting affinity volume has dimensions H × W × (2M + 1) × (2M + 1).
- 3.4 Long-term Correspondence Flow: Long-term correspondence training addresses easy or unreliable frame pairs, whose temporal spacing can make reconstruction trivial or invalid under deformation, illumination change, blur, and occlusion.The method combines scheduled sampling and forward-backward cycle consistency to improve robustness to tracker drifting.
- 3.4.1 Scheduled Sampling: Scheduled sampling recursively replaces ground-truth previous frames with predictions, annealing ground-truth use from 0.9 to 0.6 so the model learns to recover from errors.The recursive setup narrows the gap between training and inference and improves robustness to drifting.
- 3.5 Training Objectives: Training uses pixelwise cross-entropy losses for forward and backward reconstructions, weighting the forward path more heavily with α1 = 1.0 and α2 = 0.1.The colour of each pixel is quantized into 16 Lab-space classes using K-means before reconstruction is posed as classification.
4 Experiments and Analysis
Experiments evaluate self-supervised correspondence flow on DAVIS-2017 video segmentation and JHMDB keypoint tracking, including ablations, benchmark comparisons, attribute analyses, qualitative results, and additional-data training. The method outperforms prior approaches and improves further with more diverse self-supervised video data, while Kinetics limits coverage of non-human classes.
- Ablation Studies: Ablations evaluate color dropout, restricted attention, scheduled sampling, and cycle consistency on DAVIS-2017 without finetuning after training from scratch on Kinetics.The full-attention model uses about 9.2G GPU memory for a 480p image, compared with 1.4G for restricted attention; removing restricted attention reduces J and F by 6.9% and 11.6%.
- Comparison with State-of-the-art: 49.5 vs. 34.0 on J &F against video colorization and 49.5 vs. 40.7 against CycleTime, while approaching supervised ResNet50 performance at 49.5 vs. 49.7 on DAVIS-2017.The comparison reports higher values as better and attributes robustness under difficult conditions to the proposed training choices.
- Accuracy by Attributes: The proposed method outperforms previous methods across all DAVIS-2017 attribute categories, indicating robust correspondence features across varied conditions.The attribute analysis groups DAVIS-2017 testing accuracy by categories.
- Probing Upper Bound of Self-supervised Learning: Kinetics is not ideal for this evaluation because it is human-centric and most DAVIS classes, including cars and animals, are not covered.The additional-data experiment addresses this scope mismatch by adding more diverse videos.
- Probing Upper Bound of Self-supervised Learning: Increasing additional training videos improves both J and F for all sequences, with some classes comparable to or surpassing supervised learning.The additional videos are collected from YouTube using class labels without segmentation annotations during further self-supervised training.
- Keypoint Tracking on JHMDB: 11.3% average improvement in PCKinstance over previous methods [40], with better performance on the stricter PCK@.1 metric than [43].CycleTime and the proposed method are also reported as comparable to or better than supervised models when leveraging abundant video data.
5 Conclusion
The paper presents self-supervised pixel-level correspondence matching for videos using an information bottleneck and recursive training with scheduled sampling and cycle consistency. It achieves state-of-the-art video segmentation and keypoint tracking, while broader video data further improves segmentation.
- The method achieves state-of-the-art performance on video segmentation and keypoint tracking.
- An information bottleneck enables self-supervised training on standard RGB images while reducing trivial correspondence solutions.
- Recursive training with scheduled sampling and forward-backward cycle consistency addresses tracker drifting.
- Training on additional, more diverse video datasets improves video segmentation and indicates that self-supervised correspondence flow remains unsaturated.
Appendix A Network Architecture
The network uses a modified ResNet-18 with enlarged output feature maps. The appendix defines residual blocks as sequences of convolution, batch normalization, ReLU, convolution, and batch normalization.
- The architecture uses a modified ResNet-18 with enlarged output feature maps.
- A residual block consists of convolution, batch normalization, ReLU, convolution, and batch normalization with residual connections.
A.1 Failure Cases
Mask propagation fails when occlusion or leaving the frame prevents retrieval from the previous mask, and complex deformation can cause tracker drifting.
- Occlusions prevent the tracker from retrieving an object because propagation uses only the previous frame's mask.
- Objects that leave the frame are difficult to recover during mask propagation.
- Complex object deformation can cause the model to drift during tracking.