Source-linked AI summary
Joint-task Self-supervised Learning for Temporal Correspondence
Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, Ming-Hsuan Yang
TL;DR
Video correspondence learning lacks a broadly annotated way to connect region-level tracking with fine-grained pixel matching. The paper jointly learns both tasks through a shared inter-frame affinity matrix and self-supervised video signals, outperforming prior self-supervised methods across correspondence tasks and surpassing a supervised ResNet-18 affinity representation.
Problem
Most methods address either region-level or pixel-level correspondence, while datasets annotated for both tasks are scarce.
Method
The method jointly learns region localization and fine-grained matching through a shared inter-frame affinity matrix and video-based self-supervision.
Results
The method outperforms state-of-the-art self-supervised methods across video segmentation, keypoint tracking, and object tracking, and surpasses a supervised ImageNet-pretrained ResNet-18 affinity representation.
Takeaways & Limitations
Jointly modeling regions and pixels enables a self-supervised correspondence representation that supports accurate correspondences at different levels across video frames.
Takeaways & Limitations
The affinity formulation assumes paired frames contain the same contents, so newly appearing scene or object pixels can produce wrong matches.
Abstract
from arXiv · showhide
This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames. We exploit the synergy between both tasks through a shared inter-frame affinity matrix, which simultaneously models transitions between video frames at both the region- and pixel-levels. While region-level localization helps reduce ambiguities in fine-grained matching by narrowing down search regions; fine-grained matching provides bottom-up features to facilitate region-level localization. Our method outperforms the state-of-the-art self-supervised methods on a variety of visual correspondence tasks, including video-object and part-segmentation propagation, keypoint tracking, and object tracking. Our self-supervised method even surpasses the fully-supervised affinity feature representation obtained from a ResNet-18 pre-trained on the ImageNet.
1 Introduction
The paper addresses the limited integration of region-level and pixel-level video correspondence by jointly learning both through one inter-frame affinity matrix. Self-supervised temporal signals support this shared transformation, which improves localization and fine-grained matching across visual correspondence tasks.
- Visual correspondence connects multi-view representations and links static images to dynamic scenes, supporting tasks such as tracking and pixel matching.
- Most existing methods solve either region-level correspondence or pixel-level correspondence, because the tasks optimize different goals and typically require different annotations.
- The proposed framework jointly models regions and pixels through one inter-frame affinity matrix, treating both tasks as learning a shared transformation between frames.
- Temporal coherency, cycle consistency, and energy preservation provide task-agnostic self-supervision that exists naturally in videos.
- Region localization narrows the search space for fine-grained matching, while fine-grained features improve the affinity matrix and thereby localization.
- The method outperforms state-of-the-art methods across video segmentation, keypoint tracking, and object tracking, and surpasses a supervised ImageNet-pretrained ResNet-18 affinity representation.
2 Related Work
Prior correspondence work largely separates object-level tracking from fine-grained pixel matching, while self-supervised methods likewise develop along these two directions. The paper positions its approach as a joint self-supervised alternative.
- Self-supervised learning: The proposed method builds on self-supervised correspondence learning while connecting the previously separate region-level and fine-grained directions.
- Object-level correspondence: Visual tracking determines a bounding box in each frame from an annotated reference box and commonly uses tracking-by-detection or tracking-by-matching.
- Fine-grained correspondence: Fine-grained correspondence tracks individual pixels for optical flow and motion estimation, with many deep models trained on synthetic ground-truth flow.
- Fine-grained correspondence: Direct regression of pixel offsets has limited capability under dramatic appearance changes and can suffer domain shift in real-world scenarios.
- Self-supervised learning: Self-supervised correspondence methods use signals such as image or color transformation and cycle consistency for region-level or pixel-level matching.
3 Approach
The approach learns an inter-frame affinity matrix that jointly supports region localization and fine-grained matching. Self-supervised constraints exploit temporal coherence, color reconstruction, concentration, cycle consistency, and feature-energy consistency.
- Shared affinity: The inter-frame affinity matrix approximates pixel transitions between temporally coherent video frames and learns feature embeddings that associate their contents.Each affinity entry measures similarity between feature-map locations in the two frames.
- Self-supervision: Color reconstruction provides self-supervision by using grayscale-derived affinities to transform color channels between frames.The method uses ground-truth color as the supervisory signal.
- Region-level localization: Region localization samples a reference patch, finds matching parts in the target frame, and estimates its center and scale from transformed pixel locations.The estimated target region is represented by a bounding box whose center is the average transformed coordinate; scale uses average absolute distances from that center.
- Region-level localization: Concentration regularization encourages pixels from a reference patch to move together, reducing dispersal to other objects during region tracking.The method assumes that pixels within a region typically move as one unit in videos.
- Fine-grained matching: Fine-grained matching reuses the localization affinity to reconstruct target-patch color features, with an auto-encoder producing Lab-space representations.The color feature reduces errors relative to reconstructing color directly in image space.
- Fine-grained matching: Cycle consistency encourages pixels to return to their original locations after forward and backward transformations, equivalent to orthogonal affinity regularization.The same cycle-consistency idea is applied to both locations and features; feature energy is also regularized through Gram-matrix consistency.
4 Experiments
Experiments evaluate the self-supervised correspondence model across segmentation, keypoint propagation, tracking, and ablation settings. Results show strong performance against self-supervised methods and supervised ResNet-18 features, while localization and regularization improve correspondence quality.
- Instance segmentation: 13.3% higher J and 16.6% higher F than Wang et al. [51] are reported for instance segmentation propagation.The method is also 6.9% higher in J and 4.1% higher in F than ImageNet-supervised ResNet-18.
- Ablation studies: Removing localization lowers J by 2.5% and F by 0.9%, while removing concentration lowers J by 8%.Localization preserves local spatial relationships, and concentration encourages neighborhood locality during transformation.
- Ablation studies: Orthogonal regularization improves preservation of local structures by enforcing cycle-consistent forward and backward point transformations.The ablation removes this regularization and reports less effective preservation of local structures.
- Cross-task results: The method outperforms self-supervised methods and ImageNet-supervised ResNet-18 on J-HMDB keypoint propagation and VIP human-part segmentation.J-HMDB evaluates PCK for 15 human pose keypoints, while VIP evaluates semantic and instance-level part propagation.
- Cross-task results: Texture and object propagation results indicate that the learned features support precise correspondence beyond dense matching tasks.The model is integrated without fine-tuning into a correlation-filter tracking framework for object matching.
5 Conclusions
The paper learns video-frame correspondences self-supervisedly by jointly modeling region-level and pixel-level matching through a shared inter-frame affinity matrix. Experiments show effectiveness against self-supervised methods and ImageNet-supervised ResNet-18 features.
- Conclusion: The method jointly learns region-level and pixel-level correspondences through a shared inter-frame affinity matrix.The two correspondence levels facilitate each other during learning.
- Conclusion: Experimental results demonstrate effectiveness against state-of-the-art self-supervised methods and ImageNet-supervised ResNet-18 models.
A Implementation
The implementation uses Adam optimization with separate learning rates for warm-up and joint training, a fixed softmax temperature, and k-NN propagation with k = 5.
- Optimization: Adam uses a learning rate of 10^-4 during warm-up and 0.5×10^-4 during joint localization and matching training.
- Propagation settings: The softmax temperature is set to 1, and k-NN propagation uses k = 5 for all tasks.
B Texture Propagation
Texture propagation overlays a texture map on the first-frame object and propagates it through subsequent frames. The model preserves the texture, indicating precise inter-frame correspondences.
- Procedure: A texture map overlaid on the object in the first frame is propagated across the remaining video frames.
- Result: The model preserves the texture well during propagation, indicating precise correspondences between video frames.
C Instance Segmentation Propagation on DAVIS-2017
On DAVIS-2017, the method propagates instance masks while handling rapid object shape and scale changes. Inference localizes propagated points and transfers the reference mask between estimated bounding boxes.
- Results: The model remains resilient to rapid object shape and scale changes in DAVIS-2017 instance mask propagation.Examples include the horse, motorbike, and cart.
- Results: Figure 8 presents additional instance mask propagation results on the DAVIS-2017 dataset.The section also reports comparisons with state-of-the-art methods in Table 6.
- Inference procedure: During inference, each point from the reference mask is propagated by first localizing a bounding box on it in the target frame.The localization module is visualized in Figure 9.
- Inference procedure: The method refines the propagated center with mean-shift and estimates scale before transferring the reference mask between target and reference bounding boxes.Green points represent individually propagated points, while the red box indicates the estimated target bounding box.