PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint-task Self-supervised Learning for Temporal Correspondence

Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, Ming-Hsuan Yang

arXiv:1909.11895v1cs.CV

TL;DR

Video correspondence learning lacks a broadly annotated way to connect region-level tracking with fine-grained pixel matching. The paper jointly learns both tasks through a shared inter-frame affinity matrix and self-supervised video signals, outperforming prior self-supervised methods across correspondence tasks and surpassing a supervised ResNet-18 affinity representation.

  • Problem

    Most methods address either region-level or pixel-level correspondence, while datasets annotated for both tasks are scarce.

    Source: §1 · p. 1

  • Method

    The method jointly learns region localization and fine-grained matching through a shared inter-frame affinity matrix and video-based self-supervision.

    Source: §1 · pp. 1–2

  • Results

    The method outperforms state-of-the-art self-supervised methods across video segmentation, keypoint tracking, and object tracking, and surpasses a supervised ImageNet-pretrained ResNet-18 affinity representation.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Jointly modeling regions and pixels enables a self-supervised correspondence representation that supports accurate correspondences at different levels across video frames.

    Source: §5 · p. 8

    The affinity formulation assumes paired frames contain the same contents, so newly appearing scene or object pixels can produce wrong matches.

    Source: §3 · p. 3

Abstract

from arXiv · show

This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames. We exploit the synergy between both tasks through a shared inter-frame affinity matrix, which simultaneously models transitions between video frames at both the region- and pixel-levels. While region-level localization helps reduce ambiguities in fine-grained matching by narrowing down search regions; fine-grained matching provides bottom-up features to facilitate region-level localization. Our method outperforms the state-of-the-art self-supervised methods on a variety of visual correspondence tasks, including video-object and part-segmentation propagation, keypoint tracking, and object tracking. Our self-supervised method even surpasses the fully-supervised affinity feature representation obtained from a ResNet-18 pre-trained on the ImageNet.

1 Introduction

The paper addresses the limited integration of region-level and pixel-level video correspondence by jointly learning both through one inter-frame affinity matrix. Self-supervised temporal signals support this shared transformation, which improves localization and fine-grained matching across visual correspondence tasks.

  • Visual correspondence connects multi-view representations and links static images to dynamic scenes, supporting tasks such as tracking and pixel matching.Source: §1 · p. 1
  • Most existing methods solve either region-level correspondence or pixel-level correspondence, because the tasks optimize different goals and typically require different annotations.Source: §1 · p. 1
  • The proposed framework jointly models regions and pixels through one inter-frame affinity matrix, treating both tasks as learning a shared transformation between frames.Source: §1 · pp. 1–2
  • Temporal coherency, cycle consistency, and energy preservation provide task-agnostic self-supervision that exists naturally in videos.Source: §1 · pp. 1–2
  • Region localization narrows the search space for fine-grained matching, while fine-grained features improve the affinity matrix and thereby localization.Source: §1 · p. 2
  • The method outperforms state-of-the-art methods across video segmentation, keypoint tracking, and object tracking, and surpasses a supervised ImageNet-pretrained ResNet-18 affinity representation.Source: §1 · p. 2

2 Related Work

Prior correspondence work largely separates object-level tracking from fine-grained pixel matching, while self-supervised methods likewise develop along these two directions. The paper positions its approach as a joint self-supervised alternative.

Self-supervised learning

  • The proposed method builds on self-supervised correspondence learning while connecting the previously separate region-level and fine-grained directions.Source: §2 · pp. 2–3

Object-level correspondence

  • Visual tracking determines a bounding box in each frame from an annotated reference box and commonly uses tracking-by-detection or tracking-by-matching.Source: §2 · p. 2

Fine-grained correspondence

  • Fine-grained correspondence tracks individual pixels for optical flow and motion estimation, with many deep models trained on synthetic ground-truth flow.Source: §2 · p. 2
  • Direct regression of pixel offsets has limited capability under dramatic appearance changes and can suffer domain shift in real-world scenarios.Source: §2 · p. 3

Self-supervised learning

  • Self-supervised correspondence methods use signals such as image or color transformation and cycle consistency for region-level or pixel-level matching.Source: §2 · p. 3

3 Approach

The approach learns an inter-frame affinity matrix that jointly supports region localization and fine-grained matching. Self-supervised constraints exploit temporal coherence, color reconstruction, concentration, cycle consistency, and feature-energy consistency.

Shared affinity

  • The inter-frame affinity matrix approximates pixel transitions between temporally coherent video frames and learns feature embeddings that associate their contents.Each affinity entry measures similarity between feature-map locations in the two frames.Source: §3 · p. 3

Self-supervision

  • Color reconstruction provides self-supervision by using grayscale-derived affinities to transform color channels between frames.The method uses ground-truth color as the supervisory signal.Source: §3 · p. 3

Region-level localization

  • Region localization samples a reference patch, finds matching parts in the target frame, and estimates its center and scale from transformed pixel locations.The estimated target region is represented by a bounding box whose center is the average transformed coordinate; scale uses average absolute distances from that center.Source: §3 · pp. 3–4, §3.2 · pp. 4–5
  • Concentration regularization encourages pixels from a reference patch to move together, reducing dispersal to other objects during region tracking.The method assumes that pixels within a region typically move as one unit in videos.Source: §3.2 · p. 5

Fine-grained matching

  • Fine-grained matching reuses the localization affinity to reconstruct target-patch color features, with an auto-encoder producing Lab-space representations.The color feature reduces errors relative to reconstructing color directly in image space.Source: §3.3 · p. 5
  • Cycle consistency encourages pixels to return to their original locations after forward and backward transformations, equivalent to orthogonal affinity regularization.The same cycle-consistency idea is applied to both locations and features; feature energy is also regularized through Gram-matrix consistency.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate the self-supervised correspondence model across segmentation, keypoint propagation, tracking, and ablation settings. Results show strong performance against self-supervised methods and supervised ResNet-18 features, while localization and regularization improve correspondence quality.

Instance segmentation

  • 13.3% higher J and 16.6% higher F than Wang et al. [51] are reported for instance segmentation propagation.The method is also 6.9% higher in J and 4.1% higher in F than ImageNet-supervised ResNet-18.Source: §4.2 · p. 7

Ablation studies

  • Removing localization lowers J by 2.5% and F by 0.9%, while removing concentration lowers J by 8%.Localization preserves local spatial relationships, and concentration encourages neighborhood locality during transformation.Source: §4.3 · pp. 7–8
  • Orthogonal regularization improves preservation of local structures by enforcing cycle-consistent forward and backward point transformations.The ablation removes this regularization and reports less effective preservation of local structures.Source: §4.3 · p. 8

Cross-task results

  • The method outperforms self-supervised methods and ImageNet-supervised ResNet-18 on J-HMDB keypoint propagation and VIP human-part segmentation.J-HMDB evaluates PCK for 15 human pose keypoints, while VIP evaluates semantic and instance-level part propagation.Source: §4.4 · p. 8, §4.6 · p. 8
  • Texture and object propagation results indicate that the learned features support precise correspondence beyond dense matching tasks.The model is integrated without fine-tuning into a correlation-filter tracking framework for object matching.Source: §4.5 · p. 8

5 Conclusions

The paper learns video-frame correspondences self-supervisedly by jointly modeling region-level and pixel-level matching through a shared inter-frame affinity matrix. Experiments show effectiveness against self-supervised methods and ImageNet-supervised ResNet-18 features.

Conclusion

  • The method jointly learns region-level and pixel-level correspondences through a shared inter-frame affinity matrix.The two correspondence levels facilitate each other during learning.Source: §5 · p. 8
  • Experimental results demonstrate effectiveness against state-of-the-art self-supervised methods and ImageNet-supervised ResNet-18 models.Source: §5 · p. 8

A Implementation

The implementation uses Adam optimization with separate learning rates for warm-up and joint training, a fixed softmax temperature, and k-NN propagation with k = 5.

Optimization

  • Adam uses a learning rate of 10^-4 during warm-up and 0.5×10^-4 during joint localization and matching training.Source: A Implementation · p. 11

Propagation settings

B Texture Propagation

Texture propagation overlays a texture map on the first-frame object and propagates it through subsequent frames. The model preserves the texture, indicating precise inter-frame correspondences.

Procedure

Result

  • The model preserves the texture well during propagation, indicating precise correspondences between video frames.Source: B Texture Propagation · p. 11

C Instance Segmentation Propagation on DAVIS-2017

On DAVIS-2017, the method propagates instance masks while handling rapid object shape and scale changes. Inference localizes propagated points and transfers the reference mask between estimated bounding boxes.

Results

Inference procedure

  • During inference, each point from the reference mask is propagated by first localizing a bounding box on it in the target frame.The localization module is visualized in Figure 9.Source: C Instance Segmentation Propagation on… · p. 11
  • The method refines the propagated center with mean-shift and estimates scale before transferring the reference mask between target and reference bounding boxes.Green points represent individually propagated points, while the red box indicates the estimated target bounding box.Source: C Instance Segmentation Propagation on… · p. 11

Related papers

Loading 1909.11895v1…