PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Cycle-Consistency Learning

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

arXiv:1904.07846v1cs.CVcs.LG

TL;DR

Fine-grained temporal understanding is difficult to supervise because it requires extensive per-frame labels, while natural videos often lack explicit frame correspondences. TCC addresses this by learning differentiable temporal cycle-consistent embeddings across videos, which support temporal alignment and few-shot action-phase classification. With one labeled video, TCC achieves similar performance to supervised models trained with about 50 videos, and the learned alignment also supports tasks such as annotation transfer.

  • Problem

    Fine-grained temporal understanding requires per-frame supervision, but annotations are costly and natural video sequences usually lack explicit frame-by-frame correspondences.

  • Method

    TCC is a self-supervised method that learns embedding spaces by aligning video sequences of the same action with differentiable cycle-consistency losses.

  • Results

    With one labeled video, TCC achieves similar performance to supervised learning models trained with about 50 videos.

  • Takeaways & Limitations

    TCC representations support fine-grained temporal tasks and general-purpose temporal alignment without labels, including annotation transfer.

  • Takeaways & Limitations

    Kendall’s Tau assumes videos contain no repetitive frames, although this limitation is not problematic for the considered datasets.

Abstract

from arXiv · show

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using the nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .

1 Google Brain 2 DeepMind

The paper presents TCC learning, a self-supervised representation method inspired by temporal video alignment and intended for fine-grained temporal understanding.

  • TCC learning is a self-supervised representation technique inspired by temporal video alignment.Temporal video alignment finds correspondences across multiple videos despite variation.
  • The learned representations support fine-grained temporal understanding in videos.
  • Multiple videos can be aligned by matching nearest-neighbor frames in the learned embedding space.

1. Introduction

Fine-grained temporal understanding requires recognizing stages and correspondences in sequential video, but per-frame supervision is costly and incomplete. The paper therefore learns temporal correspondences self-supervised across videos and evaluates the resulting representations on temporal understanding tasks.

  • Videos contain correspondences across instances of the same process despite changes in viewpoint, scale, object style, and event speed.
  • Temporal reasoning concerns multiple process stages and their causal relations, but is less studied than action-category recognition.
  • Per-frame annotation is labor-intensive, and the exhaustive label set needed for fine-grained understanding is unclear.
  • The method learns correspondences across videos without explicit frame-by-frame alignment or supervision.
  • TCC representations are evaluated for action phase classification and continuous action-progress tracking.
  • The paper releases per-frame annotations for the Penn Action and Pouring datasets to support fine-grained video-understanding evaluation.

2. Related Work

The paper builds on cycle-consistency methods for visual correspondence and self-supervised video representation learning, while targeting temporal alignment and learning representations from raw video sequences.

  • Cycle consistency: Prior cycle-consistency methods address co-segmentation, structure from motion, and image matching.
  • Cycle consistency: Those methods generally enforce consistency on fixed feature representations, whereas TCC learns embedding spaces optimized for correspondences.
  • Cycle consistency: TCC addresses temporal rather than spatial correspondence and does not assume given correspondences.
  • Cycle-consistent representation learning: The TCC objective seeks an embedding space where similar video sequences have many one-to-one, cycle-consistent frame mappings.
  • Self-supervised representations: Soft nearest neighbors provide a differentiable component for TCC’s cycle-consistency computation.
  • Self-supervised representations: Unlike methods operating on frames from a single video or relying on existing deep features, TCC learns from multiple videos and raw sequences.

3. Cycle Consistent Representation Learning

TCC learns a shared embedding space in which similar video sequences can be temporally aligned through differentiable cycle-consistency objectives. It combines soft nearest-neighbor matching with cycle-back classification or regression to make the learned correspondences trainable by back-propagation.

  • Core idea: TCC learns an embedding space that aligns similar video sequences by maximizing cycle-consistent frame correspondences.The method seeks one-to-one mappings between sequences while avoiding the nondifferentiability of ordinary cycle-consistency computation.
  • Core idea: Each frame is encoded by φ into a per-frame embedding, producing sequences U and V for two videos.For sequences S and T, embeddings are defined as ui = φ(si; θ) and vi = φ(ti; θ).
  • Cycle-back Classification: Cycle-back classification treats each frame in U as a class and predicts whether soft nearest-neighbor matching from U to V returns to the correct frame.The return prediction uses logits based on distances between the soft neighbor ev and frames uk in U, followed by cross-entropy optimization.
  • Cycle-back Regression: Cycle-back regression incorporates temporal proximity by encouraging the return distribution β to concentrate around the original index i.A Gaussian prior and variance regularization penalize distant or diffuse cycle-back distributions.
  • Optimization: The resulting cycle-consistency formulations are differentiable and can be optimized with conventional back-propagation.Training minimizes cycle-consistency losses over randomly selected frames from sequence pairs until convergence.

4. Datasets and Evaluation

The study evaluates learned video representations on densely annotated Pouring and Penn Action data using frozen-feature probes for fine-grained temporal understanding. Evaluation covers phase recognition, continuous phase progression, and label-free temporal alignment, with Kendall’s Tau providing an alignment measure and an explicit repetitive-frame caveat.

  • Datasets: Pouring and Penn Action provide videos of humans performing actions, enabling dense temporal alignment evaluation across object-focused and sports-focused activities.The authors add key-event and phase annotations to video frames for evaluation.
  • Annotations: Key events reduce annotation effort, while every frame between consecutive events receives a phase label.Example phases include “Lifting bottle,” bounded by events such as “Hand touches bottle.”
  • Evaluation: Evaluation freezes the networks and trains SVM classifiers or linear regressors on their features, with higher scores indicating better performance.The three measures assess fine-grained temporal understanding on the validation set without additional network fine-tuning.
  • Evaluation: Phase classification accuracy measures per-frame phase prediction using an SVM trained on frame-level phase labels.The classifier is trained from the training-data phase labels and evaluated on validation features.
  • Evaluation: Phase progression measures how well embeddings capture action progress by predicting normalized distances from frames to key events with a linear regressor.The reported score is the average R-squared, whose maximum value is 1.
  • Evaluation: Kendall’s Tau evaluates temporal alignment without additional labels by comparing the ordering of nearest-neighbor frame matches across video pairs.A value of 1 indicates perfect alignment, while -1 indicates reverse-order alignment.

5. Experiments

Experiments show that TCC learns useful temporal representations from video, outperforming comparison methods across classification and fine-grained temporal tasks, especially with limited labels.

  • 5.2. Ablation of Different Cycle Consistency Losses: The variance-aware cycle-back regression loss outperforms both alternative losses across all reported metrics and defines the final TCC method.The authors use this version for the remaining experiments.
  • 5.3. Action Phase Classification: TCC features outperform Shuffle and Learn and Time-Contrastive Networks when learning visual representations from scratch on both Pouring and Penn Action.The authors attribute this partly to TCC learning across multiple videos during training.
  • 5.3. Action Phase Classification: TCC provides significant performance boosts over plain supervised learning when labeled data are limited.This pattern is reported for both Pouring and Penn Action in the from-scratch setting.
  • 5.3. Action Phase Classification: Combining cycle-consistency with TCN achieves the best performance in the self-supervised fine-tuning experiments.TCC is competitive on Penn Action and outperforms the other methods on Pouring.
  • 5.3. Action Phase Classification: With one labeled video, TCC and TCC+TCN reach the performance achieved by supervised learning with about 50 densely labeled videos.The few-shot comparison is reported for Golf Swing and Tennis Serve phase classification.
  • 5.5. Additional Applications: Nearest neighbors in the TCC embedding space support fine-grained retrieval and anomaly detection through distances from typical action trajectories.The supplied figure descriptions identify retrieval and trajectory-distance spikes during anomalous activity.
  • 5.4. Phase Progression and Kendall’s Tau: TCC features perform better on phase progression and Kendall’s Tau for both datasets when training from scratch.TCC+TCN significantly outperforms supervised and self-supervised methods on both fine-grained tasks.

6. Applications

TCC aligns related videos without supervision, enabling transfer of annotations and time-associated modalities between videos and fine-grained retrieval of action states.

  • Cross-modal transfer in Videos: Alignment across related videos enables transferring annotations or other time-associated modalities from one video to another.The paper describes transferring text annotations and sound using retrieved nearest neighbors.
  • Cross-modal transfer in Videos: Labeling one video can support annotation transfer across an entire dataset of related videos.The transferred modality may include sound, which is copied from retrieved nearest neighbors and concatenated without further postprocessing.
  • Fine-grained retrieval in Videos: Nearest neighbors support fine-grained retrieval of frames corresponding to distinct action states.Examples distinguish pouring states and baseball-pitch phases before and after the ball is pitched.

7. Conclusion

The approach learns temporally fine-grained features with limited labels and also supports general-purpose video alignment for synchronization and metadata transfer.

  • With one labeled video, TCC achieves performance similar to supervised models trained with about 50 videos.
  • TCC learns features useful for temporally fine-grained tasks and improves performance when labeled data are scarce.
  • Video alignment can synchronize multiple videos to a reference video's pace, reducing manual cuts and frame-rate changes.
  • Nearest-neighbor visual matching transfers synchronized sound or other frame-aligned metadata between videos.

C. t-SNE Visualization

The t-SNE visualizations show action trajectories and indicate that corresponding frames from different videos occupy nearby locations in the learned embedding space.

  • For each action, trajectories of four videos are shown with video-specific border colors and sampled frames linked to embedding locations.
  • Corresponding frames from different validation videos are closer in the embedding space than non-corresponding frames.
  • The learned structure supports video alignment and fine-grained retrieval using nearest neighbors.

E. Data Augmentation

Training uses shared video augmentations, while Figure 10 tracks how similarity-based alignment develops under different losses.

  • E. Data Augmentation: All training algorithms use the same augmentation pipeline: horizontal flips, brightness perturbations from −32 to 32, and contrast factors from 0.5 to 1.5.
  • F. Alignments under Different Losses: As training proceeds, brighter similarity cells emerge along the diagonal, indicating increasingly ordered alignment between two videos.
  • F. Alignments under Different Losses: Cycle-back regression produces more ordered alignment than cycle-back classification because classification does not account for time in its consistency loss.
  • E. Data Augmentation: Table 7 lists the hyperparameters used in the experiments.

H. Architecture Details

The architecture produces per-frame embeddings from convolutional features and context, while the supplied figures illustrate embedding structure, retrieval, and alignment.

  • H. Architecture Details: The encoder combines a Base Network for frame-level convolutional features with an Embedder Network that uses each frame and its context window.
  • H. Architecture Details: Figure 8 presents a t-SNE visualization of the embeddings.
  • H. Architecture Details: The Embedder Network maps frame features into a single 128-dimensional vector used by the training algorithms.
  • H. Architecture Details: Figure 9 provides fine-grained retrieval results using TCC embeddings.
  • H. Architecture Details: Figure 10 compares similarity-matrix evolution under cycle-back regression and classification losses.
  • H. Architecture Details: Table 8 describes the architectures used in the experiments and the notation for convolutional and fully connected layers.
Loading 1904.07846v1…