PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Cycle-Consistency Learning

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

arXiv:1904.07846v1cs.CVcs.LG

TL;DR

Fine-grained temporal reasoning is less studied than action-category recognition, and naturally occurring videos often lack frame-by-frame correspondences. The paper introduces self-supervised temporal cycle-consistency learning to align videos of the same action; with one labeled video, TCC achieves similar performance to supervised models trained with about 50 videos.

  • Problem

    Temporal reasoning is less studied than action-category recognition, and most real-world sequences lack natural frame-by-frame alignment.

    Source: §1 · pp. 1–2

  • Method

    TCC learns representations by aligning video sequences of the same action using differentiable cycle-consistency losses.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    With only one labeled video, TCC achieves similar performance to supervised learning models trained with about 50 videos.

    Source: §7 · p. 8

  • Takeaways & Limitations

    TCC also serves as a general-purpose temporal alignment method that works without labels and supports tasks such as annotation transfer.

    Source: §7 · p. 8

    Kendall’s tau assumes videos contain no repetitive frames, a potential caveat for slow actions or periodic motion, though the authors say it is not problematic for their datasets.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using the nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .

1. Introduction

Temporal cycle consistency (TCC) learns video representations by aligning sequences of the same action without explicit frame correspondences, and the resulting features support fine-grained temporal understanding.

1. Introduction

  • Cross-video correspondences can persist despite viewpoint, scale, container, and speed variation, linking raw sequences to high-level action phases.For pouring, examples include first touching the container and lifting it from the ground.Source: §1 · p. 1
  • Fine-grained temporal reasoning is less studied than action-category recognition [10] [42], while per-frame supervision is costly and the exhaustive label set is unclear.The authors motivate self-supervised correspondence learning as an alternative to collecting per-frame annotations.Source: §1 · pp. 1–2
  • TCC learns representations by aligning same-action video sequences through cycle-consistent, one-to-one frame mappings without requiring explicit frame correspondences.The method introduces two differentiable versions of cycle-consistency computation that can be optimized with gradient-based methods.Source: §1 · p. 2
  • The learned features support action-phase classification and continuous action-progress tracking, with performance gains when labeled data are scarce.The paper compares TCC representations with existing self-supervised video methods [27] and supervised learning.Source: §1 · p. 2
  • The work contributes per-frame annotations for Penn Action and Pouring to support evaluation of fine-grained video understanding.The authors state that these annotations will be released publicly.Source: §1 · p. 2

2. Related Work

Related work applies cycle consistency to image correspondences, video alignment, and cross-domain transformations, while other video methods use synchronized views or varied self-supervised signals. TCC instead learns a joint temporal embedding from sequences without given correspondences and uses differentiable cycle consistency as a representation-learning objective.

Cycle consistency

  • Prior cycle-consistency methods address image tasks [55], typically refining fixed features or using spatial constraints, whereas TCC learns representations for temporal correspondences.FlowWeb optimizes globally consistent dense image correspondences, while Zhou et al. use low-rank matrix recovery; Zhou et al. [55] rely on initial correspondences between 3D renderings.Source: §2 · p. 2
  • CycleGANs [57] and CyCADA [21] learn transformations across domains, while Aytar et al. [7] use nondifferentiable temporal cycle consistency for validation rather than representation learning.TCC instead learns a joint embedding space mapping frames across sequences without assuming given correspondences.Source: §2 · p. 2

Video alignment

  • Video-alignment methods use synchronization through CCA, ranking, or match classification [6], but TCN, circulant temporal encoding, and Sigurdsson et al. [37] assume corresponding events or views.These methods do not fit TCC's setting, which assumes no given correspondences between different videos.Source: §2 · p. 2

Action localization and parsing

  • Prior action-recognition, localization, parsing, and sub-activity methods [17, 38, 46, 50, 53, 24, 25, 29, 13, 33, 34, 36] rely heavily on existing deep-image [19] [39] or spatio-temporal features.[2] discovers key events unsupervised using a weak association between videos and text instructions.Source: §2 · p. 3

Soft nearest neighbours

  • Soft nearest neighbors, also used in few-shot metric learning [28] [31] [40], provide a component of TCC's differentiable cycle-consistency computation.TCC uses the soft formulation within its cycle-consistency loss.Source: §2 · p. 3

Self-supervised representations

  • Video self-supervision uses signals including temporal order [16] [27], viewpoint consistency, segment classification [22], cross-modal temporal distance [7], patch permutations [14], and visual similarity [32].Unlike methods optimizing samples independently, TCC jointly optimizes two sequences, can capture more embedding variation, and yields best results when combined with some existing unsupervised losses.Source: §2 · p. 3

3. Cycle Consistent Representation Learning

The method learns frame embeddings for temporal alignment by maximizing cycle consistency between video sequences, using differentiable losses that can be optimized with back-propagation.

3. Cycle Consistent Representation Learning

  • The core objective is to maximize one-to-one frame correspondences between similar video sequences by maximizing their cycle-consistent frames.The paper introduces two differentiable cycle-consistency losses to enable this objective to be learned with back-propagation.Source: §3 · p. 3

3.1. Cycle-consistency

  • A frame is cycle-consistent when nearest-neighbor matching from one sequence to another and back returns to the original frame.The embedding objective maximizes the number of such frames across a sequence pair.Source: §3.1 · p. 3

3.2. Cycle-back Classification

  • Cycle-back classification uses a soft nearest neighbor and classifies its return to the first sequence as the originating frame’s index.The logits are negative squared distances to frames in the first sequence, and the objective uses cross-entropy.Source: §3.2 · pp. 3–4

3.3. Cycle-back Regression

  • Cycle-back regression accounts for temporal proximity by fitting a Gaussian prior around the originating frame’s index and regularizing the return distribution’s variance.The objective minimizes normalized squared distance from the index to the distribution mean and log variance for numerical stability.Source: §3.3 · p. 4

3.4. Implementation details

  • Training minimizes cycle-consistency losses over randomly selected frames from every training sequence pair until convergence.Frames are resized to 224 × 224; the embedder aggregates frame and context-frame features with 3D convolutions and outputs 128-dimensional embeddings. The base features come from ResNet-50 [19] when using ImageNet pretraining or a VGG-M-style network [11] when training from scratch.Source: §3.4 · p. 4

4. Datasets and Evaluation

The study evaluates learned representations on Pouring and Penn Action videos using frame-level key-event and action-phase annotations. Evaluation measures phase classification, phase-progress prediction, and temporal alignment between unseen video sequences.

4. Datasets and Evaluation

  • Pouring and Penn Action videos are annotated with key events and phase labels, where each phase spans the interval between consecutive key events.Key events reduce the number of frames requiring annotation and can be less ambiguous than labeling arbitrary frames; the study uses all real Pouring videos and all but two Penn Action categories.Source: §4 · pp. 4–5, Figure 4 · p. 5

4.1. Evaluation

  • Evaluation freezes networks after training and fits SVM classifiers and linear regressors to their features, without fine-tuning.All three measures are computed on the validation set, and higher scores indicate better performance.Source: §4.1 · p. 5
  • Phase classification accuracy measures per-frame phase prediction using an SVM trained on the training-set phase labels.Source: §4.1 · p. 5
  • Phase progression evaluates whether a linear regressor can predict normalized temporal progress through an action, scored by average R-squared.Progress is approximated from each frame’s timestamps relative to key events and normalized by the video’s frame count; the maximum score is 1.Source: §4.1 · p. 5
  • Kendall’s Tau scores nearest-neighbor alignment across validation video pairs without requiring evaluation labels, with 1 indicating perfect alignment and -1 reverse-order alignment.It assumes videos have no repetitive frames; the authors state this is not a problem for their datasets.Source: §4.1 · p. 5

5. Experiments

Across experiments, TCC features support action-phase classification and fine-grained temporal tasks, while combining cycle consistency with TCN yields strong results. The experiments compare cycle-consistency losses, self-supervised baselines, and supervised learning under different labeling conditions.

5.1. Baselines

  • SaL predicts triplet order, TCN uses within-window anchor-positive n-pairs, and TCC+SaL and TCC+TCN combine weighted losses; all baselines share the same encoder.The combined-loss weights are selected from 0.25, 0.5, and 0.75.Source: §5.1 · p. 6

5.2. Ablation of Different Cycle Consistency Losses

  • The variance-aware cycle-back regression loss outperforms both other cycle-consistency losses on every metric and is used as the final TCC method.The ablation also evaluates an MSE version that ignores prediction variance.Source: Table 3 · p. 6, §5.2 · p. 6

5.3. Action Phase Classification

  • From scratch, TCC outperforms SaL and TCN on both datasets and boosts performance over supervised learning when labeled data is limited.The authors attribute TCC’s advantage to learning across multiple videos, whereas SaL and TCN operate on frames from a single video.Source: §5.3 · p. 6, Table 4 · p. 7
  • Fine-tuned TCC features are competitive on Penn Action and outperform other methods on Pouring, while TCC+TCN achieves the best performance on both datasets.Self-supervised fine-tuning also gives significant performance boosts over supervised features in the low-labeled-data regime.Source: Table 5 · p. 7, §5.3 · p. 7
  • With one labeled video, TCC and TCC+TCN match supervised learning with about 50 densely labeled videos in phase classification.Self-supervised features use the full training set of videos, while the supervised baseline uses only videos with labels.Source: Figure 5 · p. 7, §5.3 · p. 7

5.4. Phase Progression and Kendall’s Tau

  • TCC+TCN significantly outperforms supervised and self-supervised methods on phase progression and Kendall’s Tau for both datasets; from-scratch TCC also beats other features on both metrics.Kendall’s Tau measures sequence alignment using nearest-neighbor matching, and combined losses yield significantly higher Kendall’s Tau.Source: Table 6 · p. 7, §5.4 · p. 8

6. Applications

TCC alignments enable cross-video transfer and support fine-grained frame retrieval, anomaly detection, and synchronized playback.

Cross-modal transfer in Videos

  • Unsupervised alignment lets annotations transfer across related videos after labeling one video, while visual neighbors supply stitched sounds without postprocessing.The sound example transfers pouring-liquid audio by concatenating sounds from retrieved nearest neighbors.Source: §6 · p. 8

Fine-grained retrieval in Videos

  • Nearest-neighbor retrieval distinguishes fine-grained action phases, including whether a pitcher’s leg is up before or after a baseball pitch.Examples also retrieve pouring frames when a glass is half full or after the container is returned, when liquid has reached the target container.Source: §6 · p. 8

Anomaly detection

  • Distance from an ideal TCC embedding-space trajectory flags anomalous frames; in a bench-press example, distance spikes around frame 20 as activity diverges.Source: §6 · p. 8

Synchronous Playback

  • Learned alignments transfer one video’s pace to other videos of the same action for synchronous playback.Source: §6 · p. 8

7. Conclusion … B. Sound Transfer

TCC learns self-supervised features that help with temporally fine-grained tasks when labeled data are scarce, and also enables practical video alignment and synchronized-modality transfer.

7. Conclusion

  • With one labeled video, TCC achieves performance similar to supervised models trained with about 50 videos.The conclusion reports that self-supervised features significantly boost performance when labeled data are scarce.Source: §7 · p. 8
  • TCC is a general-purpose, label-free temporal alignment method that can support tasks such as annotation transfer.The paper argues that TCC is more than a proxy task for representation learning: tasks relying on alignment itself can benefit.Source: §7 · p. 8
  • The learned self-supervised features are useful for temporally fine-grained tasks, with performance gains when labeled data are scarce.Source: §7 · p. 8

A. Synchronous Playback

  • For synchronous playback, the method embeds all frames, selects a reference video, and uses dynamic time warping to match frames in each other video to its pace.The dynamic time warping step enforces per-frame temporal constraints; no other post-processing is used.Source: A. Synchronous Playback · p. 11
  • Self-supervised alignment reduces the effort of synchronizing multiple videos, a task that can otherwise require manual cuts and frame-rate changes.Source: A. Synchronous Playback · p. 11

B. Sound Transfer

  • Sound transfer copies sound blocks from nearest-neighbor frames in a video with sound and concatenates them for each target-video frame.This transfers a synchronized modality based on visual similarity.Source: B. Sound Transfer · p. 11
  • Averaging embeddings across multiple frames and concatenating their corresponding sounds reduces edge artifacts in synthesized video sound.The example’s sound changes as liquid flows into a container, which the paper presents as evidence that embeddings capture task progress; no other post-processing is used.Source: B. Sound Transfer · p. 11

C. t-SNE Visualization · D. Fine-grained Retrieval

The t-SNE visualizations show action trajectories and closer embedding locations for corresponding frames across videos, supporting alignment and nearest-neighbor fine-grained retrieval. Additional fine-grained retrieval results are presented in Figure 9.

C. t-SNE Visualization

D. Fine-grained Retrieval

E. Data Augmentation

Training uses a shared data-augmentation pipeline across all algorithms, applying random horizontal flips and brightness and contrast perturbations.

E. Data Augmentation

  • During training, all algorithms use the same augmentation pipeline: random whole-video horizontal flips, brightness offsets from −32 to 32 raw-pixel values, and contrast factors sampled uniformly from 0.5 to 1.5.Source: E. Data Augmentation · p. 11

F. Alignments under Different Losses · G. Hyperparameters

Video alignment emerges over training, and cycle-back regression produces a more ordered alignment than cycle-back classification. Table 7 lists the hyperparameters used.

F. Alignments under Different Losses

  • As training proceeds, nearest-neighbor matches become more aligned, with similarity brightening along the diagonal of the video-pair matrix.Brighter cells indicate greater similarity between frames from the two videos.Source: F. Alignments under Different Losses · p. 11
  • Cycle-back regression yields a more ordered alignment than cycle-back classification, which does not account for time when applying cycle consistency.Source: F. Alignments under Different Losses · p. 11

G. Hyperparameters

H. Architecture Details

The encoder combines a frame-level Base Network with an Embedder Network that uses each frame’s context to produce the vectors used by training algorithms. The architecture varies with the base network and uses a context window of k = 2 frames.

H. Architecture Details

  • The encoder has a Base Network for extracting convolutional features from individual frames and an Embedder Network that combines frame features with context.Table 8 describes the architectures and their parameter conventions, including downsampling choices for ResNet-50 and VGG-M models.Source: H. Architecture Details · p. 11, Table 8 · p. 14
  • The Embedder Network produces 128-dimensional vectors; the base network determines whether c4 is 1024 or 512, and all training algorithms operate on these vectors.Source: H. Architecture Details · p. 12
  • With k = 2, the frame gap is approximately 0.75 seconds (15 frames at 20 fps) for Penn Action and 0.3 seconds (9 frames at 30 fps) for Pouring.The authors report that this context size performed well on both datasets despite earlier experiments with larger k values.Source: H. Architecture Details · p. 12

Related papers

Loading 1904.07846v1…