PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-augmented Dense Predictive Coding for Video Representation Learning

Tengda Han, Weidi Xie, Andrew Zisserman

arXiv:2008.01065v1cs.CV

TL;DR

The paper addresses the challenge of learning useful representations from video alone when future frames are indeterministic and expose only one possible outcome. It introduces MemDPC, which uses compressed memory and predictive attention to form multiple future hypotheses, and reports state-of-the-art or competitive results across four downstream tasks with orders of magnitude less training data.

  • Problem

    Video-only self-supervised learning lags behind multimodal video learning, while future prediction must handle indeterministic details and multiple possible futures.

  • Method

    MemDPC augments contrastive predictive coding with Compressive Memory and predictive attention over compressed memories to construct multiple future hypotheses.

  • Results

    Across action recognition, video retrieval, scarce-annotation learning, and unintentional action classification, MemDPC achieves state-of-the-art or competitive performance using orders of magnitude less training data.

  • Takeaways & Limitations

    MemDPC supports visual-only representation learning from RGB frames, unsupervised optical flow, or both, and can be combined with two-stream networks and bidirectional RNNs.

Abstract

from arXiv · show

The objective of this paper is self-supervised learning from video, in particular for representations for action recognition. We make the following contributions: (i) We propose a new architecture and learning framework Memory-augmented Dense Predictive Coding (MemDPC) for the task. It is trained with a predictive attention mechanism over the set of compressed memories, such that any future states can always be constructed by a convex combination of the condense representations, allowing to make multiple hypotheses efficiently. (ii) We investigate visual-only self-supervised video representation learning from RGB frames, or from unsupervised optical flow, or both. (iii) We thoroughly evaluate the quality of learnt representation on four different downstream tasks: action recognition, video retrieval, learning with scarce annotations, and unintentional action classification. In all cases, we demonstrate state-of-the-art or comparable performance over other approaches with orders of magnitude fewer training data.

1 Introduction

Video-only self-supervised representation learning aims to exploit temporal information, but future prediction must handle indeterministic details and multiple possible futures. The paper addresses this with MemDPC, evaluates RGB, optical flow, and combined inputs, and tests representations across several downstream settings.

  • Video-only self-supervised learning has lagged behind multimodal approaches despite strong progress in self-supervised image representation learning.
  • Temporal information in videos provides augmentation and can help disambiguate actions such as opening versus closing.
  • Exact future prediction is indeterministic, requiring training schemes that handle unpredictable details, multiple hypotheses, and only one observed future per video.
  • Earlier approaches either select the hypothesis closest to the true observation or use contrastive learning to distinguish one future state from distractors.
  • MemDPC adds a Compressive Memory and predictive attention over compressed memories, enabling efficient construction of multiple future hypotheses.
  • The learned representations are evaluated with RGB, unsupervised optical flow, or both across action recognition, scarce-annotation learning, video retrieval, and unintentional action classification.

2 Related Work

Related work develops self-supervised video representations by exploiting spatio-temporal information, with contrastive learning forming an important line of research. Memory models provide either implicit recurrent memory or explicit attention-based memory for feed-forward models.

  • Self-supervised image learning commonly uses contrastive learning to distinguish positive and negative sample pairs.
  • Self-supervised video research exploits spatio-temporal information through multiple approaches, including visual-audio, video-narration, and future-state prediction signals.
  • Memory-model research includes recurrent internal memory, such as LSTM and GRU, and explicit memory accessed through attention-based reading or writing.

3 Methodology

MemDPC learns video representations by predicting future dense features through a shared compressive memory, enabling multiple future hypotheses. Its contrastive objective operates over spatial and temporal feature locations, while extensions support optical-flow inputs and bidirectional prediction.

  • Memory-augmented Dense Predictive Coding: MemDPC augments contrastive predictive coding with a compressive memory that represents multiple possible futures.The memory is used during self-supervised training, while the c_t embedding is used for downstream tasks.
  • Video Block Encoder and Temporal Aggregation: Video blocks are encoded into z_i features, then temporally aggregated into a context feature c_t.The paper uses recurrent neural networks for temporal aggregation, although other autoregressive models are feasible.
  • Compressive Memory: A predictive addressing mechanism converts context into probabilities over memory entries and predicts future states as their expectation.The memory bank is shared across the dataset during training, and each entry acts as a potential hypothesis.
  • Multiple Hypotheses: Predictive attention can assign probability to multiple memory slots, keeping the similarity objective high when the observed future matches different hypotheses.The memory differs from sample queues used to increase negative examples: it compresses potential hypotheses and supports sequential future prediction.
  • Contrastive Learning: Dense contrastive learning compares recursively predicted and observed future features across spatial and temporal positions.The critic is a dot product, and optimization favors aligned positive pairs from the same video over negatives.
  • Extensions: MemDPC can use RGB, optical-flow, or fused two-stream inputs and can predict in both temporal directions.The bidirectional variant shares the feature extractor, predictor, and memory bank, averaging forward and backward losses.

4 How to Evaluate Self-Supervised Learning?

The paper evaluates learned video representations through frozen-probe and end-to-end fine-tuning protocols across action classification, scarce-annotation learning, video retrieval, and unintentional-action classification. These tasks assess classification performance, data efficiency, representation geometry, and future-prediction discrepancy.

  • Evaluation protocols: Linear probes freeze the backbone and train only a linear head, whereas fine-tuning updates the entire network for the downstream task.For classification, the linear protocol trains a classifier on top of the frozen base network.
  • Action Classification: Action classification evaluates MemDPC with fixed-network linear or non-linear probes and with end-to-end fine-tuning.The fixed-network setting freezes the backbone f(.) and g(.).
  • Data Efficiency and Generalizability: Scarce-annotation evaluation fine-tunes MemDPC using 10%, 20%, or 50% of labeled training samples and reports accuracy on the same test set.This protocol examines data efficiency and generalizability under limited annotations.
  • Video Action Retrieval: Video action retrieval uses k-nearest neighbours without further training to test whether semantically similar actions are close in the learned high-dimensional space.Context features are extracted from video blocks and spatially pooled into vectors.
  • Unintentional Actions: Unintentional-action classification measures discrepancy between predicted and true features, using the resulting surprise level to identify mismatched or failing moments.The representation is evaluated both with a frozen network and after fine-tuning the entire network.

5 Experiments

Experiments evaluate MemDPC using visual-only inputs across action classification, data efficiency, video retrieval, and unintentional action classification. Across these settings, MemDPC improves representation quality, competes with or exceeds prior methods, and reduces annotation requirements.

  • Experimental setup: Self-supervised training uses unlabeled UCF101 and K400 videos, with representations evaluated on UCF101, HMDB51, and Oops.Experiments include linear, non-linear, and end-to-end fine-tuning protocols.
  • Action classification: 68.2 vs. 61.8, MemDPC initialization improves UCF101 fine-tuning accuracy over random initialization.The comparison uses identical experimental settings while varying one variable at a time.
  • Action classification: 84% classification accuracy is achieved after fusing RGB and optical-flow streams on UCF101.The ablation also reports gains from compressive memory, optical flow, and bidirectional aggregation.
  • Action classification: MemDPC surpasses previous self-supervised methods on UCF101 and HMDB51 with the same K400 visual-only pretraining data.When representations are frozen, non-linear probes outperform linear probes while remaining cheap to train.
  • Data efficiency: Less than 50% labelled data is required for MemDPC initialization to match a randomly initialized classifier trained with 100% labels, for both RGB and optical flow.MemDPC representations generalize better as labelled training samples are reduced.
  • Video retrieval: MemDPC achieves state-of-the-art retrieval on nearly all RGB metrics and significantly outperforms prior methods with optical-flow inputs.Retrieval uses self-supervised features without further fine-tuning and evaluates nearest-neighbour retrieval on UCF101 and HMDB51.
  • Unintentional action classification: MemDPC demonstrates state-of-the-art unintentional action classification, including performance above a fully supervised K700-pretrained model under fine-tuning.The model is trained on K400 and Oops videos with self-supervised learning before linear probing or end-to-end fine-tuning.

6 Conclusion

The paper presents MemDPC for self-supervised video representation learning and evaluates it across four downstream tasks. It reports state-of-the-art or competitive results using visual input alone and substantially less training data than other approaches.

  • Conclusion: MemDPC is a self-supervised video representation learning architecture built around a novel compressive memory.The framework targets representations for action recognition and handles multiple future hypotheses.
  • Conclusion: The representation is evaluated on action recognition, video retrieval, scarce-annotation learning, and unintentional action classification.These experiments assess the learned representation across four downstream tasks.
  • Conclusion: MemDPC achieves state-of-the-art or competitive performance against approaches using orders of magnitude more training data.The conclusion reports this pattern across the evaluated tasks.
  • Conclusion: The paper reports high-quality video representations learned from the visual stream alone, without additional audio or text streams.The visual inputs include RGB and optical flow settings described in the paper context.

A Architectures in detail

The architecture combines a 2D+3D ResNet encoder with a temporal ConvGRU aggregator. The detailed design processes eight five-frame video blocks and aggregates feature maps over past time steps.

  • Architecture of encoder f(.): The encoder assumes eight video blocks of five frames, with 128 × 128 frame resolution.Table 5 specifies the tensor dimensions through the encoding stages.
  • Architecture of temporal aggregator g(.): The temporal aggregator g(.) is a one-layer ConvGRU that aggregates feature maps over past T time steps.It returns a context feature with the same number of channels as its input.

B Details of unintentional action classification

The unintentional action classifier combines the current feature with a feature predicted from past inputs, then classifies three action states. Training compensates for class imbalance by oversampling rare transitioning actions.

  • Classification framework: At each time step, MemDPC produces a current feature zt and a predicted feature ˆzt from preceding inputs.The two features are concatenated before classification.
  • Classification framework: A linear classifier assigns each time step to intentional, transitioning, or unintentional action categories.The framework uses three output classes.
  • Training and testing: Transitioning actions are oversampled during training because the Oops class distribution is highly unbalanced.Testing uses a moving temporal window and summarizes predictions.
  • Training and testing: The classifier is optimized with cross-entropy loss and Adam using a 10^-3 learning rate, decayed to 10^-4 when validation loss plateaus.The learning-rate schedule is applied once.

C Video retrieval results

MemDPC implements sequential future prediction with a compressive memory bank, temporal aggregation, and a contrastive loss over predicted and desired features.

  • Architecture: MemDPC uses a 2d3d-ResNet feature extractor, ConvGRU aggregator, MLP future predictor, and compressive memory bank.The memory bank has size [k,C], with an example configuration of [1024, 256].
  • Sequential prediction: The model first aggregates past features into a context hidden state before predicting future steps sequentially.For later steps, the predicted feature is fed back into the aggregator together with the hidden state.
  • Memory prediction: Each predicted feature is a weighted sum of memory-bank entries computed with an einsum over memory probabilities and representations.The resulting feature has shape [B,C,H,W] in the example implementation.
  • Contrastive objective: The implementation stacks predictions and extracts the corresponding desired future features for contrastive comparison.Both predicted and desired tensors use the shape [B,pred_step,C,H,W] in the illustrated code.
  • Contrastive objective: The contrastive similarity matrix is flattened across batch, prediction step, and spatial locations, with diagonal entries designated as positives.The target indices are generated from the flattened matrix size B*pred_step*H*W.

E Visualization of learned memory

Visualizations suggest that MemDPC’s learned memories capture recurring visual features and that memory addressing becomes organized by action category during training.

  • Memory retrieval: Figure 8 visualizes the memory bank as a 1024-dimensional vector and highlights selected entries used for nearest-neighbor retrieval.The corresponding top-three UCF101 training neighbors are shown for memory entries 24, 37, 213, 238, and 414.
  • Memory retrieval: Single memory entries retrieve training videos whose visual content suggests captured repetitive textures or broad backgrounds.Figure 8 queries the feature space with selected entries from the memory bank and displays their top-three retrieved UCF101 videos.
  • Memory addressing: As training progresses, the network attends to different memory entries when predicting futures for different action categories.Figure 9 visualizes t-SNE clustering of the memory-addressing probability p_t+1 for 10 randomly selected UCF101 classes.
Loading 2008.01065v1…