PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning

Uta Büchler, Biagio Brattoli, Björn Ommer

arXiv:1807.11293v1cs.CV

TL;DR

Ordering-based self-supervision has generally sampled permutations randomly, despite their potential utility changing with the network’s state. This paper jointly addresses spatial and temporal ordering with a deep-reinforcement-learning policy that selects permutations adaptively, and reports favorable results across image and video tasks, including nearest-neighbor retrieval.

  • Problem

    Prior ordering-based self-supervision randomly selects permutations, although their utility for learning CNN representations may depend on the network’s current state.

    Source: §3.2 · p. 5

  • Method

    A reinforcement-learning policy selects spatial and temporal training permutations based on network state and their expected utility for improving the CNN representation.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    The adaptive spatiotemporal policy shows favorable results over the state of the art across image classification and segmentation, video action recognition, and nearest-neighbor retrieval.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The final spatiotemporal model trained with the policy reaches almost the supervised-features threshold in the reported ablation.

    Source: §4.3 · p. 13

Abstract

from arXiv · show

Self-supervised learning of convolutional neural networks can harness large amounts of cheap unlabeled data to train powerful feature representations. As surrogate task, we jointly address ordering of visual data in the spatial and temporal domain. The permutations of training samples, which are at the core of self-supervision by ordering, have so far been sampled randomly from a fixed preselected set. Based on deep reinforcement learning we propose a sampling policy that adapts to the state of the network, which is being trained. Therefore, new permutations are sampled according to their expected utility for updating the convolutional feature representation. Experimental evaluation on unsupervised and transfer learning tasks demonstrates competitive performance on standard benchmarks for image and video classification and nearest neighbor retrieval.

1 Introduction

Self-supervision uses automatically generated surrogate tasks to learn visual features from unlabeled data. The paper treats spatial and temporal ordering as related tasks and proposes using reinforcement learning to choose permutations according to their utility as the network changes during training.

1 Introduction

  • Self-supervision provides a way to learn visual features from unlabeled data by solving surrogate tasks whose targets can be obtained automatically.Examples include recovering image color, predicting basic motions, counting parts, and embedding images into text-topic spaces.Source: §1 · p. 1
  • Ordering tasks train a network to infer the original structure after visual data is shuffled, using spatial shuffling for images and temporal shuffling for videos.Because both tasks address ordering and differ in ordering dimension, the paper argues they should be addressed jointly.Source: §1 · p. 2
  • Prior ordering methods randomly selected training permutations, while this work uses reinforcement learning to propose permutations based on the network’s evolving state and improvement over time.The policy and self-supervised network are trained simultaneously, with network improvement serving as the reward signal.Source: §1 · p. 2
  • Shuffling patches at different spatial scales can affect the CNN differently, and the useful permutations may change as the network learns.The paper proposes examining prior ordering errors to identify permutations suited to the network’s current state.Source: §1 · p. 2

2 Related Work

The related work covers self-supervised representation learning, curriculum learning, and methods that adapt neural-network training. The paper distinguishes its approach through complementary spatial and temporal ordering tasks and a policy that selects permutations dynamically from the network state.

Self-Supervised Representation Learning

  • Prior self-supervised methods combine heterogeneous surrogate tasks, sometimes adding techniques such as a graph-based triplet loss, whereas this model combines related ordering tasks without additional adjustment approaches.The related-work discussion also situates the paper among single-task and multi-task self-supervision approaches.Source: §2 · p. 2

Curriculum Learning

  • Curriculum-learning methods typically set sample complexity and scheduling in advance, while this policy dynamically selects permutations from the network’s current state.The cited work includes approaches that define sample complexity from the classifier’s perspective but still manually schedule samples.Source: §2 · pp. 2–3

Meta-Learning for Deep Neural Networks

  • Like prior meta-learning methods, this approach alters network training, but its policy can directly change the input data by proposing permutations rather than scheduling fixed-label images.The discussion names learned optimization, reinforcement-learning-based sample scheduling, and gradient-based sample selection as related approaches.Source: §2 · p. 3

3 Approach

The approach learns a shared visual representation by solving spatial tile-ordering and temporal frame-ordering tasks, then uses a state-conditioned reinforcement-learning policy to choose training permutations. The policy groups permutations by their current difficulty, evaluates their effect against an extrapolated error baseline, and alternates updates with self-supervised training.

3.1 Self-Supervised Spatiotemporal Representation Learning

  • Spatial tile ordering and temporal frame ordering jointly train a shared CNN representation, with separate classifiers for tiles and frames whose gradients reach conv1.The spatial task stacks tile features in fc7, while the temporal task combines frame features with an LSTM.Source: §3.1 · p. 4

3.2 Finding an Optimal Permutation Strategy by Reinforcement Learning

  • The policy conditions permutation choices on network state because randomly sampling from a fixed candidate set ignores permutations’ differing utility for learning the representation.The state is estimated from network performance on validation samples permuted by each candidate; softmax ratios reflect permutation recoverability.Source: §3.2 · p. 5
  • The policy uses Policy Gradient to maximize reward, defined by comparing the network’s error after training with an extrapolation of prior errors.The policy proposes an action for a randomly sampled training point based on state, and the reward favors actions that improve on the baseline.Source: §3.2 · pp. 6–7
  • Grouping permutations of similar classification difficulty reduces the policy’s effective action space, while group size helps avoid overfitting to very small groups.The policy selects a group and then a permutation within it; group features include group size and the median softmax ratio.Source: §3.2 · pp. 7–8
  • Self-supervised training and policy updates alternate, and the reported policy-training overhead is approximately 40% over basic self-supervised training.Between policy updates, the network trains with proposed permutations; each episode then uses validation to determine the reward and update the policy.Source: §3.2 · p. 8

4 Experiments

Experiments evaluate self-supervised features through nearest-neighbor retrieval, transfer to image and video tasks, and ablations of joint spatial-temporal training and policy sampling. Results favor the combined tasks and policy sampling, with competitive retrieval and transfer performance.

4.1 Self-Supervised Training

  • Our model achieves the highest nearest-neighbor accuracy at every tested k on UCF-101 and Pascal VOC 2007, with Top-1 accuracy especially higher than other approaches.Evaluation uses cosine distance between pool5 features.Source: Table 1 · p. 9, §4.1 · p. 10

4.2 Transfer Capabilities of the Self-Supervised Representation

  • More than 2% over the best model with a comparable architecture and almost 4% in the linear task on ImageNet; action-recognition gains are 2.3% on UCF-101 and 2.9% on HMDB-51.The HMDB-51 videos were not seen during self-supervised training.Source: Table 2 · p. 11, §4.2 · p. 11, Table 3 · p. 11
  • On Pascal VOC, the method significantly improves over other approaches across classification, detection, and segmentation, outperforming on classification and segmentation.The comparisons use fine-tuned features; classification and detection are evaluated on VOC’07, and segmentation on VOC’12.Source: Table 4 · p. 12, §4.2 · p. 12

4.3 Ablation Study

  • After three initially uniform episodes, the policy favors high-validation-error permutations and mixes hard with intermediate groups rather than overfitting to one complexity.Permutations are grouped by validation error; the policy is updated after each episode.Source: Figure 4 · p. 13, §4.3 · p. 14
  • On VOC07 Top1 nearest-neighbor evaluation, the combined spatiotemporal model outperforms single-task spatial and temporal networks by 7% and 14%, respectively; S+T+P reaches almost the supervised-features threshold.Source: Figure 5 · p. 13, §4.3 · p. 13
  • Fine-tuning confirms S+T beats either single task, policy variants improve over baseline models, and parallel S+T yields a big gain over serial S&T.The evaluation covers Pascal VOC07 multi-object classification and UCF-101 action recognition.Source: Table 5 · p. 13, §4.3 · p. 13
  • With the policy, spatial validation error improves faster than with random permutations; without it, error improves uniformly across permutations, unlike the policy’s non-uniform decrease.Source: Figure 6 · p. 14, §4.3 · p. 14

5 Conclusion

The paper combines spatial and temporal ordering and uses an RL policy to select permutations according to their expected utility for representation learning. Its training procedure samples batches from selected permutation groups, then updates the policy using validation-derived state and reward.

5 Conclusion

  • The adaptive spatiotemporal policy samples permutations by expected utility and shows favorable results across image classification, segmentation, and video action recognition.The policy adds relatively small computational cost compared with basic training.Source: §5 · p. 14
  • The paper brings spatial and temporal ordering together as directly related self-supervision tasks.Source: §5 · p. 14

A Algorithm for Updating the Policy

  • Each episode samples a permutation group through the policy, trains the CNN on a batch of permuted samples, then updates the policy from validation-derived state and reward.The procedure updates policy weights using Eq. 9.Source: A Algorithm for Updating the Policy · p. 19

B Technical Details

The technical details specify the training schedules and input preparation for the spatiotemporal network, with separate spatial and temporal branches. The supplementary figure evaluates possible validation-set sizes.

B Technical Details

  • The spatiotemporal network trains for 350k iterations with batch size 128, starting SGD learning rate 0.001 and a tenfold reduction after 200k iterations.Source: B Technical Details · p. 19
  • The technical appendix includes a figure evaluating the optimal validation-set size.Source: Figure C.1 · p. 20
  • Spatial inputs are divided into nine 75×75 tiles, while temporal inputs use eight randomly selected frames cropped to 224×224 and resized to 75×75.The implementation uses random cropping and color jittering, with tiles or frames normalized separately.Source: B Technical Details · p. 20

C Size of Validation Set

The appendix supports a validation set of 100 images because smaller sets have larger error variability, and uses 10 permutation groups to avoid groups with similar complexity.

C Size of Validation Set

  • Validation sets of 10, 20, and 50 images show comparatively large error standard deviations, while sizes 100 and 200 differ little, motivating the choice of 100.The comparison uses five randomly sampled sets per size across checkpoints from 50k to 350k iterations.Source: C Size of Validation Set · p. 20

D Number of Groups

  • The group analysis compares permutation-complexity distributions with KS tests, treating p-values below α = 0.01 as evidence that distributions differ.Source: D Number of Groups · p. 21
  • At 15 groups, some distributions are already indistinguishable at the stated significance threshold, so 10 groups is chosen to avoid redundant complexity groups.Source: D Number of Groups · p. 21

E Baseline Error EBL Description

The policy improves performance over random permutation selection, while the inverse policy performs worse. Its reward uses an extrapolated baseline error to determine whether the validation error qualifies for positive reward.

E Baseline Error EBL Description

F How Decisive Are the Permutations?

G Permutation Selection of our Policy

The policy selects permutations according to the network state rather than following a simple curriculum based on shuffling degree. Its added computation comes from validation passes, not policy inference or updates.

G Permutation Selection of our Policy

Policy Cost Calculation

  • Policy computation is dominated by validation, which evaluates 100 samples across 1,000 permutations with the main network; policy inference and updates are negligible by comparison.Validation runs twice per episode, at its beginning and end.Source: Policy Cost Calculation · pp. 24–25
  • The policy is updated 90 times because more frequent updates add computational cost without further performance gain, relative to 350k self-supervised training iterations.Source: Policy Cost Calculation · p. 25

Performance Relative to Computation

Even after accounting for the policy’s added computation, training with the policy shows an advantage in unsupervised object classification on Pascal VOC 2007. The figure compares performance by total forward passes.

Performance Relative to Computation

  • The policy retains a big advantage in unsupervised Pascal VOC 2007 object classification even when its extra computational cost is included.The comparison accounts for total forward passes, including those used to train the policy.Source: Performance Relative to Computation · p. 25, Figure H.1 · p. 25
  • The figure plots unsupervised Pascal VOC 2007 object classification against total forward passes, with random-weight and ImageNet-trained features as references.Source: Figure H.1 · p. 25

I.1 Activations

The self-supervised model’s conv5 units show consistent activations across datasets, including units responding to eyes, faces, and landscape skies. Another unit responds to car tires across ImageNet and Pascal VOC images.

I.1 Activations

  • Conv5 units respond consistently across ImageNet, Pascal VOC, and UCF-101, with units focusing on eyes, faces, and landscape skies.The passage identifies this cross-dataset consistency as evidence of transfer capability.Source: §I.1 · p. 26, Figure I.1 · p. 26
  • A conv5 neuron responds to car wheels in images from both ImageNet and Pascal VOC.The passage describes these as tires of a four-wheel vehicle.Source: Figure I.2 · p. 26, §I.1 · p. 26

I.2 Characteristic Details

On Pascal VOC 2007, saliency maps show that the full self-supervised initialization preserves more characteristic object structure than the version without the policy. ImageNet supervision remains an upper bound.

I.2 Characteristic Details

  • The S+T+P saliency maps preserve more characteristic object details than S+T, while supervised ImageNet pre-training serves as an upper bound.The comparison follows fine-tuning on Pascal VOC 2007; random initialization is also included.Source: §I.2 · pp. 26–27, Figure I.3 · p. 27

Related papers

Loading 1807.11293v1…