Source-linked AI summary
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
Unaiza Ahsan, Rishi Madhok, Irfan Essa
TL;DR
Video action recognition motivates learning representations without large manually labeled datasets, while combining spatial and temporal cues can involve tracking or optical-flow preprocessing. Video Jigsaw trains a network to solve patch puzzles across multiple frames, and the paper reports competitive transfer results on UCF101 and HMDB51 after Kinetics pretraining.
Problem
Video action recognition needs representations learned without large manually labeled datasets, while combining spatial and temporal cues can require tracking or optical-flow patch mining.
Source: §1 · pp. 1–2Method
The method trains a network to solve jigsaw puzzles made by shuffling patches drawn from multiple video frames, jointly using spatial and temporal context.
Source: §1 · p. 2Results
Takeaways & Limitations
The permutation strategy respects spatial coherency and generates diverse permutations efficiently, including for 12 patches.
Source: §5 · p. 9The approach requires a large-scale, diverse unlabeled video dataset; the reported training used about 400,000 tuples from three frames per Kinetics video.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self-supervised approaches have used spatial context [9, 34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30]. We propose to combine spatial and temporal context in one self-supervised framework without any heavy preprocessing. We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames. So the network is trained to correctly identify the position of a patch within a video frame as well as the position of a patch over time. We also propose a novel permutation strategy that outperforms random permutations while significantly reducing computational and memory constraints. We use our trained network for transfer learning tasks such as video activity recognition and demonstrate the strength of our approach on two benchmark video action recognition datasets without using a single frame from these datasets for unsupervised pretraining of our proposed video jigsaw network.
1. Introduction
Video Jigsaw jointly uses spatial and temporal context by training a network to solve puzzles made from patches across multiple frames. It avoids explicit object tracking and optical-flow patch mining, and the authors report transfer to UCF101 and HMDB51 from Kinetics pretraining.
1. Introduction
- The task divides multiple video frames into patches, shuffles them into jigsaw puzzles, and trains the network to solve them.Figure 1 depicts three 2×2-grid frames, the shuffled set of 12 patches, and the assembled puzzle.Source: §1 · p. 2, Figure 1 · p. 2
- The joint framework exploits spatial and temporal context without explicit object tracking or optical-flow-based patch mining.The paper describes tracking or optical-flow mining as preprocessing approaches it avoids.Source: §1 · pp. 1–2
- The proposed permutation strategy constrains sampled permutations, outperforms random permutations, and is memory efficient.Source: §1 · p. 2
- The Kinetics-trained video jigsaw network demonstrates domain transfer with competitive results on UCF101 and HMDB51.The paper states that its best self-supervised model was trained on Kinetics video frames.Source: §1 · p. 2
2. Related Work
Related work spans reconstruction-based and self-supervised representation learning, including methods built around spatial patches and temporal video cues. Prior work also combines multiple cues, while the paper distinguishes its single task from multi-task approaches.
2. Related Work
- Self-supervised methods construct auxiliary tasks and pseudo-labels from unlabeled data to learn features useful for a target task.Related approaches include video prediction, exemplar CNNs, CliqueCNNs, and clustering.Source: §2 · p. 2
- Spatial-context methods use patch arrangements for supervision, including relative-location prediction and image jigsaws.Other image tasks include inpainting and colorization.Source: §2 · pp. 2–3
- Temporal-coherency methods use video cues such as tracking [58], shuffled-frame verification, and optical-flow-selected frame tuples.The paper contrasts these approaches with its use of multiple patches from multiple frames and exact permutation prediction.Source: §2 · p. 3
- Prior combined-cue methods include multi-task training and tracked-object comparisons [59], whereas this work predicts shuffled patch configurations in one task.The paper also contrasts its richer permutation labels with a prior binary spatial-and-temporal task.Source: §2 · p. 3
3. The Video Jigsaw Puzzle Problem
Video jigsaw trains a network to recover the ordering of patches drawn from three video frames, capturing both appearance and changes over time. The method uses a multi-stream network and a spatially coherent permutation sampler to generate training puzzles.
3. The Video Jigsaw Puzzle Problem
- Three 2 × 2 frames yield 12 patches, and the network learns to predict their shuffled order to capture appearance and how activity parts change over time.Source: §3 · p. 3
3.1. Training Video Jigsaw Network
- Each patch is processed by a separate stream with shared parameters; fc6 features are concatenated at fc7, and a softmax predicts one of N permutations.Source: §3.1 · p. 4
3.2. Generating Video Jigsaw Puzzles
- The proposed sampler keeps each frame’s patches together, selects permutations by Hamming distance, and reduces computation and memory while achieving similar or better transfer-learning performance.Unlike the baseline that treats all patches as one image, it prevents patches from moving between frame positions.Source: §3.2 · p. 4
4. Experiments
Experiments evaluate video-jigsaw pretraining across datasets, permutation counts, and patch sizes, then test transfer to action recognition and PASCAL VOC 2007. Kinetics-pretrained features perform competitively across video and image tasks, with qualitative visualizations also reported.
4.1. Datasets
- The experiments use UCF101 and Kinetics for pretraining and UCF101 and HMDB51 for video action-recognition evaluation.UCF101 contains 101 action categories and 13,320 videos; HMDB51 contains about 7,000 videos across 51 categories.Source: §4 · p. 5, §4.1 · p. 5
4.2. Video Jigsaw Network Training
- The training setup uses frame tuples without further preprocessing, and Kinetics tuples sample the 1st, 5th, and 10th frames of each 10-second video.UCF101 tuples are further sampled from tuples supplied by the authors of, without using their optical-flow regions.Source: §4.2 · p. 5
- Kinetics pretraining yields better UCF101 finetuning than UCF101 pretraining, despite lower video-jigsaw solving performance; the comparison also holds without UCF101 oversampling.The authors suggest oversampling may make the jigsaw task easier without improving transfer, but reduced UCF101 tuples still do not match Kinetics.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
- Increasing the number of permutations makes the jigsaw task harder but improves generalization; larger patches raise UCF101 finetuning accuracy, with little gain beyond 80 × 80.The permutation-count trend is reported for Kinetics-trained models.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7
- The spatial-coherence-preserving permutation strategy performs better or comparably to the original random strategy while generating permutations many times faster.Figure 4 compares VJ Acc and FN Acc across different numbers of permutations; the caption also reports memory and computational savings.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3. Finetuning for Action Recognition
- On HMDB51, video-jigsaw pretraining outperforms recent unsupervised pretraining approaches; on UCF101, its finetuning accuracy is comparable to state of the art, with the best network pretrained on Kinetics.Source: §4.3 · p. 7, Table 5 · p. 8
4.4. Results on PASCAL VOC 2007 Dataset
- Kinetics-trained features perform competitively against state-of-the-art image-based semi-supervised approaches and outperform most video-based semi-supervised methods on PASCAL VOC 2007.The network was trained on Kinetics videos rather than object-centric images.Source: §4.4 · p. 8
4.5. Visualization Experiments
- The first 40 conv1 filters show oriented edges, and qualitative retrievals suggest the video-trained model can identify objects in still images without activity labels.The retrieval experiment uses the model finetuned on PASCAL VOC.Source: §4.5 · p. 8, Figure 5 · p. 8
5. Conclusion
The video jigsaw framework jointly exploits spatial and temporal context without object tracking or optical-flow-based patch mining. It performs competitively on video action recognition, including outperforming the state of the art in self-supervised video action recognition on HMDB51.
5. Conclusion
- The approach outperforms the state of the art in self-supervised video action recognition on HMDB51.The conclusion also reports competitive performance on video activity recognition.Source: §5 · p. 8
- The framework jointly exploits spatial and temporal context without object tracking or optical-flow-based patch mining.Its permutation generation strategy respects spatial coherency and can efficiently generate diverse permutations for shuffling 12 patches.Source: §5 · pp. 8–9
- The authors propose a permutation-generation strategy that respects spatial coherency; using this approach, they report that diverse permutations can be generated extremely efficiently, even when shuffling 12 patches.Source: §5 · p. 9