Source-linked AI summary
Learning Image Representations by Completing Damaged Jigsaw Puzzles
Dahun Kim, Donghyeon Cho, Donggeun Yoo, In So Kweon
TL;DR
The paper asks how self-supervised representation learning can reduce reliance on human-labeled data while producing features that transfer across tasks and domains. It complicates jigsaw, inpainting, and colorization, combining them in damaged jigsaw completion, and reports state-of-the-art transfer performance on PASCAL classification and semantic segmentation. The study also notes that combining tasks with very different inputs can hinder simultaneous training and require special handling.
Problem
Human-supervised pretraining limits the scalability and adaptability of visual representations to dissimilar target tasks and domains.
Method
The method jointly trains a convolutional network to arrange shuffled patches, recover a missing patch, and colorize decolorized patches, while also complicating jigsaw, inpainting, and colorization tasks.
Results
The learned representations achieve state-of-the-art performance on PASCAL classification and semantic segmentation when transferred with AlexNet.
Takeaways & Limitations
The experiments report that the learned features generalize across PASCAL classification, detection, and segmentation, ImageNet linear classification, and nearest-neighbor search.
Takeaways & Limitations
Combining self-supervised tasks with very different inputs can hinder simultaneous training and require special handling.
Abstract
from arXiv · showhide
In this paper, we explore methods of complicating self-supervised tasks for representation learning. That is, we do severe damage to data and encourage a network to recover them. First, we complicate each of three powerful self-supervised task candidates: jigsaw puzzle, inpainting, and colorization. In addition, we introduce a novel complicated self-supervised task called "Completing damaged jigsaw puzzles" which is puzzles with one piece missing and the other pieces without color. We train a convolutional neural network not only to solve the puzzles, but also generate the missing content and colorize the puzzles. The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations. We demonstrate that complicating the self-supervised tasks improves their original versions and that our final task learns more robust and transferable representations compared to the previous methods, as well as the simple combination of our candidate tasks. Our approach achieves state-of-the-art performance in transfer learning on PASCAL classification and semantic segmentation.
1. Introduction
The paper addresses limited scalability and adaptability in supervised representation learning by making self-supervised recovery tasks more challenging. It combines damaged jigsaw, inpainting, and colorization tasks to learn transferable features for downstream vision tasks.
- Limited labeled data can cause overfitting, motivating pretraining visual features on large-scale data for target tasks.
- ImageNet classification pretraining provides strong representations but depends on human supervision and transfers less readily to dissimilar domains such as depth prediction.
- Completing damaged jigsaw puzzles removes one patch and color from the others, requiring simultaneous recovery of arrangement, missing content, and color.
- The proposed strategy raises the difficulty of jigsaw, inpainting, and colorization tasks, then transfers learned features to PASCAL classification, detection, and segmentation.
- The approach combines three complicated self-supervised tasks in one problem rather than relying on a single recovery signal.
- The learned representations achieve state-of-the-art performance on PASCAL classification and semantic segmentation after transfer with AlexNet.
2. Related works
Prior self-supervised representation methods derive supervision from damaged spatial, contextual, or channel information, while multi-task methods combine complementary signals. This paper instead complicates those tasks and trains them simultaneously, despite challenges from very different inputs.
- Unsupervised representation learning includes autoencoders and related generative models that reconstruct inputs to extract data representations.
- Self-supervised learning manufactures supervision from raw visual data and has been applied to images, video, egomotion, and sound.
- Self-supervised methods emphasize different image characteristics through spatial configuration, context, or cross-channel relations.
- Jigsaw methods infer spatial relations between patches, while inpainting recovers dropped regions from surrounding pixels.
- Colorization methods predict ab channels from the L channel, using cross-channel relations as self-supervision.
- Multi-task approaches aggregate complementary representations, but very different inputs can hinder simultaneous training and require special handling.
- This paper addresses that challenge by making jigsaw, inpainting, and colorization more difficult and incorporating them into one simultaneously trained problem.
3. Approach
The approach complicates jigsaw, inpainting, and colorization by adding spatial, contextual, and channel damage, then combines their recoveries in one self-supervised task. A shared AlexNet-based representation feeds task-specific branches whose learned features transfer to downstream tasks.
- Complicating Each Self-supervised Task: Self-supervised tasks are made harder by damaging spatial configuration, image context, and image channels before recovery.The strategy targets task-specific representations by requiring recovery from more severe damage.
- Jigsaw Puzzle: The complicated jigsaw task shuffles patches, removes one piece, and distributes L and ab channels across the remaining puzzle.Solving it requires spatial configuration and cross-channel understanding in addition to the original permutation prediction.
- Inpainting: Complicated inpainting predicts the missing region’s ab channels from surrounding pixels represented in the L channel.This combines contextual inference with cross-channel recovery rather than reconstructing all channels from intact context.
- Colorization: Complicated colorization receives only the L channel from one patch of a 2×2 puzzle and predicts its color distribution over 313 quantized ab values.Only one-quarter of the entire image is available to the colorization branch.
- Completing Damaged Jigsaw Puzzles: Completing damaged jigsaw puzzles jointly recovers shuffled patches, a discarded patch, and dropped ab channels, with missing content defined as ab generation from surrounding L pixels.The final task intermingles jigsaw, inpainting, and colorization so cross-channel knowledge supports inpainting and colorization while context is shared across tasks.
- Architecture and Losses: The model uses a 3×3, 9-tower siamese architecture with a shared AlexNet conv1–7 tower and separate jigsaw, inpainting, and colorization branches.The shared tower supplies features for transfer learning; the jigsaw branch predicts a 1000-dimensional vector, while pixel-level branches use convolutional outputs.
4. Training
The proposed network is trained on 1.3M unlabeled ImageNet images using resized inputs, extracted puzzle patches, ADAM optimization, and a scheduled learning rate.
- 1.3M unlabeled ImageNet training images are used to train the proposed network.The images come from ImageNet’s training set without annotations.
- 312×312 input images yield 140×140 patches for 2-by-2 puzzles and 85×85 patches for 3-by-3 puzzles.
- ADAM trains the network for 350K iterations with batch size 64 on a GTX 1080-Ti GPU and Intel i7 3.4GHz CPU.
5. Results and Discussions
The evaluations test complicated self-supervised tasks and their combinations through transfer to PASCAL and ImageNet. Complicating tasks consistently improves predecessors, while the combined method achieves strong downstream and representation-level performance.
- Evaluation setup: The evaluations transfer learned representations to PASCAL classification, detection, and semantic segmentation using standard benchmarks.VOC 2007 is used for classification and detection, while VOC 2012 is used for segmentation.
- Complicating tasks: Complicated self-supervised tasks consistently score higher than their predecessors in both classification and segmentation.The authors interpret this as evidence that solving more difficult tasks can extract more useful representations.
- Completing damaged jigsaw puzzles: The final method outperforms previous methods in classification and segmentation and achieves the second-best detection performance, despite grayscale-only pretraining.
- ImageNet transfer: Frozen-layer ImageNet evaluation trains linear classifiers over conv1–conv5 features to assess task-generality.
- ImageNet transfer: Conv1 performs slightly lower because training uses only the L channel, but conv4 reaches second-best and conv5 reaches state-of-the-art performance.
- Task combinations: Combining tasks increases performance at every addition, and the three-task method improves the jigsaw baseline by 2.6% in classification and 2.5% in segmentation.The simple combination of original tasks improves single-task baselines but remains below the proposed combinations.
- Nearest-neighbor analysis: Nearest-neighbor retrieval shows that single-task representations emphasize distinct cues such as pose, context, or color.Jigsaw emphasizes pose and shape, inpainting emphasizes co-occurrence and background, and colorization emphasizes color.
- Nearest-neighbor analysis: The proposed representation is more invariant to pose and intra-class color variation while retaining context-sensitive retrieval behavior.Its retrieval behavior is described as closer to ImageNet classification pretraining than the single-task baselines.
6. Conclusions
The paper studies harder self-supervised tasks for representation learning, including complicated jigsaw puzzles, inpainting, and colorization. It further introduces completing damaged jigsaw puzzles and reports broad transfer effectiveness across visual tasks.
- The paper proposes more difficult versions of jigsaw puzzles, inpainting, and colorization for self-supervised representation learning.
- Completing damaged jigsaw puzzles combines missing-content recovery and colorization in a more complex self-supervised problem.
- Experiments evaluate the learned features through PASCAL VOC classification, detection, and segmentation, ImageNet linear classification, and nearest-neighbor search.
- The reported results show that the learned features generalize across different high-level visual tasks.