PaperlayerView exact revision on arXiv

Source-linked AI summary

Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond

Krishna Kumar Singh, Hao Yu, Aron Sarmasi, Gautam Pradeep, Yong Jae Lee

arXiv:1811.02545v1cs.CV

TL;DR

Weakly supervised localization approaches can focus on the most discriminative object parts rather than the full object. Hide-and-Seek randomly hides image patches or video frames during training so models learn from other relevant content. The paper reports improved performance across multiple visual recognition tasks, while noting that patch sizes and hiding probabilities must be set as hyperparameters.

  • Problem

    Weakly supervised localization methods can fail to identify an object's full extent because they focus on its most discriminative parts.

    Source: §1 · p. 1

  • Method

    Hide-and-Seek randomly hides patches or frames during training to encourage models to learn multiple relevant object or action parts.

    Source: §1 · pp. 1–2, §5 · p. 13

  • Results

    Hide-and-Seek improves performance across varied visual recognition tasks, including localization, classification, segmentation, and recognition tasks.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    The input-only augmentation applies to multiple networks and visual tasks, and needs no hidden patches at testing.

    Source: §1 · p. 2

    Patch sizes and hiding probabilities are hyperparameters that need to be set.

    Source: §5 · p. 13

Abstract

from arXiv · show

We propose 'Hide-and-Seek' a general purpose data augmentation technique, which is complementary to existing data augmentation techniques and is beneficial for various visual recognition tasks. The key idea is to hide patches in a training image randomly, in order to force the network to seek other relevant content when the most discriminative content is hidden. Our approach only needs to modify the input image and can work with any network to improve its performance. During testing, it does not need to hide any patches. The main advantage of Hide-and-Seek over existing data augmentation techniques is its ability to improve object localization accuracy in the weakly-supervised setting, and we therefore use this task to motivate the approach. However, Hide-and-Seek is not tied only to the image localization task, and can generalize to other forms of visual input like videos, as well as other recognition tasks like image classification, temporal action localization, semantic segmentation, emotion recognition, age/gender estimation, and person re-identification. We perform extensive experiments to showcase the advantage of Hide-and-Seek on these various visual recognition problems.

1 INTRODUCTION

Hide-and-Seek randomly hides image patches during training so models must use relevant content beyond the most discriminative parts. The paper presents it as a general-purpose augmentation that improves performance across visual recognition tasks, with particular relevance to weakly supervised localization.

1 INTRODUCTION

  • Hide-and-Seek randomly hides different image patches during training, encouraging the model to learn multiple relevant object parts beyond the most discriminative one.The method complements existing augmentation techniques and modifies the input image.Source: §1 · p. 1
  • Hiding patches is needed only during training; at test time, the network receives the full image.The paper notes that using the training-data mean for hidden pixels can allow training and testing input distributions to match.Source: §1 · p. 1
  • Hide-and-Seek improves weakly supervised localization, semantic segmentation, and temporal action localization, and boosts several fully supervised recognition tasks.The reported tasks include image classification, emotion recognition, and person re-identification.Source: Table 1 · p. 2, §1 · p. 2
  • Weakly supervised localization methods can miss an object's full extent by focusing on its most discriminative parts; hiding those parts in some training samples addresses this limitation.Such methods can use image-level labels without object-location annotations.Source: §1 · pp. 1–2
  • The method applies to multiple networks and tasks, including videos, where hiding frame sequences during action-classification training encourages learning relevant action frames.The paper lists AlexNet, GoogLeNet, ResNet, VGG, and Wide Residual Network among the evaluated architectures.Source: §1 · p. 2

2 RELATED WORK

The related work contrasts Hide-and-Seek with augmentation and masking approaches, emphasizing its occlusion variety, spatial alignment, and focus on less discriminative object parts. It also presents the method as a general-purpose alternative to localization-specific approaches.

2.1 Data Augmentation

  • Hide-and-Seek preserves spatial alignment while generating occlusions, unlike cropping and flipping, and its multiple discontinuous patches offer more occlusion variety than single-patch Random Erasing.The paper notes that alignment can matter for face-based emotion recognition and age/gender estimation.Source: §2.1 · p. 2

2.2 Masking pixels or activations

  • Unlike masking methods that hide regions only at test time, Hide-and-Seek masks training inputs to encourage attention to less discriminative object parts; it uses one model and no saliency refinement.The paper also distinguishes its contiguous image-region or frame masking from dropout's random unit removal.Source: §2.2 · p. 2, Figure 2 · p. 3

2.3 Weakly-supervised object localization

  • Weakly supervised localization methods can miss an object's full extent, while Hide-and-Seek uses random training-time masking to encourage localization of multiple parts without additional data or annotations.Unlike localization-specific methods, it is presented as general-purpose and applicable across vision tasks.Source: §2.3 · p. 3

3 APPROACH

Hide-and-Seek augments training by randomly hiding image patches so networks learn from other relevant content, while full images remain visible at test time. The approach extends from image localization to temporal action localization and adjusts hidden pixel values to align training and test activations.

3.1 Hide-and-Seek for images

  • Weakly supervised localizers can focus on the most discriminative object parts; Hide-and-Seek addresses this by randomly hiding image patches during training.The method is presented for image-based visual recognition and is also applied to videos.Source: §3.1 · p. 3

3.1.1 Hiding random image patches

  • Different patches are hidden for each image and epoch, exposing the network to varied object parts; no patches are hidden during testing.The network receives the full image at test time.Source: §3.1.1 · p. 4

3.1.2 Setting the hidden pixel values

  • Setting hidden pixels to the dataset mean makes hidden-region activations match the expected fully visible response during training.This addresses the difference between training with hidden patches and testing on full images; the authors report it is crucial for similar network behavior across the two phases.Source: §3.1.2 · p. 4
  • The authors report no noticeable artifacts in AlexNet’s learned Conv1 filters after training with hidden patches.They attribute this to few sharp visible/hidden transitions and to those transitions being uninformative for the task.Source: §3.1.2 · p. 5, Figure 4 · p. 5

3.1.3 Object localization network architecture

  • For object localization, global average pooling feeds the last convolutional feature maps to a classification layer, whose class weights form a class activation map.A bounding box is obtained by thresholding the map, finding foreground connected components, and fitting a tight box to the largest component.Source: §3.1.3 · p. 5

3.2 Hide-and-Seek for videos

  • For videos, the method hides fixed-size segments of uniformly sampled frames during training, then thresholds a one-dimensional class activation map to estimate action start and end times.The map represents discriminative frames for the action class.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate Hide-and-Seek across localization, segmentation, classification, emotion and age/gender estimation, and person re-identification. The supplied passages specify datasets and evaluation protocols, but do not report comparative results.

4.1.1 Weakly-supervised object localization and semantic segmentation

  • ILSVRC 2016 validation evaluates localization with Top-1 Loc and GT-known Loc, both requiring bounding-box IoU above 50%.Top-1 Loc also requires the highest-probability predicted class to match the ground-truth class.Source: §4.1.1 · p. 5

4.1.2 Weakly-supervised semantic segmentation

  • PASCAL VOC 2012 train/validation data evaluates weakly supervised semantic segmentation using Mean IU.Source: §4.1.2 · p. 6

4.1.3 Weakly-supervised temporal action localization

  • THUMOS 2014 evaluates temporal localization on 20 annotated action classes using mAP at IoU thresholds from 0.1 to 0.5, assuming the video class is known.Source: §4.1.3 · p. 6

4.1.4 Image Classification

  • CIFAR-10, CIFAR-100, and ILSVRC evaluate image classification using top-1 accuracy on validation data.Source: §4.1.4 · p. 6

4.1.5 Emotion recognition and age/gender estimation

  • CK+ evaluates emotion classification with 10-fold cross-validation; APPA-REAL measures age error, and UTKFace evaluates gender classification.The age/gender model is pretrained on IMDB-WIKI.Source: §4.1.5 · p. 6

4.1.6 Person re-identification

  • DukeMTMC-reID and Market-1501 evaluate person re-identification using rank-1 accuracy and mAP.Source: §4.1.6 · p. 6

4.2 Implementation details

Implementation details describe task-specific networks, training protocols, and evaluation choices for the experiments. For image-based Hide-and-Seek, the authors report that patch-to-image ratios from 1/2 through 1/8 work well across tasks.

4.2.1 Weakly-supervised object localization

  • AlexNet-GAP and GoogLeNet-GAP use added convolutional layers followed by global average pooling and softmax; training uses task-specific schedules and CAM thresholds.The networks are trained with hidden patches and tested without them; localization maps average ten crops.Source: §4.2.1 · p. 6

4.2.2 Weakly-supervised semantic segmentation

  • Weakly supervised semantic segmentation applies Hide-and-Seek to DCSP using its existing training protocol and VGG-16.Source: §4.2.2 · p. 6

4.2.3 Weakly-supervised temporal action localization

  • Temporal localization uses sampled C3D features divided into 20 segments, hides each segment with probability 0.5, and thresholds the action CAM at 50% of its maximum.All continuous segments remaining after thresholding are predictions.Source: §4.2.3 · p. 6

4.2.4 Image classification

  • Image-classification experiments train ResNet models from scratch on CIFAR for 300 epochs and ImageNet for 160 epochs using SGD.Source: §4.2.4 · p. 6

4.2.5 Emotion recognition and age/gender estimation

  • Emotion recognition follows the model and training protocol in, while age and gender models are pretrained jointly on IMDB-WIKI and fine-tuned separately.Age and gender are fine-tuned on APPA-REAL and UTKFace, respectively.Source: §4.2.5 · pp. 6–7

4.2.6 Person re-identification

  • Person re-identification follows, training for 100 epochs on DukeMTMC-reID and 80 epochs on Market-1501.Source: §4.2.6 · p. 7

4.2.7 Hidden image patch size N

  • Patch-to-image ratios from 1/2 through 1/8 work well across recognition tasks; centered, tightly cropped objects work well with a 1/2 ratio.Source: §4.2.7 · p. 7

4.3 Weakly-supervised object localization

Hide-and-Seek improves weakly supervised object localization across architectures and patch settings, while also changing which pooling and hiding choices work well. Its qualitative gains have limits: nearby instances, contextual objects, and reflections can still mislead localization.

4.3.1 Quantitative object localization results

  • Across the four tested patch sizes (16, 32, 44, and 56), each performs significantly better than AlexNet-GAP on GT-known Loc; AlexNet-HaS-Mixed achieves the best Top-1 Loc results.During training, patch size is chosen randomly for each image and epoch from 16, 32, 44, and 56, as well as no hiding (full image).Source: §4.3.1 · p. 7

4.3.2 Comparison to alternate localization methods

  • 3.78 percentage points on GT-known Loc and 1.40 percentage points on Top-1 Loc for AlexNet, and 1.88 and 1.61 percentage points, respectively, for GoogLeNet, over their GAP baselines.These gains come from changing the input image without changing the network architecture.Source: §4.3.2 · p. 7, Table 3 · p. 7
  • Image-layer dropout performs substantially worse than Hide-and-Seek; applying dropout at test time improves it, but performance remains much lower.The authors report that pixel dropout can leave the most relevant object parts visible with high probability.Source: §4.3.2 · p. 7

4.3.3 Qualitative object localization results

  • Hide-and-Seek localizes more object parts than GAP, including animal bodies and tails that the baseline misses.The qualitative comparisons show GAP focusing on faces in some examples, while Hide-and-Seek also finds body regions.Source: §4.3.3 · p. 8, Figure 5 · p. 8
  • Localization can fail when nearby instances merge, contextual objects attract predictions, or a bird’s reflection is mistaken for the bird.These failure cases can produce a single box spanning multiple instances or including context.Source: §4.3.3 · pp. 8–9

4.3.4 Further Analysis of Hide-and-Seek

  • With Hide-and-Seek, max pooling markedly improves localization over its baseline and slightly outperforms average pooling with Hide-and-Seek.The authors suggest max pooling’s greater noise robustness as a possible explanation for the slight advantage.Source: §4.3.4 · p. 9, Table 4 · p. 9
  • Hiding patches in AlexNet’s first convolutional feature maps also boosts performance over AlexNet-GAP, extending the method beyond image inputs.The experiment hides grid patches and their corresponding channels with 0.5 probability.Source: §4.3.4 · p. 9, Table 5 · p. 9
  • Higher hiding probabilities leave GT-known Loc almost unchanged but reduce Top-1 Loc, whereas lower probabilities improve Top-1 Loc but reduce GT-known Loc.The authors attribute the trade-off to the number of visible pixels and the extent to which the network learns to focus beyond discriminative parts.Source: §4.3.4 · p. 9, Table 6 · p. 9

4.4 Weakly-supervised semantic image segmentation

Applying Hide-and-Seek during DCSP training improves weakly supervised semantic segmentation on PASCAL 2012 validation data.

4.4 Weakly-supervised semantic image segmentation

  • Mean IU improves from 60.80 to 61.45 when HaS-Mixed is applied during DCSP’s attention-network training.The resulting CAM focuses on more relevant object parts and produces a better pseudo-ground-truth map.Source: §4.4 · p. 10

4.5 Weakly-supervised temporal action localization

Video-HaS outperforms the full-video baseline on THUMOS validation data for temporal action localization. Qualitative examples show more complete action coverage, though contextual frames can still extend predictions beyond an action’s duration.

4.5 Weakly-supervised temporal action localization

  • Video-HaS consistently outperforms Video-full across all five IoU thresholds on THUMOS validation data.The method randomly hides temporal frame segments during training.Source: §4.5 · p. 10
  • In the examples, Video-HaS localizes more of each action’s temporal extent, while Video-full often finds only key moments.The examples include full-frame coverage for javelin throw and high jump, compared with only a thrown-javelin frame or the beginning of the jump for Video-full.Source: §4.5 · p. 10
  • Video-HaS can wrongly extend diving predictions into following swimming-pool frames, which co-occur with diving.The authors attribute this error to the network focusing on contextual frames when diving frames are hidden.Source: §4.5 · p. 10

4.6 Image classification

Hide-and-Seek is evaluated for image classification alongside standard augmentation and improves classification results across several datasets and ResNet architectures. The paper also reports higher accuracy on a set of partially occluded images.

4.6 Image classification

  • Hide-and-Seek improves classification results across CIFAR-10, CIFAR-100, and ImageNet with different ResNet architectures.The experiments also use random cropping and horizontal flips for both Full and HaS models.Source: §4.6 · pp. 10, 12, Table 8 · p. 12
  • For classification training, the method alternates between hiding a patch and showing the full image; the authors report that seeing all object parts together often is important.CIFAR uses patch size 8, while ImageNet samples patch sizes 16, 32, 44, and 56.Source: §4.6 · pp. 10, 12
  • On 200 partially occluded images from 100 random classes, AlexNet-HaS achieves 3% higher classification accuracy than AlexNet-GAP.The examples include correct predictions when salient faces are hidden.Source: §4.6 · p. 12

4.7 Semantic image segmentation

Hide-and-Seek improves semantic segmentation after ImageNet pretraining and benefits facial emotion, age, and gender estimation. Its spatial alignment is an additional advantage for face-based tasks.

4.7 Semantic image segmentation

  • Hide-and-Seek pretraining improves FCN performance on the PASCAL 2011 segmentation validation set.The comparison uses an AlexNet model pretrained with HaS-Mixed on ImageNet classification.Source: §4.7 · p. 12

4.8 Face based emotion recognition and age/gender estimation

  • Hide-and-Seek significantly improves facial emotion classification accuracy on CK+, including over a CNN with standard augmentation.The experiments use zero-bias CNNs and randomly selected patch sizes of 12, 16, 32, or 48 pixels, with hiding probability 0.5.Source: Table 10 · p. 12, §4.8 · p. 12
  • Hide-and-Seek improves both apparent- and real-age estimation on APPA-REAL and gender classification accuracy on UTKFace.The table caption reports smaller age-estimation error and higher gender-classification accuracy when HaS is combined with standard augmentation.Source: Table 11 · p. 12, §4.8 · pp. 12–13
  • Preserving pixel alignment gives Hide-and-Seek an additional advantage for facial emotion, age, and gender estimation tasks.Unlike random flipping and cropping, Hide-and-Seek preserves spatial alignment across augmented face samples.Source: §4.8 · p. 12

4.9 Person re-identification

Hide-and-Seek improves person re-identification results over Random Erasing on Market-1501, while results on DukeMTMC-reID are mixed. The best hiding probability differs between the datasets.

4.9 Person re-identification

  • Hide-and-Seek replaces Random Erasing with 64-pixel patches in person-centric 256 × 128 images, which can hide prominent body parts without hiding the whole person.Person re-identification retrieves instances of a queried person from images containing multiple people, often across occlusion, pose, or viewpoint changes.Source: §4.9 · p. 13
  • 0.7% rank-1 accuracy and 1.2% mAP gains over IDE+CamStyle+RE on Market-1501; DukeMTMC-reID rank-1 accuracy gains 1.6%, while mAP is 0.4% lower.The table summarizes the person re-identification comparison.Source: §4.9 · p. 13
  • A hide probability of 0.5 boosts both datasets over IDE+CamStyle, while 0.33 works slightly better on Market-1501.The authors attribute the dataset difference to greater clothing and body-part visibility variation in Market-1501, where hiding less exposes more of those variations.Source: §4.9 · p. 13

5 CONCLUSION

Hide-and-Seek randomly hides patches or frames in training images or videos to encourage attention to multiple relevant parts. Experiments show performance improvements across vision tasks, while patch sizes and hiding probabilities remain hyperparameters to set.

5 CONCLUSION

  • Randomly hiding patches or frames in training images or videos encourages networks to focus on multiple relevant parts of an object or action.The conclusion presents Hide-and-Seek as a simple, general-purpose visual-recognition augmentation technique.Source: §5 · p. 13
  • Extensive experiments show that Hide-and-Seek can significantly improve model performance on various vision tasks.Source: §5 · p. 13
  • Patch sizes and hiding probabilities remain hyperparameters that need to be set; dynamically learning them is proposed as future work.Source: §5 · p. 13

Related papers

Loading 1811.02545v1…