PaperlayerView exact revision on arXiv

Source-linked AI summary

Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, Tong Zhang

arXiv:1901.01703v7cs.CV

TL;DR

Single-tag training may omit image contents, while many visual tasks lack sufficient training data. Tencent ML-Images pairs a large multi-label database with ResNet-101 training, and transfer experiments across three vision tasks support the quality of its checkpoint.

  • Problem

    Single tags cannot describe all important image contents, and limited training data for tasks such as detection and segmentation makes pre-trained representations useful.

    Source: §1 · p. 1

  • Method

    The authors built Tencent ML-Images by combining Open Images and ImageNet vocabularies into a database of about 18 million images and 11,166 categories, then trained ResNet-101 with multi-label outputs.

    Source: §1 · pp. 1–2

  • Results

    Transfer learning across image classification, object detection, and semantic segmentation supports the quality of Tencent ML-Images and its pre-trained checkpoint.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Tencent ML-Images is a public multi-label resource whose checkpoint shows useful visual representations across three transfer-learning tasks.

    Source: §1 · p. 2, §6 · p. 9

    The Tencent ML-Images checkpoint fine-tuned on VOC performed worse than other checkpoints, which the authors attribute to differences in data distributions and tasks.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

In existing visual representation learning tasks, deep convolutional neural networks (CNNs) are often trained on images annotated with single tags, such as ImageNet. However, a single tag cannot describe all important contents of one image, and some useful visual information may be wasted during training. In this work, we propose to train CNNs from images annotated with multiple tags, to enhance the quality of visual representation of the trained CNN model. To this end, we build a large-scale multi-label image database with 18M images and 11K categories, dubbed Tencent ML-Images. We efficiently train the ResNet-101 model with multi-label outputs on Tencent ML-Images, taking 90 hours for 60 epochs, based on a large-scale distributed deep learning framework,i.e.,TFplus. The good quality of the visual representation of the Tencent ML-Images checkpoint is verified through three transfer learning tasks, including single-label image classification on ImageNet and Caltech-256, object detection on PASCAL VOC 2007, and semantic segmentation on PASCAL VOC 2012. The Tencent ML-Images database, the checkpoints of ResNet-101, and all the training codehave been released at https://github.com/Tencent/tencent-ml-images. It is expected to promote other vision tasks in the research and industry community.

1 INTRODUCTION

The introduction argues that single tags can miss important image content, while many vision tasks lack enough training data and need useful pretrained representations. It presents Tencent ML-Images and a ResNet-101 training pipeline, then points to transfer learning across three vision tasks as evidence of representation quality.

1 INTRODUCTION

  • Single-label annotations can miss useful information in images containing multiple objects, while collecting task-specific data for detection and segmentation can be costly.The introduction says visually similar images may receive different single labels even when they contain the same objects.Source: §1 · p. 1
  • JFT-300M improved ImageNet transfer over training from scratch, but its training took two months for four epochs and its data and checkpoint were unavailable publicly.The introduction cites 79.2% top-1 accuracy for the JFT-pretrained ResNet-101 and 77.5% for the ImageNet-trained model.Source: §1 · p. 1
  • Tencent ML-Images contains about 18 million images and 11,166 categories, built by merging Open Images and ImageNet vocabularies and removing rare or redundant categories.The authors also construct a WordNet-based semantic hierarchy and use category co-occurrence to augment source annotations.Source: §1 · pp. 1–2
  • The authors train ResNet-101 with a loss designed for severe class imbalance and distributed training, completing 60 epochs in 90 hours on 128 GPUs.The loss combines weighted cross-entropy, adaptive loss weighting, and down-sampling of negative training images in each minibatch.Source: §1 · p. 2
  • Transfer learning across image classification, object detection, and semantic segmentation yielded better results with Tencent ML-Images-pretrained checkpoints than with JFT-300M- or ImageNet-pretrained checkpoints.The introduction presents these comparisons as evidence for the quality of the database and trained checkpoints.Source: §1 · p. 2
  • The authors released Tencent ML-Images, trained ResNet-101 checkpoints, and the data preparation, pre-training, and fine-tuning code.The paper states that the release is expected to promote other vision tasks in research and industry.Source: §1 · p. 2

2 RELATED WORK

Related work contrasts single-label databases with multi-label resources used for representation learning. Earlier multi-label datasets are described as limited in scale, while larger datasets such as JFT-300M are costly to train on and unavailable to the public.

2 RELATED WORK

  • Single-label databases annotate each image with one tag, whereas multi-label databases annotate each image with multiple tags.The section discusses databases used for visual representation learning in these two categories.Source: §2 · p. 2
  • ImageNet is a widely used single-label pretraining source, but the section argues that one label may omit image content and confuse models when visually similar images receive different labels.It also notes that scene-category representations may not suit object recognition or detection.Source: §2 · p. 2
  • Earlier multi-label datasets such as MS COCO and PASCAL VOC 2007 are described as too small in scale and category vocabulary for training broadly generalizable VGG or ResNet representations.The section distinguishes these from larger multi-label databases.Source: §2 · p. 2
  • JFT-300M contains 300 million images and 18,291 categories, but its ResNet-101 training took two months for four epochs and its data and checkpoints were not published.Tencent ML-Images is presented as publicly available and trained more efficiently.Source: §2 · p. 3

3 THE TENCENT ML-IMAGES DATABASE

Tencent ML-Images combines ImageNet and Open Images into a multi-label database, with tags augmented using semantic hierarchy and category co-occurrence. Its category counts and per-image tag counts are highly imbalanced, and the authors choose an annotation setting with more missing but less noisy tags.

3.1 Image Source and Class Vocabulary

  • 17,609,752 training and 88,739 validation images span 11,166 categories, merged from ImageNet and Open Images.ImageNet categories were filtered and expanded; Open Images categories with fewer than 650 images and some abstract or redundant categories were removed.Source: §3.1 · p. 3

3.2 Tag Augmentation of Images

  • Tags are augmented with semantic ancestors and co-occurring categories, adding a category when co-occurrence exceeds 0.5 and no semantic path links the pair.The co-occurrence estimates use predictions above 0.95 from an Open Images-trained model; ancestor categories are also labeled positive.Source: §3.2 · pp. 3–4

3.3 Data Statistics

  • Category counts range from 0 to 13,217,523 images (mean 13,843), while images average 8.72 tags; the authors choose more missing but less noisy annotations.The number of tags per training image ranges from 1 to 91; noisy annotations remain in the database, and most missing tags come from ImageNet-11K.Source: §3.3 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

4 VISUAL REPRESENTATION LEARNING ON TENCENT ML-IMAGES

This section describes a ResNet-101 multi-label training pipeline for Tencent ML-Images, combining an imbalance-aware loss, image augmentation, and distributed optimization. It also defines top-k instance-level evaluation and cautions that the validation scores are unreliable.

4.1.1 Model and Loss Function

  • The weighted cross-entropy loss addresses label imbalance using higher positive-label cost and adaptive weights across sequential minibatches.The positive-label cost parameter is set to 12; loss weights decay for categories updated in sequential minibatches.Source: §4.1.1 · pp. 4–5

4.1.2 Image Pre-processing

  • Image preprocessing combines random crops and resizing with horizontal flips, rotations, color shifts, and pixel rescaling.Crops may cover as little as 0.05 of the image area, and rotations are sampled from −45 to 45 degrees.Source: §4.1.2 · p. 5

4.1.3 Training Algorithm and Hyper-parameters

  • Training uses momentum SGD with a batch size of 4,096, an eight-epoch learning-rate warm-up, and a 60-epoch schedule.The learning rate warms from 0.01 to 0.08, then decays by a factor of 0.1 every 25 epochs.Source: §4.1.3 · p. 5

4.1.4 Acceleration by Distributed Training

4.2 Evaluations and Results

  • Evaluation reports instance-level precision, recall, and F1 from top-k predictions, but the small validation set’s scores are considered unreliable.The validation set is about 1/1,200 the size of the training set, and many validation labels may be missing.Source: §4.2 · p. 6, Table 1 · p. 6

5 TRANSFER LEARNING

The paper evaluates Tencent ML-Images representations through transfer learning for ImageNet and Caltech-256 classification, PASCAL VOC object detection, and semantic segmentation. Across these tasks, the reported results favor carefully fine-tuned Tencent ML-Images checkpoints.

5.1.1 Fine-Tuning Approaches

  • Because Tencent ML-Images and ImageNet differ in feature and class distributions and annotation noise, adaptive fine-tuning changes top layers more than bottom layers; consistent rates can underperform the baseline.The authors describe bottom layers as representing low-level visual features and top-layer parameters as more dependent on training images and labels.Source: §5.1.1 · pp. 6–7, §5.1.3 · p. 7

5.1.2 Comparisons and Hyper-parameters

  • The ImageNet comparison evaluates five transfer checkpoints alongside scratch-trained MSRA and Google baselines and Google checkpoints pretrained on JFT-300M.The five authors’ checkpoints use consistent or adaptive layer-wise learning rates and differing fine-tuning image sizes.Source: §5.1.2 · p. 7, Table 2 · p. 7

5.1.3 Results

  • 80.73% top-1 and 95.5% top-5 accuracy: adaptive-rate ckpt-4 with adaptive image sizes exceeded the ImageNet baseline and Google ckpt-2.The reported gains over the authors’ baseline were 1.73% top-1 and 1% top-5 accuracy.Source: §5.1.3 · p. 7, Table 3 · p. 8

5.2 Transfer Learning to Caltech-256

  • 86.5% accuracy: the adaptively fine-tuned Tencent ML-Images checkpoint exceeded the ImageNet checkpoint’s 86% on Caltech-256.The Tencent ML-Images-only checkpoint scored below the ImageNet checkpoint.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Transfer Learning to Object Detection

  • 1.4% improvement over the authors’ ImageNet baseline: the Tencent ML-Images-pretrained checkpoint achieved similar VOC detection generalization to Google’s JFT-300M-based checkpoints.The authors report Google ckpt-2 and ckpt-3 as slightly below their Tencent-pretrained ckpt-2.Source: §5.3 · p. 8, Table 5 · p. 9

5.4 Transfer Learning to Semantic Segmentation

  • Up to 2.3% improvement over the authors’ baseline: Tencent ML-Images pretraining improved VOC segmentation, with generalization better than Google’s JFT-300M checkpoint and similar to its JFT-300M-plus-ImageNet checkpoint.The authors report Google ckpt-2 at 75.3% and Google ckpt-3 at 76.5%.Source: §5.4 · p. 9, Table 6 · p. 9

6 CONCLUSIONS

The conclusion presents Tencent ML-Images as a large-scale multi-label database and reports that transfer-learning experiments support the quality of its learned representations. The database, code, and ResNet-101 checkpoints are released for research and industry use.

6 CONCLUSIONS

  • Tencent ML-Images contains about 18M images and 11K categories, and the authors describe it as the largest-scale public multi-label image database at the time of writing.The work also presents large-scale visual representation learning with ResNet-101 and a loss function designed to alleviate severe class imbalance.Source: §6 · p. 9
  • Transfer-learning experiments on image classification, object detection, and semantic segmentation support the authors’ conclusion that the database is high quality and its pretrained checkpoint has good visual representation.Source: §6 · p. 9
  • The authors released the database, data-preparation and training code, and pretrained and fine-tuned ResNet-101 checkpoints.They state that the release is intended to promote vision tasks in research and industry.Source: §6 · p. 9

Related papers

Loading 1901.01703v7…