Source-linked AI summary
AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, Yizhou Wang, Yonggang Wang
TL;DR
Large-scale datasets with rich annotations remain limited for computer vision tasks beyond classification. AIC constructs three large datasets for human keypoint detection, attribute-based zero-shot recognition, and Chinese image captioning, providing a benchmark and pre-training resource whose overlapping annotations support cross-task investigation. The paper also identifies incomplete coverage of human expressions, actions, and non-human captioning images as future annotation directions.
Problem
Large-scale datasets with rich annotations for computer vision tasks beyond classification remain limited, including datasets for human keypoints, attributes, and captions.
Method
AIC constructs three datasets with annotations for keypoints, labels, bounding boxes, attributes, and Chinese captions across human keypoint detection, attribute-based zero-shot recognition, and image captioning.
Results
AIC provides a large-scale benchmark and pre-training resource with substantial overlap among its three datasets, enabling cross-reference between low-level annotations and higher-level captioning.
Takeaways & Limitations
The dataset supports evaluation, model improvement, pre-training, and investigation of relationships between different levels of visual information across related tasks.
Takeaways & Limitations
The human keypoint dataset currently covers skeletal keypoints only, while the Chinese captioning dataset contains images with human beings and lacks other classes.
Abstract
from arXiv · showhide
Significant progress has been achieved in Computer Vision by leveraging large-scale image datasets. However, large-scale datasets for complex Computer Vision tasks beyond classification are still limited. This paper proposed a large-scale dataset named AIC (AI Challenger) with three sub-datasets, human keypoint detection (HKD), large-scale attribute dataset (LAD) and image Chinese captioning (ICC). In this dataset, we annotate class labels (LAD), keypoint coordinate (HKD), bounding box (HKD and LAD), attribute (LAD) and caption (ICC). These rich annotations bridge the semantic gap between low-level images and high-level concepts. The proposed dataset is an effective benchmark to evaluate and improve different computational methods. In addition, for related tasks, others can also use our dataset as a new resource to pre-train their models.
1. Introduction
AIC addresses the shortage of large, richly annotated datasets for image understanding beyond classification by providing three comprehensive tasks: human keypoint detection, attribute-based zero-shot recognition, and image Chinese captioning. Its datasets combine multiple annotation types and substantial scale to support evaluation, pre-training, and cross-task research.
- Motivation: Existing large-scale vision datasets mainly support classification and recognition, while richly annotated datasets for keypoints, attributes, and captions remain limited in scale.Existing human keypoint datasets contain no more than 200k labeled images, and attribute and Chinese-caption resources are also described as small or information-limited.
- Goal: AIC targets three comprehensive tasks: human keypoint detection, attribute-based zero-shot recognition, and image Chinese captioning.These tasks address human activity understanding, learning concepts from descriptions, and modeling relationships between objects in images.
- Dataset construction: The dataset contains 300,000 keypoint images, 81,658 attribute images, and 300,000 captioning images with five Chinese captions per image.More than 95% of the keypoint and captioning images overlap, enabling investigation of jointly handling two visual tasks.
- Contributions: AIC is presented as a benchmark for the three tasks, a resource for pre-training models, and the first large-scale image Chinese captioning dataset.These contributions position the dataset for both method evaluation and model development.
2. Human Skeletal System Keypoint Detection
HKD is a large-scale human keypoint dataset designed for challenging multi-person pose understanding, with rich annotations, diverse poses, and an evaluation metric calibrated to annotation difficulty.
- Overview: HKD contains 300,000 high-resolution images with multiple persons, bounding boxes, and 14 labeled human skeletal keypoints.The dataset is intended to address the limited scale of existing human keypoint datasets.
- Data Collection and Annotation: The annotation pipeline comprises image filtering, human bounding-box labeling, and skeletal-keypoint labeling.Images are filtered for inappropriate content and overly small or indistinct human figures before annotation.
- Data Statistics: The dataset is split into training, validation, test A, and test B sets with ratios of 70%, 10%, 10%, and 10%.The corresponding image counts are 210,000, 30,000, 30,000, and 30,000.
- Data Statistics: The training set contains 378,374 human figures and nearly 5 million keypoints, of which 78.4% are labeled visible.Keypoint-type distributions are reported separately in Figure 2.
- Data Statistics: Human annotation deviation is lower for upper-body keypoints and generally higher for hips, so these deviations are incorporated into prediction-difficulty evaluation.Pose visualization after alignment shows standing and sitting as common poses alongside other configurations.
- Evaluation Metrics: HKD evaluates detections with mAP over AP values at OKS thresholds from 0.50 to 0.95, using weighted keypoint distances scaled by person size and annotation variability.Visibility flags determine which keypoints contribute to OKS.
- Baseline Model and Experiments: OpenPose achieves 0.296 mAP-12 on HKD, while the provided top-down baseline achieves 0.228 mAP-12 and 0.234 mAP-14.The baseline consists of a human detector, keypoint detector, and post-processing procedure.
3. Attribute based Zero-shot Recognition
Existing attribute datasets are limited in scale, while attribute-based zero-shot recognition uses attributes to transfer knowledge from seen to unseen classes. LAD addresses this gap with 81,658 images across 240 classes and 359 visual and semantic attributes, standardized splits, and baseline evaluations.
- Motivation: Existing attribute datasets are small, with CUB, aP&aY, SUN, and AwA totaling only 72k images.These limitations motivate a larger resource for attribute-based zero-shot recognition.
- Dataset: LAD contains 81,658 images, 240 classes, and 359 attributes for attribute-based zero-shot recognition.The classes span animals, fruits, vehicles, electronics, and hairstyles.
- Dataset: The attributes combine low-level visual properties with semantic properties such as animal habits, fruit edibility, vehicle safety, and electronics usage.The dataset defines separate attribute lists for each class subset, totaling 359 attributes.
- Dataset construction: LAD retains single-object images and annotates each foreground object with a bounding box.Images are crawled using class labels and synonyms, then filtered to match the target label.
- Evaluation protocol: Five random splits use 80% seen and 20% unseen classes, while supervised experiments use 70% training and 30% testing data.The authors recommend evaluating zero-shot recognition across all five splits and reporting mean accuracy.
- Results: MDP achieves 48.58% average recognition accuracy, compared with 43.43% for ESZSL and 33.55% for SOC.The five split accuracies are described as balanced, and SOC is around 15% lower than MDP on average.
4. Image Captioning for Chinese
The ICC dataset extends image captioning resources to Chinese by combining large-scale, multi-caption annotation with scenes emphasizing objects, relationships, and actions. Its baseline uses an encoder-decoder model adapted for Chinese, and experiments indicate that ICC-trained captions provide richer context than MSCOCO-trained captions in most cases.
- Existing caption datasets used English, motivating ICC as a large-scale Chinese image captioning resource.
- Data Annotation: ICC provides five Chinese reference captions per image from five native speakers, guided to cover objects, attributes, locations, and human actions.
- Dataset Statistics: ICC contains 1,050,000 training captions for 210,000 images and 450,000 captions across validation and two test sets totaling 90,000 images.
- Dataset Properties: More than 200 scenes and 150 actions cover common daily situations, with selection favoring complex scenes containing multiple objects.
- Baseline Model: The baseline adapts the show-and-tell encoder-decoder model by using Jieba Chinese word segmentation during preprocessing.
- Experimental Results: ICC-trained models produce more descriptive captions than MSCOCO-trained models in most illustrated cases, suggesting ICC captions provide more context information.
5. Conclusion
The paper presents AIC as a rich, large-scale resource spanning human keypoint detection, attribute-based zero-shot recognition, and image Chinese captioning. Its intersecting annotations support benchmarking, model pre-training, and cross-level analysis, while future extensions target richer human annotations and non-human captioning images.
- AIC organizes three large-volume datasets for human keypoint detection, attribute-based zero-shot recognition, and image Chinese captioning.
- Intersecting annotations allow cross-reference between low-level labels and higher-level captioning for evaluating and improving methods.
- Limitations and Future Work: The current HKD annotations cover skeletal keypoints, while ICC images contain human beings; future work proposes expressions, actions, and other image classes.