PaperlayerView exact revision on arXiv

Source-linked AI summary

DAF:re: A Challenging, Crowd-Sourced, Large-Scale, Long-Tailed Dataset For Anime Character Recognition

Edwin Arkel Rios, Wen-Huang Cheng, Bo-Cheng Lai

arXiv:2101.08674v1cs.CV

TL;DR

Anime character recognition is challenging in drawn-media domains, and this paper revises DAF and evaluates CNNs and ViTs across datasets and training settings. Results show that model comparisons depend strongly on batch and image size, while further study is needed to use ViT generalization and transfer capacities.

  • Problem

    Anime character recognition is challenging because drawn media differ from natural images, and the prior DAF split yielded only 37.3% testing accuracy for its best ResNet-18/ArcFace model.

    Source: §2.2 · p. 2, §3.1.1 · p. 3

  • Method

    The paper revises DAF as DAF:re and evaluates CNN and ViT classifiers on DAF:re and moeImouto while varying image and batch sizes.

    Source: §1 · p. 1, §3.1.1 · p. 3, §3.2 · p. 3

  • Results

    CNNs severely outperform ViTs at batch size 1024 and image size 224x224 across datasets and pretraining status, while pretrained ViTs do much better at 128x128 and batch 64 and non-pretrained ViTs are competitive.

    Source: §4 · p. 4

  • Takeaways & Limitations

    ViTs are a promising image-classification alternative, but fully using their generalization and transfer-learning capacities requires more study of hyperparameters.

    Source: §6 · p. 4

    Further study is needed on image- and batch-size effects for upstream and downstream tasks, including similar-domain and domain-adaptation settings.

    Source: §5 · p. 4

Abstract

from arXiv · show

In this work we tackle the challenging problem of anime character recognition. Anime, referring to animation produced within Japan and work derived or inspired from it. For this purpose we present DAF:re (DanbooruAnimeFaces:revamped), a large-scale, crowd-sourced, long-tailed dataset with almost 500 K images spread across more than 3000 classes. Additionally, we conduct experiments on DAF:re and similar datasets using a variety of classification models, including CNN based ResNets and self-attention based Vision Transformer (ViT). Our results give new insights into the generalization and transfer learning properties of ViT models on substantially different domain datasets from those used for the upstream pre-training, including the influence of batch and image size in their training. Additionally, we share our dataset, source-code, pre-trained checkpoints and results, as Animesion, the first end-to-end framework for large-scale anime character recognition: https://github.com/arkel23/animesion

1. INTRODUCTION

The paper motivates anime character recognition as part of building multimedia analysis systems for an influential medium, then contributes a revised dataset, model experiments, and released resources.

1. INTRODUCTION

  • Anime’s cultural and economic reach motivates robust multimedia content analysis for more efficient access, digestion, and retrieval.The paper connects this need to content recommendation systems.Source: §1 · p. 1
  • The authors revise DanbooruAnimeFaces as DAF:re to make it more tractable for anime character recognition.They also release the datasets, source code, and pretrained checkpoints.Source: §1 · p. 1
  • Experiments compare ResNets and ViTs on DAF:re and a smaller related dataset, examining ViT generalization and transfer learning across downstream tasks unlike upstream pretraining.The study also examines image-size and batch-size effects.Source: §1 · p. 1

2. BACKGROUND AND RELATED WORK

The background traces the rise of deep learning and Vision Transformers, then identifies drawn media as a distinct and comparatively underexplored setting for character recognition.

2.1. Deep Learning for Computer Vision

  • ImageNet became a de facto image-classification testbed as deep learning benefited from greater computation and data; Vision Transformers apply transformer encoders to image patches and model long-range dependencies.The paper notes ViT’s strong image-classification results across datasets.Source: §2.1 · p. 2

2.2. Computer Vision for Drawn Media

  • Drawn media differs from natural camera imagery, and CNNs’ texture bias rather than shape recognition makes it a challenging computer-vision testbed [14].Source: §2.2 · p. 2
  • Prior drawn-media research emphasized translation, synthesis, generation, and colorization [16] [17], while character recognition was mostly unexplored and manga datasets ranged from small collections to Manga109 and larger-scale recognition.Manga109 is not fully suited to anime recognition because manga is usually grayscale, unlike anime’s varied palettes and styles.Source: §2.2 · p. 2
  • Cartoon Face [21] provides a related recognition dataset, whereas this work frames recognition as standard K-label classification and embraces crowdsourced noise and stylistic variation as challenges.The authors argue that crowdsourcing also makes dataset updates and adaptation to other tasks more feasible.Source: §2.2 · p. 2

3. METHODOLOGY

The methodology describes the source datasets, the filtering and splitting used to produce DAF:re, and the model and training configurations used for experiments.

3.1. Data

  • DAF:re’s histogram for its 100 largest classes is described as long-tailed; a corresponding top-100-class histogram is also provided for moeImouto.Source: Figure 2 · p. 2, Figure 3 · p. 3

3.1.1. DAF:re

  • Prior DAF filtered Danbooru2018-derived images to single character tags, extracted YOLOv3-based head boxes, and confidence-filtered the resulting dataset to 561K images across 34K classes.Its best ResNet-18 with ArcFace achieved 37.3% testing accuracy; the authors attribute the difficulty to noisy, long-tailed, few-shot classification and drawn-media challenges.Source: §3.1.1 · p. 3
  • DAF:re retains classes with at least 20 samples and uses a 0.7/0.1/0.2 train-validation-test split, yielding 463,437 head images across 3,263 classes.The authors selected the threshold to ensure at least one sample of each class in every split while keeping the dataset challenging.Source: §3.1.1 · p. 3

3.1.2. moeImouto

  • moeImouto contains 14,395 RGB head images across 173 classes, split into training and testing sets at 0.8 and 0.2.Source: §3.1.2 · p. 3

3.2. Experiments

  • Experiments vary training epochs, image sizes, and batch sizes across shallow CNN, ResNet-18/152, and ViT configurations, using SGD with momentum and learning-rate decay.Images are normalized and randomly flipped and cropped during training; validation and testing use resizing and normalization.Source: §3.2 · pp. 3–4
  • The reported tables present classification accuracy for DAF:re at 50 epochs and moeImouto at 200 epochs, both using batch size 64 and 128x128 images.Source: Table 1 · p. 4, Table 2 · p. 4

4. RESULTS AND DISCUSSION

The section compares CNN and ViT classification results across image sizes and batch sizes. CNNs strongly outperform ViTs at 224×224 images and batch size 1024, while ViTs fare better at 128×128 and batch size 64.

4. RESULTS AND DISCUSSION

  • The reported performance metrics are validation and testing top-1 and top-5 classification accuracies.The section refers to the shallow architecture as SN, ResNet-18 as R-18, ResNet-152 as R-152, and ViTs by configuration.Source: §4 · p. 4
  • At batch size 1024 and image size 224×224, CNN architectures severely outperform ViTs across datasets, whether or not the models are pretrained.The section uses validation and testing top-1 and top-5 classification accuracy to assess performance.Source: §4 · p. 4
  • At image size 128×128 and batch size 64, pretrained ViTs obtain much better results, while non-pretrained ViTs obtain competitive results.The comparison is reported across the datasets considered.Source: §4 · p. 4

5. FUTURE WORK

The authors describe ways to extend DAF:re and identify future work on bounding boxes and ViT hyperparameter effects. The cited tables cover DAF:re and moeImouto under a 200-epoch, batch-1024, 224×224 setting.

5. FUTURE WORK

  • DAF:re can be adjusted to include more or fewer images per class and augmented with Danbooru2019 updates following the original DAF methodology.The passage describes these as possible modifications to the dataset.Source: §5 · p. 4
  • The cited classification-accuracy tables cover DAF:re and moeImouto after 200 epochs at batch size 1024 and image size 224×224.The captions identify the datasets and training settings, but provide no accuracy values in these passages.Source: Table 3 · p. 4, Table 4 · p. 4
  • DAF:re omits bounding boxes to keep this initial version manageable, with adding them identified as a next step.The original DAF included bounding boxes.Source: §5 · p. 4
  • A detailed study of ViT image-size and batch-size effects remains needed across upstream and downstream tasks, including similar-domain and domain-adaptation settings.Source: §5 · p. 4

6. CONCLUSION

The authors present DAF:re for studying anime character recognition and report experiments on DAF:re and moeImouto. They conclude that ViTs are a promising alternative to CNNs, while further hyperparameter study is needed to use their generalization and transfer-learning capacities fully.

6. CONCLUSION

  • The paper presents DAF:re as a dataset for studying anime character recognition.Source: §6 · p. 4
  • The authors conduct experiments on DAF:re and moeImouto using a variety of models.Source: §6 · p. 4
  • ViTs are a promising alternative to CNNs for image classification, but more hyperparameter study is needed to fully use their generalization and transfer-learning capacities.Source: §6 · p. 4

7. DISCLAIMER

The dataset was created to support computer-vision research on anime multimedia systems. It supplies thumbnails and does not own the images’ copyright.

7. DISCLAIMER

  • DAF:re was created to enable computer-vision study of anime multimedia systems.Source: §7 · p. 4
  • The dataset provides image thumbnails and does not own the images’ copyright.The passage compares this approach to ImageNet.Source: §7 · p. 4
  • The disclaimer describes the dataset’s contents as thumbnails rather than the original images.Source: §7 · p. 4

Related papers

Loading 2101.08674v1…