Source-linked AI summary
STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
Yuya Yoshikawa, Yutaro Shigeto, Akikazu Takeuchi
TL;DR
Japanese caption datasets are scarce, and English-to-Japanese translation can produce literal captions that do not reflect image information. The paper constructs STAIR Captions from MS-COCO images and evaluates neural caption generation trained with Japanese captions. It reports better Japanese captions than the English-generation-and-translation baseline.
Problem
Japanese caption datasets are scarce, while English-to-Japanese translation may produce literal, unnatural captions because image information is not reflected in translation.
Source: §1 · p. 1Method
Results
Ja-generator outperformed En-generator →MT, and the authors report qualitatively and quantitatively better Japanese captions with STAIR Captions.
Source: §1 · p. 1, §5.2 · p. 4Takeaways & Limitations
The authors conclude that Japanese captions can be generated by adapting an existing caption-generation method and that Japanese training captions are necessary in their experiment.
Source: §6 · p. 4The reviewed setup uses LSTM rather than the original paper's RNN, based on the original appendix's report that LSTM performed better.
Source: §4 · p. 3
Abstract
from arXiv · showhide
In recent years, automatic generation of image descriptions (captions), that is, image captioning, has attracted a great deal of attention. In this paper, we particularly consider generating Japanese captions for images. Since most available caption datasets have been constructed for English language, there are few datasets for Japanese. To tackle this problem, we construct a large-scale Japanese image caption dataset based on images from MS-COCO, which is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions for 164,062 images. In the experiment, we show that a neural network trained using STAIR Captions can generate more natural and better Japanese captions, compared to those generated using English-Japanese machine translation after generating English captions.
1 Introduction
Japanese image captioning is constrained by the scarcity of Japanese caption datasets, while translating English captions may produce literal, unnatural text. The paper introduces STAIR Captions and reports that a neural network trained on it produces better Japanese captions than English-caption generation followed by translation.
1 Introduction
- Few Japanese caption datasets were available, and translating English captions could yield literal, unnatural Japanese because the translation cannot reflect image information.The paper seeks more natural Japanese captions for images.Source: §1 · p. 1
- STAIR Captions provides Japanese captions for all MS-COCO images, creating a large-scale dataset for Japanese image captioning.The paper cites MS-COCO as Lin et al. (2014).Source: §1 · p. 1
- A neural network trained on STAIR Captions generated quantitatively and qualitatively better Japanese captions than English captions translated by machine translation.The comparison is against captions produced by first generating English and then translating into Japanese.Source: §1 · p. 1
2 Related Work
Related work includes English caption datasets, a small number of datasets in other languages, and prior work that extended MS-COCO with additional image information. The closest comparison is YJ Captions, another Japanese dataset for MS-COCO images, which covers fewer images than STAIR Captions.
2 Related Work
- English image-caption datasets include PASCAL, Flickr3k, Flickr30k, and MS-COCO [Lin et al., 2014].The passage also names additional English caption-dataset studies.Source: §2 · pp. 1–2
- MS-COCO serves image-classification and caption-generation research, and later studies added other image annotations to it.The passage describes it as a benchmark for image classification and caption generation.Source: §2 · p. 2
- A few caption datasets exist in other languages; YJ Captions is the closest Japanese comparator, but STAIR Captions covers more MS-COCO images.The paper says it compares the datasets' statistics in Section 3.Source: §2 · p. 2
3 STAIR Captions
STAIR Captions supplies five Japanese captions per image across the 2014 MS-COCO edition, with test-image captions excluded from the public release. Its annotation combines worker-written descriptions under explicit content and style rules with inspection, and its statistics show substantially greater scale than YJ Captions.
3.1 Annotation Procedure
- The dataset covers all 164,062 images in the 2014 MS-COCO edition, with five Japanese captions per image; test-image captions are excluded from the public release.The resulting total is 820,310 captions.Source: §3.1 · p. 2
- About 2,100 workers completed annotation in about half a year, following single-sentence, da/dearu-style rules and describing only visible content without opinions or emotions.Captions had to exceed 15 letters, and samples were inspected to remove captions that violated the guidelines.Source: §3.1 · p. 2
3.2 Statistics
- STAIR Captions has 6.23x as many total Japanese captions and 6.19x as many images as YJ Captions.The comparison covers overall dataset totals; Table 1 presents dataset specifications.Source: §3.2 · p. 2, Table 1 · p. 3
- STAIR Captions has a 2.69x larger vocabulary than YJ Captions, while their average sentence lengths in characters are almost the same.The paper expects the larger vocabulary to support a wider range of generated captions.Source: §3.2 · p. 3
4 Image Caption Generation
The reviewed image-captioning approach uses a CNN to extract image features and an LSTM to generate a caption sequence. The paper uses LSTM because the original study reported better performance than RNN in its appendix.
4 Image Caption Generation
- The method pairs a CNN image-feature extractor with an LSTM that generates the caption from those features.The paper reviews the approach of Karpathy and Fei-Fei (2015) for its experiments.Source: §4 · p. 3
- Caption generation maps an image to a word sequence, with output words generated until the end-of-sentence symbol appears.At each time step, the input is the embedding of the previous output word.Source: §4 · p. 3
- The experiments use LSTM rather than the original paper's RNN because its appendix reported better LSTM performance.The training phase fits the image and sequence-model parameters using the training data.Source: §4 · p. 3
5 Experiments
The experiments assess whether a neural captioning model trained on Japanese captions can generate fluent Japanese, compared with translating generated English captions. Ja-generator outperformed the translation baseline, and examples illustrate more appropriate Japanese phrasing.
5.1.1 Evaluation Measure
- BLEU, ROUGE, and CIDEr were used to evaluate caption quality.BLEU and ROUGE were originally developed for machine translation and text summarization, but are often used to measure caption quality.Source: §5.1.1 · p. 3
5.1.2 Comparison Methods
- The comparison used En-generator →MT as a machine-translation baseline and Ja-generator trained directly on STAIR Captions.Both models used the caption-generation method of Karpathy and Fei-Fei.Source: §5.1.2 · p. 3
5.1.3 Dataset Separation
- The experiment split 123,287 MS-COCO images into 113,287 training, 5,000 validation, and 5,000 test images.Hyperparameters were tuned on validation CIDEr scores, and Japanese captions were morphologically analyzed with MeCab.Source: §5.1.3 · p. 4
5.2 Results
- Ja-generator, trained with Japanese captions, outperformed En-generator →MT, the English-caption-generation and machine-translation baseline.Table 2 summarizes the experimental results; the supplied passages do not give its scores.Source: §5.1.2 · p. 3, §5.2 · p. 4, Table 2 · p. 4
- In two examples, Ja-generator produced appropriate Japanese captions where En-generator →MT yielded word-by-word or incorrect translations.The examples contrast “二階建てのバス” with “二重デッカーバ ス” and a donut caption with “食べ物の束.”Source: §5.2 · p. 4, Table 3 · p. 4
6 Conclusion
The paper introduces STAIR Captions, a large Japanese caption dataset for MS-COCO images, and reports that using it supports better Japanese caption generation than the English-caption translation approach. Future work will analyze results further and develop multilingual caption generation models.
6 Conclusion
- STAIR Captions provides 820,310 Japanese captions for all images in MS-COCO, and the authors state that, to their knowledge, it is currently the largest Japanese image-caption dataset.Source: §6 · p. 4
- The experiment found that a neural captioning model using STAIR Captions generated quantitatively and qualitatively better Japanese captions than English-caption translation.The authors also report the necessity of Japanese captions and that an existing caption-generation method can be adapted to generate Japanese captions.Source: §6 · p. 4
- Future work will analyze the experimental results in greater detail and develop multilingual caption-generation models using Japanese and English captions.Source: §6 · p. 4