PaperlayerView exact revision on arXiv

Source-linked AI summary

COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval

Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, Jieping Xu

arXiv:1805.08661v2cs.CLcs.CV

TL;DR

Cross-lingual image annotation lacks sufficiently broad Chinese resources and effective methods. The paper introduces COCO-CN, a manually annotated Chinese-English extension of MS-COCO with recommendation-assisted collection and task-specific baselines. Its dataset and methods support cross-lingual tagging, captioning, and retrieval, with experiments establishing their viability.

  • Problem

    Chinese cross-lingual image resources are limited in scale or scope, and effective methods for cross-lingual image tagging and mixed manual/machine-translated caption data remain underdeveloped.

  • Method

    The paper builds COCO-CN by adding manually written Chinese sentences and tags to MS-COCO, using recommendation-assisted annotation and task-specific cross-lingual baseline models.

  • Results

    Experiments across image tagging, captioning, and retrieval justify the viability of COCO-CN and its baseline methods.

  • Takeaways & Limitations

    COCO-CN provides a unified platform for advancing cross-lingual image annotation and retrieval research.

Abstract

from arXiv · show

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For more effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20,342 images annotated with 27,218 Chinese sentences and 70,993 tags, COCO-CN is currently the largest Chinese-English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn

I. INTRODUCTION

The paper identifies gaps in Chinese cross-lingual image resources and methods, then introduces COCO-CN, a manually annotated extension of MS-COCO supported by recommendation-assisted collection and task-specific baselines.

  • Research gaps: Existing Chinese datasets are either small or biased toward human activities, while effective cross-lingual image tagging remains unclear.The paper also notes limited work combining manually written and machine-translated target-language sentences.
  • Dataset contribution: COCO-CN extends MS-COCO with manually written Chinese sentences and tags for cross-lingual image annotation and retrieval.The dataset targets Chinese alongside existing English annotations and is described as the largest Chinese-English dataset in this setting.
  • Annotation system: The annotation system recommends relevant sentences and tags during remote, independent manual annotation.User clicks and edits are logged, and the system uses separate content-based recommendation modules for sentences and tags.
  • Task-specific methods: The paper evaluates cross-lingual image tagging, captioning, and retrieval using Cascading MLP, Sequential Learning, and Enhanced Word2VisualVec.These models are designed to learn from bilingual resources in the three target tasks.

II. RELATED WORK

Prior Chinese image datasets differ in scale, language coverage, or visual scope, motivating COCO-CN's broader cross-lingual and multi-task design.

  • Chinese datasets: Flickr8k-CN reannotates Flickr8k with five Chinese sentences per image, but its sentences average only 8 Chinese characters.The dataset was created through local crowdsourcing without reported annotation guidelines or quality control.
  • Chinese datasets: Flickr30k-CN provides Chinese annotation only for the Flickr30k test set by translating 5,000 English sentences.This limits its manually annotated coverage relative to a full training resource.
  • Chinese datasets: AIC-ICC contains 240k Internet images and 1.2 million crowdsourced sentences but is monolingual and focused on human activities.Models trained on it tend to generate someone-doing-something descriptions even when test images contain no person.
  • COCO-CN: COCO-CN combines Chinese and English sentences with tags, supporting cross-lingual image tagging, captioning, and retrieval.Its tags also support multi-label image classification and multimodal tasks such as tag-assisted captioning.
  • Annotation construction: COCO-CN uses recommendation modules to assist manual sentence and tag annotation in a web-based system.The system is designed for remote independent participation, while its recommendation models are developed specifically for Chinese content.

2) Chinese Sentence Recommendation:

The system recommends Chinese sentences and tags from image content to assist manual annotation. Sentence retrieval uses W2VV, while tag recommendation learns from a large translated Flickr resource.

  • Sentence recommendation: The sentence module retrieves the most relevant Chinese sentence from a 120k-sentence machine-translated pool.The pool is translated from MS-COCO English sentences using Baidu translation.
  • Sentence recommendation: The system recommends sentences for editing rather than direct selection, encouraging personal descriptions and supporting novel images without aligned bilingual annotations.This design deliberately preserves unaligned bilingual descriptions as a practical learning setting.
  • Sentence recommendation: Annotators receive the top five retrieved sentences after image–sentence relevance is computed on the fly.W2VV encodes text and predicts a visual feature vector for cross-modal matching.
  • Tag recommendation: The tag module predicts multiple Chinese tags using one million Flickr images whose English tag vocabulary was manually translated and filtered.Tags lacking correspondence to specific objects, scenes, or events were excluded.
  • Tag recommendation: The resulting Chinese tag vocabulary contains 1,951 classes covering 2,085 English and 2,367 Chinese words.The Flickr training data were denoised by removing overly tagged images and images with fewer than two vocabulary tags, yielding 800k images.

B. COCO-CN Annotation Process

COCO-CN uses guided, independently performed annotation with inspection-based quality control, producing diverse Chinese sentences and tags that add concepts beyond translated MS-COCO captions.

  • Annotation guidelines: Annotators were instructed to cover the main objects, actions, and scene, and to provide at least one complementary tag.Examples of well- and badly labeled data supported the guidelines.
  • Dataset statistics: The sentence vocabulary has 7,096 distinct words, while 4,867 distinct tags include 1,918 words outside the sentence vocabulary.Manual tags therefore capture concepts not fully represented by the sentences.
  • Quality control: Nearly 10% of images were double-checked, and annotators entering the watch list three times were excluded with their annotations removed.The inspection committee sampled annotation results for manual quality checks.
  • Dataset scale: 20,342 images contain 22,218 Chinese sentences and 70,993 tags, with randomly assigned images forming a representative MS-COCO subset.An additional 5,000 MS-COCO English sentences were manually translated into Chinese for captioning evaluation.
  • Annotation diversity: 45.3% of keywords in a COCO-CN sentence are novel relative to translated MS-COCO sentences, including 39.9% of nouns, 51.9% of verbs, and 69.5% of adjectives.The comparison uses nouns, verbs, and adjectives as keywords.
  • Annotation quality: Human annotations differ substantially from machine translations, whose BLEU-4, METEOR, ROUGE-L, and CIDEr scores are 16.1, 27.6, 42.9, and 60.0.The human vocabulary contains 7,096 words versus 14,219 in the five-times-larger machine-translated corpus.

2) The Influence of Recommendation:

Recommendation reduced annotation effort, with tags accepted more often than sentences and sentence recommendations lowering the average edit distance from 16.8 to 15.0.

  • Tag recommendation: Annotators accepted approximately 2.7 of five suggested tags per image, for a 54% acceptance rate.The tag ‘猫’ was accepted 649 out of 828 times.
  • Sentence recommendation: 31% of 22,218 sentences were typed after annotators clicked recommendations, including 5,796 edited sentences.Sentence recommendation was less effective than tag recommendation.
  • Sentence recommendation: The average Levenshtein distance decreased from 16.8 without recommendation to 15.0 with recommendation.The distance counts single Chinese-character insertions, deletions, or substitutions needed to transform one sentence into another.
  • Experimental setup: The shared experiments use random 1,000-image validation and test sets, with the remaining 18,342 images for training.A 100-image subset of the test set was reserved for human evaluation.

B. Task I: Cross-Lingual Image Tagging

Cascading MLP transfers English tag semantics into Chinese prediction by combining an English-tag semantic feature with image features. It tops automated and human tagging evaluations and exploits complementary cross-lingual data without requiring aligned bilingual annotations.

  • Model: Cascading MLP first predicts English tags, then concatenates that output with the image CNN feature to predict Chinese tags.The architecture consists of two sequential MLPs.
  • Model: The first MLP is trained on MS-COCO to transform image features into a 512-dimensional English-tag semantic feature.The English vocabulary contains 512 tags selected from MS-COCO captions.
  • Evaluation: Automated evaluation reports Precision, Recall, and F-measure at top 5, averaged per image across test images.These are the criteria used to compare the five tagging models.
  • Automated results: Cascading MLP tops automated tagging performance, while COCO-CN alone outperforms COCO-MT despite COCO-MT using 6.6 times as many training images.The authors attribute the advantage of combining datasets to complementary information.
  • Cross-lingual learning: Cascading MLP outperforms Multi-task MLP and remains more flexible because it does not require aligned bilingual annotations per training image.The same model also performs better than Multi-task MLP on Flickr8k-CN.
  • Human results: Cascading MLP is again the best model in human evaluation on the independent NUS-WIDE100 test set.NUS-WIDE100 contains 100 randomly sampled images independent of the training datasets.

C. Task II: Cross-Lingual Image Captioning

COCO-CN is evaluated for Chinese and cross-lingual image captioning against existing datasets and cross-lingual training strategies. The experiments compare manually written, machine-translated, and combined resources using automatic and human evaluation.

  • Dataset and training setup: Each COCO-CN test image has at least one manually written Chinese sentence plus five manually translated Chinese references.The resulting test ground truth contains six Chinese sentences per image.
  • Dataset and training setup: COCO-CN is compared with Flickr8k-CN, AIC-ICC, and machine-translated COCO-MT using separately trained Show and Tell models.Training minimizes cross-entropy loss under a standard supervised-learning setup.
  • Cross-lingual training: Sequential Learning first trains on COCO-MT and then continues training on COCO-CN to combine larger vocabulary coverage with manually written sentence quality.The method uses a joint vocabulary and is framed as a more effective use of cross-lingual resources than existing alternatives.
  • Cross-lingual training: The comparison includes Transfer learning and Artificial token models, previously proposed for learning from cross-lingual resources.Both alternatives were originally developed for Japanese caption generation and are implemented here for Chinese captioning.
  • Evaluation: Caption quality is assessed with BLEU, METEOR, ROUGE-L, and CIDEr, supplemented by human ratings of relevance and fluency.The paper notes that automatic metrics have weak correlations with human judgments.

2) Results:

COCO-CN supports more general Chinese captioning than smaller or biased alternatives, while Sequential Learning best leverages machine-translated scale and manually written quality. Human and automatic evaluations favor the proposed cross-lingual approaches in different respects.

  • Automated evaluation: COCO-CN outperforms Flickr8k-CN and AIC-ICC in automated captioning evaluation, while COCO-MT supplies a larger vocabulary but machine-translation errors reduce naturalness.Flickr8k-CN is limited in size, whereas AIC-ICC is biased toward images about human beings.
  • Human evaluation: COCO-MT receives fluency scores of 4.50 on COCO-CN test100 and 4.36 on NUS-WIDE100, the lowest among the compared models.The passage attributes these scores to translation errors affecting accuracy and grammar.
  • Human evaluation: Sequential Learning achieves the best relevance score in both automatic and human evaluation.Its fluency score is 4.76 on COCO-CN test100, below the COCO-CN model there and AIC-ICC on NUS-WIDE100.
  • Qualitative comparison: COCO-CN produces captions covering general visual content, whereas AIC-ICC can describe unrelated human activities because of its dataset bias.A qualitative example contrasts a cat-on-laptop image with AIC-ICC’s prediction about a woman watching a computer.

D. Task III: Cross-Lingual Image Retrieval

The retrieval task maps Chinese queries to images whose descriptions are in English, requiring shared representations across languages and modalities. The system compares text-only, image-only, and combined similarity strategies, with an enhanced W2VV model for sentence-to-visual projection.

  • Retrieval setting: The test set contains 1,000 Chinese queries, one manually written sentence per COCO-CN test image, paired with each image’s first English MS-COCO sentence.Each query has one known relevant image for evaluation.
  • Retrieval models: The three approaches are cross-lingual matching, cross-modal matching, and combined cross-lingual-and-modal matching.Cross-lingual matching uses Chinese queries and English descriptions; cross-modal matching uses Chinese queries and images.
  • Retrieval models: W2VV vectorizes sentences with bag-of-words, Word2Vec, and GRU representations before projecting them into visual feature space.The enhanced model addresses the GRU limitation of relying only on the final hidden vector.
  • Enhanced W2VV: The enhancement adds soft attention after the GRU, assigning adaptive weights to hidden vectors across all time steps.The attention layer sums the hidden vectors using weights computed from each vector.
  • Enhanced W2VV: The retrieval objective uses a margin-based contrastive loss with Euclidean distance between image and sentence vectors, using margin 2.The loss pulls relevant pairs closer and pushes negative pairs farther apart.
  • Similarity computation: Chinese and English models are trained separately to transform sentences into ResNeXt-101 features, whose cosine similarity defines cross-lingual and cross-modal scores.COCO-CN supplies Chinese training sentences, while MS-COCO supplies English sentences for the same images.
  • Retrieval setting: Given a Chinese query, retrieval ranks images using similarities between the query and English descriptions, images, or their convex combination.The combination weight for cross-lingual and cross-modal matching is optimized on the validation set.

2) Results:

Enhanced W2VV improves cross-lingual image retrieval over the original model, and combining cross-lingual with cross-modal evidence performs best. The enhanced model is especially useful for queries requiring multiple visual concepts.

  • Overall retrieval results: Enhanced W2VV outperforms original W2VV, supporting the joint use of attention and contrastive loss in this retrieval setting.The comparison is reported in Table X and concerns the different retrieval approaches.
  • Overall retrieval results: CLM matching achieves the best retrieval performance, while CM matching is less effective than CL matching.CLM combines cross-lingual and cross-modal similarities.
  • Qualitative analysis: Enhanced W2VV better models complex queries requiring cooccurrence of multiple visual concepts.The paper notes an exception where emphasizing the query’s ending phrase, “red ring,” helps the original W2VV.

E. Discussion

The paper positions COCO-CN as a high-quality cross-lingual resource with baseline methods for tagging, captioning, and retrieval, while identifying annotation scalability and future algorithmic work as open issues.

  • Empirical outcome: COCO-CN produced noticeably better models for multiple tasks than existing datasets.The discussion attributes this outcome to high-quality annotations, while noting that crowd sourcing would have been cheaper and more scalable.
  • Annotation system: 54 percent of recommended tags were accepted, indicating that recommendation-assisted annotation reduced annotators’ workload.Sentence recommendation also helped, but was less effective than tag recommendation.
  • Dataset and platform: COCO-CN provides a unified platform for cross-lingual image tagging, captioning, and retrieval with rich bilingual annotations.Its 655 labels and 4.4 labels per image complement its bilingual tags and sentences.
  • Baseline methods: Cascading MLP, Sequential Learning, and enhanced W2VV provide task-specific baselines for cross-lingual tagging, captioning, and retrieval.The authors describe Sequential Learning as the first solution combining machine-translated and manual annotations for cross-lingual image captioning.
  • Evaluation scope: The incomplete test-set ground truth supports qualitative conclusions about which cross-lingual image-tagging model is better.The authors also present enhanced W2VV as a new baseline for cross-lingual image retrieval.

APPENDIX

The appendix examines feature choices and tag-augmented captioning, reporting that ResNeXt-101 performs best across tasks and that tags improve captioning performance.

  • Choice of image features: ResNeXt-101 features perform best across image tagging, captioning, and retrieval compared with GoogLeNet and ResNet-152 features.All three features were evaluated in the same task-specific model settings.
  • Image captioning with tags: Including image tags as input gives better image-captioning performance.The experiment reuses a COCO-CN captioning model and compares captioning with and without tags.
Loading 1805.08661v2…