Source-linked AI summary
A Corpus for Multilingual Document Classification in Eight Languages
Holger Schwenk, Xian Li
TL;DR
Cross-lingual document classification lacks a broadly multilingual, balanced evaluation resource. The paper introduces MLDoc, an eight-language RCV2-based framework with balanced class coverage and multilingual word- and sentence-embedding baselines; sentence embeddings are overall more robust in zero-shot transfer, while targeted transfer can substantially improve results.
Problem
Existing RCV2 benchmarks cover few languages and have substantially different class-prior distributions, complicating evaluation of multilingual transfer.
Method
The paper constructs a balanced eight-language RCV2 evaluation framework and evaluates multilingual word- and sentence-embedding classifiers under zero-shot and targeted transfer settings.
Results
Multilingual sentence-embedding systems are overall more robust in zero-shot transfer, while using target-language development data can produce important improvements, particularly when training on Italian.
Takeaways & Limitations
MLDoc provides a freely available framework with baseline results for cross-lingual document classification across all eight languages.
Takeaways & Limitations
Spanish and Russian lack enough data for the full requested training sizes, and Japanese is excluded from one comparison because joint sentence embeddings are unavailable.
Abstract
from arXiv · showhide
Cross-lingual document classification aims at training a document classifier on resources in one language and transferring it to a different language without any additional resources. Several approaches have been proposed in the literature and the current best practice is to evaluate them on a subset of the Reuters Corpus Volume 2. However, this subset covers only few languages (English, German, French and Spanish) and almost all published works focus on the the transfer between English and German. In addition, we have observed that the class prior distributions differ significantly between the languages. We argue that this complicates the evaluation of the multilinguality. In this paper, we propose a new subset of the Reuters corpus with balanced class priors for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover languages which are very different with respect to syntax, morphology, etc. We provide strong baselines for all language transfer directions using multilingual word and sentence embeddings respectively. Our goal is to offer a freely available framework to evaluate cross-lingual document classification, and we hope to foster by these means, research in this important area.
1. Introduction
Cross-lingual document classification transfers a classifier trained in one language to another without target-language resources. The paper positions language-independent word or document representations as the basis for this transfer and seeks a clear multilingual evaluation environment.
- 1. Introduction: Cross-lingual document classification transfers a classifier trained in one language to another without requiring resources in the transfer language.The approach relies on representations of words or whole documents that are independent of language.
- 1. Introduction: The paper compares multilingual word embeddings with multilingual sentence embeddings for highly multilingual document classification.
2. Corpus description
The paper builds an RCV2-based corpus and evaluation framework that addresses language coverage and class-prior imbalance in existing cross-lingual benchmarks. MLDoc samples uniform class coverage across eight languages, provides multiple data sizes and splits, and supplies extraction tools rather than redistributing RCV2.
- 2. Corpus description: MLDoc uses the multilingual RCV2 corpus, whose 487,000 news stories span thirteen languages and four hierarchical top-level categories.The full RCV2 corpus cannot be redistributed, so the authors provide tools to extract MLDoc subsets.
- 2.1. Cross-lingual document classification: Existing RCV2 evaluation subsets focused mainly on English–German transfer, while their class-prior distributions varied across languages and could produce misleading transfer comparisons.Reported prior work covered English, German, French, and Spanish, with earlier transfer accuracies exceeding 90% for English/German but falling below 80% for EN/FR and 70% for EN/ES.
- 2.2. Multilingual document classification: MLDoc samples the same number of examples for each class and language to provide uniform class coverage.
- 2.2. Multilingual document classification: The corpus supports German, English, Spanish, French, Italian, Japanese, Russian, and Chinese, with reference baselines for all languages.
- 2.2. Multilingual document classification: Each language provides training sets of 1k, 2k, 5k, and 10k stories, plus 1k development and 4k test documents.Spanish and Russian have smaller maximum available datasets because insufficient data exists for the full requested sizes.
3. Baseline results
The paper evaluates multilingual word- and sentence-embedding baselines for zero-shot, targeted, and joint multilingual document classification on MLDoc. Sentence embeddings are more robust across languages in zero-shot transfer, targeted transfer can substantially improve results, and joint multilingual training improves all languages over transfer settings.
- 3. Baseline results: The evaluation compares zero-shot systems selected on the source-language development corpus with targeted systems selected using the target-language development corpus.Zero-shot transfer uses no target-language resources, whereas targeted transfer tailors the system to one transfer language.
- 3. Baseline results: The baselines cover complementary approaches: a CNN over multilingual word embeddings and an MLP over multilingual sentence representations.The word-embedding baseline uses MultiCCA embeddings, while the sentence-embedding systems map multiple languages into a shared representation space.
- 3.3. Zero-shot cross-lingual document classification: Multilingual sentence embeddings are more robust overall in zero-shot transfer, scoring best for four of seven languages, while MultiCCA word embeddings score best for three.Sentence embeddings also support strong transfer between very different languages, including Chinese and Russian.
- 3.4. Targeted cross-lingual document classification: Target-language development data can yield important targeted-transfer improvements, particularly when training on Italian.Targeted systems use the target-language development corpus for model selection and are specifically optimized for that transfer language.
- 3.5. Joint multilingual document classification: Joint multilingual training with 200 examples per language leads to important improvements for all languages compared with zero-shot or targeted transfer learning.The classification accuracies are reported in Table 6 for five-language train and test sets.
4. Conclusion
The paper introduces MLDoc, a freely available eight-language framework for cross-lingual document classification. It extends earlier Reuters-based evaluation and provides baseline results for transfer between all eight languages using multilingual word and sentence embeddings.
- 4. Conclusion: MLDoc extends Reuters-based cross-lingual evaluation to eight languages and provides baseline results for transfer between all language pairs.The framework uses multilingual word and sentence embeddings and is freely available through the project repository.