Source-linked AI summary
The Multilingual Amazon Reviews Corpus
Phillip Keung, Yichao Lu, György Szarvas, Noah A. Smith
TL;DR
Large-scale multilingual text-classification datasets have notable gaps in size, language handling, accessibility, and reproducible splits. MARC addresses these gaps with a curated six-language Amazon review corpus and mBERT baselines for supervised and zero-shot classification, using MAE to reflect ordinal ratings.
Problem
Large-scale multilingual text classification datasets are rare, and existing corpora can be small, inaccessible, lack language identity, or lack reproducible versions and splits.
Method
The paper curates, filters, and processes Amazon reviews in six languages, provides standardized splits, and fine-tunes multilingual BERT for supervised and zero-shot classification baselines.
Results
MARC provides a large public benchmark with many samples in six languages, well-defined training, development, and test splits, and supervised and zero-shot baseline results.
Takeaways & Limitations
MARC is intended as a resource for multilingual text-classification research, while MAE is encouraged because star ratings are ordinal.
Takeaways & Limitations
Existing comparison resources remain constrained: Reuters corpora are proprietary and relatively small, while Yelp lacks language identity and stable downloadable versions.
Abstract
from arXiv · showhide
We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification. The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019. Each record in the dataset contains the review text, the review title, the star rating, an anonymized reviewer ID, an anonymized product ID, and the coarse-grained product category (e.g., 'books', 'appliances', etc.) The corpus is balanced across the 5 possible star ratings, so each rating constitutes 20% of the reviews in each language. For each language, there are 200,000, 5,000, and 5,000 reviews in the training, development, and test sets, respectively. We report baseline results for supervised text classification and zero-shot cross-lingual transfer learning by fine-tuning a multilingual BERT model on reviews data. We propose the use of mean absolute error (MAE) instead of classification accuracy for this task, since MAE accounts for the ordinal nature of the ratings.
1 Introduction
MARC addresses gaps in multilingual text classification resources by releasing a large, multilingual Amazon review corpus with standardized splits and access. It is designed to support supervised and cross-lingual research.
- Motivation: Large-scale multilingual classification datasets are rare, while existing resources are small, inaccessible, lack language identity, or lack reproducible splits.RCV2 averages approximately 37,000 training examples per language; its smallest language has 1,794 examples.
- Contributions: MARC provides 200,000 training reviews for each corpus language.
- Contributions: Language detection algorithms associate reviews with their correct language with high probability.
- Contributions: MARC is distributed on AWS Open Datasets for non-commercial access by research groups.
- Contributions: The corpus uses clearly defined training, development, and test sets unlike previous Amazon review datasets.
2 Data preparation
The corpus was assembled from verified Amazon reviews across six marketplaces, then constrained, language-filtered, vocabulary-filtered, and normalized for release.
- Collection: Reviews came from six marketplaces and languages and were submitted between November 1, 2015 and November 1, 2019.Only reviews with verified purchases were included.
- Sampling: Sampling limits restrict reviews per product and reviewer, require at least two product reviews, and require reviews to contain at least 20 characters.
- Filtering: Language detection retained only reviews written in the target language because marketplace and review languages can differ.
- Filtering: A vocabulary filter excludes reviews containing tokens absent from at least 20 other reviews.Jieba and KyTea were used for Chinese and Japanese segmentation only during filtering.
- Processing: Reviews are truncated at 2,000 characters, with newlines and tabs removed; HTML is rendered as UTF-8.
- Processing: Product and reviewer identifiers are anonymized by mapping them to unique randomly generated integers.
- Processing: The released data includes labels for 30 common product types, while remaining categories are mapped to ‘other’.
3 Corpus Characteristics
The corpus balances star ratings and provides fixed training, development, and test splits, while product-category distributions vary substantially across languages.
- 200,000 training, 5,000 development, and 5,000 test reviews are provided for each language.
- Each of the five star ratings constitutes exactly 20% of the corpus.
- The numbers of unique products and reviewers are broadly similar across languages.
- Chinese reviews are especially concentrated in the books category, illustrating substantial cross-language category variation.
4 Baseline Results
The baselines fine-tune cased multilingual BERT for supervised and zero-shot multilingual rating classification, evaluating ordinal error alongside accuracy. More English training data generally helps both same-language and zero-shot performance, with diminishing returns.
- MAE is the primary metric because it penalizes rating errors according to their ordinal distance.
- mBERT baselines use review text, with some settings also including the title and product category, and predict ratings from the CLS embedding.
- Supervised Text Classification: Supervised experiments evaluate fine-grained five-star and binarized rating classification across matching training and evaluation languages.
- Zero-shot Text Classification: Zero-shot experiments fine-tune mBERT on one source language and test on non-source languages using review body, title, and product category inputs.
- Zero-shot Text Classification: Increasing English training data generally improves English test and non-English zero-shot MAE, but the gains diminish at larger data amounts.
5 Conclusion
MARC is presented as a large public benchmark designed to support multilingual text-classification research. It combines careful data processing, six languages, defined splits, corpus analysis, and supervised and zero-shot baselines.
- MARC is a curated subset of Amazon reviews specifically designed for multilingual text classification research.
- The corpus is described as the largest public benchmark dataset for training and evaluating multilingual text-classification models.
- The authors provide six languages, well-defined training, development, and test splits, and processing intended to minimize dataset noise.
- The paper analyzes corpus characteristics and reports supervised and zero-shot cross-lingual classification baselines.