Source-linked AI summary
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, Prem Natarajan
TL;DR
Multilingual NLU lacks realistic, natural labeled data across many languages. MASSIVE addresses this gap by localizing English SLURP into a 1M-example parallel dataset spanning 51 languages and evaluates XLM-R and mT5 baselines. Zero-shot exact match is 25–37 points worse than full-dataset training, with larger variation across locales.
Problem
Multilingual NLU lacks realistic, natural labeled data for training and evaluation across many languages.
Method
MASSIVE localizes the English-only SLURP dataset into a parallel corpus spanning 51 languages, with modeling baselines from XLM-R and mT5.
Results
Zero-shot exact match performance is 25–37 points worse than full-dataset training, and locale variation is substantially greater in the zero-shot setup.
Takeaways & Limitations
MASSIVE provides a broad benchmark for multilingual NLU and supports cross-linguistic analyses of device-directed language.
Abstract
from arXiv · showhide
We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.
1 Introduction and Description
MASSIVE addresses the shortage of realistic, natural labeled data for multilingual NLU by providing a large parallel dataset for virtual-assistant tasks across diverse languages. It localizes English SLURP data and includes public data, code, and models.
- Motivation: MASSIVE focuses on NLU, a component of spoken language understanding that supports voice-based virtual assistants.In SLU, raw audio is first converted to text before NLU is performed.
- Motivation: Labeled multilingual NLU data is scarce, especially data that is realistic and natural in each language.Human vetting is typically needed for naturalness and is often costly.
- Dataset contribution: MASSIVE contains 1M realistic, human-created virtual-assistant utterances spanning 51 languages, 60 intents, 55 slot types, and 18 domains.The released splits include 587k training, 104k development, and 152k test utterances; 153k were held out for a competition.
- Construction: The dataset was created by localizing the English-only SLURP NLU dataset in parallel across languages.The paper also reports modeling results for massively multilingual NLU, where one model handles incoming languages.
2 Related Work
Prior multilingual NLU benchmarks were limited in language coverage, domains, or task scope, while creating realistic schema-specific annotations remains costly. MASSIVE builds on SLURP to provide a broader multilingual benchmark with released baselines.
- Gap: Existing multilingual NLU evaluation datasets had not kept pace with multilingual pretrained models and often required costly new data and schema-specific annotations.The challenge includes identifying enough speakers for data generation and quality control.
- Prior benchmarks: MultiATIS++ extended NLU across nine languages but remained limited to airline bookings.Other prior datasets covered narrower virtual-assistant domains such as alarms, reminders, and weather.
- Contribution: MASSIVE releases the dataset together with baselines from large pretrained models fine-tuned for slot and intent prediction.The paper situates these baselines within earlier projection, alignment, translation, and multilingual pretrained-model approaches.
3 Language Selection and Linguistic Analysis
MASSIVE selects languages to maximize typological, script, resource, and practical diversity, then analyzes linguistic patterns in device-directed utterances. Its coverage supports comparisons of questions, imperatives, and other mood and word-order phenomena.
- 3.1 Language Selection: Language selection considered worker cost and availability, virtual-assistant coverage, typological genera, internet-resource proxies, and script diversity.The selection sought languages spanning the breadth of resource availability and increasing diversity for tokenization and normalization research.
- 3.1 Language Selection: MASSIVE contains 50 new text corpora representing 49 spoken languages, organized across 14 language families including isolates.Mandarin Chinese appears twice, using traditional and simplified characters.
- 3.2 Scripts: The dataset uses 21 distinct scripts, including Latin, Arabic, Cyrillic, and scripts unique to individual languages.Fourteen scripts are unique to a single language in the dataset, while Ge’ez and Chinese are represented despite broader real-world use.
- Linguistic analysis: MASSIVE is dominated by interrogatives and imperatives because its utterances are directed at a device rather than a person.The parallel corpus creates opportunities for cross-linguistic comparison of these less-studied sentence forms.
- Word order: In MASSIVE, 39 languages are subject-initial, three are verb-initial, and no object-initial languages are represented.Five languages have no preferred word order and four lack word-order data.
- 3.5 Imperative Marking: Imperative marking varies widely: 33 languages use verb morphology, 10 lack special imperative morphology, and 18 distinguish singular or plural addressees.The dataset also includes languages with specialized hortative, optative, and prohibitive forms.
4 Collection Setup and Execution
MASSIVE uses a multilingual vendor workflow that separates slot localization from full-utterance localization, followed by judgments of semantic and linguistic quality. Piloting and worker screening support the collection process, while the two-step design reduces annotation burden.
- Collection setup: The collection used a customized Amazon MTurk workflow involving multilingual vendors selected partly by resource availability and cost.The initial five-vendor pool was reduced to three, with cross-vendor support used for some languages.
- Worker selection: Workers were screened with language-specific fluency tests for translation and vendor assessments for judgment tasks.An initial pilot in three languages improved instructions, reporting methods, and resolution of open questions.
- Workflow: Each SLURP utterance was processed at the locale level across train, development, test, and held-out splits through sequential task and judgment workflows.The workflow began with slot translation or localization before full-phrase localization.
- Slot workflow: The first worker translated or localized highlighted slot values, retaining names or proper nouns when appropriate.Workers could translate, localize, or keep slot values unchanged, and this decision was recorded in dataset metadata.
- Phrase workflow: A second worker localized the full phrase using the slot output and could modify or remove slots while aligning grammatical forms.The two-step process avoids requiring workers to handle nuanced span notation directly.
- Quality control: Three workers judged semantic intent and slot alignment, grammaticality, naturalness, spelling, and language identification.English or mixed-language utterances were accepted when natural for the target language.
5 Model Benchmarking
MASSIVE benchmarks multilingual NLU models using XLM-R and two mT5 architectures for intent classification and slot filling, including full-data and English-only zero-shot settings. Full-dataset training substantially outperforms zero-shot training, while performance varies more across locales in the zero-shot setting.
- Setup: XLM-R and mT5 were fine-tuned on MASSIVE and evaluated using intent accuracy, micro-averaged slot F1, and exact match accuracy.The evaluated mT5 variants were Encoder-Only and Text-to-Text; XLM-R used separate intent and slot classification heads.
- Setup: 128 hyperparameter-tuning trials were performed for each model using Tree of Parzen Estimators and ASHA scheduling.The models were trained with Adam, selecting checkpoints by overall exact match accuracy across locales.
- Results and Analysis: Artificial spacing can improve Thai performance, while character spacing is necessary for assigning slots in Japanese and Chinese.The paper leaves more sophisticated spacing techniques for future work.
- Results and Analysis: Zero-shot exact match performance is 25-37 points worse than full-dataset training, and locale variation is substantially greater.For mT5 Text-to-Text, the highest-to-lowest locale gap expands from 15 points with full training to 44 points with zero-shot training.
- Results and Analysis: Pretraining-data correlations in zero-shot XLM-R are 0.54 for exact match, 0.58 for intent accuracy, and 0.46 for micro-averaged slot F1.With full-dataset training, these correlations decrease to 0.42, 0.47, and 0.24, respectively.
- Results and Analysis: Japanese exact match accuracy fell from 58.3% with full-dataset training to 9.4% in zero-shot training.The authors identify individually spaced characters, which differ from pretrained tokenizer inputs, as a likely major cause.
6 Conclusion
The paper releases MASSIVE, a multilingual NLU dataset spanning 51 typologically diverse languages. It is intended to support innovation in multilingual NLU, other NLP tasks, and linguistic analysis.
- Conclusion: MASSIVE is a multilingual NLU dataset spanning 51 typologically diverse languages.The dataset is released publicly as a resource for research.
- Conclusion: The authors hope MASSIVE will encourage innovations in massively multilingual NLU, machine translation, and linguistic analyses such as imperative morphologies.These are stated as anticipated uses and research directions.
A Additional Linguistic Characteristics
The paper directs readers to Table 4 for additional linguistic characteristics of the languages in MASSIVE.
- Additional Linguistic Characteristics: Additional linguistic characteristics of the MASSIVE languages are provided in Table 4.The supplied passage points to the table without specifying individual characteristics.
B The Collection System
The collection system is documented through screenshots of the workflow presented in Figures 1, 2, and 3.
- The Collection System: Screenshots from the collection workflow are provided in Figures 1, 2, and 3.The passage identifies the figures but does not describe their contents.
C Hyperparameters
The hyperparameter search spaces and selected hyperparameters for the models are reported in Tables 5 and 6.
- Tables 5 and 6 report the hyperparameter search spaces and chosen hyperparameters for the three models.
D Results for All Languages
Results for all languages are reported separately for exact match accuracy, intent accuracy, and micro-averaged slot-filling F1.
- Table 7 reports exact match accuracy for every language across the three models and two training setups.
- Table 8 reports intent accuracy for every language across the three models and two training setups.
- Table 9 reports micro-averaged slot-filling F1 for every language across the three models and two training setups.
E A summary of model performance on language characteristics
The paper summarizes mT5 Text-to-Text performance across linguistic characteristics, while the accompanying tables and figures document dataset characteristics, model configurations, and language-level metrics.
- mT5 Text-to-Text performance is summarized across language characteristics in Figures 4 and 5.The categories are sorted by exact match accuracy, with language counts shown in lower-panel bar charts.
- Table 4 provides additional linguistic characteristics of the MASSIVE languages.
- Table 5 documents the full-dataset hyperparameter search space, sampling technique, and chosen hyperparameter for the three models.
- Table 6 documents the zero-shot hyperparameter search space, sampling technique, and chosen hyperparameter for the three models.
- Tables 7 and 8 report language-level exact match accuracy and intent accuracy for the three models under full-dataset and zero-shot setups.
- Tables 9 reports language-level micro-averaged slot-filling F1 for the three models under full-dataset and zero-shot setups.