Source-linked AI summary
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, Anna Korhonen
TL;DR
Commonsense reasoning systems need challenging multilingual evaluations that connect everyday situations with causal world knowledge across languages. XCOPA provides an 11-language, typologically diverse benchmark and evaluates multilingual transfer and adaptation strategies. The results show that translation-based transfer can outperform multilingual model transfer, while adaptation targets resource-lean unseen languages.
Problem
Existing commonsense reasoning datasets were largely English-only, leaving a comprehensive multilingual benchmark missing despite cultural variation in typical situations.
Method
XCOPA creates an aligned causal commonsense benchmark by translating and re-annotating English COPA across 11 typologically diverse languages, then benchmarks multilingual models and resource-lean adaptation.
Results
Translation-based transfer largely outpaces multilingual model transfer, while the best setup reaches 63 accuracy in COPA-only fine-tuning and 80 after sequential SIQA + COPA fine-tuning.
Takeaways & Limitations
XCOPA enables cross-lingual comparisons and tests model robustness across diverse typological features, including languages unseen during pretraining.
Takeaways & Limitations
English COPA scenarios inevitably include concepts, social norms, institutions, and idioms that may sound unusual or foreign in target languages.
Abstract
from arXiv · showhide
In order to simulate human language capacity, natural language processing systems must be able to reason about the dynamics of everyday situations, including their possible causes and effects. Moreover, they should be able to generalise the acquired world knowledge to new languages, modulo cultural differences. Advances in machine reasoning and cross-lingual transfer depend on the availability of challenging evaluation benchmarks. Motivated by both demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages, which includes resource-poor languages like Eastern Apurímac Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods based on multilingual pretraining and zero-shot fine-tuning falls short compared to translation-based transfer. Finally, we propose strategies to adapt multilingual models to out-of-sample resource-lean languages where only a small corpus or a bilingual dictionary is available, and report substantial improvements over the random baseline. The XCOPA dataset is freely available at github.com/cambridgeltl/xcopa.
1 Introduction
Commonsense reasoning requires world knowledge and plausibility judgments beyond textual entailment, but multilingual commonsense benchmarks remain scarce and largely English-only. XCOPA addresses this gap with an 11-language, typologically diverse evaluation set and studies cross-lingual transfer, including adaptation to unseen languages.
- Commonsense reasoning connects textual premises to plausible hypotheses using implicit world knowledge about situations, causality, and social conventions.
- Existing commonsense evaluations have largely been limited to English, leaving a comprehensive multilingual benchmark for commonsense reasoning missing.
- XCOPA translates and re-annotates English COPA validation and test sets into 11 target languages selected across distinct language families and geographical macro-areas.
- The benchmark evaluates pretrained multilingual models for cross-lingual commonsense transfer and investigates adaptation when target languages were unseen during pretraining.
- XCOPA results indicate limitations of current state-of-the-art multilingual models in cross-lingual transfer for complex reasoning tasks.
2 Annotation Design
XCOPA prioritizes typological diversity and natural target-language expression while aligning instances and labels across languages for comparable evaluation. Its design also addresses translation quality, cultural adaptation, and robustness to linguistic features that training data may underrepresent.
- Design Objectives: The dataset design balances typological diversity, cross-language comparability, and high-quality, natural, idiomatic monolingual versions.
- Language Sampling: Variety sampling distributes linguistic phenomena uniformly and includes outliers, so XCOPA evaluates model resilience to features unlikely to appear in training data.
- Language Sampling: The authors quantify sample diversity with typology, family, and geography indices based on typological features, distinct families, and macro-area entropy.
- Language Sampling: XCOPA samples 11 languages from distinct families and five geographical macro-areas, achieving the most diverse sample on typology, family, and geography among compared datasets.
- Final Dataset: Each language contains 100 validation and 500 test instances, framed as binary choices between two alternatives for a premise and CAUSE or RESULT prompt.
- Final Dataset: Majority labels required agreement from at least six translators and matched original English COPA development labels perfectly, with high agreement across languages.
3 Qualitative Analysis
The analysis examines cultural, grammatical, and annotation challenges in translating COPA scenarios across languages. It describes translation strategies and finds that most annotation disagreements were culturally independent and associated with genuine ambiguity.
- Cultural Context: Cultural differences make English COPA scenarios unfamiliar in some target languages, especially when concepts, institutions, norms, or idioms lack equivalents.Examples include bowling, traffic regulations, mortgages, and expressions such as “put the caller on hold.”
- Cultural Context: Translators addressed culturally unfamiliar concepts through paraphrases, similar-concept substitutions, or phonetically transcribed loan words.For example, “faucet” was replaced with “pipe” in Tamil and Haitian Creole.
- Grammatical Tense: Languages without grammatical tense express temporality through aspect, mood, lexical time expressions, or pragmatic context, complicating causal interpretation.The dataset therefore favored unmarked variants in tenseless languages, using situational context to establish temporal relations.
- Label Discrepancies: 87.5% of annotation disagreements appeared culturally independent and involved genuinely ambiguous cases.A follow-up Italian experiment obtained Fleiss’ κ scores of 0.926 on validation and 0.917 on test instances.
4 Experiments and Results
The experiments benchmark multilingual models and transfer strategies on XCOPA, showing limitations in multilingual zero-shot transfer and benefits from translation and lightweight adaptation for unseen languages.
- Experimental setup: XCOPA evaluates causal reasoning by selecting the more plausible answer for a premise and CAUSE or RESULT prompt.
- Experimental setup: The experiments vary cross-lingual transfer method, multilingual encoder, and training-data source, including SIQA, COPA, and target-language validation.
- Model and data results: 63 accuracy in COPA-only fine-tuning and 80 after sequential SIQA + COPA fine-tuning are reported for monolingual English BERT, exceeding multilingual transfer results.
- Model and data results: XLM-R outperforms MBERT and USE across setups, with especially pronounced gains when models are first fine-tuned on SIQA.
- Transfer methods: Multilingual model transfer fluctuates around random-level accuracy for out-of-sample Haitian Creole and Quechua, while translation-based transfer often performs better.
- Adaptation: All adaptation methods substantially improve XLM-R Base with sufficiently large SIQA fine-tuning datasets, and bilingual dictionaries can provide further benefits for Haitian Creole and Quechua.
5 Related Work
Commonsense reasoning benchmarks have expanded across several dimensions, but multilingual coverage remains limited. XCOPA addresses this gap with cross-lingual evaluation and reports results for transfer methods and unseen-language adaptation.
- Commonsense benchmarks cover abductive, social, physical, visual, and other reasoning facets, alongside multiple-choice and reading-comprehension formats.
- The benchmark’s scope is bounded because unseen languages must use scripts already represented in the pretrained models, such as Latin script for Haitian Creole and Quechua.
- XCOPA provides a multilingual benchmark for causal commonsense reasoning, with detailed per-language results and adversarial evaluations hiding premises or prompts.
- Adaptation experiments compare transfer variants for out-of-sample languages using XCOPA accuracy scores.
- Multilingual evaluation suites exist for many natural-language-understanding tasks, including lexical similarity, document classification, sentiment, inference, question answering, and paraphrase identification.
6 Conclusion and Future Work
XCOPA is an aligned, typologically diverse benchmark for causal commonsense reasoning across 11 languages. Experiments show that translation-based transfer outperforms multilingual model transfer in many settings, while lightweight adaptation improves performance for unseen languages.
- XCOPA aligns causal commonsense instances across 11 languages selected for typological, geographic, and genealogical diversity.
- Translation-based transfer generally outpaces multilingual model transfer, while adversarial masking shows that robust causal reasoning is required.
- Lightweight adaptations using small monolingual corpora and bilingual dictionaries yield notable gains for out-of-sample languages.
- The authors position XCOPA as a challenging evaluation set for further multilingual commonsense reasoning and cross-lingual transfer research.
A Detailed Translation Guidelines
XCOPA translations were produced by one qualified translator per language under guidelines preserving lexical, temporal, and causal relations. Where direct translation was unavailable, translators used culturally appropriate alternatives while retaining causal roles.
- Each language was translated by one native speaker fluent in English and meeting a minimum undergraduate education requirement.
- Translations preserve correspondence between repeated lexical items across the premise and answer choices.
- Guidelines preserve event chronology and temporal extension through appropriate tense and aspect choices, including perfective–imperfective distinctions.
- For concepts without exact translations, translators prefer understandable loanwords, then descriptive paraphrases, then culturally familiar substitutes with similar causal roles.
B Why is Grammatical Tense Problematic for XCOPA?
XCOPA’s narrative scenarios rely on temporal distinctions that languages encode differently. Translation therefore requires language-specific choices about tense, aspect, discourse grounding, and event perspective.
- COPA scenarios describe past events in a narrative register using past simple or past continuous forms, without universally marking hypothesis anteriority.
- Because temporal marking reflects discourse grounding and speaker perspective, translators choose the most natural tense or aspect for each context.
- Languages differ in how they express perfective–imperfective distinctions; Haitian Creole uses te for bounded past events and ap for continuous aspect.
- The benchmark’s language sample and pretrained-model inventory provide the multilingual context for evaluating these translation and modeling challenges.
C Hyper-Parameter Search
The experiments searched model-specific learning rates, dropout rates, and batch sizes, then used development-set evaluation for early stopping and standardized Adam optimization settings.
- Hyper-parameter grid: MBERT and XLM-R were tuned over learning rates of 5 · 10^-6, 10^-5, and 3 · 10^-5, dropout rates of 0 and 0.1, and batch sizes of 4 and 8.The dropout was applied to the transformer output layer and feed-forward scoring network hidden layer.
- Hyper-parameter grid: USE was tuned over learning rates of 10^-3, 10^-4, and 10^-5.
- Evaluation and optimization: Training was evaluated every 500 updates for SIQA and every 10 updates for COPA, stopping after 10 consecutive evaluations without improvement.
- Evaluation and optimization: All setups used Adam with ϵ = 10^-8, no weight decay or warm-up, and gradient-norm clipping at 1.
D Full Results (Per Language)
Table 8 reports detailed results for multilingual model transfer across every XCOPA language and five evaluation setups.
- Per-language results: Table 8 contains per-language multilingual-transfer results for all XCOPA languages.
- Per-language results: The reported evaluation setups are CO-ZS, CO-TLV, SI-ZS, SI+CO-ZS, and SI+CO-TLV.
E Code and Dependencies
The implementation uses HuggingFace Transformers and USE, with code publicly available and otherwise relying on standard Python scientific-computing libraries.
- Frameworks: The code is built on HuggingFace Transformers and the multilingual Universal Sentence Encoder.
- Code availability: The source code is available at github.com/cambridgeltl/xcopa.
- Dependencies: Beyond the listed frameworks and pretrained models, the implementation relies only on standard Python scientific-computing libraries such as numpy.