Source-linked AI summary
CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
Libo Qin, Minheng Ni, Yue Zhang, Wanxiang Che
TL;DR
Zero-shot cross-lingual models such as mBERT can struggle with inconsistent contextualized subword representations across languages. CoSDA-ML addresses this by dynamically generating multilingual code-switched data for mBERT fine-tuning, and experiments across five tasks show significant improvements over mBERT and XLM baselines.
Problem
mBERT’s multilingual training uses monolingual contexts, which can produce inconsistent contextualized subword representations across languages.
Method
CoSDA-ML dynamically generates multilingual code-switched data to fine-tune mBERT and align representations from a source language with multiple target languages.
Results
CoSDA-ML consistently and significantly outperforms mBERT and XLM baselines across five zero-shot cross-lingual tasks.
Takeaways & Limitations
One training process supports multiple target languages without parallel sentences, while the resulting model remains as simple to use as mBERT.
Takeaways & Limitations
The augmentation may randomly select an incorrect word-to-word translation when a source word has multiple target-language translations.
Abstract
from arXiv · showhide
Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT), have shown success in a variety of zero-shot cross-lingual tasks. However, these models are limited by having inconsistent contextualized representations of subwords across different languages. Existing work addresses this issue by bilingual projection and fine-tuning technique. We propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT, which encourages model to align representations from source and multiple target languages once by mixing their context information. Compared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages. Experimental results on five tasks with 19 languages show that our method leads to significantly improved performances for all the tasks compared with mBERT.
1 Introduction
Zero-shot cross-lingual transfer addresses scarce labeled data by applying knowledge from resource-rich languages, but mBERT’s monolingual contexts can produce inconsistent cross-lingual subword representations. CoSDA-ML fine-tunes mBERT with dynamically generated multilingual code-switched data to align source and multiple target languages.
- Zero-shot cross-lingual learning transfers knowledge from resource-rich languages without requiring labeled data in the target language.
- mBERT trains shared subword embeddings and other parameters on merged raw sentences from multiple languages.
- Monolingual training contexts can yield inconsistent contextualized subword representations across languages.
- CoSDA-ML dynamically constructs multilingual code-switched data from English sentences and bilingual dictionaries to fine-tune mBERT.
- The augmentation randomly selects sentences, words, and target languages, mixing context information to align source and target word vectors.
- CoSDA-ML uses one training process for multiple target languages, does not require parallel sentences, and improves all five evaluated tasks over mBERT.
2 Background
The paper applies mBERT to zero-shot classification and sequence labeling by transferring an English-trained model directly to unlabeled target languages. Classification uses the sequence representation, while sequence labeling classifies tokens from their final hidden states.
- mBERT is trained on Wikipedia from 104 languages with a shared WordPiece vocabulary that enables cross-language embedding sharing.
- Classification: For classification, mBERT adds special tokens, encodes the sequence, and uses hCLS as input to a task-specific classification layer.
- Classification: The classification layer is jointly fine-tuned with all mBERT parameters by maximizing the correct-label log-probability.
- Sequence labeling: For sequence labeling, final token hidden states feed a softmax layer, with the first sub-token representation used to classify each word.
- Zero-shot transfer: Zero-shot transfer assumes labeled English training data and applies the trained model directly to target languages without labeled target-language data.
3 Method
The method dynamically augments source-language training batches with multilingual code-switched data, fine-tunes mBERT once, and applies the resulting model directly to target languages.
- Training and adaptation: The framework fine-tunes mBERT with dynamically generated multilingual code-switched training data before zero-shot testing.The process has two stages: augmented fine-tuning followed by direct application to target languages.
- Data augmentation algorithm: Algorithm 1 generates augmented data by selecting sentences, selecting words, and replacing selected words with translations.These operations are performed using source training data, bilingual dictionaries, replacement ratios, and target-language sets.
- Data augmentation algorithm: Sentence selection randomly chooses sentences for code mixing while leaving unselected sentences in the original language.The selected sentences proceed to word and replacement selection.
- Data augmentation algorithm: For selected sentences, the method randomly chooses words and then randomly selects target languages for their dictionary-based replacements.Different target languages can be mixed within the generated code-switched data.
- Data augmentation algorithm: Randomly selecting translations can produce replacements that are not guaranteed to be contextually correct, which the authors treat as data augmentation.A source word may have multiple target-language translations, and one is chosen randomly.
- Training and adaptation: Dynamic per-batch augmentation exposes the model to different code-switched data and is intended to align source words with target-language replacements through shared context.The method mixes context information while increasing the distribution of training instances.
4 Experiments
The experiments evaluate CoSDA-ML across 19 languages, five tasks, multiple baselines, data regimes, and encoder settings. Results show broad gains over mBERT and XLM, robustness with limited data, benefits from dynamic sampling, and closer cross-lingual representations.
- Experimental Setup: 19 languages and five zero-shot cross-lingual tasks are used to evaluate CoSDA-ML against mBERT and XLM.The evaluation covers natural language inference, sentiment classification, document classification, natural language understanding, and dialogue state tracking.
- Overall Results: CoSDA-ML outperforms mBERT and XLM by a large margin and achieves state-of-the-art performance across all evaluated tasks.The authors report significance testing and mark improvements with p < 0.01.
- Data Efficiency: Using only 1/10 of the training data, CoSDA-ML performs better than Attention-Informed Mixed Training using 100% of the training data.The method also consistently outperforms that baseline across all reported training-data sizes.
- Ablation Analysis: Dynamic augmentation outperforms static augmentation in all tasks by generating more varied multilingual code-switched data during batch training.The authors attribute the improvement to aligning more word representations across multiple languages.
- Representation Analysis: CoSDA-ML brings representations from different languages with the same intent closer together and makes them overlap in t-SNE visualization.The mBERT visualization shows nearly no overlap between languages for the same intent.
5 Related Work
Related work addresses zero-shot cross-lingual transfer through multilingual word embeddings and data augmentation. CoSDA-ML differs by dynamically generating multilingual code-switched data during training for multiple target languages.
- Zero-Shot Cross-Lingual Transfer: Earlier zero-shot transfer work learned cross-lingual word embeddings through joint training or post-training mappings of monolingual embeddings.The cited literature surveys both families of approaches.
- Data Augmentation: Recent augmentation work translates selected words into target languages, including attention-based selection of important words.Attention-Informed Mixed Training translates only one word into each augmented sentence.
- Data Augmentation: CoSDA-ML dynamically augments data in each epoch and generates multilingual code-switched data, enabling one training process followed by testing across target languages.This contrasts with augmentation that focuses on translating one word or a single target language.
6 Conclusion
The paper proposes multilingual code-switching augmentation to fine-tune mBERT and align source- and target-language representations. Across five tasks, the method consistently and significantly outperforms mBERT and XLM, while remaining applicable to base encoder models.
- Contribution: The proposed framework generates multilingual code-switching data to fine-tune mBERT and align representations from source and multiple target languages.The framework is designed to mix context information across languages during fine-tuning.
- Findings: Experiments on five tasks show that CoSDA-ML consistently and significantly outperforms mBERT and XLM baselines.The conclusion summarizes the cross-task evaluation result without narrowing it to a single benchmark.
- Implications: The method is flexible and can be used to fine-tune all base encoder models.The authors identify multilingual language modeling as future work for investigating more general multilingual contextual embeddings.