Source-linked AI summary
Choosing Transfer Languages for Cross-Lingual Learning
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig
TL;DR
Choosing a transfer language for a low-resource NLP task is difficult because common heuristics may overlook relevant linguistic and corpus factors. The paper introduces LANGRANK, which learns to rank candidate languages from these features and finds better recommendations than single-feature heuristics across four NLP tasks. The authors also analyze feature importance to inform educated guesses, while cautioning that models trained on limited language sets may not generalize broadly.
Problem
The best transfer language for a particular low-resource task language is unclear, and heuristic selection may consider only isolated language or dataset properties.
Method
LANGRANK formulates transfer-language selection as ranking and uses linguistic attributes and corpus statistics to predict candidate-language performance.
Results
Across machine translation, entity linking, POS tagging, and dependency parsing, LANGRANK recommends better transfer languages than single-feature heuristics.
Takeaways & Limitations
Analyzing learned feature importance provides insights that may inform future ad hoc transfer-language selection even without LANGRANK.
Takeaways & Limitations
Because rankers are trained on only a few languages for particular tasks, they may not generalize to different language sets or NLP tasks.
Abstract
from arXiv · showhide
Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages. However, given a particular task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usually the intuition of the experimenter. Since a large number of features contribute to the success of cross-lingual transfer (including phylogenetic similarity, typological properties, lexical overlap, or size of available data), even the most enlightened experimenter rarely considers all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer languages as a ranking problem, and build models that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines considering single features in isolation, and glean insights on what features are most informative for each different NLP tasks, which may inform future ad hoc selection even without use of our method. Code, data, and pre-trained models are available at https://github.com/neulab/langrank
1 Introduction
Cross-lingual transfer can improve NLP for low-resource languages, but selecting the best transfer language remains difficult. LANGRANK treats selection as a ranking problem using linguistic and corpus features.
- Low-resource NLP often lacks sufficient training data, motivating transfer from high-resource languages.
- Transfer-language choices have traditionally relied on heuristics such as language-family membership, phylogenetic distance, or task-relevant typological properties.
- LANGRANK represents candidate transfer languages with typological information and corpus statistics, then ranks them by expected task performance.
- Training requires exhaustive transfer experiments, but inference rapidly predicts promising languages from easily obtained linguistic and dataset features.
- Experiments evaluate ranking models across machine translation, entity linking, POS tagging, and dependency parsing against heuristic baselines.
2 Problem Formulation
The paper defines transfer-language selection as predicting the ranking of candidate languages by their expected scores without evaluating every candidate at test time. Exhaustive experiments provide training data, while the ranking model approximates their ordering.
- A task language is the language of interest, while a transfer language supplies additional training data to aid the task.
- For each task–transfer pair, a model is trained, evaluated on test data, and assigned a resulting score.
- The brute-force optimum is found by exhaustively training and evaluating models for every potential transfer language.
- This exhaustive search is impractical because training many models is computationally expensive and language resources may need to be gathered first.
- LANGRANK learns to predict candidate-language rankings from exhaustive sweeps over training task languages and their observed transfer scores.
3 Ranking Features
LANGRANK combines dataset-dependent statistics with dataset-independent linguistic distances to represent each task–transfer language pair. Feature availability varies by task and resource conditions.
- Language pairs and corpora are represented with dataset-dependent and dataset-independent feature classes.
- Dataset-dependent features: Dataset size includes transfer and task training-example counts and their ratio, indicating how much larger the transfer corpus is.
- Dataset-dependent features: Type-token ratio measures lexical diversity, while the distance between languages’ ratios may roughly indicate morphological similarity.
- Dataset-dependent features: Word and subword overlap measure vocabulary similarity, with task-specific omissions for entity linking, POS tagging, and dependency parsing.
- Dataset-independent features: Dataset-independent features include geographic, genetic, inventory, syntactic, phonological, and combined featural distances from linguistic databases.
4 Ranking Model
The paper uses learning-to-rank methods to model transfer-language selection. It chooses gradient-boosted decision trees trained with LambdaRank, combining ranking performance with interpretability.
- Transfer-language selection is formulated as a standard learning-to-rank problem with multiple possible models and learning algorithms.
- The chosen model is a gradient-boosted decision tree ensemble trained with LambdaRank.
- GBDT is selected for empirical performance in settings with few features and limited data, as well as interpretability.
5 Experimental Settings
The experiments evaluate LANGRANK for selecting transfer languages across four NLP tasks, using cross-validation and ranking quality measured by NDCG@3. They compare feature-based ranking models with single-feature baselines.
- Experimental Tasks: Experiments cover machine translation, entity linking, part-of-speech tagging, and dependency parsing.The task settings use multilingual TED talks, language-linked Wikipedia titles, Universal Dependencies, and zero-shot or joint-training transfer depending on the task.
- Evaluation Protocol: The study uses leave-one-out cross-validation, treating each language in turn as the test task language and the remaining languages as training and transfer candidates.Each fold leaves one language out, trains a ranking model on the remaining languages, and predicts transfer-language ranking scores for the held-out language.
- Evaluation Protocol: NDCG@3 evaluates whether LANGRANK recommends the best transfer language within the top three positions.The relevance signal retains the top 10 transfer languages, while NDCG compares predicted and gold rankings and equals 1 for an exact ranking match.
- Baselines and Models: Table 1 reports average NDCG@3 across cross-validation folds for models using all features or dataset-dependent and dataset-independent feature subsets.These models are compared with baselines that sort transfer languages using individual dataset statistics or linguistic-distance features.
- Baselines and Models: LANGRANK models achieve higher ranking quality than single-feature heuristics across all four tasks.The comparison includes dataset-dependent features such as word overlap and dataset size, and linguistic-distance features such as phylogenetic and typological distances.
6 Results and Analysis
LANGRANK generally improves transfer-language selection over single-feature heuristics by combining linguistic and dataset features, while revealing task-specific feature patterns and generalization limits.
- Main ranking results: LANGRANK consistently surpasses single-feature heuristics in ranking quality across all tasks, with the best MT and POS models achieving nearly double the best baseline.The ranking metric is NDCG.
- Feature configurations: Dataset-dependent features improve LANGRANK predictions over linguistic distance alone, although the strongest feature configuration varies by task.All features perform best for EL and POS, dataset-only features for MT, and both settings for DEP.
- Feature configurations: Linguistic-only LANGRANK still outperforms every heuristic baseline on all tasks, suggesting utility before task-specific resources are collected.The authors connect this result to informing data-creation decisions.
- Top-K transfer selection: For MT, LANGRANK recommendations consistently yield better transfer models than the best baseline, whereas POS results are generally comparable.Figure 2 evaluates the best score obtainable from the top K recommendations using BLEU for MT and accuracy for POS.
- Top-K transfer selection: In EL, LANGRANK selects the correct top-1 transfer language for 7 of 9 task languages, but fails to generalize to Telugu and Uyghur lacking typologically similar training languages.Beyond the top three predictions, baseline averages can appear better, though averaging may be misleading.
- Task-specific feature analysis: Feature importance differs by task: MT emphasizes dataset size ratio and word overlap, POS emphasizes dataset size and TTR distance, and DEP emphasizes geographic, genetic, and word-overlap features.EL emphasizes geographic and syntactic distance because its datasets lack rich dataset-dependent features.
7 Related Work
Prior work studies cross-lingual transfer, typologically informed modeling, and data selection, but does not directly learn which pivot or transfer language is best.
- Large-scale pivot-language searches have explored transfer-language selection without learning to predict the best pivot.
- Typological information has been incorporated into language modeling, phonetic representation learning, and dependency parsing.
- Bayesian optimization and dynamic or domain-relevant data-selection methods address training-data selection rather than cross-lingual transfer-language prediction.
8 Conclusion
The paper formulates optimal transfer-language selection as a ranking problem and evaluates models across four NLP tasks. Using multiple dataset and language features yields better recommendations than single-feature approaches and reveals task-specific feature importance.
- LANGRANK ranks candidate transfer languages for machine translation, entity linking, POS tagging, and dependency parsing.
- Considering multiple dataset statistics and language attributes produces better transfer-language recommendations than considering single language or dataset features.
- Analyzing learned rankers identifies influential feature types for each NLP task, informing educated guesses without using the full method.