Source-linked AI summary
How Many Labels Does a Language Need? Annotation Budgets and Cross-Lingual Pooling for African-Language Text Classification
Bhanu Prakash Vangala, Sowmya Guda, Navya Vangala
TL;DR
African-language teams have little direct guidance on label budgets between full-data benchmarks and few-shot results, even though annotation is a dominant cost. This study measures learning curves and cross-lingual pooling and transfer across 28 tasks, finding that label needs vary by task and zero-shot transfer is generally weak.
Problem
There is little guidance on label needs between full-data benchmarks and few-shot studies, despite annotation being a dominant cost for African-language NLP.
Source: §1 · pp. 1–2Method
Results
News reaches 90% of full-data macro-F1 with 400 labels, while sentiment needs thousands; pooling helps at small budgets but hurts at larger ones, and zero-shot transfer usually recovers little.
Source: §4.1 · p. 4, §4.3 · p. 7, §5 · p. 8Takeaways & Limitations
For news tasks, plan on 400 to 800 labels and use pooled data early, then drop it after a few hundred target labels; sentiment requires thousands and may warrant a pretrained encoder once labels exist.
Source: §5 · p. 8Three subsamples per budget do not fully characterize variance at 25 labels, where one unlucky draw can miss a class.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Every text classifier for an African language begins with a budgeting question: how many labelled examples are needed, and can labels from other African languages stand in for them? We answer both questions empirically for 28 language-task pairs, news topic classification in 16 languages (MasakhaNEWS) and tweet sentiment in 12 languages (AfriSenti), using a character n-gram linear model that trains in seconds on two CPU cores with no pretrained weights and no accelerator. Monolingual learning curves at budgets from 25 to several thousand labels show that topic classification reaches 90\% of its full-data macro-F1 with about 400 labels in the median language, while sentiment is still improving at the full training size in 11 of 12 languages and needs thousands of labels. Pooling the full training data of the other languages in the benchmark is worth a great deal at small budgets and nothing at large ones: at 25 target labels it adds 0.20 macro-F1 on average for news (up to 0.43 for Lingala) and 0.08 for sentiment, the gain decays to zero by 800 labels, and at full size pooling hurts in 9 of 16 and 8 of 12 languages. Twenty-five target labels plus pooled data match what 100 to 400 monolingual labels achieve for most news languages. A complete zero-shot transfer matrix shows that transfer without any target labels recovers a median of only 13\% (news) and 4\% (sentiment) of the gap between a majority-class predictor and the in-language model, with the exceptions explained by shared script (Amharic and Tigrinya), shared lexicon (English and Nigerian Pidgin, the Arabic dialects), or a shared label prior rather than by language family. We release code that regenerates every number from the public benchmark files and translate the results into concrete annotation guidance for teams building African-language classifiers without GPUs.
1 Introduction
The paper measures the label budgets African-language classifiers need and asks whether data from other African languages can substitute for target-language labels. It finds that budgets and the usefulness of cross-lingual data differ by task, while providing practical measurements for low-compute teams.
1 Introduction
- The study addresses the gap between full-data benchmarks and few-shot studies by measuring how many labels African-language classifiers need.The authors frame annotation as a dominant cost, paid for each new language and task.Source: §1 · pp. 1–2
- Across 28 language-task pairs, the study measures learning curves using a character n-gram TF-IDF linear SVM without pretrained weights or an accelerator.It covers news topic classification in 16 MasakhaNEWS languages and tweet sentiment in 12 AfriSenti languages.Source: §1 · p. 2
- Pooling other benchmark languages’ data and zero-shot transfer are evaluated as alternatives to spending on target-language annotation.Zero-shot transfer is measured for every ordered pair across scripts and language families.Source: §1 · p. 2
- News classification reaches most full-data performance with a few hundred target labels, while sentiment needs thousands; pooling helps only at the smallest budgets and zero-shot transfer mostly does not work.Where zero-shot transfer appears to work, the paper attributes it to shared script, vocabulary, or label distribution rather than linguistic relatedness.Source: §1 · p. 2
- The contributions include per-language label-budget estimates, controlled pooling measurements, a complete zero-shot transfer matrix, and code that regenerates results from public benchmark files.Source: §1 · p. 2
2 Related Work
Related work covers African-language benchmarks and pretrained models, low-compute methods, and prior studies of learning curves and cross-lingual transfer. This study is positioned as a low-compute complement that measures label budgets without pretrained models or GPUs.
African-language benchmarks and models
- African-language community efforts produced benchmarks for named entity recognition, news topics, and sentiment, alongside adapted pretrained encoders.The cited MasakhaNEWS few-shot result reaches about 93% of full-supervision performance with ten examples per label, using a large pretrained model and GPU fine-tuning.Source: §2 · p. 2
Low compute
- Prior work describes a low-resource double bind in which languages with the least data are served by communities with the least compute [Ahia et al., 2021].Simple low-compute approaches remain competitive in evaluations of African-language adaptation [Leong et al., 2023].Source: §2 · pp. 2–3
Learning curves and transfer
- Learning curves support sample-size planning, while active learning seeks to reduce annotation budgets.Source: §2 · p. 3
- Prior transfer studies find that a few target labels can recover losses or matter more than the transfer source; this study extends that line to settings without pretrained models or GPUs.The paper also distinguishes its explicit accounting of label budgets by language from earlier cross-lingual evaluation work.Source: §2 · p. 3
3 Setup
The setup uses official train and test splits from two African-language classification benchmarks, a single character n-gram linear model, and monolingual, pooled, and zero-shot configurations. Label budgets are estimated from monolingual learning curves, with Figure 1 illustrating the news-task comparison.
3 Setup
- MasakhaNEWS covers 16 languages with four to seven topic labels, while AfriSenti covers 12 languages with three sentiment labels; both use official splits and test macro-F1.Training-set sizes range from 608 to 3,309 articles for MasakhaNEWS and from 804 to 14,172 tweets for AfriSenti.Source: §3 · p. 3
- The model uses character n-grams of lengths 2 to 5, TF-IDF features, and a linear SVM, without tokenization, normalization, or language-specific resources.Character n-grams accommodate the Ge’ez, Arabic, and Latin scripts in the data.Source: §3 · p. 3
- At budgets from 25 to 800 labels and at full size, monolingual models use target examples alone, while pooled models add all other languages’ full training sets without tags or reweighting.The study also trains pooled models with zero target examples and evaluates every source-target pair in a transfer matrix.Source: §3 · p. 3
- The fitted label-budget estimate n95 is the smallest budget at which an inverse power-law fit reaches 95% of full-data macro-F1.The study also reports the first measured budget reaching that level without curve fitting.Source: §3 · p. 3
- Figure 1 plots official-test macro-F1 against target-label budget for monolingual and pooled MasakhaNEWS models; bars show standard deviations and the dotted line shows pooled zero-shot.Source: Figure 1 · p. 4
4 Results
Across the 28 language-task pairs, pooling substantially helps at small target-label budgets but fades by 800 labels, while zero-shot transfer usually recovers little of the in-language performance gap. News reaches most of its full-data performance with hundreds of labels, whereas sentiment remains unsaturated at full training size in most languages; all experiments run on two CPU cores without pretrained models or GPUs.
4.1 Monolingual learning curves and label budgets
- With 400 labels, news reaches 90% of full-data macro-F1 on average; sentiment reaches 78%, and raw n95 remains at full training size in 11 of 12 languages.Amharic’s sentiment curve is dominated by a train-test label shift: 52% neutral in training versus 67% negative in test.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
4.2 Pooling other languages: large gains at small budgets, none at large ones
- At 25 target labels, pooling adds 0.20 macro-F1 for news and 0.08 for sentiment, but gains vanish by 800 and full-size pooling hurts 9 of 16 and 8 of 12 languages.For news, 25 target labels plus pooling match 100 to 400 monolingual labels in 12 of 16 languages; shared n-gram boundaries can become harmful once target data can set them.Source: Figure 3 · p. 5, §4.2 · pp. 5–6
4.3 Zero-shot transfer: script and lexicon, not family
- Off-diagonal transfer recovers a median 13% of the majority-to-own-model gap for news and 4% for sentiment; standout cases reflect shared script, lexicon, or label-prior artifacts rather than family membership.The matrix compares each source-language model tested on a target language against the target’s majority-class predictor and full-data model.Source: Figure 4 · p. 7, §4.3 · p. 7
4.4 Cost
- Monolingual models train in 0.1 to 6 seconds per language and pooled models in 18 to 53 seconds on two CPU cores with 7 GB of memory.Generating the paper’s results took about two and a half hours on the same two cores, without a GPU, pretrained model, or external service.Source: §4.4 · pp. 7–8
5 Discussion
The discussion turns the measured curves into annotation guidance: news reaches most of its performance with hundreds of labels, while sentiment needs thousands. It also bounds those recommendations and compares them with prior few-shot findings.
Annotation guidance
- For news, plan on 400 to 800 labels per language for 90% to 95% of performance; for sentiment, plan on thousands and treat pooling as an early boost only.The authors expect teams to consider pretrained encoders [Alabi et al., 2022, Ogueji et al., 2021] for sentiment once labels exist, and to check transfer sources’ label priors.Source: §5 · p. 8
What the curves do not show
- The findings are bounded by official test splits, one model family chosen for GPU-free use, and three subsamples per budget.At 25 labels, one unlucky draw can miss a class; active learning and distant supervision were not studied, and language-tagged or upweighted pooling might extend where pooling helps.Source: §5 · p. 8
Relation to prior findings
- MasakhaNEWS reports about 93% of full performance from ten examples per label with a large model, compared with the simple model’s 90% mark at 400 labels.The paper frames this as a choice between 40 labels and a GPU or 400 labels and a laptop, extending earlier findings to a no-pretrained-model setting [Lauscher et al., 2020] [Hedderich et al., 2020].Source: §5 · p. 8
6 Conclusion
Across 28 African-language classification tasks, a character n-gram linear model reaches most of its full-data performance with hundreds of news labels but needs thousands for sentiment. Pooling helps at small budgets but hurts at full size, while zero-shot transfer mostly recovers little of in-language performance.
6 Conclusion
- A character n-gram linear model without pretrained weights reaches most full-data performance with a few hundred labels for news and needs thousands for sentiment.The study covers 28 African-language classification tasks.Source: §6 · p. 8
- Pooling other languages’ data is worth up to 0.4 macro-F1 with a few dozen target labels but costs accuracy once the target has a few hundred.The crossover is at a few hundred labels in both tasks.Source: §6 · p. 8
- Zero-shot transfer without target labels recovers little in-language performance except where languages share a script, lexicon, or label prior.The work releases code to regenerate every number from public benchmark files, and reports running on two CPU cores in an afternoon.Source: §6 · p. 8