PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Research Progress and Linguistic Inclusion in Low-Resource Languages

Atnafu Lambebo Tonja, Tadesse Destaw Belay, Jesujoba Oluwadara Alabi, Israel Abebe Azime, Benjamin Rosman, Ehsan Shareghi, Thamar Solorio, Pontus Stenetorp, Monojit Choudhury

arXiv:2610.30296v1cs.CL

TL;DR

The paper addresses the lack of evidence on whether growth in LRL research broadens and sustains linguistic representation. It maps research and applies a four-dimensional inclusion framework; its continental analysis finds that no region scores highly across all dimensions. The results indicate that research growth and expansion alone do not ensure sustained or equitable representation.

  • Problem

    Existing output and performance measures do not show whether LRL research is becoming more inclusive, and longitudinal evidence on broader, sustained representation is lacking.

    Source: §1 · p. 1

  • Method

    The paper maps peer-reviewed open-access LRL research from 2020–2025 and assesses inclusion through research growth, representation expansion, representation continuity, and distributional equity.

    Source: §1 · p. 2

  • Results

    No continent scores highly across all four inclusion dimensions, and research growth and expansion into new languages alone do not produce sustained or equitable representation.

    Source: §4.1 · p. 8, §5 · p. 8

  • Takeaways & Limitations

    Research growth alone is not a reliable measure of progress toward equitable and inclusive language technology.

    Source: §4.1 · p. 8

    The dataset is limited to peer-reviewed open-access papers found with English keywords, and language-paper counts may overstate attention when a language is merely mentioned in a multilingual benchmark.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have accelerated research on low-resource and multilingual NLP, but their effectiveness in creating an equitable landscape remains unmapped. We analyze 1,158 peer-reviewed open-access papers published between 2020 and 2025, examining research attention to low-resource languages (LRLs) and representation across language and speaker populations. To assess linguistic inclusion beyond publication counts, we introduce the Linguistic Inclusion Framework (LIF), a multidimensional framework that measures linguistic inclusion in four dimensions: research growth, representation expansion, representation continuity, and distributional equity. While annual publications grew from 52 to 415, a 5.4X increase in their ACL Anthology publication share, this growth largely concentrated on represented languages rather than expanding coverage of underrepresented languages. Further analysis shows that languages spoken in Asia account for 64.1% of speakers across LRLs we analyze but receive only 46.7% of the language representation share (0.7X their proportional share), while European languages receive 5.3X their proportional share. At the country level, linguistically diverse regions such as sub-Saharan Africa and Southeast Asia show the lowest language coverage ratios. Applying the framework at the continental level, we find that no region scores well on all four dimensions, showing that research growth alone is not a reliable metric of progress toward equitable and inclusive language technology\footnote{We will release all collected and annotated data for research purposes}.

1 Introduction

Research on low-resource languages is growing, but existing output and performance measures do not reveal whether linguistic representation is becoming more inclusive. The paper maps research from 2020–2025 and introduces a framework for assessing inclusion across four dimensions.

1 Introduction

  • More than 7,000 languages are spoken worldwide, but only a few are represented in digital language technology.The introduction situates this imbalance amid growing LLM integration into everyday domains and limited language support.Source: §1 · p. 1
  • More papers, models, datasets, or benchmarks do not necessarily mean greater linguistic inclusion, because growth may remain concentrated in already represented languages.Existing metrics capture research output and model performance, but not whether representation is becoming more inclusive.Source: §1 · p. 1
  • Longitudinal evidence is lacking on whether rapid LRL research expansion has broadened and sustained language representation or narrowed the gap with real-world linguistic diversity.The paper frames these as open questions about model development and evaluation, resource availability, and representation.Source: §1 · p. 2
  • The paper maps peer-reviewed open-access LRL research from 2020–2025 across publication trends, research topics, modalities, contributions, and language representation.It also examines geographic and linguistic distributions, language continuity, and publicly available datasets and models for frequently studied languages.Source: §1 · p. 2
  • The Linguistic Inclusion Framework measures research progress through research growth, representation expansion, representation continuity, and distributional equity.The framework is proposed for application at country, language-family, or continent level.Source: §1 · p. 2

2 Methodology

The study collects and screens LRL research papers, applies LLM-assisted annotation to classify their scope and content, and checks a sample of annotations against human judgments. The verification reports high agreement across the annotation tasks.

2 Methodology

  • The methodology describes a data collection and annotation pipeline to answer the paper’s research questions.Papers were collected from Semantic Scholar and ACL Anthology and screened before annotation.Source: §2 · p. 2, §2.1 · p. 2

2.1 Data Collection and Filtering

  • The search collected 3,385 papers, screening reduced these to 1,403 peer-reviewed open-access papers, and scope classification retained 1,158 classified as LRL.The initial search used 52 keyword variants across Semantic Scholar and ACL Anthology; excluded types included surveys, shared tasks, programming-language publications, and non-open-access papers.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 LLM-assisted Data Annotation

  • Papers were assigned to four research-scope categories, with 1,158 classified as low-resource-language papers retained; included papers were analyzed for languages, contribution types, primary NLP topics, and primary data modality.A language was represented if it appeared in a paper’s dataset, pretraining, adaptation or fine-tuning, benchmark, evaluation, or analysis; language coverage retained resource classes 0–3.Source: §2.2 · p. 3

2.3 Human Verification

  • Human–LLM agreement reached up to 100% across five annotation tasks, with lowest agreement of 96.68% for low-resource categorization and 97.32% for language extraction.Human–human agreement reached up to 98.67%, with lows of 90.26% for low-resource categorization and 91.97% for language extraction.Source: §2.3 · p. 4

3 State of Research on LRLs

The study tracks LRL publication activity, research characteristics, language coverage, continuity, and resource trajectories from 2020 to 2025. Across these analyses, research growth and resource growth remain concentrated in already represented languages, while language coverage and attention vary across regions.

3 State of Research on LRLs

  • Research on LRLs increased in publication volume between 2020 and 2025, but publication growth alone does not establish adequate representation or support for most LRLs or their speaker communities.The analysis tracks publication trends, linguistic representation, and resource trajectories.Source: §3.1 · p. 4, §3 · p. 4

3.1 Publication Trends

  • LRL papers in the ACL Anthology subset grew 11× from 2020 to 2025, while the LRL share of annual publications rose from 0.43% to 2.33%.Total ACL Anthology publications grew 2.05×, so the increase reflected a shift in research attention, not only overall expansion.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 NLP Topics, Contributions and Modalities

  • Multilingual and cross-lingual NLP (24.8%), resources and evaluation (23.1%), and machine translation (14.8%) together account for nearly two-thirds of papers.Classical tasks including syntax, parsing, morphology, and information extraction receive relatively little attention.Source: §3.2 · p. 4, Figure 3 · p. 5
  • In 2025, knowledge method (50.1%), artifact method (24.0%), and artifact dataset (17.6%) made up 91.7% of claim statements.Conceptual contributions focused on tasks, languages, or social and human dynamics remain marginal.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Text accounted for over 80% of papers in 2025, while speech and multimodal research accounted for 4.9% and 6.6%, respectively.Other modalities remained underrepresented despite modest growth in 2025.Source: §3.2 · p. 5

3.3 Language Coverage

  • Country-level coverage falls when counting only languages represented in multiple papers, while speaker-weighted representation varies sharply, from over 100 mentions per million in Australia and Iceland to 1.6 in India.The reported values are 0.95 for Indonesia, 1.22 for Nigeria, and 0.20 mentions per million for Morocco; Asia and Africa are underrepresented relative to their share of speakers.Source: §3.3 · pp. 5–6

3.4 Language Expansion and Continuity

  • 57.2% of year-to-year transition instances involved repeated languages, versus 19.6% newly included; new-language inclusion fell to 2.1% in 2024→2025.Repeated representation rose to 68.5% in 2024→2025, indicating that growth reflected repeated attention more than consistent inclusion of previously underrepresented languages.Source: §3.4 · pp. 6–7, Figure 5 · p. 7
  • Publicly available datasets and models generally increased across six regions for the 20 most frequently studied languages, with particularly notable growth among Asian languages.Growth was comparatively limited for North and South American languages and remained concentrated in languages already well represented.Source: §3.4 · p. 7, Figure 6 · p. 7

4 Measuring Linguistic Inclusion

The Linguistic Inclusion Framework (LIF) assesses research growth, representation expansion, continuity, and distributional equity. Its continental application finds that regional strengths differ across dimensions, and no continent scores highly on all four.

4 Measuring Linguistic Inclusion

  • LIF measures research growth, representation expansion, representation continuity, and distributional equity, and can be applied to any language set with yearly publications.The framework is applicable at country, language-family, or continent level.Source: §4 · p. 7

4.1 Framework Application and Discussion

  • Africa and Oceania show the greatest research growth, while Europe shows the weakest relative increase; South America and Oceania lead in new-language expansion.Africa and North America also show substantial new-language expansion, whereas Europe and Asia have lower expansion scores because more represented languages were already present at the start of 2020.Source: §4.1 · pp. 7–8
  • Europe has the strongest language retention, followed by Asia and Africa, while South America has the lowest retention despite the highest new-language inclusion.Equity is calculated over represented languages; top-10 shares are added because equity is inflated when represented-language counts are small.Source: §4.1 · p. 8, Table 2 · p. 8
  • Attention is most widely distributed in Asia, where the top ten languages receive 43% of language-publication instances; Oceania’s top ten receive 100%.Oceania’s ten represented languages account for its full top-10 share; Europe’s top-10 share is 45.5%.Source: §4.1 · p. 8
  • No continent scores highly across all four LIF dimensions, and research growth or expansion alone does not ensure sustained, equitable language representation.Europe combines strong continuity with weakest growth; South America has the most new-language inclusion but lowest continuity.Source: §4.1 · pp. 7–8

5 Conclusion

The paper maps low-resource-language research from 2020 to 2025 and finds that publication growth was concentrated in already represented languages. Its continental LIF assessment finds that no region performs well across all four dimensions.

5 Conclusion

  • The study maps 1,158 peer-reviewed open-access papers on low-resource languages published between 2020 and 2025.The analysis asks whether research growth in the LLM era translated into broader linguistic inclusion.Source: §5 · p. 8
  • Year-to-year transitions show repeated languages increasingly dominate while new-language inclusion declines.Research growth was concentrated in a limited set of languages already represented.Source: §5 · p. 8
  • No region performs well across all four LIF dimensions when the framework is applied at the continental level.The conclusion describes publication growth as concentrated in a limited set of already represented languages.Source: §5 · p. 8

6 Limitation

The study notes limits in its literature collection, annotations, and language-reference sources. It also situates its synthesis against prior work that often focuses on particular tasks, methods, or regions, or predates the LLM era.

6 Limitation

  • The corpus is limited to peer-reviewed open-access papers found through English-keyword searches in Semantic Scholar and the ACL Anthology.This under-represents works published in other languages and closed-access venues; changing terminology may also contribute to observed growth.Source: §6 · p. 9
  • Language classifications, country mapping, and speaker counts rely on sources with gaps, especially for dialects, sign languages, and cross-border languages.Counting a language mention in a massively multilingual benchmark like a paper dedicated to one language may overstate attention.Source: §6 · p. 9

A Related Work

  • Prior surveys often address specific tasks or regions, and several broad surveys predate the LLM era, leaving limited systematic synthesis of recent LRL research.The study presents a unified, up-to-date synthesis focused on how the field evolved in the LLM era.Source: A Related Work · p. 12

B Data Collection and Keywords

The literature search combines language-model terms with terms for languages described as resource-limited or underrepresented, then adds supplementary queries. The affiliation analysis counts each paper once for each distinct author-affiliation country.

B Data Collection and Keywords

C Affiliation

  • The United States (135), Germany (82), China (77), India (72), and the United Kingdom (61) contributed the most papers by author-affiliation country.Each paper is counted once for each distinct country among its affiliations; darker map shades indicate more papers.Source: Figure 7 · p. 13

D.1 Scope filtering

The appendix describes LLM-assisted procedures for screening papers, extracting represented languages, and annotating contributions, NLP topics, and modality.

D.1 Scope filtering

  • The scope screen separates work on genuinely low-resource languages, low-resource domains, simulated scarcity, and non-low-resource studies, prioritizing meaningful inclusion of an LRL.The procedure reports category counts and percentages in Table 5.Source: §D.1 · p. 13

D.2 Language extraction

  • A language is extracted when it appears in a dataset, model training or adaptation, benchmark, evaluation, or experimental analysis, but not merely in background material.Source: §D.2 · pp. 13–14

D.3 LLM prompt for NLP Paper Contribution Annotations

  • Abstract contribution sentences are grouped into artifact and knowledge categories covering methods, datasets, tasks, language, and people.Source: §D.3 · p. 14

D.4 LLM prompt to Annotate NLP Topics

  • The topic prompt assigns NLP topics from a predefined list based on paper titles and abstracts.Source: §D.4 · p. 15

D.5 LLM prompt to annotate modality

  • The modality prompt classifies each paper's data as text, speech, image, multimodal, or other.Source: §D.5 · p. 15

E Human vs. LLM agreement analysis

Human annotators and LLM-generated annotations show high agreement across the reported annotation tasks. The accompanying guidelines define what counts as represented language and distinguish contribution types, topics, and modalities.

E Human vs. LLM agreement analysis

  • 86.02%–100% pairwise human agreement and 96.68%–100% human–LLM agreement were reported across annotation dimensions.Source: E Human vs. LLM agreement analysis · p. 15, Table 6 · p. 15
  • Human agreement for low-resource categorization ranged from 86.66% to 96.01%, reflecting the context-dependent definition of low-resource language.Agreement for language, contribution, and modality extraction ranged from 91.9% to 100% between human annotators and 97.32% to 100% between humans and LLMs; modality extraction had perfect agreement.Source: E Human vs. LLM agreement analysis · p. 15

Annotation guidelines

  • A language counts as represented when it is used in a dataset, training, benchmark, test set, built system, or analysis; background mentions and unused resources do not count.Source: Annotation guidelines · p. 16
  • Contribution labels distinguish newly introduced artifacts from reported knowledge about methods, datasets, tasks, languages, and people.Source: Annotation guidelines · p. 16
  • The guidelines define topic categories and distinguish text, speech, image, multimodal, and other data modalities.Source: Annotation guidelines · p. 17

H.1 Continental distribution of speakers and language representation

The continental comparison measures each region's share of speakers against its share of language representation in the analyzed literature. Asia and Africa are underrepresented relative to their speaker populations, while languages in other regions are comparatively overrepresented.

H.1 Continental distribution of speakers and language representation

  • The comparison calculates continental speaker and representation shares using languages in resource classes 0–3; speaker counts are combined L1 and L2 populations, and representation counts analyzed papers.Source: §H.1 · p. 18
  • 64.1% of speaker share but 46.7% of language representation came from Asian languages, or 0.7× their proportional share; Africa's corresponding shares were 30.7% and 24.9%, or 0.8×.Languages in Europe and other regions receive greater representation relative to their speaker shares.Source: §H.1 · p. 18, Figure 9 · p. 18
  • Languages in the Americas and Oceania also show relatively high representation compared with their small speaker shares, while many Asian and African languages receive comparatively less attention.Source: §H.1 · p. 18, Figure 10 · p. 18

I HuggingFace Identification and Resource Collection

The study identifies publicly accessible Hugging Face models and datasets for selected languages through multiple search routes, then validates candidates and aggregates repository availability over time.

I HuggingFace Identification and Resource Collection

  • Candidate retrieval combines ISO-code tags, language-name and alias searches, multilingual-term searches, and direct lookup of widely used multilingual resources.Results are merged and deduplicated by language, resource type, and repository identifier.Source: I HuggingFace Identification and… · pp. 18–19
  • Candidates scoring at least 5 on the relevance criteria are included automatically, while scores of 1–4 receive manual review and zero-score candidates are discarded.Source: I HuggingFace Identification and… · p. 19
  • A repository's first availability date is the earlier of its Hub creation timestamp and earliest commit, and multilingual repositories count once per target language they cover.Yearly counts include repositories first available from 2020 to 2025, with cumulative counts computed as running sums.Source: I HuggingFace Identification and… · p. 19

Related papers

Loading 2610.30296v1…