Source-linked AI summary
Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding
Khanh-Tung Tran, Barry O'Sullivan, Hoang D. Nguyen
TL;DR
Low-resource languages remain underrepresented in LLM reasoning because post-training data and multilingual alignment are limited. The paper trains models to use English for intermediate CoT while preserving target-language inputs and answers, achieving up to 28.33% improvement and releasing an Irish benchmark.
Problem
Low-resource languages lag in CoT reasoning because high-quality post-training data are scarce and multilingual models exhibit language misalignment.
Method
English-Pivoted CoT Training uses target-language inputs and answers but constrains intermediate reasoning traces to English.
Results
The approach outperforms existing techniques by up to 28.33% in low-resource scenarios and transfers reasoning capability to Irish.
Takeaways & Limitations
Separating language understanding from reasoning provides a practical pathway for multilingual reasoning without retraining extensively in every language.
Abstract
from arXiv · showhide
Recent advances have enabled Large Language Models (LLMs) to tackle reasoning tasks by generating chain-of-thought (CoT) rationales, yet these gains have largely applied to high-resource languages, leaving low-resource languages behind. In this work, we first investigate CoT techniques in extremely low-resource scenarios through previous prompting, model-editing, and fine-tuning approaches. We introduce English-Pivoted CoT Training, leveraging the insight that LLMs internally operate in a latent space aligned toward the dominant language. Given input in a low-resource language, we perform supervised fine-tuning to generate CoT in English and output the final response in the target language. Across mathematical reasoning benchmarks, our approach outperforms other baselines with up to 28.33% improvement in low-resource scenarios. Our analysis and additional experiments, including Mixed-Language CoT and Two-Stage Training, show that explicitly separating language understanding from reasoning enhances cross-lingual reasoning abilities. To facilitate future work, we also release \emph{LC2024}, the first benchmark for mathematical tasks in Irish, an extremely low-resource and endangered language. Our results and resources highlight a practical pathway to multilingual reasoning without extensive retraining in every extremely low-resource language, despite data scarcity.
Introduction
The paper addresses the underrepresentation of low-resource languages in CoT reasoning, where scarce curated data and multilingual misalignment leave models weaker than in English. It proposes English-Pivoted CoT Training and releases LC2024 to transfer reasoning to Irish while keeping target-language interaction.
- Low-resource languages lag behind English in LLM reasoning because high-quality post-training data are scarce and multilingual corpora contain misalignment and bias.
- In Irish, the same mathematical problem that supports robust reasoning in English can produce misunderstanding, failed CoT, and an incorrect answer.
- English-Pivoted CoT Training makes the model interpret target-language inputs, reason in English, and generate final responses in the target language.
- The method improves extremely low-resource reasoning by up to 28.33% and supports reasoning transfer to Irish.
- LC2024 is introduced as the first benchmark for mathematical reasoning in Irish, an endangered language.
Related Works
Prior work improves low-resource reasoning through prompting, multilingual alignment, and model editing, but often depends on extra data, modules, or parallel corpora. This paper instead aligns reasoning within the model by separating English internal CoT from target-language responses.
- English CoT prompting can match competitive native-language prompting, while other alignment methods use parallel sentences or additional multilingual encoders.
- Prior approaches commonly require extra training data, additional modules, or parallel corpora.
- Multilingual models often organize reasoning through English-centric latent representations, with middle layers associated with reasoning in a dominant language.
- The paper differs by aligning reasoning inside the model’s own representations rather than relying on external models or separate bilingual modules.
Transfer
English-Pivoted CoT Training separates target-language understanding and answering from English reasoning by constraining the intermediate CoT to English. Its training objective jointly learns English traces from target-language inputs and target-language responses conditioned on those traces.
- English-Pivoted CoT Training constrains intermediate reasoning to English while keeping both inputs and final answers in the target low-resource language.
- The model factorizes generation into a reasoning trace C followed by an answer y conditioned on the input x and trace C.
- Training data contain target-language inputs, English reasoning traces, and target-language final answers.
- The objective trains English reasoning from multilingual inputs and target-language responses based on generated reasoning, with weights α and β defaulting to 1.0.
- The method aims to align representations across languages so different-language prompts lead to shared reasoning traces while preserving target-language understanding and generation.
- English-Pivoted training focuses parameter updates more on language-comprehension layers, whereas Native CoT makes approximately 1.3-times larger updates across the model.
Experiments
Experiments evaluate English-Pivoted CoT Training against multilingual adaptation baselines across English, Irish, French, and Chinese reasoning tasks. Results show strong low-resource gains, improved Irish language understanding and cross-language representation alignment, sample efficiency, and broader—but resource-dependent—generalization.
- Main results: Up to 28.33% improvement over the r1-distill-Llama-8B baseline is achieved on Irish AIME2024, while LC2024 accuracy reaches 73.33%.The approach attains 35.00% on Irish AIME2024 with r1-distill-Llama-8B and exceeds its 63.64% LC2024 baseline by 10%.
- Language understanding: English-Pivoted CoT Training improves contextual Irish understanding, whereas native-language CoT training can sharply reduce performance on concepts-and-skills questions.The contexts-and-applications split requires deeper Irish comprehension, and the proposed approach yields substantial gains there.
- Representation analysis: English-Pivoted CoT Training preserves almost 100% alignment for Irish questions and reasoning traces, while target-language-only models fall to around 35% after CoTs are included.The result indicates that forcing reasoning in different languages produces divergent internal representations, whereas English-pivoted traces remain stable across languages.
- Sample efficiency: With only 1,000 training samples, English-Pivoted CoT Training beats the base model by over 5%, with accuracy increasing as more data are added.The method decouples reasoning from language learning, requiring comprehension fine-tuning in this low-resource setting.
- Training dynamics: Varying the CoT-loss weight shows that both reducing and over-emphasizing it degrade performance, requiring balance with the final-response objective.This analysis evaluates the trade-off between English reasoning traces and target-language responses.
- Generalization: Performance generalizes across resource regimes, averaging a 14.30% improvement, but English pivoting yields only marginal Chinese gains and can cause interference.French MGSM rises from 54.8% to 83.2%, while Chinese English pivoting slightly improves over a 65.2% baseline.
Conclusion
The paper introduces English-Pivoted CoT Training to transfer reasoning capabilities to low-resource languages by separating language understanding from reasoning. It reports gains up to 28.33% and releases LC2024 to support Irish mathematical reasoning research.
- Conclusion: English-Pivoted CoT Training transfers reasoning capabilities to low-resource languages by aligning reasoning processes across languages.The approach forces CoT traces into the dominant language while targeting low-resource language understanding and generation.
- Conclusion: Performance improves by up to 28.33% over existing techniques and generalizes across low-, medium-, and high-resource language settings.
- Conclusion: LC2024 is introduced as the first Irish mathematical reasoning benchmark, supporting future research on multilingual reasoning.
Ethics Statement
The work focuses on improving accessibility and usability for Irish and other low-resource or endangered languages. Its released data resources are drawn from public sources and licensed for noncommercial research and education.
- Ethics Statement: The work targets Irish and other low-resource or endangered languages to promote linguistic diversity and preserve cultural heritage.
- Ethics Statement: All training and evaluation data come from publicly available sources.
- Ethics Statement: LC2024 is licensed for noncommercial research and educational use only.