PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?

Renhao Pei, Yihong Liu, Sampo Pyysalo, Hinrich Schütze, Shaoxiong Ji

arXiv:2606.03782v1cs.CL

TL;DR

Extremely low-resource MT lacks parallel data, while LLMs struggle to apply grammatical information effectively. The paper generates step-by-step linguistic reasoning traces from linguistic resources and evaluates them through ICL, SFT, and RFT. Reliable traces substantially help at inference time, whereas training models to generate accurate analyses remains limited.

  • Problem

    LLMs struggle to reason over grammatical descriptions during low-resource MT, despite the importance of grammatical information for understanding such languages.

  • Method

    The paper generates sentence-specific reasoning traces from UD treebanks, dictionaries, and grammar-rule banks, then evaluates them in ICL, SFT, and RFT.

  • Results

    Linguistic reasoning traces substantially improve ICL translation performance, while SFT and RFT produce smaller and less consistent gains than ICL.

  • Takeaways & Limitations

    LLMs can leverage grammatical information when given reliable analyses, but generating those analyses accurately remains a bottleneck.

  • Takeaways & Limitations

    The RFT experiments use limited sampled generations and batch size, and their reward function does not directly reward syntactic analysis.

Abstract

from arXiv · show

Large language models (LLMs) offer a promising approach to machine translation (MT) for extremely low-resource languages by incorporating linguistic resources through in-context learning. However, LLMs often struggle to apply grammatical information effectively during translation. Inspired by recent progress in chain-of-thought reasoning, we investigate whether low-resource MT can benefit from structured intermediate steps of linguistic analysis and grammatical reasoning. We propose a pipeline for automatically generating step-by-step linguistic reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar-rule banks. We evaluate these traces in three settings: in-context learning (ICL), supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT), on Xibe and Chintang as test cases. Our results show that linguistic reasoning traces are most effective as inference-time guidance: in ICL, reliable sentence-specific traces substantially improve translation performance across most models, languages, and metrics. In contrast, using the linguistic reasoning traces as training data yields smaller and less consistent gains, as models learn the trace format but often generate erroneous content. These findings suggest that LLMs can leverage grammatical information for low-resource MT when given reliable linguistic analyses, while learning to generate such analyses remains a major bottleneck.

1 Introduction

Extremely low-resource languages often lack the parallel data needed for dedicated MT systems, despite having substantial linguistic documentation.

  • More than 7,000 languages exist worldwide, but only a small fraction have sufficient parallel data for dedicated MT systems.
  • Linguistic resources such as dictionaries, grammar books, and annotated treebanks are available for many low-resource languages.

BLEU chrF SBERT LLMaJ

The paper asks whether structured linguistic reasoning can help LLMs apply grammar during extremely low-resource MT. It introduces a trace-generation pipeline and evaluates reasoning traces through prompting and fine-tuning, finding stronger benefits from inference-time guidance.

  • ICL outperforms the training-based settings on all four metrics for Qwen3-8B on Chintang.
  • LLMs struggle to reason over grammatical descriptions during in-context MT, so grammar rules alone may not provide sufficient guidance.
  • The proposed approach decomposes translation into lexical and morphosyntactic analysis, grammar application, and composition of intermediate meanings.
  • The pipeline automatically generates step-by-step linguistic reasoning traces using UD treebanks, dictionaries, and modularized grammar rules.
  • The traces are evaluated as in-context learning, supervised fine-tuning, and reinforcement fine-tuning, each against a baseline without reasoning traces.
  • Structured traces are more effective as inference-time guidance than as training supervision, while models still struggle to generate accurate analyses themselves.

2 Related Work

Prior low-resource MT work incorporates dictionaries, grammar books, and treebanks into prompts, but retrieval and application of complex grammar remain difficult. This paper directly exploits UD structure to generate grammatical reasoning traces for extremely low-resource translation.

  • Dictionary entries consistently improve low-resource MT, whereas reported grammar-book gains may come from parallel example sentences.
  • Grammar-rule retrieval is a bottleneck, and LLMs also struggle with complex grammar rules.
  • Prior UD-based prompting does not explicitly exploit syntactic tree structure, with limited gains over baseline.
  • The paper leverages UD tree structure to generate step-by-step traces that mirror sentence-level syntactic composition.
  • Earlier reasoning-oriented MT frameworks target relatively high-resource languages and emphasize drafting, refinement, or proofreading.
  • This approach instead targets extremely low-resource languages and uses grammatical reasoning to recover basic source-sentence semantics.

3 Languages, Data and General Setup

The experiments use Xibe and Chintang resources, UD annotations, dictionaries, grammar rules, and several instruction-tuned LLMs. A modular rule bank is matched to UD-parsed sentences to construct reasoning traces for low-resource-to-English MT.

  • Languages: Xibe is a Tungusic language spoken in Northwest China with around 30,000 native speakers, while Chintang is a Sino-Tibetan language spoken in Nepal with around 5,000 speakers.
  • General setup: All experiments translate from the low-resource language into English.
  • UD treebanks: UD represents sentence structure through head–dependent relations and includes word forms, lemmas, POS tags, dependency relations, and morphological features.
  • UD treebanks: A maximum sentence length of 30 words retains 979 of 1,200 Xibe trees and all 2,289 Chintang trees.
  • Dictionaries: The dictionaries combine language-specific resources, with Xibe entries taking precedence over supplementary Manchu entries, while Chintang entries derive from UD glosses.
  • Grammar rules: Each grammar rule pairs a textual explanation with a UD-based trigger such as a dependency relation, feature–value pair, or POS tag.
  • Grammar rules: The final grammar-rule bank contains 77 rules for Xibe and 82 for Chintang.
  • Models: The experiments use instruction-tuned Qwen3 models at 4B, 8B, and 14B and Gemma 4 models at E2B, E4B, and 31B.

4 Generation of Linguistic Reasoning Traces

The pipeline converts UD trees and linguistic resources into structured, step-by-step traces that progress from word-level meanings through phrasal composition to sentence translation.

  • Trace construction: The pipeline uses UD treebanks, dictionaries, and grammar rules to generate traces progressing from lexical and morphological meanings to full-sentence translation.The traces include language-specific grammar rules, syntactic relations, and partial phrasal meanings.
  • Trace construction: Reasoning steps follow a bottom-up post-order traversal of the UD tree, combining each head with its dependents before composing larger subtrees.Dependents of a head are processed in ascending index, corresponding to left-to-right surface order.
  • Trace verbalization: Each step verbalizes tokens’ POS tags, lemmas, morphological features, triggered grammar rules, and lexical meanings.Syntactic relations are also verbalized with word order, POS tags, and UD dependency relations.
  • Trace verbalization: The procedure inserts lexical and phrasal meaning placeholders after word-level and syntactic explanations, producing traces for in-context guidance.The same filled-in intermediate translations can provide supervision for process rewards in reinforcement fine-tuning.
  • Trace completion: For training data, an LLM fills placeholders using dictionary entries and the final gold translation, but the resulting template does not mirror the LLM’s own reasoning process.Xibe requires lexical and phrasal filling, whereas Chintang requires only phrasal filling because lexical meanings are available from UD annotations.

5 In-Context Learning Experiment

The in-context experiment compares flat linguistic prompts with UD-derived reasoning traces that guide models through sentence-specific analysis before translation. Reasoning traces improve performance for most models, languages, and metrics, with especially large gains for Chintang and less consistent benefits for the smallest model.

  • Prompting setup: The baseline prompt provides dictionary entries and triggered grammar rules, while the reasoning prompt organizes the same information into a sentence-specific analytic path.Models resolve trace placeholders sequentially before producing the final translation.
  • Results: Reasoning traces substantially improve in-context MT across languages, metrics, and most models, with particularly consistent gains on SBERT.The results suggest improvements in semantic closeness to the references.
  • Results: For Xibe, improvements are more moderate but remain mostly positive.The table caption reports gains from adding reasoning traces with placeholders and notes an exception for gemma-4-E2B-it.
  • Results: Benefits are less consistent for gemma-4-E2B-it, whose BLEU decreases while SBERT increases and whose chrF and LLMaJ results are mixed across languages.The passage attributes this pattern potentially to smaller capacity and lower baseline performance, which may increase susceptibility to noise.
  • Conclusion: Overall, structured linguistic reasoning provides useful in-context guidance and substantially improves translation without additional training.The experiment evaluates the approach on the test split of the treebank.

6 Supervised Fine-Tuning Experiment

The SFT experiment tests whether models can learn linguistic reasoning from training traces by comparing translation-only and reasoning-plus-translation supervision. Reasoning supervision is less consistently beneficial than ICL guidance, partly because models reproduce trace form more readily than accurate reasoning content.

  • Experimental design: The training dataset uses completed traces, with prompts containing task instructions, dictionary entries, triggered grammar rules, and source sentences.Answers contain a reasoning trace followed by the final English translation.
  • Experimental design: The dataset is split into 80% training, 5% validation, and 15% test sets, with validation selecting checkpoints and test data used for final scores.The test set contains the same sentences as the previous ICL experiment.
  • Experimental design: SFT compares models fine-tuned with final translations alone against models trained to generate reasoning traces before final translations.Both settings are evaluated relative to unfine-tuned models on the same test sentences used in the ICL experiment.
  • Results: +4.77 BLEU, +19.22 chrF, +15.19 SBERT, and +3.42 LLMaJ are reported for Qwen3-4B-Thinking-2507 on sjo relative to the unfine-tuned baseline.For ctn, the same model reports +3.01 BLEU, +18.36 chrF, +9.09 SBERT, and +1.12 LLMaJ.
  • Interpretation: SFT improvements cannot be attributed solely to reasoning traces because SFT without reasoning also achieves strong gains from fine-tuning on final translations.The model’s relatively low baseline also means larger gains do not necessarily indicate high final performance.
  • Interpretation: Compared with ICL, SFT traces are less beneficial because models learn their format and style but often generate erroneous reasoning content.These content errors limit further improvement in final translations.

7 Reinforcement Fine-Tuning Experiment

The RFT experiment tests whether reinforcement fine-tuning can improve SFT models that produce linguistic reasoning traces. RFT yields only small, mixed performance changes, while incorrect analyses remain a bottleneck.

  • Method: RFT continues training SFT models with reasoning traces using GRPO and reward functions combining translation, format, and process checks.The translation reward uses sentence-level chrF, BLEU, and SBERT; format checks require a <think> block with at least one Step and an <answer> block.
  • RFT Results: RFT produces only small, mixed changes across metrics on sjo and ctn, without substantial improvements beyond SFT with reasoning traces.Both gains and degradations remain limited under the current RFT setup.
  • Analysis: Models learn the expected step-by-step format, but their reasoning often contains incorrect sentence structures, dependency relations, and word senses.These errors suggest that insufficient knowledge for accurately analyzing low-resource languages may be the main bottleneck.
  • SFT Results: SFT with reasoning tends to outperform SFT without reasoning more often than the reverse, although gains vary across metrics and models.Table 2 compares reasoning and non-reasoning SFT against corresponding pretrained baselines on sjo and ctn.
  • Comparison with ICL: Reliable gold-standard traces outperform model-generated analyses because incorrect analyses can propagate into final translations and limit translation quality.The RFT-on-SFT setting still lags far behind ICL, where sentence-specific traces provide reliable linguistic guidance.

8 Conclusion

The paper develops and evaluates automatically generated linguistic reasoning traces for low-resource MT. Traces help most as reliable inference-time guidance, whereas training models to generate accurate analyses remains a key bottleneck.

  • Conclusion: The pipeline generates linguistic reasoning traces and evaluates them in ICL, SFT, and RFT against corresponding baselines without traces.The evaluation uses low-resource MT settings and compares inference-time guidance with training-based supervision.
  • Conclusion: Linguistic reasoning traces are most effective as in-context guidance because they provide reliable sentence-specific analyses and substantially improve translation performance.Using the same traces as training data produces smaller and less consistent gains.
  • Conclusion: Models can reproduce the trace format during training but often generate imperfect reasoning content, limiting improvements in final translation quality.Further RFT does not bring meaningful improvements over SFT.

Limitations

The RFT results are constrained by limited exploration and reward signals that do not directly assess syntactic analysis. Future work proposes verifying predicted dependency analyses against gold UD structures.

  • RFT Exploration: Computational constraints limit sampled generations and batch size, restricting exploration of linguistic reasoning trajectories during RFT.The observed limited gains may therefore reflect an RL setup insufficient for reliably discovering correct reasoning trajectories.
  • Reward Design: The reward function primarily optimizes MT metrics and does not directly reward syntactic analysis.The intermediate process reward checks surface-level phrasal translations rather than syntax, potentially weakening learning of accurate reasoning.
  • Future Work: Future work could verify predicted dependency analyses against gold UD tree structures to provide a stronger syntactic-learning reward.More accurate syntax could improve models’ ability to exploit grammatical information for downstream translation.

Ethical Considerations

The paper reports AI assistance and implementation details for training, decoding, prompting, evaluation, and RFT reward construction. The supplied passages do not state additional ethical considerations.

  • Disclosure: The authors used ChatGPT for grammar correction, clarity, and coherence, and OpenAI Codex for code implementation assistance.They retain responsibility for the paper’s technical contributions, experimental design, analyses, and final content.
  • Reward Function: RFT uses a weighted reward combining translation quality, format compliance, and process matching, with top-level weights 0.75, 0.10, and 0.15.Translation quality is the dominant optimization target while structural compliance and intermediate reasoning are also rewarded.
  • Reward Components: The translation reward combines sentence-level chrF, sentence-level BLEU, and SBERT, while format and process rewards assess output structure and intermediate partial translations.Process matching compares generated and gold partial translations using exact match, chrF, and SBERT-based similarities.
  • Prompting: Prompting procedures fill lexical meanings and phrasal translations while preserving the prescribed reasoning-step formatting and square brackets.The prompts use dictionary entries and syntactic relations to build phrase and sentence meanings.

D Example of Erroneous Reasoning

The Xibe example contrasts an erroneous reasoning trace with a gold trace, showing that similar stepwise formats can produce different lexical, syntactic, and translation analyses.

  • Lexical analysis: The generated trace assigns ’hvlhambi’ the sense [is robbing], while the gold trace uses [steals].The generated analysis is marked as a wrong lexical sense for a polysemous word.
  • Lexical analysis: The generated trace interprets ’gidame’ as [pressing], whereas the gold trace analyzes it as [closing/shutting].The traces associate the form with different lemmas and lexical meanings.
  • Syntactic analysis: The generated trace incorrectly labels ’honggon’ as a compound of ’hvlhambi’, while the gold trace identifies it as the verb’s object.The generated syntactic relation is explicitly marked wrong; the gold analysis uses the object relation.
  • Final translation: The generated trace consequently produces ’He is robbing small bells at the ear,’ while the gold trace produces ’Cover ears while stealing a bell.’The differing final translations reflect the earlier lexical and syntactic analyses.
  • Overall comparison: The comparison shows that the generated trace can follow the required reasoning format while still containing lexical-selection and syntactic-analysis errors.The table caption states that errors are annotated separately from the corresponding correct analyses.
Loading 2606.03782v1…