PaperlayerView exact revision on arXiv

Source-linked AI summary

Reference-Grounded Data Curation for Instruction-Following Thai-English Machine Translation

Thodsaporn Chay-intr, Krittapad Harnchang, Mahannop, Thabua, Kobkrit Viriyayudhakorn, Thanaruk Theeramunkong

arXiv:2609.34770v1cs.CL

TL;DR

Thai-English machine translation must follow prompt-level rules without losing translation quality, while existing open-weight systems do not close this gap. The paper builds Grounded from constraints demonstrated in reference translations and fine-tunes ChindaMT on it. ChindaMT outperforms or matches size-matched specialists across plain and constrained evaluation, with results corroborated by external metrics, other judges, and human raters.

  • Problem

    Open-weight Thai-English systems either support instruction following without specialized translation quality or deliver translation quality without modeling prompt-level constraints.

    Source: §1 · p. 1

  • Method

    RGDC selects hard-but-learnable parallel examples, extracts constraints already satisfied by their references, and retains regenerated outputs that pass every constraint for ChindaMT training.

    Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 4

  • Results

    ChindaMT outperforms or matches size-matched translation specialists on CoreEval and BroadEval across plain and constrained translation, with findings corroborated by external metrics, cross-judges, and native-Thai raters.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The reference-grounded recipe produces an open-weight Thai-English translation family and transfers across four bases and two Qwen generations.

    Source: §7 · p. 9

    RGDC supervises only constraints demonstrated by references, leaving demands without reference examples or without a yes/no check outside its coverage.

    Source: §8 · p. 9

Abstract

from arXiv · show

Instruction-following machine translation (IF-MT) requires respecting prompt-level rules on terminology, formatting, and register. Rule compliance typically trades off against translation quality, a tension that general-purpose IF data augmentation methods do not address. We propose Reference-Grounded Data Curation, a two-phase pipeline that extracts every supervised constraint from a reference translation that already satisfies it, ensuring feasibility by construction. Phase 1 applies Instruction-Following Difficulty (IFD) scoring to retain the hardest-but-learnable instances from an English-Thai parallel pool. Phase 2 extracts constraints from each reference target and keeps only generations satisfying every constraint, yielding the 1.97M-record Grounded dataset. We fine-tune open-weight bases on Grounded to produce ChindaMT, a Thai-English translation family at 4B, 2B, and 0.8B parameters. Under length-controlled pairwise judging, ChindaMT outperforms or matches every same-size baseline at every tier on both plain translation and under explicit rules, reaching up to a 68.4% win rate against the strongest baseline. The recipe transfers cleanly across Qwen generations. We release model weights, the Grounded dataset, and evaluation suites.

1 Introduction

The paper targets Thai-English translation that follows prompt-level rules without sacrificing translation quality, a gap left between instruction-following Thai models and translation-specialized systems. It introduces reference-grounded data curation and ChindaMT, whose evaluations show gains over same-size translation specialists across plain and constrained translation.

Motivation

  • Thai-focused LLMs support instruction following but lack specialized translation quality, while multilingual MT specialists deliver translation quality without modeling prompt-level constraints.The paper states that no open-weight Thai-English model currently closes this gap.Source: §1 · p. 1

Approach

  • RGDC uses IFD to select hard-but-learnable parallel examples, then extracts reference-satisfied constraints and retains regenerated outputs that pass every constraint.The pipeline grounds supervised rules in existing reference translations.Source: §1 · pp. 1–2

Evaluation

  • The evaluation uses length-controlled pairwise win rates against same-size baselines on Plain and Constrained splits across two suites.The splits cover translation alone and translation with explicit rules, respectively.Source: §1 · p. 2

Results

  • ChindaMT outperforms or matches size-matched MT specialists at all three sizes on plain and constrained translation, with wins extending to BroadEval and corroborated by external metrics, cross-judges, and human raters.The same recipe also transfers to Qwen3-4B as a cross-generation check.Source: §1 · p. 2

Contributions

  • The authors release ChindaMT at 4B, 2B, and 0.8B parameters, together with the Grounded dataset and CoreEval and BroadEval suites.The family is open-weight.Source: §1 · pp. 1–2

2 Related Work

Prior work provides instruction-data selection and augmentation methods, but these approaches were not adapted to translation. The paper positions RGDC as reference-grounded constraint curation for Thai-English instruction-following machine translation.

2.1 Data Selection for Instruction Tuning

  • IFD selects examples where an instruction reduces but does not eliminate output loss, targeting hard-but-learnable instances; the paper adapts it to translation because difficulty is measured for the model being fine-tuned.Deita scores complexity and quality, while SuperFiltering approximates the IFD criterion with a weaker model to save cost.Source: §2.1 · p. 2

2.2 Instruction-Following Data Augmentation

  • Unlike synthesize-then-filter methods such as AutoIF and UltraIF, RGDC extracts constraints from existing references, making each constraint feasible by construction; its all-pass filter handles residual generation hygiene.General instruction-data selection and augmentation had targeted general IF benchmarks rather than translation.Source: §2.2 · p. 2

2.3 Machine Translation with LLMs

  • Prior Thai-English systems either support instruction following without specialized translation quality or provide translation quality without modeling prompt-level constraints.Standard MT metrics assess translation quality but not rule compliance.Source: §2.3 · p. 2

2.4 LLM-as-Judge Evaluation

  • Figure 1 depicts RGDC’s two phases: IFD selection from the parallel pool, followed by reference-grounded constraint extraction and all-pass filtering to form Grounded.The resulting dataset trains the ChindaMT family.Source: Figure 1 · p. 3

3 Methodology

RGDC selects difficult-but-learnable parallel examples, then builds training examples from constraints already demonstrated by their reference translations. Only regenerated outputs satisfying all extracted constraints enter Grounded.

3.1 Pipeline Overview

  • RGDC scores an English-Thai parallel pool for difficulty, then extracts constraints from retained references and keeps regenerated outputs that satisfy every constraint.The constraints are feasible by construction because the references already satisfy them.Source: §3.1 · p. 3

3.2 Phase 1: Cherry Data Selection

  • The IFD stage briefly fine-tunes a base model, scores records by output-loss ratio with and without the instruction, and retains the top decile among records with IFD at least 1.This yields about 1.75M hard-but-learnable cherry samples; IFD below 1 indicates the instruction eases prediction and is discarded.Source: §3.2 · p. 3

3.3 Phase 2: Constraint Augmentation

  • For each cherry sample, an auxiliary LLM extracts verifiable constraints that its reference target already demonstrates, then generates a yes/no question for each constraint.The extractor uses a taxonomy of five constraint categories.Source: §3.3 · p. 3, Table 1 · p. 3
  • Each sample produces a variant with 1–3 randomly selected constraints and a variant with all extracted constraints; only candidates judged YES on every constraint are retained.Retained constraints become a Rules: block in the training instruction, yielding about 1.97M records.Source: §3.3 · p. 4

4 Experimental Setup

The experiments draw on a large bilingual parallel pool, construct Grounded and two evaluation suites, and compare four ChindaMT variants with size-matched systems. Training and judging use a shared recipe and length-controlled pairwise evaluation.

4.1 Source Parallel Corpora

  • The shared source pool contains about 17.85M records across both translation directions, with evaluation items held out from both RGDC phases from the outset.Ten English-Thai parallel corpora supply the two RGDC phases.Source: §4.1 · p. 4

4.2 Grounded Training Dataset

  • Grounded contains 1.97M source, Rules: block, and constraint-compliant translation records; retained records average 3.7 constraints, and 56.5% of candidates survive.The source pool contains about 17.85M records from ten English-Thai parallel corpora.Source: §4.2 · p. 4, §4.1 · p. 4

4.3 Evaluation Datasets

  • CoreEval and BroadEval each contain 400 samples per format, split evenly by translation direction; CoreEval covers five deployment domains, while BroadEval spans ten broader domains.BroadEval includes about 25% paragraph-length inputs, and both suites have Plain and Constrained formats.Source: §4.3 · p. 4

4.4 Models Compared

  • Four ChindaMT variants share one Grounded fine-tuning recipe: Qwen3.5 bases at 4B, 2B, and 0.8B, plus Qwen3-4B for cross-generation transfer.Comparisons include size-matched Thai-capable MT specialists and each variant’s own base.Source: §4.4 · p. 4

4.5 Implementation Details

  • Phase 1 samples 40,683 records for selection training, and the four LLM-based Phase 2 steps share Qwen3.5-35B-A3B-FP8; all variants receive one epoch of full-parameter SFT without preference optimization or RL.The implementation preserves an effective batch size of 64 across model tiers.Source: §4.5 · pp. 4–5

4.6 Evaluation Protocol

  • The primary judge reports length-controlled win rate using AlpacaEval-v2’s length-bias correction, with accuracy and faithfulness prioritized before rule compliance and fluency.Because the primary judge shares the Qwen family with ChindaMT and Typhoon-Translate, comparisons are re-judged by two independent-family open-weight judges; Plain and Constrained win rates reflect different task conditions.Source: §4.6 · p. 5

5 Results

ChindaMT leads or matches same-size baselines on plain and constrained translation, with external metrics, cross-judges, and human ratings broadly corroborating the results. The section also examines external quality, validation across judges and raters, scaling, and the contributions of RGDC components.

5.1 Main Pairwise Results

  • 61.8 on Plain (shared) and 68.4 on Constrained are ChindaMT-4B’s LC% against same-size Typhoon-Translate-4B on CoreEval, where it outperforms every same-tier and larger baseline.On BroadEval, ChindaMT-4B and 2B lead every reportable size-matched comparison without a loss; ChindaMT-0.8B leads its size-matched baselines except for a tie with GemmaX2-28-2B within 0.6 of 50.Source: §5.1 · p. 5, Table 3 · p. 6

5.3 Cross-Judge and Human Validation

  • All four validation channels agree on direction, while cross-judge Constrained agreement is decisive and native-Thai raters prefer ChindaMT-4B on both Plain and Constrained pairs.Across two additional judges, judge spread is under three points on Constrained comparisons; human preference intensity is stronger on Constrained (+0.68) than Plain (+0.39).Source: §5.3 · p. 6, Table 5 · p. 8, Table 6 · p. 8

5.4 Method Lift and Parameter Scaling

  • All four Qwen fine-tunes outperform their bases, and the gains from Grounded increase as base-model parameters decrease.Within-family Plain lifts compress from +12.6 for 4B versus 2B to +6.9 for 2B versus 0.8B, while Constrained lifts remain roughly constant at +14.2 and +14.4; a Gemma-3-4B fine-tune also outperforms its base, supporting transfer across model families.Source: §5.4 · pp. 6–7

5.2 External-Benchmark Validation

  • 90.7 is ChindaMT-4B’s FLORES Larger Mean, tied with MiLMMT-46-4B; ChindaMT-2B leads its tier on FLORES (88.9) and ties for the WMT24++ lead (85.5).On WMT24++ Larger, ChindaMT-4B scores 87.8 versus HY-MT-1.5-7B at 89.4 and MiLMMT-46-4B at 87.9.Source: §5.2 · p. 5, Table 4 · p. 7

5.5 Component Ablations

  • Removing the Rules block reduces LC by +12.63 on Plain and +9.53 on Constrained relative to the full recipe, with losses above standard error on every comparison.The full recipe also gains +8.24 over no-cherry on Plain and +0.69 on Constrained; removing the all-pass filter yields smaller, consistently positive lifts of +1.09 and +1.06.Source: §5.5 · pp. 7–8

6 Analysis

The analysis frames translation quality and instruction following as a trade-off among specialized systems, while arguing that reference-grounded supervision can address both. A qualitative example contrasts systems that satisfy only one of the two demands with ChindaMT-4B, which satisfies both in that case.

6.1 Translation–IF Trade-off

  • Translation specialists can retain plain-translation parity yet lose constrained comparisons, while RGDC grounds rule and translation supervision in the same reference translations.The analysis attributes part of the gains to broad output-distribution alignment rather than a narrow instruction-following mechanism; external metrics show parity rather than dominance, with a residual HY-MT-1.5-7B edge on WMT24++ th-en news.Source: §6.1 · p. 8

6.2 Qualitative Case Analysis

  • In the example, Typhoon-Translate leaks the rule block and an EN: prefix, the Qwen3.5-4B base omits “web,” and ChindaMT-4B follows the rule while preserving that content.The example illustrates the two failure modes: accurate translation without compliance, and compliance with a faithfulness loss.Source: §6.2 · p. 8

7 Conclusion

The conclusion presents RGDC as a two-phase training-data pipeline whose reference-grounded constraints are feasible by construction. ChindaMT matches or outperforms size-matched translation specialists across both evaluation suites, and the authors release the models and data.

7 Conclusion

  • ChindaMT outperforms or matches size-matched translation specialists at every tier on CoreEval and BroadEval, without sacrificing raw translation quality on FLORES-200 or WMT24++.Two open-weight cross-judges and native Thai raters corroborate the results.Source: §7 · p. 9
  • RGDC extracts rules already satisfied by reference translations, making every supervised constraint feasible by construction.The pipeline builds the training data for ChindaMT.Source: §7 · p. 9
  • The released open-weight ChindaMT family spans 4B, 2B, and 0.8B parameters, and the recipe transfers across four bases and two Qwen generations.The release includes Grounded, evaluation suites, and code under open licenses.Source: §7 · p. 9

8 Limitations

The authors identify reference-bounded constraint coverage, reliance on one primary judge, limited language-pair and register coverage, prompt-format asymmetry, and Qwen-only base models as limitations.

8 Limitations

  • RGDC can supervise only constraints exemplified by references, excluding demands such as mandated glossaries and placeholder tokens; broader grounding or synthetic generation is future work.Covered instructions include term, length, register, and output format.Source: §8 · p. 9
  • Absolute LC values may reflect the calibration of the single primary judge, Qwen3.6-35B, despite checks using external metrics, cross-judges, and human raters.Direction agreement holds across all three judge families on every Constrained and Plain (shared) comparison.Source: §8 · p. 9
  • Experiments cover English–Thai in both directions only, and extension to other low-resource pairs and broader registers remains future work.The Grounded format matches Typhoon-Translate but differs from HY-MT and GemmaX2; Constrained evaluation therefore omits some GemmaX2 comparisons.Source: §8 · p. 9
  • All four ChindaMT bases are Qwen; transfer across Qwen3.5 and Qwen3 is shown, but transfer to other architectures remains untested.The reported Gemma-3-4B lift used Qwen-derived Phase 1 selection.Source: §8 · p. 9

9 Ethical Considerations

The authors describe the data and evaluation releases, human evaluation procedures, intended-use risks, and compute considerations.

9 Ethical Considerations

  • The Grounded dataset and evaluation suites use CC-BY-SA 4.0, while one BroadEval item retains its source's CC-BY-NC-4.0 terms.The suites include source text and model outputs but no reference translations or personally identifying information.Source: §9 · p. 9
  • Three native Thai speakers volunteered without compensation, consented after being informed of the study and withdrawal rights, and rated anonymized pairs in a minimal-risk task.No personal data were collected, and the study required no formal ethics-board review at the institution.Source: §9 · p. 9
  • ChindaMT can produce fluent but incorrect output or reflect training-data bias, so high-stakes use warrants human review.Source: §9 · p. 9
  • Model weights and code are released under Apache-2.0, with weights inheriting the Qwen base-model license; the small models keep compute modest and accessible.The Grounded dataset and evaluation suites are released under CC-BY-SA 4.0.Source: §9 · p. 9

A Detailed Comparison with Prior IF-Data-Augmentation Methods

This section compares RGDC with synthesize-then-filter augmentation and describes the compliance evaluation and English–Thai parallel-corpus pool used by the curation pipeline.

A Detailed Comparison with Prior IF-Data-Augmentation Methods

  • RGDC extracts constraints from already-compliant references, enabling an all-pass filter and 1.97M translation-specialized records—an order of magnitude beyond AutoIF and UltraIF’s general-domain corpora.The comparison covers seven axes and identifies AutoIF and UltraIF as the closest prior methods.Source: A Detailed Comparison with Prior… · p. 12

B RGDC Pipeline Prompts

  • Phase 2 chains constraint extraction, verification-question generation, constrained regeneration, and compliance scoring into one record.The prompt workflow comprises stages 2A–2D.Source: B RGDC Pipeline Prompts · p. 13

B.1 Constraint Extraction (2A)

  • Constraint extraction derives new, verifiable requirements absent from the query by comparing it with its existing reference answer.The five constraint categories match the evaluation-time taxonomy.Source: §B.1 · p. 13

B.2 Evaluation Question Generation (2B)

  • Each extracted constraint becomes a yes/no evaluation question; meaningless or embedded constraints receive an empty-string fallback and are excluded from filtering.Questions are batched into one LLM call per record, making compliance objectively checkable.Source: §B.2 · p. 14

B.3 Constrained Response Generation (2C)

  • Each record generates sampled (1–3 constraint) and all-constraint variants by appending a Rules block to the original instruction.The system prompt suppresses introductory preamble so each target remains a clean translation.Source: §B.3 · p. 14

B.4 Compliance Evaluation (2D)

  • A candidate passes only when every yes/no question receives YES, with a justification required before each verdict to reduce label noise.A single NO drops the record under the all-pass filter.Source: §B.4 · p. 15
  • The final response is checked against each evaluator requirement, with a justification and a YES/NO compliance score for each question.The evaluator prompt requires JSON output.Source: §B.4 · p. 15

C Source Parallel Corpora

  • About 17.85M records from 10 publicly available English–Thai parallel corpora are unified in Alpaca format; Phase 1 selects from this pool and Phase 2 augments the selected subset.Source: C Source Parallel Corpora · p. 15

D Grounded Dataset Record Example and Quality Audit

The section documents the Grounded record format and audit, then details evaluation protocols, metrics, and comparisons. Regenerated targets match or exceed their source-corpus references on 85.9% of audited records overall.

D Grounded Dataset Record Example and Quality Audit

  • A Grounded record stores the task, source, and Rules block in instruction, leaves input empty, and places the constraint-compliant translation in output.The example uses a Thai-to-English source and rules for professional customer-service style, one sentence, and retaining the number 35.Source: D Grounded Dataset Record Example and… · p. 15
  • 85.9% of 20,000 regenerated targets match or exceed their source-corpus references overall, with more than 80% in each direction.The audit uses CometKiwi, independent of the pairwise judge; retained constraints are 77% content, linguistic, and stylistic and 23% format and numerical.Source: D Grounded Dataset Record Example and… · pp. 15–16, Table 10 · p. 16

E Evaluation Prompt Templates

G Implementation Hyperparameters

  • Implementation specifies auxiliary-LLM decoding, shared fine-tuning, and inference settings; fine-tuning used two H100s and Phase 2 generation was the dominant cost.Fine-tuning took about 68 wall-clock hours across four variants, while Phase 2 generation ran for roughly two weeks of active generation.Source: G Implementation Hyperparameters · p. 16, Table 12 · p. 16, Table 13 · p. 16

H Evaluation Judge Prompt

  • The pairwise judge prioritizes accuracy and faithfulness, then instruction and format compliance, then fluency, and penalizes extra non-translation content.Raw win rate counts wins plus half of ties; LC% estimates preference at matched reference length to correct for length bias.Source: H Evaluation Judge Prompt · p. 17, §I.1 · p. 17, §I.2 · p. 17

I.4 Metric alignment with LC%

J Per-Direction Pairwise Win Rates

  • ChindaMT-4B and ChindaMT-2B lead all size-matched FLORES comparisons and most WMT24++ comparisons, while HY-MT has its clearest edge against the larger 7B specialist on WMT24++ Thai-to-English.Table 15 breaks CoreEval comparisons out by direction and applies the pairwise protocol to plain-translation external benchmarks.Source: J Per-Direction Pairwise Win Rates · p. 18, Table 15 · p. 19

L Per-Judge Cross-Judge Numbers

O External-Metric Ablations at 4B

The 4B ablations show that cherry selection mainly improves CoreEval Plain performance, while rule-block training contributes to external translation quality. A qualitative case illustrates a rule-format violation by Typhoon-Translate.

O External-Metric Ablations at 4B

  • For cherry-selection ablation, no-cherry slightly edges the full recipe out of domain: Mean rises from 90.65 to 90.88 on FLORES (+0.23) and from 87.81 to 88.12 on WMT24++ (+0.31).The 8.24 Plain CoreEval gain therefore reflects in-domain alignment, not a uniform quality boost.Source: O External-Metric Ablations at 4B · p. 19
  • 0.92 points on FLORES and 0.72 on WMT24++ separate full-recipe Means from no-rules Means, which are lower on every external metric in both datasets.The largest drops come from GEMBA-DA, indicating a translation-quality regression rather than only a narrow instruction-following loss.Source: O External-Metric Ablations at 4B · p. 19, Table 18 · p. 20

P Qualitative Case Transcript

  • In the Thai-to-English case, Typhoon-Translate echoes the Rules block and an EN: prefix despite the instruction to return only the translation.ChindaMT-4B and Qwen3.5-4B both follow that output-only rule in the example.Source: P Qualitative Case Transcript · p. 20

Related papers

Loading 2609.34770v1…