PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Document Translation via Format Reinforcement Learning

Haiyue Song, Johannes Eschbach-Dymanus, Hour Kaing, Sumire Honda, Hideki Tanaka, Bianka Buschbeck, Masao Utiyama

arXiv:2512.05100v1cs.CLcs.AIcs.LG

TL;DR

Structured-document translation must preserve markup as well as content, but token-level supervised fine-tuning leaves markup accuracy largely unaddressed. FormatRL applies GRPO to an SFT model with structure-aware rewards, and experiments report stronger structural fidelity without compromising translation quality across inline and complex documents.

  • Problem

    Structured-document translation requires both content quality and markup fidelity, while supervised fine-tuning's token-level objective leaves markup accuracy largely unaddressed.

    Source: §1 · p. 1

  • Method

    FormatRL fine-tunes an LLM, then applies GRPO with structure-aware rewards including TreeSim and Node-chrF to optimize structured-document translation.

    Source: §1 · pp. 1–2

  • Results

    FORMATRL improves structural fidelity without compromising translation quality across simple inline markup and complex documents; it reports average gains of 3.69 XML-Match and 2.16 XML-BLEU on software documentation across four translation directions.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    StrucAUC provides a fine-grained evaluation metric for assessing structured-document translation.

    Source: §7 · p. 9

    The study did not evaluate whether models generalize to documents containing previously unseen tags because generating diverse synthetic markup required substantially more data and budget.

    Source: §8 · p. 10

Abstract

from arXiv · show

Recent works on structured text translation remain limited to the sentence level, as they struggle to effectively handle the complex document-level XML or HTML structures. To address this, we propose \textbf{Format Reinforcement Learning (FormatRL)}, which employs Group Relative Policy Optimization on top of a supervised fine-tuning model to directly optimize novel structure-aware rewards: 1) TreeSim, which measures structural similarity between predicted and reference XML trees and 2) Node-chrF, which measures translation quality at the level of XML nodes. Additionally, we apply StrucAUC, a fine-grained metric distinguishing between minor errors and major structural failures. Experiments on the SAP software-documentation benchmark demonstrate improvements across six metrics and an analysis further shows how different reward functions contribute to improvements in both structural and translation quality.

1 Introduction

Structured-document translation must preserve markup as well as translate content, but existing LLM training leaves markup accuracy largely unaddressed. FORMATRL applies reinforcement learning with structure-aware rewards, and the paper reports improvements over supervised fine-tuning.

1 Introduction

  • Structured-document translation must preserve markup that defines layout and interactive elements, making structural fidelity as important as content quality.Source: §1 · p. 1
  • Supervised fine-tuning optimizes token-level likelihood, leaving markup accuracy largely unaddressed and making complex XML or HTML documents difficult for existing approaches.Prior detag-and-project pipelines also risk error propagation between translation and alignment modules.Source: §1 · p. 1
  • FORMATRL first fine-tunes an LLM for document translation, then applies GRPO to optimize structure-aware rewards including TreeSim and Node-chrF.TreeSim measures XML-tree structural similarity; Node-chrF assesses translation quality at the XML-node level.Source: §1 · pp. 1–2
  • StrucAUC distinguishes minor errors from major structural failures while combining translation and structural quality into one score.Source: §1 · p. 2
  • FORMATRL reports average gains across four translation directions, including 3.69 XML-Match and 2.16 XML-BLEU, and a 0.93 StrucAUC score compared with a strong supervised fine-tuning baseline.The supplied result text also begins a Content-BLEU gain but does not provide its complete value.Source: §1 · p. 2

2 Related Work

The related-work discussion covers structured-text translation and reinforcement learning. It contrasts modular and end-to-end translation approaches with the challenge of handling complex markup, then frames reward design as central to applying RL to the task.

2 Related Work

  • The related-work section focuses on structured-text translation and reinforcement learning.Source: §2 · p. 2

2.1 Structured Text Translation

  • Traditional detag-and-project systems separate translation from markup handling and can propagate errors; newer LLM approaches work on sentence-level markup but struggle with complex XML structures.The paper frames these as limits of structured-text translation approaches.Source: §2.1 · p. 2

2.2 Reinforcement Learning

  • The paper presents RL generation as reward maximization and identifies reward design for matching source structure while maintaining translation quality as the task's challenge.It claims to be the first to apply RL algorithms to structured-document translation.Source: §2.2 · p. 2

3 Method

The method translates a source document represented as an XML tree while targeting both structural identity and node-level translation correspondence. It first fine-tunes an LLM, then applies GRPO with structure-aware rewards and group-relative optimization.

3.1 Task Definition

  • The task is to translate a structured source document while preserving its tree structure and translating the text associated with corresponding nodes.Documents are represented as XML trees whose nodes have tags, parent–child edges, and possibly textual segments.Source: §3.1 · pp. 2–3

3.2 Phase I: Supervised Fine-Tuning

  • SFT first adapts a pretrained LLM using parallel structured documents, with synthetic markup added to parallel plain-text data to address data scarcity.Generated document pairs are checked for isomorphic XML trees; invalid pairs are regenerated until valid or the retry limit is reached.Source: §3.2 · p. 3

3.3 Phase II: Format Reinforcement

  • Format reinforcement starts from the SFT checkpoint and optimizes the translation model to produce structurally correct, high-quality translations.The pipeline applies GRPO with the proposed format rewards after fine-tuning.Source: §3.3 · p. 3, Figure 2 · p. 3

3.3.1 Reward Functions

  • TreeSim rewards similarity between predicted and reference XML trees, while Node-chrF evaluates translation quality at the level of individual nodes.TreeSim uses tree edit distance; Node-chrF pairs nodes by parallel depth-first traversal and scores matching-tag text with chrF, assigning zero to mismatched or unpaired nodes.Source: §3.3.1 · pp. 3–4

3.3.2 Optimization

  • GRPO generates candidate translations and compares each reward with the group’s average to estimate relative advantages for policy updates.A KL-divergence regularizer limits deviation from the supervised fine-tuned model to avoid catastrophic forgetting.Source: §3.3.2 · p. 4

4 Evaluation Metrics: StrucAUC

StrucAUC is designed to avoid XML-BLEU’s document-level zero score for even minor structural mismatches. It combines node-level translation scores with tolerance for structural edits.

4 Evaluation Metrics: StrucAUC

  • XML-BLEU can assign a document-level zero score for a minor structural mismatch, motivating the more fine-grained StrucAUC metric.StrucAUC interpolates between Node-chrF and Optimal Node-chrF to evaluate translation quality at different error-tolerance levels.Source: §4 · p. 4
  • Optimal Node-chrF aligns nodes by representing each node with its subtree and minimizing total chrF distance across a one-to-one mapping.The method uses a cost matrix of subtree chrF distances and solves the assignment problem with the Hungarian algorithm.Source: §4 · p. 4
  • After alignment, Optimal Node-chrF scores node-level text while excluding children, and assigns zero to unmatched nodes.Source: §4 · p. 5
  • StrucAUC@5 integrates scores across edit thresholds from 0 to 5, allowing minor structural errors to reduce scores proportionally rather than cause complete failure.At each threshold, documents within the edit limit contribute their Optimal Node-chrF score; the rest contribute regular Node-chrF.Source: §4 · p. 5

5 Experimental Settings

The experiments use SAP software documentation in structured and inline-markup setups, evaluate translation and structural quality with six metrics, and compare prompting, SFT, and reinforcement-learning configurations.

5.1 Dataset

  • The SAP dataset provides Japanese–English and Chinese–English document pairs, with 100 training, 95 development, and 190 test pairs per language pair.Source and target documents have the same number of lines with one-to-one linear alignment.Source: §5.1 · p. 5
  • The inline-markup setup keeps translatable spans and inline tags, unlike the structured setup, which also preserves non-translatable nodes and metadata.The two setups therefore use different reference texts.Source: §5.1 · p. 5, Figure 3 · p. 5

5.2 Evaluation

  • The evaluation uses six metrics: Content-BLEU, XML-Validity, XML-Match, XML-BLEU, Content-COMET, and StrucAUC.They cover translation, structure, and combined evaluation; bootstrap resampling with 1,000 trials is used for empirical p-values.Source: §5.2 · pp. 5–6
  • Content-BLEU and Content-COMET assess document text without XML markup, while XML-Validity and XML-Match assess parsing and exact tree identity.XML-BLEU and StrucAUC combine translation and structural evaluation.Source: §5.2 · pp. 5–6

5.3.1 Prompting Baseline

  • The prompting baseline evaluates GPT-4o and Llama-3.1-8B-Instruct with zero to five full-document exemplars, reporting the setting with the highest XML-BLEU.Greedy decoding is used, with a maximum generation length of 2,000 tokens.Source: §5.3.1 · p. 6

5.3.2 Supervised Fine-Tuning

  • The SFT experiments use Llama-3.1-8B-Instruct and synthesize 900 structured document pairs per language from ALT, using SAP examples and target-vocabulary tags as guidance.Without domain-specific guidance, the LLM may default to generic tags, causing train–test mismatches.Source: §5.3.2 · p. 6

5.3.3 Format Reinforcement

  • The reinforcement phase trains with TreeSim reward for five epochs at a learning rate of 10^-6, using early stopping and a KL penalty coefficient of 0.01.It samples eight generations per document, with an effective batch size of 64 across eight H200 GPUs.Source: §5.3.3 · p. 6

5.3.4 Computational Efficiency

  • Training takes about 2.1 hours per SFT model and 1.3 hours per GRPO model; inference takes two minutes on the test set.The implementation uses DeepSpeed ZeRO-3 and bfloat16 mixed precision.Source: §5.3.4 · p. 6

6 Results and Analysis

FORMATRL with TreeSim improves structural and combined scores over baselines on structured documents while maintaining translation quality. Results vary with markup complexity, model, reward choice, and synthetic-data mix.

6.1 Main Results

  • FORMATRL with TreeSim outperforms prompting and SFT on nearly all structured-document metrics, averaging +3.69 XML-Match, +2.16 XML-BLEU, +0.22 Content-BLEU, and +0.93 StrucAUC over SFT.The main comparison is reported across four language pairs.Source: §6.1 · pp. 6–7, Table 1 · p. 7

6.2 Results on Documents with Inline Markup

  • On inline markup, FORMATRL with TreeSim improves all metrics, but the Prompt–FORMATRL XML-Match gap narrows from 10.92 to 4.74.The authors suggest LLMs handle simpler inline structures more effectively through in-context learning than more structured documents.Source: §6.2 · pp. 7–8, Table 2 · p. 7

6.3 Comparison with GPT-4 Models

  • FORMATRL is comparable to GPT-4o on most metrics and outperforms GPT-4.1-nano and GPT-4o-mini; in a 60-output review, it better matches source-document style than GPT-4o prompting.The figure compares FORMATRL with GPT-4.1-nano, GPT-4o-mini, and GPT-4o.Source: §6.3 · p. 8, Figure 4 · p. 8

6.4 Comparison with Parse-and-Assemble

  • For En→Ja, parse-and-assemble has comparable translation quality but lower XML-Match; its full-document context setup makes training 4.2× and inference 5.7× slower than standard SFT.Parse-and-assemble also struggles with inline tags whose positions vary across target-language syntax.Source: §6.4 · p. 8, Figure 5 · p. 8

6.5 Analysis: Reward Choice

  • Most rewards improve Content-BLEU, but XML-Validity hurts translation and structure; Content-BLEU and XML-Match rewards achieve the highest gain on their respective metrics.Combining TreeSim with XML-BLEU improves Content-BLEU more than TreeSim alone.Source: §6.5 · pp. 8–9, Figure 6 · p. 8

6.6 Analysis: Reward-Metric Alignment

  • TreeSim reward yields the highest TreeSim scores, while Node-chrF reward yields the best Node-chrF scores in most directions.This is the reported reward–metric alignment for the two proposed rewards.Source: §6.6 · p. 9, Table 3 · p. 9

6.7 Analysis: Synthetic Data Strategies

  • Synthetic data alone can bring translation quality close to real-data training but drive XML-Match below 20%; moderate mixtures can help, while excessive synthetic data can degrade performance.Synthetic data without explicit in-domain examples and markup guidance was more prone to structural errors; the authors attribute translation-quality limitations to ALT’s domain difference from software documentation.Source: §6.7 · p. 9, Figure 7 · p. 9

7 Conclusion

The conclusion presents FORMATRL’s structure-aware rewards and StrucAUC, reporting improved structural fidelity without compromising translation quality across inline and complex structured documents.

7 Conclusion

  • FORMATRL uses TreeSim and Node-chrF rewards and StrucAUC to improve structural fidelity without compromising translation quality across simple inline markup and complex structured documents.The conclusion states this result across both markup settings.Source: §7 · p. 9

8 Limitations

The evaluation and generalization claims are bounded by the tag set, document-level metrics, and the limited human evaluation.

Limited Tag Set

  • Synthetic markup used only tags from the development set, and extrapolation to unseen tags was not evaluated.The authors also did not explore placeholder-tag abstraction, which could aid generalization but would require pre- and post-editing and discard human-interpretable tag semantics.Source: §8 · p. 10

Applying Sentence-level Metrics to Documents

  • BLEU and COMET-22 were applied to XML-stripped documents, but neither metric was designed or trained for document-level data.The authors cite known shortcomings for this setting.Source: §8 · p. 10

Lack of Rigorous Human Evaluation

  • The human evaluation was simple; a rigorous assessment would need multiple annotators and structured-document error instructions that capture tag mismatches, nesting errors, and severity.The authors leave this evaluation to future work.Source: §8 · p. 10

A Synthetic Data Generation

The appendix describes how GPT-4o was prompted to synthesize aligned markup and illustrates the paper’s evaluation metrics on a toy example.

A Synthetic Data Generation

  • GPT-4o was prompted to insert hierarchically structured XML into both sides of existing translation pairs while keeping markup aligned.Few-shot prompts included full source–target document pairs; the five-shot examples total about 9,800 characters, or 5,070 Llama 3.1 tokens, with length varying by language pair.Source: A Synthetic Data Generation · p. 13

B Example: Metrics Calculation

  • A toy example illustrates XML-Validity, XML-Match, XML-BLEU, StrucAUC, Node-chrF, and Optimal Node-chrF.The example shows how structural errors can cause traversal misalignment and a low Node-chrF score.Source: B Example: Metrics Calculation · p. 13

C Full Results of Structured Document Setting

Across four language pairs, most reward functions improve on supervised fine-tuning, while XML-Validity is an exception.

C Full Results of Structured Document Setting

  • Most rewards improved most metrics over SFT; TreeSim usually gave the best XML-Match, while Node-chrF had the highest combined scores in three directions.The comparison covers En→Zh, Zh→En, En→Ja, and Ja→En.Source: C Full Results of Structured Document… · p. 13

D Full Results of Inline Markup Setting

The appendix reports inline-markup results and further analyses of synthetic-data ratios, GPT comparisons, reward choice, metric computation, and implementation scope.

D Full Results of Inline Markup Setting

E Synthetic Data Performance: All Translation Directions

F Comparison with GPT

  • FORMATRL was usually better than GPT-4.1-nano, comparable to GPT-4o-mini, and below GPT-4o, which led on structural and combined metrics.The comparison presents results across all translation directions.Source: F Comparison with GPT · p. 13, Figure 11 · p. 20

G Ablation: Reward Choice on Content-COMET and XML-Match

  • Content-BLEU reward did not yield the best Content-COMET gain, while Node-chrF gave the highest Content-COMET improvement among single rewards.TreeSim gave the highest XML-Match improvement among single rewards other than XML-Match itself; the analysis notes reward overfitting to the optimized metric.Source: G Ablation: Reward Choice on… · p. 13, Figure 12 · p. 21

H Details in StrucAUC

  • StrucAUC’s Hungarian matching is O(n^3) in text nodes, but matching costs were negligible versus inference for the dataset’s documents with n<20.Tree edit operations are linear in tree size under the restricted XML grammar.Source: H Details in StrucAUC · p. 13

I Discussion

  • The study found only the SAP software-documentation dataset; GRPO required an SFT model with basic structured-translation ability, and learning rate was crucial.The authors report 1e-6 as a good balance after trying 1e-5 to 1e-7; a KL regularizer was added to the loss to prevent degenerate single-metric optimization.Source: I Discussion · pp. 13–14
Loading 2512.05100v1…