Improving Translation Faithfulness of Large Language Models via Augmenting Instructions

1 Introduction

• SWIE and OVERMISS target instruction forgetting and translation unfaithfulness in instruction-tuned open-source LLMs.SWIE strengthens instruction representations, while OVERMISS constructs contrastive data focused on over-translation and miss-translation.Source: §1 · pp. 1–2
• SWIE improved BLOOMZ-3b BLEU from 0.19 to 0.51 across four WMT22 directions and from 0.20 to 0.58 across six zero-shot directions.It also reached 0.67 average BLEU on long-sentence test sets.Source: §1 · p. 2
• OVERMISS increased LLaMA-7b WMT22 BLEU from 0.69 to 3.12 and improved average COMET by 0.48 percentage points.The dataset uses contrastive negative samples targeting over-translation and miss-translation.Source: §1 · p. 2
• Combining SWIE and OVERMISS produced up to 0.56 BLEU improvement from English to German across three backbone models.The combination extends the gains obtained by the individual methods.Source: §1 · p. 2
• Internal attention analysis found that SWIE increased the attention ratio assigned to instructions relative to the baseline.This supports the paper’s hypothesis that SWIE mitigates instruction forgetting.Source: §1 · p. 2

2.1 Machine Translation based on LLMs

• Open-source LLMs offer a lower-cost route to adapting general models for machine translation through fine-tuning.Source: §2.1 · p. 2

2.2 Instruction Tuning

• Causal LMs risk forgetting task-defining instructions because those instructions occur early while attention favors nearby tokens during decoding.Source: §2.2 · p. 3
• Existing instruction-preservation methods often assume fixed instructions or other demanding task settings, unlike machine translation’s short direction instructions.Source: §2.2 · p. 3

2.3 Hallucinations in Language Models

• Translation unfaithfulness commonly appears as over-translation, miss-translation, or inadequate source attention.Source: §2.3 · p. 3
• Existing hallucination datasets are costly when human-created or low-quality when produced by model perturbation, motivating scalable fluent negative samples.Source: §2.3 · p. 3

3.1.1 Instruction Tuning Formalization

• Standard instruction tuning uses maximum likelihood over instruction, input, and target, but local attention increases the risk that instructions are forgotten during generation.Source: §3.1.1 · p. 4

3.1.2 Causal Language Model

• Causal decoder layers combine each position with prior contextual representations under a causal mask, producing the local-focus setting addressed by SWIE.Source: §3.1.2 · p. 4

3.2 SWIE: Segment-weighted Instruction Embedding

• SWIE assigns segment IDs, extracts an instruction representation, re-parameterizes it with an adapter, and fuses it into input and response representations using segment weights.Source: §3.2 · p. 4
• SWIE selects the middle three model layers for fusing instruction features with global hidden representations.Source: §3.2 · p. 4

3.3 OVERMISS: A Natural Hallucination Dataset

• OVERMISS prompts GPT-3.5 to generate over-translation and miss-translation errors as contrastive negative samples for faithfulness training.Source: §3.3 · p. 5
• Word-level alignment evaluation indicates that generated OVERMISS samples satisfy the negative-sample requirement while preserving most source meaning.Source: §3.3 · p. 5

4 Emprical Experiments

• Experiments use BLOOM and LLaMA backbones across De⇒En, En⇒De, En⇒Zh, and Zh⇒En translation directions.Source: §4 · p. 5

4.1.1 Alpaca

• Alpaca provides the 52K-item multi-task instruction-following dataset used to fine-tune baseline models.Source: §4.1.1 · p. 5

4.1.2 Parrot-hint

• Parrot-hint is the strong baseline, combining Alpaca, WMT17-20, and MQM instruction data into 200K examples.Source: §4.1.2 · p. 5

4.1.3 OVERMISS

• OVERMISS training uses Parrot-hint for basic ability and a two-stage curriculum for mixed instruction-following data with and without translation hints.Source: §4.1.3 · p. 5

4.2.1 WMT22 Test Sets

• Evaluation includes WMT22 news-translation test sets with 1984, 2037, 2037, and 1875 samples for De⇒En, En⇒De, En⇒Zh, and Zh⇒En.Source: §4.2.1 · p. 5

4.2.2 Flores-200 Dev-test

• Flores-200 dev-test expands evaluation to multilingual translation, with 1012 samples per translation direction.Source: §4.2.2 · p. 5

4.2.3 Automatic Evaluation

• BLEU measures lexical performance, while reference-based COMET measures semantic performance.Source: §4.2.3 · p. 5

4.3 Implement details

• Training freezes embedding layers and uses DeepSpeed stage 1 to reduce memory requirements and prevent over-fitting.Source: §4.3 · p. 5

Backbone comparison

• LLaMA performs well on En⇔De but often confuses the target language on En⇔Zh, whereas BLOOM performs better on En⇔Zh than En⇔De.These results indicate differences in the backbones’ basic language translation capacities.Source: §4.4 · pp. 5–6

OVERMISS

• On LLaMA-7b, OVERMISS improved BLEU by 1.02, 1.25, 3.12, and 0.69 points across four translation directions, with an average 0.48 percentage-point COMET improvement.On FLORES, OVERMISS increased BLEU by 0.46 for En ⇒De and 3.03 for Zh ⇒En.Source: §4.4 · p. 6

SWIE

• 0.19 to 0.51 BLEU scores: SWIE outperforms Parrot-hint on BLOOMZ-3b, with obvious gains in some settings and stable slight gains in others.Source: §4.4 · p. 6

OVERMISS + SWIE

• 0.05 to 0.56 BLEU scores: combining OVERMISS and SWIE further improves En⇒De translation across all backbones and helps BLOOMZ-7b-mt in three of four directions.The authors state that the methods are not orthogonal because both target faithfulness.Source: §4.4 · p. 6

4.5 Long Sentence Translation

• Adjacent groups of 3-5 WMT22 test sentences are concatenated to create the WMT22-concat long-text test set.Source: §4.5 · pp. 6–7
• Table 3 covers translation performance on WMT22 and FLORES test sets, with bold scores marking the best performance under comparable settings.Source: Table 3 · p. 7
• 0.6725 average BLEU points: SWIE improves BLOOMZ-3b on WMT22-concat long-text translation, including a 1.49-point gain for Chinese-to-English.The results suggest instruction augmenting is better suited to long-text scenarios than the original WMT22 setting.Source: §4.5 · p. 7, Table 4 · p. 7

4.6 Zero-shot Performance

• 0.20 to 0.58 BLEU scores: SWIE improves six selected zero-shot translation directions on WMT22 test sets.The selected directions include Uk⇒En, Fr⇒De, Cs⇔En, and Ru⇔En.Source: §4.6 · p. 7

4.7 The Impact of Inference Instruction

• Inference prompts range from a basic source-to-target translation instruction to detailed guides addressing output errors and over- or miss-translation.Source: §4.7 · pp. 7–8
• Table 5 reports zero-shot BLEU performance for BLOOMZ-3b, while Table 6 compares inference prompts.Source: Table 5 · p. 8, Table 6 · p. 8
• No-error hints do not improve OVERMISS-trained models, whereas no-over, no-miss, and no-over/miss hints improve performance further.Source: §4.7 · p. 8

4.8 Faithfulness Quantification

• Faithfulness is estimated by matching source and inference words, then averaging source and target coverage rates to reflect omission and redundancy.Source: §4.8 · p. 8
• Both SWIE and OVERMISS improve translation faithfulness scores on WMT22 test sets.The scores average source-word and hypothesis-word coverage rates derived from word alignment.Source: §4.8 · p. 8, Table 7 · p. 8

• The analysis divides each translation example into instruction, input, and response spans and accumulates attention scores for every span at each token.This provides the measurement basis for diagnosing attention distribution across the sequence.Source: §5 · p. 8
• Middle layers show substantially higher accumulated attention on input spans, while bottom and top layers distribute attention more uniformly.This localizes instruction-attention inadequacy primarily to the middle layers.Source: §5 · p. 8
• SWIE produces a lower instruction-to-input ending-position attention ratio, especially in middle layers, indicating relatively higher instruction attention than the baseline.The ratio compares attention at the instruction ending position with attention at the input ending position.Source: §5 · p. 8

• The proposed methods combine an additional instruction-focused model structure with a contrastive data-construction method for machine-translation faithfulness.SWIE strengthens instruction attention, whereas OVERMISS constructs contrastive examples targeting translation faithfulness.Source: §6 · p. 8
• SWIE and OVERMISS outperform strong baselines on BLEU and COMET, while SWIE shows larger gains in zero-shot and long-text translation settings.The paper also evaluates faithfulness with cross-lingual word alignment and visualizes SWIE-induced attention shifts.Source: §6 · pp. 8–9
• Figure 4 visualizes accumulative attention scores on postinstruction and post-input positions for BLOOMZ-3b fine-tuned with the Parrot-hint dataset.The figure provides the attention-distribution context for the visual analysis.Source: Figure 4 · p. 9
• Figure 5 compares models with and without SWIE using attention ratios between postinstruction and post-input positions on BLOOMZ-3b.This visual supports the analysis of SWIE-related attention shifts.Source: Figure 5 · p. 9
• Future work includes explainable segment-weight construction, extending the data method to other tasks, and reducing inference latency.These directions mark current methodological and deployment boundaries.Source: §6 · p. 9