Source-linked AI summary
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, André F. T. Martins
TL;DR
Open LLMs have struggled to support multiple translation-related tasks without task-specific specialization. The paper addresses this gap with multilingual continued pretraining and instruction finetuning, producing TOWER models that outperform open alternatives across several tasks while competing with closed models.
Problem
Open LLMs have achieved competitive results across translation-related tasks mainly through single-task specialization, unlike closed LLMs with broader capabilities.
Method
The paper continues pretraining LLaMA-2 on multilingual monolingual and parallel data, then finetunes it on diverse translation-related instructions.
Results
TOWERINSTRUCT consistently outperforms open alternatives on multiple translation-related tasks and is competitive with closed models such as GPT-4.
Takeaways & Limitations
The released TOWER models, TOWERBLOCKS dataset, TOWEREVAL framework, and model generations support reproducible research on translation-related tasks.
Takeaways & Limitations
Grammatical error correction remains an area for improvement, with no model significantly outperforming the others on most evaluated languages; the authors hypothesize missing GEC data contributes to TOWERINSTRUCT’s average performance.
Abstract
from arXiv · showhide
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and parallel data, creating TowerBase, followed by finetuning on instructions relevant for translation processes, creating TowerInstruct. Our final model surpasses open alternatives on several tasks relevant to translation workflows and is competitive with general-purpose closed LLMs. To facilitate future research, we release the Tower models, our specialization dataset, an evaluation framework for LLMs focusing on the translation ecosystem, and a collection of model generations, including ours, on our benchmark.
1 Introduction
Open LLMs have lagged behind closed models on multiple translation-related tasks, often remaining competitive only after single-task specialization. The paper proposes an open multilingual recipe that combines continued pretraining, task-focused instruction data, and supervised finetuning.
- Motivation: Open LLMs have generally achieved competitive results on translation-related tasks only when specializing on a single task.Closed LLMs, unlike most open models, have demonstrated strong capabilities across multiple translation-related tasks.
- Approach: The recipe extends LLaMA-2 through continued pretraining on 20B tokens of multilingual monolingual and parallel data, creating TOWERBASE.Parallel data supplies an additional cross-lingual signal beyond the monolingual data used in prior work.
- Approach: TOWERBLOCKS specializes LLMs for translation-related tasks, and supervised finetuning on it produces the instruction-following TOWERINSTRUCT model.The dataset is designed for instructions relevant to translation processes.
- Results: TOWERINSTRUCT consistently achieves higher translation quality than open alternatives and is competitive with GPT-4 and GPT-3.5-turbo.It also outperforms open models on automatic post-edition, grammatical error correction, and named entity recognition.
- Resources: The authors release 7B and 13B TOWER models, TOWERBLOCKS, TOWEREVAL, and model generations to support reproduction and future research.TOWEREVAL is the evaluation framework used for the paper’s translation-related-task experiments.
2 TOWER: An Open Multilingual LLM for Translation-Related Tasks
TOWER combines multilingual continued pretraining with diverse instruction finetuning to support translation workflows beyond a single task. Its training uses multilingual text and parallel data, while TOWERBLOCKS broadens task, template, and domain coverage.
- TOWERBASE: LLaMA-2’s limited non-English pretraining motivates continued pretraining on a highly multilingual corpus for multilingual translation tasks.The paper identifies this exposure as limiting LLaMA-2’s multilingual potential.
- TOWERBASE: TOWERBASE extends LLaMA-2 with 20 billion tokens across 10 languages, mixing one-third parallel data with two-thirds monolingual data.The language mixture includes English, German, French, Dutch, Italian, Spanish, Portuguese, Korean, Russian, and Chinese.
- TOWERBASE: Monolingual data is sampled uniformly across languages and cleaned through deduplication, language identification, and KenLM perplexity filtering.The monolingual corpus is collected from the multilingual mC4 web-crawled corpus.
- TOWERBASE: Parallel data covers to-English and from-English directions and is filtered with Bicleaner and COMETKIWI-22 quality thresholds.The sources are public, and the paper gives detailed filtering thresholds in an appendix.
- TOWERBLOCKS: TOWERBLOCKS combines diverse translation-related datasets, question-answer reformulation, multiple zero- and few-shot templates, paraphrasing, dialog, and coding instructions.The dataset targets translation workflows before or after translation while also improving multilingual understanding and instruction-following.
- TOWERBLOCKS: TOWERBLOCKS prioritizes quality by using human-annotated records, excluding data from 2023 onward, and filtering questionable source-reference pairs.The paper constructs question-answer pairs from validation or older test sets and applies XCOMET-QE-ENSEMBLE filtering for reference-based tasks.
- TOWERINSTRUCT: TOWERINSTRUCT is obtained by finetuning TOWERBASE on TOWERBLOCKS with a chat-oriented dialog format and loss calculated only on answer tokens.The model is trained for four epochs with bfloat16 mixed precision and packed sequences.
3 Experiments
The experiments evaluate TOWERINSTRUCT across machine translation and translation-related tasks against open, closed, and dedicated baselines using multiple datasets and metrics. TOWERINSTRUCT 13B leads open translation systems, performs strongly on post-editing and named entity recognition, while grammatical error correction remains unresolved.
- Experimental setup: The evaluation covers machine translation on FLORES-200, WMT23, and TICO-19, plus automatic post-editing, named entity recognition, and grammatical error correction.APE uses NLLB-3.3B translations; NER uses MultiCoNER 2023; GEC uses English, Spanish, and German datasets.
- Experimental setup: Models are compared with LLaMA-2 70B, Mixtral-8x7B-Instruct, GPT-3.5-turbo, GPT-4, and task-specific NLLB-54B and ALMA-R baselines.Additional open alternatives are reported in the appendix, and generations use greedy decoding.
- Experimental setup: Evaluation combines COMET-22 and additional translation metrics with ER, ERRANT, and sequence F1, plus significance-based quality clusters and normalized Borda rankings.Significance testing uses a 95% confidence threshold; corpus-level ER and sequence F1 use bootstrap samples.
- Translation: TOWERINSTRUCT 13B is the highest-quality open translation system, often ranks first on FLORES-200, and approaches GPT-4 on WMT23 and TICO-19.It outperforms larger open models and dedicated NLLB-54B and ALMA-R across the reported comparisons.
- Translation: TOWERINSTRUCT 13B ranks first for most en→xx FLORES-200 directions, is near the top for nearly all xx→en pairs, and exceeds GPT-4 on pt→en and ru→en.The reported gains from continued pretraining are larger for en→xx directions.
- Translation-Related Tasks: TOWERINSTRUCT improves NLLB 3B translations through post-editing and outperforms open models and GPT-3.5-turbo on APE, although GPT-4 remains best.Its edits can convert oscillatory hallucinations into high-quality translations, while it edits about 30% of instances versus GPT-4’s almost 90%.
- Translation-Related Tasks: GEC remains difficult: no model significantly outperforms the others on most evaluated languages, and the authors hypothesize that TOWERBLOCKS lacks GEC data.GEC is held out from the training data and evaluated with five in-context examples.
- Translation-Related Tasks: TOWERINSTRUCT 13B surpasses GPT-4 by about 15 F1 points on multilingual NER, with similar improvements already visible in TOWERINSTRUCT 7B.The authors relate this strength to learning the token-level classification task from TOWERBLOCKS records.
4 Dissecting the training recipe
The ablations examine how continued pretraining, parallel data, and instruction finetuning affect translation and translation-related tasks. Gains are task-dependent: parallel data benefits translation, while task mixtures create transfer and interference effects.
- Training stages: Continued pretraining and supervised finetuning each improve translation performance at both model scales.TOWERBASE 7B and TOWERINSTRUCT 7B outperform LLaMA-2 13B, while TOWERINSTRUCT 7B outperforms TOWERBASE 13B.
- Task-specific effects: Parallel data and continued pretraining improve translation but do not improve automatic post-editing, where supervised finetuning remains beneficial.This contrast motivates exploring additional continued-pretraining signals for translation-related tasks.
- Continued pretraining data: Mixing monolingual and parallel data achieves the highest FLORES-200 translation quality among the continued-pretraining recipes.The mixed recipe outperforms both monolingual-only and parallel-only data.
- Continued pretraining data: Nearly one COMET-22 point of improvement on xx→en comes from the proposed recipe, compared with 0.1 points for monolingual-only data.The comparison concerns translation quality relative to base LLaMA-2.
- Data scaling: 85% of total performance gains appear by 5 billion tokens, while additional tokens continue improving translation quality with diminishing returns.At 2 billion tokens, mixing parallel and monolingual data provides more than half the improvement over the base model and surpasses monolingual-only training.
- Task interference: Adding translation-related tasks improves those tasks but decreases translation quality, whereas adding general-purpose instructions recovers translation quality.The authors hypothesize that the reduced task set encourages the model to split capacity across tasks.
5 Related Work
Prior work adapts open language models through multilingual training, domain specialization, supervised finetuning, or combinations of these approaches. Tower extends these directions by combining parallel-data continued pretraining with instruction tuning for multiple translation-related tasks.
- Specialization of LLMs: Earlier systems adapted open models to individual machine-translation tasks and achieved results competitive with closed models or dedicated systems.Xu et al. introduced a two-step LLaMA-2 approach using monolingual continued pretraining followed by translation finetuning.
- Tower’s approach: Tower combines parallel-data continued pretraining with instruction-following capabilities to support multiple translation-related tasks.This differs from approaches focused on adapting open models to a single task.
- Multilinguality in LLMs: Multilingual LLMs are built either from scratch with multilingual data or through continued pretraining on languages of interest, sometimes with vocabulary extension.The paper positions its approach within both adaptation strategies.
- Multilinguality in LLMs: Tower continues pretraining LLaMA-2 with parallel sentences, without vocabulary extension, after preliminary experiments with extension yielded negative results.The design builds on evidence that parallel data can be effective during pretraining.
- Specialization of LLMs: LLM specialization research includes training from scratch, domain-tailored continued pretraining, supervised finetuning, and combinations of the latter two.These approaches target subsets of closely related tasks or specific domains.
- Specialization of LLMs: Tower curates a supervised-finetuning dataset of high-quality and diverse instructions for translation-related tasks, drawing on instruction-tuning research.The approach prioritizes data quality and diversity.
6 Conclusion
The paper proposes a recipe that expands LLaMA-2’s multilingual capabilities and then specializes it with diverse instructions for multiple translation-related tasks. The resulting models outperform open alternatives across several tasks and are competitive with closed-source models, while the released resources support reproduction and exploration.
- Conclusion: Tower expands LLaMA-2 with highly multilingual continued pretraining and then finetunes it on high-quality, diverse instructions for translation-related tasks.The recipe targets multiple tasks rather than a single translation task.
- Conclusion: The final model consistently outperforms open alternatives on multiple translation-related tasks and is competitive with closed-source models such as GPT-4.This is the paper’s main reported performance conclusion.
- Resources: The release includes TowerBase and TowerInstruct models, TowerBlocks, TowerEval, benchmark generations, reproduction instructions, and interactive Zeno access.The models are available in 7B and 13B sizes.
- Terminology: In this paper, supervised finetuning is the terminology used for instruction tuning.The equivalence is stated explicitly in a footnote.
A Analysis of alternative decoding strategies
The decoding analysis compares beam search and minimum Bayes risk decoding with greedy decoding for TowerInstruct 13B. Both alternatives improve translation quality, with MBR decoding performing best and narrowing gaps to GPT-4 on two test sets.
- Decoding strategies: Beam search and MBR decoding are evaluated against greedy decoding for TowerInstruct 13B across the reported translation test sets.MBR uses 20 hypotheses and COMET-22 as its utility function.
- Results: Both decoding strategies consistently improve translation quality over greedy decoding, with MBR decoding achieving higher quality.The table highlights systems in the first quality cluster.
- Results: On WMT23 and TICO-19, alternative decoding strategies close the gap to GPT-4.On FLORES-200, TowerInstruct 13B appears isolated in the first quality cluster.
C Details of the continued pretraining dataset
The continued pretraining dataset applies language-specific quality thresholds to monolingual and parallel data and draws parallel sentences from multiple sources.
- Table 6 specifies language-specific perplexity floors and ceilings for monolingual-data filtering and thresholds for parallel-data filtering.The parallel-data thresholds use Bicleaner and CometKiwi-22, with to-English thresholds matching the corresponding from-English thresholds.
- Table 7 lists the parallel-data sources and the number of available language pairs.
D Details of TOWERBLOCKS
TOWERBLOCKS combines translation, evaluation, multilingual language-processing, conversational, and coding datasets, while TOWERINSTRUCT is finetuned with specified training settings and chat-format controls.
- Translation data: QT21 and ApeQuest include post-edited translation data, and MT-GenEval provides a gender translation benchmark used for general and context-aware translation.
- TOWERBLOCKS datasets: TOWERBLOCKS includes datasets for error span detection, translation ranking, terminology, multi-reference translation, named entity recognition, and paraphrase generation.These include WMT Metrics MQM18, WMT Metrics DAs, WMT21 Terminology, Tatoeba, MultiCoNER, and PAWS-X development or evaluation data.
- TOWERBLOCKS datasets: The dataset collection also contains filtered UltraChat and Glaive Code Assistant data for conversational and coding examples.Glaive Code Assistant covers questions and answers across a wide range of programming languages.
- Finetuning configuration: TOWERINSTRUCT finetuning uses bfloat16 mixed precision, packing, and the ChatML template with user and model control tokens.The hyperparameter configuration is given in Table 8, and an interaction example appears in Table 9.
F Translation full results
The appendix reports translation results across WMT23, TICO-19, and FLORES-200, using several metrics and language-pair breakdowns to compare model performance.
- WMT23 and TICO-19: Tables 10–13 report WMT23 and TICO-19 translation quality using XCOMET, COMETKIWI-22, BLEURT, and CHRF.Statistically significant models are grouped into quality clusters, with best overall and best open models specially marked.
- FLORES-200: Tables 14–17 report FLORES-200 translation quality by language pair using XCOMET, COMETKIWI-22, BLEURT, and CHRF.The tables identify statistically significant quality clusters and mark the best overall and best open models.
- Broader model comparisons: Tables 18–20 provide COMET-22 results for a wider variety of models on FLORES-200, WMT23, and TICO-19.These results are presented across the three evaluation settings named in the table captions.
G Translation-related tasks full results
The paper evaluates automatic post-editing, grammatical error correction, and named entity recognition across selected languages, with additional language-specific results in the appendix.
- Automatic post-editing: Automatic post-editing evaluates four language pairs in Table 3 because no few-shot examples were available for en→ru and zh→en.Table 21 nevertheless reports TOWERINSTRUCT, GPT-3.5-turbo, and GPT-4 results for all six language pairs, using NLLB translations as the task baseline.
- Named entity recognition: Named entity recognition covers English, German, French, Spanish, Italian, Portuguese, Russian, and Chinese.Language-specific NER F1 results are provided in Table 25.
- Grammatical error correction: Grammatical error correction is evaluated in English, German, and Spanish, with edit-rate results and ERRANT measurements.ERRANT results appear in Table 22, while language-specific edit rates appear in Table 24.
- Additional breakdowns: Additional language-pair results are reported for automatic post-editing, grammatical error correction, and named entity recognition in Tables 23–25.