PaperlayerView exact revision on arXiv

Source-linked AI summary

An Investigation of Translationese in the Generations of Multilingual Large Language Models

Maria Valentini, Téa Wright, Julisa Granados, Eliana Colunga, Katharina von der Wense

arXiv:2608.17399v1cs.CL

TL;DR

It remains unclear whether multilingual large language models generate non-English text natively or with translationese. This paper evaluates translationese markers across five languages and finds that MLLM outputs resemble translationese, especially outside English, while avoiding some obvious unnatural features.

  • Problem

    It remains unclear whether multilingual large language models generate non-English text natively or reflect translation from an internal dominant language.

  • Method

    The study evaluates established translationese indicators in MLLM generations across five languages using human-trained classifiers, feature analyses, and human annotations.

  • Results

    MLLMs generate text that resembles translationese, with effects consistently exacerbated in non-English languages, while more effectively avoiding unnatural or uncommon words.

  • Takeaways & Limitations

    MLLM-generated translationese is expressed more through distributional shifts in frequent features than through the obvious abnormal words models tend to avoid.

  • Takeaways & Limitations

    The human annotation study is small, limited to two languages, and uses annotators personally known to the authors rather than independently recruited participants.

Abstract

from arXiv · show

Text which has been translated from another language tends to carry with it evidence of translation$\unicode{x2014}$hence, it is often referred to as $\textit{translationese}$. Multilingual large language models (MLLMs) generate text in a variety of languages. However, it is still unclear if MLLMs' generations resemble internal translation (from English or, potentially, other languages) and, thus, result in translationese. Here, we ask the following research questions: (1) Does text generated by MLLMs resemble translationese? (2) How does translationese produced by MLLMs differ from translationese produced through direct translation? We leverage established indicators of translated text to evaluate text generated by state-of-the-art MLLMs in five languages, comparing to both non-translated and human-written baselines in order to isolate translationese from other kinds of interference. Through the use of high-accuracy classification models, analyses of variance on individual linguistic features, and the collection of human annotations in a subset of two languages (German and Spanish), we assess the translationese content of MLLM generations and examine the key features that distinguish MLLM-generated text from typical translation-related interference.

1 Introduction

The introduction motivates evaluating whether multilingual language-model outputs are linguistically native or reflect translation from a dominant internal language. It frames translationese as an externally observable way to investigate this question and assess naturalness.

  • Motivation: MLLMs increasingly generate non-English text, but their apparent multilingual proficiency does not establish linguistically native generation.The introduction notes abilities demonstrated by ChatGPT and multilingual design claims made for PaLM2.
  • Motivation: Naturalness matters for human-facing applications, alongside correctness, coherence, and relevance, because outputs should ideally mirror native-speaker language.Language-learning-material design is given as an example application.
  • Approach: The paper evaluates externally observable phenomena typical of translated text—translationese—to study MLLM generations.This approach avoids requiring direct access to the models’ internal generation processes.
  • Research questions: The experiments ask whether MLLM-generated text resembles translationese and how it differs from translationese produced through direct translation.The first experiment compares generated text across English, German, Spanish, and Greek with multiple reference conditions.

2 Related work

Prior work defines translationese as source-text fingerprints that persist in translation and has developed computational methods for detecting them. Research on multilingual large language models further links multilingual performance and generation to translation-related processes, including English-mediated representations and translationese arising from fine-tuning biases.

  • Translationese: Translationese is the fingerprint left by translation, reflecting adherence to source meaning while adapting to target-language grammar and potentially introducing source-language interference.These effects include general translation processes and source-language-specific marks in the target language.
  • Automatic Detection of Translationese: Computational studies automatically detect translationese by distinguishing translated from original text or training classifiers on established linguistic features.Reported approaches include support vector machines and features such as type–token indicators.
  • Multilinguality and Large Language Models: Prior work finds that MLLMs struggle with cultural nuance and with balancing universal knowledge against language-specific details, particularly for low-resource languages.These challenges are relevant to translationese because translationese is tied to linguistic fidelity.
  • Multilinguality and Large Language Models: Translating inputs into English can improve multilingual performance, while LLM translation exhibits translationese linked to supervised fine-tuning biases rather than inference-time processing alone.Another study finds that MLLMs route tokens through English representation space even when prompted in other languages.
  • Multilinguality and Large Language Models: English-mediated token representations become more prominent as pretraining diversity decreases.This finding concerns MLLMs prompted in languages other than English.

3 Human-written Data

The study builds controlled human-written and translated datasets across German, Spanish, Greek, English, and Pashto. These datasets balance native and translated text where possible and include conditions designed to separate translationese from LLM-specific effects.

  • German and Spanish: German and Spanish each contain 8000 samples, split evenly between native writing and English-translated text, with 6400/1600 train/test instances.Texts are chunked to approximately 2000 tokens and partitioned at sentence boundaries.
  • Control conditions: Google Translate conditions translate pre-translation non-native text to disentangle shared translationese effects from LLM-specific stylistic biases.Comparing human + MT with LLM + MT tests whether classifier signals persist under identical machine-translation processing.
  • Greek: Greek uses the same processing but contains 6000 samples with a 4800/1200 train/test split, reflecting its more limited resources.Greek is classified as resource class 3 on Microsoft’s 0–5 linguistic diversity index.
  • English comparison: An English comparison dataset matches the German proportions, containing 4000 original English samples and 4000 translated from ten classifier-training languages.This tests whether measured interference reflects translationese rather than language-independent LLM-generation interference.
  • Pashto: The Pashto experiment combines 500 native samples scraped from Wikipedia with 500 human-translated samples assembled from the OLDI Seed dataset.Pashto is classified as resource level 1, with very limited digital resources relative to the comparison languages.

4 Experiments

The experiments evaluate translationese in multilingual language-model generations across five languages using controlled human and machine-translated baselines. They combine surface-feature SVM classification, feature-level analyses, and native-speaker annotations to compare translationese across generation configurations.

  • Analysis methods: The analysis uses surface linguistic features to reduce topic bias, combines classification with ANOVAs, and examines individual feature coefficients for finer-grained interpretation.Feature coefficients indicate each normalized feature’s influence on the linear SVM score, while ANOVAs compare machine-generated configurations with translated and non-translated human text.
  • Translationese evaluation: A binary SVM trained separately for each language uses established translationese indicators to classify samples and measure each configuration’s Classified Positive proportion.The classifier uses Platt’s SMO algorithm and is designed to capture general translationese signals rather than artifacts of one machine-translation system.
  • Dataset construction: Prompts are matched to human-test content to minimize extraneous classification noise, while direct target-language prompts are translated and back-translated for review.German and Spanish native speakers additionally wrote task descriptions in their own words to avoid influencing outputs with translated prompt wording.
  • Human evaluation: Native speakers annotate 60 randomly sampled German and Spanish sentences individually as Translated or Non-Translated, with explanations requested for translated labels.The annotations provide a human-based comparison for the model-generated and translated configurations.

5 Results and Discussion

MLLM generations showed more translationese in German and Spanish than in English, while Greek and Pashto were more translationese-free. The results also indicate that MLLMs differ from translated text mainly through feature distributions and can avoid some translationese cues noticeable to humans, though the findings do not establish that outputs were translated.

  • Cross-lingual results: 60.2% of Gemini’s German outputs and 37.6% of Llama’s were classified as translated, exceeding English scores by 18–26%.The pattern suggests greater difficulty producing natural non-English text despite multilingual training.
  • Cross-lingual results: 45.7% was the average translated score across the two native Spanish LLM configurations, broadly resembling German results.The Llama-generated and machine-translated configuration unexpectedly outperformed its non-translated counterpart.
  • Cross-lingual results: Greek and Pashto generations were more translationese-free than Spanish or German despite having fewer data and computational resources.Possible explanations include less translated training data and greater word-order flexibility, while greater English similarity may increase interference in German and Spanish.
  • Linguistic features: −0.327 was the German feature weight for auch, making its higher frequency the strongest indicator that text had not been translated.Specific function-word frequencies were more deterministic than broader distribution- or ratio-based indicators.
  • Linguistic features: +0.1192 was the Spanish feature weight for el toward translated text, although Llama configurations averaged less than half as many articles as Gemini or human outputs.This abnormality was confirmed by Spanish ANOVA results.
  • Human annotation: None of the annotated German Gemini generations were flagged as translated by a native annotator, despite 60.2% being classifier-flagged; Llama annotations classified 37.5% as translated.Annotators mainly cited unusual words or word combinations rather than abnormal frequencies of common words, suggesting some human-noticeable cues were filtered out.

6 Conclusion

The study uses translation-marker detection and multilingual generation to assess linguistic naturalness and translation interference in MLLM text. It finds that MLLMs produce translationese, especially outside English, while avoiding more human-observable unnatural wording, and leaves its cause unresolved.

  • 6 Conclusion: The study applies state-of-the-art translation-marker detection to MLLM-generated text across diverse languages, measuring translation interference and apparent author fluency.The resulting dataset supports comparisons across different generation settings.
  • 6 Conclusion: MLLMs generate text resembling translationese, with effects consistently exacerbated in non-English languages.This establishes translationese as a recurring property of multilingual model generations rather than an isolated language-specific effect.
  • 6 Conclusion: MLLMs more effectively avoid human-observable features such as unnatural or uncommon words, likely because of human preference training.Individual linguistic features and native human annotations support this distinction.
  • 6 Conclusion: The cause of LLM-generated translationese remains unresolved between internal translation and artifacts of translated training data.Experiments using models with documented pretraining-data compositions could help distinguish whether these features are inherited.

7 Limitations · A Full Data Descriptions

The human annotation study is limited in size and participant recruitment, so its findings complement rather than replace automated metrics. The Pashto data also comes from Wikipedia, introducing potential source-specific limitations.

  • 7 Limitations: The human annotation study covers only 60 sentences per language across two languages.
  • 7 Limitations: Annotators were personally known to the authors rather than independently recruited.
  • 7 Limitations: The annotators were native speakers with university education.
  • 7 Limitations: The authors treat the human annotation results as complementary to automated metrics, not as a standalone finding.
  • 7 Limitations: Future work would benefit from a larger, independently recruited pool of annotators.
  • 7 Limitations: The native-speaker Pashto data is drawn from Wikipedia, and some articles may have originally been translated.

A.1 Training Set · A.2 Test Set

The paper reports the final distributions of both the multilingual training and test sets. Each distribution is organized by language and configuration, with Human* denoting a native speaker of the target language.

  • A.1 Training Set: Table 2 gives the final distribution of the training set.It reports sample counts for each configuration/language pairing.
  • A.1 Training Set: The training-set distribution is broken down by language.The table specifies the number of generated samples associated with each language.
  • A.1 Training Set: The training-set distribution is also broken down by configuration.This identifies sample counts for every configuration/language pairing.
  • A.1 Training Set · A.2 Test Set: In both distributions, Human* refers to a native speaker of the target language.The notation is defined in the captions for the training and test-set tables.
  • A.2 Test Set: Table 3 gives the final distribution of the test set.It reports sample counts for each configuration/language pairing.
  • A.2 Test Set: The test-set distribution is broken down by language.The table specifies the number of generated samples associated with each language.
  • A.2 Test Set: The test-set distribution is also broken down by configuration.This identifies sample counts for every configuration/language pairing.

B LLM Translation Results · C Europarl-UdS Experiments · C.1 Data and Classifier Specifications

Table 4 reports language-specific LLM translation results, measuring the percentage of samples classified as translated by an SVM. It distinguishes Gemini- and Llama-translated samples originally written by native human speakers.

  • B LLM Translation Results: Table 4 reports results separately for each language in the LLM translation settings.
  • B LLM Translation Results: The reported values are percentages of samples classified as translated by the SVM.
  • B LLM Translation Results: Gemini denotes samples written by native human speakers and translated by the Gemini LLM.
  • B LLM Translation Results: Llama denotes samples written by native human speakers and translated by the Llama LLM.
  • B LLM Translation Results: The table compares translation settings involving two LLMs, Gemini and Llama.
  • B LLM Translation Results: The evaluation uses an SVM classifier to identify samples as translated.

C.1.1 Training Set … C.3 LLM Translation Results

The merged sections describe the UdS experiment’s training and test distributions, classifier evaluation, language-specific results, and LLM translation settings. The supplied passages identify the datasets, figures, and classification measure used, but provide no underlying numerical results.

  • C.1.1 Training Set: The UdS training set is distributed across configuration/language pairings, with Human* denoting a native speaker of the target language.This distribution is summarized in Table 5.
  • C.1.2 Test Set: The UdS test set reports sample counts for each configuration/language pairing, using Human* for native target-language speakers.This distribution is summarized in Table 6.
  • C.1.3 Classifier Accuracy Scores: An SVM classifier’s accuracy is reported by language as percentages after training on Europarl-UdS data.Table 7 contains the language-specific accuracy results.
  • C.2 Results: The results section presents complete Europarl-UdS findings for English, German, and Spanish, with abbreviations referenced through Figure 2.Figures 9–11 provide the language-specific results.
  • C.2.1 English: Figure 9 gives the full English results on the Europarl-UdS data and directs readers to Figure 2 for abbreviation definitions.The supplied passage identifies the figure’s scope but not its numerical findings.
  • C.2.2 German: Figure 10 gives the full German results on the Europarl-UdS data and directs readers to Figure 2 for abbreviation definitions.The supplied passage identifies the figure’s scope but not its numerical findings.
  • C.2.3 Spanish: Figure 11 gives the full Spanish results on the Europarl-UdS data and directs readers to Figure 2 for abbreviation definitions.The supplied passage identifies the figure’s scope but not its numerical findings.
  • C.3 LLM Translation Results: The LLM translation results report the percentage of UdS samples classified as translated by the SVM for each language and translation setting.Gemini and Llama denote translations of native human-written samples produced by the corresponding LLM.

D Full Spanish Article ANOVA Results … D.4 Los

The Spanish ANOVA-results appendix presents configuration distributions for the articles el, la, las, and los. Each subsection references the same set of configuration abbreviations and corresponding article-specific figures.

  • D.1 El: The Spanish article el is examined across configurations including H, NH + T, NH + MT, NH + LT, G, G + MT, L, and L + MT.Figure 12 gives the distribution of el across these configurations.
  • D.1 El: Figure 12 specifically reports the distribution of el across the listed configurations.The figure directs readers to Figure 2 for complete dictionary abbreviations.
  • D.2 La: The Spanish article la is examined across configurations including H, NH + T, NH + MT, NH + LT, G, G + MT, L, and L + MT.Figure 13 gives the distribution of la across these configurations.
  • D.2 La: Figure 13 specifically reports the distribution of la across the listed configurations.The figure directs readers to Figure 2 for complete dictionary abbreviations.
  • D.3 Las: The Spanish article las is examined across configurations including H, NH + T, NH + MT, NH + LT, G, G + MT, L, and L + MT.Figure 14 gives the distribution of las across these configurations.
  • D.3 Las: Figure 14 specifically reports the distribution of las across the listed configurations.The figure directs readers to Figure 2 for complete dictionary abbreviations.
  • D.4 Los: The Spanish article los is examined across configurations including H, NH + T, NH + MT, NH + LT, G, G + MT, L, and L + MT.Figure 15 gives the distribution of los across these configurations.
  • D.4 Los: Figure 15 specifically reports the distribution of los across the listed configurations.The figure directs readers to Figure 2 for complete dictionary abbreviations.
Loading 2608.17399v1…