PaperlayerView exact revision on arXiv

Source-linked AI summary

TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs

Shenbin Qian, Yves Scherrer

arXiv:2609.11399v1cs.CL

TL;DR

LLM translation outputs frequently contain extra text, but the problem lacks dedicated benchmarks and systematic study. The paper analyzes translation noise, introduces TransClean, and evaluates extraction methods, finding that detection is nearly saturated while exact clean-translation extraction remains difficult. The benchmark exposes this gap while supporting future work on more robust and multilingual extraction.

  • Problem

    LLM translations often contain additional text beyond the intended translation, creating a systematic challenge for automatic evaluation and downstream integration.

  • Method

    The paper analyzes over 790,000 outputs, identifies recurring noise patterns, constructs the TransClean benchmark, and evaluates span-based and LLM-based extraction approaches.

  • Results

    Detection accuracy is nearly saturated for most methods, but the best extractor reaches 54.07% accuracy on synthetic data and 52.18% on curated data.

  • Takeaways & Limitations

    TransClean provides a systematic framework for evaluating and improving clean-translation extraction from noisy LLM outputs.

  • Takeaways & Limitations

    The benchmark's synthetic noise may not capture real-world diversity, and its English-centric construction may under-represent multilingual or language-specific noise.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for machine translation, yet their outputs often contain additional text beyond the translation itself, such as language labels, explanations or bilingual repetitions, which we term translation noise. Despite its prevalence, this problem lacks dedicated benchmarks and systematic study. We analyze over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identify 12 recurring noise patterns, which we group into formatting and content noise. Building on the observed patterns, we construct TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs, comprising 8,800 synthetically generated instances and 1,100 manually curated authentic instances. We evaluate two extraction approaches on the TransClean benchmark: 1) a span-based extraction method leveraging translation quality estimation models for span detection, and 2) an LLM-based extraction method that prompts an LLM to isolate the translation. Our benchmark and analysis provide the first systematic framework to evaluate and improve the cleanliness of LLM translation outputs.

1 Introduction

LLM translation outputs frequently include translation noise beyond the intended target translation, creating challenges for evaluation and downstream use. This work formalizes clean translation extraction, introduces TransClean, and benchmarks two extraction approaches.

  • Translation noise includes labels, explanations, source repetitions, and commentary beyond the intended target translation, distorting evaluation and downstream integration.The paper defines translation noise as any output content that is not part of the intended target translation.
  • Noise appears in 3% to 99% of outputs across evaluated LLMs, and prompting alone does not reliably guarantee clean translations.Noise prevalence and form vary substantially across models, despite prompts such as “Output the translation only.”
  • The paper studies clean translation extraction as recovering the span corresponding to the correct target-language translation from a noisy LLM output.
  • The study analyzes over 790,000 outputs from 12 LLMs across 22 language pairs, identifying 12 recurring noise patterns grouped into two categories.
  • TransClean contains 8,800 synthetically noised instances and 1,100 manually curated authentic noisy examples with silver clean translations.
  • Two benchmarked extractors use quality-estimation models for span detection or prompt an LLM to isolate the translation, with standardized evaluation metrics.

2 Noise in LLM Translation Outputs

The paper measures translation noise across diverse LLMs, prompts, and language pairs, then organizes recurring behaviors into content and formatting categories. Explanations are the most frequent pattern, while content noise generally dominates and is harder to handle than formatting noise.

  • 2.1 Generating Noisy Translations: The analysis uses 12 LLMs, 22 language pairs, and three prompt templates to examine noise prevalence and patterns.The source data includes 3,000 sentence pairs per language pair from four parallel corpora collections, totaling 66,000 instances.
  • 2.2 Noise Rate: A lightweight detector identifies explanatory text with regular expressions and wrong-language outputs with fastText language identification at a 60% confidence threshold.The detector provides a coarse estimate rather than a complete characterization of all noise types.
  • 2.2 Noise Rate: Prompt 0 produces the highest noisy-output proportion, while gemma-3-27b-it reaches noise rates up to 99.73%.Prompt 0 outputs are therefore used for subsequent noise analysis.
  • 2.3 Noise Analysis: Explanations account for approximately 33% of noisy outputs, with wrong-target-language generation contributing approximately 7%.
  • 2.3 Noise Analysis: Content-level noise constitutes most noisy translations and is generally more difficult to handle than formatting noise.

3 Benchmark Construction

TransClean combines synthetically generated and authentic noisy translations to benchmark clean translation extraction while preserving realistic noise patterns and silver clean labels.

  • TransClean contains 9,900 paired noisy and clean translations, including 8,800 synthetic instances and 1,100 authentic curated examples.The authentic subset is annotated with noise patterns and silver clean translations.
  • Synthetic data injects empirically observed noise patterns into reference translations treated as clean ground truth.The benchmark uses content, formatting, and combined noise categories.
  • Synthetic noise-pattern frequencies approximately match those observed in LLM outputs through empirical-frequency sampling.Rule-based generators create formatting artifacts, while GPT-5-mini generates content patterns and verbose preambles using few-shot prompting.
  • The synthetic dataset includes three noisy categories and a clean control split for testing whether extraction preserves clean translations.It contains 8,800 instances across four splits.
  • Manual validation inspects 100 synthetic instances across nine language pairs, while the authentic subset contains 1,100 real LLM outputs.The authentic examples are filtered from 792,000 outputs and curated as 50 instances per language pair.
  • Three LLMs annotate clean translations, with majority voting and gemma-4-31B-it resolving remaining disagreement cases.Manual inspection found gemma-4-31B-it correct on all 143 examined disagreement cases except 17 with multiple acceptable translations.

4 Translation Extraction

The paper evaluates span-based and LLM-based methods for detecting noise and extracting clean translations from noisy LLM outputs. Detection is nearly saturated, but exact clean-translation extraction remains difficult, with LLM-based methods more stable across datasets.

  • 4 Translation Extraction: The benchmark evaluates a span-based extractor using quality estimation and an LLM-based extractor that isolates the translation from noisy output.Both methods operate without reference translations during extraction, and the evaluation uses detection and extraction accuracy.
  • 4.4 Evaluation Results: Detection accuracy is nearly 100% for most methods, whereas extraction accuracy remains challenging across the synthetic and curated subsets.Table 4 reports both detection and extraction accuracy on the two benchmark subsets.
  • 4.4 Evaluation Results: 52.18% extraction accuracy on the curated subset and 54.07% on the synthetic subset are achieved by the best-performing Qwen extractor.These results indicate substantial room for improvement under strict exact-match evaluation.
  • 4.4 Evaluation Results: The span-based method drops sharply from synthetic to curated data, while LLM-based extraction is more stable across datasets and may generalize better to diverse noise patterns.Synthetic noise follows predefined patterns aligned with rule-based cleaning, whereas curated data contains authentic noise that may not match them.
  • 4.4 Evaluation Results: Qwen performs best across synthetic noise categories, while the clean split favors the span-based method, which preserves already clean translations with 100% accuracy.The clean split tests whether extraction methods modify translations that contain no noise.
  • 4.4 Evaluation Results: Combining content and formatting noise produces the lowest extraction accuracy because interactions between noise types increase difficulty for all methods.Formatting noise is the easiest noisy category to extract from.

5 Related Work

LLMs are increasingly used to generate synthetic multilingual data, extending established MT practices such as backtranslation for low-resource languages.

  • LLMs are increasingly used to generate synthetic data for NLP because of their language modeling and controllable generation capabilities.
  • In machine translation, synthetic data has long been used through backtranslation, particularly to improve performance for low-resource languages.

6 Conclusion

The paper introduces TransClean as a benchmark for systematic study of translation noise and reports that detection is easier than exact clean-translation extraction. It identifies substantial remaining room for more robust extraction methods.

  • 6 Conclusion: TransClean provides a benchmark for evaluating clean-translation extraction from noisy LLM outputs, combining synthetic and authentic noisy translations.The benchmark supports controlled evaluation and validation on realistic data.
  • 6 Conclusion: Detection is relatively straightforward, whereas extracting clean translations from noisy outputs remains challenging.
  • 6 Conclusion: Future work should develop extraction methods that better generalize across diverse noise patterns, languages, and models.

Limitations

The authors identify limitations in noise detection, taxonomy coverage, synthetic realism, language diversity, and exact-match evaluation. These constraints motivate improved detection methods, broader multilingual data, and complementary evaluation metrics.

  • Limitations: Rule-based noise-rate estimation may miss unrecognized patterns or produce false positives because it relies on predefined patterns, English keyword matching, and automatic language identification.The detector enables scalable analysis but is not fully reliable.
  • Limitations: The noise taxonomy may be incomplete because pattern discovery was assisted by an LLM and new models or prompts may generate different noise types.
  • Limitations: Synthetic noise may underrepresent real-world diversity, while the English-centric benchmark may under-represent multilingual or language-specific noise phenomena.
  • Limitations: Exact-match accuracy can penalize semantically correct paraphrases, motivating edit-based or semantic-similarity metrics as complementary measures.Aya's paraphrases of already clean translations illustrate the mismatch between exact matching and semantic correctness.

Ethical Considerations

The study uses publicly accessible datasets under stated licensing agreements and presumes that they contain no sensitive or personally identifiable information. The authors report that AI tools supported writing and coding, while responsibility for the paper remains with the authors.

  • Ethical Considerations: The study relies on publicly accessible datasets and states that their use follows the licensing agreements specified for the source repositories.
  • Ethical Considerations: The authors presume that the repositories contain no sensitive or personally identifiable information and therefore report no significant ethical risks from their use.
  • Ethical Considerations: AI tools were used only to support writing quality and coding, while the authors retain responsibility for the intellectual content, analyses, and conclusions.

A Appendix: Additional Tables for Data and Models

The appendix documents inference settings and a rule-based detector that uses regular expressions to identify explanatory, meta-linguistic, and formatting content in translation outputs.

  • Inference settings: Inference used default temperature and top_p values of 1, with most models run through vLLM on four NVIDIA GH200 GPUs without quantization.DeepSeek-V3.2-Exp and t5gemma-xl-xl-prefixlm-it instead used their respective API or HuggingFace Transformers.
  • Rule-based detection: Regular expressions detect explanatory or meta-linguistic phrases, labels, reasoning markers, markup, headings, list formatting, and long parenthetical content.The patterns include cues such as “translation:”, “explanation”, “analysis”, reasoning tags, XML-like tags, Markdown headings, numbered lists, and long parentheses.
  • Rule-based detection: Additional detector patterns target translation-related labels, literal or contextual explanations, conversational phrases, special formatting, and structured output markers.These patterns include source/target labels, phrases such as “means” or “refers to,” assistant-style closings, code-like reasoning tags, and Markdown or HTML-style formatting.
  • Rule-based detection: The detector also flags explicit reasoning markup, Markdown headings, HTML-like tags, numbered or bulleted lists, and long parenthetical spans.These cues are designed to identify content that may extend beyond the clean translation.

D.1 An Example of the Synthetic Subset

The synthetic-subset example shows a German source with an English translation and a French target reference, illustrating a wrong-language noise pattern.

  • Synthetic example: The example pairs the German source “Die Wasserqualität hat sich in den letzten Jahrzehnten deutlich verbessert.” with an English output rather than the French target.The English output states that water quality greatly improved over the past decades.

D.2 An Example of the Curated Subset

The curated-subset example illustrates bilingual-output noise and documents the benchmark’s curation, synthetic-data statistics, extraction prompts, and implementation settings.

  • Curated example: A curated English-to-Chinese example contains both an English translation label and the Chinese translation, making bilingual output the primary noise pattern.The record also lists language-prefix noise and includes multiple silver annotations differing in spacing.
  • Appendix tables: Table A.1 reports test-set sizes and sources for each language pair, while Table A.2 lists model names, architectures, sizes, and tuning variants.The supplied captions identify the scope of both appendix tables but do not provide their cell values.
  • Extraction implementation: LLM-based extraction used temperature 0 and top_p 1.0 on four NVIDIA GH200 GPUs, while span-based extraction used COMET-KIWI on one NVIDIA A100 GPU.Processing all 9,900 instances took approximately 25 minutes for the LLM method and about 70 minutes for the span-based method.
  • Noise statistics: Synthetic noise data include 4,400 instances across patterns such as explanations, verbose preambles, language prefixes, alternatives, bilingual output, and wrong language.The listed pattern counts also include off-topic content, code blocks, extra punctuation, special formatting, translation prefixes, and cultural notes.
  • Extraction prompts: The extraction prompt instructs an LLM to isolate only the clean translation while identifying the noise pattern when extra content is present.It explicitly covers labels, explanations, cultural notes, alternatives, bilingual output, formatting, wrong language, and off-topic content.
Loading 2609.11399v1…