PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study

Eleftherios Kalogeros, Athanasios Ntalakas, Manolis Gergatsoulis, Paschalis Nikolaou, Sotiria-Lito Alexaki

arXiv:2608.28776v1cs.CL

TL;DR

This study examines whether multilingual sentence embeddings can detect fine-grained errors in English–Greek translations. Using a human-reviewed contrastive dataset and category-level evaluation, it finds that COMETKiwi outperforms the embedding models overall, while their complementary error sensitivities support combined evaluation frameworks.

  • Problem

    It remains unclear whether multilingual sentence embeddings can distinguish correct translations from minimally modified errors that preserve most lexical content but alter meaning.

  • Method

    The study evaluates five multilingual embedding models and COMETKiwi on 1,850 human-reviewed English–Greek contrastive examples from FLORES+ using contrastive accuracy and score margins.

  • Results

    COMETKiwi achieved 94.49% overall contrastive accuracy, versus 89.30% for the best embedding model, while embeddings detected explicit factual and lexical changes more reliably than tense-aspect and pronoun-coreference errors.

  • Takeaways & Limitations

    Multilingual sentence embeddings provide useful semantic adequacy signals but are better suited to broader translation-evaluation frameworks than to standalone use, alongside complementary MT-specific quality estimation.

  • Takeaways & Limitations

    The contrastive examples are not fully independent because 1,850 examples derive from 1,212 unique FLORES+ sentence pairs, limiting the basis for inferential analyses.

Abstract

from arXiv · show

Multilingual sentence embeddings are increasingly used to estimate semantic similarity across languages, yet their sensitivity to fine-grained translation errors remains insufficiently understood. This study investigates whether general-purpose multilingual embedding models can distinguish correct English-Greek translations from minimally modified erroneous alternatives. A contrastive dataset was developed from FLORES+ sentence-aligned reference translations and reviewed by two translation experts. It contains 1,850 examples across ten core and five exploratory error categories, covering factual, lexical-semantic, grammatical, relational, referential, and discourse-level phenomena. Five multilingual sentence-embedding models (BGE-M3, Multilingual E5, Multilingual MPNet, LaBSE, and Jina Embeddings v3) were evaluated using cosine similarity between each English source sentence and its correct and erroneous Greek translations. A reference-free COMETKiwi model was also evaluated as an MT quality-estimation baseline. Performance was assessed through contrastive accuracy and score margins for category-specific sensitivity. BGE-M3 achieved the highest accuracy among embedding models at 89.30 percent, while COMETKiwi achieved 94.49 percent. Embedding models detected explicit factual and lexical changes more reliably than tense-and-aspect and pronoun-coreference errors. COMETKiwi improved performance on several difficult categories, including tense and aspect, pronoun and coreference, and semantic-role errors, but showed lower sensitivity to date-and-time errors and underperformed the embedding models on numbers. The results show complementary error-sensitivity profiles: multilingual sentence embeddings provide useful semantic adequacy signals but are better suited as components of broader translation-evaluation frameworks than as standalone metrics.

1 Introduction

The study tests whether multilingual sentence embeddings can detect minimally modified translation errors in English–Greek and whether they are best used alone or within broader evaluation frameworks.

  • Research motivation: The study examines whether multilingual sentence embeddings distinguish correct English–Greek translations from minimally modified erroneous alternatives.Greek provides a challenging setting because of rich morphology, grammatical gender, overt agreement, flexible word order, and discourse-dependent reference.
  • Dataset and evaluation: The contrastive dataset contains 1,850 English–Greek examples across 15 factual, lexical-semantic, grammatical, relational, referential, and discourse-level error categories.Each example pairs an English source with a correct Greek translation and a minimally modified erroneous alternative.
  • Dataset and evaluation: Five multilingual embedding models are evaluated with contrastive accuracy and similarity margins, alongside reference-free COMETKiwi quality estimation.Category-level analysis identifies shared and method-specific strengths and weaknesses.
  • Research questions: The introduction frames the central question as whether embeddings can serve as standalone translation-evaluation metrics or semantic components within broader frameworks.This concern follows from the possibility that high cosine similarity can conceal critical factual, grammatical, referential, or discourse-level errors.
  • Contribution: The work contributes a category-based empirical analysis of five models and a human-reviewed contrastive challenge set for English–Greek translation evaluation.It focuses on distinctions preserved or lost in embedding spaces rather than only aggregate model performance.

2 Preliminaries: Multilingual Sentence-Embedding Models

The study compares five multilingual sentence-embedding approaches that map semantically related sentences across languages into shared representations, while differing in training objectives and architecture.

  • Embedding foundations: Multilingual sentence embeddings represent sentences as dense vectors and map semantically equivalent sentences across languages into a shared vector space.Cosine similarity is used to measure proximity between representations.
  • Model families: BGE-M3 supports dense, sparse, and multi-vector representations and was developed for multilingual and cross-lingual retrieval across more than 100 languages.
  • Model families: Multilingual E5 is trained through large-scale contrastive learning for retrieval, classification, clustering, and cross-lingual text matching.
  • Model families: Multilingual MPNet is trained on parallel and paraphrase data for semantic similarity, clustering, and semantic search across more than 50 languages.It produces 768-dimensional sentence representations.
  • Model families: LaBSE targets language-independent sentence representations, whereas Jina Embeddings v3 uses a multi-task, adapter-based architecture with long-context and Matryoshka representations.Together, the five models provide complementary retrieval-, paraphrase-, alignment-, and multi-task-oriented approaches.

3 Related Work

Prior work supplies error taxonomies, contrastive challenge sets, multilingual resources, and MT-specific metrics, while this study applies contrastive category analysis directly to general-purpose sentence encoders.

  • Research landscape: Previous research spans translation-error taxonomies, contrastive challenge sets, analyses of specific phenomena, and multilingual evaluation resources.
  • Contrastive evaluation: ACES evaluates controlled correct–incorrect translation pairs across 68 phenomena by checking whether a metric ranks the correct translation higher.The present study adopts this contrastive principle for multilingual sentence encoders rather than complete MT metrics.
  • MT evaluation metrics: COMET predicts translation quality from source, candidate, and reference information, while reference-free COMETKiwi uses only the source and candidate.
  • Error phenomena: Prior analyses motivate categories such as negation, numbers, dates, named entities, terminology, omissions, additions, modality, and semantic roles because fluent translations can still alter important meaning.
  • Evaluation resources: FLORES+ provides professionally translated, sentence-aligned multilingual data used here to construct the English–Greek contrastive dataset.
  • Study positioning: This study differs by testing whether multilingual sentence embeddings preserve fine-grained distinctions important for translation accuracy through controlled, category-level contrastive evaluation.

4 Methodology

The study compares correct and controlled erroneous Greek translations against the same English source using multilingual similarity scores and COMETKiwi quality scores. Accuracy and score margins quantify ranking success and category-specific error sensitivity.

  • Each example pairs an English source with correct and erroneous Greek translations encoded for cross-lingual comparison.
  • A model margin is positive when the correct translation scores higher, near zero when sensitivity is limited, and negative when the error is preferred.
  • Contrastive accuracy measures how often the correct translation receives the preferred score.
  • Mean category margins compare sensitivity across different translation-error categories.
  • COMETKiwi applies the same contrastive ranking principle, replacing cosine similarity with model-specific quality scores.
  • Contrastive accuracy is directly comparable across methods, but raw margin magnitudes are not because the scoring scales differ.

5 Experimental Setup

The experiments evaluate five multilingual embedding models and a reference-free COMETKiwi baseline using specified software, encoding, preprocessing, and model configurations. The setup preserves each checkpoint’s pooling and truncation behavior while applying model-specific settings where required.

  • Five multilingual embedding models are evaluated alongside the reference-free COMETKiwi quality-estimation baseline.
  • The experiments use Sentence Transformers, Transformers, Tokenizers, and Hugging Face Hub configurations reported with checkpoints, parameter counts, and maximum input lengths.
  • Embeddings are generated on CUDA with batch size 16, checkpoint-provided pooling, L2 normalization, cosine similarity, and default truncation.
  • Multilingual E5 receives the query: prefix, Jina Embeddings v3 uses its text-matching adapter, and BGE-M3 uses its standard interface without an extra prompt.
  • Greek inputs are Unicode NFC-normalized and whitespace-trimmed, while English inputs are whitespace-trimmed without additional linguistic preprocessing.

6 Dataset Construction

The dataset combines FLORES+ English–Greek references with controlled, expert-reviewed translation errors across core and exploratory categories. Its 1,850 contrastive examples derive from 1,212 unique sentence pairs and include uneven exploratory-category sizes caused by data availability.

  • 6.1 Source Data: The source pool contains 1,212 aligned English–Greek FLORES+ sentence pairs used to construct 1,850 contrastive examples.
  • 6.1 Source Data: Each contrastive unit contains an English source, a Greek reference, and one erroneous Greek variant created by introducing an intended error.
  • 6.1 Source Data: FLORES+ references are treated as high-quality references rather than unique ideal translations, and 18 questioned references were retained unchanged for consistency.
  • 6.2 Error Categories: The inventory includes ten core categories with 150 examples each and five exploratory categories with 50–75 examples each.
  • 6.1 Source Data: Two translation experts created and approved variants for intended error type, semantic difference, fluency, and naturalness.
  • 6.3 Dataset Statistics: The dataset contains 1,500 core-category examples and 350 exploratory examples because only 50 suitable unit- or currency-bearing sentences were available.

7 Experimental Results

The sentence-embedding models varied substantially in overall and category-level ability to rank correct Greek translations above minimally modified errors. COMETKiwi performed better overall, but its category strengths complemented rather than uniformly replaced those of the embedding models.

  • Overall performance: 89.30% was BGE-M3’s highest overall contrastive accuracy, while Jina Embeddings v3 and LaBSE each achieved 87.73%.Multilingual MPNet had the largest raw mean margin at 0.0387, whereas Jina Embeddings v3 had the largest raw median margin at 0.0213.
  • Category-level accuracy: Above 95% accuracy was common for explicit factual and lexical changes, including numbers, named entities, hallucinations, and dates and times.BGE-M3 reached 99.33% for named entities and hallucinations and 100% for dates and times.
  • Category-level accuracy: Tense-and-aspect accuracy ranged from 47.33% to 71.33%, while pronoun-and-coreference accuracy ranged from 40.00% to 61.33%.Discourse-connective accuracy also varied sharply, from 48.00% for Multilingual E5 to 98.67% for LaBSE.
  • Contrastive margins: The smallest within-model margins consistently occurred for tense-and-aspect and pronoun–coreference errors.Tense-and-aspect margins ranged from 0.0005 for Multilingual E5 to 0.0056 for LaBSE; pronoun–coreference margins ranged from −0.0001 to 0.0020.
  • COMETKiwi comparison: 94.49% was COMETKiwi’s overall contrastive accuracy, exceeding all five sentence-embedding models across 1,850 examples.COMETKiwi correctly ranked 1,748 examples, with 102 incorrect rankings and no ties.
  • COMETKiwi comparison: COMETKiwi reached 82.67% for tense-and-aspect and 89.33% for pronoun-and-coreference errors, but scored 72.00% on dates and times and 89.33% on numbers.These differences support complementary error-sensitivity profiles between dedicated MT quality estimation and general-purpose multilingual embeddings.

8 Discussion

The models showed category-dependent sensitivity: embeddings handled salient factual and lexical changes better than fine-grained grammatical and referential errors, while COMETKiwi provided complementary strengths. Overall comparisons therefore favor category-level evaluation and combined systems over reliance on a single metric.

  • Error-category sensitivity: Explicit factual and lexical changes, including named entities, hallucinations, numbers, and dates, generally produced stronger embedding discrimination than localized grammatical and referential errors.These changes affected salient semantic content and tended to yield larger contrastive margins across several architectures.
  • Error-category sensitivity: 47.33–71.33% accuracy for tense-and-aspect and 40.00–61.33% for pronoun–coreference errors showed persistent embedding blind spots.Jina Embeddings v3 produced margins of 0.0051 and 0.0020 for these categories, respectively, while BGE-M3 reached 61.33% on pronoun and coreference.
  • Overall model behaviour: 89.30% overall contrastive accuracy made BGE-M3 the strongest embedding model, while Multilingual MPNet and Jina Embeddings v3 led raw mean and median margins, respectively.Raw margin magnitudes are not directly calibrated across embedding spaces, so they are most informative within models and error categories.
  • Comparison with COMETKiwi: 94.49% contrastive accuracy made COMETKiwi stronger overall than every embedding model and improved tense-and-aspect, pronoun–coreference, and semantic-role detection.It reached 82.67% for tense-and-aspect, 89.33% for pronoun and coreference, and 99.33% for semantic-role errors.
  • Comparison with COMETKiwi: 72.00% accuracy on date-and-time errors and lower number accuracy showed that COMETKiwi was not uniformly superior to the embeddings.All five embedding models exceeded 93% on date-and-time errors, and each outperformed COMETKiwi on numbers.
  • Limitations: The 1,850 examples depend partly on 1,212 unique FLORES+ sentence pairs, so future inferential analyses should cluster resampling by source pair.The dependence does not affect the descriptive comparisons reported in the study.
  • Limitations: FLORES+ mainly represents general-domain informational prose, limiting direct generalization to literary, poetic, conversational, or domain-specific translation.Further experiments are needed for stylistically more complex texts.
  • Limitations: The baseline comparison includes COMETKiwi but omits reference-based neural and simple lexical metrics, so COMETKiwi is not representative of all MT evaluation approaches.Broader metric comparisons should examine category-level agreement and complementary error sensitivity.

9 Conclusion and Future Work

The study evaluates five multilingual embedding models against COMETKiwi on controlled English–Greek translation errors and finds complementary category-specific strengths. Future work will test additional language pairs and combine model signals to assess whether joint features improve error detection.

  • Conclusion: BGE-M3 achieved the highest embedding contrastive accuracy at 89.30%, while Multilingual MPNet and Jina Embeddings v3 led raw mean and median margins.Jina Embeddings v3 matched LaBSE in overall accuracy.
  • Conclusion: 94.49% overall contrastive accuracy made COMETKiwi stronger overall, but its category-specific weaknesses confirmed complementary signals rather than uniform superiority.COMETKiwi improved difficult grammatical, referential, and semantic-role categories but was less reliable for dates and numbers.
  • Future work: Future work will extend the contrastive dataset to French–Greek and Italian–Greek using the same error categories and evaluation procedure.These experiments will test whether observed strengths and blind spots are language-specific or consistent across multilingual settings.
  • Future work: Future supervised experiments will combine embedding scores, contrastive margins, and MT-specific evaluation features to test joint error-detection performance.The aim is to evaluate whether combined signals improve detection overall and within individual error categories.
Loading 2608.28776v1…