PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors

Jiahui Liang, Lifeng Han

arXiv:2607.00848v3cs.CL

TL;DR

Metaphor translation is challenging for MT and NLP, and existing analysis is coarse-grained and does not address error severity. MetaHOPE adapts HOPE into a metaphor-oriented annotation framework with five error categories and a five-level severity scale. The pilot reports Pearson correlations of [0.536, 0.726, 0.333] and exact agreement of [76.9%, 70.8%, 61.5%].

  • Problem

    Metaphor translation is challenging for MT and NLP, and existing analysis is coarse-grained and does not address error severity.

  • Method

    MetaHOPE adapts HOPE into a metaphor-oriented annotation framework with five error categories and a five-level severity scale.

  • Results

    The pilot reports Pearson correlations of [0.536, 0.726, 0.333] and exact agreement of [76.9%, 70.8%, 61.5%].

  • Takeaways & Limitations

    MetaHOPE provides a severity-aware way to investigate metaphor translation errors across MT systems and translation directions.

  • Takeaways & Limitations

    The framework’s motivation is bounded by the coarse-grained nature of prior error categories, which do not address severity.

Abstract

from arXiv · show

In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present challenges for machine translation (MT) and natural language understanding and processing (NLU, NLP), because it presents the features of semantic complexity, contextual dependency, and cultural embeddings that can lead to ambiguity issues for NLP models. To investigate how state-of-the-art NLP models perform on translating metaphors, we select three representative systems, i.e., GoogleMT, GPT5.4, and Hunyuan-7b as Neural MT (NMT) models and LLMs. We used two human-annotated metaphor corpora, including VUAMC and PSUCMC for English-to-Chinese and Chinese-to-English translation purposes. The original corpora we used are monolingual, where we carried out error annotation using the MetaHOPE framework, and also produced the human post-edited gold reference for bilingual use as a new resource. We believe the MetaHOPE evaluation framework for metaphor translation annotation, the parallel corpora resources, and the error analysis on SOTA automatic translation models can be useful and shed some light for the field of metaphor translation study. We share our resources publicly at github.com/Jiahui84/MetaHOPE

1 Introduction

Metaphor translation remains difficult because metaphors depend on semantic complexity, context, and cultural knowledge, while general MT improvements do not necessarily transfer to metaphor translation. MetaHOPE addresses limited fine-grained, severity-aware error analysis by adapting HOPE for systematic metaphor-translation evaluation.

  • Metaphor translation remains challenging for MT and NLP because interpretation requires contextual awareness, sociocultural knowledge, and conceptual reasoning.
  • Overly literal translation, especially for idioms and collocations, often fails to capture intended figurative meaning.
  • Existing work emphasizes strategies or broad quality dimensions, while fine-grained error analysis remains limited and prior categories are coarse-grained and do not address severity.
  • MetaHOPE adapts HOPE into a metaphor-oriented framework with five error categories and a five-level severity scale for systematic error identification and assessment.

2 Background and Related Work

Prior metaphor-translation research has emphasized strategies, preservation, and conceptual mapping, while MT evaluation has often relied on sentence-level quality assessment. MetaHOPE addresses this gap with context-aware, segment-level, metaphor-sensitive error analysis for MT and LLM outputs.

  • Metaphor-translation research has examined substitution, preservation, deletion, paraphrase, and conceptual mapping across linguistic and cultural contexts.
  • Recent MT and LLM studies have explored metaphor or idiom translation, but often focus on strategies, identification, or conceptual labeling rather than systematic translation-quality evaluation.
  • Sentence-level MT evaluations can overlook discourse-dependent problems because translations acceptable in isolation may become inaccurate in broader context.
  • MetaHOPE translates documents with context first, then extracts segments for annotation while preserving contextual awareness.
  • The study evaluates English-Chinese bidirectional metaphor translation and contrasts its segment-level framework with prior MRW-level analyses.

3 MetaHOPE Methodology

MetaHOPE adapts the HOPE framework into a five-category, severity-aware methodology for context-aware metaphor translation evaluation. Its workflow combines document-level MT, alignment, pilot refinement, human annotation, post-editing, scoring, and qualitative analysis.

  • MetaHOPE manually aligns segments and metaphor-related words, conducts pilot studies to refine annotation policies, and performs large-scale annotation of system outputs.
  • The workflow produces post-edited human references, MetaHOPE score tables, and qualitative analyses of metaphor-related error types and system behavior.
  • The adapted taxonomy retains five metaphor-focused categories: Impact, Required Adaptation is Missing, Mistranslation, Style, and Proof-reading error.
  • These categories cover literalness and structural shifts, missing cultural or idiomatic adaptation, meaning mismatch, loss of metaphorical effect, and awkward expression.
  • The method keeps five severity levels—minor, medium, major, severe, and critical—instead of the original exponential score range.

4 Experimental Evaluation

The evaluation uses extracted and aligned metaphor data with independent human annotation and agreement assessment. The methodology includes annotator training backgrounds, pilot development, and segment- and token-level reliability measures.

  • Metaphor-related words are extracted by part of speech, segmented, and aligned from source to target language before annotation.
  • Annotations are conducted independently using an instruction manual by linguistically and translation-trained annotators.
  • The pilot study uses six annotators to support development of the annotation process.
  • Inter-annotator agreement is evaluated at the segment level by summing IMP, RAM, MIS, STL, and PRF into SEGS.

11. From these two tables, the strongest agreement

The evaluation finds that metaphor-related errors constitute a substantial share of translation errors, while system behavior and annotator agreement differ across models. GPT-5.4 yields the highest agreement, whereas Hunyuan produces freer but more variable outputs.

  • Error statistics: Metaphor-caused errors account for 91.7% to 93.8% of errors in GoogleMT and GPT-5.4, and 61.8% in Hunyuan-LLM-7B.
  • Qualitative analysis: GoogleMT and GPT5.4 are more fact-tracking, whereas Hunyuan-llm-7b is more flexible and can produce native-sounding translations alongside additions, reductions, or hallucinations.
  • Qualitative analysis: Examples show Hunyuan can improve native phrasing or localization but may add unsupported information, omit source meaning, or lose financially relevant distinctions.
  • Agreement analysis: GPT-5.4 has the highest metaphor-level agreement, while Hunyuan-7B has the lowest exact agreement, consistent with its freer reformulations.
  • Agreement analysis: The strongest meaningful error-type agreement appears for MIS and RAM, while PRF and STL exact agreement is inflated because annotators usually assigned zero.
  • Agreement analysis: Average agreement across three annotators is more reliable than one annotator, particularly for GoogleMT and Hunyuan.

5 Conclusions

MetaHOPE adapts HOPE into a severity-aware framework for evaluating metaphor translation in NMT and LLMs. A pilot across three systems and two translation directions identified systematic differences and showed that difficult metaphors can reduce annotation consistency.

  • MetaHOPE operationalises metaphor translation quality through five error categories and severity-aware scoring for empirical evaluation.The categories are Impact, Required Adaptation Missing, Mistranslation, Style, and Proofreading Error.
  • GoogleMT and GPT-5.4 tended to preserve source wording more conservatively, whereas Hunyuan-LLM-7B showed greater flexibility and localization but sometimes sacrificed factual consistency or produced hallucinations.
  • Harder-to-interpret metaphor translations reduced inter-annotator consistency, underscoring the need for clear guidelines and cognitively informed evaluation criteria.

A Research Highlights

The paper highlights a gap in evaluating figurative translation, addresses it with context-aware resources and annotation, and situates the work in metaphor-rich news discourse. Its pilot also contrasts conservative fact-tracking with flexible localization and hallucination.

  • MetaHOPE fills an evaluation gap by adapting HOPE to assess metaphor-specific translation severity beyond general MQM or BLEU/COMET scores.
  • The study translates full segments or documents before extracting sentences, avoiding artificial isolation from surrounding context.
  • Post-edited references derived from VUAMC and PSUCMC create a parallel resource for English–Chinese MT research.
  • The pilot illustrates a trade-off between GoogleMT and GPT-5.4’s rigid fact-tracking and Hunyuan-7B’s over-flexible localization or hallucination.
  • News is selected because metaphor-related expressions are frequent there and can shape how events are represented, interpreted, and culturally reframed.The paper notes that news contains the second-highest frequency of metaphor-related expressions among the compared registers.
  • VUAMC and PSUCMC provide comparable English and Chinese source corpora using MIPVU-based lexical metaphor annotation and strong reported agreement.VUAMC covers about 190,000 lexical units, while PSUCMC contains 30,012 words; both report Fleiss’ Kappa above 0.8.

D MetaHOPE Error Types with Examples (Phase-1)

The Phase-1 pilot defines a focused dataset and annotation scope for examining metaphor translation patterns. It uses contextual input, content-word analysis, normalized part-of-speech categories, and a manageable 200-sentence sample per direction.

  • Only metaphorically used nouns, verbs, adjectives, and adverbs are analysed, while grammatical function words are excluded.
  • Chinese- and English-specific part-of-speech labels are normalized into broader categories according to contextual meaning and syntactic function.
  • Translation models receive larger text segments rather than isolated sentences so annotation better reflects context-dependent metaphor interpretation and human translation practice.
  • The pilot uses 200 metaphor-containing sentences for each translation direction to balance analytical reliability with detailed manual analysis.The paper treats fewer than 200 sentences as insufficient for statistically reflecting MT quality and regards 200 as practically feasible for in-depth annotation.
  • The pilot corpus statistics report language, segment, sentence-length, metaphor-sentence, and metaphor-related-word distributions for the extracted datasets.

E.3 Aligning Metaphor Related Words (Source, MT.output)

The alignment procedure evaluates how metaphor-related meaning is functionally represented across languages rather than requiring lexical matching. Phase-1 examples distinguish error types such as agency shifts, missing adaptation, literal unintelligibility, attenuation, and unnatural phrasing.

  • The alignment task identifies how each metaphor-related word’s meaning is realised in translation through semantic-functional correspondence rather than strict lexical equivalence.This accommodates context-sensitive, culturally dependent reformulation across linguistic systems.
  • A passive translation of “The government cracked down on protesters” can suppress the agent and alter the distribution of agency, illustrating an Impact error.
  • Paraphrasing a metaphor instead of adapting it idiomatically can partially retain figurative meaning while omitting metaphorical and cultural adaptation.
  • Literal translation may produce an unintelligible expression and lose the intended meaning, while weakened metaphorical intensity produces stylistic attenuation.
  • Understandable but unnatural Chinese is treated as a fluency problem, and current guidelines do not penalize metaphorical paraphrase by default.
  • MetaHOPE annotators inspect prepared aligned data and target texts to determine how metaphor-related meaning is conveyed in each translation.

I Implications and Future Work

MetaHOPE bridges cognitive metaphor theory and empirical MT evaluation through fine-grained, metaphor-sensitive error analysis. The proof-of-concept study also documents an annotation workflow and identifies directions for broader validation and automation.

  • Implications: MetaHOPE enables fine-grained analysis of how metaphorical meaning is preserved, distorted, weakened, adapted, or lost during translation.It offers a middle ground between metaphor-translation research and broad sentence-level MT quality metrics.
  • Future Work: The study is a proof of concept focused on news-domain data, with future work planned on full-scale testing, guideline refinement, cross-domain generalizability, metaphor types, and annotation assistance.Proposed domains include literary, fictional, academic, political, and science-communication texts.
  • Translation Patterns: The framework documents diverse metaphor translation patterns, including direct correspondence, paraphrase, distributed realization, implicit realization, and partial reformulation.These examples cover English-to-Chinese and Chinese-to-English translations.
  • Resources: The project provides updated annotation materials, including task descriptions, procedures, decision rules, error categories, severity scales, pilot annotation, and annotation requirements.The resources are made available through the open-source project website.
  • Annotation Procedure: The annotation workflow reads full context, detects errors, assigns error types and severity scores, and produces a human gold-standard reference.Multiple error categories may be assigned, with severity scores of 2, 4, 6, 8, or 10.

J.2 Error Type Definitions and Examples (updated)

The updated MetaHOPE framework defines metaphor-specific error categories and illustrates them with bilingual examples. Pilot analyses also show that agreement varies by system and error type, with unresolved disagreement over omissions, categorization, and severity.

  • Error Type Definitions: MetaHOPE distinguishes Impact, Required Adaptation Missing, Mistranslation, and Style errors according to their effects on meaning, communicative force, target-language norms, and expression.The updated definitions are grounded in metaphor-translation perspectives and supported by bidirectional English-Chinese examples.
  • Impact Error: Impact errors preserve basic meaning but weaken communicative force, emphasis, emotional effect, evaluative stance, or rhetorical impact.Examples include translating “lashed out” as merely expressing dissatisfaction and shifting focus from an acting company to affected jobs.
  • Required Adaptation Missing: Required Adaptation Missing errors arise when a source metaphor needs a conventional, idiomatic, or cultural target-language adaptation that the translation omits.Examples contrast literal or neutral renderings with established expressions such as Chinese 势头 or English “a shot in the arm.”
  • Mistranslation: Mistranslation errors distort metaphorical meaning through incorrect interpretation, conceptual mapping, or contextual transfer, often producing literal readings of idioms.The examples include “on its last legs” rendered literally in Chinese and “穿小鞋” rendered as wearing small shoes.
  • Style Error: Style errors preserve referential meaning while changing register, vividness, voice, genre fit, or interpersonal function.Neutral paraphrase can remove figurative imagery, while a literary metaphor can be inappropriate in news writing.
  • Pilot Findings: Inter-annotator agreement patterns varied by system and category, while a pilot example exposed disagreement over error detection, categorization, and severity.The omission of policy-oriented framing by Hunyuan-7B remains difficult to classify as Impact, Mistranslation, or acceptable reformulation.

L Metric Related Works

The related-work discussion situates MetaHOPE alongside adaptations of HOPE to other domains and languages. An illustrative pilot case shows how annotators can assign sharply different labels and severity scores to the same metaphor translation.

  • Related Work: HOPE has been adapted to crisis translation in Italian and to Arabic translation through Ara-HOPE.These adaptations are cited as further reading for related metric work.
  • Illustrative Example: The pilot example concerns Chinese 做题, glossed as doing or solving practice questions, and Hunyuan-7B’s rendering as “prepared over three thousand practice problems.”The example compares this output with a literal gloss involving completing or solving practice questions.
  • Annotation Variation: Annotators assigned totals from 0 to 12 and interpreted the output as no error, a minor impact or semantic shift, or severe mistranslation with reduced metaphorical force.The table includes MIS, STL, and IMP penalties alongside different overall interpretations.
  • Implications: The example demonstrates disagreement in error detection, error categorization, and severity assessment, motivating refinement of the MetaHOPE annotation guidelines.Some annotators treated the issue as Impact, while others treated it as Mistranslation or another error type.
Loading 2607.00848v3…