PaperlayerView exact revision on arXiv

Source-linked AI summary

DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains

Xiying Zhao, Zhoufutu Wen, Zhixuan Chen, Jingzhe Ding, Jianpeng Jiao, Shuai Li, Xi Li, Danni Liang, Shengda Long, Qianqian Liu, Xianbo Wu, Hongwan Gao, Xiang Gao, Liang Hu, Jiashuo Liu, Mengyun Liu, Weiran Shi, Chenghao Yang, Qianyu Yang, Xuanliang Zhang, Ge Zhang, Wenhao Huang, Yuwen Tang

arXiv:2511.10984v2cs.CLcs.AI

TL;DR

Existing translation evaluation largely measures individual sentences, leaving discourse coherence and specialist terminology underassessed. DiscoX and Metric-S address this gap, but even the strongest LLM still trails professional translators.

  • Problem

    Existing translation benchmarks predominantly assess sentences, leaving discourse-level coherence, domain-intensive terminology, and expert stylistic standards insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    The paper introduces DiscoX, a 200-case, seven-domain Chinese–English benchmark, and Metric-S, a reference-free multi-agent evaluator of accuracy, fluency, and appropriateness.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Metric-S reaches 70.3% consistency with human judgments on DiscoX versus XCOMET-QE's 34.7%, while the strongest LLM still trails professional translators.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    DiscoX serves as a realistic, challenging stress test for professional translation.

    Source: §1 · p. 3

    Models struggle with DiscoX's long inputs under strict character limits, sometimes producing disordered content, summaries, or omissions.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

The evaluation of discourse-level translation in expert domains remains inadequate, despite its centrality to knowledge dissemination and cross-lingual scholarly communication. While these translations demand discourse-level coherence and strict terminological precision, current evaluation methods predominantly focus on segment-level accuracy and fluency. To address this limitation, we introduce DiscoX, a new benchmark for discourse-level and expert-level Chinese-English translation. It comprises 200 professionally-curated texts from 7 domains, with an average length exceeding 1700 tokens. To evaluate performance on DiscoX, we also develop Metric-S, a reference-free system that provides fine-grained automatic assessments across accuracy, fluency, and appropriateness. Metric-S demonstrates strong consistency with human judgments, significantly outperforming existing metrics. Our experiments reveal a remarkable performance gap: even the most advanced LLMs still trail human experts on these tasks. This finding validates the difficulty of DiscoX and underscores the challenges that remain in achieving professional-grade machine translation. The proposed benchmark and evaluation system provide a robust framework for more rigorous evaluation, facilitating future advancements in LLM-based translation.

1 Introduction

DiscoX targets a gap in translation evaluation: existing benchmarks focus on short segments rather than discourse coherence, specialized terminology, and expert style. The paper introduces a long-form Chinese–English benchmark and a reference-free evaluation system, then shows that current LLMs still lag human experts.

1 Introduction

  • Segment-focused benchmarks such as WMT[28], FLORES[9], and Redtrans Bench[12] miss discourse coherence, domain terminology, and expert style; failures can distort arguments, weaken legal force [4], or jeopardize safety.The risks arise in scientific articles, legal contracts, and technical manuals, respectively.Source: §1 · pp. 1–2
  • Even GPT-5-high lags professional translators on DiscoX, and no model balances accuracy, fluency, and appropriateness at a human level.The authors describe DiscoX as a realistic stress test for discourse-heavy and domain-intensive translation.Source: Figure 1 · p. 2, §1 · p. 3
  • DiscoX comprises 200 expert-curated Chinese–English cases across 7 domains, averaging 1712 tokens and built through 1,330 person-hours of work.Its rubrics capture domain-critical challenges, including context-dependent terminology.Source: §1 · p. 2
  • Metric-S uses multiple LLM agents for pre-checking, quality estimation, error deduplication and attribution, and severity weighting, reaching 70.3% human-judgment consistency versus XCOMET-QE’s 34.7%.The system evaluates accuracy, fluency, and appropriateness without references.Source: §1 · p. 2
  • The authors conduct an extensive multi-model study and use Metric-S’s explainability to reveal models’ strengths and limitations on challenging translation tasks.They also discuss system-development implications and propose recommendations for advancing translation evaluation.Source: §1 · p. 3

2 DiscoX

DiscoX is designed to evaluate coherent whole-text translation in specialized domains, where preserving logic, style, and terminology requires subject-matter expertise. Its construction combines expert curation and difficulty filtering to produce a long-text benchmark spanning seven fields and two translation directions.

2 DiscoX

  • DiscoX targets discourse-level translation of complete texts as coherent units and expert-level translation requiring accurate communication of specialized concepts and terminology.Its three-stage pipeline sources texts from two primary and seven secondary domains.Source: §2 · pp. 3–4

2.1 Data Construction

  • A 133-professional team built DiscoX through annotation, quality filtering, and expert selection, narrowing 665 candidate tasks to 200.Texts had to reflect professional scenarios, meet minimum length requirements, and support unambiguous rubrics; the initial pool averaged 9.38 rubrics per task. Both tested SOTA LLMs had to fail at least eight predefined rubrics for a task to advance, after which experts selected about 30% and refined texts and rubrics.Source: §2.1 · p. 4

2.2 DiscoX Composition

  • 200 tasks span Academic (121) and Non-Academic (79) across seven fields and en→zh/zh→en, with an average input length of 1712.17 tokens.The benchmark comprises tasks in two primary domains, subdivided into seven secondary domains; its texts are substantially longer than those in typical segment-level benchmarks.Source: §2.2 · p. 5, Table 2 · p. 5

3 Metric-S: Automatic Evaluation System for DiscoX

Metric-S is a multi-agent LLM evaluation system for discourse-level translation that assesses accuracy, fluency, and appropriateness, filters invalid translations, and resolves overlapping errors before severity-weighted scoring. Its final score combines the three dimensions, weighting accuracy at 60 points and fluency and appropriateness at 20 points each.

3 Metric-S: Automatic Evaluation System for DiscoX

  • Metric-S uses multi-agent LLM judges to score accuracy, fluency, and appropriateness, then deduplicates errors by root cause before severity-weighted scoring.The workflow first filters invalid outputs and computes a final score from the unique errors identified.Source: §3 · p. 5, Figure 4 · p. 6

3.1 Instruction Following Check

  • An instruction-following judge gives zero and excludes outputs that are not valid translations, preventing summaries or continuations from entering quality scoring.Discourse-level LLM outputs may default to text continuation or summarization rather than following translation instructions.Source: §3.1 · p. 5

3.2 Quality Estimation

  • Quality estimation checks meaning, facts, tone, omissions, and domain-term rubrics; discourse fluency and coherence; and cultural, stylistic, emotional, and literary fidelity.Accuracy rubrics specify key terms such as proper nouns and domain-specific terminology; fluency is judged as if read by a native speaker.Source: §3.2 · pp. 5–6

3.3 Error deduplication and Attribution

  • Hierarchical attribution prevents double-counting by retaining root causes: extremely critical accuracy errors override, rubric violations count as accuracy, and causal analysis resolves other overlaps.For example, when a lexical choice causes disfluency, Metric-S keeps the accuracy error and discards the fluency symptom.Source: §3.3 · p. 6

3.4 DiscoX Score Calculation

  • Metric-S sums accuracy, fluency, and appropriateness scores, weighted at 60, 20, and 20 points; severity deductions are 2, 5, 10, or 50 points.The deductions correspond respectively to minor, major, critical, and extremely critical errors.Source: §3.4 · p. 6

4 Experiments

DiscoX experiments show that leading LLMs still fall short of human experts in discourse-level translation, while Metric-S aligns more closely with human judgments than XCOMET-QE. Metric-S’s ablations also indicate that its multi-component design supports its consistency.

4.1 Main Results

  • The main evaluation covered 20 systems: 7 open-source LLMs, 11 closed-source LLMs, one domain-specific LLM, and one NMT system; Gemini-2.5-Pro served as Metric-S’s judge.The model list is provided in Appendix F.Source: §4.1 · p. 7
  • GPT-5-high scored 76.66 overall versus human experts’ 80.16; GPT-5 led in accuracy, Kimi-K2 in fluency and appropriateness, and Claude-4 variants were accurate but struggled with fluency.General-purpose LLMs significantly outperformed traditional MT systems, but remained below human standards.Source: Table 3 · p. 7, §4.1 · p. 7

4.2 Effectiveness of Metric-S Evaluation System

  • Metric-S was tested on 50 sampled cases each from DiscoX and WMT 2024, using expert scores normalized with metric scores to [0, 1] and pairwise ranking consistency, including system-level SPA.Segment-level ties counted as consistent when metric-score differences were below 0.05; ChrF was evaluated only on WMT 2024 because it requires references.Source: §4.2 · pp. 7–8
  • On DiscoX, Metric-S achieved 70.3% overall consistency versus XCOMET-QE’s 34.7%; system-level zh→en consistency was 80.0% versus 10.0%.ChrF was excluded from the DiscoX comparison because it requires a reference.Source: Table 4 · p. 8, §4.2 · p. 8
  • Metric-S’s full framework achieved 90% system-level consistency, falling to 80% without error de-duplication and 20% with a single LLM judge.The authors attribute Metric-S’s effectiveness and stability to the synergistic contribution of its core components.Source: §4.2 · p. 8

5 Analysis and Discussion

LLMs show uneven translation performance across directions and domains, and thinking-enhanced models can underperform non-thinking counterparts. General-purpose LLMs outperform traditional MT and domain-specific LLMs, which struggle with long inputs and accurate terminology.

5.1 LLM Better at Chinese-to-English than English-to-Chinese

  • LLMs score higher in zh→en than en→zh translation; DeepSeek-V3’s direction gap is 34.8 points, while Doubao-1.6-T’s is 7.2 points.The lower en→zh accuracy is attributed to less abundant high-quality Chinese corpora, English-centric model architectures, and Chinese linguistic complexity.Source: §5.1 · pp. 8–9, Figure 5 · p. 9

5.2 Stronger in Academic Papers, Weaker in Literature

  • Models score higher on academic than literary translation; GPT-5-high leads in both domains, while Kimi-K2 is particularly strong in literary translation.Academic texts’ structure and logic aid translation, while literary syntax, rhetoric, and cultural nuance challenge current LLMs.Source: §5.2 · p. 9, Figure 5 · p. 9

5.3 Performance Deficit of Thinking Models

  • The non-thinking Qwen-3-235B scores 59.66 vs. 49.97 for its thinking version, with a similar, smaller deficit for Claude-4.Thinking models can over-summarize, causing omissions, or add extraneous structure, compromising faithfulness.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 Limitations of Traditional MT and Domain-Specific LLMs

  • General-purpose LLMs significantly outperform traditional MT models and domain-specific LLMs, which struggle with long inputs and domain-specific terminology.Strict character limits can lead to disordered outputs, summaries, or omissions; these systems also produce omissions and mistranslations and struggle with modern internet neologisms.Source: §5.4 · p. 10

6 Related Work

Prior translation benchmarks have broadened language, domain, and task coverage but largely remain sentence- or paragraph-level. Traditional metrics struggle to assess discourse-wide qualities and rely on references, while single-LLM judges can have coverage, accuracy, hallucination, and bias problems.

6.1 Evolution of Machine Translation Benchmarks

  • Traditional benchmarks such as WMT [28] [29] and IWSLT emphasize sentence- or paragraph-level translation, with limited domain diversity and task complexity.WMT releases annual test sets across language pairs and domains; IWSLT focuses on speech and TED talks with relatively short, well-structured content.Source: §6.1 · p. 10
  • Despite broader coverage, FLORES-101 [9], TransBench [18], and Redtrans Bench [12] remain largely sentence- or paragraph-level, leaving discourse and specialized-domain translation underassessed.FLORES-101 covers 101 languages; TransBench has 17k tasks across 33 language pairs; Redtrans Bench has 2,858 zh→en and en→zh cases. WMT 2023 discourse-level literary translation [29] also exposed low traditional-metric accuracy and questionable result credibility.Source: §6.1 · p. 10

6.2 Evaluation Metrics on Translation Tasks

  • Traditional metrics capture local adequacy and fluency but poorly represent discourse coherence, consistency, and style, weakening correlation with human judgments and making reference-dependent scores unreliable.Without references, these metrics may degrade sharply or be unusable; no single reference captures the full range of valid discourse-level translations.Source: §6.2 · p. 10
  • LLM-as-Judge methods can evaluate without strict reference dependence, but single general-purpose judges may lack comprehensive coverage and accuracy and are vulnerable to hallucinations and biases.Some researchers fine-tune smaller, specialized evaluators on extensive datasets with domain-specific criteria and illustrative examples [33].Source: §6.2 · pp. 10–11

7 Conclusion … A.2.1 Effectiveness of the workflow of Metric-SSystem

DiscoX benchmarks discourse-level, expert-level LLM translation, and Metric-S provides reference-free evaluation aligned with human judgment. Experiments show that current LLMs still underperform expert translators, while the benchmark and system are being released to support further research.

7 Conclusion

  • DiscoX is presented as the first benchmark for discourse-level, expert-level LLM translation, alongside Metric-S, a reference-free evaluation system aligned with human judgment.The paper reports that current LLMs still underperform expert translators, struggling with global coherence, domain-specific terminology, and professional style; both resources are being released to the community.Source: §7 · p. 11

A.1 Multiple Runs on DiscoX

  • Across three independent trials per model, negligible performance differences indicate consistent outputs and reliable evaluation.The passage characterizes the outputs as high-quality and interprets the trial consistency as evidence of model robustness.Source: §A.1 · p. 16, Table 6 · p. 16

A.2 Ablation experiments on Metric-S

  • Metric-S ablations were designed to test the robustness and necessity of its components.Source: §A.2 · p. 16

A.2.1 Effectiveness of the workflow of Metric-SSystem

  • The workflow ablation tests scoring without error deduplication by directly applying the different judges’ results to the final score.Source: §A.2.1 · p. 16
  • A separate ablation merges prompts for all three dimensions and uses a single LLM to judge model performance on DiscoX.Source: §A.2.1 · p. 16
  • Dimension ablations compare using accuracy alone with using all three dimensions at equal weights.Source: §A.2.1 · p. 16
  • Metric-S achieves the best alignment with human judgments among all tested settings, supporting the integrity of its framework and the need for its individual components.Source: §A.2.1 · p. 16

A.2.2 Effective of Metric-S on Different LLM Judges … B Annotator Background

Supplementary evaluations test Metric-S across judge choices, self-preference, WMT translation, and fine-grained dimensions. The paper also documents the expertise behind DiscoX’s curation.

A.2.2 Effective of Metric-S on Different LLM Judges

  • Metric-S consistently outperformed XCOMET-QE across alternative judges, with Gemini-2.5-Pro performing best, while its robustness was not specific to Gemini.Absolute Metric-S scores varied moderately with judge choice, reflecting the underlying judge capability.Source: §A.2.2 · pp. 16–17, Table 8 · p. 17

A.3 Effectiveness of self-preference bias for LLM judge

  • Gemini-2.5-Pro showed no self-preference bias, whereas o3 favored its own outputs, ranking itself first despite experts placing it third.These findings support the selection of Gemini-2.5-Pro as the evaluation judge.Source: §A.3 · p. 17, Table 9 · p. 17

A.4 Performance of Metric-S on MWT 2024 Tasks

  • On WMT 2024 en→zh translation, reference-free Metric-S achieved human-agreement rates close to reference-based metrics XCOMET and ChrF.The comparison used traditional reference-based metrics because the WMT tasks include reference answers; the supplied passages do not report specific agreement values.Source: §A.4 · p. 17, Table 10 · p. 18
  • Metric-S reports error types and severity alongside scores, providing actionable feedback to help evaluated parties target improvements.The passage describes this transparency as helping parties understand their strengths and weaknesses.Source: §A.4 · p. 18

A.5 Accuarcy of LLM judge output in different dimensions

  • Accuracy exceeded 95% on all three dimensions across 250 judgments, with experts checking whether error types and severity were correct.The study examined 50 cases with five representative judge models.Source: §A.5 · p. 18, Table 11 · p. 18

B Annotator Background

  • A 133-member dual-expertise team curated DiscoX: 115 Vertical Domain Experts sourced texts and drafted rubrics, supported by 18 Linguistic Experts overseeing linguistic accuracy and evaluation criteria.The Linguistic Experts were certified MTI graduates with 5–16 years of experience and exercised overall project oversight.Source: B Annotator Background · p. 18

C Detailed Prompt and Rubrics · D Case Analysis

The appendix specifies a translation-only system instruction and a five-stage judge prompt covering instruction following, accuracy, fluency, appropriateness, and de-duplication.

C.1 System Prompt

  • The system instruction requires translation from source to target language and forbids output beyond the translation.The judge prompt comprises instruction-following, accuracy, fluency, appropriateness, and de-duplication stages.Source: §C.1 · p. 18, §C.2 · p. 19

C.2.1 Instruction Following Judge

  • The guideline treats empty output, untranslated same-language output, and translation omission over 50% as severe failures, but accepts a response that summarizes first if it then provides a complete translation.Source: §C.2.1 · p. 19

C.2.2 Accuracy Judge, Fluency Judge and Appropriateness Judge

  • Accuracy checks compare translations against contextual checkpoints, treating precise alternatives as correct and classifying mistranslation, omission, untranslated text, and additions by severity.The rubric distinguishes ordinary word-level errors, critical sentence-level distortions, and extremely critical untranslated source text or extensive omissions.Source: §C.2.2 · pp. 20–21
  • Fluency review covers naturalness, grammar, logic, and terminology consistency, while appropriateness review checks style, emotional tone, literary quality, and cultural adaptation.Each identified issue receives its own type and severity label; the example flags an overly colloquial rendering of academic prose.Source: §C.2.2 · pp. 21–23

C.2.3 De-duplication Judge

  • De-duplication prioritizes extremely critical accuracy errors, then assigns overlaps with checkpoints to checkpoints, while leaving non-duplicate disagreements uncorrected.The judge is instructed to identify repeated penalties and attribute them rather than reassess the underlying translation.Source: §C.2.3 · p. 24

C.3 Case of Rubrics

  • Rubrics share a structure but adapt to genre-specific terminology and constraints across oncology, legal clauses, and Chan Buddhist concepts.Examples specify standard oncology terms, legally precise clause renderings, and translations of culturally specific Buddhist expressions.Source: §C.3 · pp. 25–27

C.4 Case of De-duplication

  • Hierarchical De-duplications traces each root error to one dimension; in the example, an omitted question is assigned to accuracy and duplicate fluency/style penalties are removed.The system is designed to prevent one underlying error from being penalized across multiple dimensions.Source: §C.4 · p. 28

D.1 Case of Metric Fidelity Validation … F models evaluated on DiscoX

The case analyses report that Metric-S correlates more strongly with expert judgments than other metrics, while exposing variation across models, translation directions, and reasoning modes. The section also defines Metric-S severity levels and lists the evaluated systems.

D.1 Case of Metric Fidelity Validation

  • Metric-S correlates significantly more strongly with expert human judgments than other evaluation metrics.The section introduces illustrative validation cases but provides no comparative correlation values in these passages.Source: §D.1 · p. 29
  • The examples flag omissions, conflated concepts, unsupported additions, and a missed cultural pun in the translations.The Xinjiang example’s “red” pun is described as conveying both tomato color and popularity; the literal rendering misses the latter sense.Source: §D.1 · pp. 30–31

D.2.1 Case Analysis of LLM Performance Variations Across Different Dimension

  • Claude-4 performs well on Accuracy but poorly on Fluency, whereas DeepSeek-V3 performs well on Fluency but poorly on Accuracy.A separate example notes that a fluent science-news translation uses accurate terminology, while another has correct meanings but awkward flow and a mistranslation of “mastered.”Source: §D.2.1 · pp. 31–32

D.2.2 Asymmetry in Translation Directionality: An English-to-Chinese Case Study

  • English-to-Chinese translations perform considerably worse than Chinese-to-English ones; even GPT-5-high shows a quality gap from professional human translations.The illustrated humanities case flags imprecision in rendering a classical Han Feizi saying and somewhat awkward paragraph opening.Source: §D.2.2 · p. 33

D.2.3 Case Analysis Between Thinking LLM and Non-Thinking LLM

  • Non-thinking Score: 71 (Accuracy: 45, Fluency: 6, Appropriateness: 20), versus Thinking Score: 14 (Accuracy: 0, Fluency: 4, Appropriateness: 10).The section says thinking models generally underperform in non-thinking translation tasks and are more prone to omissions or summarized translations that lose information.Source: §D.2.3 · p. 34

E Severe Levels in Different evaluation dimensions of Metric-S

  • Accuracy errors range from word- or phrase-level mistakes to critical paragraph deviations and extremely critical discourse failures; Fluency errors are minor, while Appropriateness errors are major.Source: E Severe Levels in Different… · p. 34

F models evaluated on DiscoX

  • The DiscoX evaluation roster comprises 7 open-source, 11 closed-source, 1 domain-specific LLM, and 1 NMT system.The listed systems include Kimi-K2 and DeepSeek-V3 among open-source models, GPT-5-high and Claude-4-Sonnet among closed-source models, Youdao-14B, and Google Translate.Source: F models evaluated on DiscoX · p. 35

G Correlation Framework of Evaluating alignment of different metrics with human judgments. · H Detailed results of model performance on Chinese-to-English and English-to-Chinese Tasks

The framework evaluates metric–human alignment using pairwise consistency at system and segment levels, averaging across both translation directions; detailed model results by direction appear in Table 13.

G Correlation Framework of Evaluating alignment of different metrics with human judgments.

H Detailed results of model performance on Chinese-to-English and English-to-Chinese Tasks

I Detailed results of model performance on different domains · J Model performance when using detailed instruction prompts and employing Gemini-3-Pro as the judge · K Use of LLMs

These sections report domain-level performance breakdowns, describe an evaluation setting using detailed translation instructions and Gemini-3-Pro as judge, and clarify that LLMs were used only to polish writing.

J Model performance when using detailed instruction prompts and employing Gemini-3-Pro as the judge

  • With simple translation instructions, some models produce unprofessional translations that align with preferences, hindering accurate performance evaluation.The authors respond by releasing a dataset version with detailed translation instructions.Source: J Model performance when using… · p. 35

I Detailed results of model performance on different domains

  • Table 14 breaks down model rankings and scores across seven sub-domains, categorized as academic and non-academic texts.The authors identify it as the detailed performance results for primary and secondary domains.Source: I Detailed results of model… · p. 35, Table 14 · p. 37

J Model performance when using detailed instruction prompts and employing Gemini-3-Pro as the judge

K Use of LLMs

  • LLMs were used only to polish writing by refining grammar, sentence fluency, clarity, and readability, without changing meaning or adding content.Source: K Use of LLMs · p. 38
Loading 2511.10984v2…