Source-linked AI summary
DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation
Enze Zhang, Jiaying Wang, Mengxi Xiao, Jifei Liu, Ziyan Kuang, Rui Dong, Eric Dong, Sophia Ananiadou, Min Peng, Qianqian Xie
TL;DR
LLM effectiveness in web-novel translation remains unclear, and existing benchmarks miss important genre-specific traits. The paper introduces DITING, AgentEval, and MetricAlign, and reports results on evaluation metrics and fourteen translation models.
Problem
LLM effectiveness in web-novel translation remains largely unexplored, while existing overlap-based benchmarks miss narrative coherence and cultural intent.
Source: §1 · p. 1Method
DITING assesses six narrative and cultural dimensions using over 18K expert-annotated pairs, alongside AgentEval’s deliberative evaluation and MetricAlign’s 300 annotated pairs.
Source: §1 · p. 2Results
Chinese-trained LLMs outperform larger foreign counterparts, DeepSeek-V3 produces the most faithful and stylistically coherent translations, and AgentEval correlates strongest with expert annotations.
Source: §1 · p. 2Takeaways & Limitations
The work establishes a framework for evaluating LLM web-novel translation through narrative and cultural fidelity beyond conventional surface metrics.
Source: §4 · p. 8The meta-evaluation dataset is limited in size, and the framework focuses primarily on sentence-level analysis, overlooking document-level narrative coherence.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) have substantially advanced machine translation (MT), yet their effectiveness in translating web novels remains unclear. Existing benchmarks rely on surface-level metrics that fail to capture the distinctive traits of this genre. To address these gaps, we introduce DITING, the first comprehensive evaluation framework for web novel translation, assessing narrative and cultural fidelity across six dimensions: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety, supported by over 18K expert-annotated Chinese-English sentence pairs. We further propose AgentEval, a reasoning-driven multi-agent evaluation framework that simulates expert deliberation to assess translation quality beyond lexical overlap, achieving the highest correlation with human judgments among seven tested automatic metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation dataset of 300 sentence pairs annotated with error labels and scalar quality scores. Comprehensive evaluation of fourteen open, closed, and commercial models reveals that Chinese-trained LLMs surpass larger foreign counterparts, and that DeepSeek-V3 delivers the most faithful and stylistically coherent translations. Our work establishes a new paradigm for exploring LLM-based web novel translation and provides public resources to advance future research.
1 Introduction
Web-novel translation poses narrative, stylistic, and cultural challenges that remain underexplored for LLMs and inadequately captured by surface metrics. The introduction presents DITING, AgentEval, and MetricAlign, then reports results across fourteen translation models.
1 Introduction
- Web-novel translation remains underexplored despite its informal styles, character interactions, and culturally embedded expressions, prompting questions about LLMs’ ability to preserve narrative and cultural nuance.The introduction notes that translation still relies heavily on human labor, limiting scalability and accessibility.Source: §1 · pp. 1–2
- Existing benchmarks use overlap-based metrics that can miss narrative coherence, cultural intent, temporal consistency, referential consistency, and culturally sensitive content.The paper calls this broader competence narrative and cultural fidelity.Source: Figure 1 · p. 1, §1 · p. 2
- DITING evaluates six dimensions of narrative and cultural fidelity using over 18K expert-annotated Chinese-English web-novel sentence pairs.Its dimensions are idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety.Source: §1 · p. 2
- AgentEval simulates expert deliberation, while MetricAlign provides 300 expert-annotated pairs; AgentEval achieves the strongest correlation with expert annotations among the tested metrics.MetricAlign includes error labels and scalar quality scores, and the comparison covers seven representative automatic evaluation metrics.Source: §1 · p. 2
- Chinese-trained LLMs outperform larger foreign counterparts, DeepSeek-V3 produces the most faithful and stylistically coherent translations, and SOTA LLMs surpass commercial MT systems.The evaluation covers fourteen representative open-source, closed-source, and commercial translation models.Source: §1 · p. 2
2 Method
DITING defines six sentence-level tasks for evaluating web-novel translation and is supported by expert-curated Chinese–English data. MetricAlign supplies expert annotations for metric comparison, while AgentEval uses task-specific criteria and multi-agent deliberation.
2.1 DITING
- DITING introduces six evaluation tasks for web-novel translation, covering idioms, lexical ambiguity, terminology localization, tense, omitted pronouns, and cultural safety.The tasks target figurative meaning, context-dependent sense, cultural nuance, temporal coherence, referent recovery, and avoidance of harmful or culturally inappropriate output.Source: §2.1 · p. 2, §2.1.1 · p. 3
2.1.1 Task Definition
- Each DITING task is a sentence-level sequence-to-sequence problem with a task-specific function assessing preservation of a source–target phenomenon.A translation model maps a Chinese source sentence to an English target sentence, and the task index selects one of six evaluation functions.Source: §2.1.1 · pp. 2–3
- The idiom, ambiguity, and terminology tasks assess figurative intent, contextually intended sense, and preservation of meaning with cultural nuance.The terminology task distinguishes culturally localized meaning from literal translation.Source: §2.1.1 · p. 3
- The remaining tasks assess alignment of English tense with Chinese temporal cues, explicit recovery of omitted referents, and culturally safe output.Cultural Safety targets harmful, biased, or culturally inappropriate expressions.Source: §2.1.1 · pp. 3–4
2.1.2 Dataset Construction
- The DITING-CORPUS contains 18,745 expert-curated Chinese–English pairs across the six linguistic and cultural phenomena.The construction process segments and aligns chapter-level bilingual material into sentence pairs, with expert review and refinement of ambiguous or poorly expressed segments.Source: §2.1.2 · p. 4
2.2 MetricAlign
- MetricAlign comprises 300 Chinese–English sentence pairs, each source sentence translated by 25 LLMs, with three domain experts scoring outputs across six dimensions (2/1/0) and tagging targeted error types.Experts provided brief justifications for borderline cases.Source: §2.2 · p. 4
- Specific metrics achieved Agreement = 0.96 and κ = 0.94, while general metrics showed agreements of 0.90–0.91 and κ scores of 0.84–0.85.These figures report inter-annotator agreement for MetricAlign.Source: Table 4 · p. 5, §2.2 · p. 5
2.3 AgentEval
- AgentEval has two scoring agents assess each translation independently, then a judge checks consensus and may prompt further debate using shared memory.Task type selects the evaluation schema and exemplars; each rule set combines one specific metric with two general metrics.Source: §2.3 · pp. 5–6
3 Experiment
The experiments compare automatic translation metrics with expert judgments, test AgentEval across evaluator backbones, and evaluate 14 translation models on DITING. AgentEval aligns best with human judgments, while DeepSeek-V3 leads the translation-model benchmark.
3.1 Evaluation of Automatic Metrics
- AgentEvalDebate-R1 has the strongest correlation with human annotations across dimensions, outperforming the single-agent variant and seven automatic metrics.The authors report that none of the existing metrics align strongly with human judgments in this domain.Source: §3.1 · p. 6, Table 5 · p. 6
3.1.1 Results of AgentEval with Different Backbone Models
- DeepSeek-R1-based AgentEval performs best across agreement measures, and debate helps DeepSeek-V3 and GPT-4o more consistently than R1.Debate-R1 and single-agent DS-R1 have identical ICC and Agreement Rate scores; R1’s single-agent version is slightly better on general metrics, while debate scores higher on specific metrics.Source: §3.1.1 · pp. 6–7, Table 6 · p. 7
3.2 Evaluation of Translation Models
- The translation-model evaluation covers 14 proprietary, open-source, multilingual, MT-specific, and commercial systems, using context-aware prompts.Each prompt includes the preceding paragraph’s last sentence as context, with the target sentence as the translation query.Source: §3.2 · p. 7
3.2.1 Benchmark Performances
- 5.16 overall, DeepSeek-V3 leads the DITING benchmark, followed by GPT-4o at 5.09.Chinese-centric Qwen3-8B scores 3.96 versus 3.58 for LLaMA3-70B; Seed-X-PPO-7B improves by +0.65 over its instruction-tuned counterpart. Table 7 reports task-by-metric scores on a 0–2 scale.Source: §3.2.1 · p. 7, Table 7 · p. 8
- LLMs lead on idiom translation and lexical ambiguity, but zero-pronoun recovery and cultural safety remain harder; most models score at least 4.6 on tense consistency.DeepSeek-V3 leads terminology localization at 5.12, while the authors identify cross-cultural reasoning and implicit reference reconstruction as open frontiers.Source: §3.2.1 · p. 8
4 Conclusion
The paper introduces DITING and uses it with AgentEval to evaluate web novel translation. Across 14 models, Chinese-trained LLMs outperform larger foreign systems, and DeepSeek-V3 produces the most faithful and stylistically coherent translations.
4 Conclusion
- DITING evaluates web novel translation through narrative and cultural fidelity, using six dimensions and over 18K expert-annotated Chinese-English pairs.The framework is designed to capture stylistic and cultural challenges beyond conventional surface metrics.Source: §4 · p. 8
- Across 14 translation models, Chinese-trained LLMs outperform larger foreign systems, while DeepSeek-V3 yields the most faithful and stylistically coherent translations.Source: §4 · p. 8
Limitations
The authors identify limits in the meta-evaluation data, the framework’s sentence-level focus, and the current multi-agent coordination approach.
Limitations
- Source constraints limit the size of the expert-annotated meta-evaluation dataset.Source: Limitations · p. 9
- The framework focuses primarily on sentence-level analysis, overlooking document-level narrative coherence.The authors propose extending evaluation to document-level consistency and character development.Source: Limitations · p. 9
- The multi-agent framework has not yet been optimized for dynamic coordination or learning.Planned work includes a dedicated scoring model trained to internalize expert evaluation criteria.Source: Limitations · p. 9
Ethics Statement
The authors report following research ethics guidelines and note risks related to sensitive dataset content and possible misrepresentation of translation quality.
Ethics Statement
- The datasets are publicly available and contain no personally identifiable information; trained annotators worked under fair labor practices.Source: Ethics Statement · p. 9
- Sensitive dataset content and possible metric misrepresentation are risks to consider when interpreting or applying the results.The authors state that research artifacts are provided for research and educational purposes under the MIT license.Source: Ethics Statement · p. 9
A.1 Evaluation Metrics for Translation
Translation metrics range from surface-overlap and learned evaluators to document-level measures, but literary and web-novel translation expose limits in capturing style, meaning, and narrative coherence. LLM-as-a-Judge methods offer a scalable alternative, yet face reliability and coverage constraints.
A.1 Evaluation Metrics for Translation
- Existing translation metrics include BLEU and ChrF, NLI-based MENLI, trained COMET, BERT-based BLEURT and BERTScore, and document-level BLONDE and DiscoScore.Some have also been applied to literary texts, though their effectiveness there remains limited.Source: §A.1 · p. 11
- Literary and web-novel translation remains difficult to evaluate because overlap-based metrics miss stylistic expression, semantic variation, and discourse-level coherence.These shortcomings are especially pronounced with colloquial speech, internet slang, and long-range narrative dependencies.Source: §A.1 · p. 11
- LLM-based translation judges can be sensitive to prompts, hallucinate, and produce stylistic artifacts, while existing approaches remain limited in language coverage, evaluation scope, or stylistic nuance.Human-centered MQM offers detailed evaluation but is resource-intensive and difficult to scale.Source: §A.1 · p. 11
A.2 Benchmarks for Web Novels
The section describes existing Chinese–English web-novel benchmarks and the annotation practices used in DITING. It emphasizes expert input, structured scoring, and collaborative quality checks.
A.2 Benchmarks for Web Novels
- Existing BWB and GuoFeng web-novel datasets provide large bilingual corpora and document-level context, but provenance and evaluation remain concerns.Prior analysis suggests some segments may come from post-edited machine translation, while fine-grained web-novel phenomena remain rarely benchmarked.Source: §A.2 · p. 11
B.1 Annotation Guidelines
- Annotators score translations across six dimensions using a scoring sheet, with 0–2 ratings and short justifications or notes when needed.The guidelines aim to assess performance independently and minimize subjective bias.Source: §B.1 · p. 12
Annotation Procedure
- The annotation procedure compares model output with the source and reference translation, scores each metric, totals the scores, and checks the total for accuracy.The six-dimensional criteria are detailed in Tables 8 and 9.Source: Annotation Procedure · p. 12
B.2 Annotator Demography
- Three translation-studies annotators, including two professional translators and a student quality supervisor, contributed to DITING’s annotations and consistency checks.Weekly meetings and feedback-driven guideline revisions supported the collaborative workflow, which the paper credits with producing a linguistically faithful and contextually consistent benchmark.Source: §B.2 · p. 12
B.3 Annotation Example
- The paper presents representative cases to demonstrate how the annotation guidelines are applied.The examples are in Table 10.Source: §B.3 · p. 12
B.4 Annotation Process
- The annotation process is shown in Figure 3.The figure depicts the Label Studio interface used in the DITING annotation process.Source: §B.4 · p. 12
C.1 Evaluated Translation Models
The evaluated translation models span frontier and open-source LLMs as well as translation-specialized systems. The selection is intended to represent varied model types, scales, and specializations.
C.1 Evaluated Translation Models
- The model set includes GPT-4o, noted for multilingual and reasoning abilities, and DeepSeek-V3, optimized for Chinese and English tasks, coding, and mathematics, plus open-source DeepSeek-R1-70B, LLaMA3-70B, and Qwen3 variants.Source: §C.1 · pp. 12–13
- The cited selection includes Google Translate, a widely used commercial machine-translation service based on large-scale neural methods; IFLYTEK Translate, a Chinese-oriented translation system; and Seed-X-Instruct-7B, instruction-tuned for translation.The selection spans model types, scales, and specializations.Source: §C.1 · p. 13
C.2 Evaluation of Automatic Metrics
This section points readers to the configurations used to evaluate automatic metrics, a case study of framework evaluation, and the prompts and examples used in evaluation.
C.2 Evaluation of Automatic Metrics
- The automatic-metric experiment configurations are provided in Table 14.The passage identifies the table as detailing configurations for experiments verifying the effectiveness of previous metrics.Source: §C.2 · p. 13, Table 14 · p. 18
C.3 Case Study
- A case evaluated within the framework is presented in Table 11.The table shows relevant information about the case evaluation.Source: §C.3 · p. 13, Table 11 · p. 17
C.4 Prompts
- Tables 12 and 13 provide the evaluation prompts, alongside representative human-evaluation examples across six dimensions and two general metrics.The examples use 0–2 scores for task-specific accuracy, linguistic appropriateness, and information completeness.Source: §C.4 · p. 16, Table 10 · p. 16, Table 12 · p. 17