PaperlayerView exact revision on arXiv

Source-linked AI summary

Discourse Dependency: A Continuous Criterion for Translation Difficulty

Ahrii Kim, Chanjun Park, Seong-heum Kim

arXiv:2609.04959v1cs.CL

TL;DR

Machine-translation benchmarks need a difficulty axis beyond saturated quality scores and sentence-level judgments. This paper introduces discourse dependency (DDP), a source-side measure of how far referential information lies within a document, and tests it against benchmark distributions and English-Korean post-editing. DDP is one-sidedly accurate in 99.2% of segments, while high-DDP cases expose human preferences that automatic metrics miss.

  • Problem

    Existing difficulty estimates rely on quality scores and segment-level properties, leaving continuous document-level contextual reach insufficiently measured.

  • Method

    The paper defines DDP from named-entity re-mentions and pronominal coreference, then applies DDP-stratified analysis to benchmark data and five context-injection strategies in English-Korean post-editing.

  • Results

    99.2% of segments have one-sided DDP error against gold coreference; at DDP ≥ 15, raters prefer human post-editing while XCOMET detects no difference.

  • Takeaways & Limitations

    DDP shifts evaluation toward referential reach and discourse-stratified analysis, exposing contextual difficulty and metric blind spots that aggregate scores conceal.

  • Takeaways & Limitations

    The main evaluation focuses on English-Korean, so generalization to other language pairs and source languages remains open.

Abstract

from arXiv · show

Recent calls for harder machine translation benchmarks have not clarified what difficulty should mean. We argue that one meaningful and currently unmeasured axis is referential reach, the distance a segment must look back into its document to resolve the entities and pronouns it contains. We formalize this as discourse dependency (DDP), a metric-free, source-side measure computed from named entity re-mentions and pronominal coreference. Validated against gold coreference, DDP errs one-sidedly in 99.2% of segments, so a high-DDP segment is certified to require long-range context. Applying DDP to WMT24++ and WMT25 shows that both are heavily skewed toward low-DDP segments, which domain labels do not distinguish. Building on DDP, we compare five context injection strategies in an English-Korean post-editing setup, varying context size and selection. As DDP grows, no strategy keeps pace with human post-editing. On segments with DDP >= 15 raters prefer human translations, while automatic metrics register no difference. As frontier systems saturate aggregate scores, DDP shifts evaluation from how well models score to how far they can reach.

1 Introduction

The paper argues that current machine-translation difficulty estimates rely too heavily on unreliable quality metrics and sentence-level views. It introduces discourse dependency (DDP), a source-side measure of contextual reach, and uses it to expose hidden variation in benchmarks and context use.

  • Current benchmark construction often estimates difficulty from automatic translation-quality scores, even though aggregate scores increasingly fail to distinguish frontier systems.This risks optimizing evaluation around what metrics measure rather than what genuinely challenges models.
  • Translation difficulty can be relational: a segment may require information located far earlier in its document.Prior document-level MT work identifies coreference, lexical cohesion, and tense consistency as phenomena requiring beyond-sentence information.
  • DDP measures the maximum distance between successive mentions of the same entity, providing a metric-free, source-side criterion independent of models and scoring functions.It also reframes domain labels as coarse proxies for discourse dependency rather than direct measures of contextual difficulty.
  • The paper evaluates DDP through five context-injection strategies and DDP-stratified English-Korean automatic post-editing experiments.The framework examines when models benefit from or fail to exploit document context.
  • DDP-based evaluation targets discourse progression and reveals within-domain variation that topical labels conceal.The proposed benchmark perspective focuses on inter-sentential reasoning rather than selecting only difficult individual instances.

2 Related Work

Related work has pursued harder MT evaluation through metric-based filtering, synthetic challenge construction, discourse-focused tests, and context-aware modeling. The paper positions DDP as a source-side, document-level alternative that can characterize difficulty before translation and organize evaluation by contextual reach.

  • Recent harder-benchmark efforts select difficult examples or generate challenging data, but both commonly rely on automatic quality scores with known high-end limitations.Metric-derived estimates require system outputs and therefore can filter existing sets but cannot guide test-set design beforehand.
  • DDP derives difficulty from a source-side document property rather than model outputs, enabling benchmark characterization before translation exists.This distinguishes it from metric-based filtering and links evaluation to discourse dependency directly.
  • Sentence-level evaluation can obscure discourse-sensitive errors, motivating expanded human context and targeted tests for pronouns, lexical cohesion, and ellipsis.These approaches often isolate phenomena through contrastive minimal-pair preferences.
  • Context-aware MT research varies both the amount of context supplied and how that context is selected.Fixed windows remain standard, while longer input and architectural extensions show diminishing or uneven gains.

3 DDP: Discourse Dependency

DDP operationalizes referential reach by tracing named entities and pronouns through a document, then assigning each segment its maximum anchor distance. The resulting measure captures high-dependency benchmark regions while remaining a lower bound whose reliability depends on asymmetric extraction errors.

  • 3.1 Theoretical Background: A segment can be contextually demanding when pronoun referents were introduced many segments earlier, even if its local wording is simple.In the example, resolving “she” and “him” requires information from segments 3 and 1, respectively.
  • 3.1 Theoretical Background: DDP computes each anchor’s distance to its antecedent and defines segment-level DDP as the maximum anchor distance.The example table illustrates chained distance accumulation for “him,” yielding a segment DDP of 4.
  • 3.2 Operationalization: Named entities are detected through NER, while referential pronouns are identified with POS and dependency information and grouped by gender and number.Fallback head nouns provide compatible antecedents when entities are not captured by NER.
  • 3.2 Operationalization: Named-entity chains use prior mentions, while pronouns link to the most recent compatible named entity or head noun; missing compatible antecedents receive d = 1.Distances are computed independently for anchors, and anchorless or first-mention-only sentences receive ddp = 0.
  • 3.2 Operationalization: Distances are searched across the whole document in sentences, while segments only group sentence-level distances for reporting.Segmentation changes aggregation but not the underlying antecedent distances.
  • 3.2 Operationalization: DDP is a lower bound: high-DDP segments are certified to require long-range context, whereas low-DDP segments are not certified to be easy.Unmodeled discourse links and missed mentions can lower the score without generally inflating it.
  • 3.2 Operationalization: 99.2% of segments show one-sided DDP error against gold coreference, with the residual 0.8% attributed to NER false positives.The validation used OntoNotes 5.0 annotations across 80 documents and found Pearson r = 0.81 with gold distance.
  • 3.3 WMT Sets Through the Lens of DDP: Across WMT24++ and WMT25, literary texts have higher DDP than news and social texts, with WMT25 reaching literary mean ddp = 222.6 and max 482.News saturates below ddp = 50, while literary distributions extend into the high-dependency range; both benchmarks remain dominated by low to moderate dependency.

4 Experimental Setup

The study evaluates English–Korean automatic post-editing by varying the amount and selection of document context, then compares human and automatic assessments across discourse dependency.

  • Data and task: The experiment uses WMT24++ English–Korean professionally post-edited translations to isolate context use from raw generation quality.Post-editing reduces confounding from models simply translating poorly and limits benefits from memorized references.
  • Context injection strategies: APEseg provides zero context, APEseq uses the k preceding segment pairs, and APEfull uses the entire document.The sequential condition reflects the de facto standard in prior APE work.
  • Context injection strategies: APErel retrieves the k most semantically similar segment pairs, while APEexp supplies structured declarative knowledge.The retrieval strategy uses embedding-based search, whereas the declarative condition represents document properties such as genre and participants.
  • Context injection strategies: Five strategies test context size and selection while holding prompt structure constant.APEseg, APEseq, and APEfull vary context quantity; APEseq, APErel, and APEexp vary selection at k=5.
  • Evaluation: Human rankings cover two non-overlapping subsets of approximately 500 segments, with ties allowed and preference scores ranging from 1 to 4.Each segment receives two independent judgments; mean Kendall’s τ agreement is 0.37.
  • Evaluation: The analysis computes DDP-stratified preference scores, applies LOWESS smoothing, and compares human preferences with automatic metric scores.It reports absolute strategy preferences, differences relative to APEseq, standard metrics, and TER as an auxiliary edit-volume diagnostic.

5 Results

Human preferences increasingly favor professional post-edits as discourse dependency rises, while context strategies and automatic metrics fail to reflect that growing difficulty consistently.

  • Human preferences: At DDP ≲10, multiple APE strategies surpass professional human post-edits, but from DDP ≈10 onward no APE strategy catches up.Human preference for professional post-edits rises monotonically across the DDP spectrum.
  • Context strategies: APEseq consistently outperforms APEfull across nearly the full DDP range, indicating that full-document context does not improve effective context use.APEseg can match or outperform APEseq at low DDP, where segments are more self-contained and sequential context can add noise.
  • Context strategies: APEseq is preferred over both APErel and APEexp across the DDP spectrum, favoring natural sentence context over structured declarative knowledge.The near-identical APErel and APEexp trajectories suggest that context form matters more than selection criterion in this comparison.
  • Automatic metrics: Across automatic metrics, strategy preferences remain largely insensitive to DDP and receive near-identical scores despite human differentiation.XCOMET ranks APEfull above APEseq throughout the spectrum, reversing the human preference.
  • Statistical confirmation: ρ = 0.28, p = 1.2 × 10^-9, n = 498: the professional-human-versus-APEseq preference gap correlates positively with DDP.The widening human preference is statistically significant rather than confined to the highest-DDP tail.
  • Model-level patterns: Models cluster into two performance groups across DDP, with the upper group nearly indistinguishable under current metrics.The upper cluster includes GPT-5.4, GEMMA-4-31B, HYPERCLOVA-X-SEED-THINK-32B, and QWEN3.5-27B.
  • Model-level patterns: TER decreases as DDP grows across most models but stabilizes beyond DDP ≈20, partly reflecting differential filtering of generation failures.GEMINI-3-FLASH has a 26.9% overall failure rate, concentrated at low DDP; TER is therefore interpreted as edit volume rather than quality.

6 Conclusion

The paper presents DDP as a source-side, metric-free axis for evaluating discourse-dependent translation difficulty. Its results indicate that selecting and analyzing data by discourse dependency reveals limitations that aggregate scores and topical domains obscure.

  • Conclusion: DDP measures contextual difficulty from the source and shifts evaluation toward whether systems can resolve discourse dependencies across documents.It distinguishes segments current systems handle from those requiring discourse-level reasoning before examining outputs.
  • Implications: The main bottleneck is discourse-aware context selection rather than context size.Models can ingest full documents but often fail to selectively attend to antecedents, while k-preceding context may omit them.
  • Implications: DDP stratification exposes metric blindness because standard metrics provide limited sensitivity to inter-sentential phenomena such as coreference and entity tracking.The paper links this limitation to sentence-level alignments and weak supervision in metric training data.
  • Implications: Organizing evaluation by discourse dependency provides a sharper accounting of progress than topical domain labels alone.DDP makes discourse dependency explicit in the source before system outputs are produced.

Limitations

The study’s limitations concern language-pair generality, incomplete discourse anchoring, NLP pipeline dependence, and possible training-data contamination.

  • Language-pair scope: The primary experiments use English-Korean, leaving generalization to other language pairs and source languages unresolved.A partial English-Chinese replication shows broadly consistent metric trends, but additional human evaluation is needed.
  • Anchor coverage: DDP covers named-entity re-mentions and pronominal coreference but excludes lexical cohesion, tense and aspect consistency, ellipsis, and bridging anaphora.Its gender-number grouping also conflates singular gender-neutral “they” with plural references.
  • NLP toolchain: 99.2% of segments are under- or exactly estimated against gold distance, while the remaining 0.8% overestimation is attributable to NER false positives.Thus, the lower-bound interpretation is an empirical near-guarantee rather than an absolute one.
  • Data contamination: WMT24++ may overlap with recent LLM training data, and residual contamination cannot be fully ruled out.The APE setup reduces the benefit of memorized references, but contamination remains particularly plausible at low DDP.

Ethics Statement

The study reports voluntary, compensated participation by expert annotators, anonymized annotations, publicly available research artifacts, and research-only intended use.

  • Human participants: Expert annotators participated voluntarily and were compensated at a fair market rate.
  • Privacy: All annotations were anonymized, and no personally identifiable information was collected.
  • Artifacts and licenses: The datasets, models, and software are publicly available for research purposes under the licenses specified by the study.The WMT24++ and WM25 test set is distributed under CC-BY 4.0, while the automatic metrics use Apache 2.0.
  • Intended use: The released benchmark and code are intended for research use, under CC-BY 4.0 and MIT, respectively.
  • AI assistance: The authors acknowledge Claude Opus 4.6 for writing refinement and code review during paper preparation.

A.1 Validation against Gold Coreference

The validation compares DDP with gold coreference distances and tests whether tool and anchor choices preserve its empirical behavior. DDP correlates strongly with gold distance, rarely overestimates, and remains informative across extraction pipelines.

  • Validation setup: 2,143 segments from 80 English documents were used to compare DDP with gold coreference distances.The documents came from news (n=45) and narrative (n=35) portions of OntoNotes 5.0.
  • Gold-distance computation: Gold distance considers mentions in the current segment whose coreference chains have antecedents in prior segments.Segments without prior-chain mentions receive dgold = 0.
  • Agreement: Pearson r = 0.81 and Spearman ρ = 0.78 show strong segment-level agreement between DDP and gold distance.Agreement is stronger on news (r = 0.86) than narrative (r = 0.74), where bridging anaphora and lexical chains are denser.
  • One-sided error: 86.4% of segments underestimate gold distance, 12.8% match it, and 0.8% overestimate it because of NER false positives.This supports DDP’s lower-bound guarantee in practice.
  • Implications: DDP-stratified conclusions rely on identifying high-dependency segments, because undercounting does not invalidate their gold-distance status.The narrative agreement gap may mean the observed human-metric divergence is underestimated.
  • Tool robustness: The Literary > News > Social ordering and zero-DDP share ordering are preserved under spaCy, Flair, and a weaker extractor.

A.3 Ablation: Anchor Extraction Tool

The ablation tests whether DDP findings depend on the anchor-extraction tool and extends metric-insensitivity analysis to English–Chinese. Results preserve domain-level patterns and show strong cross-tool agreement, while cross-lingual automatic metrics remain largely insensitive to DDP.

  • Anchor extraction agreement: Pearson r = 0.89 and Spearman ρ = 0.88 show strong agreement between the two DDP pipelines, with 69% of segments identical and 85% within ±5.Disagreement concentrates in segments with longer referential chains, where recognition differences accumulate.
  • Domain robustness: Domain means shift by less than one point, and the Literary > News > Social ordering remains preserved under both tools.Both tools retain the findings that benchmarks skew toward low-DDP segments and domain labels conflate within-domain variation.
  • Cross-lingual consistency: En–Zh metric scores remain largely flat across DDP, so automatic-metric insensitivity is not specific to Korean.Across injection strategies, most metrics also fail to separate them, and the APE–HPE gap is smaller than in En–Ko.
  • Scope boundary: Whether human preferences follow the same DDP-stratified pattern across language pairs remains open because stronger Chinese coverage and Chinese discourse properties offer competing explanations.Distinguishing these explanations requires human evaluation.

B.5 Human Evaluation Details

The human-evaluation setup compares context-injection strategies under controlled presentation and measures both preference and computational cost. Human judgments favor sequential context, while full-document context is expensive and automatic metrics remain insensitive to DDP.

  • Sampling and presentation: ≈500 segments per assumption are selected from documents with measurable inter-condition divergence, producing 1,000 total segments without overlap between subsets.APE outputs are anonymized and randomized to mitigate position bias.
  • Human evaluation: Kendall’s τ = 0.37 indicates moderate inter-annotator agreement across the independent rankings.Professional translators ranked four candidates with ties allowed and considered discourse attributes using surrounding context.
  • Inference overhead: APEfull requires 40–84× more input tokens than APEseg yet provides no consistent improvement in translation quality.Latency for API-based models includes network round-trip time and is not directly comparable to open-weight models.
  • Inference overhead: APEseq and APErel incur comparable input overhead at approximately 3× relative to APEseg.APErel is marginally higher because embedding-based retrieval selects longer segments.
  • Overall trade-off: APEseq offers the best trade-off between context effectiveness and computational cost.Open-weight latency rises approximately 1.1–1.8× from APEseg to APEseq, while APEfull reaches up to 5× for QWEN3.5-27B.
  • Metric diagnosis: Automatic metric scores remain largely flat across DDP and cluster tightly across injection strategies, unlike the growing contextual difficulty reflected in human judgments.Document-level metrics do not provide more reliable signal, and TER changes primarily reflect failures and conservative editing rather than quality gains.

C.2 Qualitative case studies

The qualitative cases show how increasing DDP exposes failures in names, pronouns, coreference, and register that fixed-window context strategies do not resolve. Human post-editing uses earlier discourse information to produce more coherent translations.

  • Named entities: Aquilo receives inconsistent Korean transliterations across strategies, showing the absence of a stable entity anchor when context is limited or mismatched.Because DDP is source-computed, it identifies the segment as high-dependency regardless of whether the model resolves the name.
  • Pronouns and coreference: In Seg. 925 with ddp = 54, APE strategies render they generically as 그들, whereas HPE resolves the referent Ivory with a proper noun.The example concerns a non-binary character for whom Korean has no direct equivalent pronoun.
  • Pronouns and coreference: HPE also names Princess Kari directly, while APE strategies render their silent friend generically, demonstrating the value of earlier character information.Both failures exceed what fixed-window context strategies can capture.
  • Register consistency: All APE strategies shift between formal and informal Korean endings without discourse motivation, while HPE maintains the status relationship between Ivory and Aquilo more coherently.The pattern persists across context sizes and selection criteria, suggesting that register coherence requires document-wide grounding.
  • Case-study setup: DDP increases from 6 to 53 and 54 across three consecutive literary segments, illustrating that translation difficulty depends on document-wide referential reach.The case study compares five context-injection strategies with human post-editing.
  • Metric visualization: Figure 10 plots LOWESS-smoothed automatic metric scores against segment-level DDP for En–Ko and En–Zh, with arrows marking each metric’s preferred direction.The corresponding En–Zh figure uses four averaged models, while the En–Ko figure uses six.
Loading 2609.04959v1…