What Level of Quality can Neural Machine Translation Attain on Literary Text?

1 Introduction

• The study assesses whether state-of-the-art MT can attain useful quality on literary text, considered the greatest challenge for MT.The authors target novels and compare literary-adapted NMT with PBSMT.Source: §1 · pp. 1–2
• Digital books make large amounts of parallel novels available for training literary-adapted MT systems.The authors argue that training on data similar to the application domain should improve performance.Source: §1 · p. 2
• NMT is of interest because prior findings suggest promise for lexically rich text and the possibility of culturally equivalent rather than literal translation.Literary texts are lexically rich, and their readers expect the reading experience—not only meaning—to be preserved.Source: §1 · p. 2
• The authors train PBSMT and NMT on over 100 million words of literary text and evaluate them on twelve widely known novels.The novels span from the 1920s to the beginning of the 21st century.Source: §1 · pp. 2–3

2 State-of-the-Art in MT of Literary Text

• Research on literary text has included figurative-language identification, while comparatively little work has applied MT to literary texts.The related work distinguishes monolingual processing of metaphor, idioms, humour, and irony from literary MT.Source: §2 · p. 3
• Poetry systems have targeted form: one constrained SMT output to preserve meter and rhyme, with BLEU decreasing by around 50% in evaluation on news.Another filtered output to match rhythmic patterns, including English iambic pentameter for Dante’s Divine Comedy.Source: §2 · p. 3
• Research on literary translation has highlighted denser referential chains and the importance of discourse features beyond the sentence.Voigt and Jurafsky (2012) reported denser reference chains in literary texts.Source: §2 · p. 3
• Qualitative analysis of general-domain MT outputs has examined French literary prose and poetry and the particular difficulties of literary translation.Another study used MT followed by non-professional post-editing for a short story, judging the workflow a low-cost alternative but with reduced quality.Source: §2 · pp. 3–4
• Earlier work evaluated PBSMT on a novel against a professional translation, while also analysing literary translatability by domain narrowness and translation freedom.For high- and average-quality passages, the MT output required only a few character edits to match the professional translation.Source: §2 · p. 4
• This study adds literary-adapted NMT, a more challenging English-to-Catalan direction, evaluation across twelve books, and analysis by textual features.The previous study used PBSMT for Spanish-to-Catalan and evaluated one book.Source: §2 · p. 4

3 Experimental Setup

• The setup describes the language pair, training and test data, and tools used in the experiments.It organizes these details across the language-pair, dataset, and tools subsections.Source: §3 · p. 4

3.1 Language Pair

• English-to-Catalan is chosen as a more challenging direction than the authors’ previous Spanish-to-Catalan work because English and Catalan belong to different language families.Catalan represents a mid-size European target language with over 200 identified English e-books available in Catalan.Source: §3.1 · p. 5

3.2.1 Training and Development Data

• Training uses 133 parallel novels with over 1 million sentence pairs, plus monolingual Catalan and English books and out-of-domain data.The monolingual corpora contain around 1,000 Catalan books and around 1,600 English books; out-of-domain data includes OpenSubtitles and web-crawled Catalan.Source: §3.2.1 · pp. 5–6

3.2.2 Test Data

• Testing uses twelve English novels and their professional Catalan translations, selected to span periods, genres, and audiences.The training set may contain other novels by authors represented in the test set, although the test novels themselves are excluded.Source: §3.2.2 · p. 6

3.3 Tools

• The experiments use open-source tools across preprocessing, MT experimentation, and evaluation.The authors describe the tools as state-of-the-art techniques used throughout these stages.Source: §3.3 · p. 7

3.3.1 Preprocessing

• Preprocessing extracts and converts e-books, sentence-splits, tokenizes, aligns, truecases, and normalizes text before filtering sentence pairs by length.Sentence alignment uses confidence thresholds above 0.3 for training and 0.5 for test data.Source: §3.3.1 · pp. 7–8

3.3.2 MT Toolkits and Evaluation

• PBSMT uses Moses and NMT uses Nematus, with default settings unless otherwise specified; evaluation uses BLEU and ranked human judgments.System significance is tested with paired bootstrap resampling at p ≤ 0.01 over 1,000 iterations.Source: §3.3.2 · p. 8

4.1 PBSMT System

• PBSMT development tuning found absolute gains of 0.57 BLEU with 4-gram NPLM, 0.75 with 5-gram NPLM, 0.4 with OSM, and 0.14 with out-of-domain data.The system combined in-domain and out-of-domain data with reordering models, OSM, and language models.Source: §4.1 · pp. 8–9, Table 4 · p. 9

4.2 NMT System

• NMT training used in-domain parallel data plus synthetic data made by translating Catalan in-domain monolingual text into English.The system was trained solely on in-domain data.Source: §4.2 · p. 9
• Development BLEU rose from 0.0251 after 3 hours to 0.2999 after 36 hours, then increased more slowly to a 0.3356 maximum before plateauing.Models were saved every 3 hours and evaluated on the development set.Source: §4.2 · p. 10, Figure 1 · p. 10
• Ensembling raised BLEU by 2.05 points over the best individual model, while 12 hours of training with fixed embeddings did not improve performance.The two ensemble scores were 0.3561 without further fixed-embedding training and 0.3555 with it.Source: §4.2 · p. 11
• 0.3948 BLEU after right-to-left reranking was 2.59 points above the previous NMT system and 17.6% above the best individual system.Subword units first raised the score to 0.3689; the subsequent reranking produced the higher score.Source: §4.2 · p. 11
• 0.3948 BLEU for the best NMT system exceeded the best PBSMT system’s 0.3473 by 4.75 points, despite a development-set comparison bias favoring PBSMT.The NMT system was an ensemble using subword units and right-to-left reranking.Source: §4.2 · p. 11

5.1 Automatic Evaluation

• BLEU scores varied widely across novels, from 0.1611 for PBSMT on Ulysses to 0.3892 for NMT on Harry Potter #7.The authors note that fiction is far from a monolithic domain.Source: §5.1 · p. 12
• NMT beat PBSMT by a statistically significant margin on every novel, with average gains of 10.67% relative and 3 points absolute.The relative improvement ranged from 3.11% for Sunset Park to 14% for Hunger Games #3.Source: Table 5 · p. 12, §5.1 · p. 12

5.1.1 Analysis

• The analysis tested whether lexical richness, training-data novelty, and average sentence length correlated with system performance or NMT’s relative improvement.The selected features were motivated by prior claims about NMT performance on lexically rich, long, or training-unrelated text.Source: §5.1.1 · pp. 12–13

Lexical richness

• TTR was calculated on a random sample of approximately 20,000 words per novel to account for differing book sizes.The sample size was slightly below the 24,233 words in the dataset’s smallest novel, The Old Man and the Sea.Source: Lexical richness · p. 13

Sentence length

• Average sentence length was calculated as each novel’s total tokens divided by its number of sentences.Both counts were reported in Table 2.Source: Sentence length · p. 13

• Novelty is measured by the ratio of unique four-grams shared by the parallel training data and a novel to the number of unique training-data four-grams.Higher overlap indicates less novelty; the analysis uses 20,000 words from randomly selected sentences per novel.Source: Novelty with respect to the training… · p. 13

• Ulysses has the highest TTR (0.276) and lowest four-gram overlap (0.216), while Sunset Park’s average sentence length (32.434) exceeds every other novel by over 10 points.Table 6 reports TTR, average sentence length, and four-gram overlap for the twelve novels.Source: Results · p. 14, Table 6 · p. 14
• No significant correlation was found between TTR and NMT’s gains over PBSMT in this setup, unlike a reported moderate correlation on transcribed speeches.The speech-study correlation was r = 0.73, measured with mTER.Source: Results · p. 15
• ρ = −0.45: NMT’s relative improvement over PBSMT decreases as sentence length increases, appearing to explain Sunset Park’s low relative improvement.Sunset Park has a much longer average sentence length than the other test novels.Source: Results · pp. 14–15, Figure 2 · p. 15
• r = 0.62 for PBSMT and r = 0.66 for NMT: both systems’ BLEU scores correlate positively with four-gram overlap, but relative improvement does not correlate significantly with overlap.The authors conclude that the expectation that less-related test data widens NMT’s advantage over PBSMT does not hold in this setup.Source: Results · pp. 14–15, Table 7 · p. 14, Figure 3 · p. 16

5.2 Human Evaluation

• Two native Catalan speakers with advanced English ranked translations for ten randomly selected passages per book, each containing ten contiguous sentences to provide context beyond the sentence level.The three books were Orwell’s 1984, Rowling’s Harry Potter #7, and Salinger’s The Catcher in the Rye; system identities were concealed from evaluators.Source: §5.2 · p. 17

5.2.1 Inter-annotator Agreement

• Inter-annotator agreement is 0.22 for Orwell, 0.18 for Rowling, and 0.38 for Salinger, ranging from slight to fair agreement.The passage characterizes Orwell and Salinger as fair agreement and Rowling as at the higher end of slight agreement.Source: §5.2.1 · p. 18

5.2.2 Pairwise Rankings

• NMT is judged equivalent in quality to human translation more often than PBSMT: 16.7% vs 7.5% for Orwell, 31.8% vs 18.1% for Rowling, and 34.3% vs 19.8% for Salinger.The authors interpret these rates as indicating that some translations could require no correction in post-editing; the figures show pairwise rankings against human translation.Source: §5.2.2 · p. 19, Figure 6 · p. 19, Figure 7 · p. 20
• NMT ranks above PBSMT most often, equal rankings form the second-largest category, and PBSMT ranks above NMT in fewer than 20% of cases in each book.The reported NMT-over-PBSMT shares are 41.4% and 54.7%, and equal-ranking shares are 27.8% and 39.4%.Source: §5.2.2 · pp. 19–20, Figure 9 · p. 21

5.2.3 Overall Human Scores

• Human translations, NMT, and PBSMT fall into significantly different clusters for all three books, ordered from highest to lowest as human translation, NMT, then PBSMT.The overall scores use TrueSkill with 1,000 ranking iterations followed by clustering at p < 0.05.Source: §5.2.3 · pp. 20–21, Figure 10 · p. 21

6 Conclusions and Future Work

• NMT outperformed PBSMT by a significant margin on all 12 novels.Both systems were trained on large amounts of parallel novels.Source: §6 · p. 22
• Of lexical richness, training-data novelty, and average sentence length, only sentence length showed a meaningful correlation with NMT’s relative improvement over PBSMT.The authors considered sentence length the main reason for one novel’s relatively low improvement, while noting that the novel was an outlier in sentence length.Source: §6 · p. 22
• For two of three books, native speakers judged NMT translations equivalent in quality to human translations in around one third of cases, versus one sixth for PBSMT.The human evaluation manually ranked translations from NMT, PBSMT, and human translators.Source: §6 · p. 22
• Future work will test MT-assisted literary translation through post-editing, assessing effects on translation effort and the resulting translation.Source: §6 · p. 22