compare-mt: A Tool for Holistic Comparison of Language Generation Systems

1 Introduction

• Metrics such as BLEU can show a gain without revealing which characteristics of the generated output changed.Natural-language outputs are nuanced, without clear yes/no distinctions about correctness.Source: §1 · p. 1
• Manual inspection takes time and can reinforce confirmation bias; purpose-built scripts require prior hypotheses and are also time-consuming to create.Those requirements may be difficult for researchers who are new to a task or not intimately familiar with it.Source: §1 · pp. 1–2
• compare-mt takes two system outputs and a reference, then produces a holistic report of aggregate statistics to guide analysis of specific examples.Developers can browse the report for salient differences between systems.Source: §1 · p. 2
• The tool compares word-level accuracies, sentence-level accuracies or counts, and salient n-grams or sentences where systems differ.It also supports analysis using labels, source-side text through alignments, and extensions to new analysis types.Source: §1 · p. 2

Aggregate Score Analysis

• NMT has higher BLEU and shorter sentence length than PBMT, with no significant RIBES difference.Aggregate analysis can also report confidence intervals and pairwise significance tests.Source: Table 1 · p. 2, §2 · p. 3

Bucketed Analysis

• Both systems perform worse on rare words; PBMT is more robust at low frequencies, while NMT does somewhat better on very high-frequency words.Word accuracy defaults to F-measure, with precision and recall also available to reveal over- or under-production.Source: Figure 2 · p. 2, §2 · p. 3
• By sentence length, PBMT does better on very long sentences and NMT on very short ones; NMT also sometimes produces extremely short sentences, while PBMT has slightly more low-score sentences.Length-difference histograms show how closely output lengths match references; sentence-level score histograms count sentences by accuracy.Source: §2 · pp. 3–4, Figure 3 · p. 3, Figure 4 · p. 3

N-gram Difference Analysis

• N-gram difference analysis ranks system-characteristic n-grams; an example links PBMT’s “hemispher” misspelling to a mistaken alignment and a possible word-stem alignment improvement.The smoothing coefficient trades preference for frequent, robust n-grams against highly characteristic ones.Source: §2 · p. 4, Table 2 · p. 4

Sentence Example Analysis

• Sentence examples show PBMT translating a low-frequency named entity more accurately and NMT generating a multi-word expression with frequent words more accurately.Concrete examples can reinforce holistic patterns or uncover phenomena that may motivate new analyses.Source: §2 · pp. 4–5, Table 3 · p. 4
• Users can also compare individual sentences where words in a selected frequency bucket differ in accuracy; examples help investigate the patterns behind bucketed results.The paper illustrates this with a frequency-one example where PBMT outperforms NMT.Source: §2 · p. 5

Label-wise Abstraction

• Arbitrary word labels enable POS-based analysis, revealing PBMT’s strength on base-form verbs and NMT’s on conjugated verbs.Labels come from external files, so users can preprocess them without modifying compare-mt; examples include morphological tags and sentiment lexicons.Source: §3 · p. 5

Source-side Analysis

• Source-side analysis uses alignments to measure whether input words were translated correctly, with frequency or labels such as POS tags as buckets.Source: §3 · p. 5

Word Likelihood Analysis

• compare-ll buckets per-word reference log likelihoods to produce aggregate statistics, complementing accuracy analysis.Source: §3 · p. 5

Extending compare-mt

• Users can extend word bucketing and scoring by implementing Bucketer and Scorer classes for additional analysis types.Scorers can be used in aggregate score analysis, sentence-bucket analysis, and sentence-example analysis.Source: §3 · pp. 5–6

Example Use-cases

• compare-mt analyses had appeared in several published papers after accelerating the authors’ result analysis and helping identify promising research directions.Source: §4 · p. 6
• Wang et al. (2018) used sentence buckets for score-by-length and sentence-count-by-length-difference analyses.Source: §4 · p. 6
• Published examples also include frequency-based word accuracy and n-gram analyses, plus comparisons of under- and overgenerated n-grams.Examples are reported for Qi et al. (2018), Sachan and Neubig (2018), Michel and Neubig (2018), and Kumar and Tsvetkov (2019).Source: §4 · p. 6

Related Research and Tools

• compare-mt analyzes patterns across entire corpora, drawing on prior holistic analysis methods and word-error, linguistic-label, reordering, and n-gram approaches.The authors state that, to their knowledge, earlier tools did not include compare-mt’s flexible sentence-bucketed analysis.Source: §5 · p. 6
• The authors cite public BSD-licensed availability and modern Python as practical advantages over unavailable or harder-to-use and extend tools.Source: §5 · pp. 6–7
• Example-by-example tools visualize individual outputs or inspect system representations, complementing compare-mt’s corpus-level analysis; broader integration is planned.Source: §5 · p. 7
• Diagnostic test sets target specific phenomena and long-tail behavior, but are often limited to particular language pairs and domains; compare-mt instead analyzes existing test sets with more flexibility and less focus.Source: §5 · p. 7

Conclusion

• compare-mt is an open-source tool for holistic analysis of machine translation and other language-generation systems.Source: §6 · p. 7
• The salient patterns it identifies may help guide further analysis.Source: §6 · p. 7
• A planned improvement is tighter integration between holistic analysis and clicking through to individual examples that illustrate each trait.Source: §6 · p. 7

• The example command generates the report containing the paper’s figures and tables.Source: A Example Command · p. 10, Figure 7 · p. 11
• The score-comparison command takes a reference file and two system-output files as inputs.Source: A Example Command · p. 11
• The configuration supplies tagged system files, a part-of-speech label set, an output directory, and names for PBMT and NMT.Source: A Example Command · p. 11