PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture Models for Diverse Machine Translation: Tricks of the Trade

Tianxiao Shen, Myle Ott, Michael Auli, Marc'Aurelio Ranzato

arXiv:1902.07816v2cs.CLcs.LG

TL;DR

Machine translation must represent multiple valid outputs, but mixture models have been underexplored and can collapse into degenerate behaviors. The paper develops a multi-reference quality-and-diversity evaluation and systematically studies mixture-model design choices. It finds that suitable configurations outperform variational and diverse-decoding alternatives while enabling interpretable control of translation styles.

  • Problem

    Machine translation is a one-to-many task, yet mixture models for diverse generation are underexplored and prone to training degeneracies.

  • Method

    The paper evaluates 32 mixture-model variants using multiple human references and compares their design choices, including EM type, priors, parameterization, responsibility updates, and dropout treatment.

  • Results

    Successfully trained mixture models consistently outperform variational NMT and diverse decoding methods across three WMT benchmarks, while different components capture interpretable translation styles.

  • Takeaways & Limitations

    Disabling dropout in responsibility computation and choosing suitable mixture configurations yields stronger quality–diversity trade-offs than variational models and heuristic diverse decoding.

  • Takeaways & Limitations

    Sub-optimal design choices can cause one component to dominate while others die, or cause all components to behave identically.

Abstract

from arXiv · show

Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models have been hardly explored in text generation applications such as machine translation. In principle, they provide a latent variable to control generation and produce a diverse set of hypotheses. In practice, however, mixture models are prone to degeneracies---often only one component gets trained or the latent variable is simply ignored. We find that disabling dropout noise in responsibility computation is critical to successful training. In addition, the design choices of parameterization, prior distribution, hard versus soft EM and online versus offline assignment can dramatically affect model performance. We develop an evaluation protocol to assess both quality and diversity of generations against multiple references, and provide an extensive empirical study of several mixture model variants. Our analysis shows that certain types of mixture models are more robust and offer the best trade-off between translation quality and diversity compared to variational models and diverse decoding approaches.\footnote{Code to reproduce the results in this paper is available at \url{https://github.com/pytorch/fairseq}}

1. Introduction

Machine translation is inherently one-to-many, motivating models and evaluations that capture multiple valid outputs. This paper studies mixture-model design choices and finds that careful training and evaluation improve the quality–diversity trade-off.

  • Machine translation has many plausible, semantically equivalent outputs because languages differ in information and grammatical structure.
  • The paper introduces an evaluation protocol using multiple references to measure both translation quality and hypothesis-set diversity.
  • The study analyzes mixture models across hard versus soft EM, parameterization, priors, responsibility updates, and regularization noise.
  • Disabling dropout during responsibility computation mitigates mixture-model degeneracies and supports expert specialization.
  • Hard mixtures produce more diverse generations, while uniform priors encourage every component to produce good translations for each source sentence.
  • Across three WMT datasets, successfully trained mixture models outperform variational NMT and diverse decoding methods while supporting interpretable translation styles.

2. Related Work

Prior work examined uncertainty and latent variables in machine translation, but evidence about mixture models remained incomplete. Earlier studies often lacked multi-reference quality-and-diversity evaluation, broad design comparisons, or strong baselines.

  • Earlier work introduced metrics and tools for translation uncertainty, while modern autoregressive NMT models capture that uncertainty only to a limited extent.
  • Latent-variable MT studies used Gaussian or discrete latent structures to represent uncertainty and influence output predictions.
  • A prior soft mixture model study did not use multi-reference datasets, analyze the full design space, or compare against variational NMT and diverse decoding baselines.
  • Related latent-variable approaches also target dialogue and image captioning, whereas this mixture model is presented as simpler than VAE- or GAN-based alternatives.

3. Mixture Models for Diverse MT

Mixture models introduce a latent expert variable to model translation uncertainty and generate diverse hypotheses, but their benefits depend on training and parameterization choices. The paper describes EM-based variants, decoding, and design choices intended to promote both expert specialization and translation quality.

  • Model formulation: Mixture models represent alternative translation hypotheses through a multinomial latent variable whose values correspond to expert components.The model decomposes p(y|x; θ) into a conditional prior over experts and an expert-specific likelihood.
  • Decoding and training: Decoding enumerates the K experts and greedily generates one hypothesis from each, making mixture decoding efficient and parallelizable.This procedure applies baseline decoding strategies to p(y|z, x; θ) separately for each expert.
  • Degeneracies: Naïve mixture models can degenerate because only one component is trained or because all components behave identically.In either case, the latent variable provides no benefit over a baseline model.
  • Model variants: The paper studies hard versus soft responsibilities, learned versus uniform priors, parameter sharing, online versus offline EM, and responsibility regularization.Hard mixtures assign each example to one expert, while uniform priors encourage all experts to produce good hypotheses for each source sentence.
  • Decoding and training: Disabling dropout during responsibility computation is important because even small amounts of E-step regularization noise can hamper consistent expert use.The paper alternates E- and M-step updates through online or offline schedules and reports that naïve dropout can make models ignore the latent variable.

4. Metrics

The evaluation protocol uses multiple references to measure translation quality and hypothesis diversity separately. BLEU assesses overall quality, while Pairwise-BLEU assesses similarity among hypotheses and helps identify degeneracies.

  • Pairwise-BLEU measures similarity among hypotheses; lower values indicate greater diversity, with human Pairwise-BLEU as the ideal comparison.
  • The protocol compares system BLEU and Pairwise-BLEU with human-reference scores to assess both quality and diversity.
  • BLEU measures the overall quality of a hypothesis set using multiple references in a leave-one-out evaluation.Human and system BLEU use comparable reference counts.
  • Very low BLEU and Pairwise-BLEU indicate degeneracy D1, where one expert handles all inputs and other latent values produce poor generations.
  • Good BLEU with extremely high Pairwise-BLEU indicates degeneracy D2, where the latent variable is ignored and experts generate nearly identical outputs.

5. Experiments

Experiments examine mixture-model design choices, dropout handling, and performance across three WMT datasets. The results identify configurations that avoid degeneracy and balance translation quality with diversity, with hMup selected for extended evaluation.

  • Dropout and degeneracy: Dropout in the E-step causes mixture models to ignore the latent variable, whereas applying dropout only during the M-step helps experts diversify.Using dropout in both steps produces very high Pairwise-BLEU and low diversity; disabling dropout entirely harms translation quality through overfitting.
  • Mixture-model design choices: Soft mixtures with shared parameters and offline responsibilities exhibit D2 degeneracy, producing nearly identical hypotheses with Pairwise-BLEU close to 100.The authors attribute this to shared mixtures failing to specialize latent embeddings under long training with initial random responsibilities.
  • Mixture-model design choices: Independent components with online responsibility assignment exhibit D1 degeneracy, because a rich-gets-richer effect leaves only one expert trained and sharply lowers BLEU.Parameter sharing alleviates this failure because updates to one expert also affect the others.
  • Mixture-model design choices: Hard assignment is more diverse than soft assignment, while a uniform prior encourages all components to produce good translations for each source sentence.Offline independent models are more diverse but lower quality, whereas online shared models offer a stronger quality-diversity balance.
  • Robust configurations: Online shared models, offline independent sMup and hMup, and offline shared hMup are robust variants that offer different quality-diversity trade-offs near human performance.Considering computational and memory cost favors shared parameters and hard EM; hMup is adopted for subsequent experiments because it is also simple to implement.
  • Large-scale evaluation: Across three WMT datasets, hMup achieves the best trade-off between translation quality and diversity compared with beam search, sampling, diverse beam search, and other baselines.Beam search has high quality but low diversity, while sampling is highly diverse but has poor BLEU; increasing the number of hypotheses generally increases diversity at the cost of quality.
  • Qualitative analysis: On Zh-En examples, hMup produces diversity in wording, word order, and clause structure beyond beam and diverse beam search, while latent states consistently specialize in translation styles.Observed preferences include past versus present tense, this versus that, and per cent versus %.

6. Conclusion

The study evaluates 32 mixture-model variants for machine translation and identifies design choices that affect degeneracy, quality, and diversity. Recommended configurations offer better quality–diversity trade-offs than variational models and heuristic diverse decoding.

  • 6. Conclusion: The study investigates 32 variants spanning EM type, prior, component parameterization, responsibility updates, and dropout treatment.These variants combine hard or soft EM, uniform or learned priors, shared or independent parameters, online or offline updates, and standard or removed dropout during responsibility computation.
  • 6. Conclusion: Mixture models can fail when only one component trains or when all components behave identically and the latent variable is ignored.These are the study’s two highlighted degeneracy modes under sub-optimal design choices.
  • 6. Conclusion: The paper’s broader contribution is insight into training deep sequence models with discrete latent variables.The authors identify mixture-model behavior as dependent on combinations of design choices.
  • 6. Conclusion: Recommended configurations provide better quality–diversity trade-offs than variational models and heuristic diverse decoding approaches.The paper presents this as the main conclusion of its empirical comparison.

A. Effect of Dropout

Dropout noise makes hard responsibility assignments unstable early in training, which can prevent latent states from specializing. The analysis therefore supports disabling dropout during responsibility computation.

  • A. Effect of Dropout: Dropout noise can cause mixture models to become degenerate, while removing it during responsibility computation mitigates this problem.The paper links consistent expert use to specialization and reports that even small regularization noise can hamper it.
  • A. Effect of Dropout: The authors hypothesize that dropout weakens dependence on the latent variable and makes the hard E-step select latent values randomly.This random selection prevents different latent states from specializing and can make the model ignore them.
  • A. Effect of Dropout: 42% of optimal latent assignments change at dropout probability 0.1 at the beginning of training.The experiment measures how often the optimal z changes after applying dropout to a randomly initialized model.
  • A. Effect of Dropout: Figure 4 plots dropout rate against the fraction of hard responsibility assignments changed under dropout noise.The experiment uses WMT’17 En-De, K = 2 latent categories, and the base Transformer architecture.

B. Another Diversity Metric

Reference coverage complements Pairwise-BLEU by measuring how many distinct references are matched by generated hypotheses. Higher coverage indicates that the hypothesis set spans more reference alternatives.

  • B. Another Diversity Metric: Reference coverage pairs each hypothesis with its best-matching reference and counts distinct references matched by at least one hypothesis.The count is averaged over all test sentences.
  • B. Another Diversity Metric: Low reference coverage means that hypotheses cluster near a few references, whereas high coverage indicates a more diverse set.The metric is intended to assess whether generated hypotheses cover most available references.
  • B. Another Diversity Metric: Reference coverage ranges from 1 to the total number of available references, giving it more intuitive numerical values than Pairwise-BLEU.The paper reports both metrics for completeness.

C. Detailed Results

Detailed comparisons show that mixture models balance translation quality and diversity, with hMup outperforming diverse decoding approaches on the WMT’17 En-De comparison. Tables also examine model variants and degeneracy-sensitive metrics.

  • C. Detailed Results: Mixture models provide a good trade-off between translation quality and diversity across three WMT benchmark datasets.Table 5 compares online, shared hMup and sMup with several baselines; hMup is more diverse, while sMup gives higher translation quality.
  • C. Detailed Results: On WMT’17 En-De, hMup produces more diverse and higher-quality translations than diverse beam search.Sampling is diverse but lower quality, beam search is high quality but low diversity, and hMup covers the largest number of references except unrestricted sampling.
  • C. Detailed Results: Corpus-level BLEU under a fixed latent state helps detect degeneracy in which some states rarely generate good hypotheses.Very low corpus-level BLEU for a state indicates that the state may be affected by degeneracy D1.
  • C. Detailed Results: Table 7 compares mixture-model design choices using Pairwise-BLEU, BLEU, and corpus-level BLEU with K = 3 components.Each configuration is run five times with different random seeds, with means and selected standard deviations reported.
Loading 1902.07816v2…