PaperlayerView exact revision on arXiv

Source-linked AI summary

(Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation

Carlos Escolano, Marta R. Costa-jussà, José A. R. Fonollosa

arXiv:1810.06351v1cs.CL

TL;DR

Universal language representation remains a goal for machine translation, and earlier multilingual approaches had not explicitly designed their architectures to achieve it. This paper combines autoencoders and encoder-decoders with a correlation-based interlingual loss, reporting state-of-the-art translation quality on WMT 2017 Turkish-English while concluding that further work is needed for the desired representation.

  • Problem

    Earlier multilingual NMT approaches had not explicitly designed their architectures to learn a universal language representation, a goal that could reduce the number of translation systems as languages grow.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    The architecture combines variational autoencoders and encoder-decoders with a correlation-based loss that aligns intermediate representations across languages.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The authors report state-of-the-art translation quality in both directions on WMT 2017 Turkish-English, while the conclusion describes quality as similar to the baseline system.

    Source: §1 · p. 1, §10 · p. 6

  • Takeaways & Limitations

    The proposed architecture trains translation systems while moving toward an intermediate representation shared across languages.

    Source: §10 · p. 6

    The authors conclude that more work is needed to produce the desired universal language representation for interlingual MT.

    Source: §10 · p. 6

Abstract

from arXiv · show

Universal language representation is the holy grail in machine translation (MT). Thanks to the new neural MT approach, it seems that there are good perspectives towards this goal. In this paper, we propose a new architecture based on combining variational autoencoders with encoder-decoders and introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we are able to train multiple encoders and decoders for each language, sharing a common universal representation. Since the final objective of this universal representation is producing close results for similar input sentences (in any language), we propose to evaluate it by encoding the same sentence in two different languages, decoding both latent representations into the same language and comparing both outputs. Preliminary results on the WMT 2017 Turkish/English task shows that the proposed architecture is capable of learning a universal language representation and simultaneously training both translation directions with state-of-the-art results.

1 Introduction

The paper targets a universal language representation for neural machine translation, which could reduce the number of translation systems as languages grow. It proposes an architecture that explicitly trains this representation alongside translation.

1 Introduction

  • NMT outperforms earlier statistical approaches on most tasks, while statistical MT can still outperform NMT in low-resource tasks.Source: §1 · p. 1
  • A universal language could support single encoders and decoders for each language, reducing the number of translation systems from quadratic to linear in the number of languages.Source: §1 · p. 1
  • Earlier multilingual NMT work examined whether encoder-decoder architectures could yield a universal representation, but did not explicitly design the architecture to achieve one.Source: §1 · p. 1
  • The proposed architecture combines variational autoencoders and self-attentive encoder-decoders with a correlation loss between intermediate representations from different languages.The loss is intended to force the system to learn a universal representation while training multiple translation systems.Source: §1 · p. 1
  • The authors report state-of-the-art translation quality in both directions on the WMT 2017 Turkish-English task while training a universal language representation.Source: §1 · p. 1

2 Related Work

The related work distinguishes the paper’s learned intermediate representation from classical interlingua and surveys multilingual NMT and related representation approaches. It positions this work around language-specific encoders and decoders and interlingual MT.

2 Related Work

  • Classical interlingua approaches seek conceptual understanding across languages, whereas this work learns a deep intermediate representation without requiring it to encode semantics.Source: §2 · pp. 1–2
  • Shared encoder-decoder work reports zero-shot translation and visual evidence that sentences in different languages may have nearby common representations.Source: §2 · p. 2
  • Dedicated multilingual systems include many-to-one, one-to-many, and one-to-one designs; related work also studies common sentence representations and unsupervised MT.The paper focuses on one-to-one systems as closest to its approach, while distinguishing its goal of interlingual MT from adapting representations to other NLP tasks; that adaptation is left for future work.Source: §2 · p. 2

3 Contribution

The paper makes universal representation an explicit part of multilingual NMT training and introduces evaluation measures for representation similarity during training and inference.

3 Contribution

  • The architecture adds correlation between universal representations to the loss, explicitly training the universal representation rather than treating it only as an outcome to evaluate.Source: §3 · p. 2
  • The paper proposes a universal language representation in an architecture that uses self-attention mechanisms.Source: §3 · p. 2
  • The work introduces correlation to compare representations during training and BLEU to compare decoded outputs from two language encodings of the same input sentences during inference.Source: §3 · p. 2

4 Background

The background introduces variational autoencoders, decomposed vector quantization, and correlated nets as techniques used in the proposed architecture.

4 Background

  • The paper draws on variational autoencoders, decomposed vector quantization, and correlated nets to develop its architecture.Source: §4 · p. 2

4.1 Variational Autoencoders

  • Variational autoencoders learn parameters of a probability distribution over intermediate representations, allowing synthetic instances to be sampled and decoded.Source: §4.1 · pp. 2–3

4.2 Decomposed Vector Quantization

  • Decomposed vector quantization represents an encoder output with the nearest vector from a table, while decomposition splits the representation across tables before concatenation.Using n tables with the same number of parameters as one table can generate K^n possible vectors of dimension D.Source: §4.2 · p. 3

4.3 Correlated Nets

  • Correlated nets use correlation information to measure distance between intermediate representations learned from two data sources.The passage identifies h(x_i) and h(y_i) as individual representations and h(X) and h(Y) as their respective means.Source: §4.3 · p. 3

5 Proposed Model Architecture

The model trains language-specific encoders and decoders around a shared intermediate representation, with a distance term encouraging representations from parallel sentences to align. Its architecture combines autoencoders and Transformers, and the authors consider alternative distances and ways to add languages.

5 Proposed Model Architecture

  • The proposed system trains one encoder and decoder per language to produce and consume a common intermediate representation, exploring discrete and continuous representations and different distance measures.This representation is intended to be compatible across the system's components.Source: §5 · p. 3
  • Each language uses a Transformer autoencoder, whose self-attention-based components can be recombined during inference without previous-step information.Figure 1 presents an architecture example in which every module is compatible with the intermediate representation.Source: §5 · p. 3, Figure 1 · p. 4
  • All components are optimized simultaneously, with both translation directions generated during training and all languages considered equally.The objective includes each language's autoencoding and cross-language translation losses, plus a distance between the intermediate representations.Source: §5 · p. 3
  • The authors test correlation distance between batch representations and maximum distance, defined as the largest difference between source and target representations.The correlation measure assesses how correlated the autoencoders' intermediate representations are for each training batch.Source: §5 · pp. 3–4
  • To add language Z, the system trains X–Z directions over a pretrained system, requiring X–Y and X–Z parallel data rather than a multilingual X–Y–Z corpus.The proposed scaling setup adds the new language through parallel data with X.Source: §5 · p. 4

6 Evaluation of the Universal Language Representation

The evaluation addresses whether representations from different languages are usable by the same decoders, since similar representations may not guarantee compatible encoders and decoders. It evaluates parallel-sentence encodings through autoencoding, translation, and a BLEU comparison between their outputs.

6 Evaluation of the Universal Language Representation

  • Similar representations may not yield compatible encoder-decoder pairs, and different training runs can have different representation distances despite similar translation outputs.The evaluation therefore focuses on decoder behavior rather than representation distance alone.Source: §6 · p. 4
  • For parallel test sentences, the method encodes both languages and runs one of the system's decoders on both encodings, producing autoencoding and translation outputs.The resulting outputs can be scored against references in both languages.Source: §6 · p. 4
  • The Interlingua BLEU measure compares the autoencoding output as reference with the translation output as hypothesis to assess how differently the decoder handles the two encodings.Figure 2 shows the measure's pipeline.Source: §6 · p. 4, Figure 2 · p. 4

7 Experimental framework

Experiments use Turkish–English parallel data with news development and test sets. The setup applies standard text preprocessing and uses a Transformer with fixed model and optimization settings.

7 Experimental framework

  • Experiments use Turkish–English parallel data from SETIMES, with newsdev2016 for development and newstest2016 for testing.Source: §7 · p. 4
  • Preprocessing limits sentences to 50 words and applies tokenization, Moses truecasing, and byte-pair encoding segmentation.Source: §7 · p. 4
  • The Transformer setup uses six blocks with eight attention heads each, a 12,000-word vocabulary, latent size 128, and Adam at a fixed 0.0001 learning rate.The model uses the Transformer with default parameters.Source: §7 · p. 4

8 Results

For translation quality, non-variational autoencoders outperform variational ones, and correlation distance outperforms maximum distance. In the representation-quality evaluation, same-language decoding is better than cross-language decoding, while low autoencoder–translation BLEU indicates that decoding from the universal representation remains distant.

8.1 Translation quality

  • The non-variational autoencoder outperforms the variational autoencoder by more than 1 BLEU point in both translation directions.The comparison is reported against the WMT 2017 results using parallel data only.Source: §8.1 · p. 5, Table 1 · p. 5
  • Correlation distance improves translation results by approximately 1.5 BLEU in both directions over maximum distance and performs similarly to the one-direction Transformer baseline.Source: §8.1 · p. 5, Table 1 · p. 5

8.2 Universal language representation quality

  • The representation-quality test uses univ+corr, the best-performing model from Table 1, to compare decoder performance across the two encoders’ representations.Source: §8.2 · p. 5
  • On univ+corr, the decoder produces better BLEU when given its own language’s encoder representation than when given the other language’s representation.Table 2 compares autoencoder and MT output BLEU.Source: §8.2 · p. 5, Table 2 · p. 5
  • The autoencoder–translation BLEU is proposed as a measure of universal-language quality; low scores indicate that decoding from the universal representation remains distant.Source: §8.2 · p. 5

9 Visualization

The visualization uses UMAP to show sentence representations, with Turkish and English forming separate clusters. Parallel sentences do not have a constant distance in the representation space.

9 Visualization

  • UMAP provides a nonlinear dimensionality reduction for visualizing the sentence-representation space.Source: §9 · p. 5
  • Turkish and English representations appear in different clusters, rather than being inseparable as expected if both encoders produced the same representations.Source: §9 · p. 5, Figure 3 · p. 5
  • The distance between parallel sentences varies across the representation space.Source: §9 · p. 5

10 Conclusions

The proposed translation architecture uses a common intermediate representation, and its quality is reported as similar to the baseline. A same-language decoding comparison indicates that more work is needed for interlingual MT, while exploiting limited resources is proposed as future work.

10 Conclusions

  • The architecture includes a common intermediate representation as a training objective and explores self-attentive variational and non-variational autoencoders.Source: §10 · pp. 5–6
  • Figure 4 visualizes the sentence representation space with UMAP, coloring Turkish in yellow and English in green.Source: Figure 4 · p. 6
  • The proposed architecture achieves quality similar to the baseline while moving toward an intermediate representation.The conclusion describes this as progress toward, rather than completion of, the representation goal.Source: §10 · p. 6
  • Comparing outputs from two language encodings under the same decoding shows that more work is needed for the desired interlingual representation.Source: §10 · p. 6
  • Adding encoders and decoders using parallel data for one language of a trained system is proposed as further work to exploit limited resources.The proposed extension would use the learned representation to translate to and from languages already in the system.Source: §10 · p. 6

Related papers

Loading 1810.06351v1…