Source-linked AI summary
Bilingual-GAN: A Step Towards Parallel Text Generation
Ahmad Rashid, Alan Do-Omri, Md. Akmal Haidar, Qun Liu, Mehdi Rezagholizadeh
TL;DR
The paper addresses how to generate parallel sentences in two languages while reducing reliance on parallel training data. Bilingual-GAN learns a shared latent space with shared encoders, back-translation, and a shared decoder, then uses a GAN to generate latent codes for either language. Experiments on English-French Europarl and Multi30k show fluent generated sentences with parallelism in both languages, while the authors identify quality and parallelism improvements as future work.
Problem
Bilingual-GAN addresses the challenge of generating parallel bilingual text when translation systems generally require large amounts of parallel data.
Method
The model trains a shared encoder-decoder translation unit with back-translation, then trains a GAN to mimic the shared latent space and generate code decoded into either language.
Results
The authors report fluent generated sentences exhibiting parallelism in English and French across quantitative and qualitative evaluation.
Takeaways & Limitations
The work provides a joint latent-space formulation of neural machine translation and neural text generation from a bilingual perspective.
Takeaways & Limitations
The authors identify improving generated-sentence quality and increasing parallelism, especially without parallel NMT training data, as future work.
Abstract
from arXiv · showhide
Latent space based GAN methods and attention based sequence to sequence models have achieved impressive results in text generation and unsupervised machine translation respectively. Leveraging the two domains, we propose an adversarial latent space based model capable of generating parallel sentences in two languages concurrently and translating bidirectionally. The bilingual generation goal is achieved by sampling from the latent space that is shared between both languages. First two denoising autoencoders are trained, with shared encoders and back-translation to enforce a shared latent state between the two languages. The decoder is shared for the two translation directions. Next, a GAN is trained to generate synthetic "code" mimicking the languages' shared latent space. This code is then fed into the decoder to generate text in either language. We perform our experiments on Europarl and Multi30k datasets, on the English-French language pair, and document our performance using both supervised and unsupervised machine translation.
1 Introduction
Bilingual-GAN combines shared latent representations with adversarial generation to produce parallel sentences in two languages, including when training uses only monolingual corpora. It builds on attention-based translation and latent-space text generation to generate from a shared bilingual space.
- Contributions: Bilingual-GAN learns a shared latent space between two languages and generates text from that space in either language.
- Motivation: Attention-based NMT performs strongly but generally requires massive parallel data, motivating methods that operate without parallel data.Shared autoencoders and cross-lingual word embeddings had shown promise for reducing this requirement.
- Background: GAN-based text generation avoids some conventional autoregressive limitations but faces nondifferentiability because text outputs are discrete.Using softmax outputs instead can make real and generated text easy for the discriminator to distinguish.
- Contributions: Bilingual-GAN generates parallel sentences concurrently in two languages from a shared latent space.The model is designed to translate bidirectionally and generate in either language.
- Contributions: Its GAN is designed to match the latent distribution produced by an attention-based NMT encoder.
- Contributions: The model explores parallel-sentence generation using only monolingual corpora.
2 Related Work
Related work on unsupervised translation learns shared latent representations and uses back-translation for pseudo-supervision. GAN-based NLP methods address broader generation tasks, but discrete text makes gradient-based adversarial training difficult and motivates latent-code approaches.
- Unsupervised Translation: Unsupervised translation systems use language models, shared latent representations, and back-translation to provide pseudo-supervision without parallel corpora.
- GANs in NLP: Prior GAN-based NLP research spans machine translation, dialogue, question answering, and natural language generation.
- GANs in NLP: Discrete words prevent straightforward gradient propagation through a text generator, making GAN training for NLP challenging.
- GANs in NLP: Latent-code methods address this challenge by deriving a continuous latent representation of text.
3 Methodology
Bilingual-GAN combines a shared two-language translation unit with a GAN that learns the encoder’s latent-state manifold. Generated latent codes are decoded into either language, while reconstruction, cross-domain, and adversarial losses support bilingual translation and generation.
- 3.1 Translation Unit: The translation unit uses a shared encoder and decoder across two languages, with separate embedding tables and language-specific decoder projection layers.The two corpora need not be sentence translations of each other.
- 3.1 Translation Unit: Three losses train the encoder-decoder pair to reconstruct inputs, translate across languages, and produce language-independent codes.The reconstruction objective is a standard autoencoder loss, while the other losses address translation and latent alignment.
- 3.1.1 Supervision: Cross-domain training translates a sentence into the other language and decodes it back into the original language for comparison with the input.Unsupervised training uses word-by-word translations from nearest neighbors in a multilingual embedding space; aligned data instead provides ground-truth translations.
- 3.2 Bilingual Text Generation Unit: The bilingual generator learns the translation encoder’s latent-state manifold with an Improved Wasserstein GAN gradient-penalty objective.The discriminator compares real encoder codes with generated codes, and the gradient penalty uses interpolated latent codes with λ = 10.
- 3.2 Bilingual Text Generation Unit: The encoder uses bidirectional latent states concatenated depth-wise to represent the input as a latent-state matrix rather than a single code vector.This representation exploits the attention-based architecture and bidirectional encoder.
- 3.2.2 Training: The GAN is trained on encoder outputs as real codes and reshaped upsampled noise as fake codes, then the generated code is decoded in either language.The generator uses one linear layer and five one-dimensional convolutional layers with residual connections; the pretrained decoder performs bilingual decoding.
4 Experiments
Experiments evaluate bilingual translation and text generation on Europarl and Multi30k for English–French, using supervised and unsupervised settings. Results cover BLEU, perplexity, and human judgments of fluency and parallelism.
- Datasets and settings: Experiments use Europarl and Multi30k English–French data, with supervised and unsupervised training settings.Europarl provides filtered parallel sentences, while Multi30k supplies paired French and English image captions.
- System specifications: The NMT unit uses 300-dimensional embeddings, bidirectional LSTM encoders, and an attention-equipped LSTM decoder.The discriminator is a three-layer feed-forward network when adversarial loss is used.
- Translation: Translation results show that removing adversarial loss helps, two encoder layers are beneficial, and the new concatenation method yields a small improvement.The authors attribute the concatenation improvement to combining forward and backward states for each input word.
- Text generation: Corpus-level BLEU indicates fluent English and French generation, with Europarl Bilingual-GAN scoring much higher than the baseline.The authors note lower Multi30k BLEU scores because its test set is smaller.
- Text generation: Forward perplexity analysis shows that lower perplexity for synthetic Europarl samples can coincide with reduced diversity relative to the training set.Models with lower forward perplexities tend to have higher reverse perplexities; this trend is not observed for Multi30k.
- Human evaluation: Human evaluation finds some parallelism even in unsupervised generation, while supervised models produce more parallel sentences.Bilingual judges rated fluency and parallelism on separate five-point scales.
5 Conclusion
Bilingual-GAN frames neural machine translation and text generation as a joint bilingual problem, using a shared space to generate fluent, parallel sentences in two languages. The authors present this as a step toward modeling bilingual conceptual generation.
- Bilingual-GAN models neural machine translation and neural text generation as a joint problem from a bilingual perspective.
- The model provides a principled way to generate from a shared latent space.
- Quantitative and qualitative evaluations show that the model generates fluent sentences exhibiting parallelism in both target languages.
- Future work targets better sentence quality, greater parallelism without parallel NMT training data, and additional languages.