Source-linked AI summary
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao
TL;DR
Existing Transformer-based language models lack an explicit compact latent space for abstractly controlling generation and organizing sentence representations. Optimus pre-trains a universal sentence latent space with variational auto-encoder objectives and fine-tunes it for understanding and generation, reporting broad task improvements and new state-of-the-art VAE language-modeling performance. The model also supports guided generation and more adaptable representations, while remaining under-trained and limited to moderate-length sentence control.
Problem
Transformer-based language models lack explicit compact latent structures for abstract control, while earlier VAE models have limited capacity and suffer from KL vanishing.
Method
Optimus pre-trains a large-scale sentence-level variational auto-encoder on a large text corpus to create a universal latent space, then supports language understanding and generation.
Results
Optimus reports lower perplexity than GPT-2 on standard benchmarks, strong guided-generation performance, improved BERT feature-based understanding, and new state-of-the-art VAE language-modeling results.
Takeaways & Limitations
Pre-training a meaningful latent space can reduce KL vanishing and provide more structured, adaptable sentence representations with abstract generation control.
Takeaways & Limitations
The pre-trained language VAE remains under-trained, and current control is limited to sentences of moderate length.
Abstract
from arXiv · showhide
When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model, Optimus. A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks. We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.
1 Introduction
Existing pre-trained language models support understanding or generation but lack explicit compact latent structure for abstract control. Optimus pre-trains a universal sentence latent space to combine these capabilities and improve adaptation across language tasks.
- Transformer-based models improve many NLP tasks but lack explicit compact latent structures for abstract control of language generation and representation.
- VAEs represent sentences in low-dimensional spaces that support feature regularization, vector manipulation, and guided generation.Latent variables govern lower-level word-by-word generation.
- Optimus is a large-scale pre-trained deep latent-variable language model that organizes sentences in a universal latent space.It is pre-trained with sentence-level variational auto-encoder objectives on a large text corpus.
- Optimus combines VAE, BERT, and GPT strengths, supporting language understanding and generation while learning more structured and adaptable representations than BERT.Its latent vectors also support control of high-level semantics in generation, unlike GPT-2's limited abstract guidance.
- The work contributes latent-vector injection schemes, a practical BERT/GPT-2 integration recipe, and evidence that large-scale pre-training reduces KL vanishing.It also reports improved performance across several language tasks and four VAE language-modeling datasets.
2 Related Work
Earlier sentence representation and language VAE approaches had limited capacity or failed to learn smooth, interpretable latent spaces. Optimus addresses this gap with a large Transformer-scale VAE and large-corpus latent-space pre-training.
- Earlier sequence encoders and related models generally did not learn smooth, interpretable feature spaces for sentence encoding or novel sentence generation.
- Prior language VAE work had limited modeling capacity and empirical performance, partly because of KL vanishing.Existing mitigation efforts included KL schedules, decoder changes, and related techniques.
- Earlier language VAE efforts used simple LSTM or shallow Transformer architectures with limited capacity.
- Optimus is presented as the first large VAE model at the scale of BERT and GPT-2, using large-corpus pre-training to reduce KL vanishing and achieve new state-of-the-art performance.
3 Background on NLMs & GPT-2
Neural language models generate each token from preceding tokens and can produce realistic text with low perplexity. However, this autoregressive dependency offers limited guidance from higher-level structures such as tense, topics, and sentiment.
- Neural language models generate a sequence token by token, conditioning each token on all preceding word tokens.Their conditional distributions are typically trained by maximum likelihood estimation.
- GPT-2 uses Transformers for these autoregressive conditionals and learns from a huge OpenWeb text corpus.GPT-2 produces realistic text and low perplexity on several benchmarks.
- GPT-3 extends the same neural language-modeling approach to 175 billion parameters and reports impressive few-shot performance across multiple language tasks.
- Because variation is modeled only through stepwise conditionals, NLMs provide limited guidance from higher-level structures such as tense, topics, or sentiment.
4 Pre-trained Latent Space Modeling
OPTIMUS pre-trains a sentence-level latent space with VAE objectives, combining BERT-based encoding and GPT-2-based generation. Its design addresses latent-space organization, controllable decoding, and the challenge of preventing KL vanishing during large-scale pre-training.
- Latent Space Modeling: OPTIMUS organizes sentences in a universal latent space that captures high-level semantics and guides symbolic sentence generation.The model supports bidirectional mapping between latent and symbolic spaces through inference and generation components.
- Latent Space Modeling: The VAE decoder samples z from a prior and generates x conditionally, separating high-level semantic decisions from lower-level lexical and syntactic generation.This hierarchical procedure makes the dependence on the latent vector explicit.
- Pre-training Objectives: OPTIMUS trains with a β-weighted objective combining reconstruction error and KL regularization, where β controls the compression–information trade-off.β=0 yields an AE variant, while β>0 yields the VAE variant with a smoother latent space.
- Model Architectures: OPTIMUS initializes its encoder and decoder from BERT and GPT-2, respectively, using sentence-level reconstruction to combine language understanding and generation.The encoder uses WPE tokenization and the decoder uses BPE to compute reconstruction loss.
- Model Architectures: The sentence representation is formed from BERT’s [CLS] state and projected into a latent vector, which GPT-2 receives through memory or embedding injection schemes.Memory supplies layer-specific vectors that GPT-2 attends to, while embedding injection adds the latent vector to token embeddings.
- Learning Procedures: Large-scale VAE pre-training is designed to address KL vanishing, in which the decoder ignores z and the model reduces to a simpler neural language model.The paper frames the objective as practicing the information bottleneck principle through a trade-off between predictive power and compactness.
5 Experimental Results
Experiments evaluate OPTIMUS across language modeling, latent-space generation, dialog and stylized response generation, and low-resource understanding. Across these settings, the model combines competitive generation with structured latent representations and controllability.
- Language Modeling: OPTIMUS achieves lower perplexity than existing VAE methods while obtaining higher mutual information and active-unit scores.These results indicate improved language modeling and representation learning, with only one epoch of dataset-specific fine-tuning.
- Language Modeling: OPTIMUS achieves lower perplexity than GPT-2 on three of four datasets, with a larger gap on SNLI.The paper attributes this pattern to prior language knowledge encoded in the latent variable, especially where the data exhibit common regularities.
- Guided Language Generation: OPTIMUS supports sentence transfer through latent arithmetic, generating xD from zD = zB − zA + zC and preserving semantic transformations between sentence pairs.Examples show changes such as singular-to-plural revision and shifts from daily-life topics to sports.
- Guided Language Generation: OPTIMUS achieves higher BLEU scores than all existing baselines for dialog response generation.The evaluation uses DailyDialog and reports precision, recall, and F1 alongside BLEU.
- Guided Language Generation: OPTIMUS achieves better performance on all metrics for stylized response generation and the best performance on all metrics for label-conditional Yelp generation.The Yelp evaluation measures content, style, and diversity through BLEU, accuracy, G-score, and Self-BLEU.
- Low-Resource Language Understanding: OPTIMUS yields better classification performance and faster adaptation than BERT in low-resource language understanding.The feature-based approach requires fewer computing resources because it updates only the classifier, and it adapts better when labeled data are scarce.
6 Discussion
OPTIMUS combines VAE, BERT, and GPT-2 capabilities through a smooth, universal latent space for language understanding and generation. The paper also identifies under-training and limited control over long sentences as current boundaries.
- OPTIMUS combines VAE, BERT, and GPT-2 strengths in a large-scale pre-trained latent-variable model for natural language.
- Its universal latent space organizes sentences and supports both language understanding and generation tasks.
- Current OPTIMUS is under-trained and has difficulty controlling long sentences because compact latent vectors restrict encoded information.
- OPTIMUS is intended to make deep generative models more practical for NLP and renew interest in them during the pre-training era.
- The VAE objective connects reconstruction and KL terms to mutual-information optimization and prior fitting, controlled by β.
B Pre-training Details
The pre-training study compares three ways to inject a sentence latent vector into GPT-2 for guided generation. Memory-token injection transfers information more efficiently than adding the vector as an embedding.
- The study compares latent-vector injection as an additional memory token, an additional embedding, or both combined.
- Lower reconstruction error per word indicates more effective information transfer from the encoder to the decoder.
- Memory-token injection is significantly more efficient than using the latent vector as an additional embedding on Yelp and PTB.
- The combined Mem+Emb scheme yields slightly better performance than the individual injection schemes.
B.2 Wikipedia Dataset
Wikipedia is pre-processed into natural sentences with a maximum sequence length of 64 for latent-space pre-training. This produces 1,990K sentences covering most of the dataset.
- Wikipedia is converted into natural sentences rather than fixed-length text sequences for latent-space modeling.
- 1,990K sentences, or 96.45% of the entire Wikipedia dataset, remain after applying the maximum sequence length of 64.
C.1 Language Modeling
The experiments evaluate language modeling, latent representations, dialog and style generation, and label-conditional generation. They use reconstruction and likelihood metrics alongside measures of latent-space activity, information, quality, and controllability.
- Language Modeling: Figure 5 compares Mem, Emb, and Mem+Emb latent injection schemes on Yelp and PTB using reconstruction error per word.
- Language Modeling: Language-modeling evaluation uses ELBO, perplexity, and importance-weighted perplexity, where higher ELBO and lower PPL indicate better sentence fit.
- Language Modeling: Latent representations are assessed with active units, mutual information, and posterior-prior KL divergence.
- Dialog Response Generation: SpaceFusion interpolates latent samples between context and response representations to support dialog response generation.
- Stylized Response Generation: StyleFusion interpolates a style representation with an autoencoder representation and adds the resulting biased-target loss to dialog training.
- Label-Conditional Text Generation: Label-conditional generation fine-tunes and freezes the VAE, then trains a conditional GAN to generate latent vectors from noise and labels.
- Label-Conditional Text Generation: The conditional generator produces sentence outlines in latent space while the decoder fills in their details.
- Label-Conditional Text Generation: The label-conditional evaluation reports BLEU, attribute Accuracy, and G-score, with G-score defined as their geometric mean and treated as most important.
C.3 Latent space interpolation & arithmetic operation
OPTIMUS organizes sentences in a universal latent space that supports arithmetic transformations and smooth interpolation, while remaining limited for long or complex sentences.
- Arithmetic operation: OPTIMUS encodes source and target sentences, applies z_D = z_B − z_A + z_C, and decodes an output sentence reflecting the source-to-target transition.Observed transfers include singular-to-plural subjects and daily-life-to-sport topic changes.
- Latent space interpolation: Interpolating z_τ = z_1·(1−τ)+z_2·τ with τ from 0 to 1 produces sentences with smooth semantic evolution.The paper reports smooth and meaningful interpolation for almost arbitrary input sentence pairs.
- Limitations: The compact latent vectors have difficulty representing long or complex sentences because they restrict the amount of encoded information.The paper suggests more sophisticated latent-space designs as a possible improvement.
- Interactive use: OPTIMUS provides a demo for controllable sentence generation using arithmetic and interpolation operations.
C.4 Ablation study on VAE & AE objectives
The ablation study compares VAE and AE objectives, finding smoother interpolation with VAE and better GLUE performance than AE under the feature-based approach.
- Interpolation: VAE produces smoother sentence interpolation results than AE.
- GLUE classification: Both pre-training objectives outperform BERT on large datasets, while the VAE objective performs better than the AE objective on GLUE.The comparison uses a feature-based approach.
- Interpretation: The study attributes the classification advantage to smooth regularization of the latent space.