Source-linked AI summary
A Mutual Information Maximization Perspective of Language Representation Learning
Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling, Lei Yu, Zihang Dai, Dani Yogatama
TL;DR
Language representation methods lack a unified account connecting classical embeddings, contextual encoders, and self-supervised learning objectives. The paper recasts these methods as mutual-information lower-bound optimization, develops a framework for new objectives, and demonstrates a sentence-to-n-gram task whose combination with masked language modeling improves results, especially on selected tasks.
Problem
The paper addresses the need for a unified perspective on classical word embeddings, contextual representations, and self-supervised objectives.
Method
The paper formulates Skip-gram, BERT, and XLNet as InfoNCE-based mutual-information objectives and constructs a global-sentence-to-n-gram objective combined with masked language modeling.
Results
The combined objective performs better overall on GLUE and SQuAD, particularly on question answering and linguistic acceptability tasks.
Takeaways & Limitations
The framework provides a principled basis for understanding, designing, and combining self-supervised language representation objectives.
Takeaways & Limitations
Exhaustive comparisons with related span-based methods such as SpanBERT and MASS are left for future work.
Abstract
from arXiv · showhide
We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).
1 INTRODUCTION
The paper reframes language representation learning as mutual-information maximization, unifying classical and contextual methods. This framework also supports principled construction and combination of self-supervised objectives.
- Representation learning methods maximize a lower bound on mutual information between different parts of a word sequence.
- Contrastive learning divides inputs into multiple views and maximizes mutual information between their encoded representations using other inputs as negatives.
- Skip-gram, masked language modeling, and permutation language modeling fit within the same mutual-information framework.
- The framework provides a basis for designing and combining self-supervised objectives for improved language representations.
- A proposed sentence-to-n-gram objective, combined with masked language modeling, performs better particularly on question answering and linguistic acceptability tasks.
2 MUTUAL INFORMATION MAXIMIZATION
Mutual information measures dependence between two views, but direct optimization is generally intractable for neural encoders. The paper therefore uses InfoNCE, a negative-sampling lower bound that connects contrastive learning with cross-entropy.
- Mutual information quantifies how much knowing one random variable reduces uncertainty about another.
- The training objective learns parameters of a function that maximize mutual information between different views of input data.
- Direct mutual-information maximization is generally intractable for neural-network encoders, motivating optimization of the InfoNCE lower bound.
- InfoNCE uses one positive sample and negative samples to approximate the partition function, with a tighter bound when more negative samples are included.
- InfoNCE is related to cross-entropy and is also known as contrastive learning.
3 MODELS
Skip-gram, BERT, and XLNet can all be expressed as InfoNCE objectives over different sentence views, sampling schemes, and encoder architectures. Their main distinctions lie in how views are formed and how context is encoded.
- Unified formulation: Skip-gram, BERT, and XLNet are instances of InfoNCE for learning representations from different parts of a sequence.
- Skip-gram: Skip-gram predicts a context word from an input word using embedding lookup functions and either full-vocabulary softmax or negative sampling.
- BERT: BERT masks tokens and predicts the original word from the perturbed sequence using a Transformer hidden state and the full vocabulary.
- Contextual vs. non-contextual: Skip-gram and BERT optimize similar objectives, but BERT uses a context-dependent Transformer while Skip-gram uses a word-embedding lookup.
- XLNet: XLNet considers all factorization permutations and implements permutation language modeling through attention masking and two-stream self-attention.
4 INFOWORD
INFOWORD extends the mutual-information framework by combining sentence/span representation learning with masked language modeling, while connecting language objectives to methods from other domains. Its design uses contrastive learning over masked sequences and n-grams, with a weighted objective balancing the two components.
- Unifying framework: Skip-gram, BERT, and XLNet are shown to be different instances of InfoNCE within a mutual-information maximization framework.The models differ in their sentence views, data distributions, and encoder architectures.
- Cross-domain motivation: The framework draws on mutual-information methods from computer vision, audio processing, and reinforcement learning to design language representation objectives.The paper specifically discusses Deep InfoMax as inspiration for learning from sequences.
- Global and local representations: The proposed task maximizes mutual information between a global sentence representation and local word or n-gram representations.The global representation is the first token’s contextual hidden state, while local representations are encoded words or n-grams.
- Global and local representations: Using a masked sequence avoids a trivial task that would arise if the global representation were computed from all words, including the target span.For an n-gram target, the input view masks positions i through j and the second view is the corresponding n-gram.
- INFOWORD objective: INFOWORD combines the sentence/span objective with masked language modeling to improve both sentence and word representations.Its masked language modeling term uses negative samples drawn from the unigram distribution instead of the full vocabulary.
- INFOWORD objective: The overall INFOWORD objective is a weighted combination of the masked language modeling and deep-information-maximization terms.λ_MLM and λ_DIM control the contribution of the two terms.
5 EXPERIMENTS
The experiments compare BERT, BERT-NCE, and INFOWORD across GLUE and SQuAD using BERT-like Transformer architectures. INFOWORD adds IDIM, a self-supervised objective for sentence and span representations, and consistently outperforms BERT-NCE, especially with fewer training examples.
- Experimental setup: The evaluation compares three models—BERT, BERT-NCE, and INFOWORD—using BERTBASE- and BERTLARGE-like Transformer architectures.BERT-NCE removes next sentence prediction and uses negative sampling, while INFOWORD adds IDIM to BERT-NCE.
- Experimental setup: The models are evaluated on GLUE classification tasks and SQuAD 1.1 span-based reading comprehension.Each dataset uses a task-specific decoder and fine-tuning of the pretrained models.
- Results: INFOWORD improves overall GLUE and SQuAD results over BERT-NCE, particularly on question answering and linguistic acceptability tasks involving longer phrases.The improvement is attributed to adding IDIM to the training objective.
- Results: INFOWORD consistently outperforms BERT-NCE on SQuAD development-set experiments, with the largest performance gap when the dataset is smallest.The comparison varies both the percentage of training examples and λDIM.
- Discussion: The paper relates IDIM to span-based representation learning and notes that exhaustive comparisons with SpanBERT and MASS remain future work.INFOWORD is described as simpler to train and as exhibiting similar trends to SpanBERT.
- Discussion: The framework suggests exploring higher-order or skip n-grams, syntactic and semantic parses, and better negative samples as future directions.These directions follow from extending the structured views used in contrastive learning.
6 CONCLUSION
The conclusion presents mutual information maximization as a unifying perspective on language representation learning and uses it to construct and evaluate a new self-supervised task.
- Conclusion: The paper analyzes state-of-the-art language representation learning methods from a mutual information maximization perspective.This perspective connects classical and modern word representation models with methods from other representation-learning domains.
- Conclusion: The framework constructs a self-supervised task that maximizes mutual information between global sentence representations and local sentence representations.The task is evaluated through experiments on GLUE and SQuAD.
A NEXT SENTENCE PREDICTION
The next sentence prediction objective in BERT can be viewed as contrastive learning, with negative sentences used to distinguish consecutive from non-consecutive sentence pairs.
- Next sentence prediction: BERT next sentence prediction classifies whether two sentences are consecutive, using randomly sampled second sentences as negative examples.The negative second sentence is sampled from the corpus 50% of the time.
- Next sentence prediction: The objective uses a discriminator over encoded representations of concatenated sentence pairs.The discriminator returns a score for the pair and is parameterized by φ.
- Next sentence prediction: BERT’s binary-classifier formulation is described as local Noise Contrastive Estimation because it distinguishes real sentence pairs from noisy samples.Summing over all possible negative sentences is intractable, motivating the classifier approximation.
- Next sentence prediction: InfoNCE provides a global NCE alternative by sampling negative sentences and combining them with the positive sentence.The encoder representation is obtained from the final hidden state of the first token in the concatenated sequence.
B HYPERPARAMETERS
The experiments use fixed optimization settings for pretraining and tune selected learning rates and batch sizes separately for GLUE and SQuAD.
- Pretraining: Pretraining uses Adam with batch size 1024, maximum sequence length 512, 400,000 steps, 18,000 warmup steps, and weight decay 0.01.The BASE and LARGE variants use learning rates 4e−4 and 1e−4, respectively.
- Pretraining: The objective weights set λMLM to 1.0 and tune λDIM over {0.4, 0.6, 0.8, 1.0}.These settings apply to the pretraining variants described in the experiments.
- GLUE: GLUE fine-tuning uses maximum sequence length 128, selects learning rates and batch sizes on development sets, and trains CoLA for 4 epochs and other tasks for 10.Each hyperparameter configuration is run five times before test evaluation.
- SQuAD: SQuAD fine-tuning uses maximum sequence length 512, four epochs, and development-set selection among five learning rates and two batch sizes.The selected settings are chosen separately for SQuAD.
C QUESTION ANSWERING DECODER
The question-answering decoder predicts answer spans by scoring context tokens as possible start and end positions. It computes end probabilities analogously and selects the span with the highest product of start and end probabilities.
- The decoder encodes each token in a context paragraph of length M and uses these representations for span prediction.
- The start probability assigns a score to each context token using its encoded representation and wstart.
- Two parameter vectors, wstart and wend, model the start and end positions of the answer span.
- The end-index probability is computed analogously using wend, and the predicted answer is the span maximizing the product of start and end probabilities.