Source-linked AI summary
An Unsupervised Sentence Embedding Method by Mutual Information Maximization
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, Lidong Bing
TL;DR
BERT is inefficient for large-scale sentence-pair tasks, while SBERT depends on labeled sentence-pair data that may not exist in new domains. The paper introduces IS-BERT, which combines a lightweight BERT-based encoder with mutual-information-maximizing self-supervision. IS-BERT outperforms unsupervised baselines, surpasses SBERT without in-domain labels, and is competitive with supervised methods in various tasks.
Problem
BERT requires combinatorially many sentence-pair evaluations, while SBERT’s reliance on high-quality labeled sentence pairs limits use in label-scarce or domain-shifted settings.
Method
IS-BERT adds a lightweight feature extractor above BERT and maximizes mutual information between global sentence embeddings and local context embeddings using self-supervised contrastive learning.
Results
IS-BERT significantly outperforms unsupervised baselines, substantially outperforms SBERT without task-specific labeled data, and is competitive with supervised methods across various tasks.
Takeaways & Limitations
IS-BERT can be trained directly on task-specific raw text, supporting sentence embedding in domains without available labeled data.
Takeaways & Limitations
Prior supervised sentence embedding models can be problematic to port when high-quality target-domain labels are unavailable or text distributions differ significantly from NLI data.
Abstract
from arXiv · showhide
BERT is inefficient for sentence-pair tasks such as clustering or semantic search as it needs to evaluate combinatorially many sentence pairs which is very time-consuming. Sentence BERT (SBERT) attempted to solve this challenge by learning semantically meaningful representations of single sentences, such that similarity comparison can be easily accessed. However, SBERT is trained on corpus with high-quality labeled sentence pairs, which limits its application to tasks where labeled data is extremely scarce. In this paper, we propose a lightweight extension on top of BERT and a novel self-supervised learning objective based on mutual information maximization strategies to derive meaningful sentence embeddings in an unsupervised manner. Unlike SBERT, our method is not restricted by the availability of labeled data, such that it can be applied on different domain-specific corpus. Experimental results show that the proposed method significantly outperforms other unsupervised sentence embedding baselines on common semantic textual similarity (STS) tasks and downstream supervised tasks. It also outperforms SBERT in a setting where in-domain labeled data is not available, and achieves performance competitive with supervised methods on various tasks.
1 Introduction
BERT is computationally inefficient for sentence-pair tasks, motivating sentence embeddings that support direct similarity comparison. IS-BERT addresses limited labeled-data availability with a lightweight BERT extension and mutual-information-based self-supervision.
- Motivation: BERT requires combinatorially many sentence-pair evaluations, making similarity search over large collections computationally expensive.Finding the most similar pair among 10k sentences requires about 50 million inference computations.
- Motivation: SBERT improves sentence-pair efficiency by learning semantically meaningful fixed-size sentence representations, but its standard pooling alternatives produce unsatisfactory embeddings.SBERT further fine-tunes BERT for sentence embeddings after averaging token representations or using the [CLS] output proved unsatisfactory.
- Motivation: Unsupervised sentence representation learning is needed for unlabeled or partially labeled datasets, yet prior context-prediction methods trail supervised systems on many tasks.Skip-thought and FastSent use contextual prediction as self-supervision, but their performance remains far behind supervised methods.
- Proposed method: IS-BERT adds a lightweight feature extractor above BERT and trains sentence embeddings by maximizing mutual information between global sentence representations and local contexts.CNN layers with mean-over-time pooling produce global embeddings, while local contexts from other sentences serve as contrastive negatives.
- Evaluation and results: IS-BERT is evaluated on Semantic Textual Similarity and SentEval tasks, including comparisons with unsupervised baselines and SBERT without task-specific labeled data.The introduction reports significant gains over unsupervised baselines and substantially better performance than SBERT when task-specific labels are unavailable.
2 Related Work
Sentence embedding research includes unsupervised methods trained on unlabeled text and supervised methods trained on labeled sentence pairs. Domain shift and unavailable high-quality labels motivate unsupervised approaches, including mutual-information-based representation learning.
- Research directions: Sentence embedding methods mainly divide into unsupervised approaches using unlabeled sentences and supervised approaches using labeled sentences.Some methods combine both types of data.
- Unsupervised sentence embedding: Unsupervised methods for unordered sentences learn from internal sentence structure using models such as recursive and denoising auto-encoders or paragraph vectors.These methods operate on individual sentences rather than sentence order.
- Unsupervised sentence embedding: Methods for ordered sentences use the distributional hypothesis that sentences have similar semantics to their contexts.They predict contextual information or distinguish target sentences from contrastive alternatives.
- Supervised sentence embedding: Supervised sentence embedding models use labeled datasets such as SNLI and Multi-Genre NLI to train architectures including siamese networks and transformers.Examples include InferSent, Universal Sentence Encoder, and SBERT.
- Transfer limitations: Supervised models can be difficult to transfer to new domains when high-quality labels are unavailable or the target text distribution differs substantially from NLI data.These conditions motivate methods that learn sentence representations without labels.
- Mutual information: Mutual-information-based unsupervised representation learning connects to longstanding principles and reconstruction-based generative models, although mutual information is historically difficult to compute.The cited foundations include the informax principle, ICA, auto-encoders, and GANs.
3 Model
IS-BERT learns sentence embeddings by combining BERT with multi-scale CNN token representations and maximizing mutual information between global sentence and local token representations. A discriminator-based Jensen–Shannon estimator trains the encoder so sentence embeddings retain information shared across local contexts while distinguishing sentences.
- Model Architecture: IS-BERT encodes sentences with BERT, then applies parallel CNNs using different window sizes to capture local n-gram dependencies.The resulting token representations concatenate features from multiple contextual ranges.
- MI Maximization Learning: The discriminator receives global–local representation pairs and distinguishes pairs originating from the same sentence from negative pairs.Within a batch, local contexts from the same sentence are positives, while contexts from other sentences provide negatives.
- Model Architecture: Mean-over-time pooling converts the local token representations into a fixed-dimensional global sentence embedding.Pooling introduces no additional parameters, so sentence and token representations share the encoder parameters.
- MI Maximization Learning: The learning objective maximizes mutual information between each global sentence representation and its local token representations using a Jensen–Shannon estimator.The estimator uses a neural discriminator that scores pairs of global and local representations.
- MI Maximization Learning: Maximizing the estimator encourages sentence embeddings to capture information shared across local segments of a sentence but distinct from other sentences.This objective is intended to produce expressive sentence representations without labeled data.
4 Experiment
The experiments evaluate IS-BERT on unsupervised STS, domain-specific AFS, supervised SentEval tasks, and supervised STSb, comparing it with unsupervised and supervised baselines. Results show strong unsupervised performance, domain flexibility, and benefits from self-supervised pretraining before fine-tuning.
- Evaluation setup: The evaluation covers unsupervised STS, domain-specific AFS, supervised SentEval tasks, and supervised STSb.STS measures sentence-similarity prediction, while SentEval measures sentence-embedding effectiveness on supervised downstream tasks.
- Evaluation setup: The STS comparisons use cosine similarity between sentence embeddings, avoiding BERT’s time-consuming regression evaluation for sentence pairs.Table 1 reports Spearman rank correlation ρ between embedding cosine similarity and gold STS labels, with ρ ×100 reported.
- Unsupervised STS: IS-BERT-NLI significantly outperforms other unsupervised baselines on average and surpasses InferSent in 5 of 7 STS tasks.It remains generally below USE and SBERT, which use labeled NLI data, but is comparable to them on STS13 and STS15.
- Argument Facet Similarity: IS-BERT-AFS clearly outperforms other models without task-specific labeled data, while SBERT-NLI and InferSent perform worst on the AFS task.The authors attribute this pattern to distributional differences between NLI training data and AFS data.
- SentEval: Supervised methods outperform unsupervised baselines on SentEval, but IS-BERT-task beats other unsupervised baselines on 6 of 7 tasks.IS-BERT-task is also on par with InferSent and USE, both supervised baselines trained on NLI.
- Supervised STS: Adding self-supervised learning before fine-tuning improves IS-BERT-STSb performance by more than 10% over direct fine-tuning.IS-BERT-STSb (ssl+ft) also outperforms BERT and SBERT, whereas direct IS-BERT fine-tuning performs much worse than SBERT.
5 Conclusions
IS-BERT provides unsupervised sentence representations using mutual-information maximization, outperforming unsupervised baselines and remaining competitive with supervised methods in some settings.
- IS-BERT uses a novel mutual-information maximization objective for unsupervised sentence representation learning.
- IS-BERT outperforms all unsupervised sentence-embedding baselines across various tasks.
- IS-BERT is competitive with supervised sentence-embedding methods in certain scenarios.
- IS-BERT can be trained on task-specific corpora without labels, unlike SBERT when the target domain differs substantially from its training data.
- The paper identifies semi-supervised learning and cross-domain transferability as future research directions.