Source-linked AI summary
On the Sentence Embeddings from Pre-trained Language Models
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, Lei Li
TL;DR
Untuned BERT sentence embeddings poorly capture semantic similarity because their semantic information is not fully exploited in an anisotropic, nonsmooth space. The paper analyzes this problem and applies unsupervised normalizing flows to map embeddings into a smooth isotropic Gaussian space, producing gains across semantic similarity benchmarks.
Problem
Untuned BERT sentence embeddings perform poorly on semantic textual similarity, and the paper investigates whether their semantic information is missing or insufficiently exploited.
Method
BERT-flow learns an invertible normalizing-flow mapping that transforms fixed BERT sentence embeddings into a smooth, isotropic Gaussian distribution without external supervision.
Results
BERT-flow consistently improved BERT by up to 12.70 points, averaging 8.16 points in Spearman correlation across 7 semantic textual similarity benchmarks without downstream supervision.
Takeaways & Limitations
The analysis identifies anisotropic, nonsmooth embedding geometry as a source of problematic similarity, while flow-based calibration improves performance and can remedy excessive lexical correlation.
Takeaways & Limitations
Flow calibration is learned over the full target dataset, including train, validation, and test sentences, though it uses no provided labels.
Abstract
from arXiv · showhide
Pre-trained contextual representations like BERT have achieved great success in natural language processing. However, the sentence embeddings from the pre-trained language models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We first reveal the theoretical connection between the masked language model pre-training objective and the semantic similarity task theoretically, and then analyze the BERT sentence embeddings empirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks. The code is available at https://github.com/bohanli/BERT-flow.
1 Introduction
BERT sentence embeddings without fine-tuning perform poorly on semantic similarity despite BERT’s broader NLP success. The paper attributes this gap to underexploited semantic information and proposes unsupervised flow-based calibration.
- BERT sentence embeddings without fine-tuning can underperform even averaged GloVe embeddings, hindering use where labeled data is costly or unavailable.
- The paper asks whether BERT embeddings lack semantic information or whether their semantic information is difficult to exploit without external supervision.
- Theoretical analysis connects BERT pretraining with semantic similarity, while empirical probing finds a nonsmooth, poorly defined sentence-embedding space.
- BERT-flow uses an invertible normalizing-flow mapping to transform BERT sentence embeddings into a smooth, isotropic Gaussian distribution while keeping BERT fixed.
- 12.70 points maximum and 8.16 points average improvement were obtained across 7 semantic textual similarity benchmarks in Spearman correlation without downstream supervision.
- On natural language inference and QNLI, the method outperformed sentence-BERT embeddings and showed superiority without task-specific supervision.
2 Understanding the Sentence Embedding Space of BERT
The paper examines why BERT’s theoretically meaningful contextual embeddings yield weak sentence similarity. It links the failure to anisotropy, frequency bias, sparse regions, and poorly defined areas of the embedding space.
- BERT sentence embeddings lag behind state-of-the-art semantic embeddings and can be less competitive than averaged GloVe on STS-B.
- BERT sentence vectors are commonly formed by averaging contextual embeddings or using the [CLS] embedding, with averaging reported as consistently better.
- BERT pretraining: Masked language modeling predicts masked tokens from corrupted bidirectional contexts, reducing pretraining to conditional token-distribution modeling.
- Semantic connection: Context-word dot products encode co-occurrence statistics, with PMI serving as a mathematical surrogate for word-level semantic similarity.
- Semantic connection: Pretraining can implicitly relate context embeddings through higher-order co-occurrence when different contexts connect to the same words.
- Anisotropic space: Anisotropy concentrates embeddings in a narrow cone, while word frequency further biases their spatial distribution.
- Anisotropic space: High-frequency words cluster densely near the origin, whereas low-frequency words lie farther away and disperse sparsely.
- Anisotropic space: Sparse low-frequency regions create holes where averaged sentence embeddings may land, making semantic meaning poorly defined and similarity problematic.
3 Proposed Method: BERT-flow
BERT-flow calibrates BERT sentence embeddings by learning an invertible mapping to a standard Gaussian latent space without changing BERT. The Gaussian target provides isotropy and full-space density, while flows preserve information and enable unsupervised likelihood training.
- Core approach: BERT-flow uses an invertible mapping from BERT’s embedding space to a standard Gaussian latent space to calibrate sentence representations.The invertibility condition preserves mutual information between embeddings and data examples.
- Gaussian target: Standard Gaussian distributions provide isotropy, with angularly uniform density and samples that can be viewed as uniformly distributed on a unit sphere after normalization.
- Gaussian target: Mapping to an isotropic distribution can flatten the embedding space’s singular spectrum and suppress dominating word-frequency-related directions.
- Gaussian target: The Gaussian density covers the entire real space, avoiding poorly defined “hole” regions in probability.
- Flow model: The method instantiates the mapping with a flow-based generative model, an invertible transformation between latent space Z and observed space U.The observable space consists of extracted BERT sentence embeddings.
- Training: Training maximizes the likelihood of generating BERT sentence embeddings from a standard Gaussian latent variable in a fully unsupervised manner.Only flow parameters are optimized; BERT parameters remain unchanged, and the learned inverse maps embeddings to Gaussian representations without information loss.
4 Experiments
Experiments evaluate BERT-flow on semantic textual similarity and question-answer entailment, with and without NLI supervision. Across these settings, flow-based calibration improves sentence-embedding performance and reduces excessive lexical-similarity effects.
- Semantic Textual Similarity: Experiments cover seven semantic textual similarity benchmarks, using cosine similarity between sentence embeddings and reporting Spearman correlation with human similarity labels.The benchmarks include STS-B, SICK-R, and STS tasks from 2012–2016.
- Semantic Textual Similarity: Without NLI supervision, BERT-flow improves BERTbase and BERTlarge by 5.88 and 8.16 average points, respectively.Learning flows on target datasets generally yields larger improvements than learning flows on NLI, except on SICK-R.
- Semantic Textual Similarity: With NLI supervision, the isotropic embedding space consistently improves SBERT baselines in most cases and outperforms state-of-the-art SBERT/SRoBERTa results by a large margin.The flow parameters remain learned without supervision labels.
- Unsupervised Question-Answer Entailment: BERT-flow consistently improves AUC on the QNLI validation set, with target-dataset flow learning outperforming flow learning on NLI.QNLI evaluates question-answer entailment without task-specific supervision for the sentence embeddings.
- Comparison with Other Embedding Calibration Baselines: Standard normalization improves STS-B but trails nulling top-k singular vectors, while combining both methods yields further gains; BERT-flow still performs much better.The authors attribute the advantage to invertible isotropic mapping without discarding information.
- Semantic Similarity Versus Lexical Similarity: BERT-induced similarity correlates strongly with edit distance, whereas gold-standard and flow-induced similarities show less evident lexical correlation.For sentence pairs with edit distance ≤4, BERT’s excessive lexical proximity is especially pronounced; flow suppresses this influence.
5 Conclusion and Future Work
The paper identifies deficiencies in BERT sentence embeddings for semantic textual similarity and proposes flow-based calibration to improve performance.
- The paper investigates deficiencies in BERT sentence embeddings for semantic textual similarity.
- It proposes a flow-based calibration method to address these deficiencies.
- The authors identify flow-based generative representation learning as a direction for future work.
A Mathematical Formula of the Invertible Mapping
The flow model is composed of stacked invertible transformations, including additive coupling layers with explicitly computable inverses.
- A flow-based model stacks multiple invertible transformation layers as f = f1 ◦ f2 ◦ . . . ◦ fK.
- Each transformation in the approach uses an additive coupling layer.
- The coupling function gψ can use a deep neural network to increase expressiveness.
- The inverse mapping is explicitly written, supporting invertibility of the transformation.
B Implementation Details
Implementation uses the official TensorFlow BERT codebase and flow-specific training settings, with sequence length limited to 64 and hyperparameters tuned only initially.
- The implementation adopts the official TensorFlow BERT codebase and clips maximum sequence length to 64.
- The flow models train for 1 epoch on target datasets and 0.15 epoch on NLI datasets using Adam.
- Hyperparameters are tuned on the STS-B development set before being reused across other datasets.
- Table 7 reports the flow hyperparameters.
C Results with Different Random Seeds
The random-seed analysis reports variability in NLI fine-tuning but finds that the proposed flow-based method consistently improves results.
- The experiment uses 5 runs with different random seeds in the NLI-supervised STS-B setting.
- Results with standard deviation and median are reported in Table 8.
- Despite non-negligible NLI-finetuning variance, the flow-based method consistently improves performance.
- Table 8 presents results obtained with different random seeds.