PaperlayerView exact revision on arXiv

Source-linked AI summary

AnglE-optimized Text Embeddings

Xianming Li, Jing Li

arXiv:2309.12871v9cs.CLcs.AIcs.LG

TL;DR

Existing STS embedding models commonly rely on cosine objectives whose saturation zones can cause vanishing gradients and hinder learning. AnglE adds angle optimization in complex space and evaluates it across short- and long-text STS, including limited-label settings. It outperforms baseline STS models across these scenarios and supports effective embeddings for varied applications.

  • Problem

    Cosine-based supervised STS objectives have saturation zones that can cause gradient vanishing and hinder learning subtle distinctions between texts.

  • Method

    AnglE divides embeddings into real and imaginary parts in complex space and optimizes their angle difference alongside cosine similarity.

  • Results

    AnglE outperforms SOTA STS models on transfer and non-transfer tasks across short- and long-text datasets, with 73.55% average Spearman correlation versus 68.03% for SBERT in non-transfer tasks.

  • Takeaways & Limitations

    Angle optimization produces high-quality embeddings that work across short- and long-text STS and varied scenarios, including LLM-supervised settings.

Abstract

from arXiv · show

High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.

1 INTRODUCTION

The paper identifies cosine saturation as a gradient-vanishing problem for supervised STS and proposes AnglE, which adds complex-space angle optimization. It broadens evaluation with long-text data and reports stronger performance across STS settings.

  • Cosine saturation zones can produce near-zero gradients, hindering optimization and subtle distinctions between texts.Binary similarity labels in datasets such as MRPC and QQP can fall within these saturation zones.
  • AnglE mitigates cosine saturation by optimizing both cosine similarity and angle differences in a complex representation space.Embeddings are divided into real and imaginary parts, and the normalized angle difference becomes an optimization objective.
  • The paper introduces a roughly 21K-sample long-text STS dataset from GitHub Issues, using duplicate issues as positives and non-duplicates as negatives.This addresses the short-text focus of existing STS benchmarks.
  • AnglE outperforms SOTA STS models on both transfer and non-transfer tasks across short- and long-text datasets.Its non-transfer average Spearman correlation is 73.55%, compared with 68.03% for SBERT.
  • Extensive experiments show that AnglE improves text embedding quality across varied STS scenarios, including settings with limited domain labels and LLM-annotated data.The paper also reports benefits for downstream retrieval applications.

2 RELATED WORK

Prior work includes unsupervised and supervised approaches for improving text embeddings, but most existing models optimize cosine similarity without addressing its saturation zones. AnglE is presented to address this gap.

  • Existing text-embedding research includes unsupervised augmentation, flow-based mapping, whitening, and supervised learning with NLI or STS data.
  • Most existing models optimize cosine similarity while neglecting the negative effect of cosine saturation zones.
  • The paper proposes an angle-optimized text embedding model to improve embedding quality.

3 METHODOLOGY

AnglE combines cosine similarity, in-batch negative, and complex-space angle objectives to optimize text representations. Its angle objective addresses cosine saturation by optimizing normalized angle differences between similarity-ranked pairs.

  • 3.2 COSINE OBJECTIVE: The cosine objective optimizes representation similarity so high-similarity pairs exceed low-similarity pairs.It uses a temperature hyperparameter and cosine similarity between representations.
  • 3.3 IN-BATCH NEGATIVE OBJECTIVE: The in-batch negative objective uses supervised positive pairs and batch negatives to improve generalization through data augmentation.The method accounts for potentially identical sentences in a batch that are not explicitly labeled as positives.
  • 3.4 ANGLE OBJECTIVE: The angle objective represents paired embeddings in complex space and optimizes their normalized angle difference.Real and imaginary components are obtained by chunking representations, and complex division provides the angle difference.
  • 3.4 ANGLE OBJECTIVE: Complex-space angle optimization preserves a distinct optimization signal when cosine differences approach zero in saturation zones.The method computes angle differences from complex division and minimizes them for higher-similarity pairs relative to lower-similarity pairs.
  • 3 METHODOLOGY: AnglE combines cosine, in-batch negative, and angle objectives into a weighted final loss.The final objective is L = w1 * Lcos + w2 * Libn + w3 * Langle, with constant weights.

4 EXPERIMENT

The experiments evaluate AnglE on short- and long-text STS, transfer and non-transfer settings, ablations, LLM-supervised learning, and embedding distributions. Across these evaluations, AnglE generally outperforms the compared baselines and supports the value of angle optimization.

  • 4.3 MAIN RESULTS: AnglE outperforms prior models in transfer STS, with AnglE-BERT and AnglE-LLaMA gaining 0.80% and 0.72% average score over SimCSE-BERT and SimCSE-LLaMA.The transfer evaluation uses seven STS benchmark datasets after training on MNLI and SNLI.
  • 4.3 MAIN RESULTS: AnglE consistently outperforms SBERT in non-transfer STS, achieving an absolute gain of 5.52% across four short-text and one long-text dataset.The evaluation reports Spearman correlation and includes the GitHub Issues Similarity Dataset.
  • 4.4 ABLATION STUDY: AnglE’s performance drops more without angle optimization than without the in-batch negative objective, indicating that angle optimization contributes more to the improvement.The ablation study evaluates Spearman correlations on the STS-B test set.
  • 4.4 ABLATION STUDY: LLM-supervised AnglE outperforms unsupervised contrastive baselines, with an ensemble of LLMs achieving the best results.The approach uses LLMs as annotators to create pseudo-supervised data for AnglE training.
  • 4.5 DISCUSSION AND ANALYSIS: AnglE’s cosine-similarity distribution more closely matches the gold STS-B distribution than those of SimCSE and SBERT, including patterns in cosine saturation zones.The analysis focuses on sentence-pair similarities in the STS-B test set.

5 CONCLUSION AND FUTURE WORK

The paper presents AnglE as a complex-space angle-optimized embedding model and introduces a long-text STS dataset plus LLM-supervised learning. Experiments report better performance than baselines across short- and long-text STS and varied scenarios.

  • 5 CONCLUSION AND FUTURE WORK: The GitHub Issues Similarity Dataset extends STS evaluation to long texts, while LLM-supervised learning addresses scarce domain-supervised data.The conclusion presents both as extensions supporting broader evaluation and application scenarios.
  • 5 CONCLUSION AND FUTURE WORK: Extensive experiments report that AnglE outperforms baselines and handles both short- and long-text STS tasks across various scenarios.The authors identify real-world application as future work.

A.1 LIST OF OPEN-SOURCE PROJECTS IN GITHUB ISSUE SIMILARITY DATASET

The GitHub Issues Similarity Dataset was collected from open-source projects on GitHub. The appendix lists the repositories used for this collection.

  • A.1 LIST OF OPEN-SOURCE PROJECTS IN GITHUB ISSUE SIMILARITY DATASET: The dataset’s GitHub issues were collected from 55 repositories.The listed projects include widely used open-source software repositories.

A.2 TRANSFER TASK EXPERIMENT

The transfer-task experiment evaluates sentence embeddings on seven downstream task types using accuracy. AnglE achieves the strongest overall transfer performance among the compared models.

  • A.2 TRANSFER TASK EXPERIMENT: AnglE achieves the best average accuracy and 6 of 7 best results in the transfer-task evaluation.The comparison is reported in Table 6 across seven downstream tasks.
  • A.2 TRANSFER TASK EXPERIMENT: AnglE improves 4.34% over DiffCSE and 4.48% over SimCSE in the transfer tasks.The reported metric is accuracy.
Loading 2309.12871v9…