PaperlayerView exact revision on arXiv

Source-linked AI summary

TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Kexin Wang, Nils Reimers, Iryna Gurevych

arXiv:2104.06979v3cs.CL

TL;DR

Sentence-embedding methods often lack labeled data, while STS-centered evaluation leaves their behavior on domain-specific tasks unclear. The paper proposes TSDAE, which trains a Transformer encoder through sequential denoising reconstruction and evaluates it across heterogeneous tasks and settings. TSDAE outperforms other approaches, including MLM, and can match or surpass supervised models in the reported comparisons.

  • Problem

    Labeled data is scarce and expensive, and STS-focused evaluation does not establish performance on domain-specific tasks.

  • Method

    TSDAE uses a modified Transformer encoder-decoder that reconstructs original sentences from corrupted inputs through a fixed-size sentence representation.

  • Results

    TSDAE significantly outperforms other approaches across heterogeneous tasks and settings, including MLM, and can perform on-par with or outperform supervised USE-large.

  • Takeaways & Limitations

    TSDAE is presented as an effective unsupervised method for sentence embeddings, domain adaptation, and pre-training.

  • Takeaways & Limitations

    STS performance does not necessarily correlate with performance on specific downstream tasks, limiting STS as the primary evaluation of unsupervised sentence embeddings.

Abstract

from arXiv · show

Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.

1 Introduction

TSDAE addresses the scarcity and expense of labeled data with an unsupervised Transformer-based sequential denoising auto-encoder. The paper broadens evaluation beyond STS and reports strong results across heterogeneous tasks, including gains of up to 6.4 points and performance comparable to supervised USE-large.

  • Labeled sentence-embedding data is often unavailable or expensive, motivating methods trained on unlabeled corpora.
  • TSDAE encodes corrupted sentences into fixed-sized vectors and trains a decoder to reconstruct the originals, while inference uses only the encoder.
  • STS-only evaluation leaves performance on domain-specific tasks unclear because its datasets lack domain knowledge and do not reliably predict downstream performance.
  • 6.4 points is the reported maximum improvement over previous state-of-the-art unsupervised approaches across three heterogeneous tasks.
  • TSDAE performs on-par with or better than supervised USE-large and works well for domain adaptation and pre-training compared with MLM.
  • The paper compares recent unsupervised methods across varied tasks and domains rather than focusing primarily on STS.

2 Related Work

Prior sentence-embedding work includes supervised and unsupervised Transformer-based objectives, but evaluation has largely centered on STS. The paper therefore emphasizes comparisons across specific tasks and domains, while restricting training methods to those using single sentences.

  • Supervised sentence embeddings learn from labeled sentence relations, including similarity, natural language inference, question answering, and conversational-context datasets.
  • Unsupervised methods train only on unlabeled corpora and include CT, BERT-flow, and SimCSE with distinct objectives.
  • DeCLUTR requires long documents and sentence-level contexts, whereas the considered methods require only independent sentences.
  • STS performance does not necessarily correlate with downstream-task performance, leaving specific-task and domain performance unresolved.

3 Sequential Denoising Auto-Encoder

TSDAE combines denoising auto-encoding with a modified Transformer encoder-decoder that forces reconstruction through a fixed-size sentence representation. The decoder’s restricted access creates a bottleneck intended to make the representation meaningful.

  • 3.1 Training Objective: SDAE corrupts input sentences, encodes them into fixed-sized vectors, and reconstructs the original inputs.
  • 3.1 Training Objective: The training objective maximizes reconstruction likelihood over the training corpus, with damaged sentences supplied to the model.
  • 3.2 TSDAE: Unlike a standard Transformer encoder-decoder, TSDAE’s decoder receives only the fixed-size sentence representation, not all contextualized encoder word embeddings.
  • 3.2 TSDAE: TSDAE confines both cross-attention keys and values to the sentence embedding.
  • 3.2 TSDAE: The reported best configuration uses word deletion at ratio 0.6, the [CLS] output as representation, and tied encoder-decoder parameters.

4 Evaluation

The evaluation broadens unsupervised sentence-embedding assessment beyond STS to heterogeneous-domain retrieval, re-ranking, and paraphrase tasks, while also testing settings with available labeled data.

  • Motivation: Previous approaches were primarily evaluated on STS, whose performance does not correlate with downstream task performance.STS also lacks domain-specific knowledge and uses an artificial score distribution unlike many real-world tasks.
  • Evaluation settings: The study evaluates unsupervised sentence embeddings in unsupervised learning, domain adaptation, and pre-training settings.Domain adaptation combines unlabeled target-task sentences with labeled NLI and STS data in two training orders.
  • Tasks and datasets: Evaluation covers Information Retrieval, Re-Ranking, and Paraphrase Identification across heterogeneous specialized domains and text styles.The datasets include AskUbuntu, CQADupStack, TwitterPara, and SciDocs.
  • Tasks and datasets: AskUbuntu re-ranks 20 candidate questions using MAP, while CQADupStack retrieves duplicate questions from a large pool using MAP@100.AskUbuntu uses BM25 candidates; CQADupStack contains 12 StackExchange forums and trains one model across them.
  • Tasks and datasets: TwitterPara evaluates tweet paraphrase scores with Average Precision, and SciDocs identifies up to 5 relevant paper titles from up to 30 candidates using MAP.SciDocs includes Cite, Co-Cite, Co-Read, and Co-View tasks.
  • Evaluation procedure: Sentence vectors are compared with cosine similarity, and scores across sub-datasets or sub-tasks are averaged.The evaluation uses titles for AskUbuntu, CQADupStack, and SciDocs.

5 Experiments

The experiments compare TSDAE with unsupervised baselines, supervised pretrained models, and in-domain supervised SBERT across the proposed benchmarks.

  • Baseline Methods: TSDAE is compared with CT, SimCSE, BERT-flow, embedding baselines, pretrained models, and in-domain supervised SBERT.Scores are averaged over 5 random seeds, with BERT-base-uncased generally used as the base Transformer.
  • Supervised upper bound: In-domain supervised SBERT uses MNRL for labeled relevant pairs, while TwitterPara uses MSE for labeled relevance scores.MNRL treats labeled pairs as positives and other in-batch combinations as negatives.
  • Baseline Methods: Masked Language Model derives sentence embeddings by mean-pooling output token embeddings after predicting masked input words.The setup uses 100K training steps, batch size 8, and learning rate 5e-5.
  • Contrastive Tension (CT): Contrastive Tension trains two independent encoders with identical sentences as relevant examples and random sentences as irrelevant examples.Its objective uses binary cross-entropy, and CT achieves state-of-the-art unsupervised performance on STS datasets.
  • Baseline Methods: SimCSE creates positive pairs with different dropout masks and uses in-batch negatives through a shared encoder.Its objective is equivalent to applying the MNRL loss after feeding each batch through the encoder twice.
  • Baseline Methods: BERT-flow debiases pooled pretrained sentence embeddings by fitting a trainable flow network to a standard Gaussian distribution.Because it does not update the underlying Transformer, BERT-flow is reported only for unsupervised learning and domain adaptation NLI+STS → target task.

6 Results

TSDAE outperforms prior unsupervised sentence-embedding methods across diverse task settings and also provides effective domain adaptation and pre-training. Results further show that STS performance does not reliably predict performance on domain-specific tasks.

  • Domain Adaptation: TSDAE improves performance when target-domain training precedes labeled NLI+STS training, with an average gain of 1.3 points.All unsupervised methods improve compared with training only on the target domain.
  • Pre-training: TSDAE works well for pre-training, outperforming MLM on all datasets except AskUbuntu and retaining an advantage with larger labeled training sets.CT and SimCSE perform substantially worse as pre-training strategies than TSDAE, MLM, or the pre-trained SBERT-nli-stsb model.
  • Evaluation: Table 2 reports average precision averaged over five random seeds and marks cases where TSDAE beats CT and SimCSE in all five runs.The table compares unsupervised and supervised out-of-the-box approaches, with in-domain USE-large scores separately identified.
  • Unsupervised learning: TSDAE and MLM outperform CT and SimCSE on domain-specific real-world tasks, despite CT and SimCSE performing strongly on STS data.The authors attribute this pattern to differences in input modification and evaluation characteristics.

7 Analysis

The analyses examine data efficiency, pre-training with different corpus sizes, and which word types influence sentence-pair similarity. TSDAE often performs well with limited unlabeled data, while nouns are generally the most influential content words across approaches.

  • Experimental setup: The analysis uses development sets except for TwitterPara, which uses its test set because no development split was released.Hyperparameters are selected up-front without tuning to a particular dataset.
  • 7.1 Influence of Corpus Size: Figure 2 compares TSDAE, MLM, CT, and SimCSE+SBERT as labeled training data increases, using SBERT from the standard BERT-base-uncased checkpoint as a reference.The plotted setup first pre-trains on unlabeled sentences and then applies in-domain supervised training.
  • 7.1 Influence of Corpus Size: TSDAE often outperforms previous unsupervised methods with as few as 1,000 unlabeled sentences.Experiments vary corpus sizes from 128 to 65,536 sentences and train for up to 100K steps.
  • 7.1 Influence of Corpus Size: Approximately 10K unlabeled sentences are generally sufficient to tune pre-trained transformers to a specific domain.Performance usually stagnates at 10K sentences, except for CQADupStack, whose varied forums require broader domain coverage.
  • 7.2 Relevant Content Words: Nouns are by far the most relevant content words, while prepositions and determiners have little influence on model predictions.The result is averaged over four datasets.
  • 7.2 Relevant Content Words: Unsupervised methods show no significant differences from the other approaches in the POS-tag relevance analysis.The analysis identifies the word that most reduces cosine similarity for a relevant sentence pair, then aggregates its POS tag.

8 Discussion

The discussion compares encoder-decoder checkpoints for TSDAE and finds that single encoder checkpoints, particularly BERT, generalize better than heavily autoencoder-pretrained encoder-decoder models.

  • 8 Discussion: Starting TSDAE from BERT reaches a better balance between loss fitting and generalization than training from scratch.The discussion concludes that single encoder checkpoints can use pretrained knowledge while avoiding overfitting.

9 Conclusion

The paper proposes TSDAE and evaluates it across heterogeneous tasks and settings, reporting strong performance while arguing that STS-only evaluation is insufficient.

  • 9 Conclusion: TSDAE is evaluated against recent unsupervised methods on four heterogeneous tasks in unsupervised learning, domain adaptation, and pre-training settings.The evaluation is designed to cover specialized domains rather than only general-domain similarity.
  • 9 Conclusion: TSDAE performs well across the selected tasks and settings, significantly outperforming other approaches.The conclusion summarizes consistent performance across the three evaluation settings.
  • 9 Conclusion: STS-only evaluation is insufficient because strong STS performance does not correlate with performance on specific downstream tasks.The paper therefore treats task- and domain-specific evaluation as necessary for assessing unsupervised sentence embeddings.
  • 9 Conclusion: Deletion noise is selected as the best noise type, with a deletion ratio of 0.6 identified as optimal during configuration search.The experiments compare noise types at a fixed ratio before tuning the deletion ratio.
  • 9 Conclusion: CLS pooling is chosen because it performs similarly to mean pooling while preserving positional information.The final configuration also ties encoder and decoder parameters, improving the validation score to 79.15.

B Experiment Settings

The experiments implement and train TSDAE and comparison systems with specified optimization settings, while avoiding in-domain labeled early stopping in unsupervised experiments.

  • B Experiment Settings: TSDAE, CT, and BERT-flow use 100K training steps and batch size 8, with method-specific optimizers and initial learning rates.The initial learning rates are 3e-5, 1e-5, and 1e-6 for TSDAE, CT, and BERT-flow, respectively.
  • B Experiment Settings: Unsupervised experiments replace in-domain labeled early stopping with a fixed number of training steps.This setting reflects the expense of obtaining labeled data in the intended real-world scenario.
  • B Experiment Settings: In-domain supervised models train for up to 20K steps with batch size 64, cosine similarity, and validation-based early stopping.Scores for in-domain unsupervised and supervised models are averaged over five random seeds.
  • B Experiment Settings: BM25 uses the Elasticsearch implementation with default settings.
  • B Experiment Settings: The study uses documented model checkpoints for its experimental comparisons.The checkpoint configurations are summarized in Table 8.

C Results of Other Checkpoints

Checkpoint choice and supervision requirements materially affect sentence-embedding performance: BERT checkpoints work best for TSDAE, while supervised methods can surpass unsupervised performance with limited annotation.

  • C Results of Other Checkpoints: BERT checkpoints achieve better results across methods, and TSDAE significantly outperforms other methods when using them.The authors hypothesize that BERT’s next sentence prediction pre-training guides learning from sentence-level contexts.
  • C Results of Other Checkpoints: SimCSE retains official early stopping on the STS-B development set, unlike the in-domain evaluation protocol.STS-B is not treated as an in-domain dataset for the task- and domain-specific evaluation.
  • C Results of Other Checkpoints: In-domain supervised approaches significantly outperform unsupervised approaches when sufficient labeled data is available.The paper frames the amount of annotation needed to exceed unsupervised performance as an intersection point.
  • C Results of Other Checkpoints: 140–6k labeled sentence pairs are required for in-domain supervised SBERT to match TSDAE, depending on the dataset.SciDocs requires only 464 labeled examples, while multi-domain CQADupStack and TwitterParaphrase require more data than single-domain AskUbuntu.
  • C Results of Other Checkpoints: 100 annotated tweet pairs take about 20 minutes, so matching TSDAE requires 0.5–20 hours for one annotator.The corresponding estimate is 2.5–100 hours for five crowd annotators.
  • C Results of Other Checkpoints: The reported analyses cover average precision, STS Spearman correlation, corpus-size effects, and POS-tag effects across the selected datasets.The cited figures and tables document these evaluation dimensions without reporting additional outcomes in the supplied passages.
Loading 2104.06979v3…