Source-linked AI summary
Learning Semantic Textual Similarity from Conversations
Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil
TL;DR
The paper addresses sentence-level semantic similarity by learning representations from conversational input-response pairs. It adds supervised NLI through multitask training and reports strong performance across STS and CQA, including the best performance among neural STS models.
Problem
The paper seeks better sentence-level semantic similarity representations by exploiting conversational relationships between inputs and responses.
Method
The method trains an unsupervised input-response model on conversations and optionally shares its encoder with supervised NLI training.
Results
The adapted multitask model achieves the best performance among neural models on STS, with r=0.835 on development data and 0.808 on test data.
Takeaways & Limitations
Conversationally learned representations are competitive on STS and CQA, and multitask training further improves their reported performance.
Takeaways & Limitations
Performance remains sensitive to domain-specific semantic distinctions, and the authors note that more data in those domains could improve model choices.
Abstract
from arXiv · showhide
We present a novel approach to learn representations for sentence-level semantic similarity using conversational data. Our method trains an unsupervised model to predict conversational input-response pairs. The resulting sentence embeddings perform well on the semantic textual similarity (STS) benchmark and SemEval 2017's Community Question Answering (CQA) question similarity subtask. Performance is further improved by introducing multitask training combining the conversational input-response prediction task and a natural language inference task. Extensive experiments show the proposed model achieves the best performance among all neural models on the STS benchmark and is competitive with the state-of-the-art feature engineered and mixed systems in both tasks.
1 Introduction
The paper learns sentence-level semantic similarity from conversational input-response relationships, where similar meanings tend to receive similar responses. It evaluates unsupervised conversational representations and improves them through multitask training with NLI.
- Motivation: Semantically similar inputs tend to have similar response distributions, while similar wording can still correspond to different meanings and responses.For example, age-related questions share responses, whereas “How are you?” and “How old are you?” do not.
- Approach: The proposed unsupervised approach trains models to predict conversational input-response relationships and uses their internal sentence representations for semantic similarity.
- Multitask learning: Multitask training combines conversational response prediction with supervised Natural Language Inference data to augment the learned representations.
- Results: Unsupervised conversational training performs well on STS and CQA question similarity, while adding SNLI yields further improvements and state-of-the-art performance among neural STS models.
2 Approach
The approach frames representation learning as conversational response selection, using shared sentence encoders and a response-scoring network. It compares DAN and Transformer encoders and adds a shared multitask NLI pathway.
- Response prediction: Conversational learning is formulated as selecting the correct response from randomly sampled alternatives given an input.
- Response prediction: Input and response sentences become fixed-length vectors, whose dot product is converted into response probabilities and trained by maximizing correct-response loglikelihood.
- Model architecture: The input-response architecture uses tied encoder parameters, while a feed-forward network transforms response embeddings before scoring.
- Sentence encoders: The DAN averages word and bigram embeddings before a DNN, whereas the Transformer uses self-attention and averages contextualized outputs into a fixed-length vector.
- Sentence encoders: LSTM and BiLSTM encoders performed worse than the Transformer in the reported experiments.
- Multitask learning: Multitask learning shares encoders between conversational response prediction and NLI, providing a complementary signal from mostly nonconversational sentence pairs.
3 Dataset
The dataset consists of large-scale, structured Reddit conversations from which filtered parent-child comment pairs provide conversational input-response training examples.
- Reddit corpus: The Reddit corpus spans 2007–2016 and contains 133 million posts and 2.4 billion comments with reply metadata.
- Reddit corpus: The corpus is treated as suitable for training because its comments are mostly conversational and structurally linked by reply relationships.
- Pair extraction: Parent comments and their replies are extracted as input-response pairs, producing around 600 million pairs after filtering noisy comments.
4 Experiments
Experiments evaluate response prediction and the learned encoders, using Reddit-trained and Reddit+SNLI models under specified optimization and architectural settings.
- Evaluation plan: The experiments first evaluate response prediction with different encoders, then supervised multitask results, and finally STS Benchmark performance.
- Model variants: The Reddit model uses conversational input-response pairs, while the multitask model combines Reddit training with SNLI.
- Training configuration: Reddit training uses SGD with batch size 128 and learning rate 0.01 for 40 million steps, while multitask training uses 30 million steps.
- Encoder configuration: The DAN uses a three-layer DNN with 300, 300, and 500 hidden units, while the Transformer uses six attention layers, eight heads, hidden size 512, and filter size 2048.
4.2 Response Prediction
The response-prediction evaluation compares DAN and transformer encoders using precision at N on Reddit test examples. The transformer performs better across the reported values and is used in the remaining experiments.
- Precision at N evaluates whether the true response ranks among the top N responses against 99 randomly selected negatives.The evaluation reports P@1, P@3, and P@10.
- The transformer encoder outperforms the DAN baseline for every reported value of N.The comparison covers N=1, 3, and 10.
- 65.7% P@1 is achieved by the transformer encoder, compared with 56.1% for DAN.
- The experiments use the transformer encoder thereafter because of its greater performance.
4.3 SNLI
The multitask model is evaluated on SNLI because it shares an encoder between Reddit response prediction and SNLI classification. It reaches 84.1% accuracy, close to InferSent, while jointly optimizing both tasks and improving STS performance.
- The multitask model shares an encoder between Reddit response prediction and SNLI classification, so it is evaluated on SNLI.
- 84.1% SNLI test accuracy is achieved by the multitask model, close to InferSent’s performance.
- Unlike InferSent, the proposed model learns all parameters from scratch, including word embeddings, because the Reddit dataset is sufficiently large.
- Multitask training balances Reddit response prediction and SNLI classification, whereas InferSent optimizes SNLI alone.
- The multitask model performs better on STS, and the authors suspect Reddit data helps prevent overfitting to SNLI and improves transfer performance.
4.4 STS Benchmark
The models learn sentence representations from conversational response prediction and evaluate them on STS using direct and adapted similarity scoring. Multitask training with SNLI produces the strongest neural performance, while STS adaptation further improves results.
- STS evaluates sentence embeddings against human similarity judgments for 8,628 sentence pairs spanning captions, news, and forums.Each pair receives a human-labeled similarity score from 0 to 5.
- The evaluation compares out-of-the-box embeddings with versions adapted using a transformation matrix trained on STS data.Both configurations map scores to the range of human labels.
- The untuned Reddit model is competitive with many neural representation models and keeps semantically similar text close in embedding space.
- 0.782 Pearson’s r is achieved by the out-of-the-box Reddit+SNLI model on the STS test set, surpassing InferSent and approaching CNN (HCTI).Its dev-set score is 0.814.
- 0.808 Pearson’s r is achieved by the adapted multitask model on the STS test set, the best performance among neural models.The corresponding dev-set score is 0.835, and results are competitive with feature-engineered and mixed systems.
- Adding SNLI improves the captions category by about 8% over the Reddit baseline and by 5% after STS tuning.The authors attribute the captions improvement to SNLI’s image-caption content.
4.5 How Much Data for the Supervised Task?
The study tests how much supervised SNLI data is needed to adapt Reddit-trained representations. Performance rises quickly with initial SNLI data and largely plateaus after using 40%.
- The experiments train Reddit-plus-SNLI multitask models with 10% to 90% of the full SNLI dataset.The goal is to estimate the supervised data needed for effective adaptation.
- Pearson’s r increases rapidly when SNLI data is first added across all four evaluated tasks.
- With only 10% of SNLI, captions reach around 0.85 Pearson’s r on both dev and test sets.
- Performance mostly flattens after 40% of SNLI, with only slight improvement from additional data.This indicates that Reddit-trained encoders can be adapted efficiently with a small amount of in-domain data.
4.6 CQA Subtask B
CQA Subtask B ranks forum questions by similarity to an original question. Without task-specific tuning or contextual features, the Reddit+SNLI model outperforms the official primary-run winner cited by the paper.
- CQA Subtask B ranks ten retrieved forum questions according to their similarity to an original question.Mean average precision is used to evaluate the ranking.
- Both “PerfectMatch” and “Relevant” questions are treated as good questions that should rank above “Irrelevant” questions.
- The models rank questions using cosine similarity between independently encoded original and related-question vectors, without interaction or contextual features.
- Without tuning on task-provided training data, both evaluated models show competitive performance.
- Reddit+SNLI outperforms SimBow-primary, which ranked first in the official 2017 task ranking.
5 Related Work
Prior STS work progressed from lexical and surface-based methods to competitive deep learning approaches. This paper extends conversational learning from response prediction toward general-purpose semantic representations, alongside related sentence-encoding methods.
- Early STS methods emphasized lexical semantics, surface-form matching, and basic syntactic similarity.
- Deep learning methods later became competitive on STS, including approaches based on general-purpose sentence encoders and cosine similarity.
- The official CQA ranking uses only each team’s primary run, excluding contrastive runs.
- Conversational data had previously been used effectively for email response prediction, which this work extends to broader semantic information.
- The approach is related to Skip-Thought vectors, which learn sentence representations through neighboring-sentence prediction.
6 Conclusion
The paper shows that conversational response prediction can learn useful sentence encoders for semantic similarity, with multitask Reddit+SNLI training achieving the strongest reported neural STS result.
- Response prediction over conversations learns sentence encoders that perform well on sentence-level semantic textual similarity.
- The basic Reddit conversation model is competitive with existing sentence-level encoders on public STS tasks.
- Multitask training on Reddit and SNLI achieves state-of-the-art performance among sentence-encoding-based models on the STS Benchmark.
- Without task-specific training, the Reddit and Reddit+SNLI models are competitive on CQA subtask B.