Source-linked AI summary
ConveRT: Efficient and Accurate Conversational Representations from Transformers
Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, Ivan Vulić
TL;DR
Conversational AI needs pretrained representations that reduce the computational and annotation burdens of large general-purpose encoders. ConveRT uses a compact dual-encoder pretrained on response selection, and achieves strong response-selection and transfer results while remaining inexpensive and fast to train.
Problem
Large pretrained models are computationally expensive, while conversational systems require domain-specific labelled data for scaling and maintenance.
Method
ConveRT pretrains a compact dual encoder on conversational response selection using subword representations, quantization, and extended dialog history.
Results
ConveRT achieves state-of-the-art response-selection performance, including 71.8% on Reddit, and transfers strongly to intent classification across three datasets.
Takeaways & Limitations
Quantized ConveRT models occupy 59MB and 73MB, while pretraining takes 18 hours and costs roughly $85.
Takeaways & Limitations
The paper assumes English and uses Reddit as general-domain conversational pretraining data.
Abstract
from arXiv · showhide
General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.
1 Introduction
ConveRT addresses the computational and data-scaling challenges of conversational AI with a compact pretrained response-selection encoder. It targets effective, affordable, and portable conversational representations that can transfer to other dialog tasks.
- Task-oriented dialog systems require domain-specific labels for intents, dialog states, and entities, making scaling and maintenance challenging.
- Response selection pretraining uses large unlabelled conversational data to learn representations suited to conversational cues and retrieval-based dialog.
- ConveRT combines a compact dual-encoder architecture with subword representations, quantization, and transformer-style blocks.
- 59MB ConveRT is substantially smaller than the 444MB previous state-of-the-art dual encoder.
- Pretrained conversational representations can transfer to downstream dialog tasks such as intent detection and value extraction.
- Multi-context ConveRT incorporates earlier dialog history while remaining compact at 73MB and improving established response-selection benchmarks.
2 Methodology
ConveRT is a compact dual-encoder pretrained for response selection, using subword parameterization, transformer blocks, quantization, and shared encoders. Its multi-context variant adds earlier conversation turns while preserving efficient retrieval.
- ConveRT uses subword representations, transformer-style blocks, and quantization to build an effective, affordable, and quick-to-train dual encoder.
- The model uses a shared subword vocabulary for input and response encoding, including 31,476 subwords and 1,000 out-of-vocabulary buckets.
- The architecture uses relative attention biases and bounded attention ranges to model longer sequences and distant dependencies efficiently.
- The encoder transforms input and response vectors through three feed-forward hidden layers into separate L2-normalized 512-dimensional representations.
- Response relevance is scored by cosine similarity between input and response encodings, with in-batch nonmatching responses serving as negatives.
- Multi-context ConveRT concatenates up to 10 earlier messages and combines objectives for immediate, earlier, and averaged conversation contexts.
3 Experimental Setup
Experiments evaluate ConveRT on response selection across Reddit, AmazonQA, and DSTC7-UBUNTU, and on intent classification across three domains. The setup compares compact pretrained representations with standard and dual-encoder baselines.
- Pretraining uses 727M Reddit input-response pairs derived from 3.7B comments, with 654M pairs reserved for training.
- ConveRT pretraining takes 18 hours on 12 Tesla K80 GPU nodes and costs roughly $85 on Google Cloud Platform.
- Baselines include TF-IDF, USE-LARGE, BERT-LARGE, USE-QA, and POLYAI-DUAL, covering keyword, standard pretrained, and dual-encoder systems.
- Response selection is evaluated on Reddit, AmazonQA, and DSTC7-UBUNTU using Recall@k with N=100 and k=1.
- Intent classification experiments use pretrained ConveRT encodings with a classifier and compare against USE-LARGE and BERT-LARGE.
- The experimental design includes fixed-encoder transfer for ConveRT and USE-LARGE, but full fine-tuning for BERT-LARGE.
4 Results and Discussion
ConveRT combines compact architectural and quantization choices with response-selection pretraining to reduce model size and training cost while maintaining strong performance. Across response selection and intent classification, multi-context modeling and transferable encodings achieve competitive or state-of-the-art results.
- Model Size, Training Time, Cost: ConveRT is 59MB, compared with 444MB for the previous state-of-the-art dual encoder.Its compactness uses 8-bit embedding quantization, quantization-aware training, subword-level parameterization, and pruned self-attention.
- Model Size, Training Time, Cost: 18 hours on 12 16GB T4 GPUs is the reported pretraining time for ConveRT.The paper contrasts this with 90 hours for DistilBERT on 8 V100 GPUs and one day for RoBERTa on 1024 V100 GPUs.
- Response Selection on Reddit: 71.8% is the multi-context ConveRT score for Reddit response selection, a state-of-the-art result.Replacing extra contexts with an empty string yields 65.6%, while using only the interaction sub-network yields 65.7%.
- Other Response Selection Tasks: Fine-tuned ConveRT reaches a new state-of-the-art score on AMAZONQA, while fine-tuned POLYAI-DUAL cannot match its performance.AMAZONQA supports only single-context response selection.
- Other Response Selection Tasks: Multi-context ConveRT outperforms the best-scoring system from the official DSTC7 challenge and can match or surpass another BERT-based dual encoder.The BERT-based dual encoder uses a BERT-base architecture, while its training is roughly 5× slower.
5 Conclusion
The paper introduces ConveRT as a lightweight Transformer-backed dual-encoder model for neural response selection, achieving state-of-the-art results across response-selection and intent-classification tasks. Its quantized versions reduce model size and training cost while pretrained models are publicly released.
- ConveRT achieves state-of-the-art performance across response selection and transfer-learning tasks for intent classification.
- Quantized ConveRT and multi-context ConveRT occupy 59 MB and 73 MB, respectively, and train for 18 hours at an estimated cost of 85 USD.
- The authors publicly release pretrained ConveRT models to support further developments in retrieval-based task-oriented dialog.