Source-linked AI summary
Efficient Intent Detection with Dual Sentence Encoders
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, Ivan Vulić
TL;DR
New-domain conversational systems need intent detectors that work with limited labeled data and modest resources. The paper uses fixed conversational dual-encoder representations from USE and ConveRT with lightweight classifiers, finding stronger performance than BERT-based detectors across three datasets, particularly in few-shot settings, alongside CPU-scale training and hyperparameter stability.
Problem
Expanding task-oriented conversational systems to new domains is resource-intensive because effective intent detection still needs expert knowledge and domain-specific labeled data in few-shot settings.
Method
The paper builds intent detectors from fixed USE and ConveRT sentence representations, applying a lightweight MLP classifier and optionally concatenating both encoders.
Results
USE+ConveRT detectors achieve peak performance across three diverse intent datasets, outperform BERT-backed detectors, and can be trained on CPUs in several minutes.
Takeaways & Limitations
Dual encoders provide an efficient and hyperparameter-stable basis for intent detection in low-data regimes, allowing annotated examples to be used directly for training.
Takeaways & Limitations
Increasing dual-model size and depth may yield further gains, but would reduce efficiency and training speed.
Abstract
from arXiv · showhide
Building conversational systems in new domains and with added functionality requires resource-efficient models that work under low-data regimes (i.e., in few-shot setups). Motivated by these requirements, we introduce intent detection methods backed by pretrained dual sentence encoders such as USE and ConveRT. We demonstrate the usefulness and wide applicability of the proposed intent detectors, showing that: 1) they outperform intent detectors based on fine-tuning the full BERT-Large model or using BERT as a fixed black-box encoder on three diverse intent detection data sets; 2) the gains are especially pronounced in few-shot setups (i.e., with only 10 or 30 annotated examples per intent); 3) our intent detectors can be trained in a matter of minutes on a single CPU; and 4) they are stable across different hyperparameter settings. In hope of facilitating and democratizing research focused on intention detection, we release our code, as well as a new challenging single-domain intent detection dataset comprising 13,083 annotated examples over 77 intents.
1 Introduction
Intent detection classifies varied user utterances into predefined intents, but expanding conversational systems to new domains remains resource-intensive under few-shot data constraints. The paper addresses this challenge with efficient dual sentence encoders and reports strong performance, practical training speed, and hyperparameter stability.
- Intent detection maps varied natural-language utterances into predefined intent classes for task-oriented conversational systems.Getting the intent wrong is described as the first point of failure of a conversational agent.
- Few-shot intent detection remains difficult because new domains require expert knowledge and domain-specific labeled data.The paper frames commercial low-data settings as having only several examples available per intent.
- USE- and ConveRT-based intent detectors outperform BERT-backed detectors across three diverse datasets, especially in few-shot scenarios.The comparison includes fixed BERT encodings and full-model BERT fine-tuning.
- USE+ConveRT detectors can be trained on a regular laptop CPU in several minutes and remain largely invariant to hyperparameter changes.The reported invariance can allow annotated examples to be used for training rather than held-out validation.
- BANKING77 contributes 13,083 annotated banking examples spanning 77 fine-grained intents, with code and data released online.It is presented as a challenging single-domain dataset.
2 Methodology: Intent Detection with Dual Sentence Encoders
The paper motivates conversational dual sentence encoders as efficient alternatives to language-model-based encoders for intent detection. Its method uses fixed USE and ConveRT representations with a lightweight classifier, optionally combining both encoders.
- Pretrained Sentence Encoders: ConveRT is a compact dual encoder, exemplifying the efficiency motivation behind the proposed approach.The passage reports a 59MB model pretrained in less than a day on 12 GPUs.
- Pretrained Sentence Encoders: BERT-based adaptation typically fine-tunes the full pretrained model, but this process is computationally intensive and still requires task-specific data.The paper identifies these costs as limitations of the standard adaptation approach.
- Dual Sentence Encoders and Conversational Pretraining: Conversational response-selection pretraining aligns sentence encoders with conversational tasks and encodes useful conversational cues.Dual encoders learn interactions between contexts and relevant responses.
- Intent Detection with dual Encoders: The proposed intent detectors use fixed USE and ConveRT sentence representations rather than fine-tuning the entire encoder.This design targets performance, efficiency, and few-shot applicability.
- Intent Detection with dual Encoders: A single-hidden-layer ReLU MLP followed by softmax classifies the fixed sentence representations.The classifier can consume either one encoder’s representation or concatenated USE+ConveRT vectors.
3 New Dataset: BANKING77
BANKING77 addresses limitations of earlier intent datasets by providing a challenging, fine-grained benchmark focused on one banking domain. Its 13,083 queries cover 77 intents, including partially overlapping categories that require fine-grained decisions.
- Earlier intent datasets with fewer than 10 classes oversimplified intent detection and did not emulate commercial-system conditions.More recent datasets expanded coverage but used multiple domains.
- BANKING77 contains 13,083 customer-service queries labeled with 77 fine-grained banking intents.The dataset introduces a single-domain focus not present in HWU64 and CLINC150.
- Unlike HWU64 and CLINC150, BANKING77 concentrates many fine-grained intents within a single domain.The paper proposes this focus because multi-domain examples may not capture each domain’s full in-the-wild complexity.
- Partially overlapping categories such as reverted top-up and failed top-up require fine-grained intent decisions.Individual word semantics are not always sufficient to identify the correct intent.
4 Experimental Setup
The experiments evaluate intent detectors across three datasets and multiple data regimes using accuracy as the main measure. They compare fixed and fine-tuned pretrained encoders, with a lightweight MLP setup and released training subsets supporting reproducibility.
- Few-Shot Setups: The study evaluates wide-scale few-shot classification by sampling smaller subsets from three intent detection datasets.The motivation is to approximate commercial development when expanding to a new domain or task with limited data.
- MLP Design: The MLP uses one 512-dimensional hidden layer, SGD with learning rate 0.7 and linear decay, dropout 0.75, and 500 iterations.These settings are used unless hyperparameters are explicitly varied.
- Models in Comparison: The comparison includes fixed BERT features, full-model BERT fine-tuning, and fixed dual-encoder representations.BERT-FIXED uses mean-pooled sequence-output embeddings, while BERT-TUNED uses the [CLS] output.
- Models in Comparison: USE and ConveRT are evaluated as pretrained dual encoders, while BERT-large provides a 24-layer, 1024-dimensional, 340M-parameter reference.ConveRT is described as lighter than BERT.
- Accuracy is reported for 10 examples per intent, 30 examples per intent, and full training data.The main results table reports accuracy scores across all three datasets and data setups.
5 Results and Discussion
Dual sentence encoders deliver competitive or superior intent-detection performance, particularly in few-shot settings, while remaining efficient and robust across hyperparameter choices. Their advantages over BERT include faster encoding, CPU viability, and stronger fixed-encoder performance, although some hardware and efficiency trade-offs remain.
- Results: USE+CONVERT achieves peak performance across the board on CLINC150, suggesting that the two dual models capture complementary information.The fixed-representation design permits straightforward combination of USE and CONVERT.
- Few-Shot Scenarios: USE+CONVERT improves over BERT-TUNED by +1.77, +1.33, and +0.97 accuracy points on BANKING77, CLINC150, and HWU64 with 10 examples per intent.The combined model also performs better with 30 examples per intent across all three datasets.
- Invariance to Hyperparameters: Dual-model intent detectors show limited accuracy fluctuation across hyperparameter regimes, unlike BERT-FIXED outlier runs and BERT-TUNED failures to converge in 2% of runs.The tested regimes vary dropout, hidden-layer count, hidden dimensionality, and optimizer.
- Resource Efficiency: BERT sentence encoding is around 20 times slower on CPU and roughly 3 times slower on GPU than the dual-model encoders.The comparison measures average encoding time for batches of 15 sentences.
- Resource Efficiency: Few-shot intent classifiers based on pretrained dual models can be trained and executed on CPUs without large models requiring GPUs or TPUs.For full-data training with a single hidden-layer neural network, GPU overhead can exceed the speed-up from faster encoding and computation.
- Further Discussion: Conversational response-selection pretraining transfers useful representations to intent detection, whereas BERT is competitive mainly when fine-tuned rather than used as a fixed encoder.The authors propose investigating hybrid strategies combining dual sentence encoders and language-model-based encoders.
6 Conclusion
The proposed dual-encoder intent classifiers achieve strong results in low-data settings while remaining lightweight, and the work contributes a new banking-domain dataset and released resources.
- Dual-encoder intent classifiers yield state-of-the-art results on three diverse English intent classification datasets.
- The work introduces a banking-domain dataset containing 13,083 annotated examples across 77 intents.
- The proposed models outperform fully fine-tuned BERT-based classifiers especially in few-shot scenarios with limited annotated examples.
- Their lightweight resource requirements support intent-classifier development without access to large computational resources.
- The authors plan to extend the efficient dual-encoder detectors to other languages and investigate transfer to low-resource settings.
- The code and datasets are released online at github.com/PolyAI-LDN/polyai-models.