Source-linked AI summary
Few-Shot Out of Domain Intent Detection with Covariance Corrected Mahalanobis Distance
Jayasimha Talur, Oleg Smirnov, Paul Missault
TL;DR
Few-shot OOD intent detection is needed when conversational-agent data and auxiliary OOD resources are scarce, yet standard Mahalanobis distance performs poorly in this regime. The paper analyzes this issue and proposes covariance-corrected Mahalanobis detection using robust covariance estimators. Across the reported experiments, robust estimators consistently outperform the MLE baseline in few-shot settings, with Shrinkage excelling at 5-shot and 10-shot training.
Problem
Existing OOD-intent methods require large auxiliary datasets, while standard Mahalanobis detection performs poorly in the few-shot settings common to data-scarce conversational agents.
Method
The paper replaces MLE covariance estimation in Mahalanobis OOD detection with robust covariance estimators and evaluates the resulting score-based detectors.
Results
Robust covariance estimators outperform other OOD detection techniques on 3 out of 4 datasets across all reported metrics, while covariance correction beats MLE in 5-shot settings.
Takeaways & Limitations
Shrinkage is the strongest reported estimator in 5-shot and 10-shot settings across various datasets, and the approach is computationally cheap.
Abstract
from arXiv · showhide
Conversational agents like chatbots and voice assistants are trained to understand and respond to user intents. On encountering an utterance with an intent different from the ones they have been trained on, these agents are expected to classify the intent as `unknown' or `out of domain'. This problem is known as out of domain (OOD) intent detection. Podolskiy et al. (2021), showed that Mahalanobis distance can be used effectively for identifying OOD intents, outperforming competing approaches. However, their method fails to outperform the baselines in the practically important few-shot setting. In this paper we analyze the reason for low performance and propose a covariance corrected Mahalanobis distance for detecting out-of-domain intents.
1 Introduction
OOD intent detection helps conversational agents distinguish unfamiliar utterances and trigger fallback responses, but existing methods generally assume abundant auxiliary data. This paper targets the practically important few-shot setting, where training and validation data are scarce and standard Mahalanobis detection performs poorly.
- OOD detection classifies utterances as in-domain or out-of-domain and supports fallback responses such as clarification or “I don’t know”.
- Existing OOD-intent methods commonly require either out-of-domain examples or a large unlabeled corpus.
- Domain-specific chatbots may lack both auxiliary OOD data and large unlabeled domain-specific corpora during bootstrapping.
- Customer-configured conversational agents often have scarce training and validation data.
- Mahalanobis distance on RoBERTa embeddings can outperform baselines without additional data, but performs poorly in low-resource settings.
- The paper identifies the source of low performance and proposes a covariance-corrected method for few-shot OOD intent detection.
2 Preliminaries and Related Work
OOD detection can use either additional data or score functions, and this paper focuses on score-based methods that avoid auxiliary OOD data. It examines Mahalanobis distance, whose low-resource performance is limited despite its broader use and effectiveness with larger datasets.
- OOD intent detection frames each test utterance as either out-of-domain or in-domain.
- Data-centric approaches use additional OOD examples, whereas score-based approaches decide using features, logits, or related model scores.
- A score-based detector assigns an utterance a scalar G(x), classifying it as OOD when G(x) >= T and IND when G(x) < T.
- Mahalanobis OOD detection computes each test feature’s minimum class-wise covariance-adjusted distance and uses that distance as G(x).
- Mahalanobis distance is used in pattern recognition, anomaly detection, and adversarial-example detection, but its performance degrades rapidly in low-resource settings.
- No score-based methods were specifically designed for few-shot out-of-domain intent detection before this work.
3 Methodology
The methodology analyzes covariance estimation in few-shot Mahalanobis detection and replaces the conventional estimator with robust alternatives. The analysis links poor performance to singular covariance estimates and tests how covariance-data size affects OOD performance.
- Motivation and analysis: On ROSTD, AUC improves sharply when covariance computation uses up to 400 data points, with only modest improvement afterward.
- Motivation and analysis: In few-shot settings, covariance rank is bounded by min(n −1, d), so n ≪d makes the covariance matrix singular and its pseudoinverse potentially harmful.
- Motivation and analysis: The experiment fine-tunes RoBERTa in a 5-shot setting, extracts full-dataset features, and varies the sample size used for covariance computation.
- Motivation and analysis: The covariance-size experiment supports both sufficient feature representation for IND/OOD separation and a contribution from non-invertible covariance to poor few-shot performance.
- Covariance correction: The proposed robust estimators combine covariance information with structural assumptions to improve estimation when n ≪d relative to MLE.
- Covariance correction: Van Ness retains only diagonal covariance elements, setting non-diagonal elements to zero.
- Covariance correction: Shrinkage combines a singular covariance matrix with a diagonal target whose entries equal the mean eigenvalue of the original covariance estimate.
- Covariance correction: Ledoit-Wolf selects a shrinkage coefficient by minimizing expected mean square error against the unobserved true covariance matrix.
4 Experiments
The experiments fine-tune RoBERTa and evaluate covariance-corrected Mahalanobis methods against established OOD detection baselines across several intent datasets and few-shot settings.
- Datasets and metrics: The study compares MLE, Van Ness, Shrinkage, and Ledoit-Wolf covariance estimators with Gradient Norm, Energy, and MSP baselines.The covariance estimators are evaluated as drop-in replacements within Mahalanobis distance-based OOD detection.
- Training procedure: RoBERTa is fine-tuned for intent classification using cross-entropy loss, with experiments repeated 20 times using newly sampled training sets.The pretrained model uses AdamW, a 2e−5 learning rate, linear decay, and 45 epochs.
- Training procedure: During X-shot training, each covariance matrix uses X · Nc sampled training points, while RoBERTa embeddings have dimensionality d = 768.The covariance matrix is therefore estimated from only the few-shot examples available for the selected classes.
- Datasets and metrics: The evaluation covers CLINC150, ROSTD, ROSTD-COARSE, and SNIPS, spanning datasets with different numbers of intents and OOD configurations.CLINC150 contains 150 intents across 10 domains; ROSTD-COARSE uses three intent classes, and SNIPS contains seven intents.
- Results: In the 5-shot ROSTD results, Shrinkage achieves 94.39±2.88 AUC, while MLE achieves 85.13±5.41 AUC.The corresponding PR ROC values are 83.32±7.79 for Shrinkage and 58.49±10.74 for MLE.
- Datasets and metrics: OOD detection is evaluated as binary classification using AUC, PR ROC, and FPR@95%TPR, reporting both OOD-positive and OOD-negative conventions.For FPR, the threshold is selected to achieve a 95% true-positive rate.
5 Results
Covariance-corrected Mahalanobis methods outperform MLE and other OOD detection techniques in few-shot evaluations, with the strongest gains appearing when training data or intent classes are limited.
- Benchmark results: Covariance-corrected methods outperform other OOD detection techniques on 3 out of 4 datasets across all reported metrics.The compared corrected estimators are Shrinkage, Ledoit-Wolf, and Van Ness.
- Benchmark results: In the 5-shot setting, covariance correction outperforms MLE on all datasets.This establishes the broad few-shot comparison before considering larger covariance-estimation sample sizes.
- Class-count effects: Shrinkage outperforms MLE by a large margin with few intent classes and converges toward MLE performance as the number of classes increases.The CLINC150 analysis also reports declining OOD detection effectiveness as the number of in-domain classes grows.
- Class-count effects: Increasing CLINC150 from 3 to 20 in-domain classes reduces the minimum distance from an OOD query to its closest IND centroid by 20%.Using all 150 classes reduces the average IND-OOD Euclidean distance by 37%, increasing confusion near a fixed score threshold.
6 Conclusion
The paper concludes that robust covariance estimators improve few-shot Mahalanobis OOD detection, especially with Shrinkage, while retaining a lightweight implementation.
- Conclusion: Robust covariance estimators consistently outperform the Maximum Likelihood estimator baseline in few-shot settings.The conclusion identifies Shrinkage as strongest in 5-shot and 10-shot experiments across various datasets.
- Conclusion: Shrinkage excels in 5-shot and 10-shot settings across various datasets.The conclusion presents this as the strongest estimator finding from the experiments.
- Conclusion: The suggested approach adds one matrix-vector multiplication per class and requires neither auxiliary data nor training-procedure modifications.This is the stated computational and data-use overhead of the approach.