Source-linked AI summary
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
TL;DR
Intent-classification benchmarks generally assume every query belongs to a supported class, leaving out-of-scope handling under-evaluated. The paper builds a 23,700-query dataset spanning 150 intents and evaluates classifiers and out-of-scope schemes, finding strong in-scope performance but weak out-of-scope detection. It concludes that more out-of-scope data tends to help, although such data is difficult and expensive to generate.
Problem
Existing intent-classification datasets cover supported labels but provide little analysis of queries that fall outside a system’s intent classes.
Method
The paper constructs a crowdsourced dataset of 23,700 queries across 150 intents and evaluates benchmark classifiers with three out-of-scope prediction schemes.
Results
Classifiers perform well on in-scope intents, but all evaluated methods struggle with out-of-scope queries; the best reported out-of-scope recall is 66%.
Takeaways & Limitations
The dataset provides a broader benchmark for developing more robust task-oriented dialog systems that handle out-of-scope queries.
Takeaways & Limitations
Out-of-scope data is difficult and expensive to generate because collection requires expert knowledge to verify scope boundaries.
Abstract
from arXiv · showhide
Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.
1 Introduction
Task-oriented dialog systems need intent classifiers that can recognize when queries fall outside supported actions. This paper introduces a broad dataset and evaluation focused on that under-studied out-of-scope challenge.
- Motivation: Existing intent-classification datasets cover few intents and do not support systematic analysis of out-of-scope queries.These limitations leave evaluations poorly matched to deployed systems with fixed, finite intent sets.
- Motivation: Out-of-scope queries are inevitable because users may not know the capabilities defined by a system’s fixed intent classes.Correct detection helps prevent wrong actions and can reveal potential directions for system development.
- Contribution: The dataset contains 23,700 short, unstructured user-style queries spanning 150 in-scope intents plus out-of-scope queries.It is designed to analyze intent classification with particular attention to out-of-scope handling.
- Results: BERT achieves at least 96% in-scope accuracy under limited-data and imbalanced conditions, while the best out-of-scope recall reaches only 66%.The results show a pronounced contrast between performance on known classes and out-of-scope queries.
2 Dataset
The paper introduces a crowdsourced dataset spanning 150 intents and 10 domains, with separately collected out-of-scope queries designed to resemble real user inputs. Its split and data-condition choices model limited, uneven, and out-of-scope data availability.
- In-scope collection: Crowdsourcing used scoping, rephrase, and scenario tasks to define intents and collect 100 training, 20 validation, and 30 test queries per intent.Intent definitions were manually grouped from questions and commands written in the style of interacting with an AI assistant.
- Out-of-scope collection: Out-of-scope queries came from worker mistakes and prompts based on topic areas outside the supported intents.Using the same crowdsourcing method made these queries similar in style to in-scope examples.
- Out-of-scope collection: Only roughly 69% of queries collected with out-of-scope prompts yielded out-of-scope examples, reflecting the difficulty of expert-verified collection.Of 1,200 out-of-scope queries, 100 were used for validation, 100 for training, and 1,000 for testing.
- Preprocessing and partitioning: Queries from each in-scope crowd worker stayed within one split, while out-of-scope data was split by task prompt to reduce unrealistic train-test similarity.Duplicates were removed and replaced, and all queries were downcased with final punctuation removed.
- Dataset conditions: Small, Imbalanced, and OOS+ vary in-scope training quantity, class balance, and out-of-scope training quantity to represent constrained production data.OOS+ increases out-of-scope training examples from 100 to 250.
3 Benchmark Evaluation
The evaluation compares benchmark classifiers and three schemes for detecting out-of-scope queries under multiple data conditions. It measures in-scope accuracy separately from out-of-scope recall, emphasizing missed out-of-scope cases because they can trigger incorrect responses.
- Evaluation setup: The evaluation tests a range of classifier models and out-of-scope prediction schemes across dataset conditions.The classifier set includes SVM, MLP, FastText, CNN, and BERT.
- Out-of-scope schemes: The oos-train scheme adds a 151st out-of-scope intent, oos-threshold applies a probability threshold, and oos-binary separates scope detection from intent classification.These schemes represent distinct ways to identify queries outside the 150 supported intents.
- Out-of-scope schemes: The oos-binary evaluation addresses class imbalance through in-scope undersampling or augmentation with Wikipedia sentences.The undersampling strategy uses 1,000 in-scope queries against 250 out-of-scope queries.
- Development trade-offs: Oos-train and oos-binary require curated out-of-scope training data, whereas oos-threshold can apply to any model producing probabilities.The threshold is selected using the highest validation score while treating out-of-scope as its own intent.
- Metrics: The evaluation reports accuracy over 150 intents and recall on out-of-scope queries.Recall is used because predicting an out-of-scope query as in-scope can produce a completely wrong response, whereas a false out-of-scope prediction triggers a fallback.
4 Results
Across evaluation settings, models classify in-scope intents more reliably than out-of-scope queries. More out-of-scope training data generally improves out-of-scope performance, but the gains remain limited and can reduce performance under some approaches.
- BERT is consistently the best approach for in-scope classification, followed by MLP.
- Out-of-scope performance is much lower than in-scope performance across all methods.
- Increasing the relative number of out-of-scope training queries improves out-of-scope recall, although performance remains low relative to in-scope accuracy.The OOS+ setting adds 150 out-of-scope queries to the full training set.
- Out-of-scope recall tends to be higher on the Full dataset with oos-threshold, while several models suffer greatly on limited datasets.BERT and MLP are the top oos-threshold performers; FastText and Rasa show especially erratic results.
- Undersampling lowers in-scope accuracy for all models, while out-of-scope recall improves over oos-train on Full but not OOS+.Augmenting the out-of-scope training set appears to improve both in-scope and out-of-scope performance compared with undersampling, though out-of-scope performance remains weak.
5 Prior Work
Prior intent-classification resources generally cover all examples with in-scope labels or focus on different task properties. They offer limited intent breadth or lack a feasible way to evaluate out-of-scope performance.
- Most existing analyses and datasets do not consider out-of-scope data, instead assigning labels intended to cover all possible queries.Out-of-distribution detection uses excluded datasets or classes, producing samples from a small set of coherent classes unlike this setting.
- Snips and Liu et al. provide conversational-agent request datasets, but Snips has few intents and many examples per intent.Snips also includes a low-data variation with 70 training queries per intent, where performance drops slightly.
- Other datasets impose constrained training data or use labels for response types rather than action intents.TREC includes many labels but only questions, no commands; Braun et al. cover very few intents.
- None of the other datasets summarized in Table 4 offer a feasible way to evaluate out-of-scope performance.
- DSTC datasets contain chatbot-style queries but focus on state tracking, with most or all queries in-scope.The paper instead focuses on both in-scope and out-of-scope queries for virtual-assistant intent decisions.
6 Conclusion
The paper introduces a dataset and evaluates intent classification together with out-of-scope prediction. It finds that BERT performs better on in-scope classification, while all investigated methods struggle with out-of-scope queries.
- The paper analyzes intent classification and out-of-scope prediction using a new dataset with carefully collected out-of-scope data.
- Models incorporating more out-of-scope training data tend to improve out-of-scope performance, but such data is expensive and difficult to generate.
- All datasets introduced in the paper are publicly available through the project repository.