Source-linked AI summary
Benchmarking Natural Language Understanding Services for building Conversational Agents
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, Verena Rieser
TL;DR
Accessible NLU services simplify conversational-agent development, but users lacked reliable, wide-coverage evidence for choosing among them. The paper compares four services on a large annotated dataset spanning 21 domains. Watson significantly outperforms the others on Intent classification but performs significantly worse on Entity Type recognition because of low Precision.
Problem
Reliable, third-party evaluations broad enough to compare commonly used NLU platforms were lacking, making platform choice difficult.
Method
The paper systematically evaluates Dialogflow, LUIS, Watson, and Rasa using a 25k-utterance dataset annotated with Intent and Named Entity specifications.
Results
Watson significantly outperforms the other platforms on Intent classification but performs significantly worse on Entity Type recognition because of low Precision.
Takeaways & Limitations
Performance is similar on combined F-scores, but separating Intent and Entity Type recognition reveals important platform differences.
Takeaways & Limitations
The evaluation excludes Watson’s recently added Contextual Entity annotation tool and includes noisy, ambiguity-affected Entity annotations.
Abstract
from arXiv · showhide
We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.
1 Introduction
NLU platforms make conversational-agent development more accessible by mapping utterances to structured Intent or Dialogue Act representations with entities. The paper addresses the difficulty of choosing among platforms by introducing a systematic evaluation and public dataset.
- NLU maps user utterances to abstract Intent or Dialogue Act representations and associated Entity arguments.Together, these representations specify an utterance’s semantic content within a dialogue domain.
- A newly observed Watson Contextual Entity annotation tool is not included in the evaluation.
- The paper evaluates Rasa, Watson, LUIS, and Dialogflow using a new dataset of 25k annotated user utterances.The dataset contains Intent and Named Entity specifications and is released with the evaluation toolkit.
2 Related Work
The paper positions its evaluation as the first wide-coverage comparison of NLU services, addressing limited breadth in earlier benchmarks. It covers more domains, Intent types, and Entity types than the closest cited work.
- Earlier NLU evaluations lacked breadth across Intent types, Entity types, and studied domains.The closest cited work evaluated six services across only three domains.
- This study compares four commonly used NLU services on a dataset spanning 21 domains, 64 Intents, and 54 Entity types.
3 Natural Language Understanding Services
NLU services classify user Intents and recognize Named Entities, but the evaluated platforms do not support Multiple Intents or dialogue context. The paper compares commercial and open-source options with differing input and output formats.
- NLU services typically perform Intent or Dialogue Act classification and Named Entity recognition.
- The evaluated platforms include commercial services such as Dialogflow, LUIS, and Watson, alongside open-source Rasa.
- None of the four platforms support Multiple Intents when one utterance expresses more than one action.The paper identifies this as potentially significant for spoken dialogue.
- None of the platforms use dialogue context for Intent classification or Named Entity recognition.The paper links this limitation to understanding context-dependent elliptical or fragment utterances.
4 Data Collection and Annotation
The authors collected crowdsourced utterances for a home-assistant robot across many scenarios, then annotated entities and converted the data for each platform. The final corpus contains 25,716 utterances with moderate annotation agreement.
- The dataset targets a home-assistant robot across scenarios including alarms, music, calendars, news, recommendations, transport, and weather.
- Turkers answered scenario-specific questions and selected possible entities from predefined lists, while annotation errors were manually corrected.
- 25,716 utterances were annotated for 64 Intents and 54 Entity Types.
- Entity annotation achieved moderate inter-annotator agreement, with Fleiss’s Kappa κ = 0.69.Three students annotated overlapping samples for agreement measurement.
5 Evaluation Experiments
The evaluation compares four NLU platforms using a balanced sub-corpus and 10-fold cross-validation, with platform versions and processing pipelines specified.
- Dataset and setup: The experiments compare Dialogflow, LUIS, Watson, and Rasa on a sub-corpus covering 64 Intents and 54 Entity Types.The sub-corpus contains 11,036 utterances after limiting LUIS training data.
- Dataset and setup: 11,036 utterances form the evaluation sub-corpus, sampled at 190 instances per Intent where available.Some Intents had slightly fewer than 190 instances because of the available data.
- Evaluation procedure: 10-fold cross-validation uses 90% of the sub-corpus for training and 10% for testing in each fold.The same cross-validation design is reported for the evaluation experiments.
- Dataset caveats: The study notes that Intent and Entity distributions were unbalanced, but treats this as affecting all four platforms equally.The authors therefore state that imbalance does not confound the reported comparisons.
- Platform configurations: Rasa was evaluated with version 0.10.5 using a spaCy/sklearn pipeline combining Conditional Random Fields for NER and scikit-learn for Intent classification.Rasa also provides other built-in processing components, but this evaluation used the specified pipeline.
6 Results & Discussion
Across the evaluated platforms, Watson performs best on Intent classification but worst on Entity Type recognition, while combined F-scores are generally similar except for Watson.
- Analysis: The evaluation uses micro-average scores for Intent and Entity Type classification over 10-fold cross-validation and compares platforms with pairwise t-tests.Confusion matrices provide the Precision, Recall, and F1 calculations.
- Intent classification: Watson’s Intent F1 score is 0.882 and significantly exceeds the other platforms, while Dialogflow, LUIS, and Rasa do not differ significantly.The significance tests report p < 0.05 and large or very large Cohen’s D effect sizes for Watson’s advantage.
- Entity Type recognition: LUIS achieves the highest Entity F1 score at 0.777, while Watson’s Entity performance is significantly lower because of very low Precision.The paper attributes Watson’s low Precision to producing many Entity candidates and consequently many False Positives.
- Combined performance: All NLU services have similar combined F1 scores except Watson, whose combined score is significantly lower.The reported difference has p < 0.05 with large or very large effect sizes.
- Scope limitation: The reported evaluation does not include Watson’s recently added Contextual Entity annotation tool.The omission is noted both in the experimental results and as a qualification on Watson’s evaluation.
- Data limitations: The dataset contains imbalanced distributions and noisy Entity annotations caused by ambiguities that the simplified annotation scheme could not capture.For example, “play space invaders please” could be labeled with either a song-name or game entity.
7 Conclusion
The paper contributes a broad, released NLU dataset and a comparative evaluation of four services. Results are similar on combined F-scores but diverge by task, with Watson stronger for Intents and weaker for Entity Types.
- Contributions: The released home-robot dataset covers 21 domains, 64 Intents, and 54 Entity Types.It is presented as a large NLU dataset grounded in a real-world use case.
- Contributions: The paper comparatively evaluates Dialogflow, LUIS, Watson, and open-source Rasa on this dataset.The evaluation covers both commercial and open-source NLU services.
- Main findings: The services achieve similar combined F-scores, but Watson has higher Intent scores and lower Entity Type scores.The lower Entity Type scores are attributed to Watson’s high number of false positives in Entity predictions.
- Future work: Future work targets improved data quality and spoken utterances to study the impact of ASR errors on NLU performance.The authors also plan to observe how data-quality improvements affect performance.
Appendix
The appendix documents the platforms’ training examples, annotation formats, data distributions, and confusion-matrix summaries used in the evaluation.
- Training-data examples: The appendix provides example training data for Rasa, LUIS, Watson, and Dialogflow, including intents and annotated entities.Examples include utterances paired with intent labels and entity annotations.
- Confusion matrices: Tables 7 and 8 provide confusion-matrix summaries for Intent and Entity Type classification, respectively.The matrices support calculation of Precision, Recall, and F1 scores.