Source-linked AI summary
A Literature Survey of Recent Advances in Chatbots
Guendalina Caldarini, Sardar Jaf, Kenneth McGarry
TL;DR
Chatbot research faces challenges in implementation and evaluation, including limitations of current dialogue metrics and human assessment. This paper surveys chatbot literature, implementation methods, applications, datasets, and evaluation frameworks, identifies challenges and gaps, and recommends directions for future research.
Problem
Evaluating chatbots is challenging because goals differ across dialogue systems, while n-gram metrics may not reflect equally effective responses without shared wording.
Method
The paper surveys chatbot literature and analyzes implementation methods, language models, applications, datasets, and evaluation frameworks, with attention to deep learning algorithms.
Results
The survey identifies current chatbot challenges, limitations, and literature gaps, and compares evaluation metrics while discussing further steps for improving chatbot evaluation.
Takeaways & Limitations
The paper recommends developing reliable automatic evaluation methods that assess correctness, coherence, and cohesiveness across multiple dialogue turns.
Takeaways & Limitations
Information Retrieval chatbots lack a learned AI model, while current chatbot systems remain unable to simulate human speech because of faulty dialogue modeling and limited open-access domain-specific data.
Abstract
from arXiv · showhide
Chatbots are intelligent conversational computer systems designed to mimic human conversation to enable automated online guidance and support. The increased benefits of chatbots led to their wide adoption by many industries in order to provide virtual assistance to customers. Chatbots utilise methods and algorithms from two Artificial Intelligence domains: Natural Language Processing and Machine Learning. However, there are many challenges and limitations in their application. In this survey we review recent advances on chatbots, where Artificial Intelligence and Natural Language processing are used. We highlight the main challenges and limitations of current work and make recommendations for future research investigation.
1. Introduction
Chatbots are conversational computer programs used across many fields to provide automated assistance, support, entertainment, and companionship. Recent AI and NLP advances have expanded their capabilities, but contextual and emotional understanding, gender bias, and evaluation remain important challenges.
- Chatbots process natural-language user input and generate a relevant output, enabling automated dialogue with many potential users simultaneously.
- Chatbots support applications ranging from education and e-commerce to healthcare and entertainment, including customer assistance and social connection.
- Recent AI and NLP developments have made chatbots easier to implement, more flexible to maintain, and better able to mimic human conversation.
- Human-chatbot interaction remains limited by weaker contextual and emotional understanding than humans and by gender-stereotyped implementation and application.
- The survey reviews chatbot literature and deep-learning implementation methods, identifies challenges and limitations, and recommends directions for future research.
2. Chatbots Background
Chatbots evolved from rule-based pattern matching toward AI-, machine-learning-, and NLP-based systems as computational capabilities and language technologies advanced. This evolution expanded their applications, while domain dependence remained a key limitation of manually written rules.
- ELIZA, developed in 1966, used keyword matching and linguistic rules to transform user input into an answer.
- A.L.I.C.E. used AIML topics, categories, matching rules, and templates to create an expandable chatbot knowledge base.
- ChatScript succeeded AIML and shifted chatbot technology toward semantic analysis and understanding.
- Rules and pattern matching are domain dependent and inflexible because they require manually written rules for specific domains.
- Deep-learning advances expanded chatbot applications, including voice-based personal assistants integrated into consumer devices and social chatbots designed for companionship.
3. Methodology
The literature survey used a two-stage process: gathering relevant chatbot research and then analyzing the retrieved articles to identify key aspects of the field.
- The first stage identified search terms and suitable research databases before collecting articles on chatbots.
- The second stage analyzed the retrieved articles to determine the key aspects covered by chatbot research.
Search Terms and Databases Identification
The survey gathered chatbot literature from multiple scholarly databases, repositories, and search engines using expanding search terms and successive relevance filters. It then reviewed 316 articles to classify the field’s main themes, including evolution, implementation, datasets, and evaluation.
- Search Terms and Databases Identification: The search covered IEEE, ScienceDirect, Springer, arXiv, Google Scholar, and JSTOR, selected for broad peer-reviewed coverage across relevant fields.
- Search Terms and Databases Identification: The initial “chatbot” query expanded to related terms including conversational modelling, conversation systems, conversational agents, and embodied conversational agents.
- Search Terms and Databases Identification: The researchers restricted the main analysis to relevant articles published between 2007 and 2021 and further filtered results by title.
- Search Terms and Databases Identification: Selection emphasized concrete chatbot technical implementations, development, literature reviews, and surveys while excluding marketing and social studies or peripheral chatbot uses.
- Search Terms and Databases Identification: Bibliographic searching added 201 articles to 115 initially retrieved articles, producing 316 articles for review.
- Reviewing Articles: The review analyzed abstracts and article content to identify recurring categories, including chatbot history and evolution, applications, implementations, datasets, and evaluation.
4. Literature Review Analysis
The survey organizes chatbot research around implementation approaches, datasets, evaluation methods, and applications, highlighting trade-offs across rule-based, retrieval, and generative systems. It identifies recurring limitations in data requirements, contextual response generation, and evaluation, while observing a divide between social and task-oriented chatbots.
- The literature review covers implementation approaches, public datasets, chatbot evaluation methods, and applications across different domains.
- Implementation approaches: Information Retrieval systems select responses from predefined knowledge bases, but building and matching against those bases can require substantial time and resources.Larger datasets expand the knowledge base while making correct answer matching more challenging.
- Implementation approaches: Deep Neural Networks can improve retrieval by ranking question-answer pairs associated with reformulated previous conversation turns.Merged rankings use prior queries to incorporate contextual information when retrieving an answer.
- Implementation approaches: Sequence-to-Sequence models generalize across domains but compress inputs into fixed-length context vectors, causing information loss and vague responses for longer sentences.The model can be trained end-to-end on different datasets without domain-specific knowledge.
- Evaluation: Human evaluation captures multiple interaction dimensions but is costly, time-consuming, difficult to scale, and subject to rater bias, whereas automated metrics are more efficient but lack established standards and holistic accuracy.Automated evaluation commonly uses BLEU, METEOR, and TER, although these metrics were developed primarily for other NLP tasks.
- Evaluation: N-gram metrics can undervalue effective dialogue responses with little lexical overlap, motivating adversarial evaluation based on how easily model outputs can be distinguished from human responses.The survey also notes that machine learning requires substantial training data, while domain-specific chatbot data may be confidential and unavailable openly.
- Applications and trends: The survey finds a divide in which deep-learning systems trained on large open-domain datasets are usually social or companion chatbots, while task-oriented systems typically use smaller domain-specific datasets and retrieval or rule-based methods.Task-oriented systems may combine Information Retrieval and Rule-based approaches, particularly where domain data is confidential.
5. Related Works
Previous chatbot surveys covered design, history, technology, evaluation, and domain applications. This survey distinguishes itself by emphasizing recent advances in chatbot design and implementation alongside major challenges and limitations.
- Earlier surveys examined chatbot design, implementation, history, background, evaluation methods, and domain-specific applications.
- This survey extends prior work by discussing advances in chatbot design and implementation and identifying major challenges and limitations.
- One reviewed study traced chatbot development historically, categorized technologies, and discussed modern architectures and platforms.
6. Discussion
The discussion identifies persistent gaps in chatbot architectures, industry applicability, evaluation, and survey coverage. It argues that context-aware, resource-efficient models and more reliable evaluation are needed within the documented scope.
- Current chatbot language models struggle to mimic human conversation because next-step dialogue modelling neglects prior turns, conversational context, and shared information.
- Information Retrieval chatbots remain widely used for coherent domain-specific responses, but lack a learned AI model and face scarce open-access domain data.
- Large models are difficult for industry to apply because they require substantial computational power and exceptionally large training datasets, while tailored closed-domain solutions are preferred.
- Chatbot evaluation lacks a common reference framework across automatic metrics and human judgments, limiting reliable testing and comparison.
- Recent surveys provide limited analysis of Transformers, comparative evaluation metrics, and distinctions between industry-oriented and social or companion chatbots.
- The field combines Deep Learning and Natural Language Processing and has expanded across industrial, companion, and assistant applications.
7. Conclusions
The survey analyzes chatbot language models, applications, datasets, and evaluation frameworks while synthesizing persistent implementation and research gaps. It concludes that more human-like architectures, scalable evaluation, and flexible models remain needed.
- The paper surveys relevant chatbot literature and analyzes language models, applications, datasets, evaluation frameworks, challenges, limitations, and research gaps.
- Current AI chatbots cannot yet simulate human speech, with the discussion attributing this to faulty dialogue modelling and limited open-access domain-specific data.
- Information Retrieval chatbots lack a learned AI model, while large models require substantial computing power and training data.
- Human evaluation is expensive, time-consuming, difficult to scale, biased, and incoherent, motivating improved automatic evaluation approaches.
- Future work should investigate smaller, flexible models, improved language models for industry-specific applications, more human-like architectures, and improved evaluation frameworks.