PaperlayerView exact revision on arXiv

Source-linked AI summary

A Complete Survey on LLM-based AI Chatbots

Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, Chaoning Zhang

arXiv:2406.16937v2cs.CLcs.AI

TL;DR

LLM-based chatbots build on expanding data and advances in language modeling, but earlier conversational systems had limited context, accuracy, and scalability. This survey synthesizes chatbot evolution, applications, challenges, and improvement strategies, concluding that these systems have broad sectoral potential alongside technical, ethical, and misuse concerns.

  • Problem

    Earlier conversational AI had limited contextual understanding, inaccurate responses, weak language comprehension, and problematic scalability, while the expanding LLM-based chatbot landscape required an organized overview.

  • Method

    The paper surveys foundational chatbots, LLM development, current and emerging LLM-based chatbots, sectoral applications, challenges, and strategies for improving effectiveness and reliability.

  • Results

    The survey finds LLM-based chatbots deployed across education, research, healthcare, finance, and other sectors, with capabilities including human-language interaction and handling vast information streams.

  • Takeaways & Limitations

    The paper presents LLM-based chatbots as broadly useful tools whose future advancement depends on greater efficiency, sustainability, reliability, and responsibility.

Abstract

from arXiv · show

The past few decades have witnessed an upsurge in data, forming the foundation for data-hungry, learning-based AI technology. Conversational agents, often referred to as AI chatbots, rely heavily on such data to train large language models (LLMs) and generate new content (knowledge) in response to user prompts. With the advent of OpenAI's ChatGPT, LLM-based chatbots have set new standards in the AI community. This paper presents a complete survey of the evolution and deployment of LLM-based chatbots in various sectors. We first summarize the development of foundational chatbots, followed by the evolution of LLMs, and then provide an overview of LLM-based chatbots currently in use and those in the development phase. Recognizing AI chatbots as tools for generating new knowledge, we explore their diverse applications across various industries. We then discuss the open challenges, considering how the data used to train the LLMs and the misuse of the generated knowledge can cause several issues. Finally, we explore the future outlook to augment their efficiency and reliability in numerous applications. By addressing key milestones and the present-day context of LLM-based chatbots, our survey invites readers to delve deeper into this realm, reflecting on how their next generation will reshape conversational AI.

I. INTRODUCTION

LLM-based chatbots emerged from rapid data growth and advances in language models, addressing limitations of earlier conversational systems. This survey organizes their development, applications, challenges, and future improvements across sectors.

  • Motivation: Global data creation reached around 120 zettabytes in 2023 and is projected to reach 147 zettabytes in 2024 and surpass 180 zettabytes by 2025.The survey links this expanding data ecosystem to advances in AI and the emergence of LLMs.
  • Motivation: LLMs improved chatbot context relevance, language understanding, and ability to handle vast information streams across education, research, healthcare, and other sectors.These capabilities contributed to growing adoption of LLM-based chatbots.
  • Motivation: Earlier chatbots suffered from limited contextual understanding, inaccurate responses, weak language comprehension, and poor scalability across industries.These limitations often produced robotic and disjointed user experiences.
  • Research gap: Existing literature often focused on individual chatbots, limited domains, or insufficient technical and misuse analysis.Reported shortcomings included weak categorization, limited depth, and inadequate visual or analytical support.
  • Contribution: This survey covers multiple chatbots, including BARD, Bing Chat, and Claude, and provides detailed taxonomies of applications and challenges across sectors.Its stated aim is to offer an organized, up-to-date guide for understanding and researching LLM-based chatbots.
  • Recent evolution: ChatGPT’s release was followed by a sharp rise in public interest, GPT-4, Google BARD, and other LLM-based chatbots under development.The survey describes this period as a new era of AI-driven interaction.

A. Existing Surveys, Reviews, and Case Studies

Existing reviews and case studies examine chatbot impacts, applications, and ethics, but their scope is often limited by chatbot coverage, domain breadth, or issue categorization. The survey broadens this coverage across models, sectors, and challenge subcategories.

  • Prior literature: Prior work examined AI chatbots in academic research, education, online learning, and ethical or misuse-related contexts.These studies included reviews and case studies of chatbot effects on assessment integrity, research, and learning.
  • Prior literature: Existing studies often centered on specific chatbots or limited application areas rather than surveying the broader LLM-based chatbot landscape.The survey identifies this narrower scope as a distinction from its own coverage.
  • Survey contribution: The survey expands coverage to BARD, Bing Chat, Claude, multiple application domains, and challenges divided into several subcategories.Table I is presented as a comparative summary of the reviewed literature.

B. Our Contributions

The survey examines chatbot evolution, applications, challenges, and technical and ethical improvements, covering diverse LLM-based models across sectors. It organizes applications through a detailed taxonomy and frames challenges around knowledge and data.

  • The survey asks how chatbots evolved from simple automated systems to LLM-based variants and how LLM advances redefined their capabilities.
  • It investigates applications across sectors and their effects on operational dynamics and user interactions.
  • It examines technical, ethical, and misuse challenges affecting chatbot performance, reliability, and responsible use.
  • The survey covers ChatGPT, BARD, Bing Chat, and many other LLM-based models rather than focusing on a single chatbot.
  • Its detailed application taxonomy organizes chatbot use across education, research, healthcare, software engineering, and finance.
  • The discussion frames chatbot interaction with extensive training data and subsequent generation of new knowledge as central to the survey.

II. OVERVIEW

The overview traces chatbots from early rule- and pattern-based systems through voice assistants to LLM-based chatbots. It presents this history as a progression toward more detailed and nuanced responses enabled by large-scale pretrained transformers.

  • The overview links early chatbots, LLM development, and LLM-based chatbots as successive stages in chatbot evolution.
  • The Turing Test provided an early foundation by asking whether a human could distinguish a machine from another human through textual conversation.
  • ELIZA used keyword identification and pattern matching without understanding input meanings, despite appearing human-like to users.
  • Racter generated English prose, while Jabberwacky stored dialogue phrases and selected replies from a dynamically expanding database.
  • A.L.I.C.E. expanded chatbot knowledge bases and used Artificial Intelligence Markup Language to establish conversational guidelines.
  • Watson, XiaoICE, and Tay demonstrated expanded conversational applications, emotional interaction, and risks including offensive outputs that led to Tay’s shutdown.
  • The rise of LLMs around 2020 enabled chatbots trained on large pretrained transformers to produce more detailed and nuanced responses.

B. Large Language Models (LLMs)

LLMs use transformer-based architectures and large-scale training to generate and process human language, while instruction tuning and chain-of-thought prompting extend their ability to handle unfamiliar and multi-step tasks. Despite strong text-generation capabilities, they remain limited by hallucinations, weak semantic understanding, and difficulties maintaining long-range coherence.

  • LLM overview: LLMs are transformer language models trained on enormous text datasets with billions of parameters.The survey describes GPT as the first LLM to attract broad attention, followed by progressively larger models including GPT-2, GPT-3, and GPT-3.5.
  • Architecture: The vanilla Transformer uses encoder-decoder processing, while GPT-series models use autoregressive generation with unidirectional attention.The encoder builds representations through multi-head self-attention, and the decoder generates outputs autoregressively; GPT models adapt this design.
  • Self-attention: Self-attention computes query-key similarities, scales them by 1 √d_k, normalizes them with SoftMax, and aggregates value vectors into updated representations.Q, K, and V represent the current element, other elements, and information to aggregate, respectively.
  • Training and prompting: Instruction-shaped multi-task training improves generalization to unfamiliar tasks, while chain-of-thought prompting supports tasks requiring multiple reasoning steps.These capabilities are presented as extensions of the fine-tuning process rather than as replacements for pretraining.
  • Limitations: LLMs can generate coherent text but lack semantic understanding, sometimes hallucinate factual errors, and may lose coherence across long passages.Their local planning is strong for adjacent sentences, whereas global planning and sustained relevance remain difficult.

C. LLM-based Chatbots

LLM-based chatbots have rapidly expanded from ChatGPT to competing systems such as BARD, Bing Chat, Claude, and Ernie Bot, alongside additional systems under development. Their capabilities differ in areas including access to current information, citation support, reasoning, safety, and output constraints.

  • Chatbots in use: ChatGPT, launched in November 2022 as a fine-tuned version of GPT-3.5, generates human-like replies from internet-trained language knowledge.The survey identifies ChatGPT as a leading chatbot in the current market.
  • Chatbots in use: BARD uses an optimized LaMDA variant and initially offered more recent information than ChatGPT’s knowledge limited to data available through 2021.The survey also notes that ChatGPT later gained web-search access and an updated knowledge base.
  • Chatbots in use: Bing Chat combines GPT-4 with live internet access and citation-supported responses, allowing users to validate information.It was introduced shortly after BARD and initially held an advantage over ChatGPT in these capabilities.
  • Chatbots in use: Claude 2 emphasizes reasoning and safer responses through constitutional AI, but its knowledge base lacks real-time information after 2022.Claude Instant is described as faster and lighter, while Claude 2 is positioned as the stronger reasoning system.
  • Chatbots in use: Ernie Bot is trained on web, voice, search, image, and knowledge-graph data, but early testing found hallucinations and elementary math errors.The survey reports a knowledge graph containing 550 billion facts.
  • Systems under development: Sparrow and Grok are identified as promising systems under development, while BlenderBot’s restrictive output constraints may reduce creativity and user satisfaction.The survey connects responsible progress with continued research, collaboration, and ethically sound development.

III. APPLICATIONS

LLM-based chatbots are presented as knowledge-generating tools with applications across education and other sectors. In education, they support learning, writing, teaching, feedback, translation, accessibility, and individualized engagement.

  • Cross-sector applications: LLM-based chatbots generate knowledge and are becoming integral across sectors including education, research, and healthcare.The survey associates their use with efficiency, accuracy, and personalized engagement.
  • Learning: In education, ChatGPT and Bing Chat can act as objects-to-think-with that encourage participation and support STEM learning.The survey describes potential roles across K–12, undergraduate, and graduate education.
  • Academic writing: ChatGPT improves academic writing by correcting punctuation, spelling, and grammar while supporting style development, content depth, and reader engagement.These uses are described as improving both content quality and individual writing skills.
  • Teaching: ChatGPT can assist instructors with curriculum outlines, course materials, language translation, assessment tasks, and grading.The survey groups these functions into teaching preparation and assessment support.
  • Additional educational uses: Educational chatbots provide grammar exercises, interactive discussions, instant feedback, translation, reading support, text summaries, word explanations, and speech-based accessibility tools.Speech-to-text and text-to-speech integration is described as potentially benefiting visually impaired learners.

B. Research

LLM-based chatbots support research through literature exploration, paraphrasing, data analysis, and idea generation, while healthcare applications extend to question answering, education, and treatment suggestions. Reported studies show useful performance but also identify accuracy and plagiarism concerns.

  • Literature Review and Paraphrasing: LLM-based chatbots assist literature reviews by locating related papers and extracting insights from large scholarly collections.Semantic Scholar has indexed 200 million publications, illustrating the scale of the literature-search problem.
  • Literature Review and Paraphrasing: BARD paraphrased Metaverse abstracts with a 12% plagiarism-matching rate, compared with 1% for its answer to “What is the Metaverse?”The experiment used ten Metaverse articles published from 2021–2023 and checked outputs with iThenticate.
  • Data Analysis: Chatbot-supported data analysis can improve research efficiency, but extensive documents may cause data-flow, latency, and runtime problems.Segmenting documents and using a memory buffer can address these issues, while custom agents can combine textual and numerical data.
  • Idea Generation: LLM-based chatbots support research idea generation by stimulating creativity and suggesting ways to expand existing ideas.The surveyed literature presents these systems as tools for students and teachers as well as researchers.
  • Idea Generation: AI tools were once viewed as suitable mainly for simple tasks, yet reported findings describe substantial research improvements and creativity sometimes exceeding average human performance.The passage cautions that these tools are not always accurate.
  • Healthcare: In healthcare, LLM-based chatbots address medical question answering, patient education, and treatment suggestions across multiple clinical settings.The survey discusses ChatGPT, Claude, GPT-4, Med-PaLM 2, BARD, and Bing Chat.

D. Miscellaneous Applications

Beyond education, research, and healthcare, LLM-based chatbots are applied to software engineering and finance. Their reported uses include conversational coding support, debugging, financial analysis, investment advice, and portfolio recommendations.

  • Software Engineering: LLM-based chatbots let developers express intent conversationally while supporting code writing, debugging, and software testing.This differs from command-based support, which requires users to specify methods more explicitly.
  • Software Engineering: Most evaluated chatbots generated correct Java matrix-multiplication code on the first attempt, while BARD required human assistance.YouChat produced the fastest code at 446 ms.
  • Software Engineering: Chatbots can explain and summarize code, improving clarity and making software-engineering expertise more accessible.The survey connects these capabilities with productivity gains across programming tasks.
  • Finance: Finance applications include financial-data analysis, trend detection, market-opinion analysis, investment recommendations, and multilingual investment advice.The survey describes these systems as supporting both customers and employees.
  • Finance: Bing Chat analyzed BIST100 documents from 2019 to 2022 to recommend six companies and suggest portfolio composition based on portfolio size.The passage presents these outputs as support for analysts’ strategic decision-making.
  • Cross-sector applications: The survey presents LLM-based chatbots as increasingly embedded across sectors in response to evolving consumer preferences and demand for interactive technologies.Table IV provides an overview of chatbot applications across sectors.

IV. OPEN CHALLENGES

The survey organizes open challenges around technical, ethical, and misuse perspectives, with technical limitations including stale knowledge, weak reasoning, hallucinations, and inconsistent responses. These problems constrain reliability across changing and complex tasks.

  • Challenge Framework: The survey frames these limitations within a broader challenge structure spanning technical, ethical, and misuse perspectives on knowledge and data.Table V organizes the challenge categories around those viewpoints.
  • Knowledge Recency: Updating chatbot knowledge is costly, risks catastrophic forgetting, and remains constrained by limited access to diverse, high-quality data.The survey identifies knowledge recency as a continuing limitation for information beyond the latest training data.
  • Logical Reasoning: Chatbots perform worse on multi-step reasoning than single-step reasoning and do not reason like humans.On the cited assessment, ChatGPT-3.5, ChatGPT-4, and Bing Chat scored 40.0%, 64.5%, and 60.0% on multi-step questions.
  • Hallucination: In treatment-related evaluations, 12.5% of responses were identified as hallucinations because they did not align with recommended treatments.The survey also reports inaccurate or fictitious references in some atrial-fibrillation answers.
  • Consistency and Self-awareness: LLM-based chatbots may generate different responses to the same input, creating a consistency problem that researchers address through prompt engineering.The survey separately notes that these systems lack self-awareness, emotions, and subjective experiences.

B. From an Ethical Perspective

The survey identifies ethical and practical risks in LLM-based chatbots, including opacity, bias, privacy threats, unfairness, academic misuse, over-reliance, and misinformation. It emphasizes transparency, responsible use, critical evaluation, and protection of users.

  • Ethical Issues: LLM-based chatbots raise ethical concerns involving transparency, bias, privacy risks, and unfairness.
  • Transparency: Black-box generation and unverified training data make chatbot outputs difficult to explain and can produce inaccurate domain-specific responses.
  • Bias: Biased training data and user inputs can reproduce social inequalities, including harmful medical recommendations and gendered professional suggestions.
  • Privacy and Unfairness: Sensitive training data and weak safeguards create privacy risks, while limited non-English coverage can marginalize users of other languages.
  • Misuse: Academic misuse and over-reliance may compromise evaluation standards and weaken students’ critical and independent thinking.
  • Misinformation: Because chatbot responses may present wrong information as true, users should cite generated content, preserve independent judgment, and validate accuracy.

V. FUTURE OUTLOOK

The future outlook centers on making LLM-based chatbots more efficient, sustainable, effective, and capable across modalities. The survey highlights compression, green AI, prompt engineering, and multimodal integration as key directions.

  • Model Compression and Optimization: Future chatbot development should improve efficiency through model compression and optimization while addressing the deployment costs of very large models.Models from 3 billion to approximately 1760 billion parameters show rising MMLU scores, but models over 100 billion parameters create storage, distribution, and deployment challenges.
  • Model Compression and Optimization: Model scaling improves capabilities such as in-context learning and chain-of-thought prompting, but practical deployment requires smaller or more efficient models.
  • Green AI: Green AI is needed because training and operating large chatbot models depend on extensive computing resources and can carry biases from multiple data sources.
  • Prompt Engineering: Prompt engineering improves chatbot interaction by structuring context, instructions, format, and optional elements to support more relevant and accurate responses.
  • Multimodality: Multimodal chatbots extend interaction beyond text by interpreting images, generating visual descriptions, and processing documents, although capabilities differ across systems.
  • Conclusion: The survey frames efficiency, sustainability, prompt quality, and multimodal capabilities as central directions for future LLM-based chatbots.

B. Ethical Guidelines and Responsible Usage

Responsible chatbot use requires privacy protection, language inclusion, fairness, and explicit academic and medical protocols. The survey recommends safeguards that preserve accountability while recognizing current limitations in transparency, bias, and explainability.

  • Privacy and Data Protection: Responsible deployment must protect sensitive user and patient data through strict privacy and security measures, including healthcare-compliant anonymization.
  • Language Diversity and Equal Rights: Limited non-English datasets create performance gaps, requiring inclusive systems that support diverse languages, users, disabilities, and multimodal interaction.
  • Language Diversity and Equal Rights: Bias monitoring must continue after training so discriminatory patterns can be identified and corrected to support equal access to services.
  • Academic and Medical Protocols: Educational use should rely on accountability, integrity, transparency, and honesty rules rather than banning chatbots outright.
  • Academic and Medical Protocols: Medical students and educators should recognize privacy, copyright, transparency, and bias limitations while citing chatbot use responsibly.
  • Academic and Medical Protocols: Explainability and transparency remain insufficiently established under current and proposed international regulatory frameworks, especially for medical use.
  • Conclusion: Ethical advancement depends on safeguarding privacy, ensuring language diversity and equal rights, and establishing academic and medical protocols.
  • Conclusion: The survey connects these guidelines to its broader review of chatbot evolution, architectures, systems, applications, and challenges.
Loading 2406.16937v2…