Source-linked AI summary
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, Bao Ge
TL;DR
ChatGPT research has expanded rapidly across NLP and other domains, motivating a broad synthesis of its capabilities, applications, and ethical implications. This paper surveys the GPT series and analyzes 194 arXiv papers using trend, word-cloud, and field-distribution analyses. It finds growing interest and broad application potential, while identifying limitations that future models should address.
Problem
The expanding ChatGPT literature requires a comprehensive account of its capabilities, applications, ethical concerns, limitations, and future research directions.
Method
The paper surveys ChatGPT-related research and analyzes 194 arXiv papers through trend, word-cloud, and cross-field distribution analyses.
Results
The findings show rapidly growing interest in ChatGPT, broad potential across domains, and improved adaptability and performance associated with large-scale pre-training and RLHF.
Takeaways & Limitations
The survey provides insights for leveraging large language models across applications and for guiding future development toward addressing identified limitations.
Takeaways & Limitations
ChatGPT may lack current knowledge and fact-checking ability, producing plausible but nonsensical responses from unreliable or outdated training data.
Abstract
from arXiv · showhide
This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and GPT-4) research, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) have played significant roles in enhancing LLMs' adaptability and performance. We performed an in-depth analysis of 194 relevant papers on arXiv, encompassing trend analysis, word cloud representation, and distribution analysis across various application domains. The findings reveal a significant and increasing interest in ChatGPT-related research, predominantly centered on direct natural language processing applications, while also demonstrating considerable potential in areas ranging from education and history to mathematics, medicine, and physics. This study endeavors to furnish insights into ChatGPT's capabilities, potential implications, ethical concerns, and offer direction for future advancements in this field.
1. Introduction
ChatGPT builds on GPT-series advances in large-scale pre-training, instruction fine-tuning, and RLHF, enabling versatile NLP performance and applications. This paper surveys its research landscape, analyzing trends and 194 arXiv papers while considering capabilities, impacts, ethics, limitations, and future directions.
- Background: ChatGPT, including GPT-3.5 and GPT-4, belongs to the GPT series of large language models developed through large-scale text pre-training.These models demonstrate performance across translation, summarization, and question-answering tasks.
- Background: Instruction fine-tuning and RLHF help language models adapt across NLP tasks and align responses with human preferences and values.ChatGPT succeeds InstructGPT and incorporates these developments.
- Paper scope: 194 arXiv papers mentioning ChatGPT were analyzed through research-trend, word-cloud, and cross-field distribution analyses.The corpus was counted as of April 1, 2023.
- Paper scope: The paper surveys ChatGPT research and potential applications, examining capabilities, ethical considerations, limitations, and future model-development directions.The survey covers diverse domains and evaluates findings from existing studies.
2. Related work of ChatGPT
The related-work section organizes existing ChatGPT research and uses visual analyses to represent its structure and recurring themes.
- Related work: The section reviews research on ChatGPT applications and ethics as part of the paper’s broader survey framework.Figure 4 is described as showing the overall framework of this section.
- Related work: A word cloud summarizes the terms appearing across all 194 analyzed papers.The visualization is presented as Figure 2.
2.1. Application of ChatGPT
The surveyed applications span education, mathematics, physics, medicine, communication, and literature search, with results varying substantially by task and domain. Studies also report limitations in factual grounding, consistency, and integration into time-sensitive systems.
- Education: ChatGPT supports educational question answering across academic and conceptual subjects, including physics, mathematics, chemistry, philosophy, and religion.Users can ask open-ended and analytical questions to examine its capabilities.
- Mathematics: ChatGPT performed best on simple set-theory and logic problems but poorly on Olympiad problem solving in the GHOSTS evaluation.Most Olympiad scores were 2 points, with only two 4-point scores out of 5.
- Physics: In a college-physics concept test, ChatGPT scored approximately 50%–65%, compared with a student average of 56%.The test covered kinematics and Newtonian mechanics and included 415 students.
- Medicine: ChatGPT’s causal-discovery evaluation for neuropathic-pain diagnosis found limited understanding beyond textual training data and inconsistent answers across repeated inquiries.The study distinguishes language-pattern understanding from underlying knowledge.
- Communication: ChatGPT can serve as an intelligent consulting assistant for ordered-importance semantic communication by ranking message-word importance before unequal error protection.The ranking is used within the communication system’s transmission strategy.
- Literature search: ChatGPT generated higher-accuracy Boolean queries than advanced query-generation methods, but with reduced recall.The trade-off was considered acceptable for some time-limited rapid reviews.
2.1.2. Text Classification
ChatGPT is applied to genre recognition, affective-computing classification, stance detection, and implicit hate-speech detection. Its usefulness is tempered by weaknesses on rare or out-of-vocabulary words and substantial computational requirements.
- Applications: ChatGPT’s text-classification applications include genre recognition, personality prediction, sentiment analysis, suicide-ideation detection, stance detection, and implicit hate-speech detection.The studies use zero-shot prompting or task-specific prompts across multiple datasets and languages.
- Implicit hate speech: 636 of 795 sampled tweets, or 80%, were correctly recognized in implicit hate-speech classification.The remaining classifications were 18.4% non-hate speech and 1.6% uncertain; Mechanical Turk reclassification was consistent for those categories.
- Limitations: ChatGPT struggles with rare or out-of-vocabulary words because it relies heavily on training-data distributions.This limitation affects classification tasks beyond the reported application examples.
- Limitations: The computational resources required to train and use ChatGPT can limit deployment in some text-classification applications.
2.1.3. Text Generation
The survey organizes ChatGPT text generation research by output length, covering phrases, sentences, and paragraphs, alongside code generation applications and limitations. These studies show useful generation capabilities but also quality, domain, and reliability constraints.
- Text generation: The survey categorizes ChatGPT text generation into phrases, sentences, and paragraphs based on generated word count.This structure organizes the reviewed literature by output length.
- Phrases: ChatGPT supports phrase generation for automated dataset labeling and other workflows requiring semantic or linguistic variation.The reviewed work uses ChatGPT to generate labels where datasets lack shared tokens.
- Sentences: Sentence-generation studies use ChatGPT for dialogue creation and text simplification, but performance can remain poor on test data.One dialogue-generation dataset study reported poor test-set performance, while another applied ChatGPT to simplify radiology reports.
- Paragraphs: Paragraph-generation evaluations report that ChatGPT may underperform fine-tuned models on summarization metrics and cross-lingual datasets.Reported metrics include R_1, R_2, R_L, and B_S.
- Code generation: ChatGPT can generate viable code and multiple programming solutions, supporting code explanation, translation, and more complex programming tasks.The reviewed studies include GPTCOMCARE and Jupyter-style code generation experiments.
- Code generation: Code generation remains constrained by programming-language coverage, need for manual optimization, and dependence on clear, reliable natural-language inputs.Generated code may require editing and may be unreliable when prompts contain errors, ambiguity, or inconsistency.
2.1.5. Inference
The reviewed research examines ChatGPT for inference and data-processing tasks, including reasoning, visualization, extraction, quality assessment, augmentation, and multimodal processing. Results indicate broad utility alongside uncertainty, domain-knowledge, privacy, and evaluation limitations.
- Inference: Inference studies examine inductive reasoning, decision-making, spatial reasoning, and ambiguity detection using ChatGPT.The literature includes tasks that infer meaning from text and evaluate reasoning under structured prompts.
- Inference: ChatGPT can reach correct answers through incorrect reasoning and may make suboptimal decisions on simple reasoning problems.The reviewed decision-making studies report uncertainty in the reasoning process.
- Data visualization: ChatGPT supports natural-language-to-code visualization workflows, but ambiguity in natural language remains a challenge.The reviewed applications include structured datasets and Jupyter-based graph generation.
- Information extraction: ChatGPT-based information extraction decomposes complex tasks into multiple question-answering rounds before producing structured results.ChatIE uses multi-round questioning, while ICL-D3IE iteratively updates formatted presentations.
- Information extraction: 63.99% F1 was reported for named entity recognition after a framework trained local models, compared with 23.37% previously.The cited study also reports improvement from 75% for a relationship-extraction task, though the supplied passage does not provide the paired final value.
- Quality assessment: 88.0% system-level accuracy was reported for GEMBA on the MQM2022 translation-quality test set, while ChatGPT accuracy exceeded 80%.GEMBA evaluates each translation fragment and averages scores into a system-level result; the passage notes this scope requires further improvement.
- Data processing: ChatGPT is used for data augmentation and multimodal processing, but domain knowledge, sensor-data handling, privacy, security, and prompt interaction remain concerns.The survey reports potential across visualization, extraction, enhancement, assessment, and multimodal tasks.
2.1.7. Human-ChatGPT Collaboration
The survey presents human-ChatGPT collaboration as joint work in which humans contribute expertise and decisions while ChatGPT provides language-based assistance. Applications include software architecture and game design, with positive feedback and reuse reported for the latter.
- Human-ChatGPT Collaboration: Human-machine collaboration combines human expertise, creativity, and decision-making with machine automation, scalability, and computing power.ChatGPT contributes natural-language understanding, suggestions, task completion, and adaptation from feedback.
- Software architecture: ChatGPT can support software architects by transforming software stories into feasible architecture diagrams through continuous interaction.The method also applies SAAM to evaluate architecture components and generate evaluation outputs.
- Game design: Three game design tasks produced positive feedback, and the framework showed reusability across free-text-described design tasks.The reported application scope is any design task that can be described in free text.
- Future directions: Future collaboration research may improve ChatGPT’s handling of nonverbal cues such as tone of voice and body language.The stated aim is more effective interaction with people.
2.1.8. ChatGPT Integration
ChatGPT can be integrated into software and other systems to improve communication and accessibility, including generating multiple code solutions. However, language differences, nondeterminism, and latency constrain integration in precision- or time-sensitive settings.
- ChatGPT Integration: Integration combines systems or software components, and ChatGPT can serve as a component or tool for communication between them.Natural-language interaction can reduce specialized knowledge or training requirements for non-technical users.
- Software integration: GPTCOMCARE generated multiple source-code solutions for programming queries, improving reported code-solution quality, diversity, and development efficiency.The study describes reduced time and effort for software development.
- Integration challenges: Integration is limited by language barriers, terminology differences, nondeterministic responses, and slow processing for time-sensitive data.These constraints matter most when systems require precise, reproducible results or operate in time-critical environments such as traffic.
2.1.9. Medical Applications
ChatGPT-related systems show promise across medical imaging workflows, including diagnosis, report generation, and de-identification. However, technical complexity and insufficient specialized medical knowledge constrain reliable integration.
- ChatGPT supports radiologists through image annotation, lesion detection and classification, feedback, recommendations, and report generation.
- ChatCAD integrates LLMs with computer-aided diagnosis networks for diagnosis, lesion segmentation, and report generation.
- ImpressionGPT uses domain-specific, individualized dynamic contexts to help radiologists compose impressions from findings.
- AD-AutoGPT generates data collection, processing, and analysis pipelines from text prompts, supporting Alzheimer’s disease discourse analysis.
- DeID-GPT demonstrates promising medical-report de-identification capabilities compared with other LLMs.
- Medical-imaging integration remains constrained because text interfaces may not convey detailed anatomy and ChatGPT lacks specialized medical knowledge for precise analysis.
2.2. AI Ethics
The survey describes ethical risks involving plagiarism, cheating, political and ideological tendencies, and rapidly changing regulation. It also points to norms and stakeholder responsibilities as ways to support safer use.
- ChatGPT’s scalable text generation raises concerns about plagiarism in academic, scientific, and news writing and about cheating in examinations.
- Studies found political and ethical tendencies in ChatGPT, including a pro-environment, left-wing liberal orientation in political tests.
- ChatGPT may perform slightly better than the current SOTA language model while retaining shortcomings across four evaluated aspects.
- Regulatory discussions assign responsibility across value-chain stakeholders and propose strategies for more comprehensive laws governing generative AI.
- Political and ethical tendencies could influence users’ behavior and decision-making, motivating norms and limitations for safer use.
3. Evaluation
Evaluations portray ChatGPT as a broadly capable but uneven general-purpose model. Its performance varies by task and generally remains below state-of-the-art systems, despite strengths in several language tasks and improved comprehension with prompting.
- Evaluations span multitask, multilingual, multimodal, and eight common NLP application tasks across public datasets.
- ChatGPT’s average quality is 56.5% versus 73.7% for SOTA models, with higher performance variability reflected by standard deviations of 23.3% versus 16.7%.
- Zero-shot evaluation across 20 datasets and seven task categories assesses whether ChatGPT functions as a general NLP task solver.
- Reasoning performance varies: ChatGPT performs well in arithmetic and some analogy, causal, commonsense, deductive, and abductive tasks but only averagely in mathematical-symbol and logical reasoning.
- ChatGPT performs similarly to GPT-3.5 and BERT-style models on sentiment analysis, but subjective emotion tasks reduce its average quality relative to SOTA.
- ChatGPT performs well on natural-language inference and better than GPT-3.5 on question answering, dialogue, and summarization, supporting its qualification as a general-purpose model.
- ChatGPT struggles with mathematical, temporal, spatial, multi-hop, named-entity, negative-connotation, and neutral-similarity tasks.
- Zero-shot performance is comparable to fine-tuned BERT and GPT-3.5 models, while advanced prompting improves comprehension without surpassing SOTA models.
4. Discussion
The discussion identifies limitations in timeliness, understanding, computation, safety, bias, and privacy, while outlining training, real-time data, domain adaptation, governance, and broader application as future directions.
- 4.1. Limitations: Historical training data up to 2021 limits real-time understanding, while unreliable sources and absent fact-checking can produce plausible but nonsensical responses.
- 4.1. Limitations: Ambiguous queries and specialized abbreviations can trigger comprehension biases, incorrect responses, and vacuous responses.
- 4.1. Limitations: Large-scale training and inference require substantial computation and electricity, increasing energy consumption and carbon emissions and restricting deployment.
- 4.1. Limitations: Designed restrictions can be evaded through prompts, enabling unhealthy content or illicit commercial use.
- 4.1. Limitations: Pre-training data can produce political and ideological biases, requiring caution in public applications such as education and publicity.
- 4.1. Limitations: Privacy and data-security concerns grow with user adoption, personal-information collection, and potential uploads of private multimodal data.
- 4.2. Future Directions: Future work should refine training and data filtering, reduce computational costs, improve context awareness, and integrate real-time data streams.
- 4.2. Future Directions: Interdisciplinary governance, public awareness, and localized domain-specific training are proposed for safer deployment, especially in medical applications.
5. Conclusion
ChatGPT research interest is growing rapidly, with potential applications spanning natural language processing and many other domains. The survey attributes ChatGPT’s success partly to large-scale pre-training on extensive internet data.
- ChatGPT-related research interest is growing rapidly, with potential applications across a wide range of domains.
- Large-scale pre-training captures knowledge from the internet and enables ChatGPT to learn from massive amounts of data.