Source-linked AI summary
ChatGPT: A Meta-Analysis after 2.5 Months
Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, Steffen Eger
TL;DR
The paper addresses the lack of hard evidence on how ChatGPT is perceived across social media and scientific literature. It analyzes over 300,000 tweets and more than 150 papers, finding broadly positive assessments but also declining social-media sentiment, language differences, and domain-specific concerns. The authors conclude that ChatGPT is viewed as both an opportunity and a threat, with mixed assessments in education.
Problem
There is little hard evidence about ChatGPT’s perception across social media and scientific papers.
Method
The paper analyzes over 300,000 tweets and more than 150 scientific papers from Arxiv and SemanticScholar.
Results
ChatGPT is generally viewed as high quality and positively, while its social-media perception has slightly declined, is more negative outside English, and varies across scientific domains.
Takeaways & Limitations
The findings can inform public debate and the future development of ChatGPT.
Takeaways & Limitations
The analysis relies on error-prone automated tools, hashtag-biased tweet selection, subjective annotations, nondeterministic paper retrieval, and titles or abstracts for paper annotation.
Abstract
from arXiv · showhide
ChatGPT, a chatbot developed by OpenAI, has gained widespread popularity and media attention since its release in November 2022. However, little hard evidence is available regarding its perception in various sources. In this paper, we analyze over 300,000 tweets and more than 150 scientific papers to investigate how ChatGPT is perceived and discussed. Our findings show that ChatGPT is generally viewed as of high quality, with positive sentiment and emotions of joy dominating in social media. Its perception has slightly decreased since its debut, however, with joy decreasing and (negative) surprise on the rise, and it is perceived more negatively in languages other than English. In recent scientific papers, ChatGPT is characterized as a great opportunity across various fields including the medical domain, but also as a threat concerning ethics and receives mixed assessments for education. Our comprehensive meta-analysis of ChatGPT's current perception after 2.5 months since its release can contribute to shaping the public debate and informing its future development. We make our data available.
1 Introduction
The paper addresses limited hard evidence about how ChatGPT is perceived across social media and scientific writing. It finds broadly positive assessments, alongside a slight decline in social-media perception and more negative sentiment outside English.
- Motivation: The study fills a gap in hard evidence by analyzing ChatGPT’s perception across Twitter and scientific papers.It examines over 300,000 tweets and more than 150 papers.
- Main findings: ChatGPT is generally characterized as high quality, with positive sentiment and joy dominating social-media reactions.
- Main findings: Scientific papers portray ChatGPT as a major opportunity across fields, including medicine and writing, but also as an ethical threat.
- Main findings: Education receives mixed assessments, with ChatGPT described both as an opportunity and a threat to academic integrity.
- Main findings: Its social-media perception has slightly declined since debut, with joy decreasing and surprise increasing.
- Main findings: ChatGPT is perceived more negatively in languages other than English.
2 Analyses
Across social-media analyses, ChatGPT was generally perceived positively, but sentiment varied by language, topic, and time. Positive reactions and joy remained prominent while overall positivity and joy declined, surprise increased, and manual checks identified classification errors and changing concerns.
- Sentiment analysis: The sentiment classifier was applied to English-translated tweets, with negative, neutral, and positive labels; its English F1-score was 71%.The model’s performance varied across languages, motivating English as the sole input language.
- Sentiment over time and language: Average sentiment declined mildly from about 1.15 to 1.10, while English tweets remained more positive than non-English tweets.The English–non-English difference narrowed over time, but English retained the more positive perception.
- Sentiment over time and language: Positive tweets decreased and neutral tweets increased over time, whereas negative tweets stayed stable, suggesting a less euphoric public assessment after the initial hype.Small short-term sentiment increases followed each of the three included ChatGPT releases.
- Sentiment over time and language: English tweets had the most positive sentiment, while English, German, and French trends declined and Spanish and Japanese trends rose from low starting points.Topic composition was examined to help explain these language differences.
- Topics and sentiment: The five major topic classes covered 86.3% of tweets, and English had more business and science-and-technology content, topics associated with fewer negative views.The topic model covered science and technology, learning and education, news and social concern, diaries and daily life, and business and entrepreneurs.
- Emotion and qualitative analysis: Joy generally decreased after release while surprise increased, and post-update samples contained admiration alongside concerns about inaccuracies, detectability, bias, misinformation, job loss, and unethical use.Among manually examined tweets, 14 of 20 first-period users expressed admiration, while 13 of 20 second-period negative tweets expressed frustration.
3 Related work
The paper situates its analysis alongside prior work on Twitter reception and ChatGPT failure cases, extending earlier social-media-focused evidence.
- 3 Related work: Haque et al. found overwhelmingly positive Twitter reception after about two weeks, while Borji catalogued ChatGPT failure cases.Compared with these studies, the paper uses larger samples and also examines scientific papers.
4 Conclusion
The paper analyzes over 300k tweets and more than 150 scientific papers, finding generally positive perceptions of ChatGPT alongside ethical concerns and mixed educational assessments. It concludes that these findings can inform public debate and future development, while recognizing the need for longer-term research.
- Over 300k tweets and more than 150 scientific papers were analyzed to assess ChatGPT’s perception, changes over time, and reported strengths and limitations.
- ChatGPT was generally perceived positively and as high quality, with joy dominating associated social-media emotions.
- Scientific papers characterized ChatGPT as a great opportunity across fields, including medicine, but also as an ethical threat and a mixed prospect for education.
- The findings contribute to shaping public debate and informing ChatGPT’s future development.
- Future research should examine longer time spans, content popularity, additional dimensions, actor expertise, geographic and demographic distributions, and real societal impacts.
5 Limitations and Ethical Considerations
The analysis has methodological and sampling limitations that constrain how confidently its perception findings should be interpreted.
- The study relies on error-prone sentiment, emotion, machine-translation, and other NLP systems, while hashtag-based tweet selection may introduce bias.