Source-linked AI summary
Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
Samia Kabir, David N. Udo-Imeh, Bonan Kou, Tianyi Zhang
TL;DR
The paper addresses the lack of comprehensive evidence about ChatGPT’s programming answers and studies them against Stack Overflow responses. Using mixed manual, linguistic, and user analyses, it finds frequent misinformation and verbosity alongside language features that attract user preference. The findings support greater caution and improved communication of incorrectness in programming assistance.
Problem
No comprehensive study had examined the quality, characteristics, and human perception of ChatGPT answers to programming questions compared with human Stack Overflow answers.
Method
The study combines manual analysis, linguistic analysis, and a user study of ChatGPT answers to Stack Overflow questions.
Results
ChatGPT answers frequently contain errors and are unnecessarily verbose, yet users sometimes prefer them because of their comprehensive and articulated language.
Takeaways & Limitations
The findings emphasize caution when using ChatGPT for programming and the need for methods that identify, communicate, and mitigate misinformation.
Takeaways & Limitations
The manual analysis is subjective, the user study has sample-size and participant-bias limitations, and the main analysis used free GPT-3.5 responses.
Abstract
from arXiv · showhide
Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.
1 INTRODUCTION
The study addresses the lack of comprehensive evidence about ChatGPT’s programming answers by combining quality, linguistic, and user-perception analyses against Stack Overflow answers.
- 1 INTRODUCTION: The study combines manual, linguistic, and user analyses to characterize ChatGPT answers to programming questions and compare them with human Stack Overflow answers.It examines correctness and quality, linguistic features, and how programmers choose between the two answer sources.
- 1 INTRODUCTION: 52% of ChatGPT answers contain misinformation, while 77% are verbose.The manual analysis assessed correctness, consistency, comprehensiveness, and conciseness across 517 sampled Stack Overflow questions.
- 1 INTRODUCTION: ChatGPT answers use more formal and analytical language and portray less negative sentiment than human Stack Overflow answers.These differences were identified through LIWC and sentiment analysis on 2,000 randomly sampled questions.
- 1 INTRODUCTION: Participants preferred ChatGPT answers 35% of the time and overlooked misinformation 39% of the time.Comprehensiveness and articulated language structures helped explain preferences for some incorrect ChatGPT answers.
- 1 INTRODUCTION: The findings identify misinformation and unnecessary verbosity as risks while motivating methods to detect and mitigate incorrect ChatGPT programming answers.The authors also discuss design implications and future directions for countering misinformation.
2 RELATED WORK
Prior work establishes Stack Overflow’s importance for programming help and documents limitations of both online Q&A and AI assistance, while leaving ChatGPT’s programming-answer characteristics insufficiently studied.
- 2 RELATED WORK: LLM-generated content can contain hallucinations and misinformation that may mislead people and disrupt societal functions.This concern motivates examination of misinformation in programming assistance.
- 2 RELATED WORK: ChatGPT is popular across domains but can fabricate facts and generate low-quality or misleading information.Before this study, its characteristics and human perception in programming answers had not been investigated comprehensively.
- 2 RELATED WORK: Stack Overflow supports code reviews, conceptual questions, and rapid community answers, but developers must switch away from IDEs and may encounter toxic or discouraging interactions.Prior studies also link answer sentiment to voting and follow-up participation.
- 2 RELATED WORK: AI pair-programming tools have shifted developers toward code understanding and quick suggestions, with web searches often used for documentation or verification.ChatGPT additionally supports conversational questions, feedback, explanations, and debugging beyond code completion.
3 RESEARCH QUESTIONS
The study asks how ChatGPT differs from Stack Overflow answers in quality, what fine-grained errors occur, and how question characteristics and human perceptions relate to those differences.
- 3 RESEARCH QUESTIONS: The research questions are intended to deepen understanding of ChatGPT answer characteristics and human perception while informing methods to counter programming misinformation.The stated rationale connects empirical characterization with interaction and computational design.
- 3 RESEARCH QUESTIONS: RQ1 quantifies differences between ChatGPT and Stack Overflow answers in correctness, consistency, conciseness, and comprehensiveness.The question targets both correctness and multiple quality dimensions.
- 3 RESEARCH QUESTIONS: RQ2 develops a taxonomy of fine-grained issues, including conceptual, code, and terminology errors associated with hallucinations.The goal is qualitative analysis of the specific problems underlying aggregate quality measures.
4 METHODOLOGY
The methodology uses stratified sampling, manual coding, automated linguistic analysis, and a controlled user study to compare ChatGPT with human Stack Overflow answers.
- 4 METHODOLOGY: The study combines manual analysis, linguistic and sentiment analysis, and user studies with semi-structured interviews involving 12 participants.Manual analysis addresses RQ1–RQ3, automated analyses address RQ4–RQ5, and user studies address RQ6–RQ8.
- 4 METHODOLOGY: 517 questions were sampled across popularity, recency, and question type, with an additional random sample of 2,000 questions for linguistic analysis.Question types were conceptual, how-to, and debugging; sampled types were manually validated.
- 4 METHODOLOGY: For each sampled question, researchers prompted free ChatGPT GPT-3.5 using the Stack Overflow title, body, and tags, then compared responses with accepted human answers.Separate chat sessions were used for individual questions, and the free version was chosen because it represented most of the target population.
- 4 METHODOLOGY: Manual coding assessed correctness against accepted answers and external resources, along with consistency, comprehensiveness, and conciseness using a refined multi-label codebook.The codebook covered factual, conceptual, code, and terminological errors; inter-rater agreement reached Fleiss’s kappa 0.83.
- 4 METHODOLOGY: LIWC compared linguistic features across 2,000 ChatGPT–Stack Overflow answer pairs, while the user study standardized presentation and randomized answer order.Participants could skip unfamiliar questions, consult external resources, and validate generated code without accessing the original Stack Overflow answer.
5 MANUAL ANALYSIS RESULTS
Manual analysis of 517 ChatGPT answers found substantial correctness and quality problems alongside broad coverage, with answer characteristics varying by question properties.
- 5.1 RQ1: Overall Correctness and Quality: 52% of ChatGPT answers contained incorrect information, 78% were inconsistent with human answers, 35% lacked comprehensiveness, and 77% contained redundant, irrelevant, or unnecessary information.ChatGPT answers averaged 266.43 tokens versus 213.80 for human answers, a significant mean difference of 52.63 tokens.
- 5.2 RQ2: A Taxonomy of Fine-Grained Issues in ChatGPT Answers: Conceptual errors were most common at 54%, followed by factual errors at 36%, code errors at 28%, and terminology errors at 12%.These error categories were not mutually exclusive.
- 5.2 RQ2: A Taxonomy of Fine-Grained Issues in ChatGPT Answers: 48% of ChatGPT answers had no fine-grained factual, conceptual, code, or terminology errors, while code errors were mainly wrong logic or wrong API, library, or function usage.Wrong logic accounted for 48% of code errors and wrong API, library, or function usage for 39%.
- 5.3 RQ3: Effects of Question Type: Answers were consistently comprehensive across question categories, but verbosity remained widespread, with popular-question answers not concise 84% of the time.Average and not-popular questions were not concise 74% and 72% of the time, respectively.
- 5.3 RQ3: Effects of Question Type: Question popularity, type, and recency affected answer quality: popular and older posts were less incorrect and more verbose, while debugging answers were more inconsistent but less verbose.Conceptual and how-to answers were the most verbose.
6 LINGUISTIC ANALYSIS RESULTS
Linguistic analysis of 2,000 answer pairs found that ChatGPT answers used a more formal, analytical, goal-oriented style and expressed less negative emotion than human answers.
- 6.1 RQ4: Linguistic Characteristics: ChatGPT answers contained more analytical and clout-related language, whereas human answers showed more emotional, perceptual, and cognitive language features.ChatGPT also conveyed stronger drives, affiliation, achievement, and power, while conveying risks less often.
- 6.1 RQ4: Linguistic Characteristics: ChatGPT answers were more formal, expressed more analytic thinking, showcased more effort toward goals, and exhibited less negative emotion than human answers.Human answers were more informal and casual, while ChatGPT answers used fewer swear words, netspeak, nonfluencies, and fillers.
- 6.2 RQ5: Sentiment Analysis: 85.35% of ChatGPT answers expressed positive sentiment, compared with 73.30% of Stack Overflow answers; the sentiment distributions differed significantly (χ2 = 186.84, df=3, p<0.001).Only 0.1% of ChatGPT answers were negative, versus 1.05% of human answers.
7 USER STUDY RESULTS
In a 12-programmer user study, participants rated human answers as more correct, concise, and useful, yet still preferred ChatGPT answers in about one-third of cases and often missed misinformation.
- 7.1 RQ6: Differentiating ChatGPT answers from human answers: Human answers received higher ratings for correctness, conciseness, and usefulness, while ChatGPT received a slightly higher but nonsignificant comprehensiveness rating.Mean correctness ratings were 4.41 for human answers and 3.21 for ChatGPT; usefulness ratings were 4.21 and 3.42.
- 7.1 RQ6: Differentiating ChatGPT answers from human answers: Participants correctly identified the machine-generated answer 80.75% of the time, using cues such as formal language, structured writing, answer length, and unusual errors.Language style was reported by 10 of 12 participants as helpful for identification.
- 7.2 RQ7: Assessing Answer Correctness: Participants identified incorrect answers only 60.66% of the time, failing 39.34% of the time.Comprehensive, well-articulated, and human-like explanations contributed to trusting incorrect ChatGPT answers.
- 7.2 RQ7: Assessing Answer Correctness: Ten of 12 participants emphasized verifying ChatGPT answers before use and suggested official documentation links and in-situ code execution to support validation.
- 7.3 RQ8: Factors for User Preference: Participants preferred Stack Overflow answers 65.18% of the time and ChatGPT answers 34.82% of the time, with correctness and answer quality among the main preference factors.Among ChatGPT preferences, 77.27% of the selected answers were incorrect.
8 DISCUSSION AND FUTURE WORK
The discussion highlights a tension: ChatGPT offers convenient, well-articulated answers, yet frequent misinformation can mislead programmers and affect software and education.
- Why Do Users Prefer ChatGPT Responses?: Participants preferred ChatGPT answers 34.82% of the time, including answers containing misinformation, because polite, comprehensive, articulated responses appeared convincing.Users especially overlooked misinformation when answers were difficult to verify through execution or documentation.
- What Is at Stake and What Does the Future Hold?: Seemingly correct misinformation may lead to suboptimal design decisions, software defects, and longer-term risks to software, cyberinfrastructure, and STEM learning.The authors also warn that learners may absorb incorrect concepts and compromise their ability to assess veracity.
- The Silver Lining: ChatGPT remains convenient because it provides immediate answers, and 48% of analyzed answers were completely correct.The authors frame this convenience as a benefit alongside the need to mitigate risks from incorrect information.
- What Is at Stake and What Does the Future Hold?: The authors call for awareness and new strategies and policies to address risks from incorrect ChatGPT-generated information.The discussion links these actions to programming practice, education, and broader societal consequences.
8.4 What Further Actions are Needed to Address Misinformation in ChatGPT?
The authors argue that misinformation requires interventions beyond prompt changes or generic warnings, including uncertainty communication, developer training, stronger review, and clearer accountability.
- 8.4.1 Limitations of Existing Approaches: Existing hallucination-mitigation methods address factual errors but may not correct conceptual errors caused by weak programming-concept understanding and semantic reasoning.The authors therefore call for sustainable, special-purpose solutions targeting conceptual errors.
- 8.4.1 Limitations of Existing Approaches: Prompt engineering remains insufficient for special cases, is not sustainable, and shifts responsibility for AI errors toward users.Different answers to identical prompts further complicate verification and make reliable correction difficult.
- 8.4.1 Limitations of Existing Approaches: A generic warning is insufficient because answers vary in the severity of their incorrectness, while confidence displays alone may not adequately communicate programming uncertainty and risk.The authors recommend investigating more effective communication and visualization methods for model uncertainty.
- 8.4.2 Implications for Software Industry: Software companies should train developers to verify and test ChatGPT-assisted code, especially for high-stakes tasks, and adapt code reviews to AI-specific mistakes.The recommendations include continuous testing and security checking for code produced with AI assistance.
- 8.4.5 Implications for Code Reviewers and Teaching Staff in STEM: Distinct ChatGPT linguistic features could help teaching staff and plagiarism tools recognize machine-generated answers, while curricula can use incorrect outputs as learning exercises.The authors mention verbosity, formal language, and title-body-summary structure as examples of such features.
- 8.4.4 Ethical and Policy Implications: Responsible adoption also requires policies distinguishing human and LLM roles and clarifying accountability for AI errors in software decision-making.The authors connect this need to ethical and human-rights considerations in responsible AI.
9 LIMITATIONS
The study’s limitations concern subjective manual judgments, restricted model and prompting settings, stochastic outputs, and remaining bias and generalizability constraints.
- 9 LIMITATIONS: The manual analysis is subjective, although multiple labelers, agreement checks, and iterative discussions were used to mitigate this limitation.The user study also has sample-size and participant-bias limitations; answer sources and formatting were anonymized and standardized.
- 9 LIMITATIONS: The manual analysis used free GPT-3.5 responses, limiting direct generalizability to newer ChatGPT versions.In a small comparison, GPT-4 answered 6 of 21 previously incorrect questions correctly, while 15 remained incorrect with similar error types.
- 9 LIMITATIONS: The study analyzed only initial ChatGPT answers, excluding follow-up interaction that might change answer correctness but would require question-specific prompting and greater effort.The authors note that interaction is not guaranteed to improve answers and may require multiple rounds.
- 9 LIMITATIONS: Using original Stack Overflow questions as prompts leaves the effects of alternative prompting strategies untested.The authors state that prompt design depends on the problem and user, and propose systematic future investigation.
- 9 LIMITATIONS: Because ChatGPT is stochastic, considering one initial answer per question leaves repeated-output variability unmeasured.Running ChatGPT five times per question would have multiplied analysis effort and reduced study scale.
- 9 LIMITATIONS: Despite mitigation efforts, some human bias and generalizability limitations persist.The authors identify these as continuing constraints on interpreting the study.
10 CONCLUSION
The study combines manual, linguistic, and user analyses to characterize ChatGPT answers to Stack Overflow questions. It finds frequent incorrectness and other quality problems, while users can still favor seemingly credible ChatGPT responses.
- 10 CONCLUSION: ChatGPT produces incorrect answers more than 50% of the time and also exhibits verbosity and inconsistency.Manual analysis identified many conceptual and logical errors in the answers.
- 10 CONCLUSION: ChatGPT answers use very formal language and rarely portray negative sentiment.
- 10 CONCLUSION: Users rate human Stack Overflow answers more favorably overall but sometimes prefer incorrect ChatGPT answers because of articulated language and seemingly correct logic.The study therefore emphasizes caution and awareness when using ChatGPT answers for programming tasks.
- 10 CONCLUSION: The findings motivate further research on identifying, understanding, rectifying, and mitigating errors in large language models.
A EXAMPLE OF INCORRECT CHATGPT RESPONSE
The example concerns a C++ program that crashes while accessing an array by user-provided index. The displayed ChatGPT response incorrectly proposes a conjunction for an impossible out-of-range condition, whereas the relevant valid-index test uses a disjunction.
- A EXAMPLE OF INCORRECT CHATGPT RESPONSE: The example’s C++ question asks how to end a do-while program after an invalid array index causes a crash.
- A EXAMPLE OF INCORRECT CHATGPT RESPONSE: The ChatGPT response incorrectly changes the bounds check to `i < 0 && i >= 10`, a condition no integer can satisfy.The response claims this would detect an out-of-range index, but the conjunction is logically impossible.
- A EXAMPLE OF INCORRECT CHATGPT RESPONSE: The correct in-range condition is `i >= 0 && i < 10`; equivalently, an out-of-range test requires `i < 0 || i >= 10`.Using the original `i >= 0 || i < 10` condition makes the test true for every number, allowing out-of-bounds access.
B EXAMPLE OF VERBOSE CHATGPT RESPONSE
The example presents several JavaScript checks for whether a variable is initialized, emphasizing that the appropriate method depends on the expected value and use case.
- The response compares truthiness, definedness, and null-or-undefined checks rather than identifying one universally correct method.It frames the choice around the variable’s expected value and the specific use case.
- if (elem) tests truthiness, so it can behave unexpectedly when falsy values such as zero, empty strings, NaN, null, or undefined are involved.
- if (typeof elem !== 'undefined') checks whether elem has been declared in the current scope, including when it holds a falsy value.
- if (elem != null) excludes null and undefined but still accepts other falsy values such as empty strings and zero.
- For object-property existence, the response recommends an explicit in check to avoid errors caused by falsy values.