Source-linked AI summary
How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study
Fadel M. Megahed, Ying-Ju Chen, Joshua A. Ferris, Sven Knoth, L. Allison Jones-Farmer
TL;DR
The paper addresses whether generative AI can augment SPC practice, learning, and research. It evaluates ChatGPT through structured prompts involving code, explanations, and knowledge generation, finding strong performance on structured tasks but weaknesses on nuanced tasks and a need for validation.
Problem
The paper examines whether emerging generative AI tools can usefully augment SPC practitioners, educators, and researchers despite their early development and potential for misuse.
Method
The study evaluates ChatGPT’s responses to structured prompts covering SPC code generation, concept explanation, and knowledge creation.
Results
ChatGPT performs well on structured tasks such as code translation and explaining well-known concepts but struggles with nuanced explanations and creating code from scratch.
Takeaways & Limitations
Generative AI may improve SPC practitioners’, educators’, and researchers’ efficiency and productivity, but outputs must be validated and used with other methods.
Takeaways & Limitations
The assessment focuses primarily on ChatGPT and its underlying GPT-3.5 engine as a benchmark for future evaluations.
Abstract
from arXiv · showhide
Generative Artificial Intelligence (AI) models such as OpenAI's ChatGPT have the potential to revolutionize Statistical Process Control (SPC) practice, learning, and research. However, these tools are in the early stages of development and can be easily misused or misunderstood. In this paper, we give an overview of the development of Generative AI. Specifically, we explore ChatGPT's ability to provide code, explain basic concepts, and create knowledge related to SPC practice, learning, and research. By investigating responses to structured prompts, we highlight the benefits and limitations of the results. Our study indicates that the current version of ChatGPT performs well for structured tasks, such as translating code from one language to another and explaining well-known concepts but struggles with more nuanced tasks, such as explaining less widely known terms and creating code from scratch. We find that using new AI tools may help practitioners, educators, and researchers to be more efficient and productive. However, in their current stages of development, some results are misleading and wrong. Overall, the use of generative AI models in SPC must be properly validated and used in conjunction with other methods to ensure accurate results.
1 Preface
The paper examines what large language models such as ChatGPT can and cannot do for SPC practice, learning, and research, focusing on their current capabilities and limitations.
- 1 Preface: The paper focuses on what large language models can and cannot do now because the technology is rapidly evolving and difficult to forecast.The authors also note that LLM research spans model details and ethical or legal considerations.
- 1 Preface: The study evaluates whether ChatGPT-like models can support SPC practice, learning/training, and research.The paper explicitly excludes ethical, legal, and philosophical issues from its examination.
- 1 Preface: ChatGPT’s usefulness is examined across applications relevant to practitioners, educators, and researchers.The paper frames the evaluation around whether LLM-based tools can augment these roles.
2 Introduction
The introduction situates generative AI within rapidly advancing language-model technology and motivates a focused evaluation of ChatGPT for SPC-related work.
- 2 Introduction: Generative AI differs from traditional predictive machine learning because it generates new content rather than analyzing existing datasets.Its outputs are stochastic, so the same input prompt can produce different content.
- 2 Introduction: Large language models can exhibit emergent abilities as model size increases, including multi-step arithmetic, college-level examinations, and word-meaning identification.State-of-the-art models can contain up to 175 billion parameters.
- 2 Introduction: ChatGPT and related tools have broadened access to generative AI by providing conversational language understanding and generation.Generative AI applications include translation, summarization, and content generation.
- 2 Introduction: Recent generative AI advances and adoption have created substantial business and investment hype.The cited estimate projects at least 10% greater efficiency or creativity for knowledge and creative workers, while investment reached $1.37 billion in 2022.
- 2 Introduction: The paper asks what generative LLM-based AI tools can do now to augment SPC practitioners, educators, and researchers.ChatGPT is selected because it is widely known and can generate both text and code.
A Representative Sample of Major Generative AI Developments from 2020 to Jan 2023
The section presents a representative sample of major generative AI developments and the public adoption associated with these tools through January 2023.
- A Representative Sample of Major Generative AI Developments from 2020 to Jan 2023: The figure’s representative sample covers major large-language-model releases and publications from the last three years.The source data are publicly available release dates of major AI developments.
- A Representative Sample of Major Generative AI Developments from 2020 to Jan 2023: The figure contrasts the unprecedented adoption of ChatGPT with adoption patterns for popular technology platforms.The caption identifies this as a comparison of ChatGPT adoption with other platforms.
- A Representative Sample of Major Generative AI Developments from 2020 to Jan 2023: Figure 1 links rapid generative-AI developments with public enthusiasm for trying and adopting these tools.The figure is described as covering both development and associated hype.
3 Background
The background traces language models from probabilistic sequence modeling through Transformer architectures, GPT-3, and ChatGPT’s conversational specialization.
- 3 Background: Language models assign probabilities to word sequences, building on Markov’s early demonstration that letter pairs are not independent.The section connects this probabilistic foundation to modern language-model development.
- 3 Background: Transformers use self-attention and parallel computing to weight important data components while reducing training time.The architecture established a foundation for current large language models.
- 3 Background: GPT-3 is an autoregressive language model trained with deep learning to generate human-like text.Its pretraining used almost 500 billion tokens and the model had 175 billion parameters.
- 3 Background: GPT-3 learns unlabeled text patterns by predicting the next word and optimizing model parameters with gradient descent.The model does not distinguish fact from fiction through labels; it learns relationships in the training data.
- 3 Background: Codex is a programming-focused, fine-tuned GPT-3 model that powers tools such as GitHub Copilot.Codex was trained using selected files from 54 million public GitHub repositories.
- 3 Background: ChatGPT is a pretrained GPT-3 instance optimized to generate human-like conversational responses.Its training process is presented in Figure 2.
4 Designing our Exploratory Study
The exploratory study evaluates ChatGPT across SPC practice, learning, and research through code-generation, concept-explanation, and knowledge-creation tasks, with responses assessed by experienced educators, authors, and statisticians. Its qualitative methods are shaped by evaluator experience and by the non-repeatability of generative-AI responses.
- The study examines ChatGPT’s ability to provide code, explain basic concepts, and create knowledge for SPC practice, learning, and research.
- Prompts cover control-chart creation, code explanation and translation, simulation development, SPC-term explanations, and new knowledge assets.Knowledge-creation tasks include an SPC framework, a course syllabus, and open research issues.
- Experienced educators, authors, and statisticians evaluated generated code by running it in R or Python, correcting execution errors, and assessing output correctness.The team also compared ChatGPT responses with current textbooks, although the supplied passage is truncated before describing that comparison fully.
- The methods are qualitative and influenced by the evaluators’ SPC experience and biases.
- ChatGPT responses to identical prompts are not repeated, so the study documents interactions through screenshots and related materials.The non-repeatability of responses is identified as a characteristic of generative-AI models such as ChatGPT.
5 On the Use of ChatGPT by SPC Practitioners
ChatGPT can generate useful SPC code, explanations, and practitioner-oriented frameworks, but its outputs require careful validation because it can produce executable errors, conceptual inaccuracies, and insufficient statistical detail.
- Code: ChatGPT generated R code for an Xbar chart, but incorrect qcc() arguments caused an error and initially produced the wrong chart.Changing chart to TRUE removed the error but yielded an individual chart; changing type from ‘xbar.one’ to ‘xbar’ produced the desired chart.
- Code: An alternate tidyverse implementation contained grouping, undefined-object, and geom_errorbar errors, and used inconsistent two-sigma versus three-sigma limits.The authors modified the code so its resulting limits matched those from qcc(), apart from rounding.
- Explanations: ChatGPT substantially mischaracterized Phase 1 and Phase 2 control charting by confusing retrospective model-building with prospective monitoring and reducing Phase 2 to variation monitoring.The paper states that Phase 1 includes retrospective process analysis and model development, whereas Phase 2 monitors prospectively for departures from that model.
- Explanations: ChatGPT often explained familiar SPC ideas correctly but mishandled important qualifications and boundaries in zero-state ARL, univariate, multivariate, and profile monitoring.Its zero-state ARL response described in-control ARL without explaining when counting begins, while other explanations omitted broader applicable measures or methods.
- Knowledge Creation: ChatGPT produced reasonable SPC frameworks and communication guidance, but its responses lacked sufficient statistical detail without human-supplied context.The authors contrast the one-sided prompting of an AI chatbot with a consultant’s ability to ask questions and tailor the response to the audience.
6 Learning SPC Fundamentals with ChatGPT
ChatGPT supported several SPC learning tasks, especially explaining existing code, translating code between languages, and generating course materials. However, its responses also contained coding errors, omissions, and imprecise explanations that required expert review.
- Code explanation: ChatGPT comprehensively explained uncommented R code, identifying its steps, functions, and use in creating an X-bar control chart.It omitted the meaning of the center line and dashed lower and upper control limits.
- Code translation: ChatGPT successfully converted tidyverse-heavy code into a base R approach and produced an equivalent plot.The conversion used rowMeans and a custom range function.
- Code translation: The generated base R code incorrectly used dplyr’s row_number() and consequently disrupted subgroup time ordering in the plot.The authors corrected the issue with row.names().
- Code translation: The Python translation of the original uncommented R code worked directly without modification.The prompt used the original code rather than the slightly incorrect base R version.
- SPC concepts: ChatGPT’s five-phase SPC explanation was not incorrect, but it omitted measurement-system analysis and practical guidance for observations outside control limits.The authors found its explanations of phases 4 and 5 sufficient.
- Course design: ChatGPT generated a largely reasonable 15-week undergraduate SPC syllabus covering theory, applications, software, control charts, capability analysis, improvement, experiments, and projects.The authors judged the course description, learning outcomes, and topic schedule adoptable without major changes, although “multi-vari study” was unclear.
7 On the Use of ChatGPT in SPC Research
The research-focused prompts showed that ChatGPT could identify some established SPC methods and generate executable-looking code, but its research outputs could be materially incorrect or fail in implementation. Its responses were also imprecise when handling specialized SPC concepts and open research issues.
- Code generation: ChatGPT’s EWMA Monte Carlo code ran without errors but produced incorrect ARL estimates because of truncated run lengths and improperly estimated control limits.The reported values were about 260.8, 255.8, 262.9, and 260.2, whereas corrected results were expected to be substantially different.
- Method identification: ChatGPT correctly named the mathematical method underlying ewma.arl() as Markov chain approximation.The answer followed a prompt asking about an R function for approximating two-sided EWMA-chart ARL.
- Code generation: The inverse EWMA task had generally correct solution steps, but the generated uniroot() code failed with “f(lower) is NA” because of an adverse interval choice.The task sought the control-limit factor for an in-control ARL of 500.
- Concept explanation: ChatGPT’s explanation of practitioner-to-practitioner variability was long and imprecise because it conflated variability across methods with variability under a fixed method configuration.The authors distinguish the literature’s concept from differences in chart type, subgroup design, limit calculation, outlier treatment, and software.
- Research ideation: ChatGPT generated a reasonable set of themes about open SPC research issues, but the authors viewed its main value as supporting high-level research-idea generation or validation.They cautioned that ideas could be stale because the GPT-3.5 model finished training in early 2022.
8 Discussion and Conclusions
The study finds that ChatGPT can support SPC code translation and brainstorming, but its outputs vary and may lack the precision required for reliable explanations or ground-truth tasks. The authors therefore frame generative AI as a developing tool whose SPC applications require validation and further investigation.
- Our Perspective on ChatGPT’s Ability: ChatGPT’s responses can vary in wording, content, and number of issues across similar prompts, benefiting open-ended tasks but complicating situations requiring a ground truth.The authors call for further study of prompts with singular answers.
- Our Perspective on ChatGPT’s Ability: ChatGPT appears more proficient in Python than R, can explain some functions, and can translate widely used functions between programming languages.The authors suggest code translation may help deploy SPC methods in production, but any generated code requires validation.
- Our Perspective on ChatGPT’s Ability: The current version of ChatGPT lacks the precision needed to correctly, consistently, and succinctly explain SPC concepts.The paper reports both lower-quality explanations and made-up descriptions, including for the “bigfish” dataset.
- Our Perspective on ChatGPT’s Ability: ChatGPT can serve as an idea-generation or brainstorming tool, although the emphasis assigned to topics can differ when no single answer exists.This finding applies to responses about syllabi and open research areas.
- Our Perspective on ChatGPT’s Ability: The authors recommend validating ChatGPT outputs with multiple sources and experts, including checking both execution and expected behavior for generated code.They do not favor ChatGPT for non-code explanations but consider it appropriate for knowledge creation when treated as initial brainstorming.
- Open Questions: The paper identifies open SPC questions concerning output-quality evaluation, bias, privacy and data security, customization, and the ethical use of generative AI.The authors propose evaluating variable outputs, examining bias after release, and testing whether SPC-specific training improves answers.
- Discussion and Conclusions: Generative AI is presented as a rapidly evolving technology that may change SPC practice, learning, and research while currently improving efficiency for tasks such as code translation and brainstorming.The authors describe current models as being in their infancy and expect their use to expand.
Online Materials
The online materials document the expository study’s ChatGPT interactions, figures, and code-related assessments.
- Online Materials: The GitHub repository contains study code and data, screenshots of ChatGPT interactions, practice-section code assessments, and videos of prompt responses.These materials document the study’s outputs, which cannot be reproduced exactly because ChatGPT responses are inherently stochastic.