PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice

Vinay Pursnani, Yusuf Sermet, Ibrahim Demir

arXiv:2304.12198v1cs.CYcs.AI

TL;DR

Evidence on ChatGPT’s engineering performance, particularly in environmental engineering, was limited, despite growing interest in AI for education and licensing exams. This study assessed ChatGPT on FE Environmental Exam questions using different models and noninvasive prompt refinements. The refined GPT-4 model reached 75.37% overall accuracy, while the authors note limits on interpreting this as a passing result.

  • Problem

    Research had not established ChatGPT’s performance on engineering questions, especially in environmental engineering, a domain involving complex mathematics and problem-solving.

    Source: §1.1 · p. 4

  • Method

    The study tested ChatGPT models on FE Environmental Exam questions presented verbatim or with noninvasive prompt refinements.

    Source: §1 · p. 2, §2.4 · p. 7

  • Results

    The GPT-4 base model’s accuracy rose by a mean 8.95% with noninvasive prompt modifications, resulting in 75.37% overall accuracy.

    Source: §4 · p. 16

  • Takeaways & Limitations

    The findings provide a basis for further research on responsible and effective integration of AI models in education.

    Source: §1 · p. 2

    Because NCEES publishes no tangible pass/fail score, the study cannot establish whether ChatGPT passes the FE Exam.

    Source: §2.5 · p. 8

Abstract

from arXiv · show

In recent years, advancements in artificial intelligence (AI) have led to the development of large language models like GPT-4, demonstrating potential applications in various fields, including education. This study investigates the feasibility and effectiveness of using ChatGPT, a GPT-4 based model, in achieving satisfactory performance on the Fundamentals of Engineering (FE) Environmental Exam. This study further shows a significant improvement in the model's accuracy when answering FE exam questions through noninvasive prompt modifications, substantiating the utility of prompt modification as a viable approach to enhance AI performance in educational contexts. Furthermore, the findings reflect remarkable improvements in mathematical capabilities across successive iterations of ChatGPT models, showcasing their potential in solving complex engineering problems. Our paper also explores future research directions, emphasizing the importance of addressing AI challenges in education, enhancing accessibility and inclusion for diverse student populations, and developing AI-resistant exam questions to maintain examination integrity. By evaluating the performance of ChatGPT in the context of the FE Environmental Exam, this study contributes valuable insights into the potential applications and limitations of large language models in educational settings. As AI continues to evolve, these findings offer a foundation for further research into the responsible and effective integration of AI models across various disciplines, ultimately optimizing the learning experience and improving student outcomes.

1. Introduction

The FE Environmental Exam is a consequential professional licensing assessment, while AI use in education and testing raises questions about exam credibility. Prior research had not established ChatGPT’s performance in engineering, particularly environmental engineering, motivating this study of its exam performance and potential uses.

1. Introduction

  • The FE Environmental Exam covers environmental topics and is an important certification for professionals seeking licensure and career advancement.It assesses areas including air and water quality, waste management, regulations, and sustainability.Source: §1 · p. 2
  • AI’s growing role in education and testing makes its possible use in licensing exams an issue requiring investigation, including concerns about exam credibility.The passage notes arguments about cheating and accountability as well as potential gains in objectivity and fairness.Source: §1 · p. 2
  • The study examines ChatGPT’s potential on the FE Environmental Exam, including differences among models and the effect of noninvasive prompt modifications.It also frames the work as informing responsible integration of large language models in education.Source: §1 · p. 2

1.1. Literature Review

  • Prior studies examined ChatGPT and other AI tools in several educational domains, but evidence on ChatGPT’s engineering performance remained limited.The review describes work in medicine, computer science, and business alongside the engineering research gap.Source: §1.1 · pp. 3–4
  • Environmental engineering’s breadth and multidisciplinary character leave a specific gap in evidence about ChatGPT’s performance on its questions.Evaluating the model in this domain can inform judgments about its applicability and limitations.Source: §1.1 · p. 4

2. Methodology

The study assesses ChatGPT on FE Environmental Exam questions using verbatim and noninvasively refined prompts, while documenting the dataset, model context, evaluation process, and limits on interpreting results. Because the exam has no published tangible pass/fail score and the dataset and model outputs have constraints, the study avoids definitive claims about passing or generalizability.

2.1. Scope and Objectives

  • The study asks how accurately ChatGPT answers FE Environmental CBT questions, whether it is feasible for preparation and assessment, and what benefits and concerns accompany AI use in licensing exams.Its stated stakeholders include engineering students, professionals, and NCEES.Source: §2.1 · p. 5

2.2. AI Background

  • ChatGPT is described as a conversational AI model trained on large text collections, with successive GPT iterations intended to improve capabilities including mathematical performance.The text also notes that model accuracy can vary by task and data.Source: §2.2 · pp. 5–6

2.3. Exam Overview and Data Collection

  • The 134-question practice dataset was faculty-curated, but its subject-area distribution was not representative of an actual exam.The dataset was prepared for students to find questions, worked solutions, and problem-solving videos.Source: §2.3 · pp. 6–7, Figure 1 · p. 7

2.4. Study Design

  • The study compared verbatim questions across GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4 with noninvasively refined questions tested on GPT-4.A separate GPT-3.5-Turbo experiment examined temperature; questions were run in unique calls, and results were autograded then manually validated.Source: §2.4 · pp. 7–8, Figure 2 · p. 8

2.5. Study Limitations

  • Because NCEES does not publish a tangible pass/fail score, the study cannot establish whether ChatGPT passes the FE exam.The authors also could not compare section-specific success with human test takers because diagnostic reports are not public.Source: §2.5 · p. 8
  • The dataset may be small or unrepresentative, lacks point-and-click questions, and may include questions previously posted online.The authors also note sensitivity to prompt wording and random outputs, so reproduced accuracy rates may fluctuate.Source: §2.5 · pp. 8–9

3. Results

On this 134-question dataset, refined GPT-4-base accuracy exceeded unrefined GPT-4-base accuracy, while performance varied by exam section. Prompt refinements improved overall accuracy but did not help uniformly, and mathematical results varied across model generations and calculation-intensive questions.

3. Results

  • GPT-4-base accuracy rose from 66.42% to 75.37% with refinement on the dataset; no quantitative comparison with human test-takers was made.The questions were not invasively modified for ChatGPT, and the model had not been trained on the dataset beforehand.Source: §3 · p. 10

3.1. Effects of Temperature Configuration in Engineering Questions

  • With GPT-3.5-Turbo, accuracy was highest at TEMP=0.75 (0.59) and lowest at TEMP=0.5 (0.47); TEMP=1 and TEMP=0 each scored 0.56, and TEMP=0.25 scored 0.53.The analysis covered all 134 dataset questions; the authors note that other contexts or sections may require further temperature testing.Source: §3.1 · p. 11, Figure 5 · p. 11

3.2. Utility of Noninvasive Prompt Refinements

  • Only the clear-and-concise, answer-with-explanation system-message refinement proved beneficial; LaTeX formatting and textual diagram descriptions did not yield significant or consistently accurate improvements.The study used the system-message approach, which asks for an explanation and an educated choice from the options when appropriate.Source: §3.2 · p. 12
  • Refined prompts increased mean accuracy by 8.95 percentage points versus unmodified prompts, with gains across theoretical concepts, problem-solving tasks, and practical applications.The refinements were associated with more concise and accurate answers and fewer irrelevant or ambiguous responses.Source: §3.2 · pp. 12–13, Figure 6 · p. 13
  • Refinement did not improve Air Quality and Control, Ground Water Soil and Sediments, Solid and Hazardous Wastes, Surface Water Resources and Hydrology, or Thermodynamics, and performance decreased in Ethics and Professional Practice and Water and Wastewater.The authors call for section- or topic-tailored refinements; they suggest image-description needs and multi-topic or hydrogeological complexity may limit performance in some sections.Source: §3.2 · p. 14

3.3. Reliability and Mathematical Capability of ChatGPT for Engineering Questions

  • Weighted accuracy across math-heavy sections rose from 41.38% (GPT-3.5-legacy) to 62.07% (GPT-3.5-turbo), 65.51% (refined GPT-3.5-turbo), 65.51% (GPT-4-base), and 86.21% (refined GPT-4-base).The math-heavy set included Math, Probability and Statistics, and Engineering Economics (n=29).Source: §3.3 · p. 14, Figure 7 · p. 15
  • Refined GPT-4-base could identify relevant formulas but remained unreliable on multi-step calculations, while some correct GPT-3.5 multiple-choice answers appeared fortuitous.The authors identify multi-step calculations and intricate mathematical operations as constraints, with fluctuating accuracy particularly in Engineering Economics and Mathematics.Source: §3.3 · pp. 14–15

4. Discussion and Conclusions

GPT-4 without vision reached 66.42% accuracy on the FE Environmental Exam dataset, rising to 75.37% with noninvasive prompt modifications; comparisons also found improved mathematical capabilities in later ChatGPT models. The discussion weighs educational uses against reliability and exam-integrity concerns, and proposes research on tools, learning support, integrity, and inclusion.

4. Discussion and Conclusions

  • Noninvasive prompt modifications raised GPT-4 base-model mean accuracy by 8.95% to 75.37% on the FE Environmental Exam.The unmodified base model without vision scored 66.42% overall.Source: §4 · p. 16
  • Later ChatGPT generations showed improved mathematical capabilities for engineering applications, with GPT-4 without vision outperforming predecessors.The discussion notes occasional calculation errors despite correct formula representations, and describes Python execution as producing a correct answer in an exam example.Source: §4 · p. 16, §4.1 · p. 17
  • The authors warn that AI exam use raises concerns about exam credibility, accountability, response validity, and training-data errors.They call for careful investigation to ensure licensing-exam use does not undermine credibility or create new preparation challenges.Source: §4 · pp. 16–17

4.1. Future Work and Opportunities

  • Future work proposes integrating code interpreters and specialized engineering tools, as well as using ChatGPT explanations and generated practice-question variations for FE preparation.The authors note that prompting ChatGPT to write Python code yielded a correct answer in an example where calculations were sometimes mistaken.Source: §4.1 · p. 17
  • The authors propose research on AI-assisted exam preparation and assessment, cheating detection, and the effects of AI on licensing-exam credibility and fairness.Suggested cheating-detection features include keystroke analysis, biometrics, and facial recognition.Source: §4.1 · pp. 17–18
  • AI-resistant custom exam questions are proposed as a way to maintain examination integrity to some degree while retaining AI's educational benefits.The paper informally calls these questions “Captcha for exam questions.”Source: §4.1 · p. 18
  • The authors propose preparing students for AI-related biases and misuse, while exploring multilingual and tailored materials to improve accessibility for diverse learners.They identify disadvantaged engineers and people whose first language is not English as potential beneficiaries of tailored educational materials.Source: §4.1 · p. 18

Data Availability

The dataset's raw questions are confidential because of copyright assurances. The authors report no external funding.

Data Availability

  • The dataset provider was assured that raw questions would remain confidential because the questions are copyrighted.The authors state that the data are not available.Source: Data Availability · p. 18
  • The authors credit Vinay Pursnani with software, methodology, data curation, formal analysis, and validation.Yusuf Sermet and Ibrahim Demir are also credited with specified contributions.Source: Data Availability · p. 19

Funding

Appendix

The appendix compares the FE Environmental Exam's section distributions with those in the study dataset. The listed subject rows show that some dataset proportions differ from the exam's ranges.

Appendix

  • The table lists each subject's exam weight and question range alongside its dataset weight and question count.It compares section distributions rather than reporting model-performance results.Source: Appendix · p. 22, Table A1 · p. 22
  • Mathematics accounts for 4.5–7.2% of exam questions (5–8 questions), compared with 11.19% (15 questions) in the dataset.Source: Appendix · p. 22
  • Water and Wastewater accounts for 13–19% of exam questions (12–18 questions), compared with 3.73% (5 questions) in the dataset.Source: Appendix · p. 22
Loading 2304.12198v1…