PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correctness: How Novice Software Engineering Students Exercise Oversight of Generative AI

Ali Shakiba, Thomas Chaffey

arXiv:2610.32118v2cs.SEcs.HC

TL;DR

Evidence about novice students’ engineering judgements during GenAI use remains limited, even as their work shifts toward evaluating and supervising generated outputs. The paper analyses journaled interactions and reported interventions using a multidimensional coding framework. Students’ reported oversight extends beyond detecting errors to judging fitness for purpose and acting on evaluations.

  • Problem

    Less is known about which characteristics novice students identify in AI outputs, why they intervene, and how their judgements translate into responses during authentic software engineering work.

    Source: §I · p. 1, §II · p. 3

  • Method

    The study analyzes AI Journal interactions and student-declared interventions from first-year coursework, coding output characteristics, rationales, actions, and operational mechanisms.

    Source: §I · p. 2, §II · p. 3, B. Dataset and Unit of Analysis · p. 4

  • Results

    Reported oversight extended beyond error detection: students judged engineering fitness and responded through artefact modification, prompt redirection, clarification seeking, or rejection.

    Source: §VII · p. 9

  • Takeaways & Limitations

    Understanding novice oversight requires attention to both students’ fitness-for-purpose judgements and how they respond when outputs do not meet engineering-task needs.

    Source: §VII · p. 9

    The journal captures student-declared rather than independently observed oversight, and intervention reporting was required only for outputs rated imperfect.

    Source: §VI · p. 8

Abstract

from arXiv · show

Generative AI (GenAI) is shifting software engineering students' work from producing artefacts towards evaluating and supervising AI-generated outputs. However, little is known about what characteristics students identify in outputs when reporting interventions, or how they exercise oversight during authentic software engineering work. We analysed 962 student-AI interaction turns documented by 145 first-year software engineering students through structured AI Journals, including 172 student-declared interventions. Using a multi-dimensional coding framework, we characterised the output characteristics identified in these reported interventions, students' reasons for intervening, the actions they took, and how those actions were enacted. Fitness problems were identified in 74.4% of interventions, compared with 16.9% involving correctness problems, indicating that oversight more often concerned whether plausible AI outputs were suitable for the engineering task than whether they contained explicit errors. Among classifiable interventions, students most frequently modified generated outputs (51.0%) or refined their prompts (28.2%), with both response types commonly associated with engineering fit. Within-student analysis further showed that outputs perceived as containing minor flaws were associated with lower evaluation confidence than outputs perceived as perfect. Taken together, these findings characterise novice AI oversight as a process that extends beyond error detection to encompass both judgement of fitness for purpose and active intervention in AI-assisted engineering work.

I. INTRODUCTION

As GenAI shifts students toward supervising generated artefacts, novices must judge whether fluent outputs merit scrutiny and how to respond. The paper examines reported interventions in authentic coursework to characterize students’ judgements and actions.

I. INTRODUCTION

  • Novices may struggle to evaluate fluent, plausible outputs, creating risks of cognitive offloading and uncritical adoption.The introduction frames oversight as deciding when outputs require scrutiny and how to respond.Source: §I · p. 1
  • Evaluative judgement concerns assessing work quality, while human-in-the-loop perspectives examine how such judgement becomes actions such as modifying or rejecting AI outputs.The paper presents these as complementary lenses on student oversight.Source: §I · p. 1
  • Prior process research documents prompting, inspection, verification, modification, and iteration, but the engineering judgement underlying reported interventions remains less understood.The gap includes which output characteristics students identify, why they intervene, and how judgements translate into responses, especially beyond code correctness.Source: §I · p. 2
  • The study analyses AI Journal entries from 145 first-year students in a design-first, low-code course, coding interventions across output characteristics, rationales, actions, and mechanisms.It also examines how confidence varies with perceived output correctness.Source: §I · p. 2
  • The paper contributes an empirical characterization distinguishing correctness from fitness problems, a multidimensional framework of oversight, and educational implications for evaluating artefact fitness and making judgement visible.The paper also outlines its related-work, methods, results, discussion, limitations, and conclusion sections.Source: §I · p. 2

II. RELATED WORK

Prior research describes how people interact with GenAI, particularly in programming, but leaves the engineering judgements behind reported interventions less understood. This study extends attention to whether outputs fit broader engineering contexts and how students act on their evaluations.

II. RELATED WORK

  • Process-level studies describe prompting, inspecting, verifying, modifying, and iterating with GenAI, while documenting varied novice engagement.Examples include acceleration versus exploration and repeated cycles of requesting code revisions.Source: §II · p. 2
  • Much prior process evidence concerns code tasks assessed against function or specifications, whereas upstream engineering artefacts also require judgements about completeness, scope, stakeholder alignment, and constraints.A plausible artefact may still be unsuitable for its engineering context.Source: §II · pp. 2–3
  • Evaluative judgement frames AI oversight as assessing whether plausible outputs are fit for purpose, a capability that may remain invisible in final artefacts or assessment.It develops through standards, comparison, and criterion-referenced feedback rather than tool access alone.Source: §II · p. 3
  • Human-in-the-loop perspectives emphasize meaningful opportunities to evaluate, redirect, modify, override, or reject outputs, rather than merely assigning people a nominal supervisory role.Meaningful involvement depends on retaining the capacity and opportunity to exercise judgement when intervention is needed.Source: §II · p. 3
  • Research leaves less understood which artefact characteristics prompt intervention, why students intervene, and how their judgements translate into responses, particularly beyond code-generation tasks.The study addresses this gap through analysis of reported oversight across characteristics, rationales, actions, and operational mechanisms.Source: §II · p. 3

III. METHODOLOGY

The study took place in a first-year, design-first, low-code software engineering course where contextual suitability and system constraints were salient. Structured journals recorded student-AI interactions and reflective evaluations, with intervention descriptions required for outputs judged imperfect.

A. Research Context and the AI Journal

  • The research was conducted in a first-year University of Sydney course emphasizing requirements, stakeholder analysis, design, and prototyping over conventional source-code programming.Students used tools such as PowerApps and Figma; coursework often involved contextual suitability and system constraints.Source: A. Research Context and the AI Journal · p. 3
  • Students completed structured AI Journals as a hurdle requirement rather than a graded component, documenting prompts, AI responses, interaction purpose, accuracy assessments, verification, and five-point evaluation confidence.The ungraded status reduced, but did not eliminate, incentives to optimize reflections for marks.Source: A. Research Context and the AI Journal · p. 3
  • Students had to describe interventions when they rated outputs as containing minor flaws or major hallucination errors; journals also captured reflections on interaction processes and perceived utility.The records combine documented interactions with self-reported evaluations, interventions, and reflections.Source: A. Research Context and the AI Journal · pp. 3–4

B. Dataset and Unit of Analysis

The analysis uses student-declared interventions for its first two research questions and self-reported confidence ratings for the third. Its coding framework separates what prompted an intervention from why and how students responded.

B. Dataset and Unit of Analysis

  • The dataset contains 962 student-AI interaction turns from 145 students across ten weeks, with prompts, responses, and reflective fields.For RQ1 and RQ2, interventions were required descriptions when students rated outputs as having minor flaws or major hallucinations.Source: B. Dataset and Unit of Analysis · p. 4
  • RQ1 and RQ2 analyze student-declared interventions alongside the corresponding prompt, response, and reflective fields, while RQ3 relates five-point confidence ratings to perceived output correctness.Intervention declarations are analyzed only when students evaluated an output as containing minor flaws or major hallucinations.Source: B. Dataset and Unit of Analysis · p. 4

C. Human Oversight Coding Framework

  • The coding framework was developed iteratively by reviewing intervention records, drafting codes and decision rules, then refining categories to resolve ambiguities.Source: C. Human Oversight Coding Framework · p. 4
  • The framework distinguishes four dimensions: output trigger, engineering intent and rationale, intervention type, and operational mechanism.Together they capture what prompted action, why the student acted, what action was taken, and how it was enacted.Source: C. Human Oversight Coding Framework · p. 4
  • Triggers reflect students’ characterizations, distinguishing correctness problems from fitness problems and cases with no identifiable failure.Correctness concerns validity, while fitness concerns suitability for project scope, stakeholder needs, platform constraints, or assessment requirements; both codes could apply.Source: C. Human Oversight Coding Framework · pp. 4–5

D. Coding Procedure

The authors reconciled independent coding of interventions, then analysed triggers, action patterns, and students’ confidence in evaluating AI outputs using procedures tailored to each research question.

D. Coding Procedure

  • 90.6% agreement was achieved for intervention type coding, and disagreements across dimensions were resolved through discussion before analysis.IT agreement was Cohen’s κ = 0.865 (Gwet’s AC1 = 0.886); exact-set agreement was 94.7% for OM, 91.3% for IR, and 92.1% for TR.Source: D. Coding Procedure · p. 5

E. Data Analysis

  • For RQ1, trigger prevalence was calculated across interventions because multiple triggers could be assigned to one intervention.Journal records were also examined for representative examples that contextualised the trigger patterns.Source: E. Data Analysis · p. 5
  • For RQ2, the four substantive action types were compared after excluding insufficiently described actions, while rationale and mechanism codes were analysed as multi-label co-occurrences.Co-occurrences were normalised within intervention type, and a dimension received no code when evidence was insufficient.Source: E. Data Analysis · p. 5
  • For RQ3, inferential comparisons included outputs rated perfect or as having minor flaws, but excluded five major-hallucination cases as too few for reliable estimation.Confidence was centred within student for descriptive visualisation.Source: E. Data Analysis · p. 5
  • A student-random-intercept mixed-effects model separated within-student from between-student associations between perceived flaws and evaluation confidence.The within-student component estimates the association independently of students’ general tendency to rate outputs as flawed.Source: E. Data Analysis · p. 5
  • An ordinal-logistic sensitivity analysis accounted for confidence’s five-point ordinal scale and reported the corresponding odds ratio alongside the mixed-model estimate.Standard errors were clustered by student.Source: E. Data Analysis · p. 5

IV. RESULTS

Among student-declared interventions, fitness problems were identified more often than correctness problems. Fitness problems concerned task suitability, including missing context and other mismatches with engineering needs.

A. RQ1: Characteristics of AI Outputs Identified in Reported Interventions

  • 74.4% of interventions involved a fitness problem, compared with 16.9% involving a correctness problem; 15.1% had no identifiable failure.Correctness and fitness could co-occur, so these percentages do not sum to 100%.Source: A. RQ1: Characteristics of AI Outputs… · p. 5
  • Fitness problems included outputs that were generic, incomplete, unnecessarily complex, insufficiently contextualised, or misaligned with intended system scope.Such outputs were not necessarily incorrect, but were unsuitable for the immediate engineering task.Source: A. RQ1: Characteristics of AI Outputs… · p. 5
  • A student designing a medical-device workflow added device and task information after finding the generated response lacked project-specific context.Source: A. RQ1: Characteristics of AI Outputs… · p. 6

B. RQ2: Student Oversight Behaviours

Students most often modified AI-generated outputs or refined prompts, with these actions commonly linked to engineering fit. Clarification seeking instead centred on comprehensibility and specification adjustment.

B. RQ2: Student Oversight Behaviours

  • 51.0% of classifiable interventions were output modifications and 28.2% were prompt refinements; clarification seeking accounted for 13.4% and rejection for 7.4%.The distribution excludes 23 interventions with insufficient descriptions; Figure 1 shows intervention types across all 172 declared interventions, including IT5.Source: Figure 1 · p. 6, B. RQ2: Student Oversight Behaviours · p. 6
  • Engineering fit was identified in 63.2% of output modifications and 69.0% of prompt refinements, while comprehensibility characterized 85.0% of clarification-seeking interventions.Source: B. RQ2: Student Oversight Behaviours · p. 6
  • Output transformation characterized 88.2% of output modifications; prompt refinement combined transformation and specification adjustment, while clarification seeking most often involved specification adjustment.Specification adjustment occurred in 75.0% of clarification-seeking interventions; evidence grounding and error diagnosis were comparatively uncommon overall.Source: B. RQ2: Student Oversight Behaviours · p. 6
  • Across response types, students most often exercised oversight by modifying generated outputs or refining prompts; clarification seeking had a distinct orientation toward comprehensibility.Source: B. RQ2: Student Oversight Behaviours · p. 6

C. RQ3: Confidence and Perceived Output Correctness

Students reported lower evaluation confidence when they perceived minor flaws rather than perfect outputs. Major hallucination errors were rare and too few for inferential comparisons.

C. RQ3: Confidence and Perceived Output Correctness

  • 790 outputs (82.2%) were rated PERFECT, 167 (17.3%) MINOR FLAWS, and five (0.5%) MAJOR HALLUCINATION ERRORS across 962 turns.The five major-error cases, reported by four students, had a mean raw confidence score of 4.60 out of 5 and were retained for descriptive presentation only.Source: C. RQ3: Confidence and Perceived… · pp. 6–7
  • A 0.43-point within-student confidence reduction was associated with outputs perceived as containing minor flaws rather than perfect (βW = −0.434, 95% CI: [−0.543, −0.325], p < 0.001).The within-student association remained robust when confidence was modelled as ordinal (OR = 0.289, 95% CI: [0.165, 0.507], p < 0.001).Source: C. RQ3: Confidence and Perceived… · p. 6
  • Students who more frequently rated outputs as containing minor flaws also tended to report lower confidence overall (βB = −0.721, 95% CI: [−1.095, −0.348], p < 0.001).Figure 3 displays within-student mean-centred confidence by perceived correctness; zero marks a student’s mean, with negative and positive values indicating lower- and higher-than-usual confidence.Source: C. RQ3: Confidence and Perceived… · p. 6, Figure 3 · p. 7

A. Oversight as Judgement of Fitness

Novice oversight focused more often on whether AI outputs fit engineering requirements than on whether they were correct. Students acted on that judgement through multiple forms of intervention, while perceived minor flaws were associated with lower confidence.

A. Oversight as Judgement of Fitness

  • Fitness problems occurred in 74.4% of interventions, compared with correctness problems in 16.9%, extending oversight beyond error detection to suitability for the intended engineering use.Relevant considerations include project scope, stakeholder needs, platform constraints, and implementation context.Source: A. Oversight as Judgement of Fitness · p. 7

B. Oversight as Active Engineering Agency

  • Output Modification (51.0%) and Prompt Refinement (28.2%) were the most common classifiable interventions and were most frequently associated with Engineering Fit.Modification puts the student in control of the artefact, while prompt refinement changes the constraints and context guiding AI output; clarification seeking was associated with comprehensibility.Source: B. Oversight as Active Engineering… · pp. 7–8

C. Evaluative Confidence and Imperfect Outputs

  • Within students, outputs perceived as containing minor flaws were associated with lower confidence than outputs perceived as perfect, and the association remained robust with ordinal modelling.Students who more often perceived minor flaws also tended to report lower confidence overall; only five major-hallucination cases were available, so their confidence response could not be determined.Source: C. Evaluative Confidence and Imperfect… · p. 8

D. Implications for Software Engineering Education

The paper recommends making oversight an explicit software engineering competency and assessing the judgement behind AI-assisted work. Its findings are bounded by self-reported, scaffolded journal activity and a single course setting.

D. Implications for Software Engineering Education

  • Instruction should develop judgement of fitness for purpose and a range of strategies, while process-oriented assessment can make students’ oversight visible for feedback.Students can document what prompted an intervention, why they intervened, and how they responded; evaluating only the final artefact offers limited evidence of critical evaluation.Source: D. Implications for Software… · p. 8

VI. LIMITATIONS

  • AI Journals capture student-declared oversight in a scaffolded reflective activity, so reported interventions may be incomplete and absence of a report does not establish that none occurred.Intervention reporting was required for outputs rated non-perfect, potentially excluding adaptations to outputs rated perfect.Source: §VI · p. 8
  • Category distinctions involved interpretive judgement, and high inter-rater agreement does not eliminate that judgement.The framework was applied through independent coding with explicit rules, reliability assessment, and discussion to resolve disagreements.Source: §VI · p. 8
  • The confidence analysis uses students’ self-reported perceptions and may miss oversight of outputs initially rated perfect; five severe-error cases support no inference about confidence under severe failure.The study was conducted in one first-year, design-first, low-code course, so findings should not be assumed to generalise directly to programming-intensive courses or professional settings.Source: §VI · p. 9

VII. CONCLUSION

In authentic semester-long coursework, novice oversight encompassed both error detection and judgments of engineering fitness, followed by interventions such as modifying outputs or redirecting the AI. The authors call for triangulating journal accounts with interaction traces and testing whether the correctness–fitness distinction persists in programming-intensive and professional contexts.

VII. CONCLUSION

  • Students reported lower evaluation confidence for outputs perceived to contain minor flaws than for perfect outputs.Reported responses also included modifying generated artefacts, refining prompts, seeking clarification, and rejecting outputs.Source: §VII · p. 9
  • Novice oversight extended beyond error detection to judging engineering fitness and responding through modification, prompt redirection, clarification, or rejection.The study examined oversight during authentic, semester-long coursework.Source: §VII · p. 9
  • Future work should triangulate reflective journal accounts with direct interaction traces and test whether the correctness–fitness distinction persists in programming-intensive and professional settings.Source: §VII · p. 9

ETHICS

  • The study received approval from the University of Sydney Human Research Ethics Committee.Approval No. HREC 2025/HE001132-substudy 022 v1a.Source: ETHICS · p. 9

DATA AVAILABILITY

  • Student interaction records are not publicly available because ethics approval restricts dissemination to de-identified aggregate results; the coding framework and procedures appear in Section III.Source: DATA AVAILABILITY · p. 9
Loading 2610.32118v2…