A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions

• The study addresses limited understanding of how developers validate and repair LLM-generated code, including whether provenance changes their strategies and performance.Source: §I · pp. 1–2
• Eye tracking, IDE tracking, workload questionnaires, and interviews were combined to capture developers’ behavioral and cognitive processes during validation and repair.Source: §I · p. 2
• The authors conducted a lab study with 28 participants validating, repairing, and integrating Copilot-generated code across three projects, with informed and non-informed provenance groups.Source: §I · p. 2
• Developers often fail to recognize LLM-generated code unless explicitly informed, although it typically has stronger syntax but weaker logic performance than human-written code.Source: §I · p. 2
• Developers generally used similar validation and repair strategies for LLM-generated and human-written code, but frequently switched between code and comments, focused differently, and deleted and rewrote code.Source: §I · p. 2
• Provenance awareness improved validation and repair outcomes while increasing search efforts, Copilot use, and cognitive workload because developers attended more closely to errors and integration challenges.Source: §I · p. 2

A. Empirical Study of LLM Code Generation Tools

• Offline evaluations show strong but incomplete code-generation performance, motivating further study of how developers use LLM tools and how those tools affect developers.Source: A. Empirical Study of LLM Code… · p. 3
• Prior studies describe Copilot use patterns, usability issues, and substantial time spent validating and editing generated suggestions, while this study focuses specifically on validation and repair.Source: A. Empirical Study of LLM Code… · p. 3

B. Debugging Strategies for Software Developers

• Debugging research models activities such as code comprehension, fault localization, repair, and verification, but generated code may require different processes because it differs from human-written code.Source: B. Debugging Strategies for Software… · p. 3

C. Eye Tracking in Software Engineering

• Eye tracking provides gaze-based indicators of visual attention and cognitive processes, and can be integrated with IDE interaction logs to study software development behavior.Source: C. Eye Tracking in Software Engineering · p. 3
• Earlier provenance studies found behavioral differences for code labeled as human- or machine-produced; this work applies IDE and eye tracking to provenance effects during validation and repair.Source: C. Eye Tracking in Software Engineering · p. 3

III. STUDY DESIGN

• Twenty-eight participants received identical prompts and Copilot-generated code, then validated and repaired it with Copilot assistance under randomly assigned provenance-awareness conditions.Source: §III · pp. 3–4

A. Programming Tasks

• Figure 2 illustrates ZooSystem bugs 6125, Parameter Value, and 6112, Wrong Component, in a generated Lion constructor.Source: Figure 2 · p. 4, A. Programming Tasks · p. 4
• The three Java tasks represented algorithm design, GUI design, and object-oriented programming, with Copilot generating complete programs or selected subroutines.Source: A. Programming Tasks · p. 4
• The ZooSystem task included generated management functions and bugs classified using Beizer’s taxonomy, with subtasks potentially containing multiple bug types.Source: A. Programming Tasks · p. 4

B. Participants

• The sample comprised 28 university participants with varied Java experience, though most were undergraduate or graduate computer-science students.Source: B. Participants · p. 4

C. Study Settings

• Participants were randomly split into 14-person Informed and Non-Informed groups, using IntelliJ IDEA, Copilot, and standardized laboratory eye-tracking conditions.Source: C. Study Settings · p. 4

D. Protocol

• Each session lasted about two hours; participants completed a tutorial, calibrated the eye tracker, and had 20 minutes per task without Internet access.Source: D. Protocol · p. 5
• After each task, NASA-TLX measured self-reported workload, and a final semi-structured interview examined code evaluations, repair strategies, tool use, and provenance awareness.Source: D. Protocol · p. 5

E. Data Collection Setup

• CodeGRITS recorded IDE features, keyboard and mouse events, file contents, console output, screen timestamps, and eye-tracking data for behavioral analysis.Source: E. Data Collection Setup · p. 5, 1) IDE Tracking · p. 5

2) Eye Tracking:

• CodeGRITS mapped gaze data to semantic source-code entities, including tokens and abstract syntax-tree nodes.The system sampled gaze at 60 Hz and associated each sample with source-code semantics.Source: 2) Eye Tracking · p. 5
• Each gaze record included screen coordinates, pupil diameters, validity codes, and timestamps, with invalid data excluded from analysis.Source: 2) Eye Tracking · p. 5
• The mapping pipeline converted relative editor locations into file lines and columns, then identified token types and AST hierarchies through IntelliJ PSI.Source: 2) Eye Tracking · p. 5
• Listing 2 illustrates the collected gaze data together with its computed location, token, and AST information.Source: 2) Eye Tracking · p. 6

IV. RESEARCH METHODS

• The paper analyzes the data collected from the study using the methods described in the following subsections.Source: §IV · p. 6

1) Gaze Pattern Metrics:

• Fixations were identified as gazes on the same token lasting over 200 ms, while transitions between fixations within 50 ms were treated as saccades.Source: 1) Gaze Pattern Metrics · p. 6
• Gaze-pattern analysis measured fixation time and count, average fixation duration, saccade time and count, average saccade length, and the saccade-fixation ratio.The ratio represents the balance between exploration and exploitation during programming.Source: 1) Gaze Pattern Metrics · p. 6

2) Developer Behavior Categorization:

• The data collector’s low-level IDE events were filtered and aggregated into higher-level developer-behavior categories.For example, run-related and debugger-related events were grouped, while events without clear behavioral significance were discarded.Source: 2) Developer Behavior Categorization · p. 6
• Eye-tracking fixations were classified by whether developers looked at code, comments, or documents, producing three reading-behavior categories.Source: 2) Developer Behavior Categorization · p. 6
• IDE interaction categories were independently reviewed and reconciled by the authors to establish cohesive behavior definitions.Source: 2) Developer Behavior Categorization · p. 6

B. Qualitative Analysis

• The study used open coding and thematic analysis of interview transcripts to refine themes into qualitative findings.Source: B. Qualitative Analysis · p. 6

A. RQ1. Developers’ Perceptions of LLM-Generated Code

• Almost 80% (11/14) of uninformed participants did not realize that the validated code had been generated by an LLM.Without explicit information, developers might not consider the code’s provenance or may implicitly assume human authorship.Source: A. RQ1. Developers’ Perceptions of… · p. 6
• Participants commonly praised LLM-generated code for its coding style, readability, detailed naming, and comments compared with human-written code.Eleven of 28 participants specifically described its style and readability favorably.Source: A. RQ1. Developers’ Perceptions of… · pp. 6–7
• Participants reported distinctive LLM errors, including hardcoded examples, repetitive structures, missing context, and undefined variables.Source: A. RQ1. Developers’ Perceptions of… · p. 7
• Participants perceived LLMs as relatively stronger on syntax and details but more error-prone in logic than human developers.Source: A. RQ1. Developers’ Perceptions of… · p. 7
• LLM-generated code was perceived as limited in front-end tasks because Copilot lacked multimodal understanding of visual layouts.Source: A. RQ1. Developers’ Perceptions of… · p. 7

1) General Strategies for Validation and Repair:

• Participants commonly began by comprehending the code, tracing control flow, running outputs, and using IDE navigation or debugging tools to locate bugs.Context comprehension was reported by 11/28 participants, while GotoDeclaration and FindUsages were used 1.81 times per task on average.Source: 1) General Strategies for Validation… · pp. 7–8
• 7.483 > 2.362, inserting print statements and running the entire code was more common than using the IDE debugger (p-value = 3.45e-5 < 0.01).Participants used outputs both to inspect final behavior and to test intermediate execution states.Source: 1) General Strategies for Validation… · p. 8

2) Difference for LLM-Generated Code:

• Developers frequently switched fixations between code and comments to resolve mismatches between Copilot prompts and generated code.Participants described comparing expected code behavior with the input prompt used to generate it.Source: 2) Difference for LLM-Generated Code · p. 8
• Participants sometimes viewed Copilot’s different reasoning angle as complementary, because its mistakes could differ from their own debugging traps.This was reported as a way to combine human and LLM perspectives when searching for solutions.Source: 2) Difference for LLM-Generated Code · p. 8
• When generated code contained many erroneous statements, participants often deleted or commented out most of it and rewrote the code themselves.Small local edits were considered insufficient for some generated layouts and other substantially flawed code.Source: 2) Difference for LLM-Generated Code · p. 8
• Participants focused more on overall logic for LLM-generated code, while trusting repetitive code more and associating human errors with implementation details.They also reported weaker confidence in LLM reasoning because Copilot could not explain how it generated the code.Source: 2) Difference for LLM-Generated Code · pp. 8–9

3) Role of Copilot in Validation and Repair:

• Copilot generated more characters than keystroke typing on average (362.84 > 131.31, p-value = 0.0007 < 0.01), but not more than clipboard operations (362.84 > 324.41, p-value = 0.86 > 0.10).Participants reported using Copilot for idea exploration, syntax suggestions, and faster coding.Source: 3) Role of Copilot in Validation and… · p. 9
• Participants used Copilot-generated inline comments to help understand code and investigate whether individual lines contained bugs.This extended Copilot’s role beyond generating executable code to clarifying existing code during repair.Source: 3) Role of Copilot in Validation and… · p. 9

1) Bug Fixing Success Rate:

• Informed participants had a higher average bug-fixing success rate than non-informed participants (0.577 > 0.446; Student’s t-test, p-value = 0.0417 < 0.05), performing better on 8 bugs, the same on 3, and worse on 1.Source: 1) Bug Fixing Success Rate · p. 9

2) Developer Behaviors:

• The Informed group spent more time on code relative to comments and less time reading the original document.The code/comment ratio was 0.233 versus 0.179, while original-document reading time was 48.6 versus 67.4.Source: 2) Developer Behaviors · p. 9
• Informed participants used tracing features more frequently and showed higher saccade time, indicating greater search effort.Tracing-feature use was 4.606 versus 2.784, while saccade time was 2.303 versus 1.705.Source: 2) Developer Behaviors · p. 9, Figure 6 · p. 10
• Informed participants used Copilot more frequently and the clipboard less frequently than non-informed participants.Copilot use was 11.21 versus 6.081, whereas clipboard use was 7.758 versus 11.32.Source: 2) Developer Behaviors · pp. 9–10

3) Cognitive Workload:

• Informed participants reported higher effort and exhibited longer fixation time and average fixation duration, indicating higher cognitive workload.Effort was 6.394 versus 5.595, fixation time was 254.3 versus 206.7, and average fixation duration was 0.493 versus 0.438.Source: 3) Cognitive Workload · p. 10, Figure 6 · p. 10
• Participants’ interpretations of provenance awareness and workload were mixed, and the between-subjects design may introduce recall bias.Some participants expected awareness to increase workload, while another expected it to reduce workload by narrowing attention to specific logic details.Source: 3) Cognitive Workload · p. 10

VI. IMPLICATIONS

• Labeling LLM-generated code may help developers validate and manage it more effectively by focusing attention on generation prompts and anticipated issues.The paper also suggests tools for recognizing and revising the underlying prompts.Source: §VI · p. 10
• Because LLM-generated code is stronger in syntax than logic, future tools should give developers greater control over generated logic.Hierarchical prompt decomposition is offered as one example of this direction.Source: §VI · p. 10
• LLMs’ limited understanding of visual layouts constrains GUI code generation and motivates multimodal validation across text, images, audio, and layouts.A participant attributed randomly generated calculator buttons to Copilot’s inability to understand the frontend layout.Source: §VI · pp. 10–11
• Frequent switching between code and prompts motivates interfaces that connect them directly to reduce switching costs.The proposed direction is visualization that links prompts with the generated code.Source: §VI · p. 11

VII. THREATS TO VALIDITY

• The three lightweight Java tasks may not capture the complexity and diversity of real-world programming work.This limits how broadly the findings can be generalized beyond the selected task setting.Source: §VII · p. 11
• Because participants were mostly computer-science students, their experience may not represent professional developers.The paper recommends longitudinal studies in actual software-development settings.Source: §VII · p. 11
• Eye-tracking drift after prolonged use was mitigated through recalibration and supplemented with NASA-TLX workload reports.The study also relied on documented eye-tracking metrics and analyses.Source: §VII · p. 11

VIII. CONCLUSION AND FUTURE WORK

• Without explicit notification, developers often fail to recognize code provenance, while LLM-generated code combines readable style with errors differing from typical human errors.The findings motivate tools and methods tailored to validation and repair in the LLM era.Source: §VIII · p. 11