Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks

• Software development includes reading, understanding, and modifying existing code, so coding-puzzle performance alone does not establish usefulness in real-world workflows.The introduction also notes limited human-centered research on these tools in software development workflows.Source: §1 · p. 2
• The study asks whether ChatGPT benefits coding and development task outcomes, and how people interact with it while completing tasks.The outcomes considered include efficiency, solution quality, and subjective task load.Source: §1 · p. 2
• With 109 participants in a controlled experiment, ChatGPT helped with coding puzzles but had limited effectiveness for typical development tasks.The study also observed participant interactions and related them to outcomes.Source: §1 · p. 2
• The paper contributes a quantitative evaluation, a qualitative account of interactions, and implications for software engineering research, practice, and education.The authors describe the evaluation as the first large-scale controlled experiment of LLM capabilities in software engineering, to their knowledge.Source: §1 · pp. 2–3

2.1 AIGC and LLM

• Generative AI progressed from early sequential models through deep learning, while transformers enabled large-scale pretraining and became central to LLMs.The section also describes RLHF as aligning generated content with human preferences and identifies ChatGPT and GPT-4 as built using this approach.Source: §2.1 · p. 3

2.2 Human-AI Collaboration

• Human–AI collaboration research spans several domains, while LLM work addresses real-world concerns including bias, opacity, and controllability.Prior AI-assisted coding studies report that tools do not necessarily improve task time or success, although users may value useful starting points and saved search effort.Source: §2.2 · pp. 3–4

2.3 LLM and Software Engineering

• Software engineering research progressed from early natural-language code generation to LLM-based code understanding and generation, but generated code can remain imperfect and difficult to control.A cited survey reported that developers can have trouble getting tools to meet functional or non-functional requirements.Source: §2.3 · p. 4

3 EXPERIMENT DESIGN

• The experiment followed social-science design guidelines and software-engineering practices, with Fig. 1 presenting the conditions and groups.Source: §3 · p. 4, Figure 1 · p. 5

3.1.1 Experiment Tasks.

• Coding Puzzles involved two randomly selected, medium-difficulty problems; the development task asked participants to fix two local bugs in a small GitHub project.Participants could use internet resources, and the bug-fixing task involved coding, testing, comprehension, and domain knowledge.Source: §3.1.1 · p. 5

3.1 Experiment Tasks and Group Setup

• The study selected tasks that fit the session time, required meaningful problem-solving, and represented real-world issues.The tasks were termed Coding Puzzles and Typical Development Task; participants used Python in both.Source: §3.1.1 · p. 6

3.1.2 Experiment Group Setup.

• Participants were randomly assigned to one of four groups crossing task type with ChatGPT access in a 2 × 2 between-subject design.Assignment used arrival-time-based random numbers, though group sizes varied slightly.Source: §3.1.2 · p. 6

3.2 Experiment Environment

• Task-specific environments supported online judging for puzzles and local editing and testing for the development task, with GPT-3.5 accounts available to ChatGPT groups.Internet access remained available to all participants throughout the experiment.Source: §3.2 · p. 6

3.3 Participants

• The 109 participants were computing majors from a Beijing ICT university, selected for at least one year of professional software-development experience.Most were postgraduate students, and participants were randomly assigned to groups.Source: §3.3 · pp. 6–7

3.4 Experiment Process

• Participants received task briefings and environment familiarization before working for up to 75 minutes, then completed a six-question survey.Five survey questions measured task load, and one asked about perceived AI effects.Source: §3.4 · p. 7

3.5 Pilot Experiment

• Three pilot sessions used the initial protocol without randomized assignment; observed behavior, performance, and feedback informed small design adjustments.The authors reported that the design was generally effective.Source: §3.5 · p. 8

4.1 Collected Data and Measurements

• Task measures covered completion time, test-case-based solution quality, subjective perception, and NASA-TLX task load.Screen histories and voluntary participant narratives supplied qualitative evidence about interactions with ChatGPT.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Data Analysis Strategy

• Quantitative group comparisons used Mann–Whitney U tests because most values violated normality assumptions, while interaction histories were coded and categorized by two authors.The interaction coding used a four-part record of start time, end time, prompt, and response; the authors’ independent categorization achieved 0.92 inter-rater reliability.Source: §4.2 · pp. 8–9

5.1 RQ1: Benefits (or Losses) of Working with ChatGPT

• ChatGPT reduced coding-puzzle completion time significantly, but its small time reduction on typical development tasks was not significant.Coding-puzzle times averaged 2196.7 versus 2692.6 seconds (p = 0.026); typical-task times were 2054.9 versus 2147.3 seconds (p = 0.708).Source: §5.1.1 · p. 9, Figure 3 · p. 10
• Neither coding puzzles nor typical development tasks showed a statistically significant ChatGPT-related improvement in solution quality.For coding puzzles, average quality was 86.15 versus 77.87 (p = 0.39); typical-task averages were 73.68 versus 71.46 (p = 0.95).Source: §5.1.2 · p. 9, Figure 4 · p. 10

5.1.3 Subjective Perception and Task Load.

• Participants generally viewed ChatGPT as a productivity aid, although direct task outcomes did not support that perception.The overall mean productivity perception was 16.12 on a 0–20 scale, while the with-ChatGPT typical-development group averaged 14.36.Source: §5.1.3 · pp. 10–11, Table 2 · p. 11
• ChatGPT did not significantly change subjective task load in either task, except that coding-puzzle participants rated their performance more positively.The performance rating difference for coding puzzles was significant (W = 524.5, p = 0.023, r = 0.50); other task-load aspects showed no significant differences.Source: Figure 5 · p. 11

5.2.1 Using ChatGPT in Coding Puzzles: Replacing Searching by Generating? Not for Everyone.

• In coding puzzles, some participants delegated the work to ChatGPT, while many generated an initial solution and then reviewed and fixed it themselves.Five of 31 participants relied on ChatGPT to do everything; 17 of 31 actively reviewed and edited generated code, and six of 31 used online searching in an iterative coding-debugging-searching cycle.Source: §5.2.1 · p. 12

5.2.2 Using ChatGPT in Typical Development Work: Treat It as My Life-Saver or A Colleague?

• Typical-development interactions ranged from broad, unsuccessful delegation to iterative collaboration and focused, minimal use of ChatGPT.Eight of 25 participants tended to hand over the task and often struggled; 12 of 25 used ChatGPT throughout as a colleague, while five of 25 used it minimally after first understanding the code themselves.Source: §5.2.2 · pp. 13–14

5.3 Summary of Findings

• Overall, findings combined puzzle-efficiency gains and positive attitudes with no quality gains, alongside varied interaction strategies in typical development work.The study also reports that ChatGPT may replace code searching for simple puzzles, though some participants retained traditional approaches.Source: §5.3 · p. 14

6.1 Discussion of the Findings

• The discussion frames software creation as involving conceptual work beyond representing code, connecting that distinction to the study’s contrast between coding puzzles and typical development tasks.Brooks’ cited argument says specification, design, and testing of the conceptual construct are harder than representing it.Source: §6.1 · pp. 14–15
• ChatGPT improved efficiency on coding puzzles, but typical development tasks demanding software-system conceptual models remained difficult; the pattern echoes Brooks’ distinction between conceptual and representational work.Participants who trusted the AI less and engaged more with prompt format got more predictable responses; the study also identified possible potential for replacing source-code search tools.Source: §6.1 · pp. 14–15

6.2 Implication to Research

• Because the study covered a very limited task set, further SE research should examine collaboration dynamics, especially in multimodal tasks such as architectural-design diagram generation.The authors also encourage collaboration with CSCW researchers and further empirical work on SE areas where LLMs need more sophistication.Source: §6.2 · p. 15

6.3 Implication to Practices

• Practice implications include learning to write effective prompts and designing prompt systems, with collective trial-and-error experiences as one proposed way to build guidance.The authors also suggest LLMs could reduce specialized-knowledge requirements and potentially broaden participation in software engineering.Source: §6.3 · pp. 15–16

6.4 Implication to SE Education

• The authors propose that SE education should develop a shared agenda for evaluating LLMs’ effects on curricula and addressing questions about core knowledge, design-code links, and human–AI responsibilities.The passage presents these as open questions rather than settled curriculum changes.Source: §6.4 · p. 16

Construct validity

• Most constructs used objective or well-established measures, but Task Load was assessed indirectly with NASA-TLX rather than biometrics.The authors characterize the resulting construct-validity threat as minimal.Source: §7 · p. 16

Internal validity

• Random assignment and standardized procedures supported internal validity, but the study could not control every relevant factor, including participant differences, possible plagiarism, attitudes, and work environments.The authors cite dual-display settings as one example of a potentially influential environmental difference.Source: §7 · p. 16

External validity

• External validity is limited by testing only one task type with mostly graduate students, and the authors caution against extending findings to other settings or later AI tools.The 102 graduate students were from an elite ICT university, and the study used GPT-3.5-series models.Source: §7 · pp. 16–17

Concluding remarks

• ChatGPT helped improve efficiency in solving coding puzzles, but the experiment found it still lacked sufficient capability to fully support typical software-development activities.The authors report 109 participants and characterize the work as a controlled experiment.Source: §8 · p. 17
• For typical development tasks, the study found no overwhelming ChatGPT superiority, including on simple bug-fixing tasks, and reports that it neither improved efficiency nor solution quality there.The authors describe sophisticated use and articulate input as potentially beneficial, not as guaranteed to improve outcomes.Source: §8 · p. 17
• The authors report that continuous, purposeful, specific communication might have positive effects and argue that the experiment rebuts claims that LLMs would soon replace developers in typical development tasks.They call for further research and mechanisms to promote effective collaboration.Source: §8 · p. 17

Data availability

• The experiment protocol and materials, Task 2 source code, and anonymous quantitative-analysis dataset are available at figshare.The passage provides the DOI 10.6084/m9.figshare.24166248.Source: DATA AVAILABILITY · p. 17
• Raw videos and screen histories are available upon request only to researchers who have obtained IRB approval from their own institutions.Source: DATA AVAILABILITY · p. 17
• Task 2 is the typical-development task, and its source code is included with the available materials.Source: DATA AVAILABILITY · p. 17

A THE FIVE CODING PUZZLES USED IN THE EXPERIMENT

• The experiment uses a pool of five coding puzzles for Task 1.Each participant solves two puzzles randomly drawn from the pool.Source: A THE FIVE CODING PUZZLES USED IN THE… · p. 17
• Each Task 1 participant solves only two of the five puzzles.The two puzzles are randomly drawn.Source: A THE FIVE CODING PUZZLES USED IN THE… · p. 17

A.1 Project Leader-board

• Students are ranked by descending total progress, then by descending total problems solved.Both measures determine whether students receive the same rank.Source: §A.1 · p. 18
• Students tied on both progress and problems solved share the same rank.Source: §A.1 · p. 18
• When outputting the leaderboard, students are listed in ascending username order.Source: §A.1 · p. 18
• For each test set, the output gives each student's rank, username, total progress, and total problems solved.Source: §A.1 · p. 18

A.2 Analyzing Valve Status in the Workshop

• The valve-status puzzle converts one input byte into eight Boolean valve states, with each list position corresponding to a bit.Source: §A.2 · p. 18

A.3 EAN13 Barcode Verification

• The EAN13 puzzle checks whether a 13-digit barcode passes verification using a check digit derived from the first 12 digits.Inputs that are not 13 digits produce a barcode error; valid-length codes are reported as passed or failed.Source: §A.3 · p. 18

A.4 Survival Test on a Pirate Ship

• The pirate-ship puzzle outputs the elimination order when every M-th person is removed until K people remain.Source: §A.4 · p. 19

A.5 The Simplest Word

• The simplest-word puzzle selects the word with the lowest average score when multiple learners grade the same word.Source: §A.5 · p. 19