Source-linked AI summary
Practical Implementation Report on Introducing Spec-Driven Development Using AI Agents in Software Development PBL
Hidetake Tanaka, Hiroshi Igaki, Kazumasa Shimari, Kiyoshi Honda, Naoki Fukuyasu
TL;DR
The paper addresses limited evidence on incorporating AI agents into team-based software engineering education. It applies Spec-Driven Development in an undergraduate SDPBL course and finds phase- and team-varying AI use alongside higher implementation throughput, while emphasizing the need to monitor code comprehension.
Problem
Few studies have comprehensively investigated how AI agents affect team development across multiple phases in SDPBL, despite their growing educational importance.
Method
The study implements a four-phase SDD workflow in a third-year SDPBL course and analyzes surveys, added lines of code, and instructor-assessed code comprehension.
Results
AI usage varied across development phases and teams, while students in 2024–2025 added more lines of code than students in 2022–2023; the AI-use/comprehension trend was not statistically significant.
Takeaways & Limitations
Regular monitoring of code comprehension and proactive instructor feedback are important for maintaining educational effectiveness when introducing AI agents into SDPBL.
Takeaways & Limitations
The study lacked a concurrent non-AI control group, so year-to-year changes and cohort or educational differences cannot be ruled out as confounders.
Abstract
from arXiv · showhide
In recent years, autonomous AI agents such as GitHub Copilot and Claude Code have been rapidly gaining popularity. This study reports on the practical implementation of Spec-Driven Development, a software development methodology premised on AI agents, within a Software Development Project-Based Learning (SDPBL) course for third-year undergraduate students. We defined a workflow consisting of four phases, namely investigation, planning, implementation, and review. We also established an environment tailored for the SDPBL course where AI agents generate documentation and code during each phase. We analyzed the results from three perspectives, namely students' subjective AI usage, implementation throughput, and code comprehension. The analysis reveals that AI usage patterns varied across development phases and teams. Moreover, while AI agent utilization increased implementation throughput, it also tended to encourage students to proceed with development without fully understanding the code. This study demonstrates that regular verification of code comprehension by instructors and appropriate feedback are essential for maintaining educational effectiveness when introducing SDD into SDPBL.
1 Introduction
This study examines how AI agents can be incorporated into SDPBL through Spec-Driven Development, focusing on usage across phases, implementation throughput, and code comprehension. It addresses the limited practical evidence on AI-agent-supported team development while considering both benefits and risks for student learning.
- Few studies have examined how AI agents affect team-based development across multiple phases in SDPBL.The study therefore investigates AI-agent utilization in a software development education context.
- The study introduces Spec-Driven Development, in which detailed specifications guide AI agents while developers understand and verify generated artifacts.This process is intended to maintain code quality and consistency throughout development.
- The paper aims to identify both benefits and drawbacks of AI-agent use in team development education.The stated concerns include implementation throughput, development support, proceeding without adequate understanding, and the burden of AI-agent usage.
- The analysis covers students’ AI utilization, implementation throughput measured by added lines of code, and code comprehension.The research questions examine phase-specific AI use, throughput effects, and the relationship between AI use and students’ ability to understand their code.
- AI usage was measured on a five-point scale from no use to very helpful across investigation, planning, implementation, review, and related activities.Weekly surveys captured students’ perceived effectiveness of AI tools in each development phase.
2 Related Work
Related work contrasts rapid but weakly verified AI-assisted development with Spec-Driven Development’s structured specifications and continuous verification. It also identifies a need for broader evidence about AI agents in team-based software engineering education.
- LLM-Based Software Development: Vibe Coding enables rapid prototyping through dialogue with generative AI but makes code quality and consistency harder to ensure.The difficulty arises because developers do not directly verify implementation details.
- LLM-Based Software Development: LLM-generated code may use generic designs that overlook project-specific conventions and architectural patterns, contributing to technical debt.
- AI Agent-Based Software Development: AI agents have been applied to tasks including bug fixing, requirements definition, design, implementation, testing, and code review.RepairAgent, for example, autonomously performed bug-fixing tasks and repaired 164 bugs in Defects4J evaluations.
- Spec-Driven Development: Spec-Driven Development uses structured specifications that describe architecture, coding conventions, security requirements, and testing strategies before code generation.Its workflow includes sequential document generation and developer verification during requirements analysis, design, and implementation planning.
- Spec-Driven Development: In SDD, developers remain responsible for understanding and verifying generated documents and source code, making it more expertise-dependent than Vibe Coding.
- Generative AI and Software Development Education: Prior educational research reports conflicting views on generative AI and identifies student usage profiles, with instructional intervention improving exam performance and usage patterns.
- Generative AI and Software Development Education: Few studies have comprehensively investigated AI-agent effectiveness across development phases in team-based projects such as SDPBL.This gap motivates the paper’s practical application and analysis of SDD in a third-year undergraduate course.
3 Application of Spec-Driven Development to SDPBL
The course applied SDD through a four-phase workflow and a prepared team environment in which AI agents generated documentation and code. Evaluation combined surveys, repository-based added LOC, and instructor-assessed code comprehension.
- Course Setting: The course was a 14-lecture elective in which third-year students worked in teams of three to four to develop a web application with a database.Teams used Visual Studio Code, Git, GitHub, GitHub Flow, and weekly progress reports.
- Application Workflow: The SDD workflow divided development into investigation, planning, implementation, and review phases.AI agents generated artifacts such as investigation reports and implementation plans in each phase.
- Course Environment: Each team received a Visual Studio Code template project with AI-generated artifacts stored in docs/ and phase-specific reports organized in subdirectories.The template also included a project-specific copilot-instructions.md file containing workflow and coding rules.
- Demonstration and Verification: Instructors demonstrated a complete workflow cycle, verifying the investigation report, plan, implementation, and review outputs before students applied the process.Students were told they must be able to explain AI-generated code, while review reports assessed functionality, security, code quality, and documentation consistency.
- Evaluation Methodology: Weekly surveys recorded overall AI use and perceived helpfulness from 0 to 4 for investigation, planning, implementation, review, debugging, and explanation.
- Evaluation Methodology: Added LOC were aggregated from each student’s main-branch commits during each lecture period, excluding files that were clearly not new implementation.
- Evaluation Methodology: Instructors assessed students’ understanding of committed code and architecture through explanations scored from 0 for no understanding to 3 for full understanding.
4 Results
AI use varied substantially across development phases and teams, while later academic years generally showed higher implementation throughput. Higher AI usage was also associated with weaker code comprehension in some periods, although comprehension improved after weekly instructor checks.
- RQ1: All teams used AI to some extent for Implementation and Debugging, whereas Investigation, Planning, and Review showed larger differences between teams.Teams 25A and 25B used AI effectively across phases; Team 25C used it less, and Team 25D varied by phase.
- RQ2: Seven of eight lectures showed statistically significant year-to-year differences in added LOC, with earlier years generally below later years.Only lec11 showed no significant differences; significant lectures were lec06, lec07, lec08, lec09, lec10, lec12, and lec13.
- RQ2: 2022 had significantly fewer added LOC than 2024 in lec06, lec07, lec08, lec10, and lec12, while 2023 was lower than 2024 in lec08 and lec13.No significant comparison showed 2024 or 2025 below 2022 or 2023.
- RQ3: Students at AI usage level 0 maintained consistently high code comprehension, whereas level 3 students showed very little comprehension, especially in lec09–lec10.Team 25C, which used AI least during Implementation, also maintained consistently high comprehension across the analyzed periods.
- RQ3: Code comprehension fluctuated for most teams, dropped in lec09 when AI usage became very high, and generally improved in lec11–lec12 after weekly one-on-one instructor checks.The authors observed improvement trends in the Late period regardless of AI usage and emphasize mechanisms for assessing understanding of AI-generated code.
5 Discussion
The discussion highlights increased implementation throughput alongside unresolved concerns about students’ code comprehension, motivating continuous monitoring and timely intervention. It also notes validity constraints from the study design, metrics, subjective assessments, and limited setting.
- Balancing Code Comprehension and AI Usage: AI usage was not statistically associated with code comprehension across students, although non-users tended to show high comprehension scores.Limited sample size, high individual variability, and instructor interventions may have influenced this result.
- Balancing Code Comprehension and AI Usage: Implementation throughput, measured as added LOC per student, increased from 2022 to 2024–2025.The increase may accelerate development, but does not necessarily indicate deeper learning or thorough understanding of generated code.
- Considerations for Introducing AI Agents in SDPBL: Instructors should establish foundations before extensive AI use and continuously monitor comprehension with timely intervention.The course used an initial tutorial period, while lec09 illustrated the need for prompt intervention after comprehension declined alongside high AI usage.
- Considerations for Introducing AI Agents in SDPBL: Students should be accountable for explaining the logic of AI-generated code so implementation speed does not compromise understanding.One-on-one interviews provide opportunities for students to demonstrate responsibility for code they did not manually write.
- Considerations for Introducing AI Agents in SDPBL: Standardized project templates can help students initiate AI-agent-supported development and reduce cognitive load during environment setup.The templates included AI instructions, specification templates, and development rules such as coding conventions.
- Threats to Validity: Causal attribution is limited because the study lacked a concurrent non-AI control group and compared against historical data.Year-to-year differences in student skills, cohort size, and educational conditions may confound observed changes.
- Threats to Validity: Added LOC does not fully capture software-development efficiency, while comprehension scores rely on instructors’ subjective evaluations.The dialogue-based evaluations can assess understanding depth, but lack the rigor of objective tests.
- Threats to Validity: Generalizability is constrained by the single Java and Spring Boot course and its four participating teams.The findings are relevant mainly to similar PBL courses and team developments using widely adopted technologies.
6 Conclusion
The study examined AI-agent integration in a third-year SDPBL course through AI usage, added lines of code, and code comprehension. It found higher implementation throughput but no statistically significant comprehension correlation, supporting careful educational management and further study.
- The study analyzed SDPBL student activities using self-reported AI usage, added lines of code, and code comprehension levels.
- Students added more lines of code in 2024–2025 than in 2022–2023, indicating increased implementation throughput.
- No statistically significant correlation was found between AI usage and code comprehension.One-on-one interviews and timely interventions may have helped students maintain comprehension of AI-generated code.
- Regular comprehension monitoring and proactive feedback should accompany AI-agent adoption to maintain educational effectiveness.
- Future work should assess understanding of AI-generated code, optimize workflows balancing assistance with learning, and examine long-term effects.