Generative Artificial Intelligence for Software Engineering -- A Research Agenda

1. Introduction

• GenAI tools are being explored for coding, testing, requirements engineering, software processes, and project management, while making AI-generated content more accessible to professionals.The paper notes that code generation and test-case optimization directly benefit from recent LLMs [1] [2] [3] [4].Source: §1 · p. 2
• Nondeterministic outputs, sensitivity to input settings, and hallucinations remain challenges to reliable GenAI use in software engineering.The paper says automation may be closer when these challenges are sufficiently addressed.Source: §1 · p. 3
• The research agenda combines a literature review and focus groups to describe current work and open challenges across eleven areas of software engineering.The authors do not claim a comprehensive literature review given the topic’s rapid pace; focus groups contribute practical expectations about GenAI’s future roles.Source: §1 · p. 3
• Two international workshops provided the setting for focus groups on AI-assisted agile development and GenAI for requirements engineering.The authors say the identified challenges draw on literature gaps as well as co-authors’ practical insights.Source: §1 · pp. 3–4

2.1. Terminologies

• The paper describes GenAI as using generative modeling to learn patterns from examples and produce synthetic artifacts, and defines prompt engineering as refining instructions for desired LLM outputs.It also introduces AI-generated content, fine-tuning, few-shot training, and prompts as relevant terms.Source: §2.1 · pp. 4–5
• The terminology section distinguishes GPT, NLP, language models, and LLMs, while noting that the threshold for calling a model “large” is unclear.The paper uses LLM despite that vagueness because the term is widespread.Source: §2.1 · p. 5

2.2. History of GenAI

• The history traces generative AI from early language systems and neural networks through machine learning, deep learning, GANs, and transformer-based models.The paper also mentions applications including text generation, image synthesis, music composition, and software implementation.Source: §2.2 · pp. 6–7

2.3. Fundametals on LLMs

• Language models predict likely next words; the Transformer architecture helped renew interest in language modeling and led to contextualized embeddings and GPTs.The paper describes scaling parameter counts and training data as a recent approach to improving model performance.Source: §2.3 · p. 7
• Model sizes have grown faster than GPU memory, making fine-tuning large models for smaller tasks impractical for most users; in-context learning is constrained to few examples and can perform inconsistently.Expanding context size significantly increases computational costs.Source: §2.3 · pp. 7–8
• Parameter-efficient fine-tuning addresses resource constraints by training only a subset of existing or added model parameters, with methods trading off efficiency, speed, output quality, and overhead.Figure 1 presents several optimal-parameter fine-tuning methods.Source: §2.3 · p. 8, Figure 1 · p. 8

3.1. Literature review

• The authors use focused, periodic literature reviews rather than a comprehensive systematic review because the topic changes rapidly and relevant work includes gray literature.They aimed to provide an agenda promptly and avoid claiming comprehensiveness or repeatability.Source: §3.1 · p. 9
• The literature search used Google Scholar and Scopus, gray-literature sources, and forward and backward citation snowballing; the reported search was current to October 2023.Google Scholar was selected for free access and coverage of gray literature.Source: §3.1 · p. 9

3.2. Focus groups

• Four focus groups ran from April to September 2023 with software engineering researchers; sessions lasted two to three hours and used notes, Miro boards, and recordings.The paper reports participant demographics in Table 1.Source: §3.2 · pp. 10, 12, Table 1 · p. 11
• The first two focus groups explored GenAI opportunities and research topics, while the later groups reviewed, revised, and prioritized candidate questions.The first group used SWEBOK areas to start discussion and formed eleven categories; the third reviewed 121 candidate questions.Source: §3.2 · p. 12
• The final focus group retained 78 research questions in eleven categories, excluding questions judged not practically important or meaningful, but participants did not reach consensus on a complete ranking.The authors therefore did not present rankings.Source: §3.2 · pp. 12–13

3.3. Threats to validity

• Focus-group threats include group dynamics, social acceptability, hidden agendas, secrecy, and limited comprehension; the authors used structured agendas, contribution rules, and anonymous online input to mitigate some risks.Follow-up sessions and between-session reflection were used to address comprehension limits.Source: §3.3 · pp. 13–14

4.1.1. Historical Context.

• Requirements work remains difficult because stakeholder needs evolve and agile practice faces gaps in documentation, customer access, non-functional requirements, and automated support.Source: §4.1.1 · p. 15

4.1.2. Key RQs.

• GenAI research questions span elicitation, specification, regulatory validation, change prediction, and threats across requirements engineering stages.Source: §4.1.2 · p. 15

4.1.3. State-of-the-art

• Early RE studies examine prompt patterns and generated requirements, reporting that ChatGPT requirements tend to be abstract yet consistent and understandable [46].Source: §4.1.3 · p. 16
• Other work investigates requirements retrieval, domain-concept extraction, ambiguity detection, interactive use-case refinement, and automated question-answering [53].Source: §4.1.3 · p. 16
• Classification studies report NoRBERT outperforming prior approaches on most tasks and PRCBERT performing moderately better than NoRBERT; GPT-4 has also been evaluated for goal modeling.Source: §4.1.3 · p. 17

4.1.4. Challenges and Limitations

• Across RE stages, models may miss domain context, introduce bias, produce unclear specifications, overlook guideline nuances, struggle with analysis, and require team validation.Source: §4.1.4 · pp. 17–18
• Task-specific fine-tuning is constrained by scarce public RE data, while safety-critical domains raise ethical and security concerns and unchecked training-data bias can skew representation.Source: §4.1.4 · pp. 17–18

4.1.5. Future Prospects.

• Prospects include real-time human-AI refinement, transparent visualizations, privacy-preserving fine-tuning, and responsible models designed to address bias and produce representative requirements.Source: §4.1.5 · pp. 18–19

4.2.1. Historical Context

• Design decisions require evaluating trade-offs among factors affecting quality attributes, yet existing automated approaches are not widely used in industry.Source: §4.2.1 · p. 19

4.2.2. Key RQs

• The open questions cover pattern selection, human-AI decision-making, risks and reliability, continuous design and refactoring, interface generation, quality-attribute trade-offs, and evolving architectures.Source: §4.2.2 · pp. 19–20

4.2.3. State-of-the-art

• Early studies explore architect collaboration, pattern recommendations, microservice identification, and architecture degradation, but incorrect ChatGPT advice may mislead developers [70].Source: §4.2.3 · p. 20

4.2.4. Challenges and Limitations

• Evidence is limited by the lack of industrial studies; controlled evaluations leave practical usefulness unclear, and real projects may lack sufficient input for accurate answers.Source: §4.2.4 · pp. 20–21

4.2.5. Future Prospects

• Future work should establish GenAI's role in continuous design workflows and examine engagement approaches, including prompt patterns [73].Source: §4.2.5 · p. 21

4.3.1. Historical Context

• Implementation supports productivity but faces legacy compatibility, security and privacy, scalability, and workforce challenges; code generation aims to produce code from natural-language requirements.The section gives a quick-sort request and reports that its complex LeetCode solution passed all test cases.Source: §4.3.1 · pp. 21–22

4.3.2. Key RQs

• The implementation research questions address integrating coding assistance, specializing models, ensuring correctness and security, protecting private data, validating requirements, and meeting legal constraints.They also ask about user intent, cost-effective training, and natural-language coding interfaces.Source: §4.3.2 · pp. 22–23

4.3.3. State-of-the-art

• Prior work studies code-generation pipelines, collaborative LLM agents, failure categories, explainability, and performance differences among tools and input settings.Other approaches cover code search, retrieval, and API recommendation.Source: §4.3.3 · pp. 23–24
• Research also examines comment generation and workplace use, including acceleration and exploration modes, while studies raise nondeterminism and security concerns.Programmers may prefer Copilot without necessarily achieving higher work performance.Source: §4.3.3 · pp. 24–25

4.3.4. Relevant technologies

• The proposed model-building flow selects CodeGen-6B-Mono and CodeT5-Large, fine-tunes on downstream data with LoRA, and provides an interface and secure project storage.Figure 7 presents the code-generation flow.Source: §4.3.4 · pp. 25–26, Figure 7 · p. 27

4.3.5. Challenges and Limitations

• Implementation challenges include interpreting heterogeneous requirements, since logical rules, examples, and natural language may each incompletely express user intent.The section also identifies model size and complexity as barriers to practical fine-tuning.Source: §4.3.5 · p. 28
• Project-specific code generation remains difficult: retraining can demand substantial hardware, time, and storage, while project-only training may sacrifice general knowledge.Fine-tuning methods aim to adapt general knowledge to specific problems under hardware constraints.Source: §4.3.5 · pp. 28–29

4.3.6. Future Prospects

• Future prospects include adaptive, project-tailored tools and broader access to coding, alongside hardware optimizations and wider community participation.The section points to quantization and pruning as ways to address memory, computation, and energy constraints.Source: §4.3.6 · pp. 29–30

4.4.1. Historical Context

• Software quality assurance includes testing, but traditional test design is laborious and resource-intensive; GenAI may generate diverse requirement-based test data and analyze existing data.The section presents these capabilities as possible ways to address SQA demands and test-data flaws.Source: §4.4.1 · pp. 30–31

4.4.2. Key RQs

• The proposed questions cover code review, generating tests from requirements, integrating GenAI into QA pipelines, automating acceptance criteria, and maintaining reliable test suites as requirements change.They also ask how to produce test scenarios and representative tests.Source: §4.4.2 · pp. 31–32

4.4.3. State-of-the-art

• Prior work explores GenAI for testing and quality tasks, but models may be poorly suited to specialized SQA work and the literature offers no clear implementation guidelines.Studies examine test generation, usability testing, code analysis, and related activities.Source: §4.4.3 · pp. 32–34
• One ChatGPT-based system exceeded 70% efficiency in generating test suites, while another study found only 24.8% of ChatGPT-generated tests executed successfully.The studies also report correctness issues among generated tests.Source: §4.4.3 · p. 34

4.4.4. Challenges and Limitations

• Current GenAI testing faces correctness, hallucination, code-semantics, and prompt-sensitivity issues; most studies are experimental, leaving real-world effectiveness unclear.Human intervention remains necessary, and training specialized models may require resources that some companies cannot afford.Source: §4.4.4 · pp. 35–36

4.4.5. Future Prospects

• Future work should broaden SQA tasks, evaluate prompt strategies, and test newer models on different datasets, with experts validating domain-dependent outputs.Suggested tasks include test planning, test requirements, acceptance and integration testing, and other quality attributes.Source: §4.4.5 · pp. 36–37

4.5.1. Historical Context

• Maintenance is costly and often requires engineers to understand, translate, refactor, or repair code in systems that may lack adequate testing and documentation.The section notes that maintainability is frequently neglected and that engineers often rely on knowledge embedded in source code.Source: §4.5.1 · p. 37

4.5.2. Key RQs

• The research questions cover translation correctness and codebase migration, monolith-to-microservices refactoring, program repair, and automation of smells, anomalies, bug triage, and clone detection.They also ask how multiple LLMs or specialized ML models could be combined for automatic repair.Source: §4.5.2 · pp. 37–38

4.5.3. State-of-the-art

• For code summarization, a cited study found few-shot Codex with ten samples significantly outperformed fine-tuned models, while summaries may support comprehension and documentation updates.The section reports that 40% to 60% of maintenance effort is spent understanding systems.Source: §4.5.3 · p. 38
• For refactoring, Codex and CodeGen produced speedups exceeding 2.5× for over 25% of investigated C++ and Python programs, and GPT-4 improved code quality by complexity metrics [129] [1].The speedups persisted for C++ programs compiled with O3 optimization.Source: §4.5.3 · p. 39
• For repair, state-of-the-art LLMs reportedly outperform existing APR techniques, yet off-the-shelf models struggle with plausible real-world security fixes [131].Copilot may help identify vulnerabilities but falls short at fixing them.Source: §4.5.3 · p. 39

4.5.4. Challenges and Limitations

• GenAI can handle syntax-oriented tasks such as summarization and repair, but semantic understanding remains difficult and limits performance in realistic scenarios [122] [134].Translation is also constrained by context windows, scarce high-quality code pairs, and the risk that minor errors make generated code nonfunctional.Source: §4.5.4 · p. 40

4.5.5. Future Prospects

• LLMs remain assistants rather than replacements for maintainers, while proposed future directions include collaborative models and combining LLMs with formal verification or self-debugging for vulnerability work [122] [134].Collaborative LLMs assign subtasks to distinct model “experts” [82].Source: §4.5.5 · pp. 40–41

4.6.1. Historical Context.

• Although individual developers have begun using GenAI for assistance, company-level endorsement and implementation are lagging.The section situates GenAI alongside established processes such as Agile, Waterfall, and continuous software engineering.Source: §4.6.1 · p. 41

4.6.2. Key RQs.

• Open questions span GenAI integration into development environments, CI/CD, commits and deployments, productivity across tasks and seniority levels, citizen development, and process transformation.The section asks how tools can provide real-time assistance and feedback during development.Source: §4.6.2 · pp. 41–42

4.6.3. State-of-the-art.

• Early studies examine programming assistance, prompt patterns, conversational code-grounded interactions, productivity, and process-related use cases, but broader transformation remains speculative [46] [73].Examples include runtime DevSecOps, game-design collaboration, and human-robot collaboration.Source: §4.6.3 · pp. 42–43

4.6.4. Challenges and Limitations.

• Good practices remain scarce, and the impact of GenAI on productivity and processes is poorly understood; human oversight matters because generated code can be erroneous [144].Copilot may support senior developers but challenge novices who have difficulty judging code correctness; overconfidence may reduce criticism.Source: §4.6.4 · p. 43

4.6.5. Future Prospects.

• The future role of developers may shift toward reviewing AI-written code, but which expert tasks developers will focus on remains an open question.The section also describes a vision of redirecting human attention toward conceptual tasks computers handle poorly.Source: §4.6.5 · pp. 43–44

4.7.1. Historical Context

• Engineering management covers project management, teams, leadership, budgeting, risk, communication, quality, and related practices intended to deliver projects on time, within budget, and to quality standards.Prior work identifies human-resource challenges including small organizational data sets, ethical and legal constraints, and employee responses to management.Source: §4.7.1 · p. 44

4.7.2. Key RQs

• The open questions address team size and dynamics, employee onboarding and retraining, human–AI workflows, stigma around AI assistance, and economic impacts such as costs, productivity, and return on investment.They also ask how organizations can foster trust, learning, and innovation in developer–GenAI collaboration.Source: §4.7.2 · pp. 44–45

4.7.3. State-of-the-art

• Prior work includes AI applications for project management and estimation, including a GPT-based Agile story-point approach, while a cited BERT model only marginally improved software effort-estimation performance.The section also cites a study reporting no notable productivity increases outside a specific SE focus and an industrial case study describing possible SE application contexts.Source: §4.7.3 · pp. 45, 47

4.7.4. Challenges and Limitations

• A notable gap is the lack of industrial studies, especially those with an organizational focus; complex project contexts and limited knowledge may also contribute to biased decisions and less effective projects [162].The section also reviews AI concerns in construction engineering, including explainability, security, talent, cost, ethics, and governance.Source: §4.7.4 · p. 47

4.7.5. Future Prospects

• Future prospects include AI assistance with resource allocation, risk management, task assignment, project-cost estimation, Agile sprint planning, monitoring, and documentation.The section presents these as possible areas for assistance or replacement, not established outcomes.Source: §4.7.5 · pp. 47–48

4.8.1. Historical context.

• Skill shortages, talent gaps, and gender and soft-skills gaps complicate team formation, while integrating GenAI may require developer training and could contribute to future education.The section asks how GenAI might offload repetitive tasks and support developers working alongside these tools.Source: §4.8.1 · p. 48

4.8.2. Key RQs.

• The proposed questions cover addressing talent gaps, integration and reskilling strategies, productivity by task and seniority, GenAI as an advisor, and the skills and training needed for quality outputs.The agenda relates these questions to mitigating difficulty finding software professionals and up-skilling or offloading existing developers.Source: §4.8.2 · pp. 48–49

4.8.3. State-of-the-art

• The section describes Copilot and other LLM-based tools but notes uncertainty about license compliance and internal security practices, so informed human oversight remains necessary [173] [174].Copilot generates code from prompts such as comments or function signatures using training on GitHub-hosted open-source projects.Source: §4.8.3 · p. 49

4.8.4. Challenges and limitations.

• Company reluctance to grant access to proprietary systems can hinder research participation, and findings about team productivity or dynamics require contextualized conclusions to avoid drastic decisions based on misinterpretation.The section also emphasizes informed consent, confidentiality, and anonymity for studies involving people.Source: §4.8.4 · pp. 49–50

4.8.5. Future prospects.

• Carefully applying findings across work environments can guide GenAI integration in Agile development and inform education and reskilling, indirectly contributing to mitigating the talent gap.The section anticipates progressive adaptation of work dynamics and adoption by different stakeholders.Source: §4.8.5 · p. 50

4.9.1. Historical Context.

• In software-engineering education, chatbots support students outside class with coding and problem-solving, explanations and examples, and error identification [7],; automated exercise assessment is a separate paradigm, while AI approaches find questions of similar difficulty.Automated assessment can provide immediate feedback and facilitate self-assessment and self-examination,.Source: §4.9.1 · pp. 50–51

4.9.2. Key RQs

• The agenda asks how GenAI can be integrated into curricula and classrooms, personalized for diverse learners, assessed, and used ethically, while also examining effects on skills, careers, and educator training.It also asks whether some AI uses can harm learning outcomes.Source: §4.9.2 · p. 51

4.9.3. State-of-the-art

• Prior work reports that ChatGPT can improve programming self-efficacy and motivation when students receive prompt-writing skills, and recommends AI literacy in curricula.The cited work also explores ChatGPT in software testing education.Source: §4.9.3 · p. 52

4.9.4. Challenges and Limitations

• Reported limitations span underspecified questions, nondeterministic answers, and limited domain knowledge in theory courses; programming and project work also face practical-skill, outdated-information, visual-context, and changing-project-context constraints.GenAI alone does not develop practical programming skills; project advice may require updating tool context, and real-world situations may not be fully describable as input.Source: §4.9.4 · pp. 52–53

4.9.5. Future Prospects

• Future work calls for integrating GenAI concepts and tools into SE curricula and developing collaborative teaching methods aligned with learning outcomes, alongside ethical educational-data methodologies.Proposed curriculum content includes prompt engineering, AIOps, and AI-human interaction.Source: §4.9.5 · pp. 53–54
• Researchers should develop fair assessment methods that distinguish student-generated from AI-generated work and empirically evaluate effects of AI-enabled teaching on learning outcomes and engagement.The section also calls for ethical guidance on GenAI use in SE courses.Source: §4.9.5 · p. 54

4.10.1. Historical context

• Cloud computing is presented as a precedent for technology enabling low-cost product experimentation and new startup methods, while data-dependent AI has prompted interest in ethics.Source: §4.10.1 · p. 55

4.10.2. Key RQs

• The research questions cover GenAI-enabled business models, competitiveness, adoption barriers, intellectual property and licensing, societal and ethical impacts, and business sustainability.Source: §4.10.2 · p. 55
• The section identifies unresolved economic and societal questions, including how AI-created intellectual property may be managed and monetized and how adoption may affect jobs, retraining, and resource use.Source: §4.10.2 · pp. 55–56

4.10.3. State-of-the-art

• Prior work considers workforce skills and training, GenAI’s business-model innovation opportunities, and calls for regulation in healthcare.The business-model analysis discusses value creation, new propositions, and value capture, including task automation.Source: §4.10.3 · p. 56

4.10.4. Challenges and Limitations

• The section treats these topics as interdisciplinary, connecting intellectual property to economics and philosophy and noting that countries may adopt different legislation or public policies.Source: §4.10.4 · p. 56

4.10.5. Future Prospects

• The authors anticipate new software-intensive business models, labor-force changes, and legislation, with possible tensions among social actors and across jurisdictions.They foresee both challenges for established companies and opportunities for new entrants as GenAI develops.Source: §4.10.5 · p. 57

• The agenda frames industry evaluation, reliability and correctness, explainability and decision-making integration, and sustainability as open research questions.Source: §4.11 · pp. 57–58
• Industry evaluations should bridge small empirical studies and real-world applications by assessing feasibility, usability, scalability, cost-value tradeoffs, workflows, coordination, and user experience.The section notes that models may be more effective as part of an overall SE process than on small tasks.Source: §4.11 · p. 58
• Reliability and correctness remain critical because even minor software-engineering errors can have severe consequences, while model updates and fine-tuning may alter behavior across tasks.The authors cite variation between GPT-4 versions and call for continuous evaluation amid limited update transparency [205].Source: §4.11 · p. 58
• Private companies may lack suitable domain-specific training data, while confidential or personal data raises privacy, security, access, and permission concerns [206] [203].The text calls for regular audits of privacy and security measures.Source: §4.11 · pp. 58–59
• Black-box models make decisions difficult to interpret or debug, and data-driven biases may produce unfair software without transparent human oversight [208] [172].The authors also raise automation bias and possible loss of human expertise as tasks become automated.Source: §4.11 · p. 59
• The sustainability question includes energy use and emissions: one cited study reports inference emissions for ChatGPT-like services in one year at 25 times GPT-3 training emissions.The section says the model size that jointly meets performance, environmental, and organizational objectives remains unclear.Source: §4.11 · p. 59

• The agenda characterizes GenAI in SE as a fast-moving area with opportunities and open challenges, and presents a systematic view of its state of research as of October 2023.A comprehensive view, the authors argue, spans technical, user-centric, process, managerial, education, and business concerns.Source: §5 · p. 60
• The authors say substantial theory and validation in realistic project contexts are still needed to establish GenAI in SE as a mature research area.They also identify links with sustainability, trustworthy systems, and education.Source: §5 · p. 60
• The agenda may omit relevant SE aspects, and its research questions illustrate significant concerns rather than providing an exhaustive list.The authors invite additions of topics and research questions.Source: §5 · p. 60