Source-linked AI summary
From Prompting to Engineering: A Research Agenda for Prompt Engineering in Software Engineering
Vincenzo De Martino, Giovanna Broccia, Fabiano Pecorelli, Jennifer Horkoff, Riccardo Coppola, Antonino Ferraro, Quim Motger, Emma McKenzie, Shahbaz Siddeeq
TL;DR
Prompt engineering in software engineering remains largely informal, with limited support for systematic evaluation, reuse, traceability, and governance. Based on PROMPT-SE 2026 discussions, the paper proposes a community-informed research agenda spanning five areas to support more systematic and maintainable practices.
Problem
Prompt engineering in SE is increasingly relevant but remains weakly integrated into established engineering processes, limiting systematic design, evaluation, reuse, traceability, and governance.
Method
The paper synthesizes structured community discussions from PROMPT-SE 2026 into five research areas for integrating prompt engineering into software development.
Results
The discussions identify standardization, evaluation, and lifecycle thinking as cross-cutting themes connecting five areas for SE contributions to prompt engineering.
Takeaways & Limitations
Prompts and surrounding context should be studied as engineering artifacts with concerns for evaluation, traceability, maintainability, and governance, rather than optimized only as individual formulations.
Takeaways & Limitations
Whether specification-driven prompting changes developer work, for which tasks, and with what consequences remains an open empirical question, especially during maintenance, debugging, and long-term evolution.
Abstract
from arXiv · showhide
Prompt engineering is increasingly used across Software Engineering (SE) activities, including requirements analysis, coding, testing, documentation, repository analysis, and planning. Yet prompts and related instruction artifacts are often created and evolved through task-specific and informal practices, with limited support for their systematic evaluation, management, traceability, and governance. To examine how SE can contribute to the maturation of these practices, we organized a structured community discussion at the First International Workshop on Empirical Prompt Engineering for Software Engineering (PROMPT-SE), co-located with EASE 2026. Participants discussed current prompting practices, challenges to their adoption and evaluation, and future directions for integrating prompt engineering into software development. We synthesized these discussions into five areas: prompt artifacts and standardization; evaluation and benchmarking; lifecycle integration; human-AI collaboration and skills; and governance, privacy, and technical debt. Based on these areas, we outline a research agenda to move prompt engineering from predominantly ad hoc interactions toward more systematic, maintainable, evaluable, traceable, and governable SE practices.
1. INTRODUCTION
Prompt engineering is increasingly relevant across Software Engineering activities, but its practices remain largely informal and weakly integrated into established processes. PROMPT-SE 2026 synthesized community discussions into five focus areas for moving toward systematic prompt engineering.
- Language Models support requirements engineering, development, testing, maintenance, repository analysis, documentation, and empirical research.
- Prompt engineering shapes model behavior through instructions, context, examples, constraints, and interaction protocols.These elements can affect quality, reliability, reproducibility, cost, and maintainability.
- In SE, prompts are often created through individual experience, trial and error, or task-specific adaptation, with limited support for design, evaluation, reuse, traceability, and governance.
- Moving beyond ad hoc prompting requires treating prompt-related artifacts as first-class engineering artifacts and addressing human-AI collaboration, specification, evaluation, privacy, compliance, and technical debt.
- The workshop synthesized discussions into five areas: prompt artifacts and standardization, evaluation and benchmarking, workflow integration, human-AI collaboration and skills, and governance, privacy, and technical debt.
2. WORKSHOP CONTEXT AND METHOD
PROMPT-SE 2026 used structured community discussion to identify recurring practices, adoption and evaluation challenges, and future research directions. Its resulting agenda is a community-informed synthesis rather than an exhaustive empirical taxonomy.
- The workshop brought together researchers and practitioners interested in the empirical study of prompt engineering for Software Engineering.
- Accepted contributions covered conventional commit classification, requirements goal extraction, qualitative coding, multi-agent code generation, energy-aware prompting, and novice developer support.
- Six participants were divided into three groups examining current practices, adoption and evaluation challenges, and future directions.
- Groups recorded observations, presented them in plenary, and had their notes consolidated into thematic areas by the organizers.
- The observations were synthesized into five technical and human-organizational focus areas.
3. TECHNICAL DIMENSIONS
Prompt engineering spans diverse SE contexts, and participants identified prompt artifacts, evaluation, and workflow operationalization as recurring technical concerns. Effective prompting also depends on context, domain expertise, LM knowledge, and SE judgment.
- Prompt engineering research covers commit classification, requirements analysis, qualitative SE-data coding, energy-aware prompting, novice support, and multi-agent code generation.
- Prompt artifacts and standardization: Prompt artifacts include instructions and contextual elements deliberately assembled, stored, or reused for an LM-based SE task.
- Prompt artifacts and standardization: Code-generation artifacts may encode functionality, architectural constraints, coding conventions, testing expectations, and desired-output examples.
- Prompt artifacts and standardization: Requirements and repository-analysis artifacts can incorporate stakeholder needs, domain assumptions, project history, issue reports, and organizational context.
- Human-AI collaboration and skills: Effective prompt engineering may depend on the interaction among LM-behavior knowledge, domain expertise, and SE judgment.Prompting expertise alone may be insufficient without application, project, or organizational knowledge.
The Need for Documentation and Standardization
Documentation and standardization are presented as foundations for reproducible, traceable, reusable, and maintainable prompt engineering. The proposed approach combines shared reporting principles with task-specific extensions and established SE practices.
- Without recorded prompt artifacts, it is difficult to understand why outputs were produced, relate them to original intent, or track instruction and context evolution.This limits reproducibility, traceability, and accountability.
- Relevant metadata includes task goals, model and version, prompt text, system instructions, context, examples, constraints, output format, evaluation criteria, and known limitations.
- Sensitive or high-impact tasks may additionally require data-sensitivity information, privacy constraints, human-review requirements, and links to downstream artifacts.
- Standardization should establish shared documentation and reporting principles while allowing task-specific extensions rather than imposing one template.This can make prompt artifacts easier to understand, reproduce, reuse, and compare.
- Requirements documentation, configuration management, design rationale, traceability, code review, and artifact versioning provide foundations for systematic prompt management.
3.2 Evaluating Prompt Engineering in Software Engineering
Reliable prompt engineering evaluation in Software Engineering requires examining more than prompt structure or one output, using complementary targets and criteria that reflect practical trade-offs.
- 3.2 Evaluating Prompt Engineering in Software Engineering: Teams often build confidence through expertise, manual review, testing, and informal evaluation, but these practices vary across tasks, teams, and organizations.
- What Should Be Evaluated?: Evaluation may target the prompt, output, interaction process, or generated artifacts, because useful outputs may fail to generalize across tasks, models, or contexts.
- What Should Be Evaluated?: Prompt characterization records execution dimensions such as length, readability, examples, roles, constraints, context, and generation parameters.This supports comparison of prompting strategies and can make dimensions explicit for Automatic Prompt Engineering processes.
- Multi-Dimensional Quality Criteria: A single criterion cannot capture prompt engineering quality; correctness must be considered alongside reliability, cost, latency, privacy, maintainability, reproducibility, sustainability, and trust.
- Multi-Dimensional Quality Criteria: The best prompt may be the one that achieves an acceptable trade-off among quality, effort, and risk rather than the most fluent response.
- Multi-Dimensional Quality Criteria: Non-determinism makes comparison difficult because the same prompt can produce different outputs across runs, models, or model versions.
- Multi-Dimensional Quality Criteria: Overconfidence and hallucinations are concerning because plausible but incorrect outputs can propagate into code, tests, documentation, or design decisions without proper review.
- Multi-Dimensional Quality Criteria: Prompt engineering often requires multiple iterations, consuming time, computational resources, and financial resources that should be included in evaluation.
Benchmarking and Reproducibility
Benchmarking prompt engineering in Software Engineering is hindered by heterogeneous studies and incomplete execution context, motivating realistic benchmarks, protocols, and regression-oriented evaluation.
- Heterogeneous tasks, models, datasets, prompts, metrics, and reporting practices make cross-study comparison and cumulative knowledge building difficult.
- Benchmarks and evaluation protocols should capture realistic Software Engineering scenarios rather than relying only on narrow evaluations.
- Reproducibility may require recording system prompts, conversation history, model versions, temperature settings, retrieved documents, and hidden tool interactions.
- Software Engineering can adapt empirical research, testing, quality assurance, benchmarking, and maintenance practices to evaluate prompt engineering rigorously.
- Prompt-based workflows should be monitored across model versions and evaluated for effects on downstream artifacts beyond one-shot accuracy.
- Systematic regression testing could assess whether prompt, context, model, or interaction changes preserve previously established behavior.A key open question is how to define regression when workflow behavior is non-deterministic.
- Prompt artifacts can support repeatable or partially automated workflows across requirements, planning, coding, testing, maintenance, and empirical research.
Toward Specification-Driven Development
Participants envisioned specification-driven Software Engineering in which developers express goals, constraints, requirements, and validation criteria while prompt-based artifacts remain traceable across the lifecycle.
- Specification-driven workflows would guide software-artifact generation through developer-supplied goals, constraints, requirements, and validation criteria.
- This vision may shift developer effort from direct artifact production toward specification, validation, and decision-making.
- Faster generation of alternative implementations may support earlier validation and interactive exploration of requirements and design choices.
- Lowering the barrier to artifact generation may increase the risk of producing software without sufficient understanding of quality, security, or maintainability implications.
- Whether development shifts toward “less typing, more thinking,” for which tasks, and with what consequences remains an empirical question.
- Prompt-related trace links may connect requirements, design decisions, generated outputs, tests, reviews, and subsequent revisions.
- Lifecycle process models could specify when prompts are created, reviewed, evaluated, reused, or retired and how human validation is integrated.
4. HUMAN AND ORGANIZATIONAL DIMENSIONS
Prompt engineering changes developer work and organizational responsibilities, requiring competencies in specification and evaluation alongside governance for privacy, accountability, and prompt-induced technical debt.
- LM-supported workflows may shift development from direct artifact production toward specification, review, validation, and orchestration.
- Generated artifacts may reduce developers’ familiarity with code, encourage over-reliance on model outputs, and leave artifacts insufficiently understood during maintenance and debugging.
- Effective prompt engineering requires AI-specific, domain, and organizational knowledge, because syntactic structure alone may omit relevant assumptions, constraints, or conventions.
- Education should address task specification, context selection, constraint formulation, output evaluation, privacy awareness, and recognition of model limitations.
- Software Engineering education can extend requirements, design, quality, testing, maintenance, and process instruction to cover prompts as participants in artifact production and evolution.
- Organizations must govern model access, shared information, interaction review and recording, and accountability for generated artifacts alongside privacy, security, compliance, cost, and technical debt.
- Prompt-based workflows may expose source code, requirements, logs, or organizational knowledge, creating trade-offs among external restrictions, local models, capability, cost, maintenance, and governance.
- Prompt-induced technical debt is influenced by prompts, context, generated artifacts, and human review, including debt from underspecification, missing context, weak evaluation, or insufficient review.
5. CROSS-CUTTING REFLECTIONS
Three cross-cutting themes—standardization, evaluation, and lifecycle thinking—frame the maturation of prompt engineering in Software Engineering. Together, they emphasize adaptable structure, systematic multidimensional assessment, and attention to persistent downstream artifacts.
- Standardization, evaluation, and lifecycle thinking emerged as the three cross-cutting themes connecting the workshop’s focus areas.
- Standardization: Standards should improve reproducibility, traceability, and governance without eliminating flexibility across tasks, domains, and organizations.
- Evaluation: Trustworthy evaluation requires quality models, benchmarks, human protocols, cost-aware metrics, and regression testing across prompts, models, contexts, and outputs.
- Lifecycle thinking: Prompt engineering should be assessed beyond individual interactions because generated code, tests, requirements, and explanations persist and influence later development.
6. RESEARCH AGENDA
The research agenda proposes broad questions for representing, evaluating, operationalizing, governing, and understanding prompt engineering within Software Engineering. It treats prompt artifacts as objects for systematic study across workflows, quality dimensions, organizational controls, and human roles.
- The agenda presents five broad research questions as a community-informed starting point rather than an exhaustive agenda.
- Prompt artifacts: Prompt artifacts need metadata, documentation, versioning, review, reuse, retirement, and traceability to requirements, design, code, tests, and downstream artifacts.
- Evaluation: Evaluation research should characterize prompts and executions, assess artifact, outcome, and process quality, and develop benchmarks, regression tests, and multidimensional metrics.
- Workflow integration: Operationalization research should match prompting strategies to SE activities and examine automation through prompt selection, refinement, and APE.
- Human-AI collaboration: Research should examine how LM-supported workflows redistribute specification, implementation, validation, and orchestration, and how knowledge and prolonged reliance affect developer understanding and judgment.
- Governance: Governance research should address model selection, sensitive information, human review, auditability, compliance, accountability, and prompt-induced technical debt.
7. CONCLUSION
The article identifies five areas where Software Engineering can contribute to maturing prompt engineering practices. Its central conclusion is that prompts and surrounding context should be evaluated, evolved, and governed as software-related artifacts over time.
- The article identifies five contribution areas: artifacts and standardization, evaluation, workflow integration, human-AI collaboration, and governance.
- The agenda is a community-informed starting point, not an exhaustive taxonomy, and argues against focusing only on optimizing individual prompt formulations.
- Prompt practices participate in specifying, producing, and evolving software artifacts, bringing concerns of evaluation, traceability, maintainability, and governance.
- Prompt-based practices should be built so they can be evaluated, evolved, and trusted over time.