Source-linked AI summary
Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization
Alessandro Midolo, Alessandro Giagnorio, Fiorella Zampetti, Rosalia Tufano, Gabriele Bavota, Massimiliano Di Penta
TL;DR
Code-generation prompts lack software-development-specific optimization guidelines, despite prompt elements affecting generated code quality and correctness. This paper derives 10 guidelines through test-driven prompt refinement and manual analysis, then validates them with 50 practitioners, finding that usage varies while less-used patterns can still be perceived as particularly useful.
Problem
Software development lacks prompt-engineering guidelines and tools tailored to technology, specifications, and requested solutions.
Method
The authors iteratively refine prompts using test feedback, manually analyze successful changes, and validate the resulting guideline catalog with 50 practitioners.
Results
Participants used I/O formatting and pre-post conditions more often than ambiguities and I/O examples, while perceiving the less-used patterns as particularly useful.
Takeaways & Limitations
Developers and educators can leverage the guidelines for code-specific prompt engineering, while tool creators can use them for prompt guidance and optimization.
Takeaways & Limitations
The practitioner validation measures self-reported usage and perceived usefulness rather than effectiveness, actual usage, or usefulness during development tasks.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are nowadays extensively used for various types of software engineering tasks, primarily code generation. Previous research has shown how suitable prompt engineering could help developers in improving their code generation prompts. However, so far, there do not exist specific guidelines driving developers towards writing suitable prompts for code generation. In this work, we derive and evaluate development-specific prompt optimization guidelines. First, we use an iterative, test-driven approach to automatically refine code generation prompts, and we analyze the outcome of this process to identify prompt improvement items that lead to test passes. We use such elements to elicit 10 guidelines for prompt improvement, related to better specifying I/O, pre-post conditions, providing examples, various types of details, or clarifying ambiguities. We conduct an assessment with 50 practitioners, who report their usage of the elicited prompt improvement patterns, as well as their perceived usefulness, which does not always correspond to the actual usage before knowing our guidelines. Our results lead to implications not only for practitioners and educators, but also for those aimed at creating better LLM-aided software development tools.
1 INTRODUCTION
The paper develops software-development-specific guidelines for improving code-generation prompts, addressing the lack of targeted guidance and tools. It derives 10 prompt-improvement dimensions through automated refinement and evaluates their usage and perceived usefulness in a survey of 50 practitioners.
- Motivation: LLM-based code generation can produce working solutions from simple descriptions, but prompt elements such as system instructions and assigned roles significantly affect quality and correctness.Examples include presenting the model as an expert Python or Java developer.
- Problem: Specific details about parameters, return values, pre- and post-conditions, and exceptional cases can be decisive, yet software-development prompts lack dedicated optimization guidelines and tools.The paper argues that software prompts require elements specific to development tasks.
- Method: 10 code generation prompt improvement dimensions were elicited by analyzing textual additions from automated refinement that made benchmark test cases pass.The study used BigCodeBench, HumanEval+, and MBPP+ tasks with GPT-4o-mini, Llama 3.3 70B Instruct, Qwen2.5 72B Instruct, and DeepSeek Coder V2 Instruct.
- Evaluation: 50 practitioners reported varying pattern usage: I/O formats and pre- and postconditions were common, whereas by-example prompting and linguistic improvements were less frequent.Participants also assessed the perceived usefulness of the elicited patterns.
- Implications: Participants considered I/O examples particularly useful despite using them less often, supporting the guidelines’ value for practitioners, educators, and future automated prompt recommenders.Future recommenders could identify context-dependent missing prompt elements and suggest improvements.
2 RELATED WORK
Related work spans automated prompt optimization, interactive human refinement, prompt-property analysis, and software-development tooling. This work addresses the limited transparency of predominantly black-box methods by offering empirically grounded, actionable guidelines for code-generation prompts.
- Automated optimization: Automated prompt optimization uses search, reinforcement learning, and human-feedback loops, but often provides limited transparency or actionable guidance.Ramnath et al. identify alignment and generalization as open challenges, while ProTeGi improves prompts through natural-language feedback.
- Human-guided refinement: Interactive and preference-based approaches show how human insights, conversational refinement, comparisons, and gradient-based search can improve or automate prompt crafting.Prior work contrasts interactive prompting with static prompts and develops interpretable optimization for discrete prompts.
- Prompt properties: Studies analyze prompt structure and features, linking layout, placeholders, composition, I/O examples, and method summaries to instruction following and code-generation quality.SPA predicts output characteristics without executing the LLM, enabling proactive prompt assessment.
- Development tooling: Research treats prompts as software-development artifacts, examining debugging, evolution, maintenance, documentation, realistic completion contexts, and linter-like tooling.RepoSim incorporates recent file edits and temporal patterns, while PromptSet provides developer-written prompts and motivates static analysis.
- Domain and context: Other work addresses requirements-language quality, prompt scalability, domain-specific repair, and lightweight program analysis to improve LLM-driven software tasks.Examples include linguistic-smell analysis, UML-path selection, TypeFix for Python type errors, and semantic-feature augmentation for summarization.
3 STUDY DESIGN
The study develops code-generation prompt guidelines through test-driven, iterative optimization across three Python benchmarks and four LLMs, followed by manual analysis. It evaluates optimized prompts and consolidates the findings into 10 guidelines.
- Study scope: The study targets zero-shot prompting for source-code generation across BigCodeBench, HumanEval+, and MBPP+, using four state-of-the-art LLMs.The models are GPT-4o mini, Llama 3.3 70B Instruct, Qwen2.5 72B Instruct, and DeepSeek Coder V2 Instruct.
- Research questions: The study asks which prompt information maximizes code-generation success and how the resulting guidelines reflect practitioners’ prompt-improvement strategies.RQ1 uses automated prompt optimization and manual analysis to identify elements needed for correct code generation.
- Code optimization: For each task and LLM, researchers generated 10 implementations, retained tasks with repeated failures, and iteratively revised code using test-failure feedback for up to five rounds.Generation used zero temperature and a fixed seed, while multiple outputs addressed remaining nondeterminism.
- Prompt optimization: 153, 192, 200, and 145 instances produced optimized prompts for GPT-4o mini, Llama 3.3, Qwen2.5, and DeepSeek, respectively, enabling manual guideline analysis.The optimized-prompt procedure used failing code-and-error pairs together with the test-passing code.
- Code optimization: 229, 261, 268, and 228 instances were successfully optimized by GPT-4o mini, Llama 3.3, Qwen2.5, and DeepSeek, respectively.Only tasks whose code eventually passed the tests progressed to prompt optimization.
- Guideline extraction: Manual scrutiny consolidated similar findings into 10 prompt-improvement guidelines, including clarifying input/output formats, post-conditions, and ambiguous behavior.A representative refinement made behavior explicit for positive, negative, and zero inputs.
4 RESULTS DISCUSSION
The study elicited 10 code-generation prompt guidelines spanning I/O, pre/post-conditions, requirements, algorithmic details, exceptions, terminology, and ambiguity. Practitioners reported using prompt improvements with varying frequency, especially for I/O and conditions, while requirements, exceptions, algorithmic details, and terminology were used less consistently.
- Guidelines: 23% of optimized prompts specified post-conditions, compared with 7% specifying pre-conditions and 19% specifying requirements.Examples include output guarantees, valid inputs, and explicit dependency purposes.
- Guidelines: 44% of optimized prompts clarified input/output formats, making this the second-most applied guideline.The guideline specifies data types, shapes, structures, and special or edge cases.
- Guidelines: 10 guidelines addressed code behavior and specification, including I/O formats, pre/post-conditions, requirements, exceptions, algorithmic details, terminology, and ambiguous conditions.The guidelines also covered textual and technological details intended to improve generated-code correctness.
- Practitioner usage: Participants reported performing prompt improvements across four frequency ranges: 25% of prompts (11), 25-50% (21), 50-75% (12), and over 75% (6).These figures summarize reported improvement frequency across the study participants.
- Practitioner usage: I/O specification was especially used, while pre- and post-conditions were also frequent, with most respondents applying them in at least 25-50% of prompts.Very frequent use of pre- or post-conditions, meaning over 75% of prompts, was limited to 10% of respondents.
- Practitioner usage: Exceptions, algorithmic details, and variable terminology were used less consistently, with only 10% reporting each pattern in over 75% of prompts.Algorithmic details may apply only when behavior cannot be inferred, while terminology was viewed as a linguistic optimization.
5 THREATS TO VALIDITY
The study acknowledges construct, internal, and external validity threats affecting measurement, study procedures, participant evaluation, and generalizability. Mitigations addressed nondeterminism, evaluator subjectivity, and questionnaire design, while limitations remained regarding self-reported usage, self-selection, benchmarks, Python, and participant representativeness.
- Construct validity: Construct validity is limited because the study excludes LLM test-case overfitting and measures neither guideline effectiveness nor exact actual usage.Validation instead relies on participants’ self-reported use admission.
- Internal validity: Internal validity was addressed by precautions against nondeterminism and by two evaluators reviewing and resolving inconsistent guideline-elicitation cases.The authors state that agreement by chance may not influence the study’s conclusions in RQ1.
- Internal validity: Questionnaire threats were mitigated through established survey-design guidelines and validation by authors who were not involved in questionnaire design.The participant sample showed diverse experience and expertise, but self-selection bias could not be excluded.
- External validity: External validity may be limited by the three benchmarks used—BigCodeBench, HumanEval+, and MBPP+—and by considering only Python.The benchmarks were selected as general-purpose and diverse in terms of code tasks, but their effect on generalizability cannot be excluded.
- External validity: Participants may not represent the broader developer population, particularly very expert developers, although they span substantial experience and expertise among LLM code-generation practitioners.The authors consider this coverage potentially representative enough of practitioners interacting with LLMs for code generation.
6 IMPLICATIONS
The guidelines provide practical guidance for developers using LLMs for code generation and can support prompt-engineering education. They also suggest directions for tools that analyze, recommend, and automatically improve prompts.
- Software development practitioners: The guidelines can serve as a reference for practitioners interacting with LLMs during code generation, with some optimizations broadly applicable and others context-dependent.Examples of broadly applicable optimizations concern I/O, pre- and post-conditions, examples, and textual ambiguity; requirements-related guidelines may depend on knowing the technology.
- Educators: The guidelines can form a key instructional component for teaching software engineers prompt engineering for code generation.Prompt engineering is increasingly included in software-engineering curricula.
- Tool creators and researchers: Tool creators and researchers could use the results to develop tools that analyze prompts and context, suggest improvements, and automatically repair missing or unclear elements.Such tools could inspect the surrounding context and improve prompts when needed elements are missing or unclear.
7 CONCLUSIONS AND FUTURE WORK
The paper defines 10 code-specific prompt optimization guidelines for LLM code generation and validates them through a study with 50 practitioners. The findings support implications for developers, educators, researchers, and tool creators, while motivating broader empirical evaluation and automated prompt-quality tools.
- Contributions: The authors derive 10 code-specific prompt optimization guidelines through test-driven LLM interaction and manual comparison of initial and final prompts.The iterative process targets test-passing prompts before asking the LLM to optimize them.
- Validation: A study with 50 practitioners evaluates both their existing use of the guideline patterns and their perceived usefulness.Participants report using patterns involving I/O formatting and pre-post conditions, while ambiguity-related patterns are perceived as particularly useful.
- Implications: The guidelines can support developers and educators in code-specific prompt engineering, as well as researchers and tool creators developing prompt guidance and automated code-generation tools.The proposed applications include improving code, instructing practitioners, and crafting tools for prompt guidance and optimization.
- Future Work: Future work will extend the investigation to other languages and tasks, test guideline usefulness through controlled experiments, and develop automated prompt-quality assessment and improvement tools.These directions are presented as work in progress.