PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design

Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, Douglas C. Schmidt

arXiv:2303.07839v1cs.SEcs.AI

TL;DR

The paper addresses how to use LLMs more systematically for software-engineering activities beyond direct code generation. It catalogs reusable prompt patterns and examines their application to requirements elicitation, system design, code quality, refactoring, and rapid prototyping. The paper concludes that these patterns support rapid experimentation, while effective use still requires substantial human scrutiny because LLMs can generate confident incorrect output.

  • Problem

    LLM use in software engineering lacks a systematic catalog of reusable prompt designs for recurring requirements, design, code-quality, and refactoring problems.

  • Method

    The paper documents and applies prompt patterns, including combinations and sequences, across software-engineering lifecycle activities using ChatGPT examples.

  • Results

    The explored patterns support requirements elicitation, system simulation, visual prototyping, code-quality constraints, and high-level refactoring specification.

  • Takeaways & Limitations

    A reusable catalog and guidance for combining prompt patterns can enable rapid experimentation at multiple abstraction levels throughout software engineering.

  • Takeaways & Limitations

    Effective use currently requires substantial human involvement and expertise because ChatGPT may confidently generate incorrect output.

Abstract

from arXiv · show

This paper presents prompt design techniques for software engineering, in the form of patterns, to solve common problems when using large language models (LLMs), such as ChatGPT to automate common software engineering activities, such as ensuring code is decoupled from third-party libraries and simulating a web application API before it is implemented. This paper provides two contributions to research on using LLMs for software engineering. First, it provides a catalog of patterns for software engineering that classifies patterns according to the types of problems they solve. Second, it explores several prompt patterns that have been applied to improve requirements elicitation, rapid prototyping, code quality, refactoring, and system design.

I. INTRODUCTION

The paper frames prompts as reusable instructions that program LLM interactions for software engineering, and introduces a catalog addressing requirements, design, code quality, and refactoring concerns.

  • LLMs such as ChatGPT and GitHub Copilot are increasingly used to generate code and other software-engineering artifacts.
  • Software-engineering prompts provide context and guide LLMs to generate requirements, code, documentation, build scripts, or simulated system behavior.
  • Prompt patterns are reusable prompt designs that codify solutions to recurring LLM-interaction problems, including design-principle and secure-coding constraints.
  • Prompt patterns can improve modularity and reusability by separating business logic from side effects and inserting abstractions between third-party libraries and dependent logic.
  • The paper extends prior work with a software-engineering catalog spanning system simulation, requirements, code generation, refactoring, and pattern combinations.
  • Prompt patterns use a structure analogous to software patterns, documenting intent, motivation, structure, implementation examples, and consequences.
  • A typical pattern begins with a scoping statement such as “from now on” and specifies conditional output rules for future software-engineering tasks.
  • The Output Automator example instructs ChatGPT to create a Python script whenever generated code spans multiple files, reducing manual file-editing work.

III. A CATALOG OF PROMPT PATTERNS FOR AUTOMATING SOFTWARE ENGINEERING TASKS

The catalog organizes 13 prompt patterns into four categories covering requirements, system design, code quality, and refactoring. It targets early identification of specification problems and structured improvement of software artifacts.

  • 13 prompt patterns are partitioned into four categories to help users navigate and apply them to software-engineering tasks.
  • Requirements elicitation and system design have received relatively little attention despite their importance and their association with late changes, cost, and risk.
  • Requirements Elicitation: Requirements Elicitation patterns support creating requirements, checking completeness and accuracy, and reasoning about the impact of changes.
  • System Design & Simulation: System Design & Simulation patterns address concrete specifications, domain-specific languages, alternative architectures, and early discovery of deficiencies.
  • Code Quality: Code Quality patterns target abstraction and modularity in human- and LLM-generated code, including replaceable third-party libraries through interfaces.
  • Refactoring: Refactoring patterns use LLMs’ understanding of abstract constructs, including pseudocode, to specify refactorings at a high level.
  • The catalog was developed from community-posted prompts and independently created prompts used in software-engineering work with ChatGPT.
  • Requirements Elicitation: The Requirements Simulator lets stakeholders interactively test whether desired functionality is captured and elicit missing requirements.

4) Example Implementation:

The examples show how the Requirements Simulator and Visualization Generator can support interactive requirements exploration and wireframe creation, while highlighting textual descriptions’ limits.

  • Requirements Simulator: The implementation assumes requirements are already in the LLM’s context, either entered manually or generated through earlier prompts.
  • Requirements Simulator: The Requirements Simulator treats the LLM as the system, uses requirements to guide behavior, and reports whether requested actions are possible.
  • Requirements Simulator: When an action is unsupported, the simulator writes missing requirements as user stories; when supported, it provides step-by-step instructions and implementation details.
  • Visualization Generator: Combining requirements simulation with visualization adds wireframe prompts to textual screen descriptions, supporting visual exploration of a web application.
  • Limitations: Using the Requirements Simulator in isolation can leave requirements insufficiently captured because system descriptions remain purely textual.
  • Example: The examples include a web application for socially sharing ChatGPT prompts, with wireframes generated for a screen where users delete their prompts.
  • Visualization Generator: Generated visualizations are rough interpretations, but may be easier to reason about than text and may inspire unexpected interface ideas.

3) Structure and Key Ideas:

The paper presents prompt patterns for clarifying requirements, exploring system behavior, and generating API specifications. These patterns help expose ambiguity and support earlier, more actionable design decisions.

  • Specification Disambiguation: Product managers can use follow-up prompts to turn vague requirements into more precise and actionable specifications.For example, a request to keep uploaded files secure can be expanded into access-control and encryption requirements.
  • Specification Disambiguation: Specification Disambiguation uses an LLM as an automated devil’s advocate to identify weaknesses, ambiguities, and unintended interpretations in requirements.It is intended to catch communication problems before development and can support specifications for internal teams or external audiences.
  • Specification Disambiguation: The pattern supports independent ambiguity reviews by collaborating teams before a joint development kick-off.This provides an unbiased source of discussion topics for teams integrating separate system components.
  • Pattern combinations: The pattern can be combined with Persona, API Generator, API Simulator, and Requirements Simulator patterns to uncover ambiguities through multiple representations and interactions.Transforming requirements into APIs or simulated behavior may reveal unexpected interpretations, while personas add alternative viewpoints.
  • API Generator: API Generator converts natural-language requirements into API specifications to support rapid design exploration, early formalization, and later manual refinement.The generated specification provides a starting point rather than replacing refinement.

2) Motivation:

The paper motivates API generation and simulation by the cost of manual specification and the value of testing designs before implementation. It describes a prompt structure that directs the LLM to act according to a specification and respond in a defined format.

  • Motivation: Manual API specification is costly, which can limit design exploration, delay systematic specification, or leave code as the only integration reference.These pressures motivate automating API production from requirements or system descriptions.
  • API Generator: API Generator enables teams to create and compare multiple API designs before selecting a final design.Manual API writing is described as tedious, so automation expands the set of options developers can examine.
  • API Generator: Automated API production can encourage earlier specification because developers need not manually duplicate information in both specifications and code.Code-generation tools can then be combined with specifications to accelerate implementation.
  • API Simulator: API Simulator lets developers interact with an API immediately, while the LLM generates synthetic data, sample requests, usage guidance, and error explanations.This provides a simulation-based way to explore a proposed API before implementation.
  • API Simulator: The prompt structure specifies the system’s role, the request format, and the response format for simulated API interactions.The contextual statements respectively define the specification to follow, user input, and expected output.

4) Example Implementation:

The examples show how an LLM can simulate APIs and generate usage examples from specifications or code. These prompts support natural-language scenario customization and provide compact examples for later prompting.

  • API Simulator: An API Simulator can act as a web application based on an OpenAPI specification and return HTTP responses to typed requests.The example uses plain-text HTTP requests and corresponding HTTP responses as the interaction format.
  • API Simulator: Users can customize simulated API scenarios in natural language by setting assumptions such as authorization state or the number of existing users.The LLM then responds with sample data under those assumptions.
  • API Simulator: API simulations may accept either strict HTTP requests or more flexible pseudo-data inputs that the LLM reformats.This allows users to choose between rigorous programmatic interaction and conversational exploration.
  • Few-shot Code Example Generation: Few-shot Code Example Generation creates usage examples that can be reused in prompts to remind the LLM how a system is designed and used.The pattern is motivated by token limits that make it difficult to provide an entire large system or module as context.
  • Few-shot Code Example Generation: The pattern can generate N examples for a system, optionally emphasizing public interfaces or a selected area of functionality.The examples are intended to be complete in coverage and can focus on a task such as registering new users.
  • Few-shot Code Example Generation: Code examples can convey ordering, required data, and other usage details more compactly than natural-language descriptions.The paper illustrates this with prompts requesting ten examples for an OpenAPI specification or a codebase.

5) Consequences:

The patterns support iterative system exploration, reusable usage examples, and token-efficient DSL-based design. Their benefits depend on choosing representations and syntax that remain sufficiently informative and usable.

  • Few-shot Code Example Generation: Few-shot examples can capture expected usage early and later reintroduce prior design decisions to the LLM.Combined with API simulation, developers can record interactions and supplement them with generated examples.
  • Few-shot Code Example Generation: Few-shot Code Example Generation is best suited when usage examples express constraints, assumptions, or expectations more efficiently than prose.A document such as an OpenAPI specification may sometimes be more token-efficient, even though examples can aid problem solving.
  • DSL Creation: DSL Creation lets an LLM design a domain-specific language for describing and manipulating requirements, deployment, security, or architecture concepts.The LLM can also explain the DSL and generate examples that are stored for later prompts.
  • DSL Creation: A DSL may help system design stay within LLM token limits, but syntax choices directly affect the space-efficiency of its representation.The paper contrasts more space-consuming XML-like syntax with YAML-like syntax.
  • DSL Creation: Positional conventions can improve DSL token efficiency, but they may make the language harder for unfamiliar users to interpret.Few-shot examples can teach users how to apply such a DSL and help document its usage for later prompting.

1) Intent and Context:

The Architectural Possibilities pattern helps developers explore alternative system architectures with little effort and supports rapid implementation and experimentation. The Change Request Simulation pattern helps estimate the impact and complexity of proposed system changes, but requires sufficient context.

  • Architectural Possibilities: The Architectural Possibilities pattern generates several architectures for developers to consider across modules, communication, or system tiers.The developer must clarify what “architecture” means for the desired output.
  • Architectural Possibilities: Architectural exploration can be combined with generated implementations, deployment scripts, workload tests, and cloud cost analysis.The paper gives monolithic and microservice alternatives as an example for performance-sensitive applications.
  • Change Request Simulation: The Change Request Simulation pattern helps stakeholders reason about which system aspects, modules, functions, or files a proposed change may affect.It can provide rapid feedback for conversations about schedule, cost, and risk.
  • Change Request Simulation: The pattern works best when the change is concrete and the LLM has enough system context, such as conversation history seeded by an API Generator.Insufficient context is identified as the hardest part of applying the pattern.

5) Consequences:

The patterns guide LLMs to structure code around explicit properties and to manage system-wide changes by iteratively moving between abstraction levels. They address token-limit constraints while improving clustering, modularity, and maintainability.

  • Change Request Simulation: The Change Request Simulation pattern alternates abstract and detailed analysis to identify affected features, modules, and implementation details within LLM token limits.The process zooms into individual modules or features, then repeats across high-level modules to estimate overall change impact.
  • Code Clustering: The clustering pattern restructures code into functions, classes, or modules according to properties such as side-effects, architectural tiers, or features.The desired cluster properties are specified to the LLM, which then restructures the code accordingly.
  • Code Quality: Explicit clustering context helps prevent LLM-generated code from becoming monolithic, brittle, messy, and difficult to maintain.Code decomposition affects what can be changed, extended, and maintained easily.
  • Code Clustering: A side-effect separation prompt isolates file-system, database, and network operations from functions without side-effects.The pattern can support testing and reasoning about business logic in isolation.

5) Consequences:

These patterns improve code structure by making desired modularity, dependency boundaries, and design principles explicit in prompts. Their effectiveness depends on the specificity and prior familiarity of the named guidance.

  • Dependency Decoupling: The Third-Party Library Decoupling pattern places an intermediate abstraction between business logic and external libraries so dependencies can be replaced.The prompt asks the LLM to separate business logic from underlying third-party libraries.
  • Dependency Decoupling: A single library may not provide enough information to design an abstraction that accommodates other dependencies with different architectures and interfaces.Few-shot examples of comparable libraries can help the LLM refactor the interface for broader implementability.
  • Named Design Principles: The Named Design Principles pattern uses familiar labels such as SOLID to communicate desired code structure without specifying every individual rule.SOLID expands to single responsibility, open-closed, Liskov substitution, interface segregation, and dependency inversion.

4) Example Implementation:

The paper illustrates prompt patterns that expose hidden code assumptions and connect them to future change and refactoring decisions. These prompts can identify assumptions that constrain database migration or other structural changes.

  • Hidden Assumptions Pattern: The Hidden Assumptions Pattern asks the LLM to identify assumptions embedded in a code section.The purpose is to make assumptions visible before users make decisions about using, modifying, or leveraging the code.
  • Hidden Assumptions Pattern: The pattern can optionally estimate how difficult each assumption would be to change or how likely it is to change.This refinement highlights potential liabilities in the current code structure.
  • Refactoring Applications: When an assumption is difficult to change but expected to change, the identified liability can guide an LLM refactoring request.The refactoring aims to remove the flawed assumption.
  • Refactoring Applications: A database-focused prompt asks which assumptions make code difficult to migrate from MongoDB to MySQL.The resulting list can serve as the basis for refactoring away from tight coupling to the underlying database.

5) Consequences:

The paper describes prompt patterns that give developers greater control over code refactoring without requiring explicit specification of every implementation detail. These patterns can automate refactoring for changed data formats, but may also produce broader changes than intended.

  • The assumption-identification pattern may miss assumptions outside the LLM’s supplied context, creating a risk that users treat partial results as authoritative.Its output is intended to flag possible assumptions for developer consideration rather than serve as a complete source of truth.
  • Pseudo-code lets users control the algorithm or code flow while leaving implementation details to the LLM, which is expected to produce correct, runnable code.The pattern adapts generated or refactored output to a pseudo-code template.
  • The pseudo-code defines an outline rather than exact line-level traceability, so the LLM infers the intended refactoring and maps it onto the existing code.The approach is designed to avoid requiring users to specify exact code structures.
  • Pseudo-code-guided refactoring may exceed the requested outline, including function removal and redistribution of code that changes the public interface.The broader refactoring can therefore affect externally visible code structure.
  • A schema-based pattern lets users request refactoring for a new input or output data format without specifying the exact logic changes.The LLM determines the necessary changes from the supplied format schema.

3) Structure and Key Ideas:

The paper positions prompt patterns as reusable interaction designs for applying LLM capabilities to software engineering problems. It reports potential benefits across automation and experimentation, while emphasizing that effective use still requires substantial human expertise and scrutiny.

  • Structure and Key Ideas: Prompt patterns codify reusable interaction designs for solving recurring problems in LLM-based software engineering.They complement software design patterns by focusing on interactions with an LLM rather than system structure itself.
  • Structure and Key Ideas: Prompt-pattern research complements prior work on code generation quality, security risks, bug fixing, and prompting LLMs to use outside tooling.The paper focuses specifically on interaction patterns for software engineering problems.
  • Structure and Key Ideas: Prompt patterns can help combat LLM mistakes and expose capabilities such as system simulation, API-specification generation, and assumption identification.The paper presents these capabilities as difficult to automate with existing technologies.
  • Structure and Key Ideas: The paper argues that codifying prompt catalogs and combination guidance could accelerate software engineering through rapid experimentation at multiple abstraction levels.The stated potential extends beyond code generation to tasks throughout the software engineering life-cycle.
  • Structure and Key Ideas: Effective automation currently requires significant human involvement because ChatGPT can confidently generate incorrect output, despite prompt patterns mitigating some issues.The paper identifies quality assurance and versioning as areas requiring further work.
Loading 2303.07839v1…