Source-linked AI summary
The Prompt Triangle: A Registered Report on Prompts as Hybrid Artifacts
Shalini Chakraborty, Jan-Philipp Steghöfer
TL;DR
AI coding assistants have made prompts important software-engineering artifacts, but evidence about their evolution and effects remains limited. This Stage 1 registered report uses the Prompt Triangle and a multi-source confirmatory design to test four hypotheses about prompt patterns and development outcomes. Its principal outcome is a preregistered test of whether prompt evolution, developer characteristics, RE alignment, and temporal patterns predict success; public data provide exploratory generalisability evidence.
Problem
Empirical evidence is limited on how prompts evolve as hybrid requirements-and-solutions artifacts and how their structure relates to development outcomes.
Method
The registered report uses the Prompt Triangle, four preregistered hypotheses, and controlled, community-sourced, and mined interaction data to study prompt patterns and outcomes.
Results
The study protocol specifies confirmatory tests of prompt evolution, developer characteristics, RE activity alignment, and temporal alignment as predictors of development outcomes.
Takeaways & Limitations
Treating prompts as first-class artifacts provides a foundation for systematic empirical investigation of AI-assisted software development.
Takeaways & Limitations
Public conversations may be incomplete, outdated, or biased toward noteworthy or extreme examples, so mined data are exploratory and triangulated against controlled results.
Abstract
from arXiv · showhide
AI-based coding assistants are transforming software development by shifting effort from writing code to crafting prompts that guide code generation. Despite their growing importance, little empirical evidence exists on how prompts function as software engineering artifacts. Building on prior work framing prompts as mixed artifacts combining requirement intent and solution guidance, we build on the Prompt Triangle, a conceptual model describing prompts along three components: Functionality and Quality (requirements), General Solutions (architectural guidance), and Specific Solutions (implementation constraints). This registered report presents the Stage 1 protocol for a confirmatory study investigating how prompts evolve and how their alignment with requirements engineering (RE) activities predicts development outcomes. We preregister four hypotheses examining prompt evolution, developer characteristics, RE activity alignment, and temporal patterns. We employ a controlled experiment (n=30), community uploads, and mined data using dual-coding to test whether prompting patterns predict development success.
1 Introduction
AI coding assistants make prompts central software-engineering artifacts, but empirical evidence about their evolution and effects remains limited. This registered report uses the Prompt Triangle to test preregistered hypotheses about prompt development and outcomes.
- AI coding assistants shift development effort from code authoring toward constructing prompts that specify functionality, constraints, and design preferences.
- Prompts combine requirements, architectural decisions, and implementation constraints, making them hybrid artifacts spanning requirements engineering and solution design.
- Empirical work has rarely examined how developers iteratively refine prompts or whether prompt evolution influences final system quality.
- The Prompt Triangle analyzes prompts through Functionality and Quality, General Solutions, and Specific Solutions.
- The Stage 1 registered report tests preregistered hypotheses concerning prompt evolution, developer characteristics, RE alignment, and temporal alignment.
2 Background and Related Work
Prompting is increasingly central to AI-assisted development, yet prompts often lack formal-requirements structure and remain empirically understudied as artifacts. The Prompt Triangle addresses this gap by decomposing prompts into requirement- and solution-oriented components.
- Prompt engineering guides language models toward desired outputs but involves iterative refinement, correctness and trust challenges, and translation of requirements into prompts.
- Prior research links LLMs to elicitation, validation, and traceability while providing limited evidence about prompt-component evolution and impact.
- Research Gap: Existing studies examine prompting patterns, defects, output effectiveness, and RE applications, but do not systematically model prompts as artifacts encoding requirements and solution decisions.
- Research Gap: The Prompt Triangle enables empirical analysis by separating Functionality and Quality from General Solutions and Specific Solutions.
3 Hypotheses
The study formulates four hypotheses connecting prompt composition, developer characteristics, RE activity alignment, and temporal ordering to development outcomes. H4 tests whether when alignment occurs matters beyond overall alignment.
- The Prompt Triangle distinguishes prompt content from the RE activities—validation, verification, and solution generation—that give that content its workflow purpose.
- H1 predicts that Specific Solutions increase across multi-turn sessions relative to General Solutions and Functionality and Quality.
- H2 predicts that programming experience and domain familiarity are positively associated with the proportion of Specific Solutions.
- H3 predicts higher code quality when Functionality and Quality supports validation and Specific Solutions supports verification.
- H4 predicts higher code quality for conversations combining prompt evolution with correctly ordered RE activity sequences, despite equivalent overall alignment.
4 Study Overview
The study builds a developer–AI interaction corpus from controlled experiments, community prompt exports, and public datasets. This triangulated design combines controlled quality measurement with observational evidence intended to assess generalisability.
- The data-collection strategy combines controlled experiments, community-sourced prompt exports, and mining of publicly available datasets.
- The controlled experiment uses n = 30 participants to provide unconfounded causal estimates with full quality measures.
- Community uploads target n ≥150, while mined datasets provide observational evidence for assessing generalisability.
- The planned timeline sequences task design and piloting, controlled experiments, community collection, public mining, and overlapping annotation and quality control.
5 Data Collection
The study combines controlled experiments, community uploads, and mined public conversations to examine IDE-based AI interactions. It collects prompt, code, process, and participant data, then applies standardized filtering, annotation, and power-planned analysis.
- Controlled experiment: Thirty developers complete four counterbalanced 30-minute coding tasks spanning AI-assisted implementation, refactoring, algorithm implementation, and a non-AI bug-fix baseline.Tasks are separated by five-minute breaks within a planned 2:30-hour session.
- Data collected: Each task produces chat histories, final code, completion time, recordings, think-aloud transcripts, questionnaires, and AI tool information.Participants use their preferred IDE and export conversations through native tools where available.
- Annotation: Two researchers code conversation turns for Prompt Triangle components, requirement clarity, turn type, code presence, and solution-evolution stage, targeting Cohen’s κ > 0.75.Think-aloud verbalizations are additionally coded for requirements-engineering activities, with disagreements resolved after initial alignment.
- Community and mined data: Community uploads target at least 150 conversations, while public mining targets at least 200 naturally shared conversations using keyword searches and manual verification.Included community records require complete demographics, at least three turns, and at least 100 words; mined data remain exploratory.
- Data limitations: Public conversations may be incomplete, outdated, or biased toward noteworthy examples, so mined patterns are triangulated against controlled-experiment results.The protocol records publication dates and treats incomplete conversations as valid for turn-level analysis.
6 Data Analysis
The analysis tests prompt evolution, developer-driven strategy, requirements-engineering alignment, and temporal alignment using mixed-effects, regression, ANOVA, and dual-coding procedures. Code quality combines functional correctness with structural-quality measures.
- H1 Prompt evolution: Dirichlet mixed-effects regression models prompt-component proportions across turns because Functionality and Quality, General Solutions, and Specific Solutions sum to one.The preregistered evolution hypothesis predicts increasing Specific Solutions and decreasing Functionality and Quality, assessed with likelihood-ratio tests and 95% confidence intervals.
- H2 User-driven strategy: Hierarchical regression tests whether programming experience, domain familiarity, and AI-tool experience predict the Specific Solutions proportion beyond task controls.The hypothesis requires an incremental ∆R2 ≥.10 with p < .05; beta mixed-effects regression provides a bounded-proportion sensitivity analysis.
- H3 RE activity alignment: Dual-coding assigns each prompt segment both a Prompt Triangle component and an RE activity, targeting Cohen’s κ > 0.75 before scaling annotation through automated pre-labelling.RE activities include validation, verification, and solution generation.
- Outcome measure: Code quality is the standardised mean of functional correctness, measured by predefined-test pass rate, and structural quality, measured with static-analysis metrics.Structural measures include maintainability index, cyclomatic complexity, and code smells.
- H3 RE activity alignment: Alignment regressions model quality using F&Q-validation and Specific-Solution-verification alignment while controlling for component proportions, conversation length, task type, and developer experience.Hypothesis confirmation requires at least one significant positive alignment coefficient at p < .05.
- Outcome comparisons: Repeated-measures ANOVA compares manual, low-alignment AI, and high-alignment AI conditions in the controlled experiment with n = 30.The prediction is Manual ≈ Low-alignment < High-alignment, with high alignment exceeding both at Cohen’s d ≥0.5.
- H4 Temporal alignment: Hierarchical regression tests whether early specification-oriented and late refinement-oriented alignment predicts quality beyond overall alignment.Support requires ∆R2 ≥.05, p < .05, and positive βTP; sensitivity analysis uses continuous turn-weighted alignment.
7 Threats to Validity
The study organizes threats to validity using Wohlin et al.’s framework, covering construct validity, internal validity, external validity, and reliability.
- Threat framework: Threats to validity are structured around construct validity, internal validity, external validity, and reliability.The framework is summarized in Table 2 with corresponding mitigation strategies.
8 Ethical Considerations
The protocol requires institutional review before submission and establishes consent, withdrawal, anonymisation, and secure-storage protections for participants and their data.
- Review and consent: The revised methodology will be submitted to the University of Bayreuth Institutional Review Board, and all controlled-experiment and community-upload participants must provide informed consent.Participation is voluntary and participants retain the right to withdraw.
- Privacy and security: Data protections include removing personal and proprietary identifiers and storing data securely with encryption on institutional servers.Anonymisation removes names, email addresses, company identifiers, and proprietary code patterns.