Source-linked AI summary
The EPIC Framework for Spec-Driven Development
Bruno Claudino, Ruizhe Xing, Yuanhao Zuo, Tyler Menezes, Anita Sarma, Kostadin Damevski, Zixuan Feng
TL;DR
Coding agents can fill gaps in instructions with their own assumptions, while developers lack clear criteria for judging SDD specifications. The paper develops EPIC by relating artifact quality to requirements standards and repository examples, then evaluates the framework with practitioners and repository data. Top-band repositories had 15.4% lower median code complexity per KLOC, though the repository associations cannot establish causation.
Problem
Agents may fill missing information with assumptions, while developers lack criteria for judging whether artifacts state expectations clearly enough.
Source: §1 · p. 1, §5 · p. 17Method
EPIC applies requirements-quality expectations to SDD artifacts and their cross-document relationships, organizing guidance into 40 practices across ten dimensions.
Source: §5 · p. 16, §7 · p. 18Results
Top-band repositories had 15.4% lower median code complexity per KLOC, and complexity declined across the score range (ρ= −0.206).
Source: §5 · p. 16Takeaways & Limitations
EPIC provides criteria for writing and reviewing specification and planning artifacts, while its practices can be adopted as projects grow.
Source: §5 · p. 17, §7 · p. 18Without multivariate controls or stratified analyses, potential confounders remain, so repository associations cannot establish causation.
Source: §6 · p. 18
Abstract
from arXiv · showhide
One practitioner we interviewed said their team writes "must" instead of "should" when instructing a coding agent, because the agent may treat "should" as optional. Small wording choices matter because agents often fill gaps in their instructions with their own assumptions. Spec-driven development (SDD) asks developers to write a specification, plan, and tasks before the agent writes code. SDD frameworks provide templates for these artifacts, but the templates do not help developers judge whether they have written enough or clearly enough. We studied what good SDD specifications contain. We scored the artifacts of 114 open-source SDD repositories against ISO/IEC/IEEE 29148 and derived practices from the highest-scoring ones. The resulting framework, EPIC, has 40 practices in 10 quality dimensions that guide developers in making expectations and decisions explicit in specifications, plans, and tasks for coding agents. The majority of SDD practitioners (N=15) endorsed every practice. Repositories in the top third of specification quality spent 11.8% of their commits on bug fixes, compared with 20.4% in the bottom third, and had a median of 4x as many contributors. Developers can use EPIC to find and fill the gaps in a specification before the agent does.
1 Introduction
Coding agents can silently fill gaps in instructions, making the specification a key place for developers to make expectations explicit before code exists. EPIC offers practices for SDD artifacts, and repository quality was associated with bug-fix rates, complexity, and contributor counts.
1 Introduction
- Agents may fill missing instructions with assumptions, making it difficult to distinguish developer decisions from agent decisions after code exists.The specification is the artifact developers can review before any code exists.Source: §1 · p. 1
- SDD templates show where information belongs but do not tell developers whether a specification is sufficient or consistent with its plan.Requirements standards cover requirements, but not the plans and tasks SDD adds; developers therefore lack empirically grounded guidance for SDD artifacts.Source: §1 · p. 2
- EPIC organizes 40 practices into ten quality dimensions and four categories for writing and reviewing SDD artifacts.The dimensions include three additions addressing agreement among specifications, plans, and tasks, including dependencies and constraints, separating required behavior from implementation decisions, and planning information.Source: §1 · p. 2
- 11.8% of commits in top-third repositories were bug fixes versus 20.4% in the bottom third.The top third also had 15% lower median code complexity and a median of 12.5 contributors versus 3; these are observational associations, and bug and vulnerability density results were mixed.Source: §1 · p. 2
2 Background
SDD makes natural-language specifications persistent references for human-agent collaboration, but developers lack concrete criteria for judging their quality. EPIC addresses this gap with practices for specifications, plans, and tasks.
2 Background
- Earlier requirements and specification approaches addressed explicit intent, interfaces, and assumptions, while many assumed humans would write the code.Incomplete and inconsistent requirements have been reported as common problems associated with rework and overruns.Source: §2 · p. 3
- SDD frameworks turn natural-language intent into persistent, reviewable artifacts that guide planning, implementation, and validation across sessions.Kiro, GitHub Spec Kit, and OpenSpec organize those artifacts differently while sharing that premise.Source: §2 · p. 3
- Agents may silently resolve ambiguity, and small wording differences or repeated prompts can produce materially different implementations.Underspecified requirements are also a measurable source of errors in generated code.Source: §2 · p. 3
- Prior work examines user-story quality, weaknesses in executable specifications, and desirable agent-context properties, but leaves SDD artifact practices unresolved.EPIC responds with 40 practices across ten quality dimensions for specifications, plans, and tasks.Source: §2 · p. 3
3 Methodology
The study uses mixed methods to develop and evaluate EPIC, combining artifact analysis with practitioner feedback and repository-outcome associations.
3 Methodology
- The researchers used a mixed-methods design to identify and organize practices for SDD specification and planning quality.This addressed RQ1.Source: §3 · p. 3
- Figure 1 provides an overview of the study design.The supplied caption identifies it as an overview, without specifying visual encodings.Source: Figure 1 · p. 4
- They evaluated practitioners’ views of EPIC’s clarity and usefulness and examined associations between specification quality and code-quality and development-activity indicators.Figure 1 summarizes the study design.Source: §3 · p. 4
3.1 RQ1. Developing the EPIC Framework
The researchers built EPIC by defining a quality lens, assessing SDD repository artifacts, deriving practices from high-scoring examples, and refining those practices with practitioner guidance.
3.1 RQ1. Developing the EPIC Framework
- The study combined ISO/IEC/IEEE 29148:2018 and SDD tool guidance, artifact analysis, exemplar-derived practices, and grey-literature review to develop EPIC.The four-step process aimed to identify practices for specification and planning artifact quality.Source: §3.1 · p. 4
3.1.1 Establishing a Specification Quality Lens.
- The quality lens had ten dimensions: seven retained ISO characteristics, while three adapted or synthesized guidance for SDD artifacts.The additional dimensions address dependencies and constraints, requirements abstraction versus implementation decisions, and planning and task information.Source: §3.1.1 · pp. 4–5
3.1.2 Mining OSS Repositories for SDD Artifacts.
- Repository discovery and screening produced a final dataset of 114 open-source SDD repositories.The researchers used artifact names and framework structures to identify candidates, then screened for artifacts intended to guide or constrain an AI agent’s implementation.Source: §3.1.2 · p. 5
3.1.3 Deriving Candidate Specification Practices from SDD Artifacts.
- Researchers scored artifacts on ten dimensions, calibrated the rubric, and selected each dimension’s top-scoring 20% of repositories as exemplars.They independently coded exemplar artifacts and used card sorting to define 40 recurring practices; calibration yielded 80.0% exact agreement in the first round and 83% in the second.Source: §3.1.3 · pp. 5–6
3.1.4 Refining Practices with Practitioner Guidance from Grey Literature.
- Grey-literature review refined practice definitions but added no practices or dimensions; the final EPIC framework has four categories, ten dimensions, and 40 practices.The review mapped guidance from 30 sources to the candidate practices and used explanations of why and how to revise their definitions.Source: §3.1.4 · p. 7
3.2 RQ2. Evaluating the EPIC Framework
The evaluation combined practitioner interviews about EPIC’s clarity and usefulness with repository analyses relating specification quality to code quality and development activity.
3.2.1 Practitioner Evaluation of Clarity and Usefulness.
- Fifteen SDD-experienced professionals reviewed and rated EPIC’s ten dimensions and 40 metrics, explaining their reasoning in interviews.Recruitment used professional networks and referrals; 14 participants had at least six years of development experience.Source: §3.2.1 · pp. 7–8
- Researchers inductively coded participants’ explanations, resolved coding disagreements, and stopped recruitment after two consecutive interviews added no substantive feedback.After the first 13 interviews, two already scheduled interviews confirmed the saturation assessment.Source: §3.2.1 · p. 8
3.2.2 Specification Quality and Software Outcomes.
- Specification quality was scored from 0 to 30 by summing 0–3 ratings across ten dimensions, using specification and planning artifacts from 114 repositories.SonarQube analyses used 100 repositories after exclusions; GitHub activity analyses used 109.Source: §3.2.2 · pp. 8–9
- Repository outcome analyses tested whether specification quality related to reliability, security, structural maintainability, corrective work, and development activity.The authors hypothesized lower bug and vulnerability density and cyclomatic complexity, alongside fewer bug-fix commits and higher development activity.Source: §3.2.2 · pp. 8–9
- The study compared Spearman correlations across the score range with top-versus-bottom specification-quality tertiles, using Mann–Whitney U tests and Cliff’s δ.Observed bands were bottom ≤18, middle 19–22, and top ≥23; repositories tied at cutoffs stayed together.Source: §3.2.2 · p. 9
4 Results
EPIC organizes SDD specification and planning practices into four categories, ten quality dimensions, and 40 practices. Practitioners endorsed its metrics, while repository analyses found associations between specification quality and software outcomes.
4.1 The EPIC Framework for Specification-Driven Development (SDD)
- EPIC’s 40 practices span ten quality dimensions in four categories: Expressive, Precise, Interpretable, and Compartmentalized.The categories address clear, verifiable requirements; dependencies and completeness; cross-artifact understanding; and separation of required behavior from implementation planning.Source: §4.1 · pp. 9–11
- Expressive practices clarify unresolved decisions, state measurable bounds and normative wording, separate obligations, and specify verification criteria and completion conditions.Examples include marking unresolved decisions for clarification and defining what counts as satisfying a requirement and when verification is complete.Source: §4.1 · p. 10
- Interpretable practices support conventions, consistent terminology and identifiers, alignment across artifacts, and contextual organization and guidance.Compartmentalized practices keep user-facing behavior distinct from implementation decisions while recording technical context, task outputs, interfaces, and traceability.Source: §4.1 · pp. 11–12
- A majority of participants agreed or strongly agreed with every EPIC metric, and Interpretable received the highest proportion of agreement.Figure 2 presents the framework alongside participant ratings on a five-point scale.Source: §4.1 · pp. 12, 14, Figure 2 · p. 13
4.2.2 Associations with Code Quality and Development Activity.
- 15.4% lower median cyclomatic complexity per KLOC (137.2 vs. 162.3), 11.8% vs. 20.4% bug-fix commits, and 12.5 vs. 3 median contributors distinguished top from bottom bands.The bug-fix share and contributor medians were among the repository activity differences; the complexity difference was statistically significant.Source: Table 1 · p. 15, §4.2.2 · pp. 15–16
- Complexity showed the most consistent association across analyses, while bug density and vulnerability density were supported in only one analysis each.The complexity correlation was ρ = −0.206; bug density was associated across the full sample, and vulnerability density in the top-versus-bottom comparison.Source: §4.2.2 · p. 15
5 Discussion
EPIC extends requirements quality to relationships among specifications, plans, and tasks, while practitioner accounts and repository associations suggest when its practices may matter. The discussion also outlines incremental adoption and workflow uses, while cautioning that the examples do not establish reliability and the repository results are correlational.
What EPIC Adds to Requirements Quality
- Seven of EPIC’s ten dimensions retain characteristics from ISO/IEC/IEEE 29148:2018, while the other three address a specification accompanied by a plan and tasks; none of 30 grey-literature sources added a practice.Source: §5 · p. 16
Interpreting the Evaluation Results
- Every EPIC practice received agreement or strong agreement from a majority of participants, and teams can adopt practices incrementally as project needs grow.Normative wording, exceptional behavior and technical constraints, standardized terminology, and technical context each received agreement from 14 of 15 participants.Source: §5 · p. 16
- 15.4% lower median complexity per KLOC in the top score band was associated with higher specification quality, with complexity declining across the score range.The authors present explicit boundaries, constraints, technical context, and data models as possible explanations, not established causes.Source: §5 · p. 16
- 11.8% of scanned commits in top-band repositories were bug fixes, versus 20.4% in the bottom band; the bug-fix share also declined across scores (ρ= −0.210).The authors suggest verifiability practices may help explain this association; at least 12 of 15 participants endorsed each of the practices discussed.Source: §5 · p. 16
- Top-band repositories had a median of 12.5 contributors versus 3 in the bottom band, but teams with more contributors may also invest more in specifications.The authors suggest specifications may serve as onboarding documentation and note that this explanation needs further testing.Source: §5 · p. 17
Applying EPIC in Development Workflows
- EPIC criteria can guide clarification and revisions in SDD frameworks or specification checks in CI/CD workflows.The authors also packaged the practices as an agent skill that reviews artifacts together or applies practices while drafting and revising, marking unresolved decisions for clarification.Source: §5 · p. 17
Packaging EPIC as an Agent Skill
- Figure 3 illustrates authoring: the skill added an R790 setup step omitted from its instructions and turned unresolved file policies into clarification questions, but these examples do not measure scoring or revision reliability.Future benchmarks could use EPIC to compare specifications that include, weaken, or omit a practice, measuring across models and tasks whether agents ask, decide acceptably, or violate a constraint.Source: §5 · pp. 17–18
6 Limitations
The study’s limitations include possible gaps in EPIC’s coverage, researcher judgment in repository scoring, and an evaluation that did not test EPIC in participants’ own projects. Repository-based outcome findings are exploratory because of sample constraints, potential confounders, and limited generalizability.
Limitations in developing EPIC
- EPIC may omit quality dimensions or practices relevant to SDD despite drawing on standards, tool guidance, repository artifacts, and practitioner feedback.The authors used these perspectives to reduce the risk of omissions and refine practices.Source: §6 · p. 18
- Repository scoring and exemplar selection involved researcher judgment; four researchers developed the rubric and two calibration rounds checked its consistent application.The researchers also complemented repository findings with grey-literature guidance.Source: §6 · p. 18
Limitations in evaluating EPIC
- EPIC was not tested in participants’ own projects; future evaluation is planned with developers of varied backgrounds and experience as workflows and agent capabilities evolve.The authors describe the current evaluation as an initial empirical foundation.Source: §6 · p. 18
- The 15 interview participants had varied backgrounds and development experience, were screened for SDD experience, and were shown a draft with neutral follow-up questions to reduce response bias.The authors report disagreements and explanations alongside agreement as evidence participants challenged the framework.Source: §6 · p. 18
- The limited repository sample reduces estimate precision and power to detect smaller effects; uncontrolled project characteristics may confound associations, and public English-language repositories limit generalizability.SonarQube results also vary across languages and project structures, so the authors treat findings as exploratory.Source: §6 · p. 18
7 Conclusion
EPIC provides 40 practices across four categories and ten quality dimensions to guide writing and reviewing SDD specification and planning artifacts. The authors identify in-use evaluation and agent-interpretation benchmarks as future priorities.
7 Conclusion
- EPIC comprises four categories, ten quality dimensions, and 40 practices for writing and reviewing specification and planning artifacts in SDD.The categories are Expressive, Precise, Interpretable, and Compartmentalized.Source: §7 · p. 18
- Future work will evaluate EPIC in real development settings with developers of different experience levels, including longitudinal student capstone studies.The studies are intended to examine effects on specification practices and resulting software.Source: §7 · p. 18
- Developing benchmarks for how coding agents interpret and act on EPIC’s guidance is identified as an urgent research priority.Source: §7 · p. 18
- The research artifacts are publicly available on the companion website, and EPIC is available separately.Source: Data Availability · p. 18