Source-linked AI summary
Spec Kit Agents: Context-Grounded Agentic Workflows
Pardis Taghavi, Santosh Bhavani
TL;DR
AI coding agents can produce coherent but repository-incompatible artifacts because evolving codebases require context beyond structured workflows. Spec Kit Agents adds phase-level discovery and validation around a multi-agent SDD pipeline, improving judged quality across repository tasks and increasing SWE-bench Lite performance from baseline to 58.2% Pass@1.
Problem
Structured SDD workflows still permit context blindness, where coherent intermediate artifacts conflict with existing repository APIs, paths, dependencies, or conventions.
Method
Spec Kit Agents combines PM and developer agents with read-only pre-phase repository discovery and post-phase artifact and implementation validation.
Results
+0.15 judged quality improvement was achieved across 128 runs covering 32 features in five repositories, while test compatibility remained 99.7–100%.
Takeaways & Limitations
Context-grounded orchestration improved the strongest overall workflow quality and generalized to SWE-bench Lite, where hooks raised pass rate from 56.5% to 58.2%.
Abstract
from arXiv · showhide
Spec-driven development (SDD) with AI coding agents provides a structured workflow, but agents often remain "context blind" in large, evolving repositories, leading to hallucinated APIs and architectural violations. We present Spec Kit Agents, a multi-agent SDD pipeline (with PM and developer roles) that adds phase-level, context-grounding hooks. Read-only probing hooks ground each stage (Specify, Plan, Tasks, Implement) in repository evidence, while validation hooks check intermediate artifacts against the environment. We evaluate 128 runs covering 32 features across five repositories. Context-grounding hooks improve judged quality by +0.15 on a 1-5 composite LLM-as-judge score (+3.0 percent of the full score; Wilcoxon signed-rank, p < 0.05) while maintaining 99.7-100 percent repository-level test compatibility. We further evaluate the framework on SWE-bench Lite, where augmentation hooks improve baseline by 1.7 percent, achieving 58.2 percent Pass@1.
1 Introduction
Spec-driven development structures agent work through intermediate artifacts, but repository context blindness can still make coherent plans incompatible with existing code. Spec Kit Agents adds phase-level discovery and validation to ground artifacts in repository evidence and improve judged quality while preserving test compatibility.
- End-to-end feature delivery remains brittle because agents lack current architectural, dependency, and repository-convention context across multi-step workflows.Spec-driven development externalizes specifications, plans, and task checklists to make intent explicit and auditable.
- Context blindness occurs when intermediate artifacts are internally coherent but reference nonexistent APIs, invalid paths, or incompatible architectural conventions.Late discovery during implementation or testing can force backtracking and introduce further inconsistencies.
- Spec Kit Agents adds read-only discovery before each stage and validation after each stage, including repository checks after implementation.The hooks operate outside core prompts, enabling auditable traces and selective tool access.
- +0.15 judged quality improvement was observed on a 1–5 composite LLM-as-judge score across 128 runs covering 32 feature tasks in five repositories.Repository-level test compatibility remained 99.7–100%.
2 Related Work
Related work grounds agent decisions through retrieval, tool use, feedback, and constraints, while Spec Kit Agents extends validation into intermediate artifacts. Its distinctive focus is repeated, phase-specific grounding before and after reasoning steps rather than verification only after implementation.
- Agentic systems increasingly use structured stages, specialized roles, tool-mediated reasoning, and searches over intermediate reasoning states.
- Retrieval-augmented generation and browser- or tool-augmented systems condition decisions on external evidence to support factuality and task success.
- Reliability methods use self-critique, iterative refinement, explicit constraints, and executable checks such as tests, linters, and structured validation.
- Spec Kit Agents validates SPEC, PLAN, and TASKS artifacts before code generation while retaining post-implementation checks as a final gate.This design targets hallucinated APIs, invalid paths, and architectural mismatches earlier in the workflow.
3 Method
Spec Kit Agents orchestrates PM and developer agents through a state-machine workflow that produces explicit artifacts and applies phase-scoped repository grounding. Read-only discovery supplies localized evidence, while validation checks artifacts and executes repository tests or linters after implementation.
- System Overview and Workflow: Spec Kit Agents uses a state-machine orchestrator, a PM agent for requirements and prioritization, and a developer agent for artifacts and code changes.
- System Overview and Workflow: Full workflows produce SPEC.md for requirements, PLAN.md for file-level implementation touchpoints, and TASKS.md for an executable checklist before coding.
- Context-Grounded Agentic Workflows Layer: Context-grounding hooks operate at workflow boundaries over explicit artifacts rather than being embedded in the developer’s main prompt.The layer provides phase-scoped grounding and validation.
- Context-Grounded Agentic Workflows Layer: Read-only discovery probes repository files, conventions, APIs, modules, and history before each phase to condition generation on localized context.
- Context-Grounded Agentic Workflows Layer: Post-phase validation checks artifact consistency and repository compatibility, then runs unit tests and linters after implementation to detect regressions.Earlier checks verify paths, libraries, task feasibility, and ordering before mistakes compound.
- Evaluation Design: Quality scoring uses an independently evaluated 1–5 LLM-as-judge rubric covering completeness, correctness, style, and maintainability.Prompts, tool calls, artifacts, and execution traces are logged for each run.
- Evaluation Design: The experiments compare Baseline, Augmented, Full, Full-Augmented, Discovery-only, and Validation-only workflow configurations.
4 Experiments
Spec Kit Agents is evaluated across diverse repositories, workflow configurations, ablations, latency conditions, and SWE-bench Lite. Context-grounding improves judged quality and benchmark performance, with added runtime overhead.
- Evaluation setup: 128 runs cover 32 feature tasks across five repositories and multiple change types, including APIs, configuration, modules, refactors, and tests.The repositories include FastAPI, Airflow, Dexter, Plausible Analytics, and Strapi.
- Quality results: Full-Augmented improves judged quality from 3.51 to 3.66 (+0.15) over Full, with a statistically significant paired difference (Wilcoxon signed-rank, p < 0.05).Gains appear across repositories, especially FastAPI and Plausible.
- Ablation results: The combined grounding design achieves the strongest ablation result, while Validation-only outperforms Discovery-only relative to the Full baseline.The ablation compares pre-phase discovery and post-phase validation independently before assessing their combination.
- Latency results: Context-grounding adds modest overhead in the 40-minute family and larger overhead in the 90-minute family, framing latency as a quality–runtime trade-off.Latency comparisons use completed runs and are made only within each budget family.
- SWE-bench Lite: Spec Kit Agents achieves a 56.5% pass rate without hooks and 58.2% with context-grounding hooks on SWE-bench Lite.The benchmark contains 300 real-world software engineering issues.
5 Conclusion
The conclusion presents Spec Kit Agents as a context-grounded, spec-driven workflow whose strongest quality results support explicit repository-grounded orchestration. The approach is most appropriate for higher-risk or high-complexity tasks because it adds runtime overhead.
- Conclusion: Spec Kit Agents augments Spec Kit with phase-scoped discovery and validation hooks in a multi-agent, spec-driven workflow.The workflow uses explicit repository context during its phases.
- Conclusion: Across 128 runs covering 32 features, the context-grounded full workflow achieves the strongest overall quality.The conclusion attributes the result to stronger alignment among specifications, discovered repository context, and implementations.
- Conclusion: Additional runtime overhead makes the approach most appropriate for higher-risk or high-complexity tasks.The conclusion frames this as the principal practical boundary of the approach.
A Representative Task Set
The representative task set illustrates the diversity of repositories and change types included in the custom repository evaluation.
- A Representative Task Set: Representative examples span diverse repositories and change types in the custom repository evaluation.The passage characterizes the examples as a subset of the full task set.
B.1 Repository-Level Analysis on SWE-bench Lite
SWE-bench augmentation gains vary by repository family: hooks help most when tests directly exercise the defect and less when fixes require deeper integration context or under-specified behavior.
- Augmentation helps most when tests directly exercise the underlying defect.Discovery and validation hooks can align implementations with executable checks in test-adjacent code paths.
- Augmentation is less reliable on django and matplotlib, where failures involve deeper ORM or visualization-state logic weakly exposed by local unit tests.
- SymPy-like cases can remain difficult when mathematically subtle edge conditions are under-specified by available tests.
B.2 Model and Tool Versions
The experiments separate generation, execution, and evaluation roles, while context-grounding hooks are parameterized by workflow stages and intermediate artifacts where those artifacts exist.
- Generation, execution, and evaluation roles are separated to make the pipeline explicit.
- Full and Full-Augmented run hooks at Specify, Plan, Tasks, and Implement using workflow state and SPEC.md, PLAN.md, and TASKS.md.
- Baseline and Augmented generate no intermediate artifacts and proceed directly to implementation, with implementation-stage hooks when enabled.
B.4 Configuration Files
The appendix documents reproducibility configurations, execution conditions, and a failure taxonomy covering timeouts, validation, environment, repository checks, incomplete implementations, and interruptions.
- config.yaml defines system settings, while experiments/features.yaml, experiment_runner.py, and quality_evaluator.py support task specification, orchestration, and scoring.
- Experiments ran on a MacBook Pro with Apple Silicon and 32 GB RAM, using remote inference with approximately 200–500 ms network latency.Transient rate limits were handled with exponential backoff and logged in run metadata.
- Failures are categorized as budget or human-checkpoint timeouts, artifact validation failures, execution or environment failures, repository-check failures, incomplete implementations, and rate-limited or interrupted runs.