Source-linked AI summary
Introducing Consort: A Spec-First Agent Framework for Enforced, Test-Driven Development on Live Database Branches
Kevin Hartman
TL;DR
AI coding agents turn development frameworks into control systems, creating a need to distinguish how frameworks enforce clean, correct, maintainable output. The paper introduces Consort, which uses non-editable controls, role-separated agents, and live database branches; its claim that this improves output quality remains a pre-registered hypothesis rather than a demonstrated result.
Problem
Because agents can satisfy specifications in unintended ways and may weaken tests or validate less real behavior, the paper asks how frameworks enforce clean, correct, maintainable code.
Method
Consort uses a deterministic orchestrator, separate role agents, human-approved gates, immutable tests, and test-driven execution against a live, branched database.
Results
The paper’s analytic comparison establishes that Consort enforces controls the other frameworks request, but does not establish that this produces better code.
Takeaways & Limitations
The paper argues that enforcement should reside in controls the agent cannot edit, with specialized roles supporting maintainability.
Takeaways & Limitations
Consort’s live-database enforcement requires additional database dependency and machinery, making its weight unsuitable for throwaway prototypes.
Abstract
from arXiv · showhide
When an agent writes code, the development framework becomes the control system for a non-deterministic worker. Spec-first, agent-driven frameworks have gained rapid traction since 2025; the installable ones, GitHub Spec Kit, obra/superpowers, BMAD, and GSD, and our own, all capture intent through a specification or durable planning artifacts. Since they agree on capturing intent up front, what separates them is how each enforces the engineering discipline that keeps agent-written code clean, correct, and maintainable. Every framework enforces that discipline somehow; they differ in how. We characterize three modes: enforcement by persuasion (prompt discipline the model may ignore), by front-loaded structure (strong specs, then a trusted build), and through controls the agent cannot edit (a deterministic orchestrator, human-approved gates, immutable tests, and a green result that must pass against a live, branched database). We introduce Consort, a spec-first, test-driven agent framework built on the third, enforcing that discipline through controls the agent runs inside but cannot bypass, in which a deterministic orchestrator drives separate role agents through a spec-first design lane and a test-driven build lane on a live database branch. We argue that enforcing the tests and gates in code keeps agent-written code honest and verifiable, while its specialized roles, like the human roles before them, are what make it maintainable, claims we frame as a pre-registered, testable hypothesis.
I. INTRODUCTION
As agents become code authors, development frameworks must control a probabilistic process rather than merely support human developers. Spec-driven frameworks agree on capturing intent, but differ in how they enforce clean, correct, maintainable output; Consort proposes controls the agent cannot edit.
- Motivation: AI coding changes the framework’s role from developer convenience to control system for a non-deterministic worker.The framework must constrain a probabilistic process so its output can be trusted.
- The enforcement gap: Spec-driven frameworks agree on stating intent before coding, so their key difference is how they enforce engineering discipline afterward.Agents may satisfy specifications in unintended ways, making build-time enforcement the harder problem.
- Agent failure modes: Documented failure modes include disabling tests, overusing mocks, violating constraints, and overreaching beyond intended changes.These behaviors motivate human-owned immutable tests and stronger controls.
- Motivation: AI-assisted development is associated with rising duplication, falling refactoring, more defects reaching pull requests, and lower delivery stability.The evidence is correlational and partly industry-sourced, so the paper treats maintainability risk as motivating rather than settled causation.
- Contribution: Consort classifies enforcement as persuasion, front-loaded structure, or controls the agent cannot edit, and instantiates the third mode.Its design uses a deterministic orchestrator, separate role agents, and an enforced build lane against a live, branched database.
- Scope: The comparison covers real, installable frameworks in which agents write code and does not re-derive the underlying database layer.Hosted builders and fully autonomous issue-to-pull-request agents are named but not evaluated.
II. BACKGROUND AND RELATED WORK
Spec-driven development treats a specification as the durable statement of intent, but existing frameworks differ in how long that specification remains active and whether enforcement follows it. Related evidence motivates enforcement by documenting recurrent agent violations and weak validation of real behavior.
- Spec-driven development: Spec-driven development spans spec-first, spec-anchored, and spec-as-source practices according to how the specification relates to code.Spec-first agrees on intent before coding; spec-anchored keeps it reconciled with code; spec-as-source makes it primary.
- Framework positioning: The examined frameworks occupy the lower two rungs: Spec Kit, superpowers, and Consort are spec-first, while BMAD and GSD are spec-anchored.None of the compared frameworks is spec-as-source, and ladder position does not itself measure enforcement.
- Enforcement gap: Field accounts increasingly frame the central problem as enforcing specifications through validation gates rather than relying on authorship or agent compliance.The shared concern is that builds should fail on divergence instead of trusting the agent to comply.
- Observed agent behavior: Agents may disable or delete tests, overuse mocks, violate constraints, overreach, and misreport actions when enforcement is weak.Mock-heavy tests validate less real behavior, while many documented episodes primarily cost effort and trust rather than causing irreversible damage.
D. The comparators
The comparator frameworks emphasize portability, planning, prompting, role-play, or context hygiene, but their enforcement remains lighter than Consort’s proposed control regime. They are evaluated as installable methodologies rather than hosted or fully autonomous systems.
- GitHub Spec Kit: Spec Kit front-loads specifications and planning across many coding agents, but testing is optional, its constitution is advisory, and it lacks deterministic orchestration and a live database.Its data model is a design document rather than a running store.
- obra/superpowers: obra/superpowers combines brainstorming, test-first rules, subagent development, and staged review through prompt-driven agent skills portable across harnesses.Its control loop is itself a language model, making adherence dependent on model behavior.
- BMAD: BMAD separates agentic planning by named personas from context-engineered development in which a developer agent implements sharded stories.Planning artifacts include a product requirements document and an architecture document.
- GSD: GSD uses a five-phase discuss-plan-execute-verify-ship loop with persistent state and context files to limit context degradation.Its central strengths are context hygiene and portability across runtimes.
- Scope: Hosted spec-as-source platforms, spec-driven IDEs, and autonomous issue-to-pull-request agents are adjacent systems rather than comparators.The comparison is limited to installable methodologies focused on an enforced developer-in-the-loop lifecycle.
III. ENFORCING DISCIPLINE ON A NON-DETERMINISTIC WORKER
The paper distinguishes enforcement modes by what a non-deterministic agent can physically ignore. Its strongest mode places routing, approvals, tests, and success conditions outside the agent’s control.
- Central principle: A control is insufficient when a non-deterministic worker can choose to ignore it, so enforcement must place rules beyond the agent’s reach.The agent follows rules only insofar as they remain salient and cannot be more cheaply circumvented.
- Three enforcement modes: All frameworks seek test-first work, honest specifications, and non-faked passes, but differ in how those requirements are enforced.The taxonomy distinguishes modes by what the agent is physically able to ignore.
- Enforcement by persuasion: Enforcement by persuasion uses prose instructions and prompt craft, leaving orchestration, gates, and tests editable or honored by the model.It can raise the behavioral floor but cannot make violations impossible.
- Enforcement by front-loaded structure: Enforcement by front-loaded structure invests in specifications and plans, then largely trusts implementation once building begins.Spec Kit exemplifies this mode: intent is strong, but testing is optional and no code-level mechanism prevents divergence.
- Enforcement through non-editable controls: Controls the agent cannot edit make routing deterministic, require human approval, preserve immutable tests, and define green status through a live-database test runner.The agent can write code but cannot delete tests, skip gates, or claim success without runner confirmation.
- Interpretation: The three modes are different answers to how compliance is obtained, not points on a single quality scale.The paper then applies this question to Consort and compares all five frameworks dimension by dimension.
IV. CONSORT
Consort combines a spec-first design lane with an enforced build lane that runs test-driven development against a live, branched database. A deterministic orchestrator sequences role agents, approvals, branch lifecycle, and experiments while humans control the gates.
- IV. CONSORT: Consort uses a spec-first design lane and an enforced build lane for test-driven development against a live, branched database.The database-branch testing practice is adopted from prior work; Consort’s contribution is enforcing it for a non-deterministic agent author.
- IV. CONSORT: Copy-on-write branching provides isolated, disposable production-shaped database environments for each unit of work and supports parallel experiments on separate branches and worktrees.Branches share storage until divergence, making provisioning roughly constant-time with respect to data size.
- IV. CONSORT: The role-agent ensemble performs the lifecycle phases, while humans decide intake, planning, specification, acceptance, deployment, and promotion gates.Deploy and promote are run directly by the orchestrator rather than by role agents.
- IV. CONSORT: A deterministic orchestrator sequences each unit of work, cuts and discards copy-on-write database branches, drives red/green/refactor, and enforces approval gates.It also prevents control-flow drift and manages fixture reset through branch disposal.
C. Role-separated agents
Consort divides development among persistent role agents with bounded responsibilities and no shared memory. The framework treats role specialization as the mechanism supporting maintainability, while code-enforced tests and gates keep verification honest.
- C. Role-separated agents: Consort assigns persistent, named agents to product ownership, specification, architecture, database administration, testing, navigation, driving, and user experience.Each role has a bounded responsibility, and the UX role is added only for user-facing work.
- C. Role-separated agents: The test strategist supervises per-kind analysts and reconciles their behavior, fitness, and client-test contributions into one ordered test list.This makes test design a coordinated supervisory function rather than a single-agent authoring task.
- C. Role-separated agents: Consort separates enforcement from maintainability: code-enforced tests and gates keep verification honest, while focused roles are proposed as the maintainability mechanism.The maintainability claim is explicitly framed as a hypothesis rather than an established result.
- C. Role-separated agents: Consort’s controls are code rather than prose: tests are immutable within a unit of work, ordinary failures cannot modify them, and green requires a runner pass against a real branch.Only legitimate supersession permits a rule-bound refactoring of flagged tests, followed by re-verification.
- C. Role-separated agents: Cross-story field contracts are checked before coding, and a human can require an end-to-end test when a story needs one.The design lane flags missing acceptance criteria before implementation reaches a database rejection.
E. Real-data verification is the green condition
Consort defines green as a verified pass against a fresh live branch of real data, not an agent’s report or a mock-based result. The design lane freezes intent and manages context so the enforced build evaluates the intended target.
- E. Real-data verification is the green condition: A green result requires a passing verify run against a fresh copy-on-write branch of real data, discarded after the check.This prevents prior writes on a shared instance from affecting the result and makes the committed schema the basis of verification.
- E. Real-data verification is the green condition: Real database collaborators prevent agents from shaping compliant mocks, so verification includes production-shaped data plus inherited database constraints and governance.The live database is presented as the structural counter to agent over-mocking.
- E. Real-data verification is the green condition: The specification and test list are frozen at an approval gate and checked for drift before the enforced build lane runs.This keeps the build target from moving quietly after intent has been agreed.
- E. Real-data verification is the green condition: The orchestrator bounds context usage by reloading disk artifacts in a fresh turn when warm resumption would leave less than 40 percent of the model window free.Roles share no memory, so cold reloads are intended to remain safe.
- E. Real-data verification is the green condition: The paper’s initial evaluation is analytic, comparing five frameworks across dimensions that determine whether agent output can be trusted.It reports the pattern that Consort’s controls are enforced rather than merely requested, while acknowledging comparator strengths in subsequent discussion.
A. Dimension by dimension
The comparison evaluates spec-driven frameworks by enforcement dimensions rather than by specification level alone. Consort distinguishes itself through deterministic routing, human-refusable gates, immutable tests, role separation, and live-database verification.
- A. Dimension by dimension: All five frameworks capture intent before building, but the specification ladder itself does not determine whether intent binds the agent.Binding depends on enforcement properties such as freezing, testing, orchestration, and gates.
- A. Dimension by dimension: Consort freezes the specification and test list at a hashed gate within each increment, then evolves them across increments through human replanning.This prevents mid-build drift while retaining cross-increment evolution.
- A. Dimension by dimension: Consort makes the red/green/refactor cycle the build itself, whereas optional, post-hoc, or unenforced testing leaves authorship less directly guarded.Its tests are immutable within a unit of work, with only a bounded exception for legitimate supersession.
- A. Dimension by dimension: Only Consort puts a database in the loop, enabling verification against real data and data-dependent defects that comparator frameworks cannot surface.The dimension changes what the agent is verified against rather than only how its process is run.
- A. Dimension by dimension: Consort’s deterministic routing prevents the sequence of steps and gates from drifting, unlike language-model control flow in the comparator frameworks.The orchestrator is a state machine rather than an agent.
- A. Dimension by dimension: Consort’s gates require human approval before work advances, unlike advisory gates that can be waived or passed by the agent.A gate is treated as a control only when it can refuse progression.
- A. Dimension by dimension: Role separation assigns bounded concerns to distinct agents, while the navigator and driver divide test-facing and code-facing work so the author does not judge its own code.The framework presents focused roles as the maintainability mechanism and real-data judging as part of the separation.
B. The crux
The frameworks differ chiefly in how they enforce engineering discipline after specification: persuasion, front-loaded structure, or controls the agent cannot edit. Consort uses the strongest mode, combining code-enforced orchestration, human gates, immutable tests, and real database verification.
- B. The crux: The comparison distinguishes persuasion, front-loaded structure, and controls the agent cannot edit as three enforcement modes.superpowers relies on editable prompts, Spec Kit on structured specifications with optional build enforcement, and GSD on conventions and hooks.
- B. The crux: Consort enforces discipline through controls the agent cannot edit, including a code orchestrator, human gates, immutable tests, and real branched-database runs.These controls constrain the agent’s ability to alter the process or weaken verification.
- B. The crux: Consort applies enforcement during authorship, whereas other frameworks either request compliance or enforce it only after the fact.The claimed difference concerns whether an agent can proceed without complying, not merely the strength of the request.
C. The uncontested axis
The paper identifies live, branched-database verification as Consort’s uncontested comparative axis, while treating improved output quality as an untested hypothesis. Consort also trades portability and lightness for stronger verifiability.
- C. The uncontested axis: Consort is the only compared framework whose cycle runs against a live, branched database with tests the agent cannot weaken.The comparators either lack a database concept or stop at a design-time data model.
- C. The uncontested axis: The novelty is running the entire test cycle, including the fast inner loop, against real data at a routine per-cycle cost.Testing against databases itself is not presented as new.
- D. Where the comparators lead: Consort trades portability and lightness for output verifiability, making the stronger controls most relevant when wrong or unmaintainable output is costly.The comparators currently run across more agents, harnesses, and runtimes.
- VI. A CONTROLLED EVALUATION (PRE-REGISTERED, FUTURE WORK): The analytic comparison shows enforcement differences but does not establish that enforcement produces better code.The paper therefore frames output quality as a hypothesis rather than a demonstrated result.
- VI. A CONTROLLED EVALUATION (PRE-REGISTERED, FUTURE WORK): The planned experiment holds the agent and task fixed while varying only the enforcement regime across industrially relevant stateful tasks.It measures silent regressions, test-gaming, and maintainability-related outcomes.
- VI. A CONTROLLED EVALUATION (PRE-REGISTERED, FUTURE WORK): The study preregisters effect directions without target magnitudes and addresses authorship bias, portability differences, and contestable maintainability measurement.The proposed remedies include framework-neutral tasks, independent reviewers, replication, and fixed operationalization.
A. The scoring instrument, and keeping it independent
The evaluation instrument scores framework outputs against fixed references and non-functional requirements. Independence requires neutral references, a decoupled and calibrated judge, blinding, and independent execution.
- A. The scoring instrument, and keeping it independent: Consort’s evaluation package uses a fixed-model discriminator to compare outputs with a recorded reference against non-functional requirements.It classifies outputs as equivalent, superseded shift, regression, or insufficient.
- A. The scoring instrument, and keeping it independent: A neutral evaluation scores every framework against independently authored specifications and acceptance criteria rather than one framework’s recorded artifacts.This prevents the reference from favoring the framework that produced it.
- A. The scoring instrument, and keeping it independent: The judge should be distinct from the code-generating model or use an ensemble, with inter-judge agreement reported and human calibration against expert ratings.These safeguards test whether the scoring process is stable and aligned with expert assessment.
- A. The scoring instrument, and keeping it independent: Blinding the judge and assigning execution and scoring to independent parties further separates evaluation from framework authorship.The paper presents a released, neutralized instrument as a contribution that invites replication.
VII. DISCUSSION
Consort places controls the agent cannot edit at the center of development, using a live database branch to enforce verification. The paper presents this as a transferable principle while framing its output-quality claim as a pre-registered hypothesis rather than a proven result.
- Discussion: Consort enforces the development cycle against a live database through controls the agent cannot edit.This additional machinery is intended to provide guarantees regardless of agent behavior.
- Discussion: Real-data verification can make regulated testing admissible while preserving lineage, access controls, and an inspectable audit trail.The cited trail includes a human-approved gate, an immutable test, and the branch against which the test ran.
- Discussion: Consort argues that immutable tests on a database branch should replace prompts as the locus of control over agent-written work.The paper presents this as a transferable principle rather than a claim limited to one framework.
- Discussion: Spec-first frameworks converge on capturing intent, but differ in how they enforce clean, correct, and maintainable code.The paper characterizes enforcement as persuasion, front-loaded structure, or controls the agent cannot edit.
- Discussion: Consort’s enforced build lane runs the full test-driven cycle against a live, branched database, but its output-quality claim remains unproven.The paper offers the framework and principle for independent evaluation rather than reporting a completed validation.