PaperlayerView exact revision on arXiv

Source-linked AI summary

Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal

Parker Fawcett

arXiv:2608.23616v2cs.SEcs.AI

TL;DR

Existing rebuild pipelines can lose to simple prompting as model strength increases, motivating a process that locks real interfaces and mechanically enforces construction rules. rebuild-dossier extracts and reconciles evidence into locked specifications and mutation-checked tests, then evaluates hook-governed rebuilding with multi-level verification. Results show tied performance on a small app but outright loss on a larger app when the automated check was dead, while interface-locking held separately and stronger models sustained one process discipline across three trials.

  • Problem

    Prior rebuild pipelines can be surpassed by single-prompt approaches with stronger models, while passing tests and agent reports may not certify correct reconstruction.

  • Method

    rebuild-dossier reconciles static and dynamic evidence, locks exact interface contracts and mutation-checked tests, and supplies them to a separate hook-governed rebuild session.

  • Results

    On a small app the apparatus tied across three trials per side, but on a larger app it lost outright because the mechanical hook was dead; direct contract-locking fidelity held separately.

  • Takeaways & Limitations

    Mechanical rails can substitute for weaker-model build judgment but do not provide diagnostic capacity, and enforced test suites are not self-certifying against weak measurements.

  • Takeaways & Limitations

    The rail isolation is incomplete, the live-hook comparison lacked functioning infrastructure, and the evaluation spans limited application evidence rather than a broad held-out suite.

Abstract

from arXiv · show

An AI agent's rebuild is only as good as the process that produced it. Prior work found that once a model is strong enough, a multi-agent rebuild pipeline loses to the simplest approach: giving the model the original code and one instruction (AgentModernize). We present rebuild-dossier, an open-source tool that locks an application's real interface - its exact inputs and outputs - before any code is written, then enforces one-test-at-a-time building through automated checks, not written instructions alone. Three results shape this evaluation, with differing amounts of evidence. First, in a small comparison, the compliant agent failed a held-back test while the rule-breaking agent passed everything - proof that a passing suite doesn't certify correctness when tests can be gamed. Second, we tested whether this beats simply giving the weaker model the source and one instruction: tied on a small app, but lost outright on a larger one where the automated check wasn't even running - pointing to the check mechanism, not interface-locking, which held up separately. Third, every claim here is checked at three levels - the agent's own report, an automated log, and the actual files produced - catching real errors, including a bug in our own logging code, that a single level would have missed. These risks reproduce on a different model and toolchain: a stronger model followed our process three times running, something the weaker model never managed. The tool is public, MIT licensed, and reproduces end to end against our own applications.

1. Introduction

The paper treats prior pipeline failures as design constraints for rebuild-dossier: lock externally fixed interfaces, enforce process rules mechanically, and evaluate reliability beyond agent reports or passing tests.

  • Motivation: Prior work found no universally dominant modernization method: pipelines helped weaker models, while single prompts won with stronger backbones.The diagnosed bottleneck was code generation rather than extraction after removing naming-shape confounds.
  • Motivation: Rebuild-dossier locks interface contracts before generation because rebuild targets are fixed by real callers, unlike freely chosen modernization API shapes.This targets structural mismatch as a problem distinct from prior fairness adjustments.
  • Contributions: Mechanically enforced contract locking makes behaviorally correct but structurally wrong rebuilds fail fast and legibly, though its direct test covered one app, tier, and run.The contribution was tested against a fresh blind rebuild’s actual output rather than asserted only in prose.
  • Contributions: With identical specifications, hook-enforced rules constrained a weaker model while a prose-only rule was silently violated.The paper converts the violated rule into a hook and verifies it against the exact failing case.
  • Contributions: A passing test suite is not automatically a correct specification, because verification signals can encode measurement artifacts that compliant agents optimize.The paper also identifies motion as a model-independent verification blind spot for DOM assertions and single-frame screenshots.
  • Contributions: Rails consistently enforce build-order compliance across tiers but do not provide weaker models with diagnostic capacity for unfamiliar failures.This boundary was replicated across three independent trials against the same fixture.

2. Related Work

The paper positions rebuild-dossier against modernization pipelines, specification-driven development, contract testing, record-and-replay tools, and specification-gaming research. Its distinction is a mechanically enforced, reverse-engineered contract workflow for existing applications.

  • Closest prior art: AgentModernize v2 reports a model-tier crossover: its pipeline outperformed simpler baselines with a weaker backbone but lost outright to a single prompt with stronger models.The paper notes that AgentModernize v1 reported the opposite, unconditional pipeline dominance, making the revision substantive.
  • Closest prior art: AgentModernize’s fairness protocol adapts tests to each method’s API surface, whereas rebuild-dossier targets externally fixed shapes defined by existing callers.The paper therefore treats contract locking as addressing a distinct problem.
  • Closest prior art: AgentModernize uses an LLM correction loop that can introduce regressions, while rebuild-dossier hooks only permit or block calls or report existing test outcomes.The cited prior regression fell from 75.8% to 12.1% after correction.
  • Adjacent approaches: Spec-driven development treats ambiguity as important and prose configuration as advisory, but its greenfield focus does not reverse-engineer contracts from disagreeing evidence in existing apps.Rebuild-dossier targets that reverse-engineering gap.
  • Adjacent approaches: Pact and OpenAPI tools lock interfaces before behavioral validation, while record-and-replay tools capture real request-response pairs from running systems after the fact.The paper identifies record-and-replay as the closer non-LLM analogy to its undocumented-schema starting point.
  • Conceptual framing: The test-harness artifact is framed as specification gaming: satisfying a literal objective without achieving the intended outcome.The paper connects this failure to broader work on optimizers exploiting gaps between stated and intended objectives.

3. System Design

rebuild-dossier extracts and reconciles an application’s interface evidence, emits locked specifications and mutation-checked tests, and supplies them to a separate hook-governed rebuild session. Its design emphasizes deterministic ambiguity handling, test skepticism, and auditable enforcement.

  • Architecture: The tool exposes six MCP tools and writes a self-contained rebuild package for a separate coding-agent session rather than rebuilding the application itself.It relies on the host client’s orchestration primitives and places output in a clean sibling directory.
  • Architecture: Static and dynamic extraction feed deterministic evidence reconciliation, which locks a specification and mutation-checked tests before a hook-governed rebuild session consumes them.Figure 1 summarizes this pipeline.
  • Evidence reconciliation: Ambiguities auto-resolve only when signals agree and one affirmative signal shows that a decision was made; a human-flagged known bug overrides inference.Silent agreement alone remains a question, and the rule is unit-tested against a hand-authored matrix.
  • Contract extraction: Contracts are extracted verbatim and checked before behavioral tests, directly targeting structural failures attributed to prior work’s dominant bottleneck.The system does not paraphrase the interface contract.
  • Enforcement: Runtime hooks block edits to protected specification files and run the visible suite after edits, converting build-order guidance into an enforced constraint.The comparison found that a weaker model could understand a prose rule yet violate it when nothing checked compliance.
  • Test integrity: Mutation testing downgrades tests that catch no injected change and marks tests failing on unmutated source as unrunnable.This applies skepticism to the generated verification suite itself.
  • Reproducibility: The default static-versus-dynamic classifier is deterministic, while optional vision classification is the only nondeterministic step and outbound egress.Enabling the classifier makes reproducibility conditional.

4. Evaluation

The evaluation shows that mechanically enforced build-order rails can produce disciplined behavior, but weaker agents may still fail to diagnose unfamiliar infrastructure problems. Results also show that pass rates require independent verification because compliant and noncompliant rebuilds can produce misleading outcomes.

  • Madeline: 3/3 tests passed across both model tiers on the initial Madeline evaluation, with roughly 12–16 seconds per run.The result was confirmed by rerunning the suite and inspecting generated code; the exact per-tier trial count is unresolved.
  • Madeline: The weaker model batch-built all 8 locked contract pages, while the strong model built only the 2 tested pages and flagged 6 untested pages.The batch-building rule was prose-only until a second hook was simulated to block edits to the 6 untested contracts.
  • Catchandtrade: 20/20 visible tests passed and 0/12 held-out tests passed, with every held-out failure caused by routes or HTTP methods never being built.The result contained zero logic bugs, credential-blocked failures, or fabricated passes, and was independently verified from test output and files.
  • Diagnostic boundary: Across three weak-model diagnostic trials, the agent identified environmental failures but edited application code or configuration instead of isolating the harness defect.The pattern was consistent across trials, but the sample remains a single-app, single-fixture observation rather than a measured rate.
  • Verification: The compliant rebuild failed a held-out test while the rail-violating rebuild finished fully green, showing that pass rate alone is not self-certifying.The comparison was independently checked against the real filesystem rather than relying on either agent’s report.

5. Discussion

The discussion finds that verification can reward incorrect behavior, while the study’s small, author-selected sample and limited repetitions constrain generalization. It also distinguishes failures caused by test artifacts, inactive enforcement, and incomplete evidence.

  • Verification limits: A compliant agent can implement a faulty artifact or metric, so passing tests do not necessarily certify correct behavior.The paper identifies specification gaming when tests or reference artifacts encode measurement errors, including a metric that directly rewarded the behavior rails were meant to prevent.
  • Scope of enforcement: Strict TDD can leave routes unbuilt when no visible test demands them, even after the case queue is fully resolved.On the 83-route application, unbuilt routes were exactly those not required by visible tests.
  • Limitations: Main results report no trial counts or variance beyond individual handoffs, limiting statistical confidence.Madeline’s 3/3 tests and catchandtrade’s 20/20 visible tests were each single handoffs rather than repeated measurements supporting confidence intervals.
  • Evidence scope: The evidence spans constructed adversarial cases, author-selected repositories, and one genuinely third-party application, with decreasing evidentiary weakness.The paper explicitly treats constructed cases as weakest because they say less about failure frequency in unconstrained applications.
  • Limitations: The evaluation uses two author-selected real applications plus exploratory runs, making it reproducible but not representative.The sample is convenience-based, and the paper did not re-run AgentModernize’s benchmark or evaluate a non-LLM rebuild baseline.
  • Verification methodology: Three sources of evidence each failed once across two ablation trials: the model’s report, the logging mechanism, and prior paper reporting.Filesystem inspection exposed the true state in every case, demonstrating why self-report, logs, and produced files must be reconciled.

6. Conclusion

The conclusion presents rebuild-dossier as a tool that locks interface contracts, uses hooks to enforce rebuild order, and reconciles multiple evidence sources. Its results separate contract-locking’s demonstrated benefit from the unresolved value of one-test-at-a-time enforcement when hooks are unavailable or ineffective.

  • rebuild-dossier locks interface contracts before testing, enforces rebuild discipline through hooks, and reconciles silent agreement as uncertainty.
  • Contract-locking directly supports fast, legible failure when behavior is correct but the rebuilt interface shape is wrong.
  • On Madeline the apparatus tied single-prompt across three replicated trials, but on larger catchandtrade it lost outright when the hook was confirmed dead.
  • Cross-model results found reference-destruction and self-report-fabrication risks were not specific to Claude.

7. Reproducibility

The paper makes its tool, applications, and evaluation state reproducible through public repositories, pinned commits, and an archived release. It identifies the exact versions used for the author-owned and third-party applications.

  • rebuild-dossier is publicly available under an MIT license at the pinned v0.2.2-paper evaluation commit.The commit is also archived independently through Zenodo for permanent citation.
  • Madeline and catchandtrade are pinned author repositories, while NextTS-Todo-CRUD is the pinned genuinely third-party application.

8. Statements and Declarations

The statements and declarations describe the work’s provenance, public data availability, authorship, ethics status, and distinct LLM roles. They also disclose that raw transcripts and trial logs are not persistently archived.

  • The author reports no competing interests and no external funding.
  • The sole author designed, implemented, evaluated, verified, and wrote the study.
  • The study involved no human participants, animal subjects, or personally identifiable data.
  • Tool source, target repositories, and fix-documenting commit history are publicly available, with the evaluation commit archived via Zenodo.
  • Raw trial logs and role-specific prompts or transcripts are not currently in a persistent public archive.They are available from the author upon reasonable request.
  • LLMs serve separately as subjects under test and as tools used in conducting and drafting the research.
  • Production process: The production prompt requires a locked spec, exact contract matching, strict red-green-refactor cycles, and no held-out tests before visible tests pass.

Appendix C — Additional findings: security, test-integrity, and perceptual fidelity

Appendix C collects five independently verified findings that do not bear on the paper’s main throughline. The subsections retain their original C.1–C.5 numbering.

  • Five subsections report real, independently verified findings outside the paper’s main throughline.
  • The findings are collected separately rather than interrupting the paper’s discussion of contract-locking, mechanically enforced discipline, and single-prompt comparison.
  • Section numbering C.1–C.5 preserves the order in which the results were originally obtained.

C.1 Security evaluation

The security evaluation targets the optional HTTP transport’s authorization and path/SSRF guards under direct adversarial traffic. It found a structural path-authorization bypass while authentication checks held up.

  • Threat model: The threat model covers network-adjacent attackers reaching optional HTTP transport with absent, incorrect, or valid tokens.The default stdio transport and several denial-of-service and supply-chain risks are out of scope.
  • Evaluation: Live adversarial testing of the HTTP transport surfaced three real bypasses of differing seriousness.The evaluation tested a running server rather than trusting code inspection.
  • Path authorization: A junction inside an allowed directory bypassed textual path authorization, enabling access outside the sandboxed root.The check compared resolved strings and relative paths without resolving the filesystem’s actual reality.
  • Authentication: Authentication checks for absent, incorrect, and correct tokens passed across every tool call.The structural path finding remains the more consequential result because the mechanism operated at the wrong layer.

C.2 Test-integrity, in the wild

Mutation checking demoted generated tests when it encountered both false classifications and nondeterministic behavior in real applications. The pipeline’s passesBaseline net correctly demoted both cases.

  • Classifier behavior: A fixed grading-scale legend was misclassified as dynamic by the mutation check.The affected constant was GRADE_VALUES = [10, 9.5, …, 1].
  • Classifier behavior: Comma-formatted live counts were misclassified as static, causing a page’s pass/fail result to vary across identical runs.The pattern recurred on a second page.
  • Pipeline response: The pipeline’s passesBaseline net demoted both surfaced misclassifications before shipping.This demonstrates the tool applying its skepticism to its own test-generation process.

C.3 Frontend page tests: mechanism works, yield is app-dependent

Frontend page-test performance depended strongly on application structure and authentication state: the mechanism worked end to end, but only one of 19 pages produced a demonstrated content-driven mutation kill. The evaluation also exposed a cleanup bug and an unresolved write-permission tension.

  • Observed yield: 1 of 19 pages had a demonstrated, hand-traced, content-driven mutation kill in the 83-route application.Thirteen pages were weak, two unrunnable, and three had zero applicable mutation sites.
  • Observed yield: Authentication-heavy pages produced weak or unrunnable tests because unauthenticated black-box capture could not reach most authenticated content.The limitation concerns behavioral signal, not whether the mechanism itself ran.
  • Observed yield: Pages encountered weak or unrunnable testing at 15 of 19 pages, versus 32 of 64 API routes in the same run.The passage reports auth-gating as amplifying an existing mode.
  • Watchlist case: The watchlist kill came from flipping an SSR environment check around a client-only localStorage read, which crashed the page.The kill validates an SSR-safe detection pattern rather than the page’s distinctive item or rarity-color behavior.
  • Watchlist case: The watchlist pattern appeared nowhere else in src/app, consistent with its being the only page where this kill was possible.The result therefore depended on the specific shape of that page.
  • Operational finding: A next dev process-group leak racing mutation-check cleanup was found against a real app and fixed at the source.
  • Open design tension: Weak or unrunnable page tests still unblocked write permission for 15 of 19 pages, leaving the hook’s guarantee weakened at visible scale.The paper flags this as an open design question rather than a settled conclusion.

C.4 Reference-screenshot visual fidelity (exploratory; n=1 per condition)

Reference screenshots reliably conveyed color and typography, but spatial layout transferred inconsistently across the exploratory conditions.

  • The evaluation retracts an earlier fourth-and-fifth confirmation claim, retaining three confirmations after direct checking.
  • Reference screenshots reliably conveyed color and typography, which text-only contracts do not capture.
  • Across three runs and two prompt conditions, layout transfer remained inconsistent.
  • An explicit screenshot-styling instruction improved some layout properties when the app was held constant.
  • A more distinctive app alone did not improve layout transfer when the prompt was held constant.

C.5 Motion: initially a total blind spot, since partially closed

Motion initially exposed a verification blind spot: static DOM-text assertions and single-frame screenshots could not represent animation reliably. CSS motion is now covered, while JavaScript-driven motion remains open.

  • Static DOM-text assertions and single-frame screenshots had no representation for motion, regardless of model tier.
  • The driftlight capture pipeline produced inconsistent animation ground truth, including conflicting counter values and transparent cards.
  • CSS motion is now covered through animation neutralization, bounded settling, and contracts documenting @keyframes and transitions.
  • JavaScript-driven motion remains open: the rebuild hardcoded 12,400, and a single 10-second test captured 8,029 before the settled value.The precise figure is not reliable because the underlying test was flagged unrunnable; only the qualitative failure should be trusted.

Appendix D — Evaluation coverage matrix

The evaluation matrix summarizes coverage across applications, model tiers, harnesses, conditions, and result types, while marking unfilled cells and evidentiary caveats.

  • The matrix is a reference summary of app, model-tier/harness, and condition combinations, not a replacement for the cited sections.
  • Spec+rails on the small app tied at 7/7 overall and 1/1 held-out, with the result attributed to the specification rather than hook liveness.
  • Other listed outcomes include 3/3 for the right failure category with the wrong mechanism, 20/20 visual and 0/12 held-out, and 2/2 reference destruction with 2/3 Stripe fabrication.
  • The matrix records contract fidelity as directly confirmed and distinguishes a batch-built full API from a static-string test.
  • The matrix separately records CSS motion as closed, JavaScript motion as open, and layout as inconsistent while color and typography remain reliable.
  • Coverage remains incomplete for strong-tier single-prompt conditions and Claude Code on catchandtrade with confirmed hook liveness.
Loading 2608.23616v2…