PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis

Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya Shah

arXiv:2610.37830v1cs.AIcs.HCcs.IRcs.MAcs.SE

TL;DR

The paper asks how to assess whether AI-generated smart contracts preserve intended behavior, when conventional evaluation tools and scoring choices can obscure that assessment. It evaluates execution-based fidelity and proposes an elicitation-first pipeline; its results indicate that gathering information before generation matters far more than post-generation review or repair.

  • Problem

    Evaluating synthesized contracts is difficult because English descriptions provide the requirements, common proxies measure other properties, and conventions can make systems appear to understand descriptions.

    Source: §1 · p. 1

  • Method

    The study replays transactions on generated and original contracts, tests evaluation instruments on planted errors, and evaluates a pipeline that elicits client answers before generation and checks disagreements against an executable reference model.

    Source: §3 · p. 2, §5 · pp. 7–8

  • Results

    Information gathered before generation mattered far more than post-generation stages, and a question-asking agent that also chose when to stop reached 70.6%, 5.4 points below asking all ten questions.

    Source: §4.3 · p. 4, §5 · pp. 7–8

  • Takeaways & Limitations

    In the tested configurations, verification is best used to surface disagreements for client review, while automated security repair should ask questions rather than rewrite code.

    Source: §5 · pp. 7–8

    Elicitation gains are optimistic simulator-based estimates: no human authors answered questions, and the descriptions and answers derive from the original contracts used for scoring.

    Source: §6 · p. 8

Abstract

from arXiv · show

AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked. This gap matters most for smart contracts, the programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. We ask how to check whether an AI-written contract behaves as it should, and where the requested behavior gets lost. We design an evaluation built on execution rather than opinion. We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes, compare against an answer key written without the description, and audit our own scorer until known wrong answers cannot pass. Common methods, such as AI judges and standard analysis tools, miss obvious mistakes, and scoring choices matter too. Checking only the functions every system includes hid about half the difference between pipelines. We then propose an elicitation-first architecture. Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key, and returns disagreements to the client instead of silently rewriting code. Asking first matters far more than reviewing or patching afterwards, across three AI models, while automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits, and their value can be estimated without the original contract. To our knowledge, this is the first study in smart contract synthesis to gather missing requirements before generation and check the result against an independently written executable model. The lessons extend to any setting where AI agents turn requests into programs. We release our tools, 600 tasks and all results.

1 Introduction

The paper studies how to evaluate whether synthesized smart contracts preserve requested behavior, distinguishing execution from intent and specification compliance. It tests evaluation instruments and pipeline stages to locate where information is lost.

1 Introduction

  • Smart contracts make behavioral errors consequential: they hold and transfer value autonomously, cannot be patched after deployment, and can fail expensively [5].Because each action can be replayed exactly in a sandbox, contracts also provide a concrete setting for studying underdetermined natural-language requests.Source: §1 · p. 1
  • Evaluating synthesized code is difficult because English descriptions are the only statement of intent, common proxies measure other properties, and models may succeed through conventions alone.A benchmark can therefore reward convention-following while appearing to measure understanding.Source: §1 · p. 1
  • The study measures implementation fidelity to an existing contract, which reflects author intent only when the original contract embodies what its author wanted.Its central methodological lesson is to test instruments on planted errors, control for description-free model knowledge, and replay exactly what is claimed to be checked.Source: §1 · p. 1
  • LLM judges, a static analyzer, and an LLM tester fail on planted mistakes; description-free controls expose a conventions shortcut, while shared-interface scoring and adapted replay hide omissions.The paper introduces a fidelity metric using exact replay and explicit coverage, validated with known-answer tests.Source: §1 · p. 1
  • Elicited answers raise fidelity across three model families, while post-generation stages add at most about one point and security rewriting removes required behavior.A factorial comparison separates the effect of gathered information from pipeline architecture.Source: §1 · p. 1
  • The study examines how few questions suffice, how agents choose and stop, how question value can be estimated without the original contract, and how synthetic description errors bias those estimates.The released materials include a harness, 600 contract tasks, client transcripts, reference models, generated contracts, and scores.Source: §1 · pp. 1–2

2 Related Work

Prior work evaluates generated code against trusted references, including smart-contract completion, transaction replay, and differential fuzzing. This paper's related-work contribution is a controlled study of how information, verification, and repair affect behavioral fidelity in stateful programs.

Evaluating code generation

  • Test-based code-generation benchmarks measure functional correctness against trusted references [2] [6], and stronger test suites can change model rankings [16].Smart-contract benchmarks cover function completion [22] [8], historical-transaction replay [27], and differential fuzzing against originals [9].Source: §2 · p. 2

LLMs as judges and auditors

  • LLM judges are widely used and known to be biased [31], while smart-contract auditors show high recall but low precision [4] [25].Prior repair benchmarks report that security patches often break legitimate behavior [14] [30].Source: §2 · p. 2

Oracles, clarification and user simulation

  • Prior work includes LLM-written reference oracles [29], consistency checks [10] [24], specification-guided testing [23], and clarification systems evaluated with human users [21] [11].This paper contributes a controlled study of how information, verification, and repair affect behavioral fidelity, and what such an evaluation can conclude.Source: §2 · p. 2

3 Study Design

The study uses contracts with short descriptions and fuller requirements, simulated client answers, executable reference models, and transaction replay to evaluate oracle validity and implementation fidelity. It also audits the compared pipelines and defines its uncertainty estimates and decision criteria.

3 Study Design

  • The corpus is filtered from FSM-SCG; 100 contracts support development and 500 further contracts support scale replication, with short descriptions and complete requirements generated for each.The short descriptions average about 140 words and the full requirements about 1,700 words.Source: §3 · p. 2
  • Simulated clients answer questions using GPT-5.4 and the original contract, so elicitation gains estimate recovery of hidden implementation behavior rather than measured human-author responses.The authors characterize these gains as optimistic estimates under the simulator.Source: §3 · p. 2
  • A separately prompted writer creates a Python reference model from the description, public interface, and deployment parameters; the model flags disagreements for client adjudication but never scores generated contracts.Generated contracts are always compared with the original Solidity contract.Source: §3 · p. 2
  • Oracle validity is tested by planting one-rule mistakes and checking whether the model sides more often with the untouched twin than the mutant over exposing transactions.Always-succeed, always-revert, and description-free controls test class-specific shortcuts and reliance on the description.Source: §3 · p. 2
  • Implementation fidelity replays each transaction on the original and generated contracts from their own states, counting a step correct only when outcomes and observed successful-state changes match.The required interface includes every state-changing function; contract-level transaction fidelity is averaged across contracts, with full-match and coverage measures also reported.Source: §3 · p. 2
  • Uncertainty intervals are 95% percentile bootstrap intervals over contracts with 2,000 resamples, while criteria define detectable and meaningful effects.Intervals are conditional on one generation per condition, and 36 criteria are tested without multiplicity correction.Source: §3 · pp. 2–3
  • On 500 held-out contracts, Figure 1 contrasts a pipeline that generates without client input, rewrites from security findings, and self-grades with one that elicits answers, checks a reference model, and returns disagreements to the client.Its badges report transaction-level fidelity to the original contract.Source: Figure 1 · p. 3
  • Table 1 audits instruments on planted mistakes across 200 guard removals, 40 mutants from 20 contracts, and reference models on 66 held-out contracts.It defines a caught mistake by a mutant-specific flag and reports false alarms on untouched contracts or steps.Source: Table 1 · p. 3

4 Results

The results show that evaluation tools and scoring choices can obscure whether generated contracts preserve requested behavior. Elicitation improves fidelity, but the elicitation-first pipeline outperforms alternatives at scale, while later security rewriting can remove requested behavior.

4.1 RQ1. Do common instruments measure intent?

  • The LLM judge flags 80% of untouched contracts, while requiring executable proof leaves 30% true catches; a function-name-only reference model catches 87.5% of conventional mistakes.The rubric judge correlated with three human raters at r=0.15 on ten contracts, with interval [−0.53, 0.71].Source: §4.1 · p. 3

4.2 RQ2. Where is the information lost?

  • Reference models catch 65.9% of exposed contract-specific mistakes from short descriptions versus 87.5% from complete requirements, a 21.6-point gap.The checklist recovers the full gap for reference models, but on generated contracts GPT-5.4-mini reaches 55.6% fidelity from short descriptions, 79.4% with interviewer answers, and 76.0% with checklist answers.Source: §4.2 · pp. 3–4, Figure 2 · p. 4

4.3 RQ3. How many questions does the client need to answer?

  • Three best-first questions reach 68.2% fidelity and recover about 62% of the ten-question gain, while the first three checklist questions recover 37%.Question value can be estimated without the original contract; a separation score ranks questions similarly to measured gain (Spearman ρ=0.92).Source: §4.3 · pp. 4–5, Figure 3 · p. 5

4.4 RQ4. What do the pipeline’s stages do?

  • The security audit-and-refine loop reduces fidelity from 95.3% to 82.5%, and 80% of sampled changed outcomes break a stated requirement.Of the 341 classified changes, 14% block a genuine vulnerability not requested by the requirements and 6% are undecided by them; labels are not human-verified.Source: §4.4 · p. 5, Figure 4 · p. 6
  • With identical information, the original architecture ends 21.0 points below one-shot generation and 21.5 below elicitation-first; its interaction with added information is +13.0 points.Verification and client triage show no detectable effect without answers (+0.1 points) and a small effect with answers (+0.5 points).Source: §4.4 · p. 5, Figure 5 · p. 6

4.5 RQ5. Does it hold at scale, and does the metric matter?

  • On 500 held-out contracts, elicitation-first exceeds the original pipeline by 33.5 points and one-shot generation by 18.9; the original trails one-shot by 14.6.The comparison uses the same short description and code generator; never-built outputs score zero.Source: §4.5 · pp. 5–6, Table 2 · p. 6
  • Shared-interface scoring reports a 19-point gap on 364 contracts, versus 35 points with a fixed required interface on 499 and 34 under exact replay.The audited scorer lowers fidelity by 12 to 14 points for every system, while moving no system gap by more than 1.2 points; all rules preserve rankings, but absolute fidelity depends on the rule.Source: §4.5 · pp. 6–7, Figure 7 · p. 7

4.6 Robustness of the main conclusions

  • Table 3 identifies checks that could have contradicted each conclusion by varying instruments or populations; conclusions that failed those criteria are reported as such.The checks vary sequences, generator ablations, scoring rules, runner fixes, contracts, types, and model families.Source: §4.6 · p. 7

5 Discussion

In tested configurations, information gathered before generation mattered far more than later pipeline stages; verification is best used as an alarm whose disagreements return to the client. The authors recommend stronger evaluation controls and hypothesize that some lessons may transfer to web agents.

5 Discussion

  • 18 evaluation criteria failed; targeted questioning did not beat the generic checklist by its margin, and adaptive stopping missed its margin.These failures are reported alongside successes as part of the evaluation lessons.Source: §5 · p. 7
  • The authors recommend planted-error validation, description-free controls, exact replay with coverage, auditing simulated inputs, fixed criteria, and prominent reporting of failures.Simulated inputs that could have been repaired should be reported separately, with gains treated as optimistic.Source: §5 · p. 7
  • Information gathered before generation mattered far more than later stages, and security repair should prompt client questions rather than rewrite behavior.Verification is best used as an alarm that enriches real defects, with disagreements returned to the client.Source: §5 · pp. 7–8
  • The authors hypothesize that web agents share underdetermined requests, proxy judges, and answer-leaking simulated users, and that replayable sandboxes and scored clarification could transfer.They explicitly note that this transfer has not been tested.Source: §5 · p. 8

6 Limitations

The study measures implementation fidelity under simulated, contract-derived requirements rather than author intent. Its findings are also bounded by the corpus, language, baselines, coverage, and evaluation choices.

6 Limitations

  • Descriptions and simulated client answers both derive from the original contract, so elicitation gains estimate recovery of hidden implementation behavior, not what human authors would provide.No human answered the questions; real authors may be unsure, inconsistent, or volunteer more, making elicitation gains optimistic estimates under this simulator.Source: §6 · p. 8
  • The study lacks independently authored specifications, expert-written tasks, and a controlled omission-only condition.The original contract is treated as intended behavior even when it lacks access control; security labels are LLM-generated and unverified by people.Source: §6 · p. 8
  • Coverage omits events and ether received by external accounts, while random sequences do not expose every rule.These choices limit which behavior replay can observe.Source: §6 · p. 8
  • The evaluation uses one corpus and one language, compares against the earlier pipeline and one-shot generation rather than clarification systems, and does not correct criteria for multiplicity.Generator families share a compile-repair step; the original pipeline's verification benefit is provisional, and intervals omit generation-to-generation variability.Source: §6 · p. 8

7 Future Work

Future work centers on measuring requirements from human authors, extending replay-based evaluation to autonomous agents that transact, and testing generality across settings.

7 Future Work

  • A study with people commissioning contracts should measure intent before implementation and the effort required to answer clarification questions.The authors identify this as the most important next step.Source: §7 · p. 8
  • Replayable sandboxes could evaluate wallet-using agents end to end, including whether transactions match user requests and when agents should ask before moving funds.The proposed scope also includes whether contract-writing and contract-using agents interpret a contract alike, and agents negotiating or transacting with one another.Source: §7 · p. 8
  • Further tests should use a second corpus, compare published clarification systems, repeat generations, and include other languages.The paper identifies these as ways to test how far its findings generalize.Source: §7 · p. 8
  • The authors propose phrasing each security finding as a client question to retain justified fixes without regressions.Source: §7 · p. 8

8 Conclusion

The paper argues that evaluation instruments, controls, and scoring choices can change conclusions about agentic code synthesis. Its rebuilt pipeline improves fidelity over two baselines, but sampled-test success remains low.

8 Conclusion

  • 72.9% implementation fidelity on held-out contracts compares with 53.9% for one-shot generation and 39.3% for the original pipeline, yet only 20% pass all sampled tests.The authors identify studies with human authors as a next step for addressing this gap.Source: §8 · p. 8
  • Elicited answers raised fidelity by 19 to 23 points across three generator families, while review, repair, and triage added at most about one point and security rewriting removed required behavior.Source: §8 · p. 8
  • Three to five questions recovered 60 to 75% of what a simulated client could supply, and question-value rankings were possible without the original contract.Which questions mattered depended on errors in synthetic descriptions.Source: §8 · p. 8
  • LLM judges, static analysis, and LLM testers failed on planted mistakes; description-free controls exposed convention rewards, while exact replay and known-answer tests exposed omissions.Shared-interface scoring and adapted replay had hidden omissions.Source: §8 · p. 8

A System Architecture and Codebase

The study uses a deterministic pipeline with separate execution environments for generated contracts and reference models, then evaluates fidelity and oracle validity. Its scoring rules check transaction outcomes and observable value changes, with coverage accounting for what the generated contract cannot expose.

B Pipeline Algorithm

  • The pipeline asks the client a checklist, generates and compile-repairs code, writes a reference model, compares them, and can return disagreements for client choice and repair.It repeats comparison and repair for up to two rounds, then reports static-analysis results rather than using them to decide behavior.Source: B Pipeline Algorithm · pp. 9–10

A System Architecture and Codebase

  • The architecture runs every system’s output in the same in-process EVM while reference models run in a separate sandbox, feeding fidelity and oracle-validity measurements.Model requests and outputs are cached by contract and condition so stages can resume and reported numbers can be recomputed without model calls.Source: A System Architecture and Codebase · p. 9, Figure 8 · p. 10

C Measurement Details

  • The complete-description rate is 65.2% (182 of 279) over all mutants, compared with 87.5% over exposed mutants.Outcome and value-change disagreements count, except in the conventional experiment, which uses outcomes only.Source: C Measurement Details · p. 10
  • Transaction-level fidelity averages each contract’s share of correct original-contract steps, requiring matching outcomes and observable value changes and weighting contracts equally.Unobservable original value changes make a step unverified and earn no credit; the strict contract pass requires every step correct and matching readable post-deployment values, except ether balance.Source: C Measurement Details · p. 10

D Additional Results

The additional results compare information added before generation with interventions after writing, examine question agents and client triage, and document predefined criteria and prompt design.

D Additional Results

  • With the GPT-5.4 writer, checklist answers and the complete document raise catch rates in every contract-specific mistake class, while agents on the complete document leave classes unchanged or lower.The information also lowers disagreement with untouched contracts from 21.9% to 4.2% of transactions.Source: D Additional Results · pp. 10–11
  • All question agents ask about roles most often, and the agent permitted to write its own questions is the only one that matches asking all ten.It wrote 260 follow-ups across 93 contracts.Source: D Additional Results · p. 11
  • In the 500-contract run, clients confirmed the reference model in 53% of 2,207 disagreements, the contract in 32%, and neither in 14%.Clients judged an average of 8.5 situations when disagreements reached them; the pipeline made 3.8 model calls per contract and had a 107-second median wall time.Source: D Additional Results · pp. 11–12
  • Interventions that add information before writing help in every model family and on both measures, while post-writing stages add at most about one point or can cause harm.The largest positive effect of a post-writing stage is client triage at under one point.Source: D Additional Results · p. 10, Figure 9 · p. 12

E Predefined Criteria

  • The predefined criteria distinguish effects detectable by intervals excluding zero from effects also meeting a criterion, and no multiplicity correction is applied.Eighteen of the 36 criteria were not met; ten criteria were committed together with their results.Source: E Predefined Criteria · p. 12

F Prompts

  • The short-description prompt names operations without code, while the complete document gives exact rules and boundaries; the checklist asks about limits, payments, fees, timing, roles, states, balances, and failure conditions.The simulated client answers questions in at most two plain-English sentences and chooses between randomly ordered alternatives during triage.Source: F Prompts · p. 12

G Case Study and Transcript

The case study shows how a role-related omission in a short description is corrected by asking the client, and how the resulting contract scores against the original.

G Case Study and Transcript

  • The short description suggests an administrator controls sale status and gold-list additions, but the original has no privileged role.The roles answer clarifies that any account may add addresses to the gold list or change sale status, while minting depends on list membership and a live sale.Source: G Case Study and Transcript · p. 12
  • The contract generated after all ten questions agrees with the original on outcomes and exposed values, yet 22% of steps change an unexposed value.Exact scoring credits it with 77.6%, versus 41.8% for one-shot generation and 22.4% for the original pipeline.Source: G Case Study and Transcript · p. 12
  • The case is a description error, and exact scoring penalizes changes to values the generated contract does not expose.The reported comparison is 77.6% after ten questions, 41.8% for one-shot generation, and 22.4% for the original pipeline.Source: G Case Study and Transcript · p. 12

H Resource Documentation

The resource documentation describes the released task records, model use, computational costs, and reproducibility provisions.

H Resource Documentation

  • The release covers 600 contracts: 100 development and 500 replication tasks, with originals, descriptions, generated or repaired contracts, and per-contract scores.Development records additionally include complete requirements, client answers, interviewer transcripts, and reference models.Source: H Resource Documentation · p. 12
  • Descriptions were written by Gemini 2.5 Flash, and GPT-5.4-mini generated contracts, with replication also using Claude Sonnet 5 and Gemini 3.5 Flash.GPT-5.4-mini performed compile repair across model families; the client, auditor, security classifier, and question agents used GPT-5.4.Source: H Resource Documentation · p. 12
  • On the 500-contract run, model use totaled about 4.2 million GPT-5.4, 4.5 million GPT-5.4-mini, and 0.8 million Gemini 2.5 Flash tokens.Every number can be recomputed from cached outputs, and the repository maps tables and figures to scripts.Source: H Resource Documentation · p. 12
Loading 2610.37830v1…