PaperlayerView exact revision on arXiv

Source-linked AI summary

DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis

Abhinav Rajeev Kumar, Harshit Arora, Varun Singh, Manikandan Nanjappan

arXiv:2609.11504v1cs.LGcs.SE

TL;DR

Natural-language DeFi workflows can be structurally valid yet authorize costly trades because quote-derived slippage bounds do not control self-inflicted price impact. The paper introduces DeFiFlowBench and Koan-Safe, combining multi-level evaluation with prompt-only parsing, replaceable generation, and structural safety repair. Koan-Safe improves the static safety proxy and records no unsafe executions in the saved benchmark outputs, while the authors emphasize limited execution semantics and remaining numeric-policy risks.

  • Problem

    DeFi workflow synthesis lacks evidence about whether generated workflows include protections that control trade risk, because structural validity and slippage bounds do not guarantee safe execution.

  • Method

    DeFiFlowBench scores graph coverage, configuration completeness, and declared safety predicates, then checks supported trade configurations locally; Koan-Safe adds parsing, replaceable generation, structural repair, and default safety parameters.

  • Results

    Koan-Safe’s hybrid variant scores 0.67 on the static safety proxy versus 0.33 for the best baseline, while matched enforcement comparisons produce zero unsafe executions with enforcement enabled and 14–17 when disabled.

  • Takeaways & Limitations

    The results support explicit trade protections and execution-based evaluation while distinguishing declared safety from a general safety guarantee.

  • Takeaways & Limitations

    The execution program does not traverse generated DAGs or production executors, and static checks do not establish numeric-policy validity or agreement with user intent.

Abstract

from arXiv · show

A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-language DeFi workflow synthesis. It measures graph coverage, configuration completeness, and declared safety predicates, then tests supported trade configurations on a local EVM. Direct, constrained, and few-shot prompting produce 14-19 unsafe held-out executions per configuration under a fixed 5% price-impact cap. A slippage bound derived from a quote does not prevent the price impact of the order itself. We propose Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters. On 75 held-out workflow prompts, its hybrid variant scores 0.67 on the static safety proxy, compared with 0.33 for the best baseline. Koan-Safe records no unsafe executions on the saved benchmark outputs. A matched-candidate ablation produces 14-17 unsafe executions when enforcement is disabled. Additional tests expose the limits of default injection: permissive existing thresholds can still authorize unsafe trades. A separately evaluated policy cap addresses this failure on a 36-case diagnostic grid. These results support explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.

I. INTRODUCTION

DeFi workflow synthesis must address both structural correctness and trade-risk protections: a workflow can contain expected calls yet still authorize costly self-inflicted price impact. DeFiFlowBench evaluates these protections with static safety proxies and limited execution checks, motivating Koan-Safe’s repair-based design.

  • Motivation: A quote-derived slippage minimum does not prevent a large order from causing its own price impact, so expected calls can still authorize a costly trade.The paper separates protection against quote-to-execution movement from protection against the order’s self-inflicted impact.
  • Benchmark and scope: DeFiFlowBench measures graph coverage, configuration completeness, and declared safety predicates, then separately checks supported trade configurations on a local EVM.The execution check is limited and does not execute the complete generated DAG.
  • Observed failures: Direct, constrained, and few-shot LLM configurations each produce 14–19 unsafe high-impact executions in the local checker.These results motivate separating intent parsing, candidate generation, and safety enforcement.
  • Proposed approach: Koan-Safe’s hybrid variant reaches 0.67 on the static proxy, compared with 0.33 for the best baseline.The method was frozen before the held-out split, which includes structures outside its presets.
  • Scope: The benchmark evaluates workflow-repair methods under limited execution semantics, so neither a high static score nor zero observed unsafe trades establishes a general safety guarantee.A separate stress grid tests permissive thresholds and a post-hoc policy-cap extension.

II. PROBLEM STATEMENT AND THE SAFETY LADDER

The task maps natural-language DeFi intent to a typed workflow and evaluates safety through nested static levels plus a separate execution check. The ladder distinguishes structural validity, concrete configuration, and declared protections, while emphasizing that slippage and price impact are different risks.

  • Task: A workflow specification W = (N, E, C) contains typed nodes, type-level data-flow edges, and concrete execution configuration.Benchmark annotations additionally record required safety predicates and clarification cases.
  • The safety ladder: Level 1 requires required nodes and edges, but allows extra structure and does not check acyclicity or dependency semantics.This is deliberately a lenient graph-validity score.
  • The safety ladder: Level 2 adds concrete values for every required configuration key, but does not verify user-intent alignment or numeric bounds.A template without a required trade amount fails this level.
  • The safety ladder: Level 3 requires every gold safety predicate to be declared and parameterized, including protections such as slippage bounds and price-impact gates.Predicates are derived uniformly from normalized workflows rather than self-reported by systems.
  • Execution check: The local execution check interprets configurations on a constant-product AMM and marks mined swaps unsafe above a fixed 5% impact cap.It does not traverse the generated DAG or invoke production executors.
  • Trade protections: A slippage bound protects against quote-to-execution price movement, whereas a separate sufficiently restrictive price-impact gate can block a large self-impacting trade.The distinction drives the paper’s headline safety result.

III. BENCHMARK

DeFiFlowBench combines team-authored development and held-out prompts with metamorphic safety tests, common scoring, and multiple baselines. The design targets generalization, clarification behavior, and whether safety survives risk-relevant request changes.

  • Categories and prompts: The benchmark contains 120 development prompts across swaps, limit orders, cross-chain transfers, and compositional workflows, with 15 underspecified or contradictory cases scored for clarification.The categories contain 40, 30, 30, and 20 prompts respectively.
  • Held-out test split: The separate 87-prompt held-out split tests generalization with fresh surface forms, unseen token symbols, and structures outside systems’ category presets.Examples include gasless swaps, quote-first limit orders, dashboard-augmented bridges, and genuine cross-chain swaps.
  • Metamorphic safety suite: A 34-pair metamorphic suite tests amount monotonicity, threshold tightening, waiver resistance, paraphrase invariance, and drop-field non-fabrication.Each relation compares outputs for transformed prompts without requiring a per-prompt gold label.
  • Annotations: Each prompt is paired with gold annotations for required nodes, edges, configuration keys, safety predicates, and allowed extra nodes.The annotation scheme is intended to support consistent benchmark extension.
  • Systems and scoring: The evaluation uses a shared evaluator across oracle, floor, template, Koan, LLM, and Koan-Safe systems, with retained normalized outputs for metric re-derivation.The oracle’s 1.00 static scores test scorer consistency rather than semantic trade correctness.

IV. KOAN-SAFE: STRUCTURAL REPAIR AND SAFETY DEFAULTS

Koan-Safe separates intent parsing, candidate generation, and generator-agnostic safety enforcement. The enforcement layer repairs required structure and injects conservative defaults, while held-out structures outside fixed presets test whether this design generalizes.

  • Architecture: Koan-Safe factors synthesis into an intent parser, candidate generator, and replaceable safety-enforcement layer that can be toggled and measured independently.This design treats omitted safety machinery as an enforcement problem rather than relying solely on generation.
  • Intent parser: The prompt-only parser extracts task category and trade-intent fields, then decides whether to build a workflow or request clarification.Clarification is triggered when acting would require guessing key trade intent.
  • Candidate generator: Rules, LLM, and hybrid generators feed the same enforcement layer, with the hybrid backfilling parser-extracted configuration that the model omitted.The generators differ in how they produce candidate structure and configuration.
  • Safety enforcement: The enforcement layer repairs required safety structure and injects defaults for slippage, price impact, bridge confirmations, order expiry, and bridge recipients.It does not add trade-intent fields itself.
  • Comparison integrity: Koan-Safe has no access to gold annotations, while the held-out split deliberately requires workflow structures outside its fixed presets.This setup tests generalization rather than assuming it from engineered domain knowledge.

V. EXPERIMENTAL SETUP

The evaluation combines static workflow scores with corrected local execution checks, using a frozen held-out setup and calibrated baselines.

  • The evaluation asks whether graph coverage predicts trade protection, whether Koan-Safe generalizes, and whether enforcement changes outcomes for fixed candidates.
  • Splits and models.: The held-out split contains 87 prompts, including 75 workflow prompts authored after Koan-Safe was frozen.Saved outputs are reused without resampling, and static rates use 95% Wilson intervals reflecting prompt variation.
  • Corrected execution protocol.: Evaluator v2 applies a fixed 5% impact cap independently of declared gates and rejects invalid or non-finite parameters.Ambiguous bare thresholds are also replayed under the fractional interpretation.
  • Calibration.: The oracle provides required structure and generic configuration values for a 1.00 static score, while null and random-node baselines score 0.00.These baselines calibrate scorer consistency and the floor rather than prompt fulfillment.

A. RQ1: Structure and execution measure different failures

Static structure and execution expose different failure modes: Koan-Safe improves declared safety and eliminates observed unsafe executions under the corrected replay, while comparisons remain scoped to the proxy and local semantics.

  • A. RQ1: Structure and execution measure different failures: Direct, constrained, and few-shot LLM configurations each produce 14–19 unsafe held-out executions despite their generated workflows.The slippage minimum is derived from an impact-adjusted quote and does not prevent the order’s own price impact.
  • B. RQ2: Koan-Safe improves the static safety proxy: Koan-Safe hybrid reaches 0.67 on the static proxy, while every frozen configuration records zero unsafe executions in corrected replay.The hybrid completes 26 local executions, aborts 19 trades on price impact, and leaves seven limit conditions unfilled.
  • B. RQ2: Koan-Safe improves the static safety proxy: Koan-Safe’s graph coverage generalizes unevenly: the rules variant falls to 0.52 held-out coverage, while LLM and hybrid reach 0.71 and 0.73.The system requests clarification on nine of twelve relevant prompts.
  • B. RQ2: Koan-Safe improves the static safety proxy: The hybrid does not reliably outperform the Koan-Safe LLM variant: it adds only two successes and loses none, with exploratory p = 0.5.Against safety-instructed Gemini, the hybrid succeeds alone on 26 prompts versus one baseline-only success.
  • A. RQ1: Structure and execution measure different failures: Matched-candidate enforcement raises Gemini’s LLM proxy from 2/75 to 48/75 and hybrid proxy from 1/75 to 50/75; disabling enforcement yields 14–17 unsafe executions.Both conditions use the same saved candidate, isolating the enforcement layer under the stated execution semantics.
  • B. RQ2: Koan-Safe improves the static safety proxy: Zero unsafe executions does not imply that every requested safety-policy change is respected, because Gemini variants still violate metamorphic threshold or paraphrase relations.The retained failures include two threshold relations for Gemini LLM and one paraphrase relation for Gemini hybrid.

E. Policy stress test and post-hoc repair

Stress tests show that default injection can preserve permissive thresholds, whereas a separately designed policy cap removes unsafe and non-executable outcomes on the diagnostic grid.

  • E. Policy stress test and post-hoc repair: The main held-out table reports static safety levels with 95% Wilson intervals and local-EVM outcomes, including unsafe counts and safe-rate over definite outcomes.Unsafe counts use greater-than-5% own-trade price impact regardless of gate declaration.
  • E. Policy stress test and post-hoc repair: The matched-candidate ablation compares enforcement on and off using the same saved candidate and parser, with static scores over workflow prompts and unsafe counts over supported executions.
  • E. Policy stress test and post-hoc repair: A 36-case grid yields two unsafe executions, ten safe executions, twelve protective aborts, and twelve non-executable outputs under the frozen layer.The grid spans four trade amounts and nine threshold inputs, including missing, permissive, fractional-percent, and invalid values.
  • E. Policy stress test and post-hoc repair: The policy extension produces 16 safe executions and 20 protective aborts, with no unsafe or non-executable outcomes on the same grid.It caps impact thresholds at 3%, preserves stricter valid bounds, replaces invalid thresholds, and caps slippage at 1%.
  • E. Policy stress test and post-hoc repair: The extension changes no amount, token, price, or chain field, but it was designed after the audit and is excluded from the main ranking.Its replay on saved model candidates is therefore an engineering check rather than fresh held-out evidence.
  • F. Cost and latency: The frozen enforcement function has a 2.3-microsecond median across candidates in a local benchmark.This excludes parsing, generation, and execution, and does not support billed API-cost inference.

VII. ANALYSIS

Across primary held-out runs, the execution checker found 104 unsafe outcomes among 715 definite system–prompt pairs, while static passing cases were non-unsafe under evaluator v2. These measurements remain bounded by checker semantics, threshold-unit ambiguity, and a synthetic AMM.

  • Execution outcomes: 104 of 715 definite system–prompt pairs execute unsafely, while all 266 static-proxy passes are non-unsafe under evaluator v2.The authors characterize this as empirical agreement within the checker, not a soundness proof; 345 proxy-failing pairs are also non-unsafe.
  • Execution outcomes: A declared 50% gate can pass a presence-based static check yet exceed evaluator v2’s fixed 5% impact cap.The corrected checker applies the cap independently of gate declaration.
  • Threshold sensitivity: 17 of 53 bare-threshold outputs change outcome under fractional rather than percentage-point interpretation, with seven unsafe executions under the alternative.Both analyses are retained because historical prompts did not specify numeric units.
  • Mainnet arithmetic fidelity: Across 11 token pairs and seven trade sizes, local executed outputs match the router’s integer formula with zero relative deviation.The validation uses mainnet reserves at a pinned block to seed the local AMM.
  • Mainnet arithmetic fidelity: Impact rises to approximately 9.3% at one-tenth of reserves and 33.5% at half, but this arithmetic does not validate routing, token behavior, or transaction ordering.The test uses a constant-product AMM with the same fee as the reference model.

D. Structural and configuration failures

The benchmark separates structural coverage, configuration completeness, and declared safety because systems exhibit distinct failure profiles. Koan-Safe improves declared protections and local outcomes, but novel structures, intent fidelity, and deployment-level enforcement remain outside the demonstrated scope.

  • Evaluation scope: The static proxy requires declared predicates and complete configuration, but does not certify execution or semantic fidelity.The execution check is a separate outcome-based test, and related work similarly distinguishes static proxies from end-state evaluation.
  • Structural and configuration failures: Distinct omissions in price-impact predicates, transaction monitoring, and bridge confirmation show why a single graph score cannot characterize execution readiness.Direct Gemini omits these protections on 43, 32, and 15 prompts, respectively.
  • Structural and configuration failures: Koan-Safe hybrid’s graph coverage falls from 1.00 on easy prompts to 0.77 on medium and 0.25 on hard prompts.The method reduces missing safety predicates but retains structural errors on novel workflows.
  • Reference oracle: The oracle records 43 impact aborts and 17 slippage reverts among 60 definite outcomes, but its generic parameters do not represent every intended trade.It therefore checks the scorer and execution path rather than determining which user requests should trade.
  • Implications: Structural repair and safety defaults improve coverage and local trade outcomes, yet permissive thresholds can remain unsafe and require numeric-policy validation.The paper presents explicit unit contracts and execution-path checks as additional requirements.

IX. LIMITATIONS

The evaluation is limited to configuration-level tests on a small, team-authored corpus with fixed local execution semantics. Unit conventions, model variability, annotation independence, and post-hoc policy development constrain how broadly the results should be interpreted.

  • Execution scope: The local test does not traverse the generated DAG, verify guard dominance, or support cross-chain execution, so results do not establish full-workflow safety.Compositional cases execute only their swap leg, and malformed graph structures are not fully rejected.
  • Units and evaluator dependence: Historical outputs lack specified units, and evaluator-dependent interpretations change some outcomes and metamorphic conclusions.Future generation runs should enforce an explicit unit-bearing schema.
  • Scale and annotation: The 120 development and 87 held-out prompts are team-authored, without an independent distribution shift or completed second-annotation pass.No inter-annotator agreement result is reported.
  • Model variability: Single temperature-zero generations limit claims about model variability, and the hybrid’s small numerical advantage over the LLM variant is inconclusive.The matched ablation controls saved-candidate variation but does not measure fresh-call variability.
  • Method and post-hoc extension: The frozen repair layer can retain permissive values, while the policy-cap extension was developed after inspecting evaluation results and lacks independent held-out generalization.Neither version proves semantic fidelity to every request.
  • Protocol fidelity: All transactions use mock tokens and a local constant-product AMM, excluding real routing, adversarial ordering, liquidity changes, token quirks, and bridge behavior.No live funds are used.

X. ETHICS AND RESPONSIBLE DISCLOSURE

The study is conducted offline without transactions touching live networks or real funds, and its outputs are released for independent reproduction and auditing.

  • The evaluation uses no live-network transactions or real funds and is intended to identify unsafe workflows before execution.The authors explicitly caution against executing generated DeFi workflows with real assets.
  • Released model outputs and derived metrics support independent reproduction and auditing of the reported failures.
Loading 2609.11504v1…