Source-linked AI summary
FlashScript: Compiling Recurring Agent Workflows into Reusable Neuro-Symbolic Programs
Sadra Sabouri, Hosein Hasanbeig, Adam Sage, Shreya Sinha, Cale Darling, Ashish Tiwari, Katie Garms, Souti Chattopadhyay, Sumit Gulwani
TL;DR
Recurring agents reconstruct their reasoning and actions on each run, while preserving a reviewed process for reuse remains an open need. FlashScript compiles a completed run into a neuro-symbolic program, and the evaluated programs were consistent and resource-efficient across recurring and composite tasks.
Problem
Users must repeatedly inspect reviewed agent procedures because runs reconstruct their process, and existing approaches leave consistency and generalization unresolved.
Source: §I · pp. 1–2Method
FlashScript compiles a completed run into an NSP, keeping actions in code and placing typed model calls at fixed points where judgment is needed.
Source: §I · p. 2Results
Across 120 WorkBench tasks, NSPs were correct and made identical tool calls on every run, using about 70 times fewer input tokens and a third of the runtime of either agent; on composites, the skill-guided agent’s accuracy fell to 23% while NSPs remained correct and identical.
Source: §I · p. 2, §IV · p. 4Takeaways & Limitations
NSPs turn recurring agent work into software that end users can parameterize, combine, test, maintain, and reuse.
Source: §V · p. 4
Abstract
from arXiv · showhide
Many agent tasks recur with new data or small changes, yet agents regenerate their reasoning and actions on each run, producing different results. Distilling a prior run into a skill helps the agent avoid earlier mistakes, but it still re-runs the full process each time, so runs can vary and incur that cost. We compile a successful agent run into a reusable neuro-symbolic program (NSP) with a set of guidelines we call FlashScript. Actions and data transformations become code, instance-specific values become parameters, and decisions that need model judgment become LLM calls. We identify three layers where these calls belong and three techniques that replace most of them with code. The compilation does not remove nondeterminism; it localizes it, so the deterministic program and its neural calls can be reviewed and tested separately. Across 120 WorkBench analytics tasks with Claude Haiku 4.5, the NSPs were correct and made identical tool calls on every run, using about 70 times fewer input tokens and a third of the runtime of a skill-guided agent whose runs were inconsistent. In a composite version of the tasks, the skill-guided agent's accuracy fell to 23%, and its output inconsistency increased, while the NSPs remained correct and identical across all runs. The NSPs successfully generalized to all 108 unseen instances of the same templates. Our findings suggest that NSPs from agent trajectories can be considered cumulative software assets for end-users to test, maintain, and reuse in their recurring workflows.
I. INTRODUCTION
Recurring agent workflows require users to repeatedly inspect procedures because agents reconstruct them on each run. FlashScript compiles a successful run into a reusable program that localizes model judgment, and the evaluated programs were consistent, efficient, and generalizable.
I. INTRODUCTION
- Agents reconstruct procedures on each run, so repeated executions can diverge and reviewed procedures must be inspected again.Successful runs contain procedural knowledge, but that knowledge remains buried in the session rather than carrying over to future executions.Source: §I · p. 1
- Prior trajectory methods either leave execution decisions to the model or replay fixed actions, leaving consistency and generalization unresolved.The paper identifies a missing process that users can rerun consistently while adapting to recurring workflow variations.Source: §I · pp. 1–2
- FlashScript compiles actions into code, instance-specific values into parameters, and model-dependent decisions into typed calls placed at fixed points.It identifies input, computation, and output call layers, alongside techniques to replace most calls with code.Source: §I · p. 2
- Across 120 WorkBench analytics tasks, NSPs were correct and made identical tool calls on every run, using about 70 times fewer input tokens and a third of the runtime of either agent.On composite tasks, NSPs remained correct and identical across runs while the skill-guided agent’s accuracy fell to 23%; NSPs also solved all 108 unseen template instances.Source: §I · p. 2
- NSPs localize rather than eliminate nondeterminism, making their deterministic code and neural calls separately reviewable and testable.The paper presents them as software assets end users can develop and maintain, including programs with LLM calls,.Source: §I · p. 2
II. MOTIVATING EXAMPLE
The rental-workbook example frames a recurring workflow that should preserve a reviewed process while adapting to new data or another city. In a preliminary comparison, the NSP produced identical sheets more quickly and with fewer tokens than a skill-guided agent.
II. MOTIVATING EXAMPLE
- The analyst wants recurring runs to preserve the reviewed process, generalize to new data or another city, and cost less after corrections are incorporated.The example is a weekly update to a workbook of rental listings for clients.Source: §II · p. 2
- In the sandbox, the NSP produced identical sheets in 6.7 seconds using 6.7k tokens, while the skill-guided agent varied across runs and averaged 38 seconds and 42k tokens per run.The agent varied in how it named added columns and marked removed listings; the NSP flagged the same listings.Source: §II · p. 2
III. FLASHSCRIPT
FlashScript turns a completed agent trajectory into a reusable script, fixing actions in code and reserving model calls for decisions that need judgment. It bounds those calls and uses caching, rules, and call grouping to manage variation and cost.
III. FLASHSCRIPT
- A completed trajectory becomes a script in which actions are code and reasoning becomes typed model calls made when future runs require judgment.The call operation, llm<T>, returns a value of a declared type such as a string, Boolean, or option.Source: §III · p. 2
- NSP control flow fixes when a model is queried, what it receives, and how its expected-type response is used, confining run-to-run variation to those calls.The program selects inputs and checks response types –[26].Source: §III · p. 2
- Input calls turn unstructured text into typed values, computation calls make difficult semantic decisions, and output calls render computed results in natural language.Examples include resolving “L.A.” to a sheet, judging whether a listing is a scam, and phrasing results for the task owner.Source: §III · p. 2
- NSPs balance fixed scripts’ consistency but limited input coverage against model-heavy programs’ broader variation handling but lower consistency and efficiency.Their generality depends on how many decisions they delegate to model calls.Source: §III · p. 2
- Caching reuses guarded values, codifying replaces repeatable judgments with rules, and clubbing combines independent questions into one call or runs them in parallel.Clubbing can use smaller classification models when generative capacity is unnecessary.Source: §III · pp. 2–3
IV. PRELIMINARY RESULTS
On WorkBench analytics tasks, NSPs maintained correctness and repeatable tool behavior across repeated and composite runs while using fewer tokens and less runtime. Results also show that template-based planning supports generalization, while reference trajectories can improve efficiency and reviewability.
IV. PRELIMINARY RESULTS
- The NSPs were Python scripts evaluated against vanilla and skill-guided agents on 120 analytics tasks, with each NSP and skill run three times per task.The tasks came from 12 templates; the composite evaluation used independent subtasks and instances not used to create the skill or NSP.Source: §IV · p. 3
- Across ten-subtask composites, NSPs were correct in every run, versus about 23% accuracy for the skill-guided agent; NSPs also solved all 108 unseen instances.The composite subtasks were independent, a setup the authors say likely understates accuracy drops on realistic dependent tasks.Source: Figure 2 · p. 3, §IV · pp. 3–4
- With ten subtasks, the skill-guided agent reproduced its first-run outcome in only 10% of cases, while the NSP produced the same result every run.Fig. 2 covers tasks with 1, 3, 5, and 10 independent subtasks, with ten tasks per length and three runs each.Source: Figure 2 · p. 3, §IV · p. 3
- On repeated tasks, NSP accuracy was 1.0 on every run; the vanilla agent and skill-guided agent were correct in all three runs on 79% and 96% of tasks, respectively.The eight templates requiring a data decision accounted for all 11 skilled-agent errors.Source: §IV · p. 3
- NSPs used about 70 times fewer input tokens and about a third of either agent’s runtime; caching, codifying, and clubbing reduced model calls.Caching served 37 of 49 guarded input calls, codifying replaced every computation-layer decision with a rule, and clubbing cut model questions on ten-subtask tasks from 5.8 to 2.0.Source: §IV · p. 4
- Planning instructions brought NSP accuracy to 1.00; templates supported correctness, while adding the reference run reduced calls from 1.8 to 1.5 and exposed decisions for review.Using the run alone made NSPs too specific to generalize to unseen instances.Source: §IV · p. 4
V. DISCUSSION
The discussion frames an NSP as reusable software that users can parameterize and maintain, with deterministic code separated from neural calls. It also notes that current benchmarks rarely test the nuanced judgments recurring work can require.
V. DISCUSSION
- Users can retain reusable NSP software that is parameterized, combined with other NSPs, and maintained over time.For a scam check, users can choose a heuristic or describe their criteria, then change that preference and see its effect within seconds.Source: §V · p. 4
- Table II compares NSPs written from a reference run, a request template, or both; composite values average across four task lengths.The caption identifies the conditions and averaging scope, but does not provide the table’s cell values.Source: Table II · p. 4
- The authors argue NSPs can follow a software lifecycle, with separate tests for the symbolic shell, neural calls, and the boundary between them.As a workflow settles, rule codification can replace neural calls with deterministic code.Source: §V · p. 4
- Agent benchmarks rarely capture nuanced recurring-work decisions such as inferring constraints, interpreting vague terms, weighing preferences, or framing open-ended problems.The authors identify this as a gap in current benchmark coverage.Source: §V · p. 4
VI. FUTURE PLANS
Future work will build a benchmark of recurring workflows with successive instances and real-user judgments as ground truth. It will also examine user-facing customization and ways to expose how much of a request an NSP can handle.
VI. FUTURE PLANS
- The planned benchmark will test inference of unstated constraints, interpretation of vague terms, personal preferences, and investigation without a clear success measure.Workflows will include successive instances, with real users’ judgments as ground truth.Source: §VI · p. 4
- The authors plan to study interfaces for inspecting and customizing NSPs without requiring users to read code.Planned user studies will examine which parameters users want to set and what they expect NSPs to generalize across.Source: §VI · pp. 4–5
- A proposed coverage mechanism will run requirements handled by the NSP as code and route the rest to a fallback agentic step that can update it.The authors envision coverage growing over time across scenarios and data structures.Source: §VI · p. 5