Source-linked AI summary
AgentSPEX: An Agent SPecification and EXecution Language
Pengcheng Wang, Jerry Huang, Jiarui Yao, Rui Pan, Peizhi Niu, Yaowenqi Liu, Ruida Wang, Renhao Lu, Yuwei Guo, Tong Zhang
TL;DR
Agent systems often leave workflow control implicit or tie structured workflows to Python, making authoring and maintenance difficult. AgentSPEX addresses this with a declarative, modular workflow language and harness, achieving the highest scores among compared approaches on all seven benchmarks and supporting more interpretable, accessible authoring.
Problem
Existing agent systems either leave control flow and intermediate state implicit or tightly couple structured workflows to Python, limiting maintainable workflow authoring.
Method
AgentSPEX uses declarative YAML workflows with typed steps, explicit control flow and context management, reusable submodules, and a customizable execution harness.
Results
AgentSPEX achieves the highest score among compared approaches on all 7 benchmarks, including 100% on AIME 2025 and 77.1% on SWE-Bench Verified.
Takeaways & Limitations
The paper concludes that explicit control flow, composition, and context management make agent workflows easier to author, inspect, and maintain.
Takeaways & Limitations
Support for long-context reasoning and longer-horizon tasks remains an area for future work.
Abstract
from arXiv · showhide
Language-model agent systems commonly rely on reactive prompting, in which a single instruction guides the model through an open-ended sequence of reasoning and tool-use steps, leaving control flow and intermediate state implicit and making agent behavior potentially difficult to control. Orchestration frameworks such as LangGraph, DSPy, and CrewAI impose greater structure through explicit workflow definitions, but tightly couple workflow logic with Python, making agents difficult to maintain and modify. In this paper, we introduce AgentSPEX, an Agent SPecification and EXecution Language for specifying LLM-agent workflows with explicit control flow and modular structure, along with a customizable agent harness. AgentSPEX supports typed steps, branching and loops, parallel execution, reusable submodules, and explicit state management, and these workflows execute within an agent harness that provides tool access, a sandboxed virtual environment, and support for checkpointing, verification, and logging. Furthermore, we provide a visual editor with synchronized graph and workflow views for authoring and inspection. We include ready-to-use agents for deep research and scientific research, and we evaluate AgentSPEX on 7 benchmarks. Finally, we show through a user study that AgentSPEX provides a more interpretable and accessible workflow-authoring paradigm than a popular existing agent framework.
1 Introduction
AgentSPEX addresses the limited control and Python coupling of existing agent frameworks with a declarative language for modular workflows and a customizable harness. It emphasizes explicit execution structure, context management, accessibility, reusable agents, and evaluation across seven benchmarks.
- Motivation: ReAct-style agents leave tool-use control flow and intermediate state implicit, relying on reactive execution conditioned on growing conversation history.The approach is simple to implement but depends heavily on the chosen foundation model’s capabilities.
- Motivation: Python-based frameworks provide greater execution control but make workflows difficult to modify, maintain, and share with non-programmers.The paper attributes these difficulties to tight coupling between workflow logic and Python programming, alongside steep learning curves.
- AgentSPEX: AgentSPEX uses human-readable YAML to specify agent workflows with explicit control flow, modular structure, typed steps, branching, loops, parallel execution, reusable submodules, and explicit context management.These features provide users precise control over agent behavior and context visibility without requiring execution-source modifications.
- AgentSPEX: AgentSPEX unifies skills and agents as composable workflows and gives users direct control over each step’s conversation history to improve performance, cost-efficiency, and controllability.Its executable-specification design represents workflows in declarative, human-readable YAML files.
- Evaluation and Applications: AgentSPEX includes ready-to-use agents for deep research, scientific research proposal generation, and research advising, with evaluation on 7 established benchmarks.The benchmarks span science, writing, scientific paper understanding, and software engineering.
2 AgentSPEX Design
AgentSPEX provides a lightweight, structured language for specifying agent workflows with explicit execution patterns, state and context management, reusable composition, and visual authoring. Its self-contained YAML workflows support versioning and modification by domain experts without Python orchestration code.
- Language constructs: AgentSPEX workflows use a lightweight vocabulary of primitives organized around a name, goal, optional configuration, and an ordered sequence of operations.Core operations include task, step, and workflow-control constructs such as if and for_each.
- Authoring: Self-contained YAML workflows are easy to version-control, diff, and share, while natural-language instructions let domain experts author and modify them without Python.This separates workflow authoring from orchestration code.
- State and context: Named context variables and save_as provide explicit state control, allowing outputs from one step to be templated into subsequent instructions.For example, search queries can be saved as search_queries and passed into a later step.
- Language constructs: Tasks start fresh conversations, whereas steps preserve multi-turn history, letting authors control whether information flows through variables or ongoing interaction.Tasks suit passing intermediate results via context variables; steps suit repeated tool use and instructions.
- Composition: Any workflow can invoke another through call as a submodule, pass parameters, receive return values, or be registered as a dynamically selectable skill or tool.This unified composition model supports iterative calls over collections such as lists of papers.
- Visual editor: The visual editor synchronizes an interactive flowchart with a text panel, enabling users to add, remove, rearrange, or directly modify workflow nodes before execution.Edits in either view are immediately reflected in the other.
3 Agent Harness
The AgentSPEX harness executes workflows through an interpreter and model–tool executor in a sandboxed environment, while providing observability, durability, and formal-verification mechanisms. These components support structured execution, debugging, interruption recovery, replay, and correctness checking.
- Execution Engine: The interpreter validates workflows, resolves configuration and templates, dispatches typed operations, manages nested control flow and scoping, and assigns hierarchical step identifiers.Hierarchical identifiers support checkpointing and logging across nested operations and loop iterations.
- Execution Engine: The executor runs multi-turn model–tool interaction loops through an MCP client until tool calls cease or configured tool-call or token limits are reached.Each iteration sends the current message history to the model and executes returned tool calls.
- Execution Environment: Each workflow runs in a Docker-based sandbox with isolated browser and file-system access to over 50 tools, including file operations, web search, code execution, and browser automation.The sandbox provides the external execution environment used by AgentSPEX workflows.
- Observability: The observability dashboard provides live logs of agent actions and intermediate reasoning steps for debugging and real-time monitoring at each workflow stage.The paper illustrates the dashboard during execution of a SWE-Bench Verified instance.
- Durability: The durability system checkpoints after each completed step, restores context and sandbox state to resume execution, and records traces that support selective replay of workflow runs.Checkpoints include completed step identifiers, template-variable values, prior outputs, step-level metrics, and sandbox state; traces capture model responses, tool results, and conversation state.
- Formal Verification: Explicit control flow, variable dependencies, and step boundaries enable formal verification of structural and semantic correctness using preconditions and postconditions.The verification predicate system can be defined in formal languages such as Lean or Isabelle.
4 Demos and Evaluation
AgentSPEX provides three ready-to-use research agents and achieves the highest score among compared approaches on all seven evaluated benchmarks. A 23-participant user study finds it more readable and approachable than LangGraph, while LangGraph is preferred for complex, multi-step workflows.
- Demos and evaluation: AgentSPEX includes ready-to-use agents for deep research, scientific research proposal generation, and research advising, and is evaluated on seven benchmarks across science, mathematics, writing, paper understanding, and software engineering.The deep-research workflow uses configurable breadth and depth parameters, while AI Scientist and AI Advisor implement proposal-generation and rubric-based review pipelines.
- Overall results: AgentSPEX achieves the highest score among compared approaches on all seven benchmarks, improving over the stronger science baseline by 2.8% on SciBench, 1.9% on StemEZ, and 5.5% on ChemBench.It also improves over CoT by 6.5% on ELAIPBench, reaches 100% on AIME 2025, and scores 77.1% on SWE-Bench Verified versus 76.2% for mini-SWE-agent and 74.6% for Live-SWE-agent.
- Discussion: ReAct underperforms CoT by 3.4% on ELAIPBench and 1.1% on ChemBench, whereas AgentSPEX enforces the workflow step by step instead of leaving its interpretation reactive.The reported ReAct scores on ELAIPBench are 33.8% versus 37.2% for CoT.
- Discussion: AgentSPEX’s larger gains on ChemBench (+5.5%) and ELAIPBench (+6.5%) may reflect explicit context management for extended problem statements and full-length research papers.Without explicit context controls, ReAct must carry the full conversation history forward across reasoning steps.
- User study: In a user study with 23 participants, AgentSPEX was generally favored for readability, prompting clarity, and ease of creating workflows from scratch, while most preferred LangGraph for complex, multi-step workflows.Participants described AgentSPEX as accessible to non-coders and easier to understand, whereas LangGraph was characterized as customizable and more rigorous.
5 Related Work
Related work spans open-source frameworks and closed-source skill platforms, differing in visual authoring, implementation style, context management, and runtime control. A central concern is preserving reproducibility and user control while managing degradation from growing contexts.
- Open-source agent frameworks differ in visual no-code editing, implementation style, and context management, while robust harnesses provide sandboxed environments and general computer-use tools.Table 3 summarizes these framework differences; prior work emphasizes sandboxed harnesses for agent tool access.
- Natural-language skills in Codex CLI and Claude Code enable reusable submodules but shift control flow, state management, and execution semantics to runtime model behavior.These platforms can be fast to implement, but the runtime shift can limit reproducibility and reduce user control.
- Longer contexts can cause context rot and broader performance degradation, making per-step context management a central design decision for agent frameworks.The passage identifies context rot and performance degradation as challenges motivating framework-level context management.
6 Conclusion and Future Work
AgentSPEX offers a structured, controllable framework that makes agent workflows easier to author, inspect, and maintain. Future work targets verification, automated workflow use, agentic training, multi-agent orchestration, and longer-horizon reasoning.
- Conclusion: AgentSPEX makes control flow, composition, and context management explicit, enabling LLM-agent workflows that are easier to author, inspect, and maintain.The framework aims to support more reliable, modular, and accessible agent systems.
- Future Work: Future work includes formal verification, training models to write and use workflows automatically, end-to-end agentic training pipelines, and expanded multi-agent orchestration.These directions are identified as promising extensions of the framework.
- Future Work: Future work also targets long-context reasoning and longer-horizon tasks through robust context compression and more expressive multi-agent abstractions.The paper identifies these capabilities as an exciting direction for continued development.
A Evaluation Details
AgentSPEX is evaluated on seven diverse benchmarks spanning five domains. Unless otherwise specified, evaluations use pass@1 accuracy on the full test set.
- Evaluation Details: 7 benchmarks span 5 domains, as summarized in Table 4.The exact workflows and agent harness configurations are available in the released code.
- Evaluation Details: Evaluations use pass@1 accuracy on the full test set unless otherwise specified.
A.1 SWE-Bench Verified
The section evaluates three agent systems on SWE-Bench Verified across Claude-Opus-4.5 and Claude-Opus-4.6 at high reasoning effort. Results are reported by model version, with averages presented separately and one baseline score averaged with a local reproduction.
- Setup: Three agent systems are evaluated on SWE-Bench Verified using Claude-Opus-4.5 and Claude-Opus-4.6 with high reasoning effort.Per-model results appear in Table 5, while averages are reported in Table 2.
- Setup: Table 5 breaks down SWE-Bench Verified results by model version, with all runs using high reasoning effort and temperature 1.0.The table provides model-version-specific results rather than only aggregate averages.
- Reproducibility note: 78.0% is the locally reproduced score averaged with the paper-reported result for Live-SWE-agent using Claude-Opus-4.5.This reproducibility adjustment applies specifically to Live-SWE-agent and that model version.
A.2 Science · A.3 Generative Writing
The evaluation covers science benchmarks measuring chemical and scientific problem-solving with exact match, alongside generative writing assessed by an LLM-based judging protocol. The science evaluation uses sampled or complete chemistry subsets, while WritingBench uses a sampled cross-domain subset.
- A.2 Science: ChemBench evaluates chemical knowledge and reasoning across 2,788 question–answer pairs spanning 9 domains.The study randomly samples 10 questions from each domain, producing a 90-question subset evaluated by exact match.
- A.2 Science: The ChemBench evaluation uses a 90-question subset sampled evenly across its 9 domains.Evaluation uses exact match.
- A.2 Science: SciBench provides 213 collegiate-level chemistry problems drawn from four textbook-based subsets.The subsets are Atkins’ Physical Chemistry (N=101), Chemistry by McMurry & Fay (N=33), Properties of Matter (N=47), and Quantum Chemistry (N=32), with exact-match evaluation against reference solutions.
- A.2 Science: SciBench’s chemistry evaluation spans physical chemistry, general chemistry, properties of matter, and quantum chemistry textbook problems.The benchmark uses open-ended, free-response questions and evaluates answers by exact match against reference solutions.
- A.2 Science: MMLU-Pro Stemez contributes 216 Physical Chemistry STEM problems evaluated by exact match.The problems are sourced from the Stemez website.
- A.3 Generative Writing: WritingBench covers generative-writing queries across 6 major domains and 100 subdomains.The evaluation randomly samples 20 questions from each domain, yielding a 120-question subset.
- A.3 Generative Writing: WritingBench evaluates 120 sampled questions using its official LLM-as-Judge protocol.Each response is assessed by 5 instance-specific criteria on a 10-point scale through the provided critic model.
B Observability · C Formal verification of AgentSPEX
AgentSPEX includes an observability dashboard with a log viewer, while its declarative YAML structure makes workflow plans and execution trajectories amenable to static and runtime verification. Explicit control flow, variable dependencies, and step boundaries support this verification capability.
- B Observability: Figure 4 shows an example log viewer for a SWE-Bench Verified instance.The figure illustrates the log viewer within the observability interface.
- C Formal verification of AgentSPEX: AgentSPEX’s declarative structure enables formal verification of agent workflows.The paper identifies this as a notable advantage over imperative frameworks.
- C Formal verification of AgentSPEX: Imperative frameworks make formal verification difficult because their control flow is embedded in general-purpose code.AgentSPEX is contrasted with this structure in the passage.
- C Formal verification of AgentSPEX: Explicit control flow, variable dependencies, and step boundaries are represented in the AgentSPEX YAML specification.These elements make workflow behavior more inspectable for verification.
- C Formal verification of AgentSPEX: Agent plans and execution trajectories become amenable to static and runtime verification.This follows from making workflow structure explicit in the YAML specification.
D User Study
The user study surveyed participants with prior programming experience but varied experience building agents, using questions listed in Table 6 with anonymized framework labels.
- Participants generally had prior programming experience, while their experience building agents varied.
- Table 6 lists the user-study survey questions and anonymizes the labels of the two agent frameworks.