PaperlayerView exact revision on arXiv

Source-linked AI summary

SPIN: Structural LLM Planning via Iterative Navigation for Industrial Tasks

Yusuke Ozaki, Dhaval Patel

arXiv:2605.14051v1cs.AI

TL;DR

Industrial LLM planners can produce structurally invalid or unnecessarily long workflows, making downstream execution brittle and costly. SPIN validates and repairs plans into executable DAGs, then evaluates DAG prefixes with a simulator and critic to stop when sufficient. Across AssetOpsBench and MCP Bench, it improves task quality, executability, and planning-related scores while reducing downstream execution burden, with additional internal prompting and benchmark-specific limitations.

  • Problem

    Industrial LLM planning requires executable, auditable workflows, but planner outputs can be structurally invalid or unnecessarily long, creating brittle interfaces and avoidable execution cost.

  • Method

    SPIN combines validator-based DAG repair with simulator-critic evaluation of incrementally longer DAG prefixes and early stopping.

  • Results

    SPIN improves execution quality and planning-related scores across AssetOpsBench and MCP Bench while reducing AssetOpsBench executed tasks from 1061 to 623 and tool calls per run from 11.81 to 6.82.

  • Takeaways & Limitations

    Treating planner outputs as executable interfaces and controlling execution by sufficient DAG prefixes can reduce downstream burden while maintaining or improving task quality.

  • Takeaways & Limitations

    Evaluation is limited to AssetOpsBench and its fixed tool/agent environment, while simulator accuracy depends on trajectory coverage and critic stopping has no formal correctness guarantees.

Abstract

from arXiv · show

Industrial LLM agent systems often separate planning from execution, yet LLM planners frequently produce structurally invalid or unnecessarily long workflows, leading to brittle failures and avoidable tool and API cost. We propose \texttt{SPIN}, a planning wrapper that combines validated Directed Acyclic Graph (DAG) planning with prefix based execution control. \texttt{SPIN} enforces a strict DAG contract through \texttt{\_validate\_plan\_text} and repair prompting, producing executable plans before downstream execution, and then evaluates DAG prefixes incrementally to stop when the current prefix is sufficient to answer the query. On AssetOpsBench, across 261 scenarios, \texttt{SPIN} reduces executed tasks from 1061 to 623 and improves \emph{Accomplished} from 0.638 to 0.706, while reducing tool calls from 11.81 to 6.82 per run. On MCP Bench, the same wrapper improves planning, grounding, and dependency related scores for both GPT OSS1 and Llama 4 Maverick.

1 Introduction

SPIN addresses brittle, costly industrial workflows by making planning an executable interface and controlling execution through validated plan prefixes. It combines DAG validation and repair with simulator-critic stopping to improve executability and reduce unnecessary continuation.

  • Motivation: Industrial workflows require auditable, robust multi-step decisions, but structural plan errors can cause hard execution failures and obscure underlying task difficulty.Long workflows can also incur unnecessary external-action cost when partial progress already suffices.
  • Motivation: Prior agent research improves action generation and self-correction, while planner–executor interfaces often remain implicit and external checks are needed for reliable verification and repair.This leaves domain-grounded evaluation vulnerable to formatting and interface brittleness.
  • Contribution: SPIN makes plan structure an explicit executable interface for tool-using LLM agents and evaluates it on AssetOpsBench and MCP Bench.The wrapper targets downstream consumption rather than treating plans solely as internal reasoning artifacts.
  • Contribution: SPIN validates and repairs planner output into a dependency-aware DAG, then uses simulator- and critic-based prefix control to stop when the query is answerable.The execution policy evaluates prefixes incrementally instead of unconditionally running the full workflow.
  • Contribution: SPIN combines a validator enforcing dependency-aware DAG format with a simulator-critic prefix loop that decides whether further execution is necessary.The contribution is framed as a contract-based planning wrapper for tool-using agents.

2 Related Work

The related work spans realistic tool-interface evaluation, structured planning validation, and simulator-guided search. SPIN targets industrial workflows where planner outputs become executable DAGs and execution burden is a first-class concern.

  • Industrial and tool-interface settings: AssetOpsBench evaluates industrial operations and maintenance workflows requiring specialized steps, heterogeneous tools, and large operational data sources.Planner outputs can directly affect downstream execution cost, operational burden, and recovery behavior.
  • Agent planning and validation: Prior work improves LLM agents through reasoning, reflection, tool use, structured generation, and validation or consistency mechanisms in multi-agent planning.These directions include interleaved reasoning and acting, self-correction, reinforcement learning, and grammar-constrained generation.
  • Industrial and tool-interface settings: MCP-Bench evaluates agents across heterogeneous real-world MCP tools using measures including tool selection, planning, and grounding.SPIN instead targets industrial asset-operations workflows with structural validity and execution cost as explicit criteria.
  • Simulator-guided planning: Search-oriented methods use search, tests, simulators, or reorganized reasoning-action workflows to improve planning efficiency or correctness.SPIRAL is especially related through its planner-simulator-critic decomposition.
  • Simulator-guided planning: SPIN differs by optimizing execution-centric planning: structural validity and prefix control are emphasized when downstream execution dominates cost.Evaluation therefore includes executed tasks, tool and API calls, elapsed time, and downstream failure patterns.

3 Method

SPIN uses two linked components: a validator-repair loop that produces a machine-readable DAG, and a simulator-critic loop that evaluates increasingly long prefixes and stops when sufficient.

  • System overview: SPIN first validates and repairs planner output into a machine-readable DAG, then evaluates prefixes rather than executing the entire workflow by default.The method is inspired by SPIRAL’s simulator-critic decomposition but operates over validated DAG prefixes without Monte Carlo Tree Search.
  • Plan representation: The parsed plan consists of ordered nodes containing task descriptions, assigned agents, dependencies on earlier nodes, and expected output specifications.The raw plan is serialized through aligned task, agent, dependency, and expected-output fields.
  • Validation and repair: The validation-repair loop retries planner output using detected errors and returns a valid plan or failure after the retry budget is exhausted.Algorithm 1 explicitly conditions repair prompting on the validator’s error set.
  • Validation and repair: The validator checks aligned field lengths, consecutive indices, legal references to previous nodes, and membership of assigned agents in an admissible set.These constraints define structural executability before downstream execution, rather than task-level success.
  • Prefix evaluation and stopping: For each increasingly longer prefix, the simulator predicts the current outcome and the critic judges whether it is sufficient to answer the query.The critic returns status, a Boolean sufficiency flag, and a rationale; the process defaults to the full validated plan if no earlier prefix qualifies.
  • Quality–effort trade-off: SPIN trades lower external execution effort for additional internal prompting effort from simulator and critic calls.The targeted external measures include executed tasks, tool calls, API calls, and elapsed time, while internal prompt and completion tokens may increase.

4 Experiments

SPIN is evaluated across offline plan validation, AssetOpsBench execution outcomes and failure modes, and MCP Bench quality and efficiency metrics. The experiments test whether validated DAG planning and prefix-based control improve executability, outcomes, execution burden, and robustness across models and settings.

  • Experimental setup and systems: SPIN validates planner outputs into dependency-aware DAGs, while simulator- and critic-based prefix evaluation stops execution when the current prefix is sufficient.The evaluation compares full SPIN with variants that remove simulation or weaken critic-based stopping.
  • AssetOpsBench results: On AssetOpsBench, SPIN increases Accomplished from 0.638 to 0.706 while reducing executed tasks from 1061 to 623 across 261 scenarios.Not decreases from 0.244 to 0.195, while Error remains comparable at 0.009 for BASE and 0.010 for SPIN.
  • AssetOpsBench results: SPIN reduces average tasks per run from 4.07 to 2.39, tool calls from 11.81 to 6.82, API calls from 34.05 to 19.97, and runtime from 198.44s to 143.53s.Tokens sent and received increase, reflecting additional internal simulator and critic prompting.
  • AssetOpsBench ablations: Removing the simulator more strongly degrades robustness-related failure patterns, while weakening the critic produces longer and more expensive executions.The ablations use smaller subsets than the full 261-scenario comparison, so their component-role conclusions require caution.
  • Failure mode analysis: On the matched 95-scenario set, SPIN most clearly reduces specification/state failures, especially Step Repetition, but does not clearly improve clarification or verification failures.The failure analysis indicates stronger gains in redundant-continuation control than in uncertainty-handling or verification robustness.
  • MCP Bench results: Without external memory, SPIN improves most quality-oriented MCP Bench metrics for both GPT-OSS1 and Llama 4 Maverick, while efficiency effects remain model-dependent.GPT-OSS1 Avg Rounds remains 7.06, whereas Llama 4 Maverick decreases from 17.06 to 14.00; GPT-OSS1 Tool Call Success decreases from 0.97 to 0.92.

5 Limitations

SPIN’s evaluation is bounded by the environments, trajectory data, and model-based control components used in the experiments. These constraints affect generalization, simulation accuracy, stopping reliability, and engineering cost.

  • Evaluation is limited to AssetOpsBench and its fixed tool/agent environment, leaving trade-offs untested on other benchmarks and APIs.Different tool latencies and failure characteristics may change observed quality and efficiency.
  • Simulator quality depends on logged-trajectory coverage, freshness, and representativeness, so sparse or shifting environments can produce inaccurate outcomes.Those inaccuracies may propagate into suboptimal intermediate decisions.
  • The LLM-based critic can miscalibrate early stopping, lacks formal stopping guarantees, and is sensitive to rubric design, few-shot selection, and model choice.False positives can halt too early, while false negatives reduce efficiency.
  • SPIN adds repair prompting, database retrieval, simulator, and critic components that increase engineering complexity and per-step inference overhead.

6 Conclusion

SPIN combines executable DAG validation with prefix-based execution control to reduce downstream execution burden while maintaining or improving task quality. The analyses attribute the gains mainly to progression and stopping control, while identifying weaker robustness to clarification and verification failures.

  • SPIN combines executable DAG validation with prefix-based execution control for tool-using LLM agents.
  • Across experiments, SPIN improved machine-consumable planner outputs and reduced downstream execution burden while maintaining or improving task quality.AssetOpsBench showed fewer executed tasks, tool calls, API calls, and shorter runtimes, alongside improved Accomplished.
  • SPIN’s gains primarily reflect better progression and stopping control, especially reducing redundant continuation and repetition-related failures.The simulator and critic contributed differently to state estimation and execution-length control.
  • Clarification- and verification-related failures were not substantially improved, marking a boundary of the current approach.The paper proposes conditional planning with explicit clarification and fallback branches as a next step.

A.1 Experiment Regeneration

The experiment-regeneration workflow recomputes reported tables from saved artifacts and provides optional Docker-based replay paths. Deterministic post-processing is separated from replay stages that depend on external model or API calls.

  • Deterministic table recomputation: Reported tables are recomputed from saved trajectory traces and experiment metadata without rerunning benchmark execution or invoking external APIs.The deterministic scripts read artifacts under the AssetOpsBench track1_result directories.
  • End-to-end replay: End-to-end Docker Compose replay regenerates trajectories and experiment summaries but depends on external model/API calls, so bitwise-identical outputs and exact metric equality are not guaranteed.Users must provide local API credentials and endpoints.
  • Failure mode analysis replay: Failure-mode replay also depends on external calls and is expected to produce broadly consistent rather than bitwise-identical distributions.The paper calls the variant SPIN_wo_cri although its released artifact directory is named spin_wo_cri2/work/.
  • Failure mode analysis replay: TrajFM extracts failure-mode outputs separately for each method directory and stores them under trajfm_outputs for matched-scenario analysis.
  • Figure generation: The plotting script renders failure-mode figures from precomputed aggregated rates rather than performing raw trajectory labeling or aggregation.
  • External API dependency: Exact replay may become impossible if an external provider changes model-identifier mappings or discontinues service.Only partial reproducibility from saved artifacts is guaranteed in that case.

A.2 Submission Package Structure (Software Archive / Data Archive)

The submission package contains separate software and database archives, restoration instructions, and workflow configuration needed to regenerate experiments. Reviewers can switch Track 1 variants through Docker Compose file references.

  • Archives: The reproducibility package provides AssetOpsBench.zip as a software archive and pgdata.tgz as a populated PostgreSQL data archive.
  • Archive contents: The software archive includes a repository-level readme.txt, while the data archive contains PostgreSQL directories such as PG_VERSION, base, global, and pg_wal.
  • Implementation location: The primary Track 1 workflow implementation resides under src/agent_hive/workflows/ and includes planning, validation, critic, simulator, and workflow-variant modules.
  • Configuration switching: Changing three referenced filenames in docker-compose.yml selects different Track 1 workflow variants, including baseline and SPIN configurations.The services must be restarted after editing the Compose configuration.
  • Restoration and rerun: Reviewers can restore PostgreSQL state, recreate the local environment file, rerun the Compose stack, and regenerate tables using the repository instructions.

A.3 Fixed Experimental Settings (Configuration Only)

The evaluation uses fixed scenario sets for each study component, while reported outcomes depend on several configuration choices and external API logging conditions.

  • Fixed evaluation sets: 261 scenarios are used for the end-to-end AssetOpsBench results, while MCP Bench uses 18 two-server grounding tasks.The offline DAG validation study evaluates 141 plan files per planner configuration.
  • Configuration choices: SPIN iteration budget, simulator retrieval filtering, early-stopping conditions, and database persistence can change reported aggregates.The early-stopping rule activates when the Critic returns can_answer_now=True and the Simulator predicts Accomplished or Partially_Accomplished.
  • External API conditions: External API reproductions identify the provider and model and log timestamps, model identifiers, inference parameters, and retry information.

A.4 Failure mode analysis on matched scenarios

The matched-scenario failure analysis compares four methods on the exact common utterance-ID intersection, using a predefined taxonomy spanning specification, interaction, and verification failures.

  • Scope and comparison: Failure analysis is restricted to the exact common utterance-ID intersection across BASE, SPIN, SPIN_wo_sim, and SPIN_wo_cri.After collapsing to one trajectory per utterance ID, the matched set contains 95 shared scenarios.
  • Evaluation procedure: Failure labels are assigned by an LLM-based evaluator using the trajectory, final answer, and user question to produce structured multi-label judgments.
  • Failure taxonomy: The taxonomy covers specification and state-related failures, interaction failures, and verification and completion failures.

B.1 Failure mode analysis on matched scenarios

On the matched 95-scenario set, SPIN has the lowest average failure count and substantially reduces specification and state-related failures, especially repetition, while interaction failures vary little.

  • Aggregate failures: 1.48 is SPIN’s mean failure count per scenario, versus 1.87 for BASE, 1.80 for SPIN_wo_sim, and 1.53 for SPIN_wo_cri.The comparison uses the exact common 95-scenario intersection.
  • Specification and state failures: 1.x specification and state-related failures decrease from 60 for BASE to 27 for SPIN, with SPIN_wo_sim at 41 and SPIN_wo_cri at 33.
  • Specification and state failures: 35.79% to 10.53% is the reduction in 1.3 Step Repetition from BASE to SPIN.The corresponding rates are 17.89% for SPIN_wo_sim and 12.63% for SPIN_wo_cri.
  • Verification and completion failures: 84, 78, 93, and 77 are the 3.x verification and completion failure totals for BASE, SPIN, SPIN_wo_sim, and SPIN_wo_cri, respectively.No or Incorrect Verification and Weak Verification are lower for SPIN than BASE, while Premature Termination is identical for BASE, SPIN, and SPIN_wo_cri.
  • Interaction failures: 34, 36, 37, and 35 are the 2.x interaction failure totals for BASE, SPIN, SPIN_wo_sim, and SPIN_wo_cri, respectively.Fail to Ask for Clarification ranges from 24.21% to 26.32% across methods.

D.2 What the Results Show

SPIN combines validated DAG interfaces with prefix-based execution control, improving outcomes and reducing downstream effort across AssetOpsBench and MCP Bench while adding internal prompting overhead.

  • End-to-end results: 0.638 to 0.706 is the increase in AssetOpsBench Acc, while executed tasks fall from 1061 to 623.Tool calls decrease from 11.81 to 6.82 per run, alongside lower API calls and elapsed time.
  • Simulator interpretation: The simulator’s benefits persist on additional assets absent from its retrieval database, suggesting task-level knowledge transfer beyond asset-specific memory.
  • Cross-benchmark transfer: SPIN improves planning, grounding, dependency, and efficiency scores for both GPT-OSS1 and Llama 4 Maverick on MCP Bench.For Llama 4 Maverick, average rounds decrease from 17.06 to 14.00 and prompt tokens from 212,118.22 to 125,243.06.
  • Failure analysis: 1.48 versus 1.87 is SPIN’s matched-set mean failure count compared with BASE, while 1.3 Step Repetition drops from 35.79% to 10.53%.The matched comparison covers 95 shared scenarios and indicates more stable, less repetitive progression.
  • Component roles: The simulator is the more load-bearing component for matched-set robustness, while the critic remains more directly associated with execution-length control.SPIN does not reduce clarification-related failures relative to BASE.
  • Impact: SPIN treats planner outputs as executable interfaces whose quality should be assessed through execution burden and downstream failure patterns, not only reasoning length.
  • Limitations and next steps: Conditional planning with explicit fallback branches is proposed as a next step because efficiency-oriented control does not automatically improve clarification or fallback behavior.
Loading 2605.14051v1…