Source-linked AI summary
Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution
Hyewon Suh, Thanh Minh Nguyen, Chih-Lun Lee, Darrow Hartman, Lizhao Liu, Xin Eric Wang, Ang Li, Jiachen Yang
TL;DR
Computer-use agents re-plan every step of recurring workflows, making repeated execution costly and unreliable. This paper learns reusable neuro-symbolic policies through iterative execution, diagnosis, and revision, then refines them across task variants. The policies lead across the evaluated settings while reducing per-run cost and latency and transferring to held-out instances.
Problem
Current computer-use agents plan step-by-step on every run, creating reliability concerns across repeated runs and high costs and latency on long workflows.
Source: §1 · p. 1Method
NSPI converts an agent trajectory into a policy that encodes stable decisions in code, then iteratively evaluates and revises it using judges and agent continuation.
Source: §1 · p. 2Results
NSPI policies achieve the highest Pass^3 in all four OSWorld and ScienceBoard settings, 4.5–13.9 points above the base agent, while running 15–217× cheaper and 3.4–5.1× faster per run.
Source: §1 · p. 2Takeaways & Limitations
The conclusion identifies trajectory-to-policy conversion and policy evaluation as remaining bottlenecks alongside the policies’ reuse and transfer across recurring workflows.
Source: §6 · p. 11On longer workflows, agent trajectories averaged 158 steps and scripts 970 lines, leading to mistakes when scripts recomputed every value read from the screen.
Source: §4.5 · p. 10
Abstract
from arXiv · showhide
Many computer tasks recur: the same workflow runs many times, with new inputs and from different starting states. Current computer-use agents re-plan every step of every run, which makes them costly and unreliable on such tasks. We introduce neuro-symbolic computer use, in which a recurring workflow is executed by a learned policy rather than re-derived by an agent on each run. The policy fixes the decisions that are stable across runs (ordering, variables, loops, and branches) in executable code, and delegates observation-dependent decisions, such as grounding and state checks, to neural models. We learn these policies with neuro-symbolic policy iteration: starting from one agent trajectory, it executes the policy, diagnoses failures with task-completion and step-level judges, and revises the code with a coding model informed by an agent's continuation from the point of failure, without access to the benchmark evaluator. Iterating on generated parameter and initial-state variants makes the policy reusable, and a pre-action verifier guards each state-mutating step at deployment. On OSWorld-Verified and ScienceBoard, the learned policies achieve the highest Pass^3 of all methods in all four settings, 3.6-15.8 points above the base agent, while cutting per-run cost by 15-217$\times$ and latency by 3.4-5.1$\times$. On OSWorld-Verified, policies built only on variants transfer to the held-out original tasks, exceeding AutoRPA by 8.6-17.5 points in Pass^3.
1. Introduction
Recurring workflows expose a trade-off between costly, unreliable step-by-step agent planning and brittle fixed scripts. Neuro-symbolic policies and NSPI address it by encoding stable procedure in code while retaining neural decisions for changing states.
1. Introduction
- Neural agents re-decide every action, incurring cost and reliability concerns across repeated runs, while fixed RPA scripts are cheap to repeat but break when states change.The contrast motivates a representation that reuses stable workflow structure without assuming every state is identical.Source: §1 · pp. 1–2
- Neuro-symbolic policies encode stable workflow decisions in code and delegate observation-dependent decisions to neural primitives at run time.The policy fixes the algorithm, variables, and control flow while neural primitives handle decisions such as grounding and state checks.Source: §1 · p. 2
- NSPI starts from a base agent trajectory, then executes, judges, and revises the policy using agent continuations, without benchmark-evaluator feedback during iteration.Generated parameter and initial-state variants support generalization; pre-action verifiers guard state-mutating actions at deployment.Source: §1 · p. 2
- NSPI policies achieve the highest Pass^3 across all four OSWorld and ScienceBoard settings, 4.5–13.9 points above the base agent.Policies built only on OSWorld variants also exceed AutoRPA on held-out original tasks by 8.6–17.5 points in Pass^3; per-run cost falls 15–217× and latency 3.4–5.1× versus the base agent.Source: §1 · p. 2
2. Neuro-Symbolic Computer Use
Recurring desktop tasks vary in parameters and starting states, so a reusable policy must succeed across related instances. Neuro-symbolic policies combine executable stable structure with neural primitives for decisions that depend on current observations.
2.1. Problem statement: recurring workflows and reusable policies
- A policy for a recurring task is intended to succeed across both a similar family of parameterized instructions and varying initial states.The task is modeled as an instruction executed in a digital environment, with success assessed by a held-out indicator.Source: §2.1 · p. 2
- The reliability objective is success across repeated executions, while amortized per-execution cost accounts for both policy construction and its run cost.The one-time construction cost is spread over n executions; a policy with a lower per-run cost can become cheaper after enough runs.Source: §2.1 · pp. 2–3
2.2. Neuro-symbolic policies
- Figure 1 depicts a policy that retains stable workflow structure in code, uses neural primitives for observation-dependent choices, and is reused with pre-action verification.The figure’s workflow starts from agent exploration and iteratively evaluates, diagnoses, and improves the policy.Source: Figure 1 · p. 3, Figure 2 · p. 4
- Neuro-symbolic policies are programs whose control flow and assignments combine symbolic primitives with neural calls that act on current observations.Neural primitives include GUI grounding and condition checks; symbolic operations such as shortcuts and file operations run without a model call.Source: §2.2 · p. 3
3. Neuro-Symbolic Policy Iteration
NSPI learns a task policy from an agent trajectory, iteratively tests and repairs it, and then refines it across task variants. Dual judges diagnose rollouts, while pre-action verification guards state-mutating actions during deployment.
3. Neuro-Symbolic Policy Iteration
- NSPI first learns a policy for reliable execution on a task and then seeks to generalize it beyond the instance used for learning.Its policy-iteration process alternates evaluation by dual judges with improvement by a code healer.Source: §3 · p. 4, §3.1.1 · p. 4
3.1.1. Synthesis and iterative refinement
- A base-agent trajectory initializes the policy; successive rollouts from reset states are judged, and a failed rollout triggers a revised policy.The process returns a policy when all K rollouts pass the judges, or returns the last policy after at most N iterations; the benchmark reward is reserved for evaluating the returned policy.Source: §3.1.1 · p. 4
3.1.2. Policy evaluation with a dual judge
- A task-completion judge probes the rollout’s end state, while a step-level judge identifies the first divergence and names a failure mode.The completion judge can be skipped when success cannot be checked from persistent state; in that case, the applicability gate determines whether it applies.Source: §3.1.2 · pp. 4–5
3.1.3. Policy improvement with a code healer
- After failure, the base agent continues from the failed state and a summarizer condenses its progress for a coding-model healer to revise the policy.Execution errors bypass the judges and continuation, sending the error directly to the healer.Source: §3.1.3 · p. 5
3.2. Policy generalization
- NSPI targets anticipated variation by iterating on variants that perturb task parameters and initial states before evaluating on the held-out original task.Variants can include changed dates, filenames, or settings, and states such as a closed target document or stale output; validation rejects variants that change the task goal or setup-file content.Source: §3.2 · p. 5, §3.2.1 · p. 5
3.2.1. Robust learning under workflow perturbations
- Sequentially presenting validated variants exposes hard-coded values and unhandled states, requiring the policy to handle multiple instances rather than one example.The method holds out the original task for evaluation after learning on the variants.Source: §3.2.1 · p. 5
3.2.2. Pre-action verification
- A calibrated VLM verifier checks each state-mutating primitive’s precondition before deployment and either passes or blocks the action.A block can lead to a stateful agent, another NSPI run, or human escalation; the decision uses the probability mass of “no” responses against a calibrated threshold.Source: §3.2.2 · p. 6
4. Experiments and Results
Across OSWorld and ScienceBoard, the experiments test repeated-run reliability, transfer to held-out workflow instances, and execution cost. NSPI leads the baselines on repeated-run Pass^3, transfers better than AutoRPA, and reduces per-run cost, while the long-workflow and verifier analyses expose important trade-offs.
4.2. Reliability across repeated executions
- NSPI has the highest Pass^3 in all four settings, with a 3.4-point Pass^1-to-Pass^3 drop versus 12.4 points for the base agent.Its Pass^1 remains comparable to the base agent, and its average Pass^3 advantage over the next-best method is 8.1 points.Source: Table 1 · p. 7, §4.2 · p. 7
- Batch2Code’s Pass^k curve is similarly flat but remains 3.3–33.1 points below NSPI; failed-seed initialization also yields reliable policies, at higher healing cost and time.Of 356 tasks where the base agent failed all three seed attempts, 62 final policies succeed in Pass^3; failed seeds incur 18–31% higher healing cost and 17–38% longer healing time.Source: §4.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8
- Across the four settings, NSPI reduces per-run cost by 15–217× and runs 3.4–5.1× faster than the corresponding base agents.On ScienceBoard with GPT-5.6 Terra, Figure 3 shows high Pass^k as k increases alongside efficient per-run execution cost.Source: §4.2 · p. 8, Figure 3 · p. 8
4.3. Policy generalization across workflow instances
- On held-out OSWorld tasks, NSPI reaches 55.0% Pass^3 for parameter variations and 60.0% for initial-state variations, versus AutoRPA’s 37.5% and 51.4%.Policies are refined on generated variants, then frozen for evaluation on the original task.Source: §4.3 · p. 8, Table 2 · p. 9
- Policies reading changing values from requests transfer more often; for initial-state variation, file-modifying or command-line paths transfer more often than GUI-only policies.NSPI performs better than AutoRPA within both script-based and GUI execution styles.Source: §4.3 · pp. 8–9
4.4. Efficient repeated execution and cost amortization
- Policy construction costs are recovered after 3.1–26.8 executions, while wall-clock break-even takes 8.6–16.9 executions.The authors caution that the comparison spans only two benchmarks and two base models, partially confounding task complexity with other benchmark differences.Source: §4.4 · p. 9, Table 3 · p. 9
4.5. Extension to long workflows on OSWorld-V2
- On OSWorld-V2, NSPI retains 96% of its Pass^1 at k=3, yet trails the base agent by 5.6 points in strict success and 18.5 points in partial credit.The authors attribute the performance gap to information loss when converting long agent trajectories into initial policies; trajectories averaged 158 steps and scripts 970 lines.Source: §4.5 · p. 10, Table 4 · p. 10
4.6. Pre-action verification under environment variation
- At θ=0.78, Qwen3.6-35B-A3B wrongly blocks at most 8.4% of correct rollouts but passes 70.8% of block-worthy primitives; Qwen3.8-27B has the reverse trade-off.Wrong-block rate measures rollouts of correct policies blocked at least once; wrong-pass rate measures primitives that should be blocked but are passed.Source: §4.6 · p. 10
5. Related work
Prior work ranges from agents that accumulate reusable skills to generated or replayable scripts; this work instead targets a robust code policy for each recurring task family. Its comparison with AutoRPA emphasizes Pass^k, generalization across task instances, and precondition checks.
Related work
- The policy is intended to solve an equivalence set of parametric tasks in full, with the neural agent serving as an expensive fallback rather than the primary driver.Earlier skill-learning methods build reusable APIs in natural language or short programs for an agent to invoke.Source: §5 · pp. 10–11
- Prior LLM-generated policies used human instructions and few-shot examples, and were static; this work refines policies through execution feedback and continued agent exploration.The paper also places its approach in a longer line of programs with perceptual inputs, including programmatic reinforcement learning and LLM-generated robot code.Source: §5 · p. 11
- AutoRPA converts ReAct-style exploration into replayable scripts and reduces cost while maintaining baseline pass@1; this work explicitly targets Pass^k and intra-task generalization.It also adds a precondition check for deployment.Source: §5 · p. 11
6. Conclusion
The paper presents reusable neuro-symbolic policies for recurring workflows, combining stable code with neural decisions for state-dependent actions. Across OSWorld and ScienceBoard, these policies improve repeated-execution reliability, reduce cost and latency, and transfer better than AutoRPA to held-out instances.
Conclusion
- Reusable policies preserve stable procedure in code while leaving state-dependent decisions to neural primitives.Neuro-Symbolic Policy Iteration learns and refines these policies from agent trajectories through execution, diagnosis, exploration, and revision.Source: §6 · p. 11
- Across OSWorld and ScienceBoard, the resulting policies improve repeated-execution reliability while reducing per-run cost and latency, and transfer better than AutoRPA to held-out workflow instances.The conclusion identifies trajectory-to-policy conversion and policy evaluation as remaining bottlenecks.Source: §6 · p. 11
A. Method details
The method converts an agent trajectory into a policy, refines it through judged rollouts and agent continuation, and uses separate judges and a pre-action verifier to support execution. The appendix details the primitives, initialization, iteration, diagnosis, and verification procedures.
A.1. Primitive action space
- Simulang policies combine symbolic actions that run without model calls with neural primitives for grounding and state checks, and can use Python control flow.The healer can use loops to retry clicks or branches on state checks.Source: §A.1 · p. 13
A.2. Initialization
- The converter extracts a trajectory into a script, replaces machine-specific coordinates with descriptions grounded at runtime, and substitutes task-specific values with parameters.Conditioning on the task lets it omit incidental exploratory actions from the initial policy.Source: §A.2 · p. 14
A.3. Iteration
- Each iteration runs policy rollouts, diagnoses the first judged failure, and revises the policy using an agent continuation; execution errors instead send the traceback to the healer.Rollouts and continuations share the task step budget, and the continuation serves as healer experience rather than changing the rollout verdict.Source: §A.3 · p. 14
A.4. Dual judge
- The dual judge pairs end-state checks with step-level diagnosis: task checks use read-only probes of persistent state, with an applicability gate skipping checks when success is not persistently verifiable.The two judges address limits of actor reports and trajectory-only evaluation, which can miss whether an intended result was saved or persisted.Source: §A.4 · pp. 14–15
- Observers report step outcomes from screenshots, accessibility-tree changes, or command outputs; the localizer uses those reports to identify the first divergence and a failure mode.An aggregator separates observations from interpretations in the healer's diagnostic.Source: §A.4 · pp. 15–16
A.5. Pre-action verifier
- The verifier runs before every state-mutating primitive and blocks actions when the no-answer probability meets its threshold; a block triggers the script's fallback hierarchy.The verifier also covers reversible writes, since a lost action can leave later steps operating on an unchanged page.Source: Figure 6 · p. 16, §A.5 · pp. 16–17
- The verifier input includes the primitive call, its code-comment explanation, the current screenshot or accessibility tree, and the run's executed-step history.The verifier maps this input to pass or block; its no-answer mass is derived from the top-20 token probabilities.Source: §A.5 · p. 17
- A shadow-mode calibration set selects the smallest threshold within the deployment's wrong-block budget, maximizing detection among feasible thresholds.Successful rollouts define wrong blocks; both wrong-block and detection rates are non-increasing as the threshold rises.Source: §A.5 · p. 18
B.1. Main experiment configuration
The main experiment compares NSPI with a base agent, Batch2Code, and ASI, using shared backbones within each benchmark. It estimates Pass^k from three fresh trials per task and reports construction and per-run costs separately.
B.1. Main experiment configuration
- The experiment compares NSPI with a base agent, Batch2Code, and ASI using GPT-5.6 Terra and Claude Opus 5 as backbone models.Batch2Code converts three agent trajectories without refinement, while NSPI starts from one trajectory and runs five iterations of up to five rollouts.Source: §B.1 · p. 18
- Each method receives three fresh trials per task, and Pass^1, Pass^2, and Pass^3 are estimated over the full task set.Reported cost includes provider-billed model cost and grounding-model GPU time.Source: §B.1 · p. 18
- Construction cost covers policy conversion and refinement for NSPI, conversion for Batch2Code, or online skill-library building for ASI; per-run cost covers one final trial.Source: §B.1 · p. 18
B.2.1. Task variant generation
Task variants let policies be refined across changed starting states and task parameters while withholding the original task for evaluation. Generation and validation constrain variants to preserve feasible task semantics and benchmark scoring.
B.2.1. Task variant generation
- Variants are generated by one LLM call and deterministic validation, with initial-state variants changing transient setup and parameter variants changing instruction values and corresponding evaluator constants.Initial-state setup must leave task files and the original solution unchanged; parameter variants are rejected if scoring changes or the evaluator still checks against an obsolete reference file.Source: §B.2.1 · pp. 18–19, Figure 7 · p. 19
- On OSWorld, all 332 feasible pre-filtered tasks admit initial-state variants, but only 93 admit parameter variants, largely because reference-file checks cannot be regenerated.The resulting families include 40 parameter and 35 initial-state families across ten domains, generally with three variants each.Source: §B.2.1 · p. 19
- Variants are processed sequentially with one policy carried across a family; construction uses model-judge feedback, not benchmark evaluators, and NSPI allows up to three refinement iterations.The setup uses GPT-5.6 Terra for construction roles and the same grounding model for both methods.Source: §B.2.1 · p. 19
- After construction, policies are frozen and evaluated three times from fresh environments on the original task, with construction failures assigned zero in family averages.AutoRPA is adapted to computer-use settings while retaining its exploration, reflection, continuation, verification, and rollback logic.Source: §B.2.1 · p. 19
B.3. Judge analysis details
The judge analysis finds both useful failure detection and systematic blind spots: completion judgments can disagree with task instructions, while step localization is weaker for hidden or cascading failures. The pre-action verifier is supplemented with intent comments and additional rubric rules for potentially misleading actions and targets.
B.3. Judge analysis details
- The task-completion judge was correct on 16 of 19 annotated tasks, but gave false alarms on three rollouts; the benchmark evaluator also passed one rollout that violated the task instruction.The evaluator checked only that a file named three metrics, while the annotator required copying them from the specified webpage.Source: §B.3 · p. 20
- For 10 failures visible on screen, the step-level judge’s divergence step was correct in 7, adjacent in 2, and wrong in 1; its narrative named the cause fully in 4 and partly in 4.Six additional failures occurred only inside scripts and were never visible; the judge’s localization was wrong in all six, and its suggested focus was useful in 7 of the 10 visible cases.Source: §B.3 · p. 20
- The step-level judge receives no primitive return values, so script-output errors such as missing libraries are hidden; these common failures were excluded from the sample, making reported judge counts more favorable.The judge sees policy code and screenshots, but not shell or script output or condition-check answers.Source: §B.3 · p. 20
- Late or missed clicks can shift a policy one step behind, leading later actions to hit wrong targets and the judge to localize a later symptom instead of the needed fix.A short wait can let the next click fire before its target loads.Source: §B.3 · p. 20
- Neither judge recognizes infeasible tasks; in the sample, both recommended completing requests for a nonexistent GIMP theme or a photo absent from the folder.The sample’s two infeasible tasks asked for a “Blue” theme and a photo in an empty folder.Source: §B.3 · pp. 20–21
B.4. Precondition checks
- Because most final policies lack intent comments, the verifier’s default literal restatement can make correct but context-dependent actions look destructive; the authors inject comments explaining each action’s role in the plan.On OSWorld with the Terra converter, 99.9% of gated primitives fell back to literal restatements before comment injection.Source: §B.4 · p. 21
- Verifier rubric additions exempt task-related open, close, save, and retry steps while blocking inputs aimed at closed applications or targets covered by dialogs, popups, or menus.The study reports results both with and without these additions because they change the production rubric.Source: §B.4 · p. 22
B.5. OSWorld-V2 extension
The OSWorld-V2 extension uses a June 2026 benchmark release and reports results for 107 tasks, with one task excluded because its hard-coded GitLab instance cannot be scored as intended. Construction settings differ from the main experiments, and infrastructure changes and costs are documented as part of the run context.
B.5. OSWorld-V2 extension
- The extension uses OSWorld-V2 v2026.06.24 with task definitions, evaluators, and assets unmodified, reporting 107 of 108 tasks; task 041 is excluded because its hard-coded GitLab instance conflicts with local setup and grading.Task 093 remains included as a zero because the converter produced no program.Source: §B.5 · p. 22
- During the experiment, 39 web-application tasks changed hosting versions after the benchmark authors replaced the v2026.06.24 applications with newer versions at the same addresses.The switch occurred between 2026-08-25 and 2026-09-10.Source: §B.5 · p. 22
- Compared with OSWorld and ScienceBoard, the OSWorld-V2 setup uses Claude Opus 5 as converter, DeepSeek-V4.1-Flash for continuation, and a step-level judge that cannot localize failures to a step.The healer also rewrites after a failure in the last iteration, and that rewrite is returned without judgment.Source: §B.5 · p. 23
- Thirteen infrastructure-fault reruns replaced original scores regardless of direction: four rose, one fell, and eight were unchanged.Tasks with reruns still running retain their original scores.Source: §B.5 · p. 23
- Construction averaged $8.2 per task in LLM calls and a median 2.9 VM-hours; estimated execution-time LLM costs averaged $0.03–$0.08.Counting infrastructure reruns, total LLM spend was 1.8× the construction-call amount.Source: §B.5 · p. 23
C. Constructing Policies from Unsuccessful Trajectories
NSPI can produce consistently successful policies even when its selected seed trajectory failed all three base-agent attempts. Recovering from those seeds is more expensive and slower than refining successful seeds, while successful-seed policies also improve over the base agent in most settings.
C. Constructing Policies from Unsuccessful Trajectories
- The failed-seed split includes tasks where the base agent failed all three attempts, distinguishing recovery from an unsuccessful demonstration from refinement of a successful one.The policies’ final success is measured with the benchmark evaluator.Source: C. Constructing Policies from… · p. 23
- 62 of 356 policies built from failed seeds succeeded in all three final runs under the benchmark evaluator.On tasks with a successful seed, the final plan improved over the base agent in three of four settings.Source: C. Constructing Policies from… · p. 23
- Failed seeds cost 18–31% more and took 17–38% longer to reach a policy than successful seeds; in three of four settings they also received a larger share of judge rejections.The authors say this is consistent with failed trajectories containing more incorrect or missing steps to diagnose and repair.Source: C. Constructing Policies from… · p. 23