PaperlayerView exact revision on arXiv

Source-linked AI summary

When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows

Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang

arXiv:2610.13092v1cs.AI

TL;DR

Whether local decision savings translate into efficient, reliable task completion remains unclear. Across five agents and two benchmarks, the study evaluates Jev integrations and finds that decision cost and latency fall, while task-level gains vary by workflow.

  • Problem

    Whether faster, cheaper bounded decisions translate into efficient, reliable task completion remains unclear, as workflow actions, auxiliary reasoning, interactions, and failed attempts shape resource use.

    Source: §1 · p. 1

  • Method

    Across five agent families in ALFWorld and AndroidWorld, the study compares original and Jev-integrated configurations over 4,500 executions, measuring outcomes, resource use, and execution behavior.

    Source: §1 · p. 2

  • Results

    Decision cost and latency fall across all five agents, but task gains vary: success stays close to baseline on ALFWorld and declines more substantially in two mobile agents.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Integrations should preserve progress, support recovery, and verify completion, and be assessed through successful delivery, recovery behavior, and resources per delivered outcome.

    Source: §7 · p. 8

    Evidence covers selected agent families in language-observed mobile and household-task settings, with repeated runs on fixed tasks rather than newly sampled environments or distributions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

AI agents depend on repeated decisions to accomplish tasks, with large language models (LLMs) commonly serving as their core decision makers. The recent emergence of Jev offers an opportunity to make these decisions faster and at lower cost. However, it remains unclear whether such local savings can be translated into efficient and reliable task completion. In this paper, we investigate this efficiency transfer across five agents in two representative application settings, embodied tasks in ALFWorld and mobile app use in AndroidWorld. Across 4,500 task executions, we jointly evaluate task effectiveness, decision-level and task-level resource use, and execution behavior. Experimental results show that Jev integration substantially reduces decision cost and latency across all five agents, with task-level gains varying by workflow. Task success remains close to baseline levels on ALFWorld and declines more substantially in two mobile agents. Auxiliary reasoning and additional interactions absorb decision savings, while failed recovery and premature completion limit successful delivery. Mean cost per successful outcome increases in one agent even as spending per attempt falls. These findings highlight the importance of preserving progress, supporting recovery, and verifying completion when integrating efficient decision models into agent workflows. Their practical value should be assessed through successful task delivery, recovery behavior, and resources per delivered outcome.

1 Introduction

The study asks whether faster, cheaper action-selection decisions translate into efficient, reliable task completion, and evaluates Jev integrations across five agents and two benchmarks. Decision costs fall, but workflow overhead and recovery outcomes shape whether savings reach successful delivery.

Motivation

  • Whether local savings yield efficient and reliable task completion remains unclear because actions affect later observations and choices, and auxiliary reasoning, interactions, and recovery add workflow costs.Failed attempts consume resources without delivering the requested outcome.Source: §1 · p. 1

Study approach

  • Across five agent families and ALFWorld and AndroidWorld, the study compares original and Jev-integrated configurations over 4,500 executions, measuring effectiveness, resource use, and execution behavior.Systematic trace coding characterizes failure and recovery patterns.Source: §1 · p. 2

Results

  • Decision cost and latency fall substantially across all five agents, but task-level gains vary; success stays near baseline on ALFWorld and declines more in two mobile agents.The introduction also reports that mean task costs fall across agents, while auxiliary reasoning and additional interactions limit overall speed gains.Source: §1 · p. 2
  • Lower spending per attempt can coexist with higher cost per successful outcome, while failed recovery and premature completion constrain successful delivery.The paper argues that practical value should be assessed through successful task delivery and the resources it requires.Source: §1 · p. 2

Implications

  • The findings emphasize preserving progress, supporting recovery, and verifying completion, while assessing efficient decision models by successful delivery and its resource requirements.Source: §1 · p. 2

2 Background

The background introduces Jev as a structured decision model and situates its choice interface within repeated action selection in agent workflows.

Jev

  • Jev maps shared state and instruction-defined questions to typed choices, scores, or binary probabilities, with choice questions returning probabilities over supplied candidates.The paper describes it as a System One Model for fast, structured decisions in software automation.Source: §2 · p. 2
  • Agents can provide goals, observations, and available actions as context for Jev’s candidate-based selections during a workflow.Source: §2 · p. 2
  • Prior Jev studies examine probabilistic coding of crash narratives (Rafe and Das, 2026) and adoption across public software projects (Ling et al., 2026), while this paper studies decisions within task execution.Source: §2 · p. 2

3 Related Work

Related work frames bounded action selection as one component of workflows shaped by candidate construction, execution, feedback, and recovery, and links it to resource-aware evaluation.

Agent Action Selection

  • Agent action selection maps goals, observations, and interaction history to actions, while workflows determine available candidates and condition later decisions through feedback, reflection, and memory.Bounded action selection can coexist with open-ended argument generation.Source: §3 · p. 2

Efficient Decision Models and Routing

  • Routing and cascade methods allocate requests across models, whereas this study examines bounded action selection across five agent families and its transfer to task-level efficiency.The study considers auxiliary reasoning, interaction length, recovery, and completion when characterizing that transfer.Source: §3 · p. 3

Agent Evaluation and Sequential Reliability

  • The study connects decision-level resources to total expenditure and resources per delivered success, including failed attempts, and codes recovery and completion errors.Prior work includes resource-aware evaluation and repeated trials for consistency (Kapoor et al., 2025).Source: §3 · p. 3

4 Empirical Study Design

The study evaluates Jev-based bounded action selection through task outcomes, resource use, and execution behavior, with matched within-family comparisons and trace analysis.

4 Empirical Study Design

  • The study assesses Jev integration through task effectiveness, execution efficiency, inference cost, and consistency across runs, focusing on translation from decision savings to successful delivery.Source: §4 · p. 3

4.1 Research Questions

  • The research questions ask whether Jev changes task success, when decision-level gains transfer to task-level efficiency, and what failure and recovery patterns characterize Jev-based agents.Source: §4.1 · p. 3

4.2 Experimental Setup

  • AndroidWorld uses text UI observations for 116 task types, while ALFWorld uses textual observations and all 134 valid_unseen tasks; configurations include baseline, Jev, and, in ALFWorld, LLM-Choice variants.Table 1 labels the variants B, J, and C; the LLM-Choice control uses Jev’s candidates and instructions to separate interface from decision-model comparisons.Source: §4.2 · p. 3, Table 1 · p. 4
  • Jev replaces action selection while retaining surrounding observation, history, and execution mechanisms; each configuration runs three times on fixed tasks with within-family matched budgets.Auxiliary LLMs provide open-ended text and summaries, and cross-family comparisons also reflect differing budgets and workflow designs.Source: §4.2 · pp. 3–4

4.3 Evaluation and Metrics

  • Evaluation reports benchmark task success, AndroidWorld partial-completion scores, task- and decision-level resource metrics, and cost and time per delivered success, with paired task-cluster bootstrap intervals.All task attempts, including failures, contribute to cost and time per delivered success.Source: §4.3 · p. 4

5 Results and Analysis

Jev preserves near-baseline success on ALFWorld but loses more ground in two AndroidWorld agents. It sharply reduces decision cost and latency, while workflow overhead, recovery, and completion determine how much benefit reaches successful delivery.

5.1 RQ1: Effectiveness

  • On ALFWorld, ReAct-J and Reflexion-J trail their original configurations by 1.49 and 1.74 pp, while AndroidWorld deficits reach 13.22 pp for T3A and 18.10 pp for FastAgent.The 95% task-cluster bootstrap intervals are entirely negative for T3A and FastAgent, and include zero for the other agents and both ALFWorld J–C comparisons.Source: §5.1 · p. 4, Figure 1 · p. 5
  • Success varies by task: Jev reaches 100% on three ALFWorld categories, while cleaning and longer AndroidWorld budgets show deficits; trajectories also illustrate premature stopping and failed correction.In the shortest AndroidWorld budget, T3A and FastAgent lose 12.5 and 15.3 pp; trajectory examples include an unmet Wi-Fi goal and repeated reasoning from a wrong-object selection.Source: §5.1 · pp. 4–5, Figure 2 · p. 5

5.2 RQ2: Efficiency Transfer

  • Task-level cost savings vary by workflow and transfer more consistently than speed; auxiliary work and additional interactions absorb selection savings.FastAgent’s task time rises 0.7%, and its one-tool-per-step configuration makes 44.1% more decisions; AutoDroid’s outside-model-call time rises from 164.43 to 213.37 seconds.Source: §5.2 · pp. 5–6, Figure 4 · p. 6
  • Decision cost falls 66.1%–78.4% and decision time 73.9%–81.5% across five families, while tokens per decision rise 53.5%–134.9%.Jev’s lower input price and zero output charge offset the token increase.Source: §5.2 · p. 5, Figure 3 · p. 6
  • Across attempts, cost per delivered success rises 7.7% for T3A, from 68.70 to 73.97 mUSD, despite lower spending per attempt.The increase has a 95% interval spanning zero; at observed means, lower completion offsets T3A’s 17.1% cost saving per attempt.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 RQ3: Failure and Recovery

  • In selected persistent-loss pairs, failed recovery appears in 40 of 57 Jev executions versus 3 of 57 baseline executions; ineffective search appears in 29 versus 2.The outcome analysis compares persistent Jev failures, the reverse contrast, and mixed outcomes across three runs.Source: Figure 5 · p. 7, §5.3 · p. 7
  • The results link reliable completion to correcting errors, checking action prerequisites, and verifying the requested final state.The coded behaviors and Reflexion outcomes connect reliability to how agents respond after an error.Source: §5.3 · pp. 7–8
  • Reflexion-J wins more first trials but recovers fewer initial failures than baseline, leaving lower final success across three runs.J succeeds in 11/19 initial failures versus 25/26 for B; recovery fractions condition on each configuration’s own initial failures.Source: §5.3 · p. 7, Table 4 · p. 8

6 Discussion

The discussion argues that efficient agent decisions must be paired with goal-sensitive action selection, recovery, and completion checks. It recommends evaluating progress through successful delivery and its resource requirements.

Model development

  • Train decision models to recognize goals, action prerequisites, and contradictory feedback, and to revise choices or terminate correctly.The discussion proposes contrasting successful and failed trajectories and calibrated confidence about whether available actions advance the goal.Source: §6 · p. 8
  • Assess progress by successful task delivery and required resources, alongside decision accuracy and response time.Source: §6 · p. 8

Agentic system design

  • Expose targets, prerequisites, expected state changes, and feedback in candidate interfaces; focus checks on completion and repeated lack of progress.Direct additional observation or reasoning toward unresolved decisions, where completion and recovery are at risk.Source: §6 · p. 8
  • Deployment choices should account for the original runtime bottleneck, interaction length, and expenditure per successful outcome.Auxiliary calls and interaction overhead can absorb faster-selection savings.Source: §6 · p. 8

7 Conclusion

Across five agents and two scenarios, Jev-based action selection yields workflow-dependent task gains, with success near baseline on ALFWorld and lower in two mobile agents. The conclusion emphasizes recovery and completion verification alongside successful delivery and its resource cost.

7 Conclusion

  • Task success stays close to baseline on ALFWorld but falls more substantially in two mobile agents, while decision savings produce workflow-dependent task gains.The evaluation covers five agents across two scenarios.Source: §7 · p. 8
  • Auxiliary reasoning and additional interactions absorb savings, while failed recovery and premature completion limit delivery.Mean cost per successful outcome rises in one agent despite lower spending per attempt.Source: §7 · p. 8
  • Agent workflows should preserve progress, support recovery, and verify completion, assessing efficient models by successful delivery, recovery behavior, and resources per delivered outcome.Source: §7 · p. 8

Limitations

The study’s conclusions are bounded by selected environments and configurations, fixed task instances, and measurement conditions that may affect reported resource and latency trade-offs.

Evaluation scope

  • Evaluation covers selected household-task and mobile environments with language-based observations and bounded action interfaces, not visual grounding, open-ended tool creation, or longer workflows.Three shared-seed repetitions characterize fixed task instances, not variation across newly sampled environments or task distributions.Source: Limitations · p. 9

System-level comparisons

  • Comparisons concern the evaluated integrations; differences across agent families also reflect interaction budgets and workflow design, so not every difference can be attributed to Jev.The ALFWorld LLM-Choice control uses a shared candidate interface but does not cover every integration.Source: Limitations · p. 9

Measurement conditions

  • Latency and cost depend on model versions, service conditions, and experimental prices, while hosted inference variation and accounting adjustments introduce uncertainty.Future model updates or pricing changes may alter observed trade-offs; reported costs are estimates under stated accounting rules.Source: Limitations · p. 9

A Evaluation and Statistical Methods

The section defines how task-level and action-selection resource measures are aggregated across runs and compared between configurations.

A.1 Aggregation and Resource Accounting

  • Within each run, task-level quantities are averaged over the reported task scope, while action-selection quantities are averaged per decision call.The scope may include all task instances or only official successes.Source: §A.1 · p. 11
  • Reported means and sample standard deviations summarize three runs, with 134 ALFWorld and 116 AndroidWorld instances in each full-task run.Each decision has equal weight within a run.Source: §A.1 · p. 11
  • For matched comparisons, the resource-change percentage is 100(m̄J/m̄L − 1), and successful-task pairs use instances completed by both configurations.Task and run identifiers define each pair.Source: §A.1 · p. 11
  • Cost per delivered success divides expenditure across all task–run executions by total successes; failure expenditure is the share spent on failed executions.Task expenditure includes decision and auxiliary calls.Source: §A.1 · p. 11

A.2 Task-Cluster Bootstrap

The task-cluster bootstrap estimates uncertainty in paired J-minus-control differences while retaining repeated runs of each sampled task.

A.2 Task-Cluster Bootstrap

  • 20,000 bootstrap replicates resample task identities with replacement and retain all three runs of both configurations for each sampled task.The evaluated task counts are 134 for ALFWorld and 116 for AndroidWorld.Source: §A.2 · p. 11
  • 95% intervals use the 2.5th and 97.5th percentiles, with estimates computed for success rate, task cost, task time, and resources per delivered success.The resampling describes uncertainty across the evaluated task mix conditional on fixed instances and run conditions.Source: §A.2 · p. 11

B.1 Task Effectiveness

The section specifies how effectiveness, resource use, paired effects, and execution behaviors are measured across repeated task runs.

B.1 Task Effectiveness

  • Success rates and AndroidWorld scores are reported as run means ± sample SD, while all-three success counts tasks completed in every run.Paired outcome counts retain task–run correspondence to distinguish gains, losses, and both-configuration successes.Source: §B.1 · p. 11, Table 5 · p. 12, Table 6 · p. 12

B.2 Resource Use and Successful Delivery

  • Table 9 separates task time into decision, auxiliary, and residual environment-interaction and other-overhead components, and reports auxiliary cost share.Component means are in seconds per task, pooled over three equally sized runs.Source: §B.2 · p. 12, Table 9 · p. 13

B.3 Paired Effect Estimates

  • Paired effect estimates cover all executions, matched successes, and resources per delivered success, using 20,000 task-cluster bootstrap replicates.Intervals are pointwise 95% intervals given by the 2.5th and 97.5th percentiles.Source: §B.3 · p. 12, Table 10 · p. 14

C.1 Coding Protocol and Behavioral Patterns

  • Two authors independently annotated goals, outcomes, and action records, with a third resolving disagreements using actions, observations, and final UI states.Behavior denominators include complete and partial records but exclude seven FastAgent baseline executions without usable action events.Source: §C.1 · p. 12
  • Behavior annotations cover selected outcome groups, with labels for errors such as failed recovery, premature completion, and ineffective search.Labels can overlap, and successful executions can include corrected errors; the reported mean Cohen’s κ is unweighted across seven labels.Source: §C.1 · pp. 12–14

B.2 Resource Use and Successful Delivery

  • Tables 7 and 8 report task and decision resource metrics across all tasks and successful executions, with decision averages divided by action-selection calls.The successful-task group uses each configuration’s own successful executions; cost units are 10−3 USD per task and 10−6 USD per decision.Source: §B.2 · p. 12, Table 7 · p. 13, Table 8 · p. 13

C.2 Case Evidence and Execution Records

Execution records pair official outcomes with case-level action histories, showing how recovery, current-state updates, and completion criteria shape whether workflows deliver the requested result.

C.2 Case Evidence and Execution Records

  • Each selected family–task pair contributes six executions, and run-order codes preserve each configuration’s three official outcomes.Table 12 presents outcomes for runs 1–3; Table 11 counts executions with substantive action records, and its behavior labels can overlap.Source: §C.2 · p. 14, Table 12 · p. 15, Table 11 · p. 15
  • 19 illustrative pairs—3 ALFWorld and 16 AndroidWorld, including 14 for FastAgent—span household tasks and mobile settings, records, calendars, and media.Representative failures include persisting with an incorrect target or accepting an unmet goal at termination.Source: §C.2 · p. 14
  • In cloth cleaning and Wi-Fi cases, wrong-object reasoning persists or execution ends before the required state is reached; Reflexion preserves a wrong premise across ten unsuccessful trials.ReAct-J repeatedly invokes THINK when cleaning remains unavailable; FastAgent-J declares success while Wi-Fi remains off.Source: §C.2 · p. 14
  • Recipe creation, activity reporting, and map-favorite cases show Jev succeeding through updated observations, requested output format, or the evaluator’s name-based acceptance criterion.For the activity query, both workflows obtain 105 minutes, but Jev returns the requested integer while the baseline sentence fails the evaluator.Source: §C.2 · p. 15
  • 21 mixed pairs comprise one ReAct, ten T3A, four AutoDroid, and six FastAgent tasks; 20 are AndroidWorld instances.Fixed task instances make these contrasts informative about variation within the same workflow.Source: §C.2 · p. 15
  • In two-object placement, success requires depositing the first object before collecting the second; in recipe deletion, a failed Jev run leaves the third target after its budget expires.A separate run performs 24 swipes after deleting two targets and still ends with the third target present.Source: §C.2 · pp. 15–16
  • The activity-duration completion call contains “105” despite an internal failure flag, and the evaluator accepts the returned answer; map-favorite acceptance uses the requested name.The baseline’s geographic verification attempt is separate from the map evaluator’s acceptance condition.Source: §C.2 · p. 16
Loading 2610.13092v1…