PaperlayerView exact revision on arXiv

Source-linked AI summary

LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents

Weiming Li, Helen Paik, Yulei Sui

arXiv:2608.25777v1cs.AI

TL;DR

Long-horizon GUI agents often leave persistent control information implicit in growing trajectories, making each decision depend on reconstructing task stage, evidence, and runtime feedback, especially with local backbones. LocalLSTC organizes this information by temporal scope, linking persistent state to bounded execution commitments and integrating outcomes back into state. With Qwen3.6-27B, it reaches 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena, while ablations support contributions from both control sides.

  • Problem

    Persistent task stage, accumulated evidence, and runtime feedback remain implicit in trajectory histories, requiring reconstruction at each decision and contributing to observable cross-step control failures.

  • Method

    LocalLSTC is a training-free architecture that separates persistent Long-Term Control from bounded Short-Term Execution, with L2S forming commitments and S2L updating persistent state from outcomes.

  • Results

    64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena: LocalLSTC with Qwen3.6-27B achieves the strongest local results in the reported comparison on both benchmarks.

  • Takeaways & Limitations

    The findings identify temporal organization of control information as a distinct architectural dimension for locally deployed GUI agents.

Abstract

from arXiv · show

Modern GUI-agent frameworks achieve strong desktop task performance with frontier API models, yet persistent control information often remains implicit in growing interaction trajectories. At each step, the planner reconstructs the active task stage, accumulated evidence, and runtime feedback before deciding the next action. This dependence becomes more pronounced under weaker local reasoning backbones. Across four representative state-of-the-art frameworks, replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9\% to 37.7\%. Trajectory annotation further identifies at least one control failure in 91.6\% of failed trajectories. To address this problem, we introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintaining persistent cross-step state to guide short-term execution commitments. Long-Term Control maintains the active subgoal, subgoal-aligned evidence, and runtime feedback across interactions, while Short-Term Execution realizes bounded commitments for the current step. Long-to-Short Planning forms each commitment from persistent state, and Short-to-Long Control integrates execution outcomes back into that state for progress assessment, recovery, and termination. With Qwen3.6-27B, LocalLSTC reaches 64.7\% SR-100 on OSWorld and 65.3\% on WindowsAgentArena, outperforming the strongest prior local results on both benchmarks. Ablations further support contributions from mechanisms on both sides of execution. These findings identify temporal organization of control information as a distinct architectural dimension for locally deployed GUI agents.

1 INTRODUCTION

LocalLSTC addresses the implicit cross-step control burden in locally deployed GUI agents by separating persistent control from bounded execution. Under local backbones, control failures and benchmark degradation motivate this temporal organization, which achieves strong results across two desktop benchmarks.

  • Motivation: Persistent task stage, accumulated evidence, and runtime feedback remain implicit in growing interaction histories, requiring reconstruction before each decision.Reconstruction errors can manifest as lost stage commitment, incorrect progress assessment, repeated actions, mismatched recovery, or premature termination.
  • Motivation: 60.9% to 37.7%: replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 across four representative GUI-agent frameworks.Among failed local-backbone trajectories, 91.6% contain at least one observable control failure.
  • LocalLSTC: LocalLSTC separates Long-Term Control, which preserves cross-step state, from Short-Term Execution, which represents a bounded commitment for the current step.The architecture is training-free and targets locally deployed GUI agents.
  • LocalLSTC: Long-to-Short Planning forms a subgoal-conditioned execution commitment from persistent state and current observation, while Short-to-Long Control updates state from execution outcomes.The loop supports subsequent progress assessment, recovery, and termination without recovering all control information from the full trajectory.
  • Results: 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena: LocalLSTC with Qwen3.6-27B achieves the strongest literature-reported local results in the comparison.With Qwen3.5-9B fixed, removing L2S or S2L reduces OSWorld SR-100 from 49.1% to 36.4% and 41.5%, respectively.

2 RELATED WORK

Related GUI-agent work improves capability through model adaptation, system composition, execution tools, and additional inference-time computation. Sequential-agent approaches also organize long-horizon decisions with search, subgoals, explicit execution states, and revisable plans.

  • Parameter adaptation: GUI-specific training adapts grounding, reasoning, and action-generation capabilities through specialized data, pre-training, supervised fine-tuning, or reinforcement fine-tuning.Examples include UI-TARS, Aguvis, OS-ATLAS, and AgentCPM-GUI.
  • System composition: System-level frameworks strengthen a general backbone through hierarchical planning, grounding, execution tools, candidate generation, rollout selection, and judge-based ranking.Agent S and Agent S2 use an Agent-Computer Interface and generalist–specialist coordination, while Agent S3 and GTA1 increase test-time computation.
  • Sequential control: Sequential agents structure long-horizon decisions through search, online subgoal planning, finite-state execution verification, recovery, and revisable plans.The cited approaches include Language Agent Tree Search, the Subgoal-driven Framework, Agent-SAMA, and AgentProg.

3 METHOD

LocalLSTC separates persistent cross-step control from bounded step-local execution, connecting them through planning and outcome integration. Its control cycle preserves state, validates commitments, abstracts outcomes, routes transitions, and verifies completion.

  • State Representation: LocalLSTC externalizes the active task stage, accumulated evidence, and runtime feedback into persistent control state rather than reconstructing them from the full trajectory.
  • State Representation: The Short-Term Execution Commitment contains a current execution subgoal and ordered action list, and expires after outcomes update the next control state.
  • Long-to-Short Planning: Long-to-Short Planning forms commitments from persistent state and current observations, while Short-to-Long Control incorporates execution outcomes into subsequent persistent state.
  • Short-to-Long Control: The seven-step cycle validates and executes ordered GUI, Bash, or software API actions, then abstracts outcomes into evidence for state updates.
  • Long-to-Short Planning: Proposal comparison occurs within one planner inference, with all candidates conditioned on the same persistent control state.
  • Short-to-Long Control: Routing distinguishes continue, advance, error, stall, and finish states, with final verification determining success from accumulated evidence.

4 EXPERIMENTS

The experiments measure backbone sensitivity and cross-step control failures, then evaluate LocalLSTC across benchmarks, backbones, runtime events, and ablations. Results show strong local performance and contributions from persistent-state mechanisms on both sides of execution.

  • Backbone sensitivity: Across four frameworks, replacing GPT-5 with Qwen3.5-9B lowers average OSWorld SR-100 from 60.9% to 37.7%.The decrease ranges from 21.1–25.0 points across frameworks.
  • Control failures: 91.6% of failed local-backbone trajectories contain at least one cross-step control failure, with RECOVER, STOP, and LOOP most frequent.These categories occur in 64.2%, 41.1%, and 34.4% of failed trajectories, respectively.
  • Control failures: Human verification supports annotation reliability, with 86.5% agreement on ANY labels and Cohen’s κ = 0.730.GPT-5.6-sol achieves ANY F1 scores of 86.1% and 85.2% against the two annotators.
  • Control-failure outcomes: LocalLSTC reduces OSWorld ANY control-failure incidence from 59.4% to 43.1%, a 16.3-point reduction.The task-level bootstrap 95% CI is [8.3, 24.2].
  • Benchmark performance: 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena are achieved by LocalLSTC with Qwen3.6-27B, the strongest local results in the comparison.On OSWorld, the same configuration also reaches 62.7% SR-50 and 51.4% SR-15; on WindowsAgentArena, it reaches 63.4% and 48.5%.
  • Ablations: Removing L2S or S2L lowers OSWorld SR-100 from 49.1% to 36.4% or 41.5%, respectively.The largest individual mechanism effect comes from removing SA, which reduces SR-100 by 17.5 points.
  • Runtime-event analysis: Runtime-event rates remain similar across variants, but Full achieves higher conditional evaluator scores after no-progress events.Stall or repetition incidence is 4.5%, 4.3%, and 4.6%, while conditional scores are 34.3%, 18.8%, and 11.8% for Full, w/o L2S, and w/o S2L.
  • Runtime-event analysis: After rejected termination, Full reaches a 38.9% evaluator score versus 23.1% without L2S.The analysis attributes this difference to L2S supplying stage and evidence context for the next commitment.

5 CONCLUSION

The conclusion presents LocalLSTC as temporal separation between persistent Long-Term Control and bounded Short-Term Execution. It reports benchmark gains, ablation support for both execution-boundary mechanisms, and temporal control organization as an architectural dimension for local GUI agents.

  • Conclusion: LocalLSTC separates persistent Long-Term Control from bounded Short-Term Execution to address cross-step control burden.Long-Term Control preserves state, while Short-Term Execution handles the immediate commitment.
  • Conclusion: L2S forms commitments from persistent state and current observations, while S2L incorporates outcomes into state for planning, recovery, and termination.The two mechanisms connect persistent control with step-local execution.
  • Conclusion: 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena are reported as the strongest local results on both benchmarks.The conclusion also states that ablations confirm contributions from both sides of the execution boundary.
  • Conclusion: The paper identifies temporal control organization as an architectural dimension for locally deployed GUI agents.This conclusion is stated alongside improvements in task success and reduced control failures.
  • Artifacts and disclosure: Implementation details, prompts, evaluation artifacts, and task-level outputs are described or released through an anonymous code repository.The text also records GPT-5.6-sol annotation and AI-assisted editing reviewed by the authors.

A CONTROL-FAILURE ANNOTATION PROTOCOL

The annotation protocol defines unresolved cross-step control failures at trajectory termination and evaluates multiple labels independently. Human verification and GPT-based annotations are compared on 185 blinded trajectories using agreement statistics.

  • Objective and evidence: The protocol assesses unresolved failures remaining at trajectory termination using task instructions, execution logs, and evaluator evidence.When evaluator evidence conflicts with the execution record, the protocol follows the execution record.
  • Label definitions: Five labels capture stage maintenance, progress assessment, repetition, recovery, and termination failures: STAGE, ASSESS, LOOP, RECOVER, and STOP.The labels are evaluated independently, and each applicable label is recorded at most once per trajectory.
  • Decision procedure: ANY is assigned when at least one defined control-failure label applies; OTHER marks another unresolved failure, and NONE marks no unresolved failure.Multiple control-failure labels may be assigned when a trajectory independently satisfies multiple definitions.
  • Human verification: Table 5 reports per-label prevalence and agreement, F1, and Cohen’s κ for two independent human annotators on 185 trajectories.The table organizes reliability by label rather than only by aggregate failure status.
  • GPT comparison: Table 6 reports precision, recall, F1, and Cohen’s κ for GPT-5.6-sol against each human annotator on the same 185 trajectories.Each annotator is used independently as the reference.

A.4 HUMAN VERIFICATION

Human verification evaluates annotation consistency on 185 shared OSWorld trajectories, with anonymized independent labeling and comparison against GPT-5.6-sol annotations.

  • 185 trajectories from 37 shared OSWorld tasks were independently annotated by two PhD-level AI researchers after method identities were removed.The sample included 10 multi-application tasks and 27 tasks from nine single-domain strata.
  • 86.5% agreement, 87.0% F1, and Cohen’s κ = 0.730 were achieved for the ANY label between human annotators.Across the seven original labels, Cohen’s κ ranged from 0.301 for OTHER to 0.792 for LOOP, with a macro average of 0.624.
  • GPT-5.6-sol achieved 86.1% F1 with κ = 0.686 against Annotator 1 and 85.2% F1 with κ = 0.647 against Annotator 2 for ANY.The corresponding human–human comparison was 87.0% F1 with κ = 0.730.

A.5 COMPLETE ANNOTATION RESULTS

Complete annotation results report task-level control-failure incidence across all 369 OSWorld tasks under a fixed Qwen3.5-9B backbone, with LocalLSTC showing the most favorable ANY and NONE rates.

  • 43.1% ANY incidence was the lowest among methods, while 42.3% NONE incidence was the highest for LocalLSTC.The baseline average ANY incidence was 59.4%.
  • Table 7 reports the percentage of all 369 tasks assigned each control-failure label under the fixed Qwen3.5-9B planner backbone.Bold formatting marks the highest NONE incidence and lowest ANY incidence.

B GROUPED L2S–S2L ABLATION CONFIGURATIONS

Grouped ablations isolate the effects of Long-to-Short Planning and Short-to-Long Control under the fixed Qwen3.5-9B backbone.

  • Removing L2S eliminates subgoal-conditioned commitment formation and uses task-level single-action planning while retaining task-level post-execution control and Step Abstraction.
  • Removing S2L retains the L2S commitment interface but replaces evidence-conditioned post-execution state updates with planner-intent updates.

C WINDOWSAGENTARENA RUNTIME DIAGNOSTICS

WindowsAgentArena runtime diagnostics quantify LocalLSTC’s planner activity, subgoal usage, context refinement, and final-verification behavior across 154 tasks.

  • Across all 154 tasks, LocalLSTC averaged 30.5 planner steps, 8.7 subgoals, and 2.8 context refinements per task.
  • 51.5% of tasks rejected by Final Verification subsequently succeeded after returning to execution.Final Verification rejected termination on 21.4% of tasks, followed by an average of 5.2 additional planner steps.

D IMPLEMENTATION DETAILS

LocalLSTC implements execution as a validated planner–runtime loop that constructs bounded commitments from persistent state, executes them across channels, abstracts outcomes into evidence, and updates control state.

  • Each planner call constructs a Short-Term Execution Commitment from the current observation and Long-Term Control State.
  • Planner context combines task and visual state, active subgoal, compact evidence history, runtime feedback, action channels, and operational guidance.Context adapts to GUI and bash execution modes, emphasizing visual/API information or file-system and textual evidence respectively.
  • The runtime selects the next execution mode from task cues, application priors, and the channels used during execution.GUI and software API actions select GUI, while Bash/Python actions select bash execution; the final channel determines the mode for multi-channel commitments.
  • Planner proposals contain thought, subgoal, and an ordered nonempty actions list, with continue retaining the active subgoal and other values proposing transitions.Reserved WAIT, TERMINATE, and INFEASIBLE actions encode waiting, verification requests, and infeasibility judgments, with terminal placement validated.
  • Planner outputs undergo list-level, channel-level, and schema-level validation before execution, with parsing or validation errors returned for up to three repair attempts.Canonicalization also removes bash wrappers, converts GUI scripts to executable pyautogui code, and restricts software API actions to valid callable expressions.
  • Step Abstraction converts execution outcomes into subgoal-aligned evidence containing actions, results, progress, exceptions, and environment changes.Visual changes use screenshot differencing and changed-region extraction, while bash and API records preserve status, outputs, return values, and task-relevant identifiers.
  • Transient routing states classify outcomes as finish, error, stall, advance, or continue, while three consecutive no-change steps under one subgoal produce stall.Bash steps are excluded from visual no-change aggregation, and three consecutive stalls reach runtime failure handling.

F.2 STAGE-PRESERVING RECOVERY ACROSS EXECUTION CHANNELS

The recovery examples show LocalLSTC preserving completed evidence and active subgoals while switching execution channels to correct failures and verify final task state.

  • Bash download plus GUI export produced incorrect filenames, but readback retained the valid 115 KB file and normalized its name without restarting completed stages.
  • A Thunderbird export mismatch was corrected by retaining completed export evidence, renaming the file with Bash, and completing conversion in LibreOffice Calc.
  • After numeric-type and file-state errors, a corrected script completed the spreadsheet edit, while rejected termination triggered dialog recovery and visual verification.
  • For image insertion in Writer, repeated rejected termination requests preserved the insertion subgoal until subsequent interaction exposed the required visual evidence.
Loading 2608.25777v1…