PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoGLM: Autonomous Foundation Agents for GUIs

Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, Yang Yang, Yifan Xu, Yu Yang, Yujia Wang, Yulin Xu, Zehan Qi, Yuxiao Dong, Jie Tang

arXiv:2411.00820v1cs.HCcs.AIcs.CLcs.LG

TL;DR

AutoGLM addresses the challenge of training deployable foundation agents for autonomous GUI control amid costly expert data and inefficient environmental sampling. It separates planning from grounding and uses self-evolving online curriculum reinforcement learning, achieving strong results across web and Android evaluations.

  • Problem

    Training GUI agents is constrained by costly expert trajectories, weak error recovery from imitation, and inefficient reinforcement-learning environment sampling.

    Source: §2.1 · p. 4

  • Method

    AutoGLM uses an intermediate interface to separate planning and grounding, together with self-evolving online curriculum reinforcement learning and deployment infrastructures.

    Source: §1 · p. 3, §2.2 · p. 5, §2.3 · p. 6

  • Results

    AutoGLM demonstrates strong GUI-control performance across web and Android benchmarks, including 55.2% success on VAB-WebArena-Lite and 36.2% on AndroidLab.

    Source: §1 · p. 3, §4 · p. 8

  • Takeaways & Limitations

    The results and deployments indicate AutoGLM’s potential as a practical foundation-agent system for web browsing and Android control.

    Source: §1 · p. 3, §4 · p. 8

Abstract

from arXiv · show

We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs). While foundation models excel at acquiring human knowledge, they often struggle with decision-making in dynamic real-world environments, limiting their progress toward artificial general intelligence. This limitation underscores the importance of developing foundation agents capable of learning through autonomous environmental interactions by reinforcing existing models. Focusing on Web Browser and Phone as representative GUI scenarios, we have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our approach integrates a comprehensive suite of techniques and infrastructures to create deployable agent systems suitable for user delivery. Through this development, we have derived two key insights: First, the design of an appropriate "intermediate interface" for GUI control is crucial, enabling the separation of planning and grounding behaviors, which require distinct optimization for flexibility and accuracy respectively. Second, we have developed a novel progressive training framework that enables self-evolving online curriculum reinforcement learning for AutoGLM. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains. For web browsing, AutoGLM achieves a 55.2% success rate on VAB-WebArena-Lite (improving to 59.1% with a second attempt) and 96.2% on OpenTable evaluation tasks. In Android device control, AutoGLM attains a 36.2% success rate on AndroidLab (VAB-Mobile) and 89.7% on common tasks in popular Chinese APPs.

1 Introduction

AUTOGLM targets the scarcity of decision-making data for GUI agents by developing deployable agents for web browsing and Android control. Its approach separates planning from grounding and uses self-evolving online curriculum reinforcement learning, achieving strong benchmark and real-world results.

Motivation

  • GUI environments offer parallelizable simulators, rich textual and visual inputs, safer control than embodied environments, and broad practical appeal.However, existing pretraining data contains little decision-making information because internet-scale knowledge is mainly static.Source: §1 · pp. 2–3

Approach

  • AUTOGLM addresses GUI-agent data scarcity with agents for Web Browser and Android built on the ChatGLM model family and infrastructures for user deployment.Progressive deployment is intended to teach effective assistance while helping researchers examine benefits and risks.Source: §1 · p. 3

Intermediate Interface Design

  • An intermediate interface separates planning, which needs flexibility and error recovery, from grounding, which emphasizes action accuracy.This separation enables the two behaviors to be optimized independently.Source: §1 · p. 3

Self-Evolving Online Curriculum RL

  • Self-evolving online curriculum reinforcement learning uses a progressive weak-to-strong schedule to address scarce instructions and trajectories and improve error recovery.The framework targets capabilities that are difficult to acquire through offline training alone.Source: §1 · p. 3

Results

  • 55.2% success rate on VAB-WebArena-Lite, rising to 59.1% with a second attempt, while OpenTable reaches 96.2%.On OpenTable, AUTOGLM surpasses GPT-4o at 62.6% and Agent Q at 81.7%.Source: §1 · p. 3
  • 36.2% success rate on AndroidLab exceeds GPT-4o’s 31.2% and Claude-3.5-Sonnet’s 29.0%, while common Chinese-app tasks reach 89.7%.The system is also implemented through an AccessibilityService Android application.Source: §1 · p. 3

2 AUTOGLM: Techniques and Insights

AutoGLM develops GUI foundation-agent techniques around data-efficient training, an intermediate interface separating planning from grounding, and self-evolving online curriculum RL. These techniques target decision-making, grounding accuracy, error recovery, and policy stability in Web Browser and Android settings.

2.1 Important Techniques

  • AutoGLM combines pre-training, behavior cloning, curriculum learning, reward modeling, and reinforcement learning to address scarce, costly, or inefficient agent-training data.Pre-training leverages weakly supervised decision signals and high-resolution visual inputs; BC supplies expert trajectories, while RL learns from failures but faces simulator and sample-diversity bottlenecks.Source: §2.1 · p. 4

2.2 Insight 1: Intermediate Interface Design

  • Separating planning from grounding through an intermediate interface enables independent optimization for planning flexibility and grounding accuracy.The design represents actions with natural-language element descriptions for the planner, then converts those descriptions into coordinates through a grounder.Source: §2.2 · p. 5
  • Current GUI-agent errors often arise during grounding because incorrect element identification remains more common than planning errors.The paper motivates the interface by observing that existing LLMs and LMMs are generally stronger at planning than grounding on current benchmarks.Source: §2.2 · p. 5
  • The trained grounder improves proprietary LLM/LMM planners, while grounding data can be harvested at scale from automatically constructed environmental observations.Table 1 is cited as experimental support for the usefulness of this separation.Source: §2.2 · p. 5, Table 1 · p. 5

2.3 Insight 2: Self-Evolving Online Curriculum RL

  • WebRL trains in-house planners with self-evolving online curriculum RL to address scarce task instructions and expert trajectories.Failed instructions are mutated into harder or easier tasks, filtered by a critic, and reused in later roll-outs.Source: §2.3 · pp. 5–6
  • AUTOGLM is evaluated on VAB-WebArena-Lite, where the compared methods are trained using GLM-4-9B-Base.Figure 3 identifies the benchmark and common training base; Figure 4 presents success-rate comparisons among agents.Source: Figure 3 · p. 5, Figure 4 · p. 6
  • KL-constrained policy updates and actor-confidence-filtered replay address policy distribution drift during progressive curriculum training.The ablation study reports these designs as indispensable for consistent iterative performance improvement.Source: §2.3 · p. 6

3 Results

AUTOGLM is evaluated across web and Android GUI tasks, including reproducible benchmarks and real-world user-delivery settings. It outperforms compared agents on the reported web and Android benchmarks while also achieving strong human-evaluated performance on Chinese Android apps.

3.1 Evaluated on Web

  • On VAB-WebArena-Lite, a manually verified 165-task subset of WebArena, AUTOGLM significantly advanced, narrowing the performance gap between autonomous agents and humans.Source: §3.1 · p. 6
  • AUTOGLM outperforms GPT-4o and Agent Q on the reconstructed 200-sample OpenTable real-world booking evaluation.The evaluation runs on the real OpenTable website with human evaluation; results are reported in Figure 5.Source: §3.1 · p. 6, Figure 5 · p. 6

3.2 Evaluated on Android

  • 36.2% success rate on AndroidLab is the best performance among all compared agents.AndroidLab is an interactive benchmark supporting reproducible evaluation and reinforcement-learning improvement; BC denotes behavior cloning training.Source: §3.2 · p. 7, Figure 6 · p. 7
  • 89.7% success rate on common tasks across seven Chinese Android apps demonstrates practical performance in a physical-phone user-delivery setting.Success is determined by human evaluation of complete execution trajectories using AccessibilityService applications.Source: §3.2 · p. 7, Figure 7 · p. 7
  • The Chinese-app evaluation distinguishes complete success, partial completion, and failure, showing that unfinished tasks can still assist users.Partial tasks proceed in the correct direction without fulfilling all requirements, whereas failures terminate early, get stuck, or go wrong.Source: §3.2 · pp. 7–8

4 Conclusion

AUTOGLM is a ChatGLM-based foundation-agent series for GUI operation across web browsing and Android environments. Its intermediate interface, self-evolving online curriculum reinforcement learning, benchmark results, and real-world deployments support its potential as a practical GUI agent.

4 Conclusion

  • AUTOGLM combines an intermediate interface that disentangles planning and grounding with self-evolving online curriculum reinforcement learning for robust error recovery and performance improvement.The system demonstrates 55.2% success on VAB-WebArena-Lite and 36.2% on AndroidLab, alongside browser-plugin and Android-application deployments.Source: §4 · p. 8

Related papers

Loading 2411.00820v1…