PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual GUI Agents

Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng

arXiv:2601.20732v4cs.LGcs.CV

TL;DR

Continual GUI Agents address the problem of GUI agents trained on static environments deteriorating under domain and resolution shifts. The paper introduces GUI-AiF, a reinforcement fine-tuning framework with APR-iF and ARR-iF to anchor changing interaction points and regions, and reports stronger continual generalization than baselines. Its scope is limited to a 3B model, three training datasets, three benchmarks, and domain and resolution shifts.

  • Problem

    Static GUI agents struggle to maintain grounding when domains, interaction points, layouts, and resolutions shift over time.

  • Method

    GUI-AiF extends reinforcement fine-tuning with APR-iF for diverse interaction points and ARR-iF for diverse element regions and scales.

  • Results

    GUI-AiF outperforms state-of-the-art baselines and improves forward transfer across subsequent domain and resolution tasks.

  • Takeaways & Limitations

    The work establishes a continual learning framework for GUI agents that continuously anchors interaction points and regions despite evolving GUIs.

  • Takeaways & Limitations

    The study focuses on a 3B model, three training datasets, three benchmarks, and only domain and resolution shifts, excluding broader shifts such as layout, appearance, and language.

Abstract

from arXiv · show

As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.

1. Introduction

Continual GUI Agents address the challenge of grounding under evolving domains and resolutions, where static training can fail to maintain stable interaction locations and regions. GUI-AiF uses two rewards within reinforcement fine-tuning to improve adaptation across these shifts.

  • Motivation: Static GUI training and grounding strategies struggle as new interfaces, layouts, and resolutions arrive over time.Existing agents are trained on fixed UI datasets, while real-world digital environments continually change.
  • Motivation: Domain shifts alter interaction points, while resolution shifts change element scales, making stable grounding difficult.Mobile OS interfaces tend to use more text, whereas Web OS interfaces rely more on icons.
  • GUI-AiF: GUI-AiF optimizes grounding policy rewards within the reinforcement fine-tuning paradigm for continual GUI adaptation.The framework is designed to mitigate grounding bias from static GUI tasks.
  • GUI-AiF: APR-iF rewards diverse interaction locations, while ARR-iF encourages diversity in predicted element regions and robustness to scale variations.Together, the rewards target over-adaptation to static coordinates and element sizes.
  • Continual GUI Agents: Continual GUI Agents introduce continual learning scenarios across shifting UI domains and screen resolutions.Examples include transitions from Mobile OS to Web OS and scaling from 1080p to 4K.
  • Results: GUI-AiF achieves state-of-the-art performance on ScreenSpot-V1, ScreenSpot-V2, and ScreenSpot-Pro, outperforming existing baselines.The contribution statements report results across all three benchmarks.

2. Related Works

Prior GUI-agent work relies on fine-tuning paradigms developed for fixed distributions, while continual GUI settings require adaptation to changing platforms, layouts, and resolutions. GUI-AiF extends reinforcement fine-tuning with rewards that target diverse interaction points and regions.

  • GUI Agents: GUI agents use natural-language instructions to automate digital human-computer interaction across graphical user interfaces.Prior approaches include structured representations, workflow modules, and vision-language fine-tuning.
  • Post-training: Supervised fine-tuning depends on labeled datasets and can struggle to generalize when GUI distributions change.Its cross-entropy objective matches the current task’s label distribution, which can move the model away from previously learned capability.
  • Continual GUI Agents: Current GUI agents overlook changing operating-system platforms and interface resolutions as new GUI data arrives.This motivates the Continual GUI Agents task and GUI-AiF.
  • GUI-AiF: GUI-AiF integrates rewards for predicted interaction locations and element scales into reinforcement fine-tuning.APR-iF targets interaction-point diversity, while ARR-iF targets diversity in element regions.
  • Post-training: Standard reinforcement fine-tuning uses on-policy optimization and KL divergence but targets performance on the current task distribution.It lacks adaptability under drastic shifts in visual layout, interaction domain, and resolution.

3. Method

GUI-AiF extends reinforcement fine-tuning with rewards that encourage diversity in predicted interaction points and element regions, supporting adaptation across sequential domain and resolution shifts.

  • Problem Formation: GUI-AiF defines continual GUI tasks as sequential domain or resolution shifts that require adaptation without over-adapting to any single task.The task sequences are represented as TD for distinct domains and TR for shifting interface resolutions.
  • Anchoring Point Reward in Flux: APR-iF rewards exploration of diverse predicted interaction points rather than clustering around coordinates from a single task.It computes center points from predicted bounding boxes and measures their spatial variance around the centroid.
  • Anchoring Region Reward in Flux: ARR-iF promotes diversity in predicted element regions by modeling bounding boxes as Gaussian distributions and measuring pairwise Bhattacharyya distances.The distributions’ covariance matrices encode predicted region scale and shape, while the total region reward averages separation across prediction pairs.
  • Integration Rewards: The combined reward RAiF weights APR-iF and ARR-iF to balance interaction-location and element-scale generalization.The weights α and γ control exploration intensity over diverse points and regions.
  • Reinforcement Fine-tuning with GUI-AiF: GUI-AiF integrates the combined reward with standard RFT advantages and KL regularization to counter overfitting to current UI layouts.Standard advantages compare predictions with ground truth, while RAiF encourages exploration of shifted locations and scales; the KL term limits drift from a reference policy.

4. Experiments

The experiments evaluate continual GUI learning under domain and resolution shifts across multiple benchmarks, comparing SFT and RFT baselines with GUI-AiF. Results indicate that GUI-AiF improves adaptation, forward transfer, grounding stability, and continual performance, while ablations identify the roles of its rewards and KL regularization.

  • Experimental Setup: The evaluation uses domain sequences across mobile, desktop, and web platforms, plus resolution sequences from normal to high-resolution interfaces.ScreenSpot-V1 and V2 evaluate continual domain adaptation, while ScreenSpot-Pro evaluates continual resolution performance across six software-interface categories.
  • Continual GUI Domain: GUI-AiF further improves continual domain performance over SFT and RFT baselines, including migration from mobile OS to Web OS.The authors report that SFT methods generally perform poorly, RFT baselines show some adaptation, and GUI-AiF further boosts continual performance.
  • Continual GUI Resolution: GUI-AiF outperforms both SFT and RFT baselines after normal-to-high-resolution shifts, where the baselines achieve the same final average accuracy.The authors attribute the challenge to distortion in element scales and locations and link GUI-AiF’s improvement to generalization over both aspects.
  • Reward Ablation: The full combination of APR-iF and ARR-iF performs best, indicating that generalizing interaction locations and element scales are both important for continual GUI agents.The ablation compares the full rewards with using APR-iF or ARR-iF alone on ScreenSpot-V1 and ScreenSpot-V2.
  • Forward Transfer: Forward-transfer results show GUI-AiF uses prior mobile knowledge to improve subsequent desktop and web performance, and supports generalization from normal to high resolution.The reported gains are associated with exploring diverse interaction regions rather than over-adapting to specific layouts.
  • Analysis: Removing the KL term degrades continual performance, while reward analysis shows exploration-oriented rewards can conflict with rewards focused on optimizing the current static task.APR-iF fluctuates more than ARR-iF, and the authors identify an inverse relationship between GUI-AiF rewards and conventional point-plus-coverage rewards.

5. Conclusion

The paper formalizes continual GUI agents under domain and resolution shifts and proposes GUI-AiF with APR-iF and ARR-iF to maintain grounding as GUIs evolve. Experiments show GUI-AiF outperforms state-of-the-art baselines.

  • Continual GUI Agents models GUI interaction under shifting domains and resolutions.
  • GUI-AiF uses APR-iF and ARR-iF to continuously anchor interaction points and regions despite evolving GUIs.
  • GUI-AiF mitigates over-adaptation to static grounding cues in dynamic GUI environments.
  • GUI-AiF outperforms state-of-the-art baselines and establishes the first continual learning framework for GUI agents.

Abstract

The supplementary material reports four additional analyses covering training boundaries, task order, interaction bias, and GUI visualizations.

  • Training all leveraged GUI datasets at once investigates GUI-AiF’s upper boundary.
  • Reversing the sequence of continual domain tasks tests sensitivity to task order.
  • The supplement analyzes bias between text and icon interaction.
  • Additional GUI visualization heatmaps are provided.

A. Upper Boundary of GUI-AiF

The upper-bound analysis compares GUI-AiF with completing all domain tasks and evaluates performance on SSv1 and SSv2.

  • GUI-AiF’s upper boundary is comparable to performance achieved by completing all domain tasks.The analysis evaluates continual domain-task datasets on SSv1 and SSv2.

B. Reversed Continual Domain Tasks

Reversing the continual domain sequence to Web, Desktop, and Mobile preserves the performance tendency reported in the main paper.

  • The Web, Desktop, and Mobile sequence produces continual-learning performance aligned with the main paper’s tendency.

C. Bias between Text and Icon Interaction.

GUI agents perform consistently better on text-based than icon-based interaction tasks across domains and resolutions. The authors attribute this bias to stronger OCR-based text comprehension and greater difficulty interpreting semantically ambiguous icons.

  • Text-based task performance is consistently higher than icon-based performance across domains and resolutions.
  • The authors attribute the bias to the visual branch’s reliance on OCR for text comprehension.
  • Icons remain harder because they are semantically ambiguous and lack direct textual information.

D. Visualizations of Continual GUI Agents

The paper visualizes how GUI agents’ perceived interaction regions change during continual learning across platforms, domains, and resolutions. It also reports upper-bound and reversed-task performance on ScreenSpot-V1 and ScreenSpot-V2.

  • Continual GUI visualizations: Interaction-region heatmaps visualize continual GUI-agent perception changes across web, mobile, desktop, Vivado, and Photoshop interfaces.The visualized instructions include searching for a game, playing a song, adding Vivado sources, finding a Switch console, and selecting Photoshop’s color picker.
  • Performance evaluation: Table 6 reports upper-bound GUI-AiF∗ and reversed-task GUI-AiF† performance on ScreenSpot-V1 and ScreenSpot-V2.Results are organized across Mobile, Desktop, and Web platforms, with main-paper results shown for reference and the best result bolded.
  • Text and icon bias: Figure 7 presents GUI performance bias between text and icon interaction.
Loading 2601.20732v4…