PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

Rui Yang, Qianhui Wu, Zhaoyang Wang, Hanyang Chen, Ke Yang, Hao Cheng, Huaxiu Yao, Baolin Peng, Huan Zhang, Jianfeng Gao, Tong Zhang

arXiv:2602.22190v2cs.LGcs.AIcs.CL

TL;DR

Open-source native GUI agents face scarce action-aligned reasoning data and post-training methods that can disrupt grounding or misread partially verifiable feedback. GUI-Libra addresses these issues with curated data, action-aware SFT, and conservative RL, and reports consistent gains across web and mobile benchmarks. The work supports data-efficient post-training without costly online interaction during training.

  • Problem

    Native GUI agents face scarce high-quality action-aligned reasoning data, reasoning–grounding interference in SFT, and weak offline-to-online predictability under partially verifiable RL rewards.

  • Method

    GUI-Libra releases a curated 81K reasoning dataset, mixes reasoning-then-action with direct-action supervision using action-aware token weighting, and applies KL-regularized conservative RL with success-adaptive negative-gradient scaling.

  • Results

    GUI-Libra consistently improves offline step-wise accuracy and online task completion across AndroidWorld, WebArena-Lite-v2, and Online-Mind2Web.

  • Takeaways & Limitations

    Careful data curation and GUI-specific post-training can make smaller native VLMs competitive with much larger systems using modest open-source data.

  • Takeaways & Limitations

    The study uses a relatively limited amount of existing open-source data and does not systematically study fully online interactive training.

Abstract

from arXiv · show

Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks. This gap stems from two limitations: a shortage of high-quality, action-aligned reasoning data, and the direct adoption of generic post-training pipelines that overlook the unique challenges of GUI agents. We identify two fundamental issues in these pipelines: (i) standard SFT with CoT reasoning often hurts grounding, and (ii) step-wise RLVR-tyle training faces partial verifiability, where multiple actions can be correct but only a single demonstrated action is used for verification. This makes offline step-wise metrics weak predictors of online task success. In this work, we present GUI-Libra, a tailored training recipe that addresses these challenges. First, to mitigate the scarcity of action-aligned reasoning data, we introduce a data construction and filtering pipeline and release a curated 81K GUI reasoning dataset. Second, to reconcile reasoning with grounding, we propose action-aware SFT that mixes reasoning-then-action and direct-action data and reweights tokens to emphasize action and grounding. Third, to stabilize RL under partial verifiability, we identify the overlooked importance of KL regularization in RLVR and show that a KL trust region is critical for improving offline-to-online predictability; we further introduce success-adaptive scaling to downweight unreliable negative gradients. Across diverse web and mobile benchmarks, GUI-Libra consistently improves both step-wise accuracy and end-to-end task completion. Our results suggest that carefully designed post-training and data curation can unlock significantly stronger task-solving capabilities without costly online data collection. We release our dataset, code, and models to facilitate further research on data-efficient post-training for reasoning-capable GUI agents.

1 Introduction

GUI-Libra addresses action-aligned data scarcity and generic post-training failures in native GUI agents through curated reasoning data, action-aware SFT, and conservative RL. Across web and mobile benchmarks, it improves offline step-wise accuracy and online task completion.

  • Native GUI agents remain less effective at long-horizon decision making despite progress in visual grounding and low-level action execution.
  • Existing training pipelines face scarce action-aligned reasoning data and a tension between explicit reasoning and grounding.Datasets often contain weak reasoning or noisy action labels, while chain-of-thought can degrade grounding performance.
  • GUI-Libra consistently improves offline step-wise accuracy and online task completion across AndroidWorld, WebArena-Lite-v2, and Online-Mind2Web.GUI-Libra-4B and GUI-Libra-8B improve their base models by +15.6% and +12.2% on AndroidWorld, and +4.0% and +8.7% on Online-Mind2Web.
  • GUI-Libra combines action-aware SFT with conservative RL to prioritize executable actions, constrain policy drift, and improve offline–online alignment.Its SFT mixes reasoning-then-action and direct-action supervision with token reweighting, while RL uses KL regularization and success-adaptive negative-gradient scaling.
  • 81K GUI reasoning examples are released through a scalable construction and filtering pipeline designed to improve alignment between reasoning traces and executable actions.

2 Related Work

Prior GUI-agent research develops datasets for visual grounding, multi-step interaction, and natural-language reasoning, alongside SFT-based and unified native-model training approaches. GUI-Libra contributes an openly released dataset and post-training recipes for reproducible reasoning-capable GUI agents.

  • GUI datasets span visual perception, screen parsing, and UI-element annotation for grounding-oriented training.
  • Trajectory-based datasets extend supervision beyond single-step grounding by capturing multi-step interactions and evolving UI state.
  • Reasoning-oriented datasets add natural-language rationales for observation interpretation and planning, but their annotations are often short and noisy.
  • Recent GUI models use SFT and unified native architectures to learn grounding, planning, and multi-step navigation end to end.
  • GUI-Libra contributes open-source training recipes and a high-quality released dataset to support reproducible development of reasoning-capable GUI agents.

3 Preliminaries

The paper formulates GUI interaction as a goal-conditioned partially observable decision process and describes SFT and RLVR objectives for predicting executable actions. It emphasizes that GUI rewards can be sparse and step-wise verification can be ambiguous when multiple actions are valid.

  • GUI interaction is modeled as a goal-conditioned POMDP in which a VLM policy maps instructions, histories, and observations to executable actions.States are partially observed through screenshots, and actions include operations such as clicking, typing, and scrolling.
  • Episodes terminate upon task success or a maximum horizon, with sparse success-only rewards commonly assigned at termination.
  • The paper focuses mainly on high-level navigation tasks, while grounding is treated as low-level coordinate prediction from an instruction and current observation.
  • SFT trains on expert trajectories by predicting a reasoning trace followed by an executable action as one target sequence.
  • RLVR samples candidate actions, assigns rule-based rewards for action type, values, and coordinates, and updates the policy with normalized group-relative advantages.
  • GUI step-wise rewards are ambiguous because multiple distinct actions can validly advance the task, motivating attention to KL regularization and offline–online alignment.

4 Reasoning Data Curation for GUI Agents

GUI-Libra constructs a curated GUI reasoning dataset by augmenting diverse web and mobile trajectories with structured reasoning and executable actions, then filtering noisy or mismatched samples. The resulting data supports separate SFT and RL training sets with improved quality and distribution balance.

  • Unified data format: Each sample pairs a screenshot and interaction context with a CoT trace and JSON action containing the action type and execution arguments.The context includes the system prompt, user instruction, and previous actions; arguments can include text values or click coordinates.
  • Data construction: GUI-Libra aggregates web and mobile GUI trajectories, cleans invalid traces, and constructs reasoning-annotated samples with executable structured actions.Initial cleaning removes incomplete, extremely short or long trajectories, and compound actions outside the action space.
  • Reasoning generation: Generated reasoning is improved with GUI-specific prompting that structures observation, reflection, and planning while enforcing action-related format requirements.The pipeline also compares generator models because reasoning quality depends on the chosen generator.
  • Quality filtering: Agreement filtering discards steps whose stochastic action re-prediction accuracy falls below 0.3, removing uncertain or low-quality samples.Agreement is measured using action-type matches and coordinate proximity for Click-like actions across 10 stochastic runs.
  • Quality filtering: Bounding-box verification retains a step only when its original point 2d lies inside the predicted box for the intended action target.This reduces coordinate errors and supplies bounding-box supervision for RL.
  • Dataset statistics: After filtering, the SFT set contains 81K steps from 9K trajectories, while the RL set contains 40K steps balanced against early-step and mobile-domain biases.The SFT distribution is mobile-heavy, with 14.3% of data from the web domain and Click comprising around 60% of steps.

5 GUI-Libra

GUI-Libra studies why generic reasoning and RL training can misalign GUI agents’ reasoning with grounding and online success, then proposes action-aware SFT and conservative RL to address these issues. The framework combines mixed supervision, token reweighting, KL regularization, and success-adaptive negative-gradient scaling to improve offline-to-online reliability.

  • SFT with Long CoT Hurts GUI Grounding: Long CoT responses correlate with degraded grounding accuracy, and long reasoning traces substantially reduce grounding under SFT.The study evaluates response length on ScreenSpot-v2 and compares CoT, no-CoT, and grounding-only SFT variants.
  • Action-Aware Supervised Fine-Tuning: ASFT mixes reasoning-then-action and direct-action supervision while reweighting action and grounding tokens to preserve reasoning without sacrificing execution accuracy.The weighting coefficients can recover standard SFT, CoT-free SFT, or grounding-only SFT as special cases.
  • Reinforcement Learning from Partial Verifiable Rewards: Offline step-wise metrics can poorly predict online success because policies may shift outside offline support and valid actions may differ from the single demonstrated action.The analysis identifies occupancy mismatch and off-demo validity mass as the two governing factors.
  • Reinforcement Learning from Partial Verifiable Rewards: KL regularization improves offline-to-online predictability by constraining state-distribution shift and probability movement away from demonstrated actions.The resulting offline matching score remains a more stable proxy for online task success within the trust region.
  • Success-Adaptive Negative Gradient Scaling: Success-adaptive negative gradient scaling downweights ambiguous negative updates, reducing over-penalization of valid alternatives under partial verification.The scaling becomes less conservative as policy concentration on the demonstrated action increases when κ > 0.
  • Overall Training Framework: GUI-Libra’s two-stage framework combines ASFT with conservative GRPO, improving behaviorally meaningful step-wise metrics and their connection to online task success without costly online training interaction.The framework uses static offline data while targeting gains that transfer to the policy’s own trajectories.

6 Experiments

GUI-Libra consistently improves offline grounding and navigation accuracy as well as online task completion across mobile and web benchmarks. Ablations attribute these gains to action-aware supervision, reasoning-aware training, data filtering, RL, and moderate KL regularization.

  • Offline evaluation: GUI-Libra achieves the best Pass@1 on AndroidControl-v2 and MM-Mind2Web-v2, outperforming similarly sized and several larger systems.GUI-Libra-3B improves over Qwen2.5-VL-3B by +20.9 and +14.8 on AndroidControl-v2 high- and low-level tasks, and by +19.3 on MM-Mind2Web-v2 average Pass@1.
  • Online mobile evaluation: GUI-Libra-4B and GUI-Libra-8B reach 42.6 success on AndroidWorld, surpassing several larger native models and matching strong proprietary systems.GUI-Libra-8B exceeds Qwen3-VL-8B with step-wise summary at 39.1 and is comparable to GPT-4o at 42.6.
  • Online web evaluation: GUI-Libra improves generalization on WebArena-Lite-v2 and live Online-Mind2Web despite limited web training data.GUI-Libra-7B improves WebArena-Lite-v2 average success from 4.9 to 22.6, while GUI-Libra-8B improves Online-Mind2Web overall score from 19.3 to 28.0.
  • Grounding and reasoning: Mixed-data SFT and action-aware weighting improve grounding accuracy in both reasoning and no-reasoning modes, while RL preserves grounding despite longer responses.GUI-Libra-7B reaches 89.3% grounding accuracy with reasoning versus 88.5% without reasoning; GUI-Libra-3B reaches 83.4% versus 83.2%.
  • Ablations: KL regularization improves offline-to-online alignment, with KL-trained models reaching Pearson correlation 0.89 and Spearman correlation 0.83.Without KL regularization, the corresponding correlations are 0.63 and 0.53.
  • Ablations: Filtering, ASFT, and RL each improve navigation, with moderate KL regularization strengthening both offline and online performance.ASFT raises AndroidWorld success from 3.5 to 13.0, and RL with KL = 0.001 raises it further to 25.2; filtering adds +4.5 AndroidControl-High Pass@1 in SFT.

7 Conclusion

GUI-Libra shows that competitive long-horizon navigation can be obtained from existing trajectory corpora through carefully designed data curation and post-training. Its action-aware SFT and conservative RL deliver strong, data-efficient offline and online performance without costly online interaction during training.

  • Conclusion: GUI-Libra combines action-aware SFT with conservative RL to preserve grounding under long reasoning traces and control policy drift under partially verifiable feedback.The framework is built around the curated GUI-Libra-81K dataset.
  • Conclusion: GUI-Libra achieves strong results across diverse mobile and web benchmarks with favorable data and parameter efficiency.The reported training approach does not rely on expensive online interaction during training.
  • Conclusion: The framework makes offline evaluation more reliable and more predictive of online task success, supporting data-efficient learning for interactive GUI agents.This property is presented as important for real-world deployment.

Limitations

GUI-Libra is evaluated primarily in an offline open-source-data setting and does not study fully online interactive training. Scaling to broader trajectories and online RL remains future work.

  • Scope: The study trains on existing open-source datasets and a relatively limited amount of data.The authors identify broader and more diverse open-source trajectories as a promising direction.
  • Unexplored setting: The framework is not extended to fully online, interactive training.The authors leave a systematic study of fully online training for future work because online RL can be expensive, slow, and infrastructure-intensive.
  • Context: The related-work context describes RLVR as relying on verifiable signals rather than learned reward models.This passage motivates verifiable rewards but does not establish a limitation specific to GUI-Libra.
  • Context: Prior VLM-agent research commonly adopts a two-stage SFT-then-RL paradigm for visually grounded decision-making.This provides background for the training setting addressed by the paper.

B Implementation Details

The implementation uses a structured GUI action space, full-parameter SFT or ASFT, and optional online GRPO-based RL. Evaluation combines grounding benchmarks with offline navigation measures.

  • Action Space: Each GUI action is represented as (action type, action target, value, point 2d).The fields encode the operation, target element, auxiliary arguments, and spatial coordinates when grounding is required.
  • SFT: SFT and ASFT fully tune Qwen2.5-VL and Qwen3-VL models from 3B to 8B with learning rate 1 × 10−5 and effective batch size 256.Training uses two epochs on GUI-Libra-81K or one epoch when reasoning and direct-action data are mixed.
  • RL: RL initializes from SFT or ASFT checkpoints and uses GRPO with online rollouts.The default setup uses 300 iterations, learning rate 1 × 10−6, global batch size 128, rollout group size n = 8, and KL regularization.
  • Inference: Inference uses vLLM with temperature 0.0, top-p 1.0, and up to 1024 completion tokens.The available action list can be adjusted to match the deployment environment.
  • Evaluation: Evaluation includes grounding accuracy on ScreenSpot-V2 and ScreenSpot-Pro, where success requires the predicted click coordinate to fall inside the target bounding box.ScreenSpot-V2 contains 1,269 tasks and primarily uses screenshots below 2560×1440 resolution.

C.2 Offline GUI Navigation Benchmarks

Offline GUI navigation is evaluated on multimodal web and Android benchmarks using cached demonstrations, serialized-action correctness, and filtered annotations to address action-label noise.

  • Multimodal-Mind2Web-v2: MM-Mind2Web aligns human demonstration steps with webpage screenshots and HTML/DOM states for offline multi-step web navigation.Its test split spans more than 100 websites, with cached webpages enabling fully offline evaluation.
  • Multimodal-Mind2Web-v2: MM-Mind2Web operation correctness uses exact-match F1 over the serialized action string “ActionType Value”.The value can contain typed text for Write actions or an app/website identifier for OpenApp actions.
  • Configuration: The RL configuration is summarized separately for the experiments.The supplied table caption identifies Table 14 as the unified RL configuration reference.
  • AndroidControl-v2: AndroidControl-v2 pairs step-wise instructions with mobile screenshots and demonstrated actions but contains about 20% action-type or coordinate annotation errors.The evaluation pipeline uses Qwen3-VL-32B-Instruct to filter mismatched samples.

C.3 Online GUI Navigation Benchmarks

Online evaluation measures final-state task completion on AndroidWorld, WebArena-Lite-v2, and Online-Mind2Web, while auxiliary studies examine grounding predictability and reasoning augmentation.

  • AndroidWorld: AndroidWorld evaluates whether agents reach correct final device states across 116 tasks and 20 real-world apps.A task counts as successful only when the agent outputs Terminate and the environment satisfies completion rules.
  • WebArena-Lite-v2: WebArena-Lite-v2 contains 154 locally deployed web tasks and reports averages across four runs because results have high variance.The evaluation setup also supports response actions, multiline answers, and fuzzy judging with GPT-5.
  • Online-Mind2Web: Online-Mind2Web spans 136 live websites and 300 web-agent tasks with a maximum of 30 interaction steps per task.Task assessment uses the WebJudge method backed by either o4-mini or WebJudge-7B.
  • Grounding Study: Grounding offers a single-step verifiable setting because coordinate correctness can be checked against an annotated bounding box.The study trains Qwen3-VL-4B and Qwen3-VL-8B with GRPO on a 40K downsampled GTA1 grounding dataset.
  • Grounding Study: Grounding performance improves steadily before plateauing, with ScreenSpot-V2 testing near-distribution behavior and ScreenSpot-Pro testing higher-resolution distribution shift.Unlike navigation, grounding does not exhibit an offline–online evaluation gap.
  • Reasoning Augmentation: Using identical 30K AGUVIS samples and the same Qwen2.5-VL-3B base model, GPT-4.1-generated reasoning outperforms GPT-4o by +3.7 and o4-mini by +2.1.The coordinate predictor is held fixed as UGround-7B-v1 in this controlled comparison.

D.4 Additional Metrics on Offline Benchmarks

GUI-Libra’s additional offline metrics show gains in both grounding and action-related performance across AndroidControl-v2 and MM-Mind2Web-v2. The results indicate that improvements extend beyond aggregate step accuracy to its underlying components.

  • Evaluation metrics: The evaluation reports grounding and action-type accuracy on AndroidControl-v2, and grounding accuracy and operation F1 on MM-Mind2Web-v2.These decomposed metrics separate grounding from action prediction.
  • AndroidControl-v2: GUI-Libra-8B achieves 76.3 and 95.5 Pass@1 grounding accuracy on AndroidControl-v2 high-level and low-level tasks, respectively.It also outperforms GPT-5 + UGround-7B-v1 and larger open-weight models.
  • MM-Mind2Web-v2: GUI-Libra-8B attains the highest MM-Mind2Web-v2 grounding Pass@1 across all three subsets, averaging 57.8.Qwen2.5-VL-72B is 1.4 points higher at Pass@4, which the authors attribute mainly to web-data imbalance.
  • MM-Mind2Web-v2: GUI-Libra-7B achieves the best MM-Mind2Web-v2 operation Best@1 performance at 85.1, while GUI-Libra-3B remains competitive with 32B–72B baselines.Best@N is the maximum operation F1 over N ∈ {1, 4} samples.
  • Overall findings: The additional metrics show that GUI-Libra improves both step accuracy and the grounding and action components underlying it.This provides a more complete view of the sources of improvement.

E.0.1 When does offline step-wise matching predict online success?

The section explains why offline one-step matching can fail to predict online success in long-horizon GUI navigation. The mismatch arises from state-distribution shift and unobserved valid alternatives under partial verification, while KL regularization improves predictability by constraining policy drift.

  • Predictability conditions: If C(π) is uniformly bounded and η̄π is small or stable across policies, offline matching becomes predictive of online success through an affine lower bound.These are the supported conditions under which the offline metric regains predictive value.
  • Offline-to-online mismatch: Offline matching can be unreliable because it evaluates one demonstrated action on a fixed dataset distribution, whereas online success depends on valid actions across policy-induced states over a long horizon.The theorem characterizes this mismatch using occupancy mismatch C(π) and off-demo validity mass η̄π.
  • Distribution shift and error accumulation: Small errors can compound and shift online trajectories away from offline support, allowing Moff(π) to improve while J(π) stagnates or decreases.This occurs when failures arise in states rarely or never represented in the offline data.
  • Partial verifiability: Offline matching does not determine true step validity unless off-demo validity mass is negligible or approximately invariant across compared policies.The unobserved mass can vary substantially, so matching changes may reflect probability reallocation rather than genuine correctness gains.
  • Illustrative example: A higher offline score can coincide with lower success: in the example, Moff rises from 0.2 to 0.4 while J falls from 0.9 to 0.5.Offline matching credits only the demonstrated valid action and cannot distinguish valid alternatives from invalid actions.
  • Offline overfitting: Offline demo matching can encourage concentration on the demonstrated action, reducing behavioral diversity and recovery strategies needed for interactive agents.In long-horizon navigation, early mistakes can move policies into poorly covered states and increase distribution shift.
  • KL regularization: A KL trust region limits both occupancy drift and movement away from the demonstrated action, making offline improvements more likely to reflect online-success gains.Lemma E.1 links KL regularization to distribution shift, while Lemma E.2 links it to off-demo validity mass.

F Prompt Templates

The prompt templates generate structured, GUI-specific reasoning and executable actions from screenshots, instructions, and interaction history. Separate mobile and web formats specify action fields and include worked examples of the expected reasoning-to-action response.

  • F.1 Prompt for Reasoning Augmentation: The prompt-generation pipeline uses GPT-4.1 and refines GUI-specific instructions through iterative trial-and-error, motivated by differences in reasoning quality across generator models.The guidelines encourage observation, reflection, planning, and action justification.
  • F.1 Prompt for Reasoning Augmentation: Web prompts use the same screenshot-centered reasoning procedure while supplying instruction, interaction history, and reference action fields.The web template likewise requires one structured reasoning-and-answer pair.
  • F.1 Prompt for Reasoning Augmentation: The templates use GUI-specific inputs including instructions, interaction history, reference action descriptions, and reference commands.Referenced elements may be marked by a small red hollow circle in the screenshot, but the prompt says not to use that marker as justification.
  • F.1 Prompt for Reasoning Augmentation: The reasoning-augmentation prompts instruct models to observe the screenshot, reflect on context, and decide the next action with justification.They also impose a structured one-step <think></think> plus <answer></answer> response format.
  • F.1 Prompt for Reasoning Augmentation: Mobile responses encode an action description, action type, target, and value, with action-specific rules for coordinates, text, navigation, waiting, and app launching.The mobile action vocabulary includes Click, Write, LongPress, Scroll, Swipe, NavigateHome, Wait, NavigateBack, OpenAPP, Terminate, and Answer.
  • F.2 SFT Data Example: The example begins with a GUI-agent instruction describing a task, screenshot, and action choices before presenting the human instruction and history.This frames the response as the next executable move in an interaction trajectory.
  • F.2 SFT Data Example: The SFT action schema defines fields for Answer, Click, Write, Swipe, Wait, NavigateHome, NavigateBack, and OpenApp actions.Each action specifies the relevant target, value, coordinates, or execution explanation.
  • F.2 SFT Data Example: The SFT example presents a mobile plant-search task and asks the agent to generate the next move from the screenshot, instruction, and previous actions.The expected reasoning identifies the search bar and selects it so the plant name can be entered.

G Long-Horizon Trajectory Case Studies

The trajectory case studies illustrate GUI-Libra’s behavior on long-horizon mobile and web tasks. GUI-Libra completes the demonstrated tasks, while the mobile base-model comparison shows difficulty sustaining multi-step progress.

  • AndroidWorld Task 18: GUI-Libra successfully deletes School Supplies, Religious, and Flight Tickets in AndroidWorld Task 18 by alternating iterative reasoning with grounded actions.The task is the ExpenseDeleteMultiple scenario in Pro Expense.
  • AndroidWorld Task 18: The Qwen2.5-VL-7B-Instruct base model takes more steps to discover how to delete one item and then fails to remove the second.The comparison highlights difficulty sustaining progress across the multi-step task.
  • WebArena-Lite-v2: GUI-Libra-4B successfully completes a long-horizon WebArena-Lite-v2 GitLab task involving following lahwaacz, Koushik, and Vinta Chen.The case study is presented as evidence of generalization to multi-step web interactions.
  • Trajectory comparisons: Figures 14–17 pair GUI-Libra trajectories with base-model or comparison-model trajectories for the mobile expense-deletion and web GitLab tasks.The figure captions identify the tasks and the compared models.
Loading 2602.22190v2…