Source-linked AI summary
Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents
Chengyang Gu, Le Zhang, Jingbo Zhou, Yize Chen, Yu Shi, Siqi Bao, Zheng-Fan Wu, Hua Wu, Hui Xiong
TL;DR
GUI-agent RL methods often use outcome-only supervision, missing quality differences among trajectories with the same result. LACL-GUI adds length- and divergence-aware preferences within successful and failed trajectories, consistently improving GUI task completion over prior RL-based methods.
Problem
Existing contrastive RLVR methods rely mainly on outcome labels, treating same-outcome GUI trajectories equivalently despite differences in efficiency and proximity to success.
Method
LACL-GUI uses length-aware preferences for successful trajectories and divergence-aware preferences for failed trajectories within contrastive RLVR optimization.
Results
LACL-GUI consistently improves GUI task completion across model scales; on OSWorld, LACL-GUI-8B achieves a 50.0% overall success rate, improving Qwen3-VL-8B-Thinking by 7.3%.
Takeaways & Limitations
Trajectory-level quality information provides more informative optimization signals than outcome-only supervision for GUI-agent contrastive RLVR.
Takeaways & Limitations
The first-order invariance analysis assumes group-wise statistics remain constant through stop-gradient treatment during optimization.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) have shown strong potential for automating tasks across diverse digital environments, where reinforcement learning (RL) has become a dominant training paradigm. However, widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward-gradient misalignment, leading to inefficient and unstable optimization. Recent work addresses this issue by reformulating RL with verifiable rewards (RLVR) as contrastive or classification-based objectives, which improve stability by eliminating problematic gradient behaviors. Despite this progress, existing contrastive RLVR methods rely primarily on outcome-level supervision and fail to capture fine-grained differences in trajectory quality within the same outcome category. In this paper, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive RLVR framework that incorporates trajectory-level quality signals into policy optimization. LACL-GUI introduces structured preferences within both successful and failed trajectories, encouraging concise successful executions and differentiating failure quality based on divergence from successful trajectories, while preserving optimization stability. Experiments on GUI agent benchmarks show that LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods, highlighting the value of trajectory-level supervision in contrastive RLVR.
1 INTRODUCTION
GUI agents benefit from RL-based optimization, but GRPO can misalign reward signals with gradient allocation, while contrastive RLVR methods overlook quality differences among trajectories with the same outcome. LACL-GUI addresses this gap with length-aware contrastive supervision that favors concise successful executions and distinguishes trajectory quality within outcome groups.
- Background: GUI agents powered by MLLMs have demonstrated potential for automating tasks across web, mobile, and desktop platforms, whereas supervised fine-tuning requires substantial synthetic data and may generalize poorly to unseen interfaces.This motivates reinforcement-learning approaches for broader GUI-agent training.
- Problem: GRPO provides computationally efficient policy optimization without a separate critic, but reward signals can become misaligned with gradient allocation.The resulting optimization pressure may not reflect the actual quality indicated by rewards.
- Related work: Prior contrastive or classification-based RLVR objectives improve optimization stability, robustness, and learning effectiveness by replacing scalar reward-weighted optimization with comparison-based objectives.Representative methods include REAL (Zhai et al., 2026), CLIPO (Cui et al., 2026), and Harmony (Yu et al., 2026c).
- Motivation: Existing contrastive RLVR methods mainly use outcome-level labels, treating same-outcome trajectories equivalently despite differences in execution quality, efficiency, and proximity to success.This coarse-grained supervision limits their ability to capture fine-grained trajectory quality.
- Proposed method: LACL-GUI introduces length-aware contrastive RLVR supervision for GUI agents, favoring concise successful trajectories while down-weighting redundant executions.The framework adds structured preferences within successful trajectory groups rather than relying only on outcome labels.
2 RELATED WORKS
Prior GUI agents evolved from brittle structured representations toward visually grounded MLLM-based interaction, while RL and contrastive RLVR methods seek more effective optimization signals. Existing approaches still face scalability limits from sparse outcome rewards, task-specific supervision, or auxiliary models.
- GUI Agents: GUI agents map high-level instructions to grounded actions across web, mobile, and desktop environments, evolving from HTML and DOM representations to visually grounded MLLM agents such as UI-TARS, OS-Atlas, and UGround.Structured approaches depend heavily on representation coverage and quality, which can be brittle and inconsistent across platforms.
- Reinforcement Learning: RL has become a prominent approach for improving LLM- and MLLM-agent reasoning and decision-making in multi-turn environments, following earlier RLHF methods such as DPO (Rafailov et al., 2023).RLHF derives reward signals from offline human preference data.
- Fine-Grained Credit Assignment: GUI-agent credit-assignment methods add milestone-based rewards or pretrained representation models, but require task-specific supervision or additional models that limit scalability across diverse environments.GiGPO (Feng et al., 2026) uses manually designed environment-specific rules, while other approaches estimate intermediate progress with pretrained representation models (Zheng et al., 2026; Wang et al., 2026b).
- Contrastive RLVR: Contrastive learning separates positive and negative samples, motivating RLVR objectives that replace reward-weighted optimization with discriminative or contrastive training under binary outcome rewards (Parthasarathi et al., 2025).REAL (Zhai et al., 2026) casts RLVR as binary classification with BCE loss and contrasts successful with unsuccessful trajectories; CLIPO (Cui et al., 2026) and ConSPO (Zhang et al., 2026) extend this direction.
3 METHODOLOGY
LACL-GUI formulates GUI interaction as sparse-reward sequential decision-making and trains with asynchronous, contrastive trajectory groups. Its objective adds length- and divergence-aware preferences while preserving REAL’s first-order success-versus-failure optimization.
- Problem Formulation: LACL-GUI operates in a partially observable GUI environment, where agents use screenshot observations and finite history to produce reasoning tokens and executable actions until completion or a horizon limit.Task supervision is sparse and trajectory-level: every step shares a binary success or failure outcome, with no intermediate rewards.
- Training Architecture: The asynchronous architecture collects parallel rollouts, filters mixed-outcome groups, reuses successful trajectories, and decouples collection from policy optimization.All-success or all-failure groups are discarded because they provide no discriminative contrastive supervision; the Success Buffer also supports divergence-aware failure analysis.
- Baseline Objective: REAL (Zhai et al., 2026) separates sampled trajectories by binary verifier outcomes, using adaptive logits to promote successes and suppress failures with bounded gradient allocation.The objective gives stronger updates to under-optimized positives and stronger penalties to high-logit negatives, addressing gradient misalignment.
- Length- and Divergence-Aware Objective: LACL-GUI adds within-outcome preferences: shorter successful trajectories receive greater weights, while failures with more erroneous steps receive stronger suppression than near-success failures.These perturbations encourage efficient success and preserve informative failures that diverge later from successful executions.
- Optimization Analysis: With group statistics detached, LACL-GUI preserves REAL’s first-order success-failure objective while reshaping higher-order gradient allocation and local optimization geometry.The perturbations therefore refine optimization without changing the underlying success-failure classification boundary.
4 EXPERIMENTS
Across the 222-task OSWorld subset, LACL-GUI consistently improves GUI-agent success rates over the backbone and RLVR baselines. Ablations, trajectory-length analyses, and threshold sensitivity results support its trajectory-level preference modeling and optimization stability.
- Overall Performance: LACL-GUI-8B achieves a 50.0% overall success rate, improving Qwen3-VL-8B-Thinking by 7.3% and surpassing GRPO (47.2%) and REAL (47.3%).Evaluation uses 222 OSWorld tasks selected to reduce noise from external instability.
- Effect over RL Baselines: LACL-GUI improves over REAL and GRPO by 2.7 and 2.8 percentage points on 8B, and reaches 44.5% versus 43.0% and 42.2% on 4B.The gains hold across both evaluated backbone scales.
- Ablation Study: Adding the LACL-GUI objective produces the largest ablation gain, +2.8%, raising performance from 47.2% to 50.0% after architecture components are introduced.The improvement is attributed to length-aware preference modeling and divergence-aware failure refinement.
- Trajectory Efficiency: On outcome-matched tasks, LACL-GUI reduces successful trajectory length from 9.13 to 8.19 steps and failed trajectory length from 29.87 to 26.72 steps.These correspond to reductions of 10.3% and 10.5%, respectively, using within-group divergence-aware modeling for failures.
- Threshold Sensitivity: Varying the similarity threshold from 0.80 to 0.90 changes overall success rates only marginally, from 49.5% to 50.0% and 49.9%.The stable relative ordering of failed trajectories supports reliable identification of near-successful failures despite domain-level variation.
5 CONCLUSION
LACL-GUI is a length-aware contrastive framework that models fine-grained preferences within successful and failed GUI-agent trajectories. On OSWorld, it consistently improves task completion over existing RL-based baselines, while future work targets adaptive quality estimation and broader environments.
- LACL-GUI models preferences within successful and failed trajectories, refining successful behaviors by length and distinguishing near-successful failures from early-diverging attempts.
- Sensitivity analysis evaluates the divergence-estimation threshold Θ across domains, highlighting the default setting Θ = 0.85.
- LACL-GUI consistently improves GUI task completion performance over existing RL-based baselines on the OSWorld benchmark.
- Future work will investigate more adaptive trajectory-quality estimation strategies and broader interactive environments.
A APPENDIX · A.1 PROOF OF THEOREM 1
The appendix proves that LACL-GUI’s trajectory-length perturbations preserve the original REAL objective to first order. The proof establishes weighted zero-mean identities for both outcome groups, causing linear perturbation terms to vanish while allowing higher-order redistribution of optimization emphasis.
- A.1 PROOF OF THEOREM 1: Lemma 1 establishes a weighted zero-mean property for the proposed logit perturbations in LACL-GUI.The proof defines the positive-group weighted mean trajectory length and derives the negative-group identity analogously using its weighted mean and weights.
- A.1 PROOF OF THEOREM 1: Under stop-gradient treatment of the group-wise statistics µw and µe, Theorem 1 states the first-order invariance of the LACL-GUI objective.The constants are assumed fixed during the analysis.
- A.1 PROOF OF THEOREM 1: For the positive group, a first-order Taylor expansion shows that the perturbation’s linear coefficient vanishes by Lemma 1.Thus, the positive-group loss differs from the corresponding REAL loss only beyond first order.
- A.1 PROOF OF THEOREM 1: Combining both groups yields first-order invariance for the full objective, with the perturbations affecting optimization emphasis only through higher-order behavior.The conclusion applies after separately showing cancellation in the positive and negative losses.
- A.1 PROOF OF THEOREM 1: For the negative group, applying the analogous perturbation and first-order expansion likewise eliminates the linear term through the weighted zero-mean property.The negative-group derivation follows the same algebraic structure as the positive-group proof.
- A.1 PROOF OF THEOREM 1: LACL-GUI preserves the REAL objective up to first order while redistributing optimization emphasis within each outcome group through higher-order terms.This is the theorem’s combined conclusion for the positive and negative groups.
A.2 EXPERIMENT DETAILS · A.2.1 OSWORLD BENCHMARK SETTINGS
The experiments use a filtered subset of OSWorld under controlled virtual-machine conditions, focusing on 222 tasks across eight application domains. The section also reports task-domain distribution and a trajectory-length comparison scope for tasks completed by all three methods.
- A.2 EXPERIMENT DETAILS: Experiments evaluate 222 tasks selected from the 369-task OSWorld benchmark in Baidu’s RLE virtual machine environment.The selection is intended to provide stable training and evaluation conditions.
- A.2.1 OSWORLD BENCHMARK SETTINGS: Chrome and Multi-app tasks are excluded because unstable external connectivity and application dependencies can produce non-reproducible outcomes.The supplied passage explicitly motivates exclusion for stable conditions, but its discussion of Multi-app dependencies is truncated.
- A.2.1 OSWORLD BENCHMARK SETTINGS: After filtering, the remaining 222 tasks span eight application domains.Their distribution is summarized in Table 3.
- A.2.1 OSWORLD BENCHMARK SETTINGS: Table 3 summarizes the distribution of the 222 selected OSWorld tasks across application domains.
- A.2.1 OSWORLD BENCHMARK SETTINGS: Table 4 reports average trajectory lengths on the 70 tasks successfully completed by all three methods.Its last column gives LACL-GUI’s relative change with respect to REAL.
- A.2 EXPERIMENT DETAILS: The trajectory-length comparison is restricted to tasks successfully completed by all three methods.The supplied passage does not provide the trajectory-length values.
A.2.2 EFFECT OF LENGTH-AWARE PREFERENCE MODELING
Length-aware preference modeling shortens both successful and failed trajectories when methods share the same outcome, isolating trajectory preference from success-rate differences. LACL-GUI reduces average lengths relative to REAL, though the effect varies across application domains.
- Analysis setup: The analysis compares trajectory lengths only where Qwen3-VL-8B-Thinking, REAL, and LACL-GUI share the same outcome, isolating preference effects from success-rate differences.Results are reported separately for successful and failed trajectories.
- Successful trajectories: LACL-GUI reduces successful-task trajectories from 9.13 to 8.19 steps versus REAL, a 10.3% reduction across 70 commonly solved tasks.The reduction appears in most domains, but LibreOffice Writer increases slightly and VLC increases clearly.
- Failed trajectories: LACL-GUI reduces failed-task trajectories from 29.87 to 26.72 steps versus REAL, a 10.5% reduction across 93 commonly failed tasks.Lengths decrease in six of eight domains, especially Gimp and OS, but increase in LibreOffice Writer and VS Code.
A.3 REAL-WORLD TRAJECTORIES
Real-world GIMP trajectories show that binary success or failure rewards overlook meaningful quality differences. LACL-GUI addresses this by preferring concise successful executions and modeling failed trajectories according to divergence and post-divergence errors.
- Positive Group: A 6-step LACL-GUI-8B success reaches the goal directly, whereas a 27-step REAL success contains redundant and outdated-state interactions despite the same reward.The concise trajectory follows the optimal path without exploration or recovery; the longer trajectory repeatedly clicks without progress and after the interface changes.
- Positive Group: Length-aware positive preference therefore reinforces concise successful trajectories rather than rewarding eventual completion alone.The additional interactions in the long successful trajectory do not contribute to task completion and are undesirable behaviors to reinforce.
- Negative Group: Two failed trajectories with reward 0 differ qualitatively: one terminates after four steps, while the other continues ineffective exploration after the same divergence point.The first incorrectly assumes completion before verifying the desired state; the second performs additional actions without recovering the correct execution path.
- Negative Group: LACL-GUI models failure severity by locating divergence from a successful reference and estimating erroneous steps afterward, distinguishing premature termination from unnecessary post-divergence actions.In the example, both failures diverge at the same decision point but exhibit different error severities.