Source-linked AI summary
Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents
Jiahe Ying, Wendong Bu, Kaihang Pan, Bingchen Miao, Siyu Chen, Wen Wang, Xueming Jiang, Juncheng Li, Siliang Tang
TL;DR
GUI-agent training is limited by costly annotation, weak action–intent alignment, and discarded failed trajectories. Iron addresses these issues with dual learning, SCC rewards, and hindsight reproduction, and reports stronger cross-platform performance with substantial gains on unseen web tasks.
Problem
GUI-agent training faces costly annotation, inefficient exploration because failed trajectories are discarded, and insufficient fine-grained alignment between low-level actions and underlying intent.
Method
Iron uses dual learning with a stepwise cycle-consistent reward for action–intent alignment and hindsight reproduction to repurpose failed MCTS trajectories as training data.
Results
Iron consistently outperforms the Step-DPO variant and a baseline trained with triple the SFT data across cross-platform tasks, while achieving over 25% relative improvement on unseen web scenarios.
Takeaways & Limitations
Iron provides a model-agnostic, scalable self-training pipeline that learns from successful and failed experiences and expands data volume and task diversity.
Abstract
from arXiv · showhide
Achieving virtual agents capable of automating tasks across diverse digital environments remains a pivotal challenge in Embodied AI. While Multimodal Large Language Models (MLLMs) offer enhanced visual perception and reasoning, their agentic deployment faces three challenges: costly data annotation, imprecise action-intent alignment, and inefficient exploration from discarded failed trajectories. To address these, we introduce Iron, an intent-aligned, self-improved, and annotation-efficient framework for training GUI agents. Iron employs a novel dual learning strategy that utilizes a stepwise cycle-consistent (SCC) reward to achieve fine-grained alignment between low-level actions and high-level intents, thereby improving instruction grounding and intent understanding. Concurrently, Iron introduces a hindsight reproduction mechanism to repurpose failed trajectories for training, improving both learning efficiency and task diversity. Extensive experiments demonstrate that Iron-trained generalist agents consistently improve performance on cross-environment and cross-device tasks, outperforming models trained with three times more data. Iron also achieves a substantial 25.06% relative improvement on unseen web tasks, with further gains observed on inherently complex tasks, demonstrating the feasibility of building more capable virtual agents.
1. Introduction
Iron addresses costly annotation, discarded failed trajectories, and weak action–intent alignment in GUI-agent training through dual learning, SCC rewards, and hindsight reproduction. Experiments report stronger cross-platform performance and gains on unseen and complex tasks.
- Challenges: MLLM-based GUI agents face costly sequential-data annotation, inefficient exploration from discarded failures, and inadequate alignment between low-level actions and underlying intent.Automated labeling is unreliable for complex multimodal reasoning, failures often vastly outnumber successes, and existing agents may rely on mechanical imitation.
- Iron Framework: Iron combines dual learning for instruction grounding and intent understanding with hindsight reproduction, enabling self-training from successful and failed trajectories.Instruction grounding maps an instruction I to executable operations A, while intent understanding infers intent U from A.
- Iron Framework: The stepwise cycle-consistent reward provides per-step self-assessment for fine-grained action–intent alignment without external labeling.The reward exploits the cycle-consistency between instruction grounding and intent understanding.
- Iron Framework: Hindsight reproduction converts failed trajectories into successful trajectories with revised goals and filters converted data using five metrics, increasing data volume and task diversity.The mechanism is implemented within MCTS and uses Repeatability, Logicality, Ineffectiveness, Exploratory, and Invalidity metrics.
- Experiments: 41.96% on OSWorld and 39.15% on AndroidWorld were achieved with Qwen3-VL-8B, alongside over 25% relative improvement in unseen web scenarios.Iron consistently outperformed the Step-DPO variant and a baseline trained with triple the SFT data across the reported cross-platform experiments.
2. Related Works
Prior GUI-agent methods use search and reinforcement-learning data but underuse search-tree information or intermediate values. Iron instead provides step-level rewards aligned with action intent.
- Existing Methods: GUI-agent development remains dependent on heavy human annotation, while recent MCTS-based methods use success and failure data through DPO, RFT, and PPO.These methods target cross-platform GUI operations but retain data and search-utilization limitations.
- Limitations and Gap: Trajectory-level DPO and RFT underuse search-tree information, while step-level DPO ignores intermediate step values and ProgRM lacks multidimensional reward consideration.Recent work adds multidimensional evaluation metrics, but Iron focuses specifically on step-level action–intent alignment.
3. Method
Iron combines dual learning, stepwise cycle-consistent rewards, MCTS-based hindsight reproduction, trajectory filtering, and self-training to improve GUI-agent learning without external labeling. The framework aligns actions with intents while converting exploration outcomes into higher-quality, more diverse training data.
- 3.1. Preliminary and Task Formulation: Iron formulates Instruction Grounding and Intent Understanding as complementary tasks that mutually reinforce action execution and intent comprehension.Instruction Grounding predicts executable actions from instructions, observations, and action history; Intent Understanding infers underlying intent from action sequences and observations.
- 3. Method: The complete pipeline combines imitation-learning cold start, SCC-guided MCTS exploration, hindsight reproduction, filtering, and iterative self-training.Successful trajectories provide positive supervision, while repurposed failures supplement data diversity in a continuing optimization loop.
- 3.2. Stepwise Intent-aligned Dual learning: MCTS supplies diverse trajectories through exploration, while the SCC reward addresses sparse outcome feedback by guiding action selection at individual steps.Candidate child actions receive matching scores between inferred intent and instruction, with higher scores indicating greater success likelihood.
- 3.2. Stepwise Intent-aligned Dual learning: The stepwise cycle-consistent reward provides per-step supervision by scoring semantic alignment between inferred intent and the original instruction.The reward uses the intent-inference function and normalizes semantic reward values with a Sigmoid function to [0, 1].
- 3.3. Hindsight Reproduction: Trajectory filtering uses Repeatability, Ineffectiveness, Exploratory, Logicality, and Invalidity to remove repetitive, ineffective, illogical, or invalid samples.The metrics address feedback loops, ineffective actions, exploratory behavior, human-like logical consistency, and invalid outcomes such as 404 or network errors.
- 3.3. Hindsight Reproduction: Hindsight reproduction relabels failed trajectories with the concrete intent or outcome they achieved, converting them into successful training examples with revised goals.The relabeled trajectories are filtered before being added to the training set, expanding data diversity and task coverage.
4. Experiment
Iron is evaluated as a model-agnostic framework across desktop, mobile, and unseen web environments, using multiple backbones and self-training configurations. Results report competitive cross-platform performance, improved unseen-environment transfer, stronger complex-task behavior, qualitative intent understanding, and SCC reliability.
- Experimental setup: Iron evaluates generalist agents across OSWorld, AndroidWorld, and VisualWebArena using Qwen3-VL-8B, OS-Atlas-Base-7B, and InternVL2.5-4B backbones.The setup is designed to test model-agnostic self-improvement across desktop, mobile, and web environments.
- Cross-platform performance: Iron variants consistently outperform Full models trained with three times more SFT data across evaluated platforms.The comparison also includes Step-DPO and tests whether autonomous data generation and iterative self-training provide informative training data.
- Cross-platform performance: 41.96% on OSWorld and 39.15% on AndroidWorld are achieved by Iron-30k on Qwen3-VL-8B, surpassing the Full model with fewer training samples.Expanded self-training raises Qwen3-VL-8B performance from 38.39% to 41.96% on OSWorld and from 33.47% to 39.15% on AndroidWorld.
- Unseen-environment generalization: 17.38% overall success is reached on unseen VisualWebArena by Qwen3-VL-8B Iron-30k, up from 14.31% and above the Full model.Similar gains are reported for OS-Atlas-7B and InternVL2.5-4B, including an increase from 4.08% to 5.96%.
- Reliability and intent understanding: Iron qualitatively integrates failed and corrective searches into a complete conditional intent, unlike a baseline focused on the final successful action.The case study compares OS-Atlas-7B-Iron-30k with an SFT baseline using the red-shelf and black-shelf search sequence.
- Reliability and intent understanding: SCC Match accuracy rises from 74.5% at 5k steps to 92.5% at 30k steps, while SCC scores increase from 0.45 to 0.84.Intent consistency also improves from 85.1% to 91.6%, and SCC scores correlate with CLIP instruction–intent similarity at Spearman ρ = 0.685 and Pearson r = 0.689.
5. Ablation Studies
Ablations show that Iron’s components contribute distinctly to performance, with trajectory filtering having the largest effect on both benchmarks.
- Removing Dual Learning and SCC Reward causes 5.0% to 7.0% degradation on both OSWorld and AndroidWorld.These components support action–intent alignment and provide dense optimization signals.
- Removing Hindsight Reproduction further reduces performance, highlighting its role in leveraging unsuccessful trajectories for exploration diversity.
- 65.0% performance decreases on OSWorld and 48.2% on AndroidWorld occur without trajectory filtering.The results identify data quality as critical for stable self-improvement and reliable iterative training signals.
6. Conclusion
Iron is a model-agnostic self-training framework that combines intention–action alignment with hindsight reproduction to learn from successful and failed experiences. It improves agents across diverse cross-platform tasks and challenging unseen web scenarios.
- Iron enables autonomous learning from both successful and failed real-world experiences through intention–action alignment and hindsight reproduction.
- Iron achieves considerable performance gains on diverse cross-platform tasks and substantial progress on challenging unseen web scenarios.
- Iron provides a powerful and scalable self-training method for autonomously enhancing agents’ core capabilities.