PaperlayerView exact revision on arXiv

Source-linked AI summary

MintAct: A Unified Visual Agent for Digital Environments

Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan

arXiv:2609.22083v1cs.CV

TL;DR

Visual-agent capabilities are fragmented across specialists, which are costly to serve and scale at compact sizes. MintAct unifies grounding, navigation, and visual tool use through staged training and scalable asynchronous RL infrastructure. It matches or exceeds size-matched specialists across these capabilities, with MintAct-8B scoring 48.9 on OSWorld-Verified.

  • Problem

    Visual-agent capabilities are developed and benchmarked in separate specialist models, which are costly to serve and scale, especially at compact sizes.

    Source: §1 · p. 1

  • Method

    MintAct unifies UI grounding, cross-platform navigation, and visual tool use through staged fine-tuning, per-domain RL specialist distillation, and joint asynchronous RL.

    Source: §1 · pp. 1–2

  • Results

    MintAct matches or exceeds size-matched specialists across its capabilities, and MintAct-8B scores 48.9 on OSWorld-Verified.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work provides a foundation for further scaling unified visual agents through its environments, infrastructure, and training recipe.

    Source: §5 · p. 17

    Joint RL across all environments remains future work because it requires sustaining many heterogeneous environments concurrently throughout training.

    Source: §6 · p. 17

Abstract

from arXiv · show

We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.

1 Introduction

MintAct addresses the fragmentation of visual-agent capabilities across specialist models, which are costly to serve and scale, especially at compact sizes. It unifies grounding, cross-platform navigation, and visual tool use, and reports matching or exceeding size-matched specialists across those capabilities.

1 Introduction

  • Separate domain specialists are costly to serve and scale, particularly at compact sizes suited to on-device use.Source: §1 · p. 1
  • Unification requires addressing differences in observation and action spaces, data, and environments, as naive action-set merging or data mixing can erode domain quality.Source: §1 · p. 1
  • MintAct combines per-domain RL specialists distilled into one model with joint agentic RL, supported by infrastructure for hundreds of concurrent instances and both online rollouts and offline trajectory collection.Source: §1 · p. 2
  • MintAct models match or exceed size-matched specialists across grounding, mobile, desktop, and web navigation, and visual tool use.The reported comparison holds across all three model scales.Source: §1 · p. 2
  • MintAct unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use at 2B, 4B, and 8B scales.Source: §1 · pp. 1–2

2 Related Work

Visual agents span GUI grounding and navigation as well as tool use, but these capabilities have largely been developed in separate settings. Prior asynchronous agent systems address variable rollout latency, while MintAct extends asynchronous RL to a unified, multidomain visual-agent setting.

2.1 Visual Agents

  • Most existing GUI methods specialize by platform or interaction setting, while MintAct unifies grounding and navigation across mobile, desktop, and web.Source: §2.1 · p. 3
  • Visual tool-use work ranges from image-centric operations to hybrid GUI agents; MintAct also supports multi-turn visual grounding into actions in stateful, multidomain environments.Source: §2.1 · p. 3

2.2 Asynchronous RL for Agents

  • Asynchronous RL overlaps rollout generation and policy optimization to reduce waiting on variable, expensive interactions, with prior approaches using synchronization, staleness control, or load balancing.Source: §2.2 · p. 3
  • Earlier asynchronous visual-agent systems largely target one domain or modality; MintAct handles heterogeneous backends under an explicitly controlled training distribution.Source: §2.2 · p. 4

3 A Unified Visual Agent

MintAct represents visual-agent tasks with a shared policy conditioned on prompts, observations, and interaction history. Its design combines shared screenshot grounding, domain-specific action prompts, balanced training, and staged supervised and reinforcement learning.

3.1 Problem Formulation and Overview

  • The policy conditions on the system prompt, instruction, observations, and prior reasoning and actions, then produces a reasoning trace and next action.Source: §3.1 · p. 4
  • Grounding maps an instruction to a screen location, navigation sequences interface actions across screens, and visual tool use can invoke external tools over multiple steps.Source: §3.1 · p. 4
  • UI actions use raw screenshots and normalized pixel coordinates, while domain-specific prompts select among distinct action sets instead of merging them.Source: §3.1 · p. 4, Table 1 · p. 5
  • Training proceeds from high-resolution single-step grounding SFT to low-resolution multi-step SFT, per-domain RL specialists distilled through RFT, and joint agentic RL.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Environment and Data Pipeline

MintAct's environment and data pipeline supports UI interaction and visual tool use through reusable environments, curated trajectories, and online RL tasks. The domains use different collection and curation procedures, including deterministic web replay and state-checked tool-use trajectories.

3.2 Environment and Data Pipeline

  • The pipeline combines AndroidWorld, OSWorld, Weblica, and MM-ToolSandBox with synthetic mobile and desktop environments; agents learn from teacher rollouts or direct reward-based interaction.Source: §3.2 · p. 5

3.2.1 Desktop

  • Desktop instances run in Docker containers on a decoupled HTTP infrastructure that can host over 200 concurrent environments; iterative task expansion and rollout filtering build SFT data.Source: §3.2.1 · p. 6
  • Desktop RL tasks are selected from the SFT pool by favoring tasks with mixed successes and failures near a 50% success rate.Source: §3.2.1 · p. 6

3.2.2 Mobile

  • Mobile uses AndroidWorld with over 100 concurrent RL instances; its SFT rollouts receive an additional thinking-trace labeling pass, and roughly 3k tasks are curated for RL.Source: §3.2.2 · p. 6

3.2.3 Web

  • Web environments combine locally served recorded sessions for deterministic offline replay with synthesized interactive websites spanning varied domains and interactions.Source: §3.2.3 · pp. 6–7
  • Web SFT retains about 51.7k verified successful trajectories, while synthesized environments provide a 10k-task RL suite.Source: §3.2.3 · p. 7

3.2.4 Visual Tool Use (VTU)

  • MM-ToolSandBox supports multi-turn visual interactions and more than 500 tools; search and coding tools help navigate its hierarchy, while trajectory filtering checks rubric success and environment state.Source: §3.2.4 · pp. 7–8
  • Visual tool-use scenarios vary information flow and image timing, undergo quality filtering, and yield roughly 800 RL scenarios selected for intermediate success and balanced application coverage.Source: §3.2.4 · pp. 7–8

3.3 Supervised Fine-Tuning

MintAct uses two supervised fine-tuning stages to build a unified warm-up policy, then rejection-sampling fine-tuning to distill domain specialists back into one model. The stages progress from high-resolution grounding and single-step navigation to balanced multi-step navigation and visual tool use.

3.3 Supervised Fine-Tuning

  • The two SFT warm-up stages combine static grounding and single-step datasets with multi-step trajectories from per-domain environments.Both stages optimize the autoregressive likelihood of the target reasoning trace and action; the second uses lower resolution for long histories.Source: §3.3 · p. 8

3.3.1 High-Resolution Single-step SFT

  • High-resolution single-step SFT trains grounding and navigation on grounding and GUI datasets, generating reasoning followed by coordinates for grounding and a next action for navigation.The stage caps each image at 2,116,800 pixels and uses a textual summary of past actions for navigation.Source: §3.3.1 · p. 8

3.3.2 Low-Resolution Multi-step SFT

  • Low-resolution multi-step SFT trains on four balanced domains—mobile, desktop, web, and visual tool use—with each example conditioned on its domain prompt and accumulated interaction history.The mixture is set to 25% per domain; trajectories are capped at 30 turns for mobile, desktop, and web, and 100 turns for tool use.Source: §3.3.2 · p. 9
  • For visual tool-use SFT, tool-registry-aware segmentation splits trajectories when available tools change, retaining full history while masking loss so each assistant action is supervised once.Each segment uses the updated system prompt matching the active tool set.Source: §3.3.2 · p. 9

3.3.3 RFT with Rejection Sampling

  • The resulting MintAct-SFT model already performs grounding, navigation, and visual tool use across domains and initializes the subsequent RL and RFT stages.For tool-use rejection sampling, specialists make four passes per task; mobile, desktop, and web use one pass each.Source: §3.3.2 · p. 9, §3.3.3 · p. 9
  • Rejection-sampling fine-tuning distills successful trajectories from per-domain RL specialists into one model, further improving performance across domains while retaining a single set of weights.A VLM judge filters specialist-generated trajectories before the MintAct-SFT model is fine-tuned on them.Source: §3.3.3 · p. 9

3.4 Agentic Reinforcement Learning

Agentic RL addresses slow, uneven environment interactions, long multimodal trajectories, and the need to control mixed-domain data. MintAct combines asynchronous infrastructure with trajectory and optimization strategies to manage those challenges, including domain composition in joint mobile–desktop RL.

3.4 Agentic Reinforcement Learning

  • Slow and variable GUI interactions, long multimodal histories, and mixed-domain sampling requirements make synchronous RL inefficient and asynchronous data mixtures difficult to control.Naive asynchronous collection can let faster environments dominate the training data.Source: §3.4 · p. 10

3.4.1 Agentic RL infrastructure

  • The asynchronous framework buffers rollout groups between environment-interacting rollouters and trainers, while periodic weight synchronization and bounded queues limit sample staleness.The infrastructure supports hundreds of concurrent interactions across heterogeneous environments and independently configurable rollout and training workers.Source: §3.4.1 · pp. 10–12, Figure 3 · p. 11
  • Environment pools assign each rollout an exclusive backend instance, while multimodal offloading stores historical images on disk and retains references in the queue.The trainer loads images on demand after dequeuing a trajectory group.Source: §3.4.1 · p. 11

3.4.2 RL Training Strategies

  • UI-domain RL preserves the full multi-turn interaction context; visual tool-use RL segments trajectories when tool schemas change and trains on the longest fork per trajectory.Tool-use segments retain full history while adapting the serialized schemas to the active tool registry.Source: §3.4.2 · p. 12
  • Dynamic group filtering uses a VLM judge to remove groups whose rollouts are all successful or all failed, collecting informative groups until a full mini-batch is formed.This online filtering adapts to evolving success rates and avoids updates on uniform-reward groups.Source: §3.4.2 · p. 12
  • The RL objective uses group-baseline advantages without standard-deviation normalization and combines asymmetric clipping with truncated importance weighting to address policy staleness and train–inference mismatch.Environment-failure trajectories are masked; the truncated weight has an upper bound but no lower bound, and the mask-independent normalizer means masking removes a token’s contribution rather than redistributing its weight.Source: §3.4.2 · pp. 12–13

3.4.3 Joint RL Across Heterogeneous Domains

  • Joint mobile–desktop RL controls domain composition at consumption with per-domain quotas and quota-normalized producer backpressure, rather than letting the faster domain dominate.After a bounded wait, the per-domain caps are relaxed so productive domains can backfill the batch.Source: §3.4.3 · pp. 13–14

4 Experiments

MintAct is evaluated across grounding, navigation, and visual tool-use benchmarks at multiple model scales. Results show that its unified models remain competitive with specialist models, while ablations test the training stages and synthetic environments.

4.1 Setup

  • The evaluation covers grounding, mobile, desktop, and web navigation, and visual tool use, using each benchmark’s official metric and limits of 30 navigation steps and 100 tool-use steps.Grounding includes MMBench-GUI, ScreenSpot-v2, UI-Vision, and OSWorld-G; navigation includes AndroidControl, AndroidWorld, OSWorld-Verified, Weblica, and Online-Mind2Web.Source: §4.1 · p. 14

4.2 Main Results

  • 48.9 on OSWorld-Verified; across three scales, MintAct matches or beats size-matched specialists, while MintAct-8B leads public models on OSWorld-Verified, Weblica, UI-Vision, and OS-World-G.Table 3 compares model scales and marks the best and second-best scores.Source: Table 3 · p. 14, §4.2 · p. 14

4.3 Ablation

  • A joint SFT model matches or exceeds domain SFT specialists on mobile, desktop, and tool use, while remaining in a similar range on grounding and web.Each specialist was trained only on its own domain’s data; the joint model covers all domains.Source: Table 4 · p. 15, §4.3 · p. 15
  • Stage 1 alone favors grounding but is weak on interaction; Stage 2 recovers navigation and tool use but degrades grounding, while combining both retains strong grounding and navigation.Stage 1 is high-resolution, single-step SFT; Stage 2 is low-resolution, multi-step SFT.Source: §4.3 · p. 15, Table 5 · p. 15
  • Distilling per-domain RL specialists into MintAct-RFT improves navigation and tool use while keeping one model; joint RL then improves most online benchmarks with grounding essentially unchanged.At 8B, RFT reaches 68.1 on AndroidWorld versus 67.8 for the mobile RL specialist; joint RL raises OSWorld-Verified from 43.1 to 48.9.Source: §4.3 · pp. 15–16, Table 6 · p. 16
  • Joint RL across mobile and desktop exceeds single-domain RL specialists on average for grounding and navigation, transfers to web without web data, and leaves visual tool use at the SFT level.Weblica rises from 57.5 to 60.8 and Om2W from 30.0 to 32.8.Source: §4.3 · p. 15, Table 7 · p. 16
  • Synthetic RL improves synthetic mobile and desktop benchmarks and can transfer to real AndroidWorld, OSWorld, and iOSWorld settings without real-device interaction data.The reported real-setting gains include AndroidWorld from 47.6 to 60.3, OSWorld-Verified from 33.9 to 38.6, and iOSWorld from 2.3 to 18.8 with synthetic environments.Source: §4.3 · pp. 16–17
  • Synthetic environments lose their advantage when real in-domain data is available; real in-domain RL yields larger gains on AndroidWorld and OSWorld than synthetic RL alone.AndroidWorld is 66.7 versus 59.8, and OSWorld is 48.3 versus 39.4.Source: §4.3 · p. 17

5 Conclusion

MintAct unifies grounding, multi-step navigation across mobile, desktop, and web, and visual tool use in one model. Across scales, it matches or exceeds size-matched specialists, supported by scalable RL infrastructure and a multi-stage training recipe.

5 Conclusion

  • MintAct models match or exceed size-matched per-domain specialists while unifying grounding, mobile, desktop, and web navigation, and visual tool use.The models are trained at 2B, 4B, and 8B scales.Source: §5 · p. 17
  • The RL infrastructure hosts hundreds of concurrent instances, controls the cross-domain training distribution, and remains stable under noisy feedback and off-policy drift.It serves both online RL and offline supervised fine-tuning trajectory collection.Source: §5 · p. 17
  • A multi-stage recipe warm-starts the model, sharpens it with per-domain RL and rejection-sampling distillation, then optimizes it across domains with joint RL.The paper also identifies synthetic environments as a complementary, low-cost data source where real interaction data is scarce.Source: §5 · p. 17

6 Limitation

The paper identifies limits in the scope and resource demands of joint RL, long-horizon context handling, and integration of navigation with tool use. It also flags prospective on-device/server collaboration and approximations in dynamic function calling.

6 Limitation

  • Joint RL is beneficial across mobile and desktop, but extending it to web and visual tool use is resource-intensive and left for future work.The extension requires sustaining many heterogeneous environments concurrently throughout training.Source: §6 · p. 17
  • Appending every screenshot during training makes context grow rapidly and memory use large as trajectories lengthen.Summarizing or pruning stale observations is proposed as a future direction for longer-horizon tasks.Source: §6 · p. 17
  • Visual tool use is largely separate from navigation; seamless, on-demand switching between pixel-level interaction and tool calling remains future work.The envisioned policy would invoke tools when they are more effective than direct interaction.Source: §6 · p. 18
  • On-device/server collaboration is proposed rather than evaluated: a compact local model could handle simple or privacy-sensitive tasks and hand off harder ones.The proposed arrangement could combine local latency and privacy benefits with larger-model capability.Source: §6 · p. 18
  • Dynamic tool discovery changes the prompt prefix, so the current function-calling approach segments trajectories and provides only coarse credit assignment.Assigning trajectory-level reward to segments and training on the last segment is an approximation; sampling earlier segments increases stability but uses more compute for similar results.Source: §6 · p. 18

A Synthetic Environments

Synthetic environments offer a lightweight alternative to costly live pages and virtual machines for training and evaluation. A capability flywheel mines skills from interaction traces, generates environments and tasks, and uses the resulting environments for supervised learning, reinforcement learning, and held-out evaluation.

A Synthetic Environments

  • Synthetic environments built with web technologies and rendered in a headless browser are lightweight, deterministic, durable, and faster to interact with than live environments.They run without emulators or virtual machines and can be hosted by the thousands on the training cluster without external dependencies.Source: A Synthetic Environments · p. 23
  • The capability flywheel derives target skills from real usage traces, generates environments and tasks until solvable, then uses them for supervised learning, reinforcement learning, and held-out evaluation.The figure description summarizes the three uses as teacher distillation, direct interaction to improve the agent, and benchmarking.Source: A Synthetic Environments · p. 23
  • Capability extraction uses a frontier model to over-generate fine-grained skills from trajectories, then merges synonyms and clusters them into higher-level groups.The source corpora include Android-in-the-Wild and MONDAY for mobile, and AgentNet for desktop.Source: A Synthetic Environments · p. 23
  • Each platform begins with a vetted operating-system shell, then forks and adapts it for capability groups and graded tasks using distinct synthetic user personas.Shared per-application data schemas let personas and tasks port across platforms with cosmetic changes.Source: A Synthetic Environments · p. 23
  • RL agents interact directly with the environments and receive task-completion rewards from an LLM judge; the setup uses around 16,300 tasks across 3,348 forked mobile and desktop environments.Lightweight environments run in parallel on inference nodes without separate environment-hosting machines.Source: A Synthetic Environments · pp. 23–24
  • Evaluation holds out entire capability groups from training and reports success on 160 Synthetic Mobile tasks and 140 Synthetic Desktop tasks.The same judge verifies task completion during training and evaluation.Source: A Synthetic Environments · p. 24

B SFT Setup

The SFT setup uses three stages that differ mainly in image resolution, task horizon, and data mixture. Across the stages, full fine-tuning freezes the vision encoder and multimodal projector, while the later stages use a four-domain mixture.

B SFT Setup

  • The three SFT stages differ mainly in input resolution, task horizon, and data mixture, while all optimize the standard autoregressive objective.All stages perform full fine-tuning with the vision encoder and multimodal projector frozen.Source: B SFT Setup · p. 24
  • The stages initialize successively from Qwen3-VL-Instruct, the High-res SFT model, and MintAct-SFT..Source: B SFT Setup · p. 24
  • All three stages use AdamW with zero weight decay, cosine learning-rate schedules, and 0.1 warmup.Their global batch sizes are 128, 64, and 64, respectively.Source: B SFT Setup · p. 24
  • Low-res multi-step SFT and RFT use a 25:25:25:25 domain mixture, with 30 maximum turns or 100 for visual tool use.Their maximum image pixels are 921,600 and maximum sequence lengths are 60,000.Source: B SFT Setup · p. 24

C RL Setup

MintAct's joint RL setup disables the KL penalty and uses truncated importance sampling to address rollout–training mismatch. Asynchronous training separates trainers and rollouters while controlling synchronization, staleness, and domain sampling.

C RL Setup

  • The setup sets the KL penalty to β = 0 and uses truncated importance sampling to relieve rollout–training mismatch.Source: C RL Setup · p. 25
  • Separate trainer and rollouter nodes synchronize model parameters every K updates, with message-queue capacity capped to limit sample staleness.Source: C RL Setup · p. 25
  • Screenshots are resized to a common maximum resolution, and each domain has a target sampling ratio and maximum interaction-turn count.Source: C RL Setup · p. 25

D System Prompts

The system prompts define agents’ roles, tools, and response formats across UI grounding, mobile, desktop, web, and visual tool use. They also specify how agents use preconfigured app tools and a restricted Python sandbox.

GUI grounding

  • The grounding prompt asks the agent to inspect a screenshot, interpret the instruction, and return a UI-element coordinate in a specified JSON-and-XML format.The output wraps a reasoning process in <grounding_think> tags and the coordinate in <answer> tags.Source: D System Prompts · p. 25

Mobile GUI

  • Mobile interaction uses one action per turn, followed by a new screenshot, with the agent continuing until task completion and then terminating.The prompt provides touchscreen actions such as clicking, typing, swiping, waiting, and opening apps; it requires a Thought, Action, and tool-call response.Source: D System Prompts · pp. 25–27

Desktop GUI

  • Desktop interaction uses screenshot-informed mouse and keyboard actions, with instructions to wait for slow applications and place clicks near element centers.Its response format consists of a short Action and a tool call.Source: D System Prompts · pp. 27–28

Web browsing

  • The web prompt defines browser actions for navigation and page interaction, requires one tool call per turn, and instructs the agent to stop once the task is complete.It also specifies Thought, Action, and tool-call output, with concise guidance for actions such as clicking, typing, scrolling, waiting, and answering questions.Source: D System Prompts · pp. 29–30

Visual tool use

  • Visual tool use relies on a dynamic tool registry whose schemas appear through function calling and whose tool names are randomized across examples.The fixed prompt directs the agent to discover relevant tools, verify their names and parameters, and use code execution for complex multi-step tasks.Source: D System Prompts · pp. 30–31

D System Prompts

  • Apps are already logged in and supervisor is preconfigured, so agents should skip login or signup and call supervisor functions directly when needed.The supervisor app provides credentials and profile information; its exact function signatures can be found by searching for “supervisor.”Source: D System Prompts · p. 31
  • Python runs in a sandbox that permits common libraries including json, math, datetime, numpy, matplotlib, and PIL.Use json.loads() and json.dumps(); file-based JSON functions are blocked.Source: D System Prompts · p. 31
  • The sandbox blocks file I/O, network and system access, time.sleep(), process control, and os filesystem operations.The prompt directs agents to access data through tool calls rather than files.Source: D System Prompts · p. 31
  • If code execution is blocked, agents should adjust their approach and retry.Avoid blocked operations that waste a turn.Source: D System Prompts · p. 31
Loading 2609.22083v1…