From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

1 Introduction

• Structured interface data can be unavailable, difficult to interpret, or misaligned with the visible GUI, while aligning demonstrations with task-dependent actions can be challenging.The paper motivates pixel-based observations and generic actions as an alternative to those dependencies.Source: §1 · p. 1
• PIX2ACT uses pixel screenshots and basic mouse and keyboard actions, tuning PIX2STRUCT with human demonstrations and environment interactions.The approach also uses tree search to iteratively generate new expert trajectories for training.Source: §1 · pp. 2–3
• PIX2ACT outperforms MiniWob++ human crowdworkers and improves task score nearly 4x over the best prior result in the proposed setting.The comparison baseline is CC-Net without DOM.Source: §1 · p. 3
• A pixel-only agent with generic actions reaches performance comparable to state-of-the-art agents using DOM information and a comparable number of human demonstrations.The paper describes this as the first such result on MiniWob++.Source: §1 · p. 3
• The paper adapts WebShop to pixel observations and general low-level actions, establishing its first baseline while retaining a performance gap to larger HTML-based agents with task-specific actions.The benchmark adaptation uses a unified observation and action format.Source: §1 · p. 3
• PIX2STRUCT pre-training improves behavioral-cloning task scores from 17.1 to 66.5 on MiniWob++ and from 1.1 to 46.7 on WebShop.The paper identifies screenshot-parsing pre-training as effective for pixel-based GUI instruction following.Source: §1 · p. 3
• The paper also demonstrates tree search as a relatively simple policy-improvement method for MiniWob++.The method relies on access to environment interactions and reward signals, as described in the approach.Source: §1 · p. 3

2 Environment

• The browser-based framework models GUI interaction as an MDP, with an agent receiving observations and selecting actions at each step.Selenium is used to interact programmatically with Chrome.Source: §2 · p. 3
• Observations combine a browser screenshot with the instruction, a drawn cursor, and an indicator for whether the mouse button is pressed.The cursor and button indicator provide interaction-state cues, including for dragging.Source: §2 · p. 3
• Actions use raw mouse and keyboard controls, including discretized coordinates, keys, modifiers, scrolling, and drag operations; DOM data defines neither observations nor actions.Coordinate bins provide a configurable resolution for each axis.Source: §2 · p. 3
• Episodes end at a terminal state or step limit, and reward arrives only at termination as a binary success signal or partial task score.The MiniWob++ example shows a new observation and next action at each step, with 32 coordinate bins per axis.Source: §2 · p. 3, Figure 2 · p. 4

3 Proposed Agent

• PIX2STRUCT uses an image Transformer encoder and text Transformer decoder, pre-trained to predict simplified HTML from screenshots with visually masked regions.The experiments use its 282M-parameter base variant, called once per time step.Source: §3 · p. 4
• The model receives pixel observations, optionally concatenating prior frames, but experiments use only the current screenshot; it predicts action tokens autoregressively and beam-searches candidate actions.Input resolutions are 160×210 for MiniWob++ and 800×600 for WebShop.Source: §3 · p. 4
• The greedy policy selects the highest-scoring action, avoiding actions already taken for the same observation to help prevent cycles.The paper reports this modification provides a modest performance increase.Source: §3 · p. 4

3.1 Training

• Behavioral cloning trains on human demonstrations, while tree search is used as an alternative policy-improvement approach to common reinforcement-learning algorithms.Search takes advantage of deterministic environment dynamics to look ahead at possible actions.Source: §3.1 · pp. 4–5
• MCTS estimates leaf-state values using a learned value model and greedy-policy rollouts, with a surrogate reward penalizing extra steps.The value model uses the PIX2STRUCT architecture and predicts discretized state values.Source: §3.1 · p. 5
• Successful episodes from tree search become supervised training data, and the improved model can support stronger search in repeated iterations.The fine-tuning procedure remains standard supervised learning on tree-search episodes.Source: §3.1 · p. 5

4 Benchmarks and Demonstrations

• Both benchmarks are adapted to a shared framework with pixel-based observations and generic low-level actions, and previously collected human demonstrations are mapped to those formats.The framework covers MiniWob++ and WebShop.Source: §4 · p. 5

4.1 MiniWob++

• MiniWob++ comprises over a hundred browser tasks with seeded variations, visually presented instructions, and automatic rewards at task end.Each task has an algorithm for generating variations and an instruction template.Source: §4.1 · p. 5
• MiniWob++ demonstrations came from an X11 environment, creating rendering shifts; mapping succeeded reasonably for 59 tasks, but some demonstrated behaviors and animated observations are unsupported in Selenium.For example, Selenium does not support highlighting text and dragging it into a text field.Source: §4.1 · p. 5
• PIX2ACT significantly improves over prior work on MiniWob++ and establishes the first pixel-based baseline on WebShop, with a gap to the best HTML-based WebShop methods.On MiniWob++, it performs competitively with humans and methods using DOM or HTML.Source: Figure 3 · p. 6

4.2 WebShop

• WebShop asks users to find and purchase products from over 1.1 million Amazon products based on text instructions, with reward based on similarity to the target product.Tasks involve searching, selecting results, and judging their relevance to the instruction.Source: §4.2 · p. 6
• WebShop demonstrations were converted from high-level actions into lower-level clicks, scrolling, and text entry; evaluation uses average reward over 500 test instructions.Clicks are mapped to element-center coordinates, with scrolling added when the target lies outside the viewport.Source: §4.2 · p. 6

5.1 Training Details

• MiniWoB++ training used 26K steps, two tree-search improvement iterations totaling 826K episodes, and another 26K tuning steps; WebShop used demonstrations and MiniWoB++ intermediate finetuning.The WebShop finetuning phase lasted 10K steps.Source: §5.1 · pp. 6–7
• Figure 4 ranks task-level MiniWoB++ scores for PIX2ACT and human crowdworkers by their relative performance difference.Source: Figure 4 · p. 7

5.2 Main Results

• PIX2ACT significantly outperforms the estimated CC-Net without DOM baseline on MiniWoB++; WebShop has no prior baseline in this setting.The CC-Net estimate assumes the 75% screenshot-only performance drop reported for a subset of tasks also holds on the studied subset.Source: §5.2 · p. 7

5.3 Ablations and Analysis

• On MiniWoB++, PIX2ACT is comparable to CC-Net’s best model using fewer information sources and a comparable number of demonstrations; CC-Net’s BC-only score is 38.7.CC-Net also loses approximately 10% when it lacks the environment-provided dictionary of input strings.Source: §5.3 · pp. 7–8
• On WebShop, WebGUM outperforms PIX2ACT and uses HTML, a shopping-specific action space, and a Flan-T5-XL backbone.The paper attributes some of the gap to simplified actions, direct access to page text, and Flan-T5 pretraining and instruction finetuning.Source: §5.3 · p. 8
• PIX2STRUCT initialization yields a 28.3 mean score on held-out tasks versus 7.6 with random initialization; training on those tasks yields 65.5.The nine held-out tasks were manually selected as reasonable to generalize to from the training tasks.Source: §5.3 · p. 8, Table 1 · p. 8
• Finetuning directly on WebShop without prior MiniWoB++ finetuning lowers Task Score by 4.0; the experiments do not test RL on WebShop.Source: §5.3 · pp. 7–8
• Tree-search-generated episodes improve MiniWoB++ scores, while the best greedy policy approaches the best tree-search policy without inference-time rewards or extra exploration.The authors report that further tree-search policy-improvement iterations could improve performance.Source: §5.3 · p. 8, Table 2 · p. 8

• Early GUI approaches used component properties or hierarchical structures such as DOM trees, often targeting actions at specific components; such data may be unavailable or visually misaligned.Source: §6 · p. 9
• World of Bits paired visual and structured inputs with low-level actions but performed poorly against humans; later work emphasized DOM-aware architectures and restricted actions, while CC-Net returned to visual and structured inputs with low-level actions.The paper builds on CC-Net’s use of large-scale human demonstrations and reports that PIX2ACT can outperform humans on MiniWoB++ while matching DOM-based state-of-the-art approaches.Source: §6 · p. 9
• Web-task LLM approaches commonly use HTML inputs and custom actions; some are sample-efficient, and WebGUM combines HTML-based modeling with an ImageNet-21K-pretrained vision encoder.Source: §6 · p. 9
• Mobile GUI studies often rely on Android view hierarchies or do not evaluate interactive environments, while visual-only, low-level-action agents have also been studied for Atari and Minecraft.The paper focuses on web GUIs to use a consistent environment framework.Source: §6 · p. 9

7 Limitations and Discussion

• Pixel-based pretraining has shown GUI-task effectiveness at a smaller scale, while text-based inputs remain useful when available for transferring LLM capabilities.The authors hope that scaling vision or multimodal pretraining could enable similar capabilities in pixel-based settings.Source: §7 · pp. 9–10
• Tree search for MiniWob++ relies on procedurally generated task and instruction variations, completion rewards, and the ability to reset the environment.Such capabilities may be unavailable in real-world environments, where generative instructions and approximate reward models might be needed.Source: §7 · p. 10
• The study trained and evaluated models only in offline environments; online-service deployment would require behavior checks and attention to data privacy.The paper also identifies potential spam, fraud, abuse, and security risks from service-interacting models.Source: §7 · p. 10
• Models interacting with online services could violate service terms or enable abuse that is harder to detect, so security and abuse-detection research should account for these uses.Examples include impersonation, harmful content, spam, fraud, denial-of-service attacks, and potentially solving CAPTCHAs.Source: §7 · p. 10

A.1 MiniWob++ Supported Tasks

• The evaluation supports 59 of MiniWob++’s 104 tasks, selected for feasibility of converting demonstrations and implementing actions, not for model performance.Further development of conversion and Selenium actions could potentially support additional tasks.Source: §A.1 · p. 13
• Tasks were excluded when animations lacked a -nodelay version, scrolling or copy-paste interactions were difficult to map, or Selenium did not support required interface behaviors.Other exclusions followed low demonstration-conversion success, including episodes ending after pointer-down events or frequent double-clicking.Source: §A.1 · p. 13

A.2 MiniWob++ Rendering Differences

• Human demonstrations and the Selenium environment render the same state differently, including subtle changes in font style, element size, and position.Figure 5 compares the human-demonstration screenshot on the right with the Selenium rendering on the left.Source: §A.2 · p. 13, Figure 5 · p. 14

B.1 Beam Search

• Beam search over decoder tokens returns the top-k candidate actions and their approximate probabilities for a state.The experiments use k = 8 for MiniWob++ and k = 10 for WebShop.Source: §B.1 · p. 14

B.2 Tree Search

• The surrogate reward adds a small per-step penalty to raw reward only when a terminal state exceeds the 0.8 raw-reward threshold.The penalty encourages shorter trajectories without unnecessary actions and is small enough not to incentivize sacrificing raw reward.Source: §B.2 · p. 14
• The value network approximates optimal future reward from states in human demonstrations, training on their actual future surrogate rewards mapped to discrete bins.At inference, it estimates the mean over its top three bin predictions, weighted by their probabilities.Source: §B.2 · p. 14
• The search policy uses environment interactions and rewards to select actions more optimally than greedy policy selection, with the policy and value networks constraining and prioritizing search.The policy network selects the highest approximate-probability action under the greedy policy.Source: §B.2 · p. 15
• Monte Carlo Tree Search traverses a state-action tree for 16 rounds, choosing actions by accumulated average reward plus an exploration term.The method was minimally tuned to improve mean score over the greedy policy, including on challenging tasks.Source: §B.2 · pp. 14–15
• The policy network biases exploration, which is restricted to its top 8 actions; the exploration term uses visit counts and a scalar set to 0.1.The term encourages exploration of actions selected fewer times from a state.Source: §B.2 · p. 15
• For a newly reached leaf, the method mixes the value-network estimate with up to 20 steps of greedy-policy rollout to estimate future returns.The mixture is intended to balance potentially uninformative rollouts against inaccurate value estimates, especially for states underrepresented in demonstrations.Source: §B.2 · p. 15
• Combining rollout and value-network estimates gives the best mean scores over 12 challenging MiniWob++ tasks; the value network is especially useful on longer tasks.The comparison includes rollout-only, combined, value-network-only, and greedy-policy results.Source: Table 3 · p. 15, §B.2 · p. 15
• The policy is iteratively tuned on generated trajectories filtered to raw reward of at least 0.8, while the value network remains fixed.The search tree is reused across subsequent time steps for efficiency; tuning on search trajectories required a lower learning rate and more warmup steps to address an early loss spike.Source: §B.2 · pp. 15–16

B.3 Compute Details

• Fine-tuning used 64 Google Cloud TPU v3 cores.Source: §B.3 · p. 16

C.1 Variance Estimates

• Across three MiniWob++ evaluation-seed trials, mean scores were 96.2, 96.4, and 96.1, with a standard deviation of 0.15; WebShop uses a 500-instance standard test set.MiniWob++ evaluation used 100 randomly selected seeds for each of 59 tasks.Source: §C.1 · p. 16

C.2 MiniWob++ Results Per Task

• The reported comparisons also include PIX2ACT and CC-Net scores using behavioral cloning only.Source: §C.2 · p. 16
• GlobalCNN is omitted because it reported only human-normalized success rates; prior work’s success rates are mappable to reported scores for tasks without partial rewards.Humphreys et al. (2022) is the stated exception to the prior-work reporting pattern.Source: §C.2 · p. 16
• Table 4 reports mean scores across 59 MiniWob++ tasks, comparing PIX2ACT with human crowdworkers and other approaches.The passage does not provide the table’s individual scores or identify a leading approach.Source: §C.2 · p. 16, Table 4 · p. 17