Falcon-UI: Understanding GUI Before Following User Instructions

1. Introduction

• GUI agents need to interpret both GUI context and user instructions, because dynamic interfaces change in response to predicted actions.Elements’ roles and potential responses matter alongside their visual layout.Source: §1 · p. 2
• Prior instruction-focused dataset approaches rely on costly human annotation, LVLM-generated instructions, or source-code heuristics that can limit scenario diversity and generalization.The paper notes that these methods can also depend on LVLM proficiency or unavailable and unreliable metadata.Source: §1 · p. 2
• Insight-UI Dataset uses Common Crawl pages and simulated actions to create instruction-free GUI data across 312K domains without human annotators or LVLMs.The dataset contains 434K episodes and 1,456K images, and the paper says it can be expanded to scenarios where GUI agents can operate.Source: §1 · p. 2
• Falcon-UI is pretrained on Insight-UI Dataset before fine-tuning on mobile and web GUI tasks; its 7B-parameter model achieves accuracy comparable to 72B-parameter Qwen2VL on AITZ.The paper motivates the approach with shared interaction patterns across interfaces, such as clicking an element to reveal details.Source: §1 · p. 2

2.1. Large Vision-Language Models

• Large vision-language models integrate vision with language models through cross-attention, MLPs, or Q-Former, while later systems address high-resolution inputs and curate large training datasets.The cited examples include Flamingo [1], LLaVA, InstructBLIP, LLaVA-NeXT, DeepSeek-VL [25], and Qwen2VL [33].Source: §2.1 · p. 3

2.2. GUI Agent

• GUI-agent work spans image-based navigation, element-layout modeling, grounding, and app-specific exploration across mobile and desktop interfaces.Examples include Auto-UI, CoCoAgent [27], AppAgent [22] [39], CogAgent [12], SeeClick [6], MobileVLM [35], and Ferret-UI [38].Source: §2.2 · pp. 3–4

3.1. GUI Context Understanding

• GUI-context pretraining models transitions between observations, addressing the static single-image format of prior datasets and its lack of next-state knowledge.The paper uses two formats: sequentially providing new observations after predicted actions, or presenting all observations and asking for their order and actions; high-loss examples are selected for the second format.Source: §3.1 · p. 4

3.2. Dataset Construction

• Insight-UI Dataset is collected by rendering filtered Common Crawl pages, simulating interactions, and recording observations and actions across device types and resolutions.The final dataset contains 434K episodes and 1,456K images; collection cost $200, compared with $19,200 for Rico crowdsourced annotations.Source: §3.2 · pp. 4–6, Figure 3 · p. 5

3.3. Model Optimization

• Falcon-UI pretraining combines Insight-UI GUI data with M4-Instruct [17] and 158K GPT-generated LLaVA instruction examples in a 1M-sample dataset.The model continues pretraining from Qwen2-VL-7B for one epoch, with its vision encoder frozen.Source: §3.3 · p. 6

4.1. Main Result

• On AITW, Falcon-UI improves overall performance 1% over CogAgent; it reaches state-of-the-art on AITZ and Android Control, with AITZ accuracy comparable to 72-billion-parameter Qwen2-VL.The AITZ comparison is reported in Table 3, and the Android Control comparison in Table 4.Source: §4.1 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 7
• On desktop Mind2Web, Falcon-UI significantly outperforms SeeClick using screenshot-based coordinate prediction, demonstrating generalization to desktop devices.A prediction counts as correct when its coordinate falls within the target element’s bounding box.Source: §4.1 · p. 7, Table 5 · p. 7

4.2. Ablation Study

• Single-source instruction data reduces overall GUI performance, grounding data yields modest improvement, and Insight-UI helps significantly before gains plateau beyond 100K pretraining examples.The study evaluates pretraining data mixtures on AITZ.Source: §4.2 · p. 8
• Adding grounding data to GUI-context data yields no notable improvement, whereas adding instruction-following data improves accuracy, especially when data volume exceeds 100K.The authors interpret this as GUI-context knowledge and instruction-following capability jointly enhancing GUI-agent performance.Source: §4.2 · p. 8

4.3. Further Analysis on Data Domain

• The domain study compares general Insight-UI data with 434K instruction-free contexts converted from AITW, then fine-tunes and evaluates the models on AITZ.AITW is treated as in-domain because AITZ is a cleaned subset of AITW with screenshots from the same source.Source: §4.3 · p. 8
• Both general and in-domain pretraining improve downstream performance; 100K GUI-context pretraining lets fine-tuning on 200 samples match direct fine-tuning on 2,000.The authors state that GUI-context pretraining can alleviate the need for extensive downstream data collection.Source: §4.3 · p. 8
• In-domain pretraining generally performs better when downstream data is limited, but its advantage narrows as pretraining and downstream data volumes increase.The authors suggest pretraining on extensive general data before domain-specific fine-tuning.Source: §4.3 · pp. 8–9

• The paper treats understanding GUI scenarios and interpreting user intentions as two core abilities for GUI agents.Source: §5 · p. 9
• Insight-UI Dataset is a fully auto-generated, instruction-free dataset intended to provide diverse GUI context knowledge.Source: §5 · p. 9
• Falcon-UI outperforms previous work across various benchmarks after GUI context comprehension is enhanced with Insight-UI Dataset.Source: §5 · p. 9
• Future work will combine general GUI knowledge with specific app perception to balance model performance and overall cost.Source: §5 · p. 9

• Figure 6 depicts hierarchical webpage design, with only upper-layer elements selected.Source: Figure 6 · p. 11

• Random action selection can produce repetitive or meaningless steps, so heuristic rules refine the dataset's action choices.Source: §6 · p. 11
• The initial action set contains visible clickable or inputable elements, with lower-layer elements removed when an upper layer is present.The webpage's hierarchy is represented by z-index; the upper layer is prioritized.Source: §6 · p. 11
• After an input action, the next action is selected near the input area because subsequent actions typically occur nearby.The input bounding box is expanded by its shorter edge length; elements outside it or more than two elements away in the action set are removed.Source: §6 · p. 11, Figure 7 · p. 11
• Elements shared across consecutive pages are excluded when they would repeat an action with the same result, unless the new page has no new elements.A scroll action is added before one remaining action is randomly selected for execution.Source: §6 · pp. 11–12, Figure 8 · p. 12

7.1. Training Format

• Multiple operation sequences can reach the same final observation, and intermediate operations can be redundant.One example reaches the final observation directly by clicking the title; Figure 9 also illustrates alternative routes to the same result.Source: §7.1 · p. 12, Figure 9 · p. 13
• Because navigation sequences may contain irrelevant actions, GUI-context pretraining can split long sequences into shorter tasks focused on navigation logic.The authors note that these issues are difficult to eliminate during data collection and that LVLM-based cleaning is costly and imprecise.Source: §7.1 · p. 12
• The first pretraining format supplies initial and final observations, then provides a new observation after each predicted action.The model is first trained on 2,000 randomly selected Insight-UI examples and evaluated with ground-truth inputs using teacher forcing.Source: §7.1 · p. 12
• The second pretraining format presents all observations at once and asks the model to infer their order and the actions at each step.Half of the examples with higher evaluation losses in the first-format evaluation are selected for this format.Source: §7.1 · p. 12

7.2. Training Configuration

• Falcon-UI is initialized from Qwen2-VL-7B, continually pretrained, and LoRA-fine-tuned with the vision encoder frozen.The training uses AdamW at 1e-4, a global batch size of 64, and linear warm-up for 10% of steps followed by cosine decay.Source: §7.2 · p. 12
• Pretraining uses ZeRO2 with vision resolution limited to 840 for the first 80% of steps, then ZeRO3 limited to 1540 for the remaining 20%.Examples use center-point prediction for 70% of cases and bounding-box prediction for 30%.Source: §7.2 · p. 13

8. Visualization

• Falcon-UI predictions are shown in mobile AITZ and desktop Mind2Web episodes, illustrating the reported generalization across diverse scenarios.Figure 13 shows mobile episodes; Figure 14 shows desktop episodes.Source: §8 · p. 13, Figure 13 · p. 17, Figure 14 · p. 18
• An example shows that intermediate operations can be skipped by clicking “Academic Catalog 2024-2025” to move directly from the initial to final observation.Source: Figure 10 · p. 14
• The initial-to-final screenshot prompt illustrates stepwise interaction, with “Our Services” as an example predicted click.Source: §8 · p. 15
• The first format reveals a new observation after each predicted action, while the second provides all observations initially for ordering and action prediction.Source: Figure 11 · p. 15, Figure 12 · p. 16
• A random-order screenshot episode illustrates predicting screenshot order and the action at each transition, including clicks on “Highlight Links” and “Invert Colors.”Source: §8 · p. 16
• Offline metrics may be inaccurate in complex GUI scenarios; searching for “Taylor Swift” can also yield correct results despite a marked incorrect prediction.Source: Figure 14 · p. 18