Source-linked AI summary
GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection
Zheng Wu, Chengcheng Han, Zhengxi Lu, Tianjie Ju, Yanyu Chen, Qi Gu, Xunliang Cai, Zhuosheng Zhang
TL;DR
GUI agents remain limited by insufficient GUI-operation world knowledge, while scaffolding and conventional post-training provide costly or implicit solutions. GUI-CIDER synthesizes and selects structured GUI knowledge for mid-training, and experiments report gains in both GUI understanding and task success across five benchmarks.
Problem
GUI agents’ advancing capabilities are increasingly bottlenecked by insufficient world knowledge about GUI operations, while scaffolding adds overhead and SFT/RL encode knowledge implicitly.
Method
GUI-CIDER synthesizes static planning and dynamic causal knowledge from GUI trajectories, reselects causally structured and non-redundant exemplars, and internalizes them through mid-training.
Results
GUI-CIDER improves GUI-operation understanding and downstream task success across three task-completion and two GUI-knowledge benchmarks, including a 9.70% average relative task-success improvement over post-training baselines.
Takeaways & Limitations
The results suggest that knowledge scaling through structured mid-training is a promising path toward more capable GUI agents.
Takeaways & Limitations
Due to computational resource constraints, GUI-CIDER uses LoRA instead of full parameter tuning and evaluates models ranging from 4B to 8B parameters.
Abstract
from arXiv · showhide
Despite the rapid progress of multimodal large language models in building Graphical User Interface (GUI) agents, their real-world task completion is fundamentally bottlenecked by a lack of world knowledge about GUI operations. Existing solutions typically rely on expensive multi-agent scaffolding or conventional post-training paradigms, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). However, post-training only allows agents to implicitly absorb world knowledge through action annotations or reward signals, leading to inefficient trajectory memorization rather than genuine comprehension. Therefore, an approach that enables explicit learning of this knowledge is imperative. To this end, we propose GUI-CIDER, a mid-training method that explicitly internalizes GUI world knowledge through Causal Internalization and Density-aware Exemplar Reselection. GUI-CIDER operates in three stages: (1) data synthesis, which distills static planning and dynamic causal knowledge from GUI trajectories into text; (2) exemplar reselection, which filters the corpus by rewarding causal structures and penalizing semantic redundancy; and (3) mid-training, where the refined data is used to embed the acquired knowledge. Extensive experiments on two GUI knowledge benchmarks and three task completion benchmarks demonstrate that GUI-CIDER consistently improves both the agent's understanding of GUI operations and its task success rates.The codes are available at https://github.com/Wuzheng02/GUI-CIDER.
1 Introduction
GUI-CIDER addresses GUI agents’ limited world knowledge by explicitly internalizing it through mid-training rather than relying solely on scaffolding or conventional post-training. Its three-stage pipeline synthesizes, filters, and trains on GUI knowledge, improving knowledge understanding and task completion.
- GUI agents’ capability bottleneck increasingly stems from limited world knowledge about GUI operations despite progress from SFT and RL.
- Multi-agent scaffolding can compensate for GUI agents’ world-knowledge deficiency but introduces additional overhead and scaffolding.
- Conventional SFT and RL implicitly encode GUI knowledge through action labels or rewards, encouraging trajectory memorization rather than genuine comprehension.
- GUI-CIDER explicitly internalizes GUI world knowledge through Causal Internalization and Density-aware Exemplar Reselection in a mid-training method.
- Its pipeline synthesizes static planning and dynamic causal knowledge, reselects causally strong and less redundant exemplars, and uses the refined corpus for mid-training.
- 9.70% average relative improvement in task success rate versus post-training baselines accompanied GUI-CIDER’s performance across three task-completion and two GUI-knowledge benchmarks.On the GUI knowledge benchmark, an 8B-scale agent reached a level close to Claude-Sonnet-4.5.
- Model comparisons indicate that mid-training should target general agents rather than agents excessively post-trained for the GUI domain, while ablations support the pipeline’s rationale.
- The work contributes an approximately 100M-token synthesized corpus and reports improvements in GUI-operation knowledge and task completion.
2 Related Work
Related work develops GUI agents through single-agent and multi-agent approaches and uses mid-training to extend pretrained knowledge into specialized domains. However, GUI-agent mid-training remains sparsely studied, with limited publicly available data and methodological detail.
- GUI agents operate computers, web environments, and smartphones by simulating human actions such as clicking and scrolling.
- GUI-agent construction is broadly divided into single-agent approaches and multi-agent systems.
- Mid-training bridges pre-training and post-training by extending knowledge into specialized domains while preserving general capabilities.
- Existing mid-training work commonly involves data collection, synthesis, selection, and decontamination for mathematics, question answering, and coding.
- Very little work internalizes domain knowledge for GUI agents through mid-training, and UI-Venus-1.5 did not open-source its data or provide specific details.
3 GUI-CIDER
GUI-CIDER synthesizes GUI planning and causal knowledge into text, filters examples using causal saliency and semantic density, then mid-trains agents on the refined corpus.
- GUI-CIDER comprises data synthesis, exemplar reselection, and mid-training stages for internalizing GUI world knowledge.The method converts GUI trajectories into knowledge-rich textual samples before filtering and training.
- Data Synthesis: Static planning extraction decomposes task instructions into hierarchical natural-language sub-goals that form an actionable execution graph.An expert reasoning model generates the sub-goals used as dense supervision for long-term planning.
- Data Synthesis: Dynamic causal synthesis translates GUI actions and screenshots into semantic descriptions, triggers, mechanisms, and self-contained textual rationales.The rationale stores pre- and post-state descriptions, eliminating the need for raw screenshots in the final sample.
- Exemplar Reselection: Exemplar reselection combines causal-token saliency with K-nearest-neighbor density estimation to retain informative and less redundant samples.Causal saliency is based on causal-logic token counts, while density is normalized across the synthesized corpus.
- Mid-training: Mid-training applies next-token prediction to serialized high-quality samples, embedding transition dynamics and GUI world knowledge into model parameters.The full textual sequence is trained as a plain autoregressive language-modeling stream without input-output separation.
4 Is the Retention Function g(x) a Good Function?
The retention function is designed to favor causal saliency, penalize redundancy, preserve density ordering, and amplify causal distinctions in dense regions; the paper provides theoretical support for these properties.
- The retention function is evaluated against four properties governing causal saliency, density penalties, density ordering, and causal-density interaction.The section defines these requirements and proves that GUI-CIDER’s function satisfies them.
- Causal Monotonicity: Higher causal saliency produces higher retention probability, with a strictly positive effect for samples whose density exceeds zero.This property rewards samples containing richer causal-logic structures.
- Density Penalty: Higher semantic density lowers retention probability, imposing a redundancy penalty on samples in semantically crowded regions.The proof establishes non-positive dependence on density under the stated parameter constraints.
- Density Order Preservation: The reselection rule preserves the original ordering of semantic density, so denser samples remain ordered as denser after weighting.For densities d1 < d2, the product d1 · g(x1) remains below d2 · g(x2), all else being equal.
- Density-Causal Synergy: In denser regions, increased causal saliency yields greater marginal retention benefit, helping rigorous exemplars stand out among redundant samples.The cross-partial derivative is strictly positive when λ > 0 and α > 0.
5 Experiment
GUI-CIDER is evaluated across three task-completion benchmarks and two GUI-knowledge benchmarks, improving both GUI understanding and task success. Its 8B model approaches Claude-Sonnet-4.5 on the GUI knowledge benchmark.
- Evaluation setup: GUI-CIDER is evaluated on AITZ, AndroidControl, and GUI-Odyssey for task completion, plus MMBench-GUI L1 and GUI Knowledge Bench for GUI knowledge.Task benchmarks measure action and completion outcomes, while knowledge benchmarks use multiple-choice and true-false questions.
- Task completion: Mid-training improves task completion across models of different parameter scales, and subsequent post-training preserves GUI-CIDER’s benefits.A 4B model using both stages surpasses its 8B counterpart, motivating knowledge scaling alongside parameter scaling.
- GUI understanding: GUI-CIDER-8B significantly outperforms baselines on MMBench-GUI L1, indicating stronger GUI content understanding.The table caption states that improvements appear across all evaluated platforms despite incomplete platform coverage in the synthesis data.
- GUI knowledge: 66.51 vs. 66.53: GUI-CIDER-8B reaches performance close to Claude-Sonnet-4.5 on the GUI knowledge benchmark.It also surpasses larger models on the objective subset, which assesses whether a task is truly completed.
6 Further Analysis
Further analysis examines base-model choice and the exemplar reselection stage. General models benefit from GUI-CIDER data, whereas extensively GUI-post-trained models degrade, and removing reselection substantially lowers success.
- Model comparison analysis: Increasing GUI-CIDER data consistently improves SR for the general Qwen3-VL-8B-Instruct base model.The analysis varies the amount of generated data in 20% increments on AITZ.
- Model comparison analysis: Extensively GUI-post-trained OS-Atlas-pro-7B steadily declines as more GUI-CIDER data are added, unlike the general base model.The authors attribute this to partially disrupted language representations that hinder learning new world knowledge through mid-training.
- Model comparison analysis: The analysis supports mid-training world knowledge before GUI-domain post-training as a reasonable training paradigm.This ordering follows the contrast between general-model gains and specialized-model degradation.
- Ablation study: Removing exemplar reselection causes a substantial SR drop on GUI-Odyssey.The authors attribute the decline to low-quality and redundant samples that introduce noisy supervision and encourage shortcut or hacking behaviors.
7 Conclusion
GUI-CIDER internalizes GUI world knowledge through synthesized causal and planning information plus density-aware exemplar selection. Across five benchmarks, it improves GUI operation understanding and downstream task success, supporting knowledge scaling as a promising path.
- Conclusion: GUI-CIDER synthesizes static planning knowledge and dynamic causal transition logic from GUI trajectories, then selects knowledge-rich, non-redundant exemplars.The framework uses mid-training rather than external scaffolding or direct post-training on raw trajectories.
- Conclusion: Experiments on three task-completion benchmarks and two GUI-knowledge benchmarks improve both GUI operation understanding and downstream task success.The reported findings support knowledge scaling as a promising direction for more capable GUI agents.
Limitations
GUI-CIDER’s retention rule combines density-aware sampling with causal saliency, with theoretical results describing its boundaries, monotonicity, and expected preservation of causal information. A practical limitation is that the reported mid-training uses LoRA and 4B–8B models because of computational constraints.
- Practical limitation: LoRA replaces full-parameter tuning in GUI-CIDER’s mid-training because of computational resource constraints, with experiments using 4B–8B models.The authors identify full-parameter tuning and scaling to larger models as future work.
- Retention properties: The retention function is non-decreasing in causal saliency for fixed density, and strictly increasing when λ > 0 and d(x) > 0.This property formalizes preferential retention of samples with stronger causal structure.
- Retention properties: For fixed causal saliency, the retention function is monotonically non-increasing in density, while the reweighted density score remains strictly increasing in density.These results distinguish sample retention from the ordering induced by the reweighted density score.
- Retention properties: The retention rule exhibits positive density–causal synergy: the marginal value of causal saliency becomes larger in denser regions.The stated interaction is strictly positive for λ > 0 and α > 0.
- Information preservation: Stochastic thinning retains a guaranteed fraction of the original causal signal in expectation.The expected retained corpus is characterized through Bernoulli retention indicators with mean g(x), and a lower bound is given for preserved causal information.
B.4 Stability Under Score Estimation Errors
The appendix analyzes how errors in estimated causal saliency and density affect GUI-CIDER’s retention score. It concludes that bounded estimation noise produces controlled perturbations, making the rule stable under score uncertainty.
- Stability result: Moderate perturbations in estimated causal saliency and density induce controlled perturbations in the retention score.Both quantities are estimated from synthesized text and embedding geometry.
- Stability result: The retention rule is stable under bounded score estimation noise.The result follows from a Lipschitz-stability argument based on the derivatives of g(f, d).
C Benchmark Details and Examples
The experiments use five benchmarks covering GUI knowledge, content understanding, Android navigation, realistic mobile control, and long-horizon cross-app navigation. GUI-CIDER also converts raw GUI transitions into structured, reasoning-rich textual supervision.
- Benchmark details: GUI Knowledge Bench evaluates widget functions, interface states, interaction effects, and workflow progress rather than end-to-end execution.It is presented as a diagnostic benchmark for GUI-specific knowledge across diverse GUI platforms.
- Benchmark details: MMBench-GUI L1 tests GUI content understanding, including interface content, element semantics, and relative placement.The benchmark is hierarchical and cross-platform; the L1 split is used here.
- Benchmark details: AITZ evaluates whether agents infer the next GUI action from the current screen, prior context, and task instruction.It is an Android navigation benchmark built from screen–action pairs with Chain-of-Action-Thought annotations.
- Benchmark details: AndroidControl evaluates everyday mobile-task following using human demonstrations paired with high- and low-level instructions.It targets realistic Android control with diverse data and task complexity.
- Benchmark details: GUI-Odyssey evaluates long-horizon cross-app navigation requiring history tracking, planning, and cross-app reasoning.Its multi-app setting places heavier demands on these capabilities than single-app benchmarks.
- Synthetic examples: GUI-CIDER structures each synthetic example as a step list, action intent, state descriptions, trigger, mechanism, and reasoning.The representation captures why an action is appropriate in the current GUI state, while lexical anchors help elicit and inspect reasoning-rich patterns.
F Experimental Details
The experiments use specified expert and base models, training settings, and action-space documentation, while comparing mid-training followed by post-training against direct post-training. The reported setup costs 1,400 hours on 80G GPUs.
- Model configuration: Planning uses deepseek-v4-flash, mapping and causal analysis use Qwen3-VL-32B-Instruct, and base models are Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct.These models instantiate the three synthesis modules and the mid-training base agents.
- Action space: Table 6 documents the action space used in the experiments.The supplied passage identifies the table but does not enumerate its actions.
- Training comparison: Mid-training with GUI-CIDER followed by post-training improves performance over direct post-training on the same data and test set.The comparison is reported as evidence that GUI-CIDER can further unlock the potential of the data.
- Compute: The total computational cost is 1,400 hours on 80G GPUs.The cost is reported for the experimental setup.