Source-linked AI summary
GPA: Learning GUI Process Automation from Demonstrations
Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan Li
TL;DR
Traditional RPA is brittle and costly to maintain, while VLM-based GUI agents pose reliability, controllability, latency, and privacy challenges. GPA learns a structured GUI workflow from one demonstration and replays it using geometric grounding, readiness calibration, and local deterministic execution. In a pilot comparison, GPA achieved 100% success across both difficulty levels and was roughly 10× faster than Gemini.
Problem
Traditional RPA requires brittle manually scripted selectors, while VLM-based GUI agents are non-deterministic, difficult to control, slow, and privacy-sensitive for enterprise workflows.
Method
GPA records one demonstration, represents UI states as graphs, and grounds workflow actions using visual similarity, geometric relationships, Sequential Monte Carlo localization, and readiness checking.
Results
100% success across both difficulty levels was reported for GPA, with roughly 10× faster execution than Gemini.
Takeaways & Limitations
GPA provides demonstration-based GUI process replay with deterministic local execution for interfaces where GUI elements are well-defined and stable.
Takeaways & Limitations
GPA cannot reason or make decisions during replay, so workflows requiring judgment, such as changing calendar months, may only work for the recorded date.
Abstract
from arXiv · showhide
GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.
1 Introduction
GPA addresses the implementation burden and brittleness of traditional RPA, as well as the non-determinism, limited controllability, latency, and privacy risks of VLM-based GUI agents. It learns a structured workflow from one demonstration and combines geometric grounding, deterministic execution, readiness checking, and local processing.
- Limitations of existing GUI automation: Traditional RPA requires manually defined selectors and scripted edge-case handling, while layout or resolution changes can break its brittle rules.Dynamic content, complex conditional logic, and unstructured visual data may also remain outside deterministic RPA workflows.
- Limitations of existing GUI automation: VLM-based GUI agents offer flexible interface interaction but introduce non-deterministic actions, limited controllability, high latency, and privacy risks from cloud inference.The same instruction can produce different action sequences across runs, complicating auditing and prediction.
- GPA approach: GPA learns a structured workflow from a single user demonstration without coding or selector inspection.The workflow contains ordered steps, action types, neighboring nodes, and parameterisable fields.
- GPA safeguards: GPA executes locally and deterministically, using readiness calibration to gate actions until confidence exceeds a defined threshold.The design uses lightweight local models and keeps sensitive visual data on the local machine.
- GPA approach: GPA grounds recorded actions through context-guided Sequential Monte Carlo inference, using neighboring UI elements to handle missed detections and window rescaling.This targets the UI-grounding challenge created by changing appearances, dynamic content, and inconsistent identifiers.
2 GPA: GUI Process Automation
GPA learns GUI workflows from a single demonstration, representing UI states and recorded actions structurally before replaying them on a live screen. It grounds targets with visual and geometric context, uses SMC for ambiguous or rescaled layouts, and controls execution through readiness checks and bounded retries.
- Demonstration and Workflow Building: GPA records one user run as screenshot-action pairs, builds UI graphs and step subgraphs, then compiles them into a reusable workflow template.The template includes ordered steps, action types, nearby nodes, and parameterizable fields.
- Problem Formulation: GPA formulates target grounding as predicting a current-screen click location from a demonstrated target and its neighboring nodes under layout changes.The latent variable θ = [x, y, sx, sy] captures target location and horizontal and vertical scaling between demonstration and runtime screens.
- Problem Formulation: Each UI state is represented as a graph whose nodes contain bounding boxes, OCR text, and icon embeddings, with edges linking spatially nearby elements.Graphs are constructed using UI detection, OCR, visual features, and k-nearest-neighbor spatial connections.
- Execution Control: At execution time, GPA captures and parses each screen, checks readiness, executes confident actions, and retries steps when confidence is below threshold.A finite-state controller alternates between execution and decision states, with bounded retry-or-fail behavior and a scroll-specific branch that skips unnecessary scrolling when the target is already visible.
- Grounding as Sequential Monte Carlo Localization: Direct visual matching handles clear cases, while context-guided SMC approximates multimodal posteriors when candidate correspondences are ambiguous or the layout changes.Particles are initialized from context-candidate matches and refined through likelihood tempering, reweighting, resampling, and Metropolis–Hastings rejuvenation.
3 Pilot Experiments
The pilot compares GPA with Gemini on 16 desktop GUI tasks of differing demonstration lengths, measuring runtime and success rate. GPA maintains perfect success while executing substantially faster, with reliability, latency, and long-horizon scaling explaining the gap.
- Experimental setup: The pilot evaluates GPA against Gemini on 16 desktop productivity and enterprise GUI tasks, grouped by demonstration length and measured by runtime and success rate.Simple tasks average 10.8 steps, while hard tasks average 27.27 steps.
- Pilot results: 100% success and 33.74 s versus 329.31 s runtime show GPA is roughly 10× faster while succeeding across both difficulty levels.Gemini success falls from 93.2% to 87.64% as tasks become longer.
- Reliability: GPA improves reliability by following a fixed demonstrated procedure, acting only after readiness confirmation, retrying bounded delays, and explicitly failing when mismatches remain unresolved.Gemini instead infers the correct action at every step, allowing small misidentification risks to compound.
- Latency: On-device capture, OCR, and local retrieval keep GPA steps within milliseconds to low seconds, whereas Gemini’s network and inference latency accumulates over each action.On approximately 27-step hard tasks, runtime exceeds 383 s for Gemini versus approximately 41 s for GPA.
- Scaling with length: Over 27 steps, Gemini’s per-step errors compound into an approximately 6 percentage-point success drop, while GPA’s fixed-plan execution remains at 100%.GPA adds only a cheap matching operation per step and does not introduce additional drift risk from generative action selection.
4 Related Work
Related work spans classical RPA, foundation-model GUI agents, and learning from demonstrations. These directions address GUI automation through structured workflows, multimodal perception and action, or demonstrated task specification.
- From RPA to visual GUI automation: Classical RPA is attractive for legacy systems without API integration but is most effective for structured, rule-based, relatively invariant processes.Coupling automation logic to application structure or surface behavior makes systems brittle under software and interface changes.
- Foundation-model GUI agents: Foundation-model GUI agents frame automation as multimodal decision-making over screenshots, interface cues, and grounded actions across web, mobile, and desktop settings.Surveys organize this space around perception, grounding, planning, memory, and execution.
- Foundation-model GUI agents: Web GUI research established datasets and evaluation suites including Mind2Web, WebArena, and VisualWebArena, exposing a gap between text-only agents and grounded interaction.The passage situates these benchmarks within the maturation of screenshot-based web interaction.
- Learning from demonstration: Demonstration-based work reduces reliance on zero-shot planning by using recorded workflows, clarification, demonstration parsing, experience retrieval, and action execution.Examples include HILC and LearnAct with the LearnGUI benchmark.
5 Conclusions & Limitations
GPA is presented as a single-demonstration RPA framework for local GUI automation, with CLI support for composing workflows. Its central limitation is that record-and-replay cannot reason about changing task states.
- Conclusions: GPA requires only a single demonstration and uses UI-grounding algorithms to replay GUI operations without requiring expensive LLM agents.The framework is positioned for local desktops, browsers, enterprise software, and domains with well-defined, stable GUI elements.
- Conclusions: CLI support enables GPA integration with shell scripts and composition of multiple workflows for more complex use cases.This extends the framework’s use as a command-line computer-use skill.
- Limitations: GPA cannot reason or make decisions during record-and-replay, so it cannot adapt when a workflow requires judgment about the current interface state.Date pickers work correctly only when selecting the same date used during recording.
- Future work: Future extensions include LLM-based workflow recording and self-healing, along with precondition tracking and software-state estimation for tool use.These directions address stale workflows and unsatisfied tool-call preconditions.
A.1 Locality Weight Bandwidth
GPA adapts locality weighting to the spatial distribution of context nodes using a target-centric RMS distance scale. Dense interfaces receive narrow locality, while sparse or edge-positioned targets receive broader locality.
- Locality weighting: Each context node contributes to the log-likelihood according to a locality weight whose decay depends on distance from the target.The bandwidth controls how rapidly distant nodes are down-weighted.
- Adaptive bandwidth: Dense interfaces make distant nodes less reliable, whereas sparse interfaces allow distant nodes to provide useful geometric constraints.The locality bandwidth must therefore adapt to the spatial distribution of nodes.
- Bandwidth estimation: Silverman’s rule uses RMS distance from the target to context-node centres as the bandwidth scale estimate.This makes the scale target-centric rather than merely measuring context-node spread.
- Examples: In a dense 20-node UI, RMS distance is approximately 70 px and locality bandwidth approximately 41 px, so mainly nearby nodes contribute.The reported calculation is σloc ≈1.06 × 70 × 20^-0.2 ≈41 px.
- Examples: In a sparse three-node UI, RMS distance is approximately 500 px and locality bandwidth approximately 425 px, allowing distant nodes to retain meaningful weight.A target at the edge of a node cluster also increases RMS distance and broadens the effective locality.
A.2 Entropy-Based Ambiguity Detection
GPA uses entropy over ranked candidate similarities to detect ambiguity before deciding whether to bypass SMC. Direct matching proceeds only when the top candidate is both strong and clearly separated from alternatives.
- Entropy normalization: The effective candidate count keff prevents weak tail candidates from artificially lowering normalized entropy.When only two candidates carry meaningful probability mass, keff remains 2 regardless of the raw candidate count.
- Temperature and examples: At τ = 0.02, scores [0.95, 0.40, 0.30] yield H ≈0.00, whereas near-tied candidates yield H ≈0.84–0.90.The examples classify the first distribution as unambiguous and the latter distributions as ambiguous.
- Ambiguity measure: Entropy measures ambiguity across the ranked candidate distribution rather than relying only on the top-two score gap.It accounts for the number and relative magnitudes of close competitors and overall distribution peakedness.
- Decision rule: The fast path requires s1 > smin and H < Hthr, with defaults smin = 0.9 and Hthr = 0.5.Only when both conditions hold does GPA skip the SMC particle filter.
A.3 Confidence Scoring
GPA combines likelihood confidence from context-node matches with spatial confidence from the predicted localization distribution. The likelihood component evaluates match quality relative to an appearance ceiling and a missing-node baseline, while nearby nodes receive greater weight.
- Confidence composition: Final confidence is the product C = ˜p(Z | θ) × Cspatial, combining complementary retrieval-quality measures.The multiplicative design requires both context agreement and spatial concentration to be high.
- Likelihood confidence: Per-node confidence normalizes joint geometry-and-appearance matching between the missing-node baseline and the best achievable appearance score.It approaches 1 for near-perfect geometry and becomes 0 when matching is no better than the missing baseline.
- Locality weighting: Overall likelihood confidence is a locality-weighted average that gives more influence to nearby nodes.A high value indicates geometric consistency across context matches, while a low value signals that expected matches are not found near the prediction.
A.3.2 Spatial Confidence
GPA estimates spatial confidence from the concentration of SMC particles and combines it multiplicatively with context-match likelihood. Tempered SMC uses adaptive annealing, resampling, and MCMC rejuvenation to handle peaked or multimodal localization posteriors.
- Spatial confidence: Spatial confidence measures how tightly particles agree on the target location independently of context-candidate likelihood.Tight convergence produces high spatial confidence, while wide scatter reduces it.
- Adaptive radius: The acceptance radius grows with context-node density scale using r = rbase + α · σloc, with rbase = 50 px and α = 0.2.This avoids over-penalizing wider particle spread in sparse interfaces.
- Failure-mode separation: The multiplicative confidence separates observation mismatch from posterior uncertainty: either low context likelihood or low spatial concentration keeps final confidence low.High context likelihood with scattered particles and tight convergence at a mismatched position are both treated as low-confidence cases.
- SMC algorithm: Tempered SMC gradually transitions from the prior to the true posterior by increasing β while adaptively maintaining effective sample size.Each stage performs importance reweighting, resampling, and MCMC rejuvenation to reduce particle degeneracy and improve mixing between modes.
- SMC algorithm: The sampler can stop when confidence is sufficient or stable, then output the densest particle-cluster mean as the predicted click position.The MH random walk scales steps with particle spread and retains a post-tempering floor to merge surviving sub-clusters.
- Context graph: The UI graph represents detected elements as nodes and connects spatially nearby elements, allowing neighboring context to support matching across UI states.Edges are created with k-nearest neighbors, typically k = 5, based on element centers.
B.2 Similarity Computation
GPA combines text and icon evidence for node similarity, then models scale and geometric tolerance to support matching across resized interfaces. The scale prior represents both unchanged and proportionally rescaled windows.
- Similarity computation: Textual node similarity combines fuzzy text matching and icon embedding similarity as s(vd, vc) = 0.9 · stext(vd, vc) + 0.1 · cos(ed, ec).Fuzzy matching tolerates OCR errors, while exact text matches receive a similarity of 1.0.
- Similarity computation: For pure icons or images, GPA uses only icon embedding similarity: s(vd, vc) = cos(ed, ec).This avoids applying text similarity where no textual content is available.
- Scale prior: The scale prior is a mixture of identity and ratio log-normal components, covering unchanged windows and full proportional rescaling.Per-axis ratios allow independent horizontal and vertical resizing, with symmetric penalties for shrinking and expanding.
- Geometric tolerance: Geometric tolerance accounts for context-node distance and element size, becoming stricter for small elements and more tolerant for large or distant ones.This makes spatial constraints reflect the precision of the available context evidence.
C Precheck Pipeline
The precheck pipeline speculatively processes upcoming workflow steps during environment idle time and reuses sufficiently confident cached results. Workflows generated from a single demonstration store ordered actions, parameterizable variables, metadata, and per-step execution data.
- Precheck Pipeline: The pipeline processes upcoming steps in a background thread while the environment executes the current action.This exploits settling time such as page loads or animations.
- Precheck Pipeline: When the runner finishes a step, it submits the observation, dispatches the action, and later retrieves completed precheck results with collect().The runner uses a cached result for the next step when its confidence is sufficiently high.
- Workflow Representation: A single demonstration produces a workflow template containing metadata, ordered natural-language action descriptions, and parameterizable variables with placeholder defaults.Variables such as {{recipient_email}} are extracted during building and substituted at execution time.
- Workflow Representation: The stored workflow separates configuration and step data, including variable descriptions, action types, target elements, coordinates, embeddings, window bounds, pauses, and active application names.The example references files such as workflow.yaml and steps_data.json and records per-step data by UUID.