Source-linked AI summary
SPECTRA: On-Device Cognitive Perturbation and Trajectory Analysis for Autonomous Edge-Cloud GUI Grounding
Zhan Qu, Hui Zang, Ran Chen, Tao Wang, Shengyu Zhang
TL;DR
Real-time GUI grounding calls for reliable task allocation between edge agents and the cloud, but dense interfaces can expose limits in edge-agent grounding. SPECTRA stress-tests salient visual anchors and analyzes hidden-state trajectory divergence to assess cloud requests without autoregressive decoding. Its two reported agent pairs retain 93.44% and 95.60% of cloud-only performance, with average request rates of 37.58% and 39.24%.
Problem
GUI grounding must balance reliable target selection with minimizing cloud dependency, while real-time interaction motivates deployment on edge devices.
Source: §1 · p. 2, §3.1 · p. 3Method
SPECTRA perturbs salient visual regions and analyzes hidden-state cognitive trajectories and their divergence to assess cloud requests without autoregressive decoding.
Source: §1 · p. 2, §3.2.1 · p. 4Results
93.44% and 95.60% of cloud-only performance are maintained by the GTA1-32B+InfiGUI-G1-3B and GTA1-32B+Holo1.5-3B pairs, with average request rates of 37.58% and 39.24%, respectively.
Source: §1 · p. 2Takeaways & Limitations
SPECTRA assesses cloud requests before autoregressive decoding, avoiding costly decoding computation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
The effectiveness of edge-cloud collaboration for GUI grounding depends on autonomous requesting, where the edge agent selectively offloads complex tasks to the powerful cloud. However, in visually dense scenarios, lightweight edge agents often exhibit overconfident hallucinations, leading to a misalignment between confidence and accuracy that hinders reliable autonomous requesting. To address this, we leverage the observation that an agent's cognitive instability leads to significant latent drift under minute perturbations due to steep decision boundaries. We propose SPECTRA, a lightweight autonomous request framework for edge-cloud GUI grounding, comprising (1) Saliency-Guided Targeted Perturbation and (2) Efficient Cognitive Trajectory Analysis. SPECTRA conducts a visual cognitive stress test by injecting masks into critical visual anchors and quantifies the topological divergence of the agent's high-dimensional cognitive trajectories during the prefill phase, avoiding inefficient output decoding. Experiments demonstrate that SPECTRA performs cloud request assessment without autoregressive decoding. Our GTA1-32B+InfiGUI-G1-3B and GTA1-32B+Holo1.5-3B maintain 93.44% and 95.60% of cloud-only performance with average request rates of 37.58% and 39.24%, respectively.
1 Introduction
Dense GUI distractors can make edge agents confidently select the wrong targets, undermining autonomous cloud requests. SPECTRA stress-tests visual grounding and analyzes hidden-state trajectories without autoregressive decoding; its reported agent pairs retain 93.44% and 95.60% of cloud-only performance at average request rates of 37.58% and 39.24%.
1 Introduction
- Dense GUI distractors can make edge agents overconfidently select visually similar elements, misaligning confidence with accuracy and hindering reliable autonomous requests.Examples include tightly packed icons and structurally homogeneous list items.Source: §1 · p. 2
- Fragile visual anchoring creates steep target–distractor decision boundaries, so slight visual perturbations can cause prediction inconsistency and spatial coordinate drift.Capturing output-coordinate dispersion can require multi-round decoding, whose computational cost is unacceptable for edge devices.Source: §1 · p. 2
- SPECTRA injects masks into critical visual anchors, samples hidden states as cognitive trajectories, and measures their topological divergence to trigger cloud requests without autoregressive decoding.The framework uses saliency-guided targeted perturbation and efficient cognitive trajectory analysis.Source: §1 · p. 2
- 93.44% and 95.60% of cloud-only performance are maintained by the GTA1-32B+InfiGUI-G1-3B and GTA1-32B+Holo1.5-3B pairs, with average request rates of 37.58% and 39.24%, respectively.The evaluations demonstrate request assessment without autoregressive decoding.Source: §1 · p. 2
2 Related Work
Prior work grounds GUI instructions in screen coordinates and deploys MLLM agents through visual perception, while edge-cloud collaboration seeks cloud-level performance with edge-level latency. SPECTRA's overview combines targeted visual stress tests with trajectory analysis for request decisions.
2.1 GUI Agents and Grounding
- GUI grounding maps natural-language instructions to target-element coordinates; SPECTRA's overview masks critical visual anchors and uses trajectory divergence to choose local action or cloud requests.Edge agents can lack fine-grained discrimination in visually dense GUIs, while MLLM-based agents interact through visual perception.Source: §2.1 · p. 2, Figure 2 · p. 3
2.2 Edge-Cloud Collaboration
- Edge-cloud collaboration seeks cloud-level performance with edge-level latency, but reliable autonomous requests remain challenging because output uncertainty can misrepresent correctness and repeated decoding adds overhead.Prior approaches include single-pass confidence or entropy estimates, multi-sample consistency checks, and probing internal states; existing state-probing approaches lack adaptation to visual-spatial GUI tasks.Source: §2.2 · p. 3
3 Method
SPECTRA frames edge-cloud GUI grounding as reliability-constrained request scheduling, then uses targeted visual perturbations and latent trajectory analysis to assess when to offload. Its predictor operates on parallel prefill trajectories rather than decoded outputs.
3.1 Problem Formulation
- GUI grounding maps screenshot instructions to target coordinates; collaboration schedules a low-latency, limited-capacity edge agent against a stronger, higher-cost cloud agent under a reliability-constrained request-rate objective.Source: §3.1 · p. 3
3.2 SPECTRA Design
- SPECTRA uses targeted visual stress tests and cognitive-trajectory divergence to decide whether difficult GUI tasks should be offloaded.Source: §3.2 · p. 3
3.2.1 Saliency-Guided Targeted Perturbation.
- It scores visual-token hidden states by L2 norm, smooths saliency across neighboring patches, and uses saliency-weighted independent masks to stress-test likely anchors.Source: §3.2.1 · p. 4
3.2.2 Efficient Cognitive Trajectory Analysis.
- Parallel prefill perturbations yield last-token, penultimate-layer vectors; a permutation-invariant transformer compares their pairwise relations to estimate latent dispersion without decoding.Source: §3.2.2 · pp. 4–5
- A supervised predictor learns cloud-request labels from edge correctness, then thresholds its instability score; the threshold is validation-calibrated and can be raised under cloud load.Source: §3.2.2 · p. 5
4.1 Experimental Settings
The evaluation covers three GUI grounding benchmarks, uses request-decision and collaborative-performance metrics, compares uncertainty-estimation baselines, and specifies the agents and training setup.
4.1.1 Datasets.
- MMBench-GUI, ScreenSpot-Pro, and UI-I2E-Bench test cross-platform interaction, fine-grained grounding, and implicit-instruction understanding, respectively.Together, they cover heterogeneous layouts, dense distractors, and non-intuitive descriptions.Source: §4.1.1 · p. 5
4.1.2 Evaluation Metrics.
- AUC and SRCC assess request decisions, while AUCG measures normalized collaborative gain across request rates.Higher AUCG indicates better accuracy retention at lower cloud costs.Source: §4.1.2 · pp. 5–6
4.1.3 Compared Baselines.
- The baselines span random scores, single-pass LN-Confidence, LN-Entropy, and CoT, plus multi-sampling EigenScore and Self-Consistency.Self-Consistency uses the spatial standard deviation of multiple coordinate predictions sampled at temperature 0.4 as its uncertainty score.Source: §4.1.3 · p. 6
4.1.4 Implementation Details.
- Experiments pair GTA1-32B with InfiGUI-G1-3B or Holo1.5-3B and apply perturbations to visual-encoder layers 7, 15, and 23 and the cross-modal merger.The trajectory predictor is a transformer encoder; training uses about 7.3k hybrid samples split 70%/15%/15%, with 100 epochs, AdamW, learning rate 1e-5, and batch size 32.Source: §4.1.4 · p. 6
4.2 Main Results and Analysis
SPECTRA shows strong request and collaborative performance across GUI benchmarks while limiting cloud requests and assessment overhead. Its on-device costs remain modest, and it assesses requests without extra token decoding.
4.2.1 Request Decision Performance.
- On MMBench-GUI with InfiGUI-G1-3B, SPECTRA surpasses the best baseline by 11.46% in AUC and 17.62% in SRCC.LN-Entropy performs well on ScreenSpot-Pro with Holo1.5-3B, but drops on semantically complex UI-I2E-Bench; Self-Consistency can leave overconfident hallucinations undetected.Source: §4.2.1 · pp. 6–7
- SPECTRA achieves the highest AUCG on MMBench-GUI and UI-I2E-Bench, while maintaining high efficacy across benchmarks.Collaborative gain depends on both request decisions and the cloud agent’s accuracy on offloaded tasks; on UI-I2E-Bench with Holo1.5-3B, LN-Entropy reaches 60.55% and Self-Consistency 48.26%.Source: §4.2.2 · p. 7
4.2.2 Collaborative Performance.
- 93.44% and 95.60% of cloud-only performance are maintained by GTA1-32B+InfiGUI-G1-3B and GTA1-32B+Holo1.5-3B, with average request rates of 37.58% and 39.24%, respectively.On iOS, InfiGUI-G1-3B achieves 86.80% accuracy and SPECTRA requests the cloud for 7.19% of tasks; on the OS subset of ScreenSpot-Pro, edge accuracy is 33.67% and the request rate is 70.27%.Source: §4.2.3 · p. 7
4.2.3 Performance across Domains.
- Assessment during prefill reduces edge latency by over 97% in the cloud execution path and incurs zero extra token decoding costs.On InfiGUI-G1-3B, multi-sampling requires generating over 660 extra tokens before deciding whether to request the cloud.Source: §4.2.4 · p. 7
4.2.4 Cost Analysis.
- At N=5, mean latency increases by only 36.7% relative to N=1; complete edge deployment occupies approximately 7.97 GB of VRAM.The trajectory predictor adds approximately 1.9 ms of latency and 46 MB of VRAM.Source: §4.2.4 · p. 7
4.3 Visualization Results
The visualizations show how targeted masks emphasize interactive GUI regions and how perturbation makes successful and failed cognitive trajectories more distinguishable.
4.3.1 Targeted Perturbation.
- Figure 5 shows the intermediate steps of targeted perturbation, with high-norm regions around dense interface elements and masks covering key controls rather than irrelevant backgrounds.The sliding window smooths discrete high-activation points while preserving the global GUI layout.Source: §4.3.1 · pp. 7–8
4.3.2 Cognitive Trajectories.
- After perturbation, failed samples are more dispersed than successful samples, whose trajectories remain relatively clustered.Without perturbation, the distributions differ but their boundary remains blurred; the predictor uses potential semantic separation and, more critically, amplified topological divergence.Source: §4.3.2 · p. 8
4.4 Ablation Study
The ablations find that saliency-guided perturbation, trajectory modeling, and extraction choices each affect request-decision performance.
4.4 Ablation Study
- Removing perturbation drops AUC by 3.48%, while removing the sliding window drops AUC by 2.06%; Gaussian noise, random masking, and inverse saliency also underperform.These MMBench-GUI ablations indicate that static hidden states alone are insufficient and that spatial continuity and visual priors matter.Source: §4.4.1 · p. 8, §4.4 · p. 8
4.4.1 Impact of Perturbation Strategy.
- Replacing the trajectory predictor with mean-pooled or concatenated trajectories followed by an MLP reduces AUC by 4.99% and 7.13%, respectively.The authors attribute the difference to self-attention modeling pairwise interactions among perturbed trajectories, while mean pooling obscures divergence and concatenation lacks explicit relational modeling.Source: §4.4.2 · p. 8
4.4.3 Impact of Trajectory Extraction.
- First- and middle-layer extraction yields AUCs of 58.21% and 66.85%; the last layer is 0.76% below the penultimate layer, and last-token aggregation exceeds average pooling by 3.41% in AUC.The authors suggest that the penultimate layer retains richer semantics than the next-token-specialized last layer, while the last token aggregates critical cross-modal semantics better than average pooling.Source: §4.4.3 · p. 8
5 Conclusion
SPECTRA addresses overconfident hallucinations by edge agents in visually dense GUI scenarios, where confidence and accuracy can be misaligned. It uses masked visual stress tests and prefill-phase trajectory analysis, and experiments report accurate request decisions without autoregressive decoding.
5 Conclusion
- SPECTRA targets the mismatch between edge agents’ confidence and accuracy in visually dense GUI scenarios.Source: §5 · p. 8
- Saliency-guided binary masks stress-test identified visual anchors, and trajectory analysis measures latent cognitive divergence during prefill.Source: §5 · p. 8
- Experiments across multiple benchmarks report superior request-decision accuracy without autoregressive decoding.Source: §5 · p. 8