PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

Yuhao Wang, Mu Qiao, Xindong Zhang, Yunzhi Zhuge, Lei Zhang, Huchuan Lu

arXiv:2609.10297v1cs.CV

TL;DR

Reusable GUI screenshot caches make pruning an irreversible decision before future targets are known, while visual processing grows across trajectories. TRACE orders evidence using layout, instruction relevance, and novelty, repairs spatial coverage, and contracts retired frames; across six benchmarks, it ranks first on four and second on two.

  • Problem

    With cache reuse, pruning commits visual evidence before later targets are known, while tight budgets also require preserving spatial coverage.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    TRACE combines a layout-derived interaction prior with instruction relevance and feature novelty in a nested evidence order, repairs coverage with native tokens, and monotonically contracts retired frames.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Across six GUI benchmarks, TRACE ranks first on four and second on Mind2Web and AndroidControl.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    In reusable GUI lifecycles, visual pruning must commit an evidence order before future grounding demands are known while preserving spatial coverage.

    Source: §5 · p. 9

    Under aggressive context reduction, errors often reflect insufficient localization detail, and thin or low-contrast widgets remain difficult to distinguish from background.

    Source: §C.1 · p. 40

Abstract

from arXiv · show

GUI agents accumulate high-resolution screenshots as the trajectory unfolds, increasing inference latency and memory usage. Training-free visual token pruning can reduce this cost, but cache reuse introduces a fundamental constraint. Once tokens are discarded, the corresponding visual evidence cannot be recovered without re-encoding. Pruning therefore becomes an \textit{irreversible admission decision} that must remain useful for unknown future targets while preserving coverage of operable regions under tight budgets. To address these challenges, we propose \textbf{\method{}}, a training-free framework for \emph{\textbf{T}rajectory-\textbf{r}obust \textbf{A}dmission and \textbf{C}overage-aware \textbf{E}vidence ordering}. Specifically, we combine a query-independent layout-derived interaction prior with instruction relevance and feature novelty to rank visual evidence according to both potential future utility and diversity. Then, we reserve part of the budget for native visual tokens distributed across the screen, repairing missing spatial coverage without breaking the ordering. Together, these mechanisms produce a nested token order, allowing retained visual evidence to shrink monotonically across budgets while remaining reusable throughout the trajectory. Finally, our monotone KV contraction incrementally contracts retired frames into compact session state, avoiding repeated visual encoding or pruning. Extensive experiments across six GUI benchmarks and diverse models verify the effectiveness of our proposed \method{} under tight budgets. The source code will be released.

1 Introduction

Cache reuse reduces repeated GUI-frame computation but makes token pruning an irreversible write-time commitment: retained evidence must remain useful for unknown future targets and preserve spatial coverage. TRACE addresses this with a nested evidence order and monotone contraction across a trajectory.

1 Introduction

  • Cache reuse removes repeated prefill but leaves within-frame redundancy, making pruning an irreversible write-time commitment whose historical keep sets must nest within current ones.Dropped tokens cannot serve future queries without re-encoding the frame.Source: §1 · pp. 1–2
  • Because future targets are unknown and importance-biased keeps can leave operable regions unsupported, admission needs both a query-independent layout prior and spatial coverage under tight budgets.The introduction contrasts instruction-selected tokens that can miss later targets with layout-selected tokens that retain coverage.Source: §1 · p. 2, Figure 1 · p. 2
  • TRACE combines LIP’s layout-derived interaction prior, NEO’s nested order blending prior, relevance, and novelty, NCR’s native-token coverage repair, and MKC’s monotone retirement contraction.Together, the components commit evidence once and contract it monotonically thereafter.Source: §1 · p. 2
  • MKC contracts each retiring frame to its history prefix and replays retained native rows in the next prefill, avoiding re-encoding historical frames.This makes the admitted order reusable as session state throughout the trajectory.Source: §1 · p. 2
  • Experiments across six GUI benchmarks span single-step and multi-step settings, multiple model scales, and a cross-family backbone.The introduction reports these evaluations as verifying TRACE’s effectiveness, without giving result values here.Source: §1 · p. 2

2 Related Work

Prior GUI-agent and visual-token efficiency methods reduce computation through learned perception, pruning, or merging, but can incur training costs or fail to preserve reusable, spatially covered evidence. TRACE instead admits a nested keep and contracts it as monotone session state.

2.1 Efficient GUI Agents

  • Training-based GUI methods learn cheaper perception but require extra training and do not transfer across architectures, motivating frozen-model input-side pruning and cache-side compression.Examples include CogAgent (Hong et al., 2024), ReVision (Abaskohi et al., 2026), AQuaUI (Li et al., 2026b), GUI-KV (Huang et al., 2025), and ST-Lite (Zhou et al., 2026).Source: §2.1 · p. 3
  • GUI cache methods re-score keeps at each step, so adjacent keeps may not nest and retired frames require re-encoding or re-ranking; TRACE admits a nested keep once and contracts it as monotone session state.Source: §2.1 · p. 3

2.2 Visual Token Pruning

  • Visual-token pruning ranks native tokens by attention, dispersion, or instruction relevance, while merging aggregates discarded patches into synthetic summaries.Examples include FastV, DivPrune (Alvar et al., 2025), CDPruner (Zhang et al., 2025b), VisionZip (Yang et al., 2025b), and PruMerge+ (Shang et al., 2025).Source: §2.2 · p. 3
  • Both pruning and merging selections drift across steps and require rows absent from contracted caches; neither is tailored to GUI streams, where salient-region collapse sacrifices dense-interface coverage.TRACE injects an interaction prior before prefill, repairs spatial collapse at admission, and contracts the admitted keep into monotone session state.Source: §2.2 · p. 3

3 Method

TRACE admits visual evidence once, then reuses it through nested pruning and cache contraction. It combines layout preference, instruction relevance, feature novelty, and native-token spatial coverage to support efficient GUI trajectories.

3.1 Problem Formulation

  • TRACE selects retained visual positions before the first prefill, then retires frames monotonically by removing evidence while preserving original tokens.Each frame is encoded once and contracts thereafter, making admission an irreversible decision under unknown future utility.Source: §3.1 · p. 3

3.2 Layout-derived Interaction Prior

  • LIP converts detected UI-box entropy, contrast, containment, and resonance into a soft token mass that favors layout-dense regions while keeping every token eligible.It normalizes per-token box energy into a spatial prior, then adds a unit baseline and scales the prior by α.Source: §3.2 · p. 4

3.3 Nested Evidence Ordering

  • NEO greedily orders tokens by instruction relevance and prior-weighted feature novelty, so each budget uses a nested prefix and retirement deletes a suffix.The interaction prior reweights normalized features; orthogonal residual selection balances novelty against relevance.Source: §3.3 · pp. 4–5

3.4 Native-token Coverage Repair

  • NCR repairs spatial gaps with native tokens, using stride-sampled coverage tokens for current frames and feature medoids from partitioned regions for history.Protected ordered prefixes retain the strongest evidence, while the remaining budget is allocated to coverage; history medoids remain nested within the current keep.Source: §3.4 · p. 5

3.5 Monotone KV Contraction

  • MKC retires a frame by indexing its stored embeddings down to the history prefix, then restores those rows in a merged forward pass without re-encoding them.Only the fresh frame is newly encoded at its current-frame budget.Source: §3.5 · p. 5

4 Experiments

TRACE combines interaction priors, ordered evidence selection, and coverage repair to preserve useful visual evidence across budgets and trajectories. Experiments report strong benchmark performance, module contributions, and reduced serving latency and visual KV cache.

4.1 Experimental Setup

  • Experiments use GUI-Owl-1.5 models and UI-TARS-1.5-7B across six benchmarks, measuring official accuracy for single-step tasks and Step SR for multi-step action prediction at matched visual-token budgets.Single-step budgets retain fraction r; multi-step budgets independently control current and historical tokens with c and h.Source: §4.1 · p. 6

4.2 Main Results

  • TRACE is best among existing methods, retaining 78.7% and 61.1% of dense performance on GUI-Owl-1.5-8B at mild and tight budgets.At r = 5%, it leads all three single-step benchmarks; its margin over PruneSID on SS-v2 is 20.99%, and over VisionTrim on SS-Pro is +2.08%.Source: §4.2 · pp. 6–7
  • At tight budgets, TRACE’s multi-step margins reach 1.30%–3.53%; on GUI-Owl-1.5-2B it retains 52.9% versus PruneSID’s 43.5% of dense performance on average.On UI-TARS-1.5-7B, TRACE is best at both budgets, with tight-budget margins over DivPrune of 0.79%–4.92%.Source: §4.2 · p. 7, Table 5 · p. 7, §4.3 · p. 9

4.3 Ablation Studies

  • On SS-v2 at r=10%, combining the prior and NEO reaches 56.45%, versus 32.63% for the top-k prior and 36.48% for NEO alone; NCR raises accuracy to 73.90%.Adding NCR also lifts tight-budget Mind2Web from 27.79% to 34.20%.Source: §4.3 · p. 7, Table 2 · p. 7
  • Removing any NEO scoring factor degrades accuracy on all four benchmarks, with prior removal costing up to −23.74% on SS-v2; altered ordering choices cost 3.61%–10.53% on two grounding benchmarks.The ordering tests doubled likelihood weight, log-mean-exp query pooling, and mean-row collapse.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7
  • Removing any LIP energy attribute lowers accuracy on both benchmarks, with containment and resonance contributing most; NCR stride tokens beat medoids on current-frame SS-v2 and SS-Pro.Stride tokens score 67.92% and 37.63%, versus medoids’ 58.65% and 30.11%; for history frames, stride costs 1.40% on OmniGUI, so medoids are used there.Source: Table 6 · p. 7, Table 7 · p. 7, §4.3 · p. 8
  • The prior improves four existing methods by +0.16% to +12.11%, but the best reaches only 34.36% versus TRACE’s 55.19%; without diversity, NEO spends budget on repeated glyphs.NCR reduces uncovered targets on most frames and rescues targets on 44 frames while losing them on 5; the prior alone scores 17.06%.Source: §4.3 · pp. 8–9, Figure 3 · p. 8
  • TRACE reduces TTFT from 1116.8 ms to 452.7 ms and visual KV cache from 697 MB to 292 MB, while three recent dense frames still use 90% of the dense budget under window truncation.TRACE is 2.4× faster than dense re-prefill; one dense frame still costs 1.66× TRACE, with no accuracy gain from window truncation.Source: §4.3 · p. 9, Figure 5 · p. 9

5 Conclusion … A.1 Lifecycle Contract and Method Space

TRACE treats reusable GUI visual pruning as an irreversible admission decision, combining layout priors, instruction relevance, novelty, native-token coverage repair, and monotone KV contraction. The appendix outlines the lifecycle contract and method space and points to implementation, protocol, and supplementary experimental analyses.

5 Conclusion

  • TRACE combines LIP, NEO, NCR, and MKC to create nested, coverage-aware visual-token admission and reusable session state without re-encoding retired frames.LIP derives a query-independent layout prior; NEO combines it with instruction relevance and feature novelty; NCR repairs spatial coverage with native tokens while preserving nesting.Source: §5 · p. 9
  • Because cache reuse makes pruning irreversible, TRACE orders evidence for unknown future grounding demands while preserving spatial coverage under tight budgets.The resulting nested order lets retained evidence shrink monotonically across budgets; MKC contracts retired frames into session state without re-encoding.Source: §5 · p. 9

APPENDIX

  • The appendix covers lifecycle behavior, benchmark breakdowns, ablations, serving efficiency, transfer to other backbones, and evidence scope and limitations.It also specifies lifecycle, implementation, comparison-protocol, and configuration details.Source: APPENDIX · p. 13

A.1 Lifecycle Contract and Method Space

  • The lifecycle contract requires pre-prefill keep decisions, nested retention, and later droppability of retained cache rows; Table 8 evaluates methods against these three conditions.The table scores official selection rules for the deletion-only serving path, not as a universal judgment on each method.Source: §A.1 · p. 13, Table 8 · p. 13
  • FastV fails pre-prefill and nesting conditions, merge-family methods fail cache-row droppability, and DivPrune, CDPruner, ZOO-Prune, and TRACE satisfy all three under their official rules.VisPruner and PruneSID receive △ for nesting because their quota or de-duplication rules nest only after a specified modification.Source: §A.1 · p. 13, Table 8 · p. 13
  • Prior families leave gaps across six setting axes, whereas TRACE covers multi-step GUI reuse, pre-language-model decisions, native rows, nested budgets, and no historical-frame re-encoding.Visual-token pruners lack GUI multi-step reuse; GUI input-side methods lack native rows or nested ordering; GUI cache-side methods re-score each step.Source: §A.1 · p. 13, Table 9 · p. 14

A.2 Method Specification

TRACE orders visual evidence for potential future utility and diversity, then repairs spatial coverage with native tokens so retained sets remain nested across budgets. Monotone cache contraction reuses encoded features while preserving token positions and aligning downstream states with the contracted context.

Layout-derived Interaction Prior

  • Removing containment is the costliest LIP attribute ablation (−2.91% and −1.45%), followed by resonance on ScreenSpot-v2 (−2.28%).The query-independent interaction prior uses detector boxes and combines entropy, contrast, containment, and resonance; the detector runs at admission, not during retirement.Source: Table 10 · p. 14, §A.2 · p. 14

Nested Evidence Ordering

  • TRACE combines within-frame instruction relevance with prior-weighted novelty and residual coverage into a nested token order; when relevance variance is zero, it falls back to prior-weighted residual selection.Admission forms the full-frame sequence, protects prefixes for current and history budgets, inserts native stride representatives and history medoids, then evicts eligible tail tokens to retain the budget.Source: §A.2 · pp. 14–15
  • Doubling the log-aj coefficient costs 10.53% on ScreenSpot-v2, log-mean-exp pooling costs 6.45% on ScreenSpot-Pro, and mean-row pooling costs 8.81% and 5.63% across the two suites.The adopted form uses a unit likelihood coefficient, max-pooling across query rows, and instruction-token rows in U.Source: Table 11 · p. 15, §A.2 · p. 15

Native-token Coverage Repair

  • History repair protects a prefix and partitions the raster-ordered complement into contiguous, near-equal regions, retaining each region’s native feature-space medoid to preserve nested history sets.A dynamic program minimizes the maximum within-region squared distance to the region mean using O(ghr) states and transitions, with preparation and medoid extraction costing O(L d̃), where d̃ ≤ 256.Source: §A.2 · pp. 15–16
  • On the current frame, stride reaches 67.92% on ScreenSpot-v2 and 37.63% on ScreenSpot-Pro, while history medoids reach 24.04% on OmniGUI versus 23.20% for stride.The method uses stride for current-frame spatial gaps but native feature-space medoids for the tighter history budget; on Mind2Web, the history-rule difference is 0.02%.Source: §A.2 · pp. 15–16

Monotone KV Contraction

  • Contraction preserves original multimodal RoPE positions for retained rows and recomputes rotary offsets for subsequent text from the surviving positions.It removes rows from each retired frame’s stored order while leaving text and action rows untouched before replaying the selected history tokens.Source: §A.2 · p. 16
  • On OmniGUI, contraction costs 6.4 ms per transition versus 90.8 ms to encode one screenshot, while reusing historical features and limiting visual-cache growth to O(kc + Hkh).Each transition encodes one new screenshot; retained historical frames use kh rows each, compared with N dense visual tokens, and are replayed in parallel during prefill.Source: §A.2 · p. 16
  • At 100% budget, context-consistent replay passes a byte-exact no-op test; at pruned budgets, it aligns downstream states with the contracted visual context without claiming dense-recomputation equivalence.Contraction truncates the cache at the retiring frame boundary, keeps selected native visual rows, and replays them with downstream text and action rows.Source: §A.2 · p. 17

A.3 Evaluation Protocol and Baseline Reproduction · A.4 Configuration Validation · A.4.1 Fixed-Configuration Check

The evaluation specifies reproducible baseline scoring and implementation checks, then tests TRACE’s configuration choices across budgets and tasks. The fixed-configuration check finds a small aggregate gap from per-benchmark selection, with larger differences in particular settings.

A.3 Evaluation Protocol and Baseline Reproduction

  • Nine training-free methods are compared with random and uniform pruning, and a step is correct only when both action type and target or argument are correct.Click and long-press points must lie within normalized Euclidean distance 0.04 of the gold coordinate; text and scroll arguments follow the reference protocol’s string and direction rules.Source: §A.3 · p. 17
  • All main-table experiments pass five implementation checks, while PyramidDrop and IVC-Prune are excluded because their nominal budgets do not yield comparable pruning.Checks cover byte-exact 100% no-op behavior, unit-test invariants, exact keep budgets, original-position consistency, and multi-step ledger reconstruction. PyramidDrop leaves shallow layers unpruned; IVC-Prune prunes late in the LLM, retaining dense vision encoding and most prefill computation.Source: §A.3 · p. 17
  • On ScreenSpot-v2 at r = 10%, tuned VisionTrim reaches 61.24%, while VisPruner remains 12.66% below TRACE at 73.90%.VisPruner improves from 46.07% to 58.81% as its importance ratio increases, and VisionTrim scores 44.26% → 58.65% → 61.24% as its DVTS share rises from 0.5 to 0.7 to 0.9. PruneSID scores 61.01% → 58.96% → 58.49% across NMS threshold scales 0.7, 1.0 (Official), 1.3; ZOO-Prune scores 57.47% → 56.29% → 55.90% across 16, 64 (Official), 256 directions.Source: §A.3 · pp. 17–18

A.4 Configuration Validation

  • TRACE selects a prior cap from {1, 2, 4, 6}, applies min(¯α, α⋆), and uses no prior for frames without detector support.The effective-token-count rule is used to calibrate prior strength; excessive α can concentrate evidence too narrowly and remove context needed by later actions.Source: §A.4 · p. 18
  • The unit term in mj(α) keeps every token eligible, while αNpj increases the mass of tokens in regions indicated by the interaction prior.Neff is N under uniform mass and approximately 1 when mass concentrates on one token; it reflects the effective token count rather than overall mass scale.Source: §A.4 · p. 18
  • TRACE selects current-frame dose ρcur from {0, 0.2, 0.3, 0.5} and, for multi-step runs with kh < kc, history dose ρhist from {0.05, 0.10, 0.20}.Each dataset-budget result uses one configuration selected from fixed public grids for prior cap, calibration mode, and doses.Source: §A.4 · p. 18

A.4.1 Fixed-Configuration Check

  • Across 16 benchmark-budget combinations, the selected configuration improves the median by +1.0% and beats the fixed setting in 14 combinations by less than +2%.Mind2Web Cross-Website gains 3.0% at c = 25%, while AndroidControl gains 4.1% at the tight budget; Table 12 compares the global and per-benchmark-budget configurations.Source: §A.4.1 · p. 18, Table 12 · p. 19

A.4.2 Benchmark and Budget Dependence · A.4.3 Prior and Coverage Interaction · A.4.4 Budget and Dose Sensitivity

The preferred interaction-prior strength depends on task and budget, while coverage dose and current-versus-history allocation determine whether retained evidence preserves useful spatial coverage. Across the tested settings, concentrated ordering and coverage repair address different failure modes, and excessive repair can erase the ordering.

A.4.2 Benchmark and Budget Dependence

  • Mind2Web Cross-Task Step SR rises from 43.68% to 46.86% across the prior-cap grid, while other tasks peak at different strengths, showing prior value depends on task and budget.The comparison spans candidate-based selection, coordinate grounding, and structured action prediction; Table 13 reports prior-strength sensitivity.Source: §A.4.2 · p. 19, Table 13 · p. 20
  • Candidate-based selection favors stronger caps, whereas coordinate grounding and structured action prediction generally favor milder caps that preserve surrounding context.MMBench-GUI peaks at an intermediate cap at r = 50%; ScreenSpot-v2 gains up to +2.7% with mild caps at the tightest budget, while OmniGUI and AndroidControl change little across caps.Source: §A.4.2 · p. 19, Table 13 · p. 20

A.4.3 Prior and Coverage Interaction

  • At tight budgets, a strong prior may already cover interactive regions, making extra repair compete for tokens; at milder budgets, a larger current dose can repair exposed regions.The prior concentrates evidence on likely interactive regions, while the current dose restores coverage the concentrated order leaves exposed.Source: §A.4.3 · p. 20, Table 14 · p. 20
  • The prior improves spatial grounding most on ScreenSpot-Pro, contributes little over uniform on AndroidControl, and mainly helps Mind2Web element grounding.An effective-support rule limits prior strength when concentration would remove too much context, favoring milder priors when later actions require more context.Source: §A.4.3 · p. 20, Figure 7 · p. 21
  • In the 1,000-frame OmniGUI dump, cap saturation is 18.4% of frames at the mild budget versus 70.3% at the tight budget; AndroidControl rises from 11.7% to 70.6%.These calibration diagnostics show the support rule permits stronger priors when retained context remains sufficient.Source: §A.4.3 · p. 20

A.4.4 Budget and Dose Sensitivity

  • On OmniGUI, halving the current budget costs 5.29%, versus 0.74% for halving history, supporting a larger current budget and smaller history budget.Most multi-step benchmarks use a light dose; sparse AndroidControl is an exception, requiring heavier repair on the current frame, while OmniGUI history-dose changes Step SR only slightly.Source: §A.4.4 · p. 21
  • On ScreenSpot-v2, moderate stride representatives improve greedy selection, but a full dose collapses the ordering to uniform sampling; alternative scan rules stay within 1.5% of the adopted dose.The gain comes from coverage repair itself rather than a particular scan rule; the current dose should repair coverage without removing the importance order.Source: §A.4.3 · p. 20, §A.4.4 · p. 21, Figure 8 · p. 21

A.4.5 Calibration and Robustness

TRACE remains usable with incomplete detections and the default instruction-likelihood temperature on ScreenSpot-v2 at r = 10%. Detector degradation causes a gradual accuracy decline, while moving the temperature away from τ = 1 can substantially reduce accuracy.

Detector degradation

  • With 25%, 50%, 75%, and 100% detector-box removal, accuracy falls from 73.90% to 68.55%, 64.15%, 55.82%, and 53.85%, respectively.With all detections removed, the prior becomes flat and selection relies on instruction likelihood, feature diversity, and coverage repair; accuracy remains 10.1% above uniform sampling.Source: §A.4.5 · p. 21, Figure 10 · p. 22

Instruction-likelihood temperature

  • At τ = 1, accuracy changes by only 0.9% relative to τ = 2, while τ = 0.5 and τ = 4 lose 10.5% and 7.1%, respectively.TRACE keeps τ = 1 as its default in every experiment without tuning it.Source: §A.4.5 · pp. 21–22, Figure 10 · p. 22

B Additional Experimental Evidence · B.1 Lifecycle Behavior

TRACE’s lifecycle evidence shows that its admitted visual evidence often retains later targets and can shrink to a nested history prefix without rebuilding the cache. The supplementary diagnostics also cover benchmark results, ablations, and serving measurements, though those findings are not detailed in these passages.

B Additional Experimental Evidence

  • The supplementary evidence is organized around lifecycle traces, benchmark tables, component ablations with keep visualizations, and serving measurements that separate cache reuse from selection cost.Each diagnostic addresses a different question; keep visualizations show spatial allocation, paired ablations measure score changes, and serving measurements report the evaluated execution path.Source: B Additional Experimental Evidence · p. 22

B.1 Lifecycle Behavior

  • On OmniGUI, independent re-selection requests uncached tokens on 99.5% and 99.1% of frames; forcing it onto surviving rows changes Step SR by −0.7% and −0.4%.At 65 tokens, fresh selection adds and drops 12 tokens relative to the committed prefix, and changes 23.5% of the surviving set over 1000 steps.Source: Figure 11 · p. 23, §B.1 · p. 23
  • TRACE retains 96.09% and 91.41% of later targets at mild and tight budgets; retirement admission error is 12.37% and 20.23%, versus about 72% for uniform.Existing methods perform near chance on later-target retention; the retirement comparison uses the same budgets.Source: §B.1 · p. 23
  • On ScreenSpot-v2 at r = 10%, TRACE achieves 82.5% grounding accuracy versus 40.0% without the prior; an OmniGUI episode contracts from about 50% to 10% with zero nesting violations.The grounding comparison covers 120 frames.Source: Figure 12 · p. 24, §B.1 · p. 24
  • Across the illustrated Alipay episode, the acted-on widget remains in each contracted row, preserving evidence from prior pages rather than only the latest screenshot.The six actions tap Me, Yu’ebao, Open, Participate, Accept, and Claim reward.Source: §B.1 · p. 24
  • For the displayed frames, the surviving prefix is a subset of the admitted keep, while released tokens form its complement; together, the panels reconstruct the admitted keep.The marked target remains in the survivor panel on the displayed frames; this does not imply every removed patch is irrelevant to all future actions.Source: §B.1 · pp. 24–25

B.2 Benchmark Breakdowns

The breakdowns compare benchmark accuracy and retained performance across target categories, action metrics, budgets, and model scales. TRACE’s relative advantage varies by benchmark, metric, and budget, so leading averages do not imply uniformly higher category-level or auxiliary scores.

Single-step Breakdown

  • At r = 10%, TRACE scored 73.90%, 37.63%, and 50.47% on ScreenSpot-v2, ScreenSpot-Pro, and MMBench-GUI, retaining 64.5% of dense performance on average.At r = 5%, TRACE ranked first on all three benchmark averages, although its scores fell to 55.19%, 20.30%, and 30.88%.Source: §B.2 · p. 26, Table 15 · p. 27

Multi-step Breakdown

  • Across 8B multi-step tasks, TRACE’s average retained performance fell from 93.0% to 80.4%, while its lead over the best published baseline widened on OmniGUI and Mind2Web.The leads grew from 0.74 to 2.99 points on OmniGUI and from 0.54 to 3.53 points on Mind2Web; on tight-budget AndroidControl, TRACE led ZOO-Prune by 1.30 points but uniform by 0.93 points.Source: §B.2 · p. 26, Figure 15 · p. 27, Table 16 · p. 28

Mind2Web Metric Breakdown

  • On Mind2Web, TRACE exceeded uniform in element accuracy and Step SR at both budgets, while operation F1 remained near-equal.At mild budgets, element accuracy and Step SR were 53.02% and 45.52% for TRACE versus 47.48% and 39.65% for uniform; at tight budgets, they were 39.41% and 34.20% versus 26.32% and 20.61%. Operation F1 was 84.28% versus 84.35% at mild budgets and 83.90% versus 83.49% at tight budgets.Source: §B.2 · p. 27, Table 16 · p. 28

Cross-scale Evaluation

  • On GUI-Owl-1.5-2B single-step grounding, TRACE ranked first on all three benchmark averages at r = 10% and 5%, retaining 48.0% and 28.2% of dense performance.Its ScreenSpot-Pro lead at r = 5% was only 0.25 points overall, and it trailed FastV on icon targets, showing that top benchmark rank did not mean a category-level lead.Source: §B.2 · pp. 27–28
  • On GUI-Owl-1.5-2B multi-step tasks, TRACE led OmniGUI at both budgets but trailed FastV on Mind2Web and PruneSID on AndroidControl.Its Step SR was 36.08%, 37.92%, and 56.53% at mild budgets, and 31.88%, 26.52%, and 53.72% at tight budgets, retaining 91.4% and 77.7% of dense performance on average.Source: §B.2 · p. 28

Rank stability

  • TRACE remained first on 2B ScreenSpot-v2 and MMBench-GUI at all four keep ratios, but pooled Mind2Web and AndroidControl ranks fell from first to fourth in plotted comparisons.On MMBench-GUI, full method ordering had Spearman ρ = 0.109 and 0.091 at r = 10% and 5%, although TRACE remained first.Source: §B.2 · p. 28

Target-category Breakdown

  • Target categories explain benchmark differences: at 8B ScreenSpot-Pro r = 25%, TRACE led VisionTrim on text targets but trailed on icons, while at r = 5% it led overall despite trailing on icons.At r = 5% on MMBench-GUI, TRACE also exceeded FastV in both Basic and Adv. categories, scoring 40.51% versus 25.46% and 21.36% versus 15.61%.Source: §B.2 · p. 28, Table 17 · p. 29

B.3 Ablations and Visual Evidence

This section tests how TRACE’s components affect task accuracy and examines how its layout prior and coverage repair shape retained visual evidence. The diagnostics also show where coverage can fail, especially for small targets.

Keep Maps

  • At r=10% on the same frame, TRACE keeps the target while both no-prior and uniform selections miss it.The visualization compares selection outcomes at a matched budget and links the layout prior to the resulting token allocation.Source: §B.3 · p. 29, Figure 17 · p. 32

Coverage Recall and Coupling

  • TRACE’s detector-proposal coverage recall is 90.06% on the 120-frame dump, while paired repair comparison records 44 rescued targets and 5 lost targets.Recall measures whether a detector-defined element retains at least one token, not coverage of every true interface element or task accuracy; repair reallocates a fixed budget, so some targets are lost.Source: §B.3 · pp. 30–31

Coverage Coupling

  • Adding repair raises tight-budget ScreenSpot-v2 accuracy from 38.36% to 55.19% and ScreenSpot-Pro accuracy from 28.15% to 37.63%.These comparisons are with the prior and ordering enabled.Source: §B.3 · p. 31

Ablation Summary

  • Prior removal costs 20.05 points on ScreenSpot-v2 at r=10%, 23.74 at r=5%, and 13.79 and 16.51 on ScreenSpot-Pro at r=25% and r=10%.Instruction-term removal costs 15.57 points on tight ScreenSpot-v2 and 14.17 on ScreenSpot-Pro; diversity removal costs 5.35 and 4.74 points, respectively. Removing NCR yields a 6.41-point Mind2Web delta.Source: §B.3 · pp. 31–32, Figure 18 · p. 32

MMBench Prior-strength Interaction

  • At r=25% on MMBench-GUI, scores rise from 71.06% to 72.45% across four prior strengths, with only 0.11 points gained from α=4 to α=6.The sweep varies prior weighting, not coverage repair; most of the gain occurs before the strongest setting.Source: §B.3 · pp. 32–33

Layout Prior Comparison

  • Even the best compatible shared-prior selector trails TRACE by 13.37 points at the mild budget and 20.83 at the tight budget on ScreenSpot-v2.Prior-only top-k scores are 32.63% and 17.06% at mild and tight budgets, 41.27 and 38.13 points below the complete method; shared layout information does not replace evidence ordering.Source: §B.3 · p. 33, Table 21 · p. 33, Figure 19 · p. 33

Keep Gallery and Budget Tightening

  • At r=10%, the keep gallery marks four examples HIT and the Notifications example MISS; at the tightest budget on another screen, TRACE retains 3 of 4 targets versus 1 for instruction-conditioned scoring.The Notifications control is tiny and insufficiently represented; the budget comparison holds the screen and targets fixed, and its counts are not dataset-wide accuracy rates.Source: §B.3 · pp. 33–35, Figure 20 · p. 34, Figure 21 · p. 35

Detector to Prior and Selector Comparison

  • On the detector-to-prior example, 64 regions cover 430 of 720 tokens, or about 60%, while selector comparisons show every displayed selection keeps the target.The detector produces a spatial mass field rather than a binary keep mask; ordering and coverage rules allocate the limited budget, and the selector visualization compares the surrounding footprint at matched token counts.Source: §B.3 · p. 35

B.4 Serving Efficiency · C Transfer and Scope

TRACE’s serving results show how selection cost and reusable visual state affect latency, while transfer experiments test the method on matched cache paths and a different backbone. Together, the results report transfer gains alongside benchmark-specific exceptions and budget-dependent outcomes.

B.4 Serving Efficiency

  • 453.9 ms TTFT and 2,295.8 ms end-to-end latency are TRACE’s lowest reported values; versus dense, it cuts them by 648.7 and 662.4 ms, with Step SR 48.91.TRACE incurs 67.1 ms of selection cost; the pruned methods report about 2,011 input tokens, and visual KV cache falls from 697 MB to 292 MB.Source: §B.4 · pp. 35–36, Table 22 · p. 37
  • 487.4 ms TTFT with MKC alone falls to 452.7 ms at the mild budget and 397.8 ms at the tight budget with the complete selector, while visual KV usage drops 58.1% and 68.1%.Adding repair shifts TTFT from 451.7 to 452.7 ms at mild budget and from 399.1 to 397.8 ms at tight budget, indicating negligible overhead.Source: §B.4 · p. 37, Table 23 · p. 37
  • 59.84%, 59.91%, and 60.20% are the mild-budget AndroidControl Step SR scores; at tight budget, scores are 55.45% at dose 0.3 and 56.58% at dose 0.5.The star denotes the nested history rung and is not another point on the current-dose-only curve.Source: §B.4 · p. 37, Figure 24 · p. 38
  • 5.1× is the dense curve’s TTFT relative to TRACE at step 5; TRACE latency still rises overall and fluctuates between steps.The within-episode OmniGUI plot does not independently vary history-window length; the widening gap is consistent with avoiding repeated encoding of retired frames.Source: §B.4 · p. 38, Figure 24 · p. 38

C Transfer and Scope

  • 0.88–9.62 points is TRACE’s lead over DivPrune, VisPruner, and ZOO-Prune across reported matched-budget settings on the nested MKC path.Because all methods use the same reusable cache path, the comparison reflects differences in the visual evidence admitted by their selection rules.Source: C Transfer and Scope · p. 38, Table 24 · p. 38
  • At reported retention settings, TRACE leads ScreenSpot-Pro at 25% and 5% with 16.76% and 3.04% average accuracy, but DivPrune leads MMBench-GUI and PruneSID leads several higher-retention settings.On ScreenSpot-v2, TRACE also leads at 25% and 10%; the transferred configuration’s scores change by 0.13–0.63 points with rankings unchanged.Source: C Transfer and Scope · p. 38, Table 25 · p. 39, Table 26 · p. 39
  • 1.05 and 1.18 points on OmniGUI and 2.80 and 4.92 points on Mind2Web are TRACE’s gains over the strongest baseline across two budgets.The OmniGUI comparison covers 1,521 of 2,572 steps with target locations; all methods use the same visual-context scope and UI-TARS native action grammar.Source: C Transfer and Scope · p. 39

C.1 Limitations and Failure Modes

TRACE’s limitations are that current benchmarks do not test the long temporal gaps where history reuse may help, and aggressive pruning can leave insufficient detail for precise localization or small, low-contrast widgets.

Trajectory Horizon

  • Evaluated episodes are short relative to TRACE’s five- or six-frame history horizon, and benchmarks rarely require retrieval across long frame intervals, so the benefit of longer reuse remains under-tested.Most existing GUI benchmarks retain at most two or three historical frames; systematic evaluation on longer trajectories is left for future work.Source: §C.1 · p. 39

Failure Modes

  • At r = 5%, TRACE misses 503 ScreenSpot-v2 examples solved by the unpruned model; 93.0% retain target overlap, while missing spatial evidence explains only 0.8% of the gap to dense.Most remaining errors occur when the target neighborhood is retained but lacks enough detail for precise localization.Source: §C.1 · pp. 39–40
  • Thin or low-contrast widgets fitting within one patch remain difficult to distinguish from the background after budget reduction; more patch detail or improved visual representations are more direct remedies than pruning.Figure 25 shows remaining failure cases with GUI-Owl-1.5-8B.Source: §C.1 · p. 40, Figure 25 · p. 40

Related papers

Loading 2609.10297v1…