PaperlayerView exact revision on arXiv

Source-linked AI summary

Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

Qijia Chen, Giulio Jacucci

arXiv:2608.21794v1cs.CLcs.AIcs.HC

TL;DR

GUI grounding evaluations may treat embedding similarity as semantic evidence even when UI metadata exposes the target’s visible label. The paper audits this issue by ranking same-screen elements across three benchmarks with lexical controls and five sentence encoders, finding substantial lexical coupling and limited deployable fusion gains.

  • Problem

    The paper addresses whether instruction–element embedding similarity measures semantic GUI grounding when candidate metadata re-exposes visible target labels.

  • Method

    The paper evaluates same-screen candidate ranking across AndroidControl, MoTIF, and Mind2Web using five off-the-shelf encoders, lexical baselines, label stratification, perturbations, LCC, and ESH.

  • Results

    Across three corpora, no off-the-shelf encoder consistently beats BM25 at R@1; target-text masking lowers R@1 by 22–38 percentage points, and deployable fusion captures only a small fraction of oracle headroom.

  • Takeaways & Limitations

    Embedding-based GUI evaluations should report lexical baselines, label-type stratification, target-text dependence, candidate-pool buckets, and deployable fusion separately from oracle headroom.

  • Takeaways & Limitations

    The protocol covers text-metadata candidate-ranking evaluations, not screenshot-, OCR-, or layout-only visual grounding without additional perception controls.

Abstract

from arXiv · show

GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.

1 Introduction

The paper audits whether instruction–element embedding similarity measures semantic GUI grounding or often recovers visible labels. Across three corpora, it compares encoders with lexical controls and finds strong lexical coupling in top-1 performance.

  • GUI grounding is evaluated as ranking same-screen UI elements from natural-language instructions and textual metadata.
  • High instruction–element similarity can reflect visible-label overlap rather than semantic inference from task or screen context.
  • The audit combines lexical baselines, label-type stratification, target-text perturbation, LCC, and ESH to distinguish lexical recovery from semantic grounding.
  • None of five off-the-shelf encoders consistently outperforms BM25 at R@1 across AndroidControl, MoTIF, and Mind2Web.
  • Masking exposed target text reduces R@1 by 22–38 percentage points, while encoder hit@1 is predictable from lexical rank, label type, and candidate-pool size with AUCs of 0.82–0.88.

2 Related Work

The paper connects GUI grounding evaluation to construct-validity audits and information retrieval, asking whether embedding scores contain evidence beyond visible-label recovery. It positions the work as a metric-side audit of text-metadata candidate ranking.

  • GUI grounding benchmarks test mapping natural-language instructions to UI elements, whereas this paper audits what their similarity scores measure.
  • Construct-validity work motivates checking whether scores carry the intended construct rather than relying on lexical-overlap heuristics or annotation artifacts.
  • In information retrieval, BM25 is a canonical lexical reference that should be reported alongside dense retrievers before dense similarity is treated as semantic evidence.
  • GUI candidate texts re-expose labels used to describe targets, making lexical matching both a valid grounding route and a confound for interpreting embedding similarity.
  • The work differs from model-side robustness and annotation-side cleanup audits by holding model family and benchmark fixed while auditing the evaluation metric.

3 Method

The method treats each grounded action as a same-screen candidate-ranking problem and compares lexical retrieval with five sentence encoders across mobile and web corpora. It stratifies targets by label availability and measures lexical coupling and fusion headroom with LCC and ESH.

  • Datasets and candidate pools: The datasets include AndroidControl, MoTIF, and Mind2Web, with label taxonomies distinguishing readable, metadata-only, and label-poor targets.
  • Candidate-ranking protocol: Each grounded step forms a query and same-screen candidate set, ranks the recorded positive element, and reports R@1, R@5, R@10, and MRR.
  • Datasets and candidate pools: Queries use step instructions for AndroidControl, episode goals for MoTIF, and task plus preceding action history for Mind2Web.
  • Candidate textualization: Candidate strings combine visible text and accessibility descriptions, with resource-id or class-name fallbacks when readable labels are absent.
  • Retrievers: BM25 and Jaccard provide lexical controls, while five ℓ2-normalised single-vector encoders rank candidates by cosine similarity.
  • Fusion: The oracle min-rank combiner estimates target-aware headroom, whereas true reciprocal-rank fusion provides the deployable comparison.
  • Diagnostics: LCC uses a regularised logistic regression of encoder top-1 hits on lexical rank, candidate-pool size, and label type; ESH summarises oracle and deployable fusion gains.

4 Results

Across three corpora, lexical matching remains a strong top-1 baseline, while label availability and lexical overlap substantially shape embedding-based GUI grounding results. Fine-tuning improves ranking, but supervised gains and apparent semantic recovery remain dependent on exposed target text.

  • AndroidControl: BM25 obtains R@1=0.557 and MRR=0.632 on AndroidControl clean-main, exceeding every sentence-embedding baseline on both metrics.The comparison uses 32,646 steps and a random-ranking floor of R@1=0.022.
  • AndroidControl: BGE-large, E5-large-v2, and Qwen3-Embedding-0.6B trail BM25 by 3.7, 5.0, and 7.6 pp on R@1, respectively.Embeddings perform better further down the list: BGE-large reaches R@5=0.724 and R@10=0.785.
  • Label availability: All text-only methods remain weak on label-poor targets, while resource-only rows reach only R@1=0.124 under Qwen3.Masking exposed target text reduces R@1 by 22–38 percentage points on an equal-allocation stratified sample.
  • Cross-corpus generalisation: On MoTIF and Mind2Web, no off-the-shelf embedding significantly beats BM25 at top-1, and the low-label weak regime recurs across both corpora.MoTIF’s label-poor rows achieve R@1=0.039–0.073 against random 0.042, while Mind2Web id/class-only or unlabelled rows reach R@1 ≤0.005.
  • Lexical coupling and fusion: Encoder top-1 hits remain strongly predictable from lexical rank, candidate-pool size, and label type, with LCC values of 0.835–0.882 across major encoders.Oracle ESH indicates complementary signal exists, but deployable fusion captures much less of that headroom.
  • Fine-tuning control: Fine-tuned MiniLM reaches R@1=0.694 and recovers 37.4% of BM25-miss rows, yet its LCC remains 0.877 and R@1 falls to 0.151 after target-text masking.Task supervision improves ranking without removing lexical coupling or dependence on exposed target text.
  • Lexical coupling and fusion: The min-rank oracle reaches R@1=0.706, whereas the largest deployable fusion gain is BM25 + Jaccard at R@1=0.578, only +0.023 over BM25.Oracle headroom therefore substantially overstates deployable fusion gains.

5 Discussion

The discussion argues that embedding similarity alone is insufficient evidence of semantic GUI grounding because lexical label recovery explains much of top-1 performance. It identifies limited deployable fusion benefits and recommends controls for evaluating and designing GUI grounding benchmarks.

  • Interpretation: Off-the-shelf sentence-embedding similarity is not a sufficient grounding proxy without lexical controls.Its top-1 behaviour is largely predictable from lexical rank and candidate structure, while deployable fusion gains remain below target-aware oracle headroom.
  • Interpretation: LCC(BGE-large) remains in [0.85, 0.88] across mobile and web, indicating stable lexical coupling across GUI candidate spaces.This supports interpreting high embedding scores cautiously when instructions name targets using visible words re-exposed in UI metadata.
  • Interpretation: Removing only the target candidate’s visible label drops R@1 by 22–38 percentage points across BM25 and embedding methods.The drop occurs on an equal-allocation stratified subset, showing target text dependence in the ranking evaluation.
  • Residual signal: Embeddings recover ∼15% of BM25-miss rows, but standard RRF captures only a small part of this residual signal.Aggregate combiners can lose top-1 accuracy, termed destructive fusion, because lexically coupled retrievers may disagree mainly on non-target candidates.
  • Deployment: For deployed GUI agents, the paper recommends lexical retrieval followed by semantic or multimodal reranking, or task-specific fine-tuning, rather than embeddings as the sole top-1 ranker.The same comparison against text-only lexical retrieval is recommended before multimodal improvements count as visual grounding.
  • Reporting: Evaluations should report matched lexical baselines, label-type and candidate-pool stratification, target-text perturbations, and deployable fusion separately from oracle headroom.These controls help distinguish label-explicit cases from label-poor cases where exposed candidate text is insufficient.
  • Benchmark design: Future benchmarks should annotate direct target-label mentions, balance label-explicit and context-dependent cases, and collect paired instructions with and without direct label mentions.Such native pairs complement post-hoc masking by separating direct label recovery from broader grounding.

Limitations

The protocol is scoped to text-metadata candidate-ranking evaluations and does not establish claims about visual grounding or downstream behavioral difficulty. Its conclusions concern off-the-shelf single-vector encoders, while stronger representations and learned combiners remain open directions.

  • Scope: The protocol covers enumerable UI-element candidate pools represented by accessibility-tree or DOM text, not screenshot-, OCR-, or layout-only visual grounding.Those settings require additional perception controls.
  • Scope: The paper bounds its claim to grounding-metric validity rather than downstream behavioral-difficulty prediction.Step-level embedding distances do not predict local repair actions on AndroidControl, where repair signals are sparse by design.
  • Model and fusion scope: The conclusions apply to off-the-shelf single-vector sentence encoders under candidate-ranking evaluation.Richer learned combiners remain future work because tested fusion methods leave most oracle headroom unrealised.
  • Future directions: One epoch of contrastive fine-tuning substantially improves a 22M encoder, so stronger visual representations or end-to-end agent training may change absolute performance.The measurement recommendation remains to interpret such gains relative to lexical controls and label-type breakdowns.

A Artifact use, license, and data statement

The release provides reproducible derived panels and analysis code without redistributing original datasets or checkpoints. Free-text fields are removed to address third-party content exposure, while the evaluation uses public corpora, checkpoints, and defined preprocessing choices.

  • Artifact use: The study uses five public research datasets, five public sentence-embedding checkpoints, and one public sparse-retrieval checkpoint for their released evaluation purposes.The datasets include AndroidControl, MoTIF, Mind2Web, AndroidControl-Curated, and STS-B; the sparse checkpoint is SPLADE.
  • License and release: The project releases derived per-step evaluation panels, analysis scripts, and summary reports under the MIT License, but not the original datasets or checkpoints.Users must obtain raw inputs from official sources.
  • Data statement: Free-text columns are stripped from released panels because source screens may contain third-party content such as email addresses.Join keys and numeric/categorical fields remain, and the paper states that numeric results are reproducible from the de-texted panels.
  • Preprocessing: AndroidControl preprocessing parses accessibility trees for tap, long-press, and text steps and enumerates visible UI elements with non-empty bounds.The target is resolved as the smallest-area visible element containing the action coordinate.
  • Preprocessing: The ranking pipeline uses lowercased alphanumeric tokens with a short UI stoplist for queries and candidates.This avoids boosting lexical baselines with generic UI verbs or class names shared by construction.

D Statistical inference and implementation details

The evaluation fixes model, ranking, tie-breaking, and bootstrap procedures to make cross-corpus comparisons reproducible. It also documents the coarse per-screen BM25 setting and the stratified target-label perturbation used to test text dependence.

  • Implementation: The encoders are off-the-shelf checkpoints, while BM25 uses k1=1.5, b=0.75 and reciprocal-rank fusion uses k=60.The models span MiniLM, MPNet, BGE, E5, and Qwen3-Embedding-0.6B; no encoder hyperparameters are tuned for zero-shot evaluation.
  • Inference: All paired comparisons use 2,000-resample cluster bootstraps with corpus-specific episode or annotation units and percentile 95% confidence intervals.This accounts for dependence between consecutive steps within the same task.
  • Reproducibility: Ranking ties are resolved with deterministic hash-based jitter keyed by episode, step, retriever, and candidate identifiers.Fixed seeds and released panels make rank assignment reproducible across subsets.
  • Lexical baseline: Per-screen BM25 estimates IDF from a median of 62 candidates, making IDF coarse but not responsible for the lexical advantage.In the uninformative-IDF limit, BM25 becomes length-normalised token overlap, and Jaccard tracks it within a few R@1 points on every corpus.
  • Diagnostic design: The target-text ablation blanks only the gold candidate’s text and content description across 1,000 TEXT_ONLY, 1,000 CD_ONLY, and 1,000 BOTH rows.The query, candidate pool, non-target candidates, resource ID, and class name remain fixed; the resulting average equally weights strata.

I Oracle min-rank vs. true RRF: full table

Target-aware oracle fusion substantially overstates what deployable rank fusion achieves. Simple calibrated fusion recovers only limited signal, while fine-tuning improves recovery without removing the observed lexical and target-text dependence.

  • Oracle versus deployable fusion: +0.151 is the maximum oracle R@1 lift over BM25, whereas true RRF with the same information can lose R@1.Oracle min-rank uses privileged target rank; true RRF combines per-candidate ranks and is deployable.
  • Oracle versus deployable fusion: The largest deployable top-1 lift comes from the lexical BM25 + Jaccard pair.Single-embedding RRF combinations with BGE, E5, and Qwen3 provide only small deployable gains.
  • Calibrated fusion: Validation-tuned score fusion gains +2.27 pp for MiniLM, +2.81 pp for MPNet, +1.46 pp for E5, and +1.80 pp for Qwen3 on held-out test data.BGE selects α=1.00, reducing exactly to BM25 with no held-out change; α selection is limited by the n=261 validation subset.
  • Fine-tuning: Fine-tuning uses one contrastive epoch with same-screen hard negatives and evaluates on 2,949 held-out AndroidControl clean-main rows.The protocol mirrors zero-shot per-screen candidate ranking.
  • Coupling audit: Fine-tuning improves target recovery, including BM25 misses, but hit outcomes remain predictable from lexical and structural controls.Removing the gold candidate’s exposed text sharply reduces top-1 accuracy, so improved ranking remains text-dependent.

L Construct correlation: M3 converges to Jaccard

The paper defines M3 as a step-level instruction–target cosine distance and compares its coupling with lexical Jaccard similarity across textualisation variants and encoder families. M3 closely tracks Jaccard on GUI data, while standard semantic-similarity evaluation shows weaker embedding–Jaccard agreement.

  • Construct definition: M3 is the step-level instruction–target cosine distance used to assess alignment under a sentence encoder.M1 is the corresponding episode-level analogue, but the construct here focuses on per-step alignment.
  • Construct correlation: M3 and token-level Jaccard distance correlate strongly across all five encoder families and three textualisation variants, with Spearman values of 0.68–0.82.Action-aware textualisation produces the tightest coupling for MiniLM, MPNet, and BGE-large; current textualisation is marginally tighter for E5-large.
  • Construct correlation: Model-to-model agreement is higher than embedding–Jaccard coupling, reaching pairwise Spearman values of 0.80–0.91 for the same textualisation variant.This indicates that encoder families often produce similarly ordered step-level distances even when their agreement with lexical similarity is lower.
  • Cross-domain comparison: On STS-B, all five encoders beat Jaccard against gold similarity by 16–24 points, while embedding–Jaccard agreement is 0.608–0.749.The corresponding AndroidControl clean-strict coupling range is 0.68–0.82.

N Behavioural validity boundary

The behavioural analysis tests whether alignment metrics predict local and episode-level friction proxies. Step-level M3 shows essentially null local associations, whereas full M1 has small, bounded positive aggregate associations that remain below 0.15 after controls.

  • Step-level validity: Step-level friction labels mark whether either of the next two actions is back, wait, or scroll, with robustness variants for terminal exits and broader transitions.The analysis reports marginal correlations, decile contrasts, and within-episode comparisons of high- and low-M3 steps.
  • Step-level validity: Current M3 correlations with back, non-terminal back-out, and combined friction are near zero on the no-duplicate subset of 26,832 steps.The reported Spearman values are −0.008, −0.007, and −0.009, respectively.
  • Step-level validity: The lexical Jaccard baseline is slightly larger but remains in the same direction, with combined-friction Spearman of −0.020 [−0.031, −0.008].This supports the conclusion that the step-local null is not specific to embeddings.
  • Episode-level validity: M1 is computed from an episode goal and concatenated prospective step instructions, using MiniLM-L6 and controls for episode length, instruction diversity, and lexical distance.The metric is bounded to grounding-metric validity rather than behavioural prediction.
  • Episode-level validity: Full M1 shows small positive aggregate associations: marginal Spearman is +0.092 for back frequency and +0.124 for scroll frequency.After controls, the corresponding values are +0.102 and +0.038, all below 0.15.
  • Scope boundary: Typing-correction events occur in only 82 instances across 63 of 15,283 episodes, or 0.4% episode incidence.Their sparsity prevents using them as a friction proxy in this dataset.

O Qualitative error analysis

The qualitative analysis distinguishes lexical label recovery, residual embedding recovery, and label-poor failure across de-identified examples. Additional retrieval controls show that the measurement diagnosis persists beyond the original encoder set.

  • Error regimes: Table 15 organizes examples into lexical label recovery, residual embedding recovery, and label-poor failure regimes.Ranks and label types are preserved while text fields are paraphrased, and qualitative rank summaries include BM25 and representative encoders.
  • Lexical label recovery: A visible DELETE target can be recovered lexically because the query and target expose the same label, although some encoders rank other candidates higher.BM25 and E5 rank the target first, while MiniLM and MPNet rank it 21st and BGE 11th.
  • Residual embedding recovery: An AndroidControl query for app preferences recovers SETTINGS through a close lexical or semantic variant that BM25 misses.MiniLM ranks the target first, while BM25 ranks it 145th.
  • Label-poor failure: A generic resource id lacking an item title and visible affordance causes all text rankers to miss an overflow-icon target.Ranks are poor across BM25 and encoders, including 85 for BM25 and 41 for E5.
  • Lexical label recovery: A time/weather target with matching visible text is recovered by lexical matching, while one embedding ranker is diverted to another time-related candidate.The example shows that even visible-label recovery can be disrupted by embedding similarity.
  • Residual embedding recovery: In a MoTIF shopping example, dense encoders rank first a text-labelled target that BM25 misses, illustrating residual signal beyond lexical retrieval.BM25 ranks the target 38th, while MiniLM, MPNet, BGE, and E5 each rank it first.
  • Retrieval stress tests: Qwen3 remains below BM25 at R@1 across the listed AndroidControl, MoTIF, and Mind2Web evaluations, while SPLADE is tied with BM25 on Mind2Web.Both remain predictable from lexical and candidate controls, preserving the broader diagnosis.
Loading 2608.21794v1…