PaperlayerView exact revision on arXiv

Source-linked AI summary

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Eric Wang

arXiv:2406.19263v2cs.CLcs.CV

TL;DR

GUI understanding work has largely overlooked screen reading based on user-indicated points, despite its relevance to accessible technology. ToL builds a Hierarchical Layout Tree and generates content and layout descriptions; it outperforms the reported baselines on description accuracy and shows utility in identifying incorrect mobile-navigation actions.

  • Problem

    GUI-understanding work has largely overlooked point-based screen reading, a task the paper identifies as important for accessible technology and users with visual impairments.

    Source: §1 · p. 1

  • Method

    ToL uses a screenshot and indicated point to build a Hierarchical Layout Tree, then generates descriptions of the pointed region’s content and related layout.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    ToL outperforms all reported baselines in content and layout description accuracy and demonstrates utility in identifying incorrect actions in mobile navigation trajectories.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported results indicate ToL’s potential for assisting visually impaired users with digital devices and for identifying incorrect mobile-agent actions.

    Source: §6 · p. 9

    GPT-4o server delays account for more than 80% of end-to-end evaluation time, and its expense motivates replacing it with a more efficient locally hosted model.

    Source: §7 · p. 9

Abstract

from arXiv · show

Graphical User Interfaces (GUIs) are central to our interaction with digital devices and growing efforts have been made to build models for various GUI understanding tasks. However, these efforts largely overlook an important GUI-referring task: screen reading based on user-indicated points, which we name the Screen Point-and-Read (ScreenPR) task. Currently, this task is predominantly handled by rigid accessible screen reading tools, in great need of new models driven by advancements in Multimodal Large Language Models (MLLMs). In this paper, we propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism, to address the ScreenPR task. Based on the input point coordinate and the corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout Tree. Based on the tree, our ToL agent not only comprehends the content of the indicated area but also articulates the layout and spatial relationships between elements. Such layout information is crucial for accurately interpreting information on the screen, distinguishing our ToL agent from other screen reading tools. We also thoroughly evaluate the ToL agent against other baselines on a newly proposed ScreenPR benchmark, which includes GUIs from mobile, web, and operating systems. Last but not least, we test the ToL agent on mobile GUI navigation tasks, demonstrating its utility in identifying incorrect actions along the path of agent execution trajectories. Code and data: https://screen-point-and-read.github.io

1 Introduction

Existing GUI-understanding work overlooks point-based screen reading, a task important for accessibility. The Tree-of-Lens agent combines content and layout descriptions, and is evaluated on ScreenPR and mobile navigation trajectories.

1 Introduction

  • The Screen Point-and-Read task takes a user-indicated point and screenshot as input and returns a description of the pointed region.The paper frames this overlooked GUI-referring task as important for accessible technology, including assistance for users with visual impairments.Source: §1 · p. 1
  • ToL is designed to describe both the content of a pointed region and related screen-layout information, including spatial relationships between elements.The paper notes that content alone may not distinguish identical items, such as the two “Tumbler pack” examples.Source: §1 · p. 2
  • ToL constructs a Hierarchical Layout Tree from screenshot regions, then uses the target path to create visual lens prompts for an MLLM.The tree is built with an object detector trained on the Android Screen Hierarchical Layout dataset, containing 50k bounding boxes of hierarchical regions from Android screenshots.Source: §1 · p. 2
  • ToL achieves the best performance against general and GUI-specialized MLLMs, improving content-description accuracy by over 15% and layout-description accuracy by over 30% versus vanilla textprompting GPT-4o.The ScreenPR benchmark includes 650 screenshots and 1,500 target points from web, mobile, and operating-system GUIs.Source: §1 · p. 2
  • On mobile navigation trajectories, ToL demonstrates utility in identifying incorrect actions along the execution path.The paper presents this as actionable feedback for refining mobile-agent navigation decisions.Source: §1 · p. 2
  • The authors introduce the ScreenPR benchmark to evaluate descriptions that include both content and layout details.The benchmark is part of their evaluation of ToL against other baselines.Source: §1 · p. 2

2 Related Works

Prior GUI research includes general MLLMs, task-specific GUI systems, and models addressing several grounding and referring tasks. The paper situates ToL as a layout-aware ScreenPR approach using a chain of lenses inspired by visual reasoning work.

2 Related Works

  • GUI MLLM work includes task-specific systems for navigation and referring-expression comprehension, alongside models addressing multiple grounding and referring tasks.CogAgent and Ferret-UI are examples of multi-task GUI systems discussed in this context.Source: §2 · p. 2
  • The ToL pipeline builds a Hierarchical Layout Tree from detected global and local regions, then sends target-path lenses to GPT-4o for content and layout descriptions.The lenses use various field widths as visual prompts.Source: Figure 2 · p. 3
  • ToL focuses on ScreenPR and distinguishes itself from GUI agents with referring abilities by generating layout-aware descriptions.The paper connects this focus to challenges MLLMs face in understanding complex image layouts.Source: §2 · p. 3
  • Vision reasoning work includes Chain-of-Look, See-Think-Confirm, and dynamic multi-turn visual processing; ToL applies a chain of varying-width lenses to GUI referring.The lens sequence represents GUI hierarchy and simulates a sequence of attention.Source: §2 · p. 3

3 Tree-of-Lens Agent

The ToL agent addresses point-based GUI reading by mapping a user-indicated location to content and layout descriptions. It builds a hierarchical layout tree and uses target-path lenses to prompt an MLLM.

3.1 Task Definition

  • The ScreenPR task takes a GUI screenshot and user-indicated point, then describes the targeted region’s content and related screen layout.Source: §3.1 · p. 3

3.2 Method Overview

  • The ToL agent uses screenshots and MLLMs across GUI domains to produce natural-language descriptions of a target’s content, function, and position in the layout.Source: §3.2 · pp. 3–4

3.3 Hierarchical Layout Tree Construction

  • The Hierarchical Layout Tree has a fixed three-layer structure: the whole screenshot, broader global regions, and local regions such as buttons, text fields, and icons.Source: §3.3 · p. 4
  • A DINO detector with a ResNet-50 backbone predicts local and global regions, which are linked by highest IoU to construct each tree.Source: §3.3 · p. 4
  • ASHL contains around 52k labeled boxes from 2,180 screenshots across 18 Android apps; source-code collection prunes invisible regions and merges nodes with IoU > 0.9.Source: §3.3 · p. 4

3.4 Target Path Selection and Multi-lense Prompting

  • Target-path selection finds the smallest local region containing the point and its parent global region, then presents a focused lens followed by a full-screen lens.Source: §3.4 · p. 5
  • The curated lenses and target-point coordinates are sent to GPT-4o with instructions to explain the boxes’ relationships and their location on the screen.Source: §3.4 · p. 5

4 Screen Point-and-Read Benchmark

The ScreenPR benchmark covers point-indicated regions across diverse GUI screenshots, with local regions ranging widely in area. It combines human ratings and automated cycle-consistency checks to evaluate generated content and layout descriptions.

4.1 Overview

  • 650 diverse-domain screenshots pair manually verified target points with local regions and content descriptions; Figure 4 shows point locations and local-region areas spanning 1% to over 50% of the screen.The target points are distributed to cover every screen location.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Data Collection

  • Mobile and web screenshots come from Screenspot, while operating-system screenshots come from OSWorld videos; annotators draw region boxes and manually correct GPT-4o-generated crop captions.Three student annotators draw the local-region boxes; data quality is checked with scripts and manual review.Source: §4.2 · pp. 5–6

4.3 Evaluation

  • Human judges rate content description, layout articulation, and overall preference for screen-reader descriptions shown with a screenshot and indicated point.Four response options map to scores of 100%, 66%, 33%, and 0%.Source: §4.3 · p. 6
  • For automated content evaluation, GPT-4V selects the target region from four human-annotated GUI crops using the screen reader’s generated content description.The four candidate crops contain only one target region.Source: §4.3 · p. 6
  • For automated layout evaluation, GPT-4 selects the positional relationship between two non-overlapping regions using their generated layout descriptions.Fixed choices include positional relations such as upper, lower, and left; accuracy is calculated against the regions’ ground-truth locations.Source: §4.3 · p. 6

5 Experiments

On ScreenPR, ToL leads the reported baselines in content and layout description accuracy, and its grounding components contribute to that performance. The experiments also test generalizability and use ToL descriptions to verify mobile navigation actions.

5.1 Main Results

  • ToL achieves the best content and layout description accuracy on ScreenPR, with over 15% improvement over GPT-4o in the reported evaluation.The table covers human evaluation, cycle consistency, and content-description language similarity.Source: Table 2 · p. 7, §5.1 · p. 7

5 Experiments

  • The GUI region detector trained on ASHL achieves 94.1% Average Precision and 95.9% Average Recall.Training uses a 90%/10% ASHL train/evaluation split, four NVIDIA A6000 GPUs, batch size 8, and 90 epochs.Source: §5 · pp. 6–7

5.1 Main Results

  • Human judges most favor ToL, while language-similarity score differences are relatively indistinguishable and LlaVA-NeXT receives slightly higher overall human scores than CogAgent.The authors suggest LlaVA-NeXT’s longer, more comprehensive-feeling descriptions may explain the human preference.Source: §5.1 · p. 7
  • ToL built on Qwen-VL-Plus and Gemini Pro Vision consistently outperforms each standalone MLLM in cycle consistency evaluation.The authors present this as evidence of the approach’s effectiveness and generalizability.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Ablation Study

  • Removing multi-lens prompting, the red dot, or tree-derived region marks degrades content and layout accuracy, with removing local and global marks causing a significant decrease.The ablations use cycle consistency evaluation on ScreenPR.Source: Table 4 · p. 7, §5.2 · p. 8
  • On a 20% random ScreenPR sample, ground-truth local or global regions narrow the content-accuracy gap toward human performance, but the layout gap remains and points to the MLLM as a bottleneck.The authors relate the layout finding to MLLM difficulty understanding multipanel image layouts (Fan et al., 2024).Source: §5.2 · p. 8, Table 5 · p. 8

5.3 Broader Application: Verification of Mobile Navigation Agent Actions

  • Across 10 end-to-end evaluations of 52 trajectories and 209 actions, ToL verification recognizes 92% of correct actions and identifies about 38% of incorrect actions.It detects 44% ± 0.04 of 20 incorrect execution-loop actions; Table 7 reports the best F1 score and execution-loop detection among the compared verification methods.Source: §5.3 · pp. 8–9, Table 7 · p. 9

6 Conclusion

The conclusion presents ToL as a layout-aware agent for ScreenPR and reports that it outperforms the baselines in content and layout description accuracy. It also describes its use in identifying incorrect actions by a mobile navigation agent.

6 Conclusion

  • ToL provides layout-aware descriptions of pointed screenshot regions for the ScreenPR task, which the authors frame as important for accessible technology.The paper reports that ToL outperforms all evaluated baselines in content and layout description accuracy.Source: §6 · p. 9
  • The authors report that ToL shows potential for assisting visually impaired users with digital devices.Source: §6 · p. 9
  • In a mobile GUI navigation application, ToL is used to identify incorrect actions along an agent’s execution trajectory.The conclusion describes the results as promising.Source: §6 · p. 9

7 Limitations

The authors identify latency, expense, potential harmful generated content, and the need to balance efficiency with semantic representation as limitations for application. The appendix also presents strong and weak agent-output examples across GUI domains.

7 Limitations

  • GPT-4o server delays occupy more than 80% of end-to-end evaluation, alongside high expense, making real-world application a non-trivial issue.The authors say a more efficient, locally hosted model is needed in place of GPT-4o.Source: §7 · p. 9
  • Because ToL relies on an MLLM to generate descriptions, those descriptions may contain harmful content, for which the authors say post-processing may be needed.Source: §7 · p. 9
  • The authors identify balancing application efficiency with improved semantic representation as a challenge for enabling on-device support and expanding ToL’s application areas.They characterize the current work as research-focused, with application-wise exploration still pending.Source: §7 · p. 9

A Example outputs of the ToL Agent

B Examples of the Cycle Consistency Evaluation

The appendix illustrates cycle-consistency evaluation for content and layout descriptions and identifies the prompts used by baseline models. The examples show how GPT-4o evaluates generated descriptions against choices or positions.

B Examples of the Cycle Consistency Evaluation

C Cross-evaluation Experiments with Cycle Consistency Evaluation

Under cross-evaluation, ToL consistently outperforms vanilla MLLM baselines, including when the evaluator differs from the model being tested. The section also describes prompt templates and illustrates action verification on a mobile task.

C Cross-evaluation Experiments with Cycle Consistency Evaluation

  • ToL consistently outperforms vanilla MLLM baselines when evaluation uses an auxiliary model different from the model under test.The reported cycle-consistency result includes GPT-based comparisons and a cross-evaluation setting with GPT-4o- and Gemini Pro Vision-based models.Source: C Cross-evaluation Experiments with… · p. 12

D Text Prompt templates

  • The ToL prompt uses normalized input-point coordinates substituted for {{x}} and {{y}} in its template.The GPT-4o prompt template is shown in Figure 12.Source: D Text Prompt templates · p. 12

E.1 Example

  • For a planned “Tap Settings” action, ToL describes the target regions and GPT-4o judges whether the action matches the instruction.The example concerns the second step of a task to pause a historical track in YouTube.Source: §E.1 · p. 12

E.2 Sample Strategy of Mobile Navigation Action Execution Trajectory

The mobile-navigation experiment selects low-completion trajectories across Android apps, then uses ToL region descriptions and action history to help GPT judge whether actions follow instructions. The pipeline uses point- and input-region-specific region selection, while the authors note that its validation goals remain high-level.

E.2 Sample Strategy of Mobile Navigation Action Execution Trajectory

  • Trajectories with task completion rates below 50% were selected across 11 Android apps spanning eight application domains.The sampled instructions came from WeWeb and were chosen to have little overlap with ASHL.Source: §E.2 · pp. 12, 14

E.3 Action verification Pipeline Details

  • For each trajectory step, ToL describes the target point and screenshot, including prior steps and the currently planned interaction.The descriptions are concatenated and sent to GPT-4 to assess whether actions follow the task goal.Source: §E.3 · p. 14
  • For clicks, ToL selects the smallest local and global regions containing the point; for input actions, it selects regions using separate IoU thresholds.The click-region detections are extended by 50 pixels to improve the accuracy of smaller local regions; input-region thresholds are greater than 0.4 locally and 0.1 globally.Source: §E.3 · p. 14
  • GPT-4o judges actions using current and historical action-region descriptions and the instruction, with repetition discouraging continuation and instruction alignment allowing it.Its response includes a yes-or-no decision and a reason.Source: §E.3 · p. 15
  • The action-validation setup gives the agent two high-level goals and expects it to infer details from ToL information; finer-grained validation is left for future work.The authors say that further fine-grained work could improve performance, though it is not their current focus.Source: §E.3 · p. 15

F Human Evaluation

Human judges rated ScreenPR descriptions for content, layout articulation, and overall preference. The reported ratings show that approximately 81% of samples received at least two identical judgments, with a standard deviation of 19.4%.

F Human Evaluation

  • Approximately 81% of samples received at least two identical ratings from the three human judges, with a standard deviation of 19.4%.Judges rated content description, layout articulation, and overall preference.Source: F Human Evaluation · pp. 15–16

G GUI Region Detection Model

The GUI region detector is trained for 90 epochs using COCO-pretrained weights, and validation precision shows convergence after epoch 70. Trained only on mobile data, it also shows generalization to unseen OS and web screenshots.

G GUI Region Detection Model

  • The detector uses COCO-pretrained weights and is trained for 90 epochs with a MultiStepLR scheduler stepping every 30 epochs at γ = 0.1.The reported loss at the end of training is 2.5803; Figure 10 shows the training loss.Source: G GUI Region Detection Model · p. 16
  • The detector’s validation-set average precision metrics show a clear convergence trend after epoch 70.Table 9 reports average precision by IoU and area; the passage does not give the metric values.Source: G GUI Region Detection Model · p. 16, Table 9 · p. 17
  • Trained only on mobile data, the model shows generalization ability to unseen operating-system and web domains.The evaluation uses unseen OS, web, and mobile screenshots; global regions were manually annotated for 20% of randomly selected ScreenPR data.Source: G GUI Region Detection Model · p. 16, Figure 17 · p. 19
  • Table 10 compares generated local- and global-region accuracy against human annotations at IoU@0.5, with small local regions noted as challenging.The passage provides the comparison’s scope but no accuracy values.Source: Table 10 · p. 17

Related papers

Loading 2406.19263v2…