PaperlayerView exact revision on arXiv

Source-linked AI summary

CogAgent: A Visual Language Model for GUI Agents

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang

arXiv:2312.08914v3cs.CV

TL;DR

GUI agents aim to automate tasks on digital interfaces, but existing visual-language models struggle with the tiny text and icons in high-resolution screens. CogAgent combines low- and high-resolution visual processing for GUI understanding and planning, achieving state-of-the-art results across GUI and visual question-answering benchmarks while reducing high-resolution compute costs.

  • Problem

    GUI automation requires recognizing tiny icons and text, while common low-resolution VLM inputs and the high computational cost of high-resolution images limit GUI understanding.

  • Method

    CogAgent combines a large low-resolution vision encoder with a small high-resolution cross-module to jointly model visual features efficiently.

  • Results

    CogAgent achieves state-of-the-art performance on nine visual question-answering benchmarks and tops GUI understanding and decision-making benchmarks including AITW and Mind2Web.

  • Takeaways & Limitations

    CogAgent demonstrates that a generalist VLM can outperform LLM-based GUI methods using extracted structured text while maintaining broad cross-modality performance.

Abstract

from arXiv · show

People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM, with a new version of CogAgent-9B-20241220 available at https://github.com/THUDM/CogAgent.

1. Introduction

CogAgent targets GUI-agent limitations that arise when language-only systems cannot directly represent visual and spatial interface information. It combines GUI-focused data and high-resolution processing, achieving strong GUI-navigation, VQA, and compute-efficiency results.

  • Motivation: GUI applications often lack standard APIs, contain visual and spatial information difficult to express in words, and include elements that HTML cannot parse.These constraints limit purely language-based agents in real-world GUI scenarios.
  • Contribution: CogAgent is a visual language foundation model specializing in GUI understanding and planning while retaining general cross-modality ability.It builds upon CogVLM as its base visual language model.
  • Contribution: GUI and OCR pre-training data address the distributional difference between GUI images and natural Web images.The authors construct a large-scale annotated GUI and OCR dataset for continual pre-training.
  • Results: CogAgent tops GUI understanding and decision-making benchmarks AITW and Mind2Web, outperforming LLM-based methods using extracted structured text.The reported result uses visual-agent evaluation on both representative GUI-navigation benchmarks.
  • Results: CogAgent achieves state-of-the-art generalist performance on nine visual question-answering benchmarks spanning general and text-rich tasks.The reported benchmarks include VQAv2, OK-VQA, TextVQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE.
  • Results: CogAgent-18B with 1120 × 1120 inputs uses less than half the FLOPs of CogVLM-17B with default 490 × 490 inputs.The separated high- and low-resolution branches reduce the compute cost of consuming high-resolution images.

2. Method

CogAgent extends a low-resolution VLM with a high-resolution cross-module designed for GUI text and tiny elements. The model combines parallel visual processing, cross-attention, GUI-focused pre-training, and instruction fine-tuning.

  • 2.1. Architecture: CogAgent uses CogVLM-17B as its base, retaining a low-resolution 224 × 224 branch and adding a high-resolution cross-attention module.The architecture combines a pre-trained VLM with a separate high-resolution input pathway.
  • 2.1. Architecture: Standard VLM resolutions of 224 or 490 pixels are insufficient for typical 720p-or-higher GUI screens, while high-resolution processing creates quadratic attention costs.The visual-token sequence and self-attention overhead grow with image resolution.
  • 2.1. Architecture: The high-resolution cross-module accepts 1120 × 1120 images, uses a smaller 0.30B-parameter vision encoder, and fuses its features with every decoder layer.The original decoder computations are retained while cross-attention integrates high-resolution features with hidden states.
  • 2.2. High-Resolution Cross-Module: The module supplements low-resolution features through residual cross-attention, allowing high-resolution information to be incorporated within a controlled computation budget.Cross-attention dimensions and head counts can be adjusted according to compute budget and model performance.
  • 2.3. Pre-training: The design targets high-resolution text recognition, grounding of text and objects, and specialized understanding of GUI imagery.Pre-training data are divided around these capabilities and include GUI grounding examples from the CCS400K screenshot dataset.
  • 2.4. Fine-tuning: Instruction fine-tuning uses more than two thousand human-annotated computer and mobile screenshots alongside Mind2Web and AITW data.Annotations cover screen elements, potential tasks, and operation methods in question-answering format.

3. Experiments

CogAgent is evaluated across foundational visual understanding and GUI-agent benchmarks for computers and smartphones. It achieves strong generalist VQA performance and outperforms language-based and visual-language baselines on GUI navigation tasks.

  • Foundational Visual Understanding: CogAgent achieves state-of-the-art generalist results on both general VQA benchmarks and 5 of 6 text-rich VQA benchmarks.It surpasses generalist competitors by +8.0 on TextVQA, +2.1 on ChartQA, +2.3 on InfoVQA, and +16.2 on DocVQA.
  • Foundational Visual Understanding: CogAgent scores 52.8 on MM-Vet, surpassing LLaVA-1.5 by +16.5, and reaches 85.9 on POPE-adversarial.These evaluations cover conversational-style multimodal question answering and hallucination assessment.
  • GUI Agent: Computer Interface: On Mind2Web, CogAgent surpasses LLaMA2-70B by 11.6%, 4.7%, and 6.6% on cross-website, cross-domain, and cross-task subsets.The benchmark uses webpage snapshots and prior actions to predict subsequent actions, while language-model baselines use cleansed HTML inputs.
  • GUI Agent: Smartphone Interface: On AITW, CogAgent achieves state-of-the-art performance and improves overall performance by +2.61 over the visual-language baseline Auto-UI.The comparison includes language-based methods using OCR and icon text representations and visual-language methods using screenshots.
  • GUI Agent: Smartphone Interface: AITW evaluation uses a unified model across Android subsets, with GoogleApps downsampled to 10% to reduce data imbalance.AITW contains 715k operation episodes spanning Android versions and device types.
  • GUI Agent: Smartphone Interface: Manual reassessment found that more than 40% of sampled apparent inaccuracies were actually correct alternative methods.The discrepancy reflects multiple valid pathways for accomplishing smartphone tasks.

4. Ablation Study

The ablations examine architecture, resolution, and pre-training data, showing that the high-resolution cross-module improves efficiency while preserving or improving performance, and that domain-specific data matters for GUI agents.

  • Model Architecture: High-resolution cross-modules keep computational overhead nearly linear with image-patch count, whereas the original CogVLM structure becomes dramatically more expensive at higher resolutions.At resolution 1120, the original structure can require more than 10 times the FLOPs of a cross-module design.
  • Model Architecture: More than 10× higher FLOPs occur for the original structure than for a cross-module at resolution 1120.The comparison uses FLOPs across resolutions 224, 490, 756, and 1120.
  • Model Architecture: Models with a high-resolution cross-module at resolution 756 use only half the computational resources of the original structure at resolution 490 while delivering significantly better performance.The ablation compares computational efficiency and model performance across architecture choices.
  • Model Architecture: Increasing acceptable resolution within a limited computational budget yields additional performance improvements.The cross-module permits resolution increases without proportionally increasing computational cost.
  • Pre-train Data: Captioning, OCR, GUI, and grounding data each broadly improve performance, with web and grounding data having a significant impact on Mind2Web.The data ablation sequentially adds OCR, then GUI and grounding data to image-caption pre-training.

5. Conclusion

CogAgent combines enhanced pre-training data with an efficient high-resolution architecture and achieves state-of-the-art results across VQA and GUI benchmarks, while retaining notable limitations.

  • Conclusion: CogAgent achieves state-of-the-art performance across a wide range of VQA and GUI benchmarks.The model is introduced as a VLM-based GUI agent with enhanced pre-training data construction and an efficient architecture for high-resolution input.
  • Conclusion: CogAgent still produces imprecise output coordinates and cannot process multiple images, motivating further research.These shortcomings are explicitly identified in the conclusion.

1. Details of Training Configurations

The paper reports detailed training settings and model configurations in dedicated tables.

  • Training Configurations: Training settings for pre-training and multi-task fine-tuning are reported in Table 7.The supplied passage identifies the table but does not provide its individual settings.
  • Training Configurations: CogAgent model configurations are reported in Table 8.The supplied passage identifies the table but does not provide the configuration values.

2. Details of Evaluation Datasets

The evaluation covers broad and text-rich VQA together with computer and Android GUI-agent benchmarks, using datasets that test visual reasoning, OCR, documents, charts, and UI action execution.

  • General VQA: General VQA evaluation includes VQAv2, OK-VQA, MM-Vet, and POPE, covering natural-image questions, external knowledge, integrated vision-language skills, and object hallucination.These benchmarks assess complementary generalist capabilities, including recognition, OCR, knowledge, spatial awareness, math, and hallucination detection.
  • Text-rich VQA: Text-rich VQA evaluation includes OCR-VQA, TextVQA, ST-VQA, ChartQA, InfoVQA, and DocVQA.These datasets require text recognition or reasoning over book covers, scene text, charts, infographics, and document images.
  • GUI Agent Benchmarks: Mind2Web evaluates web agents on real-world websites using open-ended tasks and annotated action sequences.Its instances contain task descriptions, action sequences, and webpage snapshots; actions pair target elements with operations such as clicking, typing, or selecting.
  • GUI Agent Benchmarks: Mind2Web reports step success rate on cross-website, cross-domain, and cross-task test sets, counting only exact target-element and operation matches as correct.The evaluation first selects a target element among top-k candidates, then predicts the operation.
  • GUI Agent Benchmarks: AITW evaluates Android device-control systems from natural-language instructions across 715k episodes, 30k instructions, four Android versions, and eight device types.AITW contains five subsets and directly evaluates manipulation of smartphone user interfaces.
  • GUI Agent Benchmarks: AITW provides screenshots rather than tree-based UI representations, while language-model agents use textual OCR and icon representations.The screenshot-only format reflects the limited availability or quality of tree-based representations in many smartphone applications.

3. Derivation of Acceleration for High-Resolution Cross-Module

CogAgent’s high-resolution cross-attention module reduces the attention cost of processing 1120 × 1120 images while retaining access to fine-grained GUI information.

  • Complexity formulation: The module combines low-resolution image tokens, high-resolution image tokens, and text tokens through cross-attention and decoder self-attention.The complexity is expressed using sequence lengths LIlo, LIhi, and LT, plus attention-head counts and per-head dimensions.
  • Complexity formulation: Replacing low-resolution images directly with high-resolution images would require O((LIhi + LT)^2 Hdec ddec) attention complexity.This is the baseline complexity used to compare against the proposed high-resolution cross-module.
  • Reduction factor: The complexity reduction factor is derived as the ratio of original to improved attention computation.The derivation accounts for cross-attention between low- and high-resolution image features and decoder self-attention over low-resolution image and text tokens.
  • Case 1: LIlo, LT ≪ LIhi: When LIlo and LT are much smaller than LIhi, the reduction factor is approximately LIhi/(LIlo + LT).This is the zeroth-order approximation for Case 1, where both LIlo/LIhi and LT/LIhi are first-order small quantities.
  • Case 2: Our implementation: With LIhi = 6400 and LIlo = 256, the implementation provides at least (6400 + LT)/(256 + LT) acceleration.The stated lower bound follows from dcross = 32, Hcross = 32, ddec = 128, and Hdec = 32.
  • Case 2: Our implementation: For pre-training stages with LT typically below 512, the high-resolution cross-module achieves over 25× reduction in computational cost.The paper presents this as an example of the module’s computational benefit.

4. Performance Analysis on AITW

AITW analysis shows that disagreements between CogAgent and annotated actions often reflect alternative valid smartphone-operation paths rather than model errors.

  • Discrepancy analysis: AITW contains large-scale Android agent episodes, and the analysis focuses on cases where CogAgent responses diverge from annotated ground truth.The authors manually verify sampled discrepancies to distinguish errors from alternative correct actions.
  • Discrepancy analysis: 42% of sampled discrepancies were manually judged to be alternative correct methods rather than errors.The paper attributes this pattern to the multiplicity of ways to accomplish smartphone functions.
  • Multiple correct answers: Examples include searching through either Google’s app or search bar, selecting relevant news immediately, and avoiding redundant annotation steps.These examples illustrate valid differences in navigation preferences and path efficiency.
  • Evaluation implication: The authors recommend evaluating task completion after action sequences in virtual environments to better reflect real-world smartphone use.This proposed evaluation would focus on whether the task is completed rather than exact agreement with one annotated sequence.

5. Samples of Pre-train Data

CogAgent’s pre-training data targets high-resolution text recognition, visual grounding, and GUI-oriented webpage understanding.

  • Data categories: The pre-training data are divided into text-recognition, visual-grounding, and webpage-HTML pair categories.Examples are provided in Figures 5, 6, and 7, respectively.
  • Text recognition: The data are designed to improve recognition of text with varied sizes, orientations, and fonts in high-resolution images.This focus addresses the fine-grained visual demands of GUI screenshots.
  • Visual grounding: Visual-grounding data support locating text and objects within images.Figure 6 shows samples of the visual-grounding pre-training data.
  • GUI understanding: Webpage-HTML pairs specialize training toward GUI imagery such as webpages.Figure 7 provides examples of the paired data.

6. Details of Fine-Tuning Data

CogAgent’s fine-tuning data combine manually collected screenshots, structured GUI annotations, grounding labels, and converted web-agent instructions, while failure analysis identifies recurring navigation errors.

  • Screenshot collection: More than ten annotators collected and labeled screenshots across apps and websites in a two-phase process.The process was designed to support varied GUI-agent tasks rather than simplistic, homogenized content.
  • Phase one annotation: In phase one, annotators selected relevant screenshots and labeled buttons, clickable areas, image-information questions, and specific operation requirements.Each app or website contributed at least 20 screenshots, with diverse interface elements and task-oriented prompts.
  • Phase two annotation: Phase two added grounding annotations for the information-extraction and operational tasks proposed in phase one.Examples include grounding profile information, clickable regions, and the control used to follow an author.
  • Mind2Web conversion: Mind2Web examples were converted into natural-language plans and immediate actions, with explicit click locations or text-entry content.The conversion required JSON outputs containing plan, action, and operation fields while avoiding HTML-only elements.
  • Operational examples: The resulting operational guide uses task goals, prior actions, and secretly specified future action sequences as examples for GUI operation planning.The examples cover selecting flights, hotels, room types, and checkout actions.
  • Failure cases: Failure cases are grouped into incorrect action prediction, incorrect coordinate prediction, and incorrect GUI observation.These categories summarize distinct ways GUI navigation can fail.
Loading 2312.08914v3…