PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileFlow: A Multimodal LLM For Mobile GUI Agent

Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, Wenhao Xu

arXiv:2407.04346v3cs.CV

TL;DR

Mobile GUI agents face challenges with Mandarin text, privacy-sensitive API use, and fixed-resolution visual inputs. MobileFlow combines hybrid visual encoders, an adapter, and GUI-specific training; the paper reports stronger performance across sectors and strong visual question answering after business-specific fine-tuning.

  • Problem

    Existing GUI agents face Mandarin-text difficulties, privacy concerns from system API use, and fixed-resolution visual encoders that can limit performance across diverse interfaces.

    Source: §1 · p. 1

  • Method

    MobileFlow combines a hybrid visual encoder, a visual-language adapter, an MoE-enhanced language model, and GUI-specific alignment and fine-tuning.

    Source: §1 · p. 2, §2 · p. 3, §2.1 · p. 3

  • Results

    MobileFlow performs better across different sectors and complex tasks, and shows strong visual question-answering after business-specific fine-tuning, standing up to larger Qwen-vl-max.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    The paper reports MobileFlow’s practical applications across business sectors and presents it as an initial step in the evolution of GUI agents.

    Source: §4 · p. 7

    The paper identifies susceptibility to hallucinations and handling multiple images as future challenges, and says MobileFlow is mainly used for mobile apps.

    Source: §4 · p. 7

Abstract

from arXiv · show

Currently, the integration of mobile Graphical User Interfaces (GUIs) is ubiquitous in most people's daily lives. And the ongoing evolution of multimodal large-scale models, such as GPT-4v, Qwen-VL-Max, has significantly bolstered the capabilities of GUI comprehension and user action analysis, showcasing the potentiality of intelligent GUI assistants. However, current GUI Agents often need to access page layout information through calling system APIs, which may pose privacy risks. Fixing GUI (such as mobile interfaces) to a certain low resolution might result in the loss of fine-grained image details. At the same time, the multimodal large models built for GUI Agents currently have poor understanding and decision-making abilities for Chinese GUI interfaces, making them difficult to apply to a large number of Chinese apps. This paper introduces MobileFlow, a multimodal large language model meticulously crafted for mobile GUI agents. Transforming from the open-source model Qwen-VL-Chat into GUI domain, MobileFlow contains approximately 21 billion parameters and is equipped with novel hybrid visual encoders, making it possible for variable resolutions of image inputs and good support for multilingual GUI. By incorporating Mixture of Experts (MoE) expansions and pioneering alignment training strategies, MobileFlow has the capacity to fully interpret image data and comprehend user instructions for GUI interaction tasks. Finally, MobileFlow outperforms Qwen-VL-Max and GPT-4v in terms of task execution by GUI agents on both public and our proposed evaluation metrics, and has been successfully deployed in real-world business contexts, proving its effectiveness for practical applications.

1 Introduction

MobileFlow addresses GUI-agent challenges in Mandarin understanding, privacy-sensitive API use, and visual encoders limited to fixed resolutions. It combines visual perception and GUI-specific training for interaction across diverse interfaces.

1 Introduction

  • MobileFlow targets Mandarin-heavy GUI applications, API-related privacy concerns, and fixed-resolution visual encoders that limit performance in varied interfaces.The paper highlights diverse “super-app” GUI scenarios as a challenge for existing agents.Source: §1 · p. 1
  • MobileFlow uses a hybrid visual encoder trained on GUI pages to extract and comprehend information across diverse interfaces without system APIs for page layout.The paper says this visual-only approach mitigates privacy-intrusion risk on user devices.Source: §1 · p. 2
  • MobileFlow combines visual and textual data through MoE and specialized GUI training, and uses chain-of-thought fine-tuning to improve accuracy.The paper describes applications including GUI understanding, step-by-step guidance, information extraction, and question answering.Source: §1 · p. 2

1.1 Related Work

  • Prior GUI agents include AppAgent, CogAgent, and MobileAgent, while related approaches vary in their reliance on APIs and UI-specific training.MobileAgent and AppAgent use existing VLMs and external modules or APIs; CogAgent avoids system API calls but its LLM was not targeted-trained for GUI tasks [2024b].Source: §1 · p. 1, §1.1 · p. 2
  • MobileFlow introduces a LayoutLMv3 visual branch pretrained on UI data that adjusts to varying aspect ratios for GUI-agent use.The related-work discussion contrasts this with visual encoders pretrained on natural scenes and other approaches that combine pretrained visual perceptors.Source: §1.1 · p. 3

2 Method

MobileFlow combines multiple visual encoders, a visual-language adapter, and an MoE-enhanced language model. GUI alignment training develops visual grounding, referring, question answering, and stepwise reasoning for interaction tasks.

2.1 Model Architecture

  • MobileFlow uses OpenCLIP ViT-B/32 and a variable-resolution UI Encoder based on LayoutLMv3, with UI-specific pretraining and aspect-ratio-preserving image encoding.For a 1216x576 image, the example encoding uses 779 patches and pads the sequence with five more.Source: §2 · p. 3
  • UI Encoder pretraining combines masked language modeling, masked image modeling, word-patch alignment, and reconstruction of interfaces after controls are replaced.Word-patch alignment predicts whether the image patch corresponding to a text word is masked.Source: §2.1 · p. 3
  • The visual-language adapter compresses each encoder’s image features to 256 tokens with cross-attention, then uses an MLP to fuse and project them into the LLM’s semantic space.The architecture also expands the language model with four identical MLP experts initialized from the original MLP.Source: §2.1 · pp. 3–4
  • MobileFlow processes GUI screenshots into global and local visual features, then aligns them with user text, OCR text, and GUI bounding-box data for the LLM.Its architecture combines a hybrid visual network, visual-language alignment, and an MoE-enhanced LLM.Source: §2.1 · p. 4

2.2 Training Formulation

  • Light GUI alignment training builds on Qwen-VL-Chat and includes tasks for grounding text in image regions, locating referred-to items, and answering questions about screen content.The question-answering task uses ScreenQA, which includes page-information extraction and page-content inference.Source: §2.2 · pp. 4–5
  • MobileFlow uses chain-of-thought during training and inference, structuring intent execution as observation, reasoning, action, and summary.The paper reports noticeable improvement in link and question-answering accuracy after adding this approach.Source: §2.2 · p. 5

3 Experiments

Experiments evaluate action prediction and visual question answering using task-success metrics across business sectors and task complexity. The reported results favor MobileFlow in action prediction, and ablations report gains from both MoE and the UI Encoder.

3.1 Metrics

  • The evaluation adds EDR to address endpoint determination, while adopting WTSR and SSR for single-step and multi-step prediction accuracy.The authors say earlier metrics were effective for discrete counting with limited samples, citing a case with 10 samples.Source: §3.1 · p. 6

3.2 Quantitative Results

  • Performance varies across business sectors, and evaluation metrics tend to degrade as task complexity increases.Table 1 reports results across six business sectors and three task-complexity levels.Source: §3.2 · p. 6, Table 1 · p. 6
  • EDR is expected to be lower than WTSR because it counts success only when a task executes fully and ends correctly; in Medical Service Apps, all successfully predicted tasks ended properly.Source: §3.2 · p. 6
  • MobileFlow shows better performance across sectors and excels at complex tasks in action prediction, despite having a smaller LLM than Qwen-vl-max.The comparison is reported in Table 2.Source: Table 2 · p. 6, §3.2 · p. 7
  • MobileFlow shows strong VQA ability, especially after business-specific fine-tuning, and is reported to stand up to larger models such as Qwen-vl-max.The VQA comparison is presented in Table 3.Source: Table 3 · p. 7, §3.2 · p. 7

3.3 Ablation Study

  • MoE improves WTSR by 4.49% and SSR by 8.17% over a dense LLM, while adding the UI Encoder improves WTSR by 6.96% and SSR by 4.47%.Source: Table 4 · p. 7, §3.3 · p. 7

4 Conclusion

MobileFlow is presented as a GUI agent for analyzing UI images, understanding instructions, and operating in practical scenarios. The conclusion notes unresolved challenges and a current focus on mobile apps.

4 Conclusion

  • MobileFlow analyzes UI images, understands user instructions, and operates in practical scenarios across diverse scenarios and business domains.The paper reports practical applications across various sectors.Source: §4 · p. 7
  • The authors identify hallucinations and handling multiple images as future challenges, and describe current use as mainly focused on mobile apps.Expansion to computers is described as potential future work.Source: §4 · p. 7

A Action Space

B MobileFlow Details

The prompt supplies an image, task information, page elements, and action history, then asks the agent to reason about the current page and produce a feasible next action or finish the task.

B.1 Prompt structure

  • The prompt presents an image alongside task information, page elements, and historical actions.Source: §B.1 · p. 10
  • The agent is instructed to describe page observations, reason about a next action, and output a feasible click, swipe, text input, or Finish.Source: §B.1 · p. 10
  • After the next action, the agent summarizes prior actions in two or three sentences without future actions or control coordinates.Source: §B.1 · p. 10

C.1 Positive Sample Determination

The evaluation labels predictions using both action-type matching and coordinate-area IoU. Figure 4 illustrates matching cases used to assess prediction accuracy.

C.1 Positive Sample Determination

  • Prediction labels combine matching the predicted action type with Intersection over Union (IoU) of coordinate areas.The combination assigns a true or false label to each prediction.Source: §C.1 · p. 10
  • Figure 4 presents matching cases for positive and negative samples.Source: Figure 4 · p. 11
  • The matching cases in Figure 4 visualize how the evaluation metrics assess prediction accuracy.Source: §C.1 · p. 11

C.3 Dataset

MobileFlow training uses separate models for UI understanding and token prediction, with curated UI data and mixed multimodal alignment and business-specific fine-tuning data.

C.3 Dataset

  • The training pipeline uses a UI Encoder for interface understanding and Qwen-vl-chat for token prediction.Source: §C.3 · p. 11
  • The UI Encoder is fine-tuned on 100K manually labeled instances curated for current business domains.Source: §C.3 · p. 11
  • Multimodal alignment combines RefCoCo[29], ScreenQa[28], Flickr30K[9], and in-house UI data, followed by supervised fine-tuning on 70k manually labeled data from 10 business sectors.Source: §C.3 · p. 11

C.4 Training and Evaluation details

UI Encoder was pretrained on UI images and fine-tuned on labeled business data; evaluation compared MobileFlow with GPT-4v using MobileAgent-compatible test interfaces.

C.4 Training and Evaluation details

  • Approximately 200k UI images were used to pre-train UI Encoder, with downstream validation on image classification and UI component recognition.Pretraining aimed to improve comprehension of fonts, images, controls, and other UI elements.Source: §C.4 · p. 11
  • UI Encoder fine-tuning used a 100K labeled business dataset for 2 epochs; experiments ran on 8 Nvidia A100 GPUs, and MobileAgent evaluation kept the model parameters intact.Source: §C.4 · p. 11
  • MobileAgent was adapted to the test data format and interface, while its model parameters remained intact; GPT-4v was used directly for token prediction.Experiments ran on 8 Nvidia A100 GPUs; Qwen-VL-Chat supervised fine-tuning used 2 epochs, with the learning rate unspecified in the passage.Source: §C.4 · p. 11

D.1 Software Testing

The paper describes MobileFlow applications in software testing, advertisement preview and audit, and small-merchant e-commerce monitoring, addressing different operational tasks in each domain.

D.1 Software Testing

  • Natural-language instructions are presented as a way to reduce testing-system complexity and programming prerequisites, handle UI-route noise and account/data differences, and analyze some alarms automatically.The paper reports software testing takes 150% longer on average than development, and describes noisy routes as a source of false alarms and frequent script updates.Source: §D.1 · p. 12

D.2 Advertisement Preview And Audit

  • MobileFlow can support advertisers previewing triggered ads and platforms monitoring trigger strategies, interaction logic, and ad content.The paper notes that multimedia ads require multi-step interactions, making manual and traditional automation costly and unstable.Source: §D.2 · p. 12

D.3 E-Commerce Operation And Monitoring

  • A MobileFlow-based tool helps small merchants monitor cross-platform and cross-date price discrepancies and competitors’ marketing events.The paper frames these tasks against the operational burden of managing stores across multiple Chinese platforms.Source: §D.3 · p. 12

E More Samples

Four figures show MobileFlow GUI-agent examples through user instructions for checking records and lists, placing an order, and booking a clinic appointment.

E More Samples

  • Figure 5 presents the instruction to check claim records.Source: Figure 5 · p. 13
  • Figure 6 presents a request to order strawberry custard from the Jiao Tong University Campus store in Haidian District.Source: Figure 6 · p. 14
  • Figure 7 presents the instruction to check the top health insurance lists.Source: Figure 7 · p. 14
  • Figure 8 presents a request for an April 14 gastroenterology appointment at Wuhou Hospital of Sichuan Modern Hospital.Source: Figure 8 · p. 15

Related papers

Loading 2407.04346v3…