Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation

1 Introduction

• Mobile agents struggle when training data lacks relevant scenarios, while manually written knowledge raises cost and makes performance sensitive to author expertise.The introduction links these limitations to incomplete scenario coverage, poor generalization, and the need for expert-authored knowledge.Source: §1 · pp. 1–2
• 36% performance improvement over existing frameworks is reported on Mobile-Knowledge, a benchmark designed to isolate knowledge-utilization efficacy.The benchmark uses straightforward tasks requiring procedural knowledge while minimizing unrelated operational abilities.Source: §1 · p. 2
• Mobile-Knowledge evaluates knowledge utilization separately from broader capabilities such as localization, planning, and decision-making.This design addresses the difficulty of measuring knowledge-injection effects in integrated mobile benchmarks.Source: §1 · p. 2
• Mobile-Agent-V injects operational knowledge by having users perform and record a task once, reducing the need for manual knowledge construction.The framework uses video as a natural, unprocessed knowledge source.Source: §1 · p. 2
• A multi-agent strategy extracts and applies video knowledge using a sliding window for long videos and a deep-reflection agent to refine decisions.The video agent adaptively selects relevant keyframes while reflection improves decision accuracy.Source: §1 · p. 2

2.1 GUI Agent

• GUI-agent frameworks use HTML parsing on the Web, visual perception in assistants, and system APIs in PC environments, while mobile automation presents distinct operational challenges.These approaches illustrate different interface access and control strategies.Source: §2.1 · p. 2
• Mobile automation agents often require operational knowledge that existing LLMs lack, motivating costly training, exploration, or manual knowledge approaches.The related work places Mobile-Agent-V within efforts to improve GUI operation across mobile environments.Source: §2.1 · p. 2

2.2 Video-guided Agent

• Video-guided agents have been studied for video comprehension, automated editing, efficient frame retrieval, and robotics training from human demonstrations.These applications show video guidance extending beyond comprehension to practical agent interaction and learning.Source: §2.2 · p. 3

3 Mobile-Agent-V

• The framework is presented as a combination of video processing, sliding-window selection, video-agent progression, decision making, and deep reflection for mobile automation.These components are described as working together to improve operational efficiency and accuracy.Source: §3 · p. 3

3.1 Framework

• Mobile-Agent-V iteratively generates, refines, and executes video-aligned actions while dynamically updating its observation window until completion or the exploration limit.The framework extracts keyframes, uses a decision agent to propose actions, applies deep reflection when needed, executes refined actions, and advances the window.Source: §3.1 · p. 3

3.2 Video Processing

• Uniform sampling followed by similarity and temporal-gap filtering produces a reduced keyframe set that removes redundancy from mobile recordings.The method first samples the video uniformly, removes highly similar consecutive frames, and filters frames separated by insufficient temporal gaps.Source: §3.2 · p. 3

3.3 Sliding Window

• The sliding window selects only keyframes relevant to the current operation, reducing MLLM input length while preserving critical information.The window is defined by start and end points, with its length controlling the decision-making input.Source: §3.3 · pp. 3–4

3.4 Decision Agent

• The decision agent derives each operation from the video window, video and user instructions, the current device screenshot, and historical operations.Its action space includes Click, Scroll, Type, Back, Home, and Done.Source: §3.4 · p. 4

3.5 Deep-Reflection Agent

• The deep-reflection agent checks whether each proposed action matches the demonstrated operation and refines discrepancies against the video trajectory.It analyzes the operation, current device state, and decision-agent output before producing a final refined action.Source: §3.5 · p. 4

3.6 Video Agent

• The video agent adaptively shifts the sliding window by analyzing before-and-after screenshots, window keyframes, and user inputs after each operation.This progression keeps the observation scope aligned with the corresponding point in the demonstrated task.Source: §3.6 · p. 4

4.1.1 Benchmark

• Mobile-Knowledge contains 30 device-specific tasks across basic, normal, and advanced instructions, while AndroidWorld-Knowledge contains 48 knowledge-intensive tasks across five applications.The two benchmarks evaluate direct knowledge utilization and generalization beyond directly demonstrated tasks, respectively.Source: §4.1.1 · p. 5

4.3.1 Mobile-Knowledge

• Up to 23.4% improvement over the best baseline appears on Mobile-Knowledge across SR, CR, and DA, with fewer steps indicating greater action efficiency.Mobile-Knowledge emphasizes procedural knowledge while minimizing planning and operational complexity.Source: §4.3.1 · p. 5, §4.1.1 · p. 5, Table 1 · p. 6

4.3.2 AndroidWorld-Knowledge

• 31.3% SR on AndroidWorld-Knowledge is at least 12.4% higher than the best baseline, demonstrating performance beyond direct video-guided tasks.The benchmark provides only one video per scenario, requiring generalization to other tasks.Source: §4.3.2 · pp. 5–6, Table 2 · p. 6

4.4.1 Generalization from Videos

• Video-misaligned instructions reduce SR and DA for normal and advanced tasks but leave basic tasks stable, while performance remains competent beyond direct instruction mapping.The analysis uses mismatched operational logic to test generalization from video demonstrations.Source: §4.4.1 · p. 6, Figure 3 · p. 7

4.4.2 Impact of Window Size

• Larger sliding windows generally improve SR, CR, and DA and reduce steps, but excessive context causes diminishing returns or metric declines.The reported decline is attributed to irrelevant information interfering with decision-making.Source: §4.4.2 · p. 6, Figure 4 · p. 7

4.4.3 Impact of Keyframe Quality

• Manually selected keyframes perform slightly better than uniform sampling with filtering, but the small gap shows that the proposed strategy preserves essential task-relevant information.The comparison evaluates redundancy avoidance and omission prevention in keyframe selection.Source: §4.4.3 · p. 6, Figure 5 · p. 7

4.4.4 Impact of Knowledge Injection Method

• 86.7% SR is achieved with video-based knowledge injection in 0.7 minutes on average, whereas manually written knowledge can require up to five minutes for expert knowledge.The comparison positions video injection as a faster alternative while retaining high performance.Source: §4.4.4 · p. 7, Table 3 · p. 7

4.5 Ablation Study

• Deep reflection improves decision-making most on complex low-performing tasks by correcting action misalignments, reducing cascading errors, and increasing robustness in challenging visual conditions.Step count changes only slightly, indicating improved precision without major action-efficiency effects.Source: §4.5 · pp. 7–8, Figure 6 · p. 8

5 Conclusion

• Mobile-Agent-V uses a sliding window to select keyframes and a deep-reflection agent to improve decision accuracy.Source: §5 · p. 8
• 23.4% Success Rate improvement on Mobile-Knowledge and 12.4% on AndroidWorld-Knowledge demonstrate superior performance.Mobile-Agent-V rivals expert-level written knowledge and reduces injection time by 86%.Source: §5 · p. 9
• The framework transforms videos into operational knowledge for a streamlined path to scalable agent development.Source: §5 · p. 9

6 Limitations

• Suboptimal video recordings may reduce the accuracy of knowledge extraction.Source: §6 · p. 9
• The sliding window may overlook essential frames during complex interactions despite improving processing efficiency.Source: §6 · p. 9
• Generalization performance depends on the range and quality of available video demonstrations.Source: §6 · p. 9

A.1 Experimental Details

• The section documents additional details about Mobile-Agent-V’s experimental setup.Source: §A.1 · p. 12
• The section documents implementation choices used in Mobile-Agent-V.Source: §A.1 · p. 12
• These details supplement the main description of Mobile-Agent-V’s experiments.Source: §A.1 · p. 12

A.1.1 Sliding Window Size Selection

• A sliding window size of 4 was selected for the experiments.Source: §A.1.1 · p. 12
• Increasing the window size to 5 produced marginal performance improvement while increasing computational cost through higher token consumption.Source: §A.1.1 · p. 12
• The size-4 setting represents a trade-off between efficiency and performance.Source: §A.1.1 · p. 12

• Frame similarity begins by converting the two input frames I1 and I2 into grayscale representations.Source: §A.1.2 · p. 12
• The grayscale frames are compared by computing their absolute pixel-wise difference.Source: §A.1.2 · p. 12
• Similarity score S is obtained by counting the nonzero pixels in the difference image D.Source: §A.1.2 · p. 12
• The approach captures frame differences while maintaining computational efficiency.Source: §A.1.2 · p. 12

A.1.3 Frame Similarity Threshold Selection

• Settings uses fs = 0.3 to retain informative text-based changes, whereas Weather uses fs = 0.5 to reduce redundant extraction amid greater visual variation.Source: §A.1.3 · p. 12

A.1.4 Step Limitations and Task Termination Criteria

• Execution limits are 10 steps for basic tasks, 15 for standard tasks, and 20 for complex tasks.Source: §A.1.4 · p. 12
• Attempts fail when they exceed the applicable step limit without success or continue with unnecessary operations beyond the instruction’s scope.Source: §A.1.4 · p. 12

A.1.5 Video Frame Concatenation for Visualization

• Up to four sliding-window frames are concatenated row-wise, indexed for tracking, and marked at the final frame to indicate termination.When fewer than four frames exist, the sequence contains only the available frames, up to three.Source: §A.1.5 · p. 12

A.1.6 Action Space Definition

• Mobile-Agent-V uses Set of Mark interaction coordinates and adds clickby-text as a supplement for pages vulnerable to XML parsing issues.Source: §A.1.6 · p. 12

A.3.1 Evaluation Tasks of Mobile-Knowledge

• Mobile-Knowledge evaluates video-aligned and video-misaligned instructions across application-categorized tasks to test linguistic robustness and real-world adaptability.Source: §A.3.1 · pp. 12–13

A.3.3 Metrics

• The evaluation uses success rate, completion rate, decision accuracy, and step count to measure end-to-end performance, step-level progress, decision precision, and execution efficiency.Source: §A.3.3 · p. 13

A.3.4 Screen Recording

• Videos were recorded on a OnePlus 7 Pro at practical rates of 30–60 Hz, without editing, with one unique optimal-path video per benchmark instruction.Source: §A.3.4 · p. 13