PaperlayerView exact revision on arXiv

Source-linked AI summary

Cradle: Empowering Foundation Agents Towards General Computer Control

Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, Zongqing Lu

arXiv:2403.03186v3cs.AI

TL;DR

Foundation agents struggle to generalize across virtual environments with differently designed observation and action spaces. Cradle uses a modular LMM framework with a unified screen-and-input interface, showing performance across games, software applications, and OSWorld.

  • Problem

    Foundation agents struggle to generalize across environments with human-designed observation and action spaces, while agent-accessible software remains limited, especially commercial closed-source games.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    Cradle operationalizes GCC with a modular LMM framework that interprets computer screens and issues keyboard and mouse actions through a unified interface.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    Cradle showed strong generalization and performance across four commercial games, five software applications, and OSWorld, including a maximal weekly total profit of 87% in Dealer’s Life 2.

    Source: §1 · p. 4

  • Takeaways & Limitations

    GCC offers a unified setting for studying foundation-agent control across diverse games and software.

    Source: §1 · p. 4, §6 · p. 12

    Cradle struggles to recognize out-of-distribution icons and complete out-of-distribution tasks, particularly in games with non-realistic styles.

    Source: §5 · p. 12

Abstract

from arXiv · show

Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. Cradle greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents' various abilities and facilitate further data collection, thus paving the way for generalist agents.

1 Introduction

Cradle proposes General Computer Control (GCC), in which agents use a unified human-style interface to interact with computers, and introduces a modular LMM-powered framework for this setting. The paper evaluates Cradle across games, software applications, and OSWorld.

Introduction

  • GCC aims to let foundation agents master any computer task using screen and audio input and keyboard and mouse output, avoiding reliance on each environment’s distinct interface.The paper motivates virtual computer environments as a path toward generalizability without the hardware, practicality, and catastrophic-failure challenges of real-world robotics.Source: §1 · p. 2
  • GCC requires multimodal alignment, precise control over a large hybrid action space, long-horizon reasoning and memory, and structured self-improving exploration.These challenges arise from partial observability and the diversity of tasks in the digital world.Source: §1 · p. 2
  • Cradle combines six modules: information gathering, self-reflection, task inference, skill curation, action planning, and memory.Together, they process multimodal observations, assess progress and failures, select or propose tasks, manage reusable skills, generate executable control operations, and retain experience.Source: §1 · p. 2
  • Video games and software applications form the two broad GCC task categories, with games testing adaptive problem-solving, long-term reasoning, learning, and precise control.The evaluation covers representative game tasks and everyday software manipulation across distinct domains.Source: §1 · pp. 2, 4
  • Across four commercial games, five software applications, and OSWorld, Cradle reports broad performance, including 40-minute RDR2 missions, a thousand-person Cities: Skylines city, and a maximal weekly total profit of 87% in Dealer’s Life 2.It also farms and harvests parsnips in Stardew Valley and operates Chrome, Outlook, CapCut, Meitu, and Feishu.Source: §1 · p. 4

2 Related Work

Computer-control benchmarks cover web and common software, while game environments form a separate research stream; API-dependent interfaces and differing action and observation spaces limit access and cross-environment consistency. Existing software and game agents retain interface or training constraints, motivating Cradle’s unified GCC exploration.

2.1 Environments and Benchmarks for Computer Control

  • Earlier computer-control benchmarks focused on web navigation, and newer ones include common software, but none of these benchmarks includes video games.The paper frames games as a missing component of computer control.Source: §2.1 · p. 4
  • Game environments range from Atari [4] and Minecraft [16] [21] to custom embodied worlds such as Habitat [37] [46] [54], advancing decision-making research, especially reinforcement learning.Source: §2.1 · p. 4
  • Reliance on open-source code or built-in APIs, costly human implementation, and divergent action and observation spaces limit agent access and cross-environment data consistency.Commercial closed-source software and games are especially difficult to expose to agents, and few agents complete tasks across multiple environments.Source: §2.1 · p. 4

2.2 LMM-based Agents for Computer Tasks

  • Software agents rely on DOM or built-in APIs, struggle to generalize, and often target static tasks that omit screenshot dynamics and timed or mode-specific input.The paper notes that these omissions hinder deployment to rapidly changing graphics such as games.Source: §2.2 · p. 5
  • Game agents often use game-specific APIs or actions, while screenshot-based approaches depend on costly labeled gameplay or human-expert demonstrations.JARVIS-1 uses a hybrid keyboard, mouse, and API action space; VPT uses screenshots and keyboard and mouse control but requires costly action-labeled videos; SIMA trains on human gameplay for 10-second tasks across ten 3D games.Source: §2.2 · p. 5
  • Cradle explores unified interaction across complex games and software under GCC, where the authors report no prior agent with superior performance and generalization across these tasks.The work presents a preliminary benchmark and framework for using diverse software as agent-capability testbeds.Source: §2.2 · p. 5

3 The CRADLE Framework

CRADLE is an LMM-powered framework that uses video observations and staged reasoning to produce executable keyboard and mouse controls; its modules support interaction, memory, and planning.

3 The CRADLE Framework

  • CRADLE uses GPT-4o as its backbone to interpret screens and changes between frames and generate precise control actions across arbitrary software.The framework is designed to combine vision and reasoning with knowledge of computer interfaces and control.Source: §3 · p. 5

3.1 Environment IO

  • CRADLE accepts screen-video observations and produces keyboard and mouse operations, using visual and textual information, learned semantic skills, and an executor to control software.Skills can be developed from tutorials, manuals, or self-exploration, and instantiated with parameters such as duration, position, and speed.Source: §3.1 · pp. 5–6

3.2 Memory

  • CRADLE maintains episodic memory of screenshots, multimodal information, actions, tasks, and reasoning, and procedural memory of skills that it can retrieve, update, or compose.Periodic summaries abstract recent episodic information into long-term summaries, while procedural memory calculates skill relevance for the current task and situation.Source: §3.2 · p. 7

3.3 Reasoning

  • CRADLE’s reasoning modules reflect on action outcomes, infer the current task, curate relevant skills, and plan parameterized action sequences for execution.Reflection can diagnose failures, support replanning, and suggest skill improvements; the staged workflow converts video into reasoning, skills, actions, and low-level controls.Source: §3.3 · pp. 7–8

4 Empirical Studies

The empirical studies test CRADLE across challenging GCC environments for decision-making, UI understanding, and manipulation, using human evaluation for diverse applications and games and OSWorld for quantitative comparison.

4.1 Experimental Settings

  • Experiments generally used five runs, 2-fps video, GPT-4o-2024-05-13, and human evaluation for games and applications; OSWorld supplied a quantitative benchmark comparison.RDR2 and Stardew Valley were paused during model responses because of game dynamism and inference latency; reported results use µ ± σ or successful runs over total runs.Source: §4.1 · p. 8, Table 4 · p. 11
  • The task suite spans 13 RDR2 storyline checkpoints plus an open-world supply mission, three Stardew Valley tasks, a week of shop management, city building, and five tasks per software application.RDR2’s first two Chapter I missions take a novice about 40 minutes; tasks were designed to test agent performance across varied environments.Source: §4.1 · pp. 8–9

4.2 Performance across Tasks and Environments

  • CRADLE succeeds on straightforward RDR2 following and location tasks, but combat and searching remain difficult; it completes the unguided Buy Supply task.Protect Dutch success falls to 20% even with Grounding DINO; indoor search is hampered by poor spatial perception.Source: Figure 7 · p. 9, §4.2 · p. 9
  • CRADLE clears many Stardew Valley farm obstacles and obtains a parsnip by watering for four days and harvesting, but struggles with precise object interaction and shopping.The agent must stand precisely in front of objects in the 2D grid, contributing to inefficiency in farming.Source: Table 1 · p. 9, §4.2 · p. 9, Figure 8 · p. 10
  • In Dealer’s Life 2, CRADLE finalized 93.6% of potential transactions, averaged two negotiation rounds per customer, and achieved +39.6% total profit, peaking at +87.4%.It generally targeted a profit rate above 50% at the initial offer.Source: Table 2 · p. 9, §4.2 · p. 9
  • In Cities: Skylines, CRADLE covered over 90% of available area with zones and built roads and power, but unreliable water connections limited growth; human corrections enabled population above 1,000.With assistance to fix small design issues, the final population can be doubled.Source: Table 3 · p. 10, Figure 10 · p. 10, §4.2 · p. 10
  • Across five software applications, CRADLE struggles with UI-item recognition and visual context, especially in non-standard layouts such as CapCut, Meitu, and Feishu.Failures include forgetting to save and misidentifying enabled versus disabled buttons; limited prior knowledge also impairs task inference and skill selection.Source: §4.2 · p. 10, Figure 11 · p. 11

4.3 Ablation Study & Baselines Comparison

  • CRADLE with GPT-4o performs best across the compared tasks, while baselines manage only simple tasks and the visionless Voyager-like method performs worst.The Cultivation ablation suggests task inference and episodic summaries help CRADLE recognize when to harvest; Voyager-like also struggles without visual input.Source: §4.3 · p. 11, Table 5 · p. 11, Table 6 · p. 12

5 Limitations and Future Work.

CRADLE remains limited by current LMM capabilities, omitted audio input, and the lack of training and broad task coverage. The authors propose technical and training improvements while warning that responsible deployment requires safeguards.

Limitations and future work

  • CRADLE struggles to recognize out-of-distribution icons and tasks, especially in non-realistic games such as Stardew Valley, and does not yet process audio alongside graphics.The authors attribute OOD difficulties to current LMM weaknesses and identify simultaneous audio and graphical input as future work.Source: §5 · p. 12
  • Frequent explicit LMM calls can increase costs and delays, though improved LMMs may allow some or all six modules to be combined into one request.The modules are described as representing a problem-solving mindset.Source: §5 · p. 12
  • Although CRADLE is evaluated without training, future work proposes RL-based exploration or supervised learning from expert demonstrations [48], following promising trained-agent results from SIMA [48].Source: §5 · p. 12
  • The study investigates only a few tasks; future work will broaden target coverage, explore complex games more deeply, and make adaptation easier for users.Source: §5 · p. 12
  • Potential misuse includes game cheating, harmful software-operation failures, and other negative agent behavior, so secure and responsible deployment requires additional safeguards or regulations.The authors note these safeguards should cover digital and physical environments.Source: §5 · p. 12

6 Conclusion

The paper introduces GCC, a unified screenshot-and-input setting for controlling diverse software, and CRADLE, an open-source framework that operates without built-in APIs. CRADLE demonstrates strong performance across complex games and common applications, while remaining imperfect on some tasks.

6 Conclusion

  • CRADLE is presented as the first framework to succeed across a diverse set of environments without relying on built-in APIs.It reasons and performs actions to accomplish real missions in complex video games and tasks in common software applications.Source: §6 · p. 12
  • GCC standardizes control of diverse games and software through screenshots as input and keyboard and mouse operations as output.The setting aims to support general foundation agents across digital-world tasks.Source: §6 · p. 12
  • CRADLE exhibits strong reasoning and action performance on real missions in complex video games and tasks in common software applications.The conclusion characterizes the framework as a pioneering effort, while noting that it still faces difficulties on certain tasks.Source: §6 · p. 12
  • CRADLE makes it feasible to turn software, especially complex games, into benchmarks for evaluating agents’ general intelligence and facilitating further data collection for self-improvement.The paper identifies framework enhancements and advances in LMMs as directions for developing more powerful agents.Source: §6 · p. 12

8 Team Members and Contributions · Appendix

The appendix identifies Cradle’s leadership and core contributors, then assigns implementation, experimentation, analysis, writing, and organizational responsibilities across the framework and evaluated environments.

8.1 Roles

  • Program leads, team leadership, framework co-leadership, advisory roles, and core contributors are identified.Program Leads: Zongqing Lu, Shuicheng Yan, and Bo An; Team Lead: Weihao Tan; Framework Co-Leads: Börje F. Karlsson and Weihao Tan; General Advisors: Xinrun Wang and Börje F. Karlsson. Core Contributors: Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Xinrun Wang, and Börje F. Karlsson.Source: §8.1 · p. 13

8.2 Detailed Contributions

  • Framework design was led by Weihao Tan, Börje F. Karlsson, Wentao Zhang, Ziluo Ding, and Xinrun Wang.Source: §8.2 · p. 13
  • The RDR2 implementation, experiments, and analysis involved a 15-person team, with Xinrun Wang providing suggestions.The team comprised Weihao Tan, Börje F. Karlsson, Wentao Zhang, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Molei Qin, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, and Yewen Li.Source: §8.2 · p. 13
  • The Stardew Valley implementation, experiments, and analysis were assigned to nine contributors, with four others providing suggestions.The contributors were Weihao Tan, Wentao Zhang, Chuqiao Zong, Yujie Wu, Xiaoqiang Chai, Haochong Xia, Yewen Li, Ruyi An, and Molei Qin; Xinrun Wang, Long Tian, Chaojie Wang, and Börje F. Karlsson provided suggestions.Source: §8.2 · p. 13
  • The Cities: Skylines implementation, experiments, and analysis were done by four contributors, with Xinrun Wang providing suggestions.The contributors were Weihao Tan, Wentao Zhang, Haochong Xia, and Ceyao Zhang.Source: §8.2 · p. 13
  • Yewen Li and Ruyi An handled Dealer’s Life 2 implementation, experiments, and analysis, with Weihao Tan, Wentao Zhang, and Xinrun Wang providing suggestions.Source: §8.2 · p. 13
  • Separate teams handled software-application and OSWorld implementation, experiments, and analysis, with additional contributors providing suggestions.For Chrome, Outlook, CapCut, Meitu, and Feishu, the team was Xinrun Xu, Börje F. Karlsson, and Xiyun Li; Weihao Tan and Xinrun Wang provided suggestions. For OSWorld, the team was Haochong Xia, Tianbao Xie, and Pengjie Gu; Weihao Tan, Xinrun Xu, Xinrun Wang, and Börje F. Karlsson provided suggestions.Source: §8.2 · p. 13
  • Weihao Tan, Xinrun Wang, Börje F. Karlsson, and five others contributed to paper writing, while Zongqing Lu, Shuicheng Yan, and Bo An provided research direction and organizational support.Writing contributors were Weihao Tan, Xinrun Wang, Börje F. Karlsson, Wentao Zhang, Xinrun Xu, Yewen Li, Ruyi An, and Chuqiao Zong; Haochong Xia, Molei Qin, and Ceyao Zhang further contributed with writers in appendix organization.Source: §8.2 · p. 13

A General Implementation … B.2 Objectives

CRADLE combines screenshot-based perception, memory, planning, and low-level keyboard-and-mouse control; the RDR2 section frames its missions as a demanding benchmark and describes the task design and key limitations.

A General Implementation

  • CRADLE uses GPT-4o as its backbone, processes execution video at 2 fps, and expresses keyboard and mouse actions through Python, including timing and coordinate attributes.Its control wrapper supports Windows and macOS and switches between the target application and agent window.Source: A General Implementation · pp. 21–22, Table 7 · p. 21
  • Procedural memory uses text-embedding-ada-002 and cosine similarity to retrieve the top K relevant skills, after which GPT-4o selects from that smaller candidate set.Skill-documentation embeddings are matched against an embedding of the task.Source: A General Implementation · p. 22
  • Episodic memory keeps screenshots and module outputs from the latest five interactions, while a recurrent summary retains long-term task, entity, and behavior history with linear-time inference.The short-term memory is refreshed with new information; the long-term summary is updated from prior summary content and recent screenshots.Source: A General Implementation · pp. 22–23
  • CRADLE uses GPT-4o for OCR and visual analysis, infers the next task from action outcomes, validates documented generated skills, and instantiates selected skills into executable action sequences.Template matching, Grounding DINO, SAM, and visual prompts augment visual grounding.Source: A General Implementation · p. 23

B.1 Introduction to RDR2

  • RDR2 offers a challenging GCC setting because its closed-source world provides no APIs or predefined automation actions, while players can follow its storyline or explore freely.Its RPG controls include movement, mouse-based view changes, combat, inventory, and manipulation.Source: §B.1 · p. 23

B.2 Objectives

  • The first two tutorial missions are divided into 8 and 5 checkpoint-based tasks, respectively, with subtasks classified as easy or hard based on failure likelihood and retry needs.Checkpoints can include failure scenarios; Table 8 describes the tasks and gives a 500-step maximum per task.Source: §B.2 · pp. 23–24, Table 8 · p. 24
  • GPT-4o’s weak spatial understanding and fine manipulation make entering buildings or reaching precise indoor locations difficult, while response latency hinders time-sensitive events such as combat.Source: §B.2 · p. 24

B.3 Implementation Details … B.4.2 Skill Curation

These sections describe how Cradle implements screenshot-driven RDR2 control, manages tasks and skills, and uses self-reflection and skill curation to address errors and game-specific challenges. Case studies illustrate these capabilities in practice.

B.3 Implementation Details

  • Cradle uses screen video and keyboard-and-mouse outputs for RDR2, extracting keyframes for GPT-4o to categorize and OCR, with visual augmentation for spatial and icon-recognition gaps.The augmentation uses Grounding DINO bounding boxes and template matching; GPT-4o provides a detailed description only for the last frame to save interactions.Source: §B.3 · pp. 25, 27, Figure 16 · p. 27
  • Self-reflection samples at most eight frames to assess action success and causes of failure, while low-resolution sequences improve detection of changes between screenshots.It can also decide whether a prolonged action should continue; the authors hypothesize that high-resolution images can obscure overall changes with local detail.Source: §B.3 · p. 27
  • Task inference assigns new tasks a long- or short-horizon label, follows each for three interactions, then returns to the stacked long-horizon task.The authors say binary horizon decisions are more robust than replanning every iteration and help prevent forgetting ongoing long-horizon tasks.Source: §B.3 · pp. 27–28
  • Action planning retrieves relevant and newly generated skills for GPT-4o to choose from, but executes only the first proposed action because later actions often hallucinate.The model also supplies reasons for its skill choices to improve accuracy, stability, and explainability.Source: §B.3 · p. 28
  • Procedural memory begins with basic movement skills, while composite follow/navigation and fight skills reduce model calls or enable faster punches.The agent is intended to learn other skills from scratch where possible; composite skills streamline gameplay rather than being required to complete tasks.Source: §B.3 · pp. 28–29

B.4 Case Studies

  • The case studies provide in-depth discussion of Cradle’s capabilities and the challenges of the GCC setting..Source: §B.4 · p. 29

B.4.1 Self-Reflection

  • After mistakenly selecting a knife for “Protect Dutch,” self-reflection identified the error, enabling the agent to choose a gun and advance the story.The game continued prompting the player to reopen the weapon wheel until the correct weapon was selected.Source: §B.4.1 · p. 29

B.4.2 Skill Curation

  • Skill curation turns detected on-screen instructions into code, and GPT-4o-generated code comments support skill retrieval; curation can regenerate code after a key or button is misrecognized.The framework provides examples of keyboard and mouse APIs, and comment quality affects retrieval and subsequent action planning.Source: §B.3 · p. 28, §B.4.2 · p. 29

B.4.3 Action Execution and Feedback

Because actions can have uncertain or unobservable effects, CRADLE must reason from partial visual feedback to decide whether to repeat a step or try another strategy.

Grounding feedback

  • Pickup attempts may fail because the agent is too far away or its inventory is full, and occlusion can hide the outcome of a successful manipulation.In RDR2, the character can stand near a gun yet still be unable to pick it up.Source: §B.4.3 · p. 31

Imprecise timing in IO-level calls

  • Minor timing mismatches can make a correct keypress fail without visible feedback, while holding a key longer may trigger a different action.Opening a cabinet with [R] may require holding the key until an on-screen progress circle completes; releasing early has no effect.Source: §B.4.3 · p. 31

Change in the semantics of key and button

  • A key or button can change meaning across situations or action steps, so an [E] press intended to hitch a horse may instead dismount it or open a door.An unintended effect can mislead the agent about what happened and distort its next-action planning.Source: §B.4.3 · pp. 31–32

Interference issues

  • Multi-step actions can be interrupted by unrelated events, such as losing aim after being shot or an in-game animation canceling the action.Without internal-state APIs or other feedback, deciding whether to repeat sub-actions or switch strategies is harder.Source: §B.4.3 · p. 32

Action feedback and retry

  • With no direct environment feedback, CRADLE must infer whether an action step needs re-execution; Figure 17 shows self-reflection correcting a failed gun-equip attempt after a knife was selected.All modules run on each iteration, although the figure shows only the relevant ones.Source: Figure 17 · p. 30, §B.4.3 · p. 32

B.5 Limitations of GPT-4o and GPT-4V … C.4.1 Self-Reflection

The passages describe GPT-4o and GPT-4V’s limits in game understanding and how Cradle’s Stardew Valley setup and self-reflection address challenges in perception, navigation, and action. They also introduce Stardew Valley’s evaluation tasks and implementation choices.

B.5 Limitations of GPT-4o and GPT-4V

  • GPT-4o improves spatial perception over GPT-4V, but both need target boxes for fine control, and GPT-4o’s mini-map localization remains unreliable.Localization failures can cause the agent to get lost or miss targets; cropping the exact icon region helps, but makes recognition challenging and unreliable.Source: §B.5 · pp. 32, 33, 35
  • Both models struggle with game-specific icons, and few-shot learning does not generalize, so Cradle matches screenshot icons against prepared templates.This template matching continuously detects and highlights appearing icons.Source: §B.5 · p. 32
  • Long screenshot histories can trigger confusion, hallucinations, and prompt-rule violations; absent a world model, agents misjudge actions, mitigated with summarized memory and added prompt rules.Cradle also makes self-reflection more flexible to address action-selection errors.Source: §B.5 · p. 35

C.1 Introduction to Stardew Valley

  • Stardew Valley offers a lightweight test of production and social behaviors across farming, mining, fishing, crafting, relationships, and town quests, balanced against health, energy, and seasonal time.The experiments use version 1.6.8.Source: §C.1 · p. 35

C.2 Objectives

  • The evaluation covers farm cleanup, crop cultivation through parsnip harvest, and shopping across maps; each task allows at most 100 steps.These tasks test precise positioning and tool choice, long-horizon memory and reasoning, and long-distance navigation, respectively.Source: §C.2 · p. 35

C.3 Implementation Details

  • For Stardew, Cradle uses 3 × 5 grid descriptions and color bands, clips toolbar items for recognition, supplies prior screenshots, and relies on composite skills with at most two skills per action.The skills support door entry, shopping, and movement; first-and-last video frames reduce request tokens, while store screenshots help verify entry.Source: §C.3 · pp. 36–37

C.4 Case Studies

  • The case studies examine Cradle’s self-reflection and task-inference processes in the GCC setting.The examples focus on game-specific behavior.Source: §C.4 · p. 38

C.4.1 Self-Reflection

  • In the Purchasing Seeds task, self-reflection detects that the character is blocked and redirects it upward, allowing it to bypass obstacles and return home.The agent initially tries to go left from the store, but obstacles require moving up first.Source: §C.4.1 · p. 38

C.4.2 Task-inference … D.5 Quantitative Evaluation

Across Stardew Valley and Dealer’s Life 2, Cradle decomposes broad tasks, applies game-specific controls, and demonstrates both successful and failed negotiations. The evaluation spans a seven-day shop-management scenario, while the supplied passages do not report its quantitative results.

C.4.2 Task-inference

  • For Stardew Valley’s parsnip task, Task Inference breaks cultivation and harvesting into tilling, planting, daily watering for four days, and harvesting.This decomposition turns a complete but vague task into specific steps to guide completion.Source: §C.4.2 · p. 39, Figure 25 · p. 39

C.5 Limitations of GPT-4o

  • GPT-4o struggles with pixel-precise positioning in Stardew Valley and recognizing blocked movement from one image; Cradle uses template-matched composite skills and compares frames in Self-Reflection.The composite skills support complex interactions such as purchasing items.Source: §C.5 · p. 39

D.1 Introduction to Dealer’s Life 2 / D.2 Objectives

  • Dealer’s Life 2 is a closed-source game without APIs; Cradle’s task is to manage the shop for a week, making it a zero-shot test of haggling and strategic decision-making.There is no tutorial, so the agent must apply prior knowledge in an unfamiliar environment and weigh item rarity, condition, and customer mood.Source: §D.2 · p. 40, §D.1 · p. 40

D.3 Implementation Details

  • Dealer’s Life 2 uses initialized atomic skills for its finite screen types and fixed buttons; planning permits one skill per action because customer responses make the next state uncertain.Self-Reflection uses the first and last video frames to help the game progress, while unknown action effects can cause severe errors.Source: §D.3 · pp. 40–41

D.4.1 Successful Negotiation

  • In a successful negotiation, Cradle used item quality and the seller’s initial proposal to set an offer near its $160 valuation, then adapted to reach a profitable deal.It adjusted when the seller signaled an inclination to leave.Source: §D.4.1 · p. 42, Figure 27 · p. 42

D.4.2 Unsuccessful Negotiation

  • In an unsuccessful negotiation, Cradle’s below-value offers and the seller’s above-value demands never converged, so the customer left without a deal.Source: §D.4.2 · p. 42, Figure 28 · p. 43

D.4.3 Acquiring and Selling of a Counterfeit Item

  • 145% profit resulted when Cradle resold an item it recognized as potentially counterfeit, a profit-first strategy that risked the character’s reputation.Cradle initially bought the item based only on the seller’s offer, without a price estimate.Source: §D.4.3 · p. 42

D.5 Quantitative Evaluation

  • Cradle’s Dealer’s Life 2 performance was evaluated over a seven-day gaming scenario using nine quantitative financial metrics.The supplied passages do not give the metric values.Source: §D.5 · p. 43

D.6 Evaluation Metrics … E.2 Objectives

Dealer’s Life 2 is evaluated with transaction, profitability, and valuation metrics. Cities: Skylines is presented as a resource-management test, with CRADLE tasked to build a functional city under a constrained first-turn setup.

D.6 Evaluation Metrics

  • Turnover Rate (TR) measures successfully traded items as a share of all items considered, reflecting transaction completion.The denominator includes both successful and unsuccessful trades.Source: §D.6 · p. 43
  • Gross Profit Margin (GPM) measures gross profit relative to sales revenue, while Return on Investment (ROI) measures profit relative to investment.Source: §D.6 · p. 43
  • Valuation Deviation (VD) compares selling price with market valuation, while Buying Price to Valuation Ratio (BPVR) and Selling Price to Valuation Ratio (SPVR) assess purchase and sale prices against valuation.Source: §D.6 · p. 43
  • Average Profit Rate (APR) summarizes overall item profitability; Maximum Return Rate (MRR) and Minimum Return Rate (mRR) capture the highest and lowest item return rates.Source: §D.6 · p. 45

E.1 Introduction to Cities: Skylines

  • Cities: Skylines tests planning across urban challenges such as traffic, budgets, services, and economic growth, with decisions affecting city development and citizen satisfaction.Population growth can bring tax revenue and unlock facilities, but also raises operating expenses and risks shortages or decline if infrastructure is inadequate.Source: §E.1 · p. 45
  • The game offers a risk-free setting for testing agent strategies in urban development and resource allocation, including transportation, energy, and waste-management policies.Source: §E.1 · p. 45
  • The study uses unmodded Cities: Skylines 1, version 1.17.1-f4, because Cities: Skylines 2 had optimization and crash problems, while its predecessor is difficult for beginners.Source: §E.1 · p. 45

E.2 Objectives

  • CRADLE’s mission is to build a city supporting as many people as possible, starting near water with a fixed viewpoint, ₡70,000, basic utilities, and a 1,000-step or budget limit.The setup targets the first milestone, Little Hamlet, at a population of 440; the objective also asks whether the city can maintain utilities and sensibly divide residential, commercial, and industrial zones.Source: §E.2 · p. 46, Figure 30 · p. 46

E.3 Evaluation Metric … F.2 Software Tasks

These sections define how Cradle’s Cities: Skylines city-building performance is assessed, describe implementation choices and failure cases, and introduce the software applications and tasks used to evaluate the agent.

E.3 Evaluation Metric

  • City completeness is measured by closed-loop roads, sufficient water and electricity, zoning over 90% of available area, and maximal population, with a separate population measure after up to three human-assisted fixes.The human assistance addresses small construction mistakes after Cradle finishes the city design.Source: §E.3 · pp. 46–47

E.4 Implementation Details

  • Cities: Skylines uses a 3 × 5 coordinate grid for visual control, a locked camera, and original screenshots for OCR to avoid coordinate overlays obscuring game text.Other modules use augmented screenshots, while information gathering reads budget, population, construction information, and error messages.Source: Figure 31 · p. 46, §E.4 · p. 47
  • Cradle self-explores toolbar objects with SAM grounding and GPT-4o descriptions to generate skills, but GPT-4o often fails to follow the supplied city blueprint and may omit key facilities.Actions are restricted to one skill at a time to prevent combining a placement preview with its confirmation.Source: §E.4 · pp. 47–49

E.5.1 Failure for Road Building.

  • Road-building failures include endpoints offset when roads extend over water and unconfirmed placements that leave road segments disconnected.The endpoint recorded for a road over water can be its sea-level projection rather than its actual endpoint.Source: §E.5.1 · p. 50, Figure 38 · p. 50

E.5.2 Failure for Sufficient Water Supply.

  • Water-supply failures stem from disconnected pipes or a missing pumping station, and the illustrated mistakes can be fixed within three operations, with significant population increases reported after correction.One city example has a local water shortage and a population of 800+.Source: §E.5.2 · pp. 50–51, Figure 39 · p. 51

F.1 Selected Software Applications

  • The software benchmark covers Chrome, Outlook, Feishu, CapCut, and Meitu, with five domain-representative tasks per application; the paper identifies CapCut, Meitu, and Feishu as first targets for an agent.The selected apps span browsing, email, media editing, and office collaboration.Source: §F.1 · p. 52

F.1.1 Brief Descriptions

  • The applications represent web browsing and email, video and photo editing, and business communication and collaboration, including Feishu’s messaging, meetings, and workflow functions.CapCut provides video-editing tools, while Meitu provides photo adjustments, retouching, filters, and effects.Source: §F.1.1 · p. 52

F.2 Software Tasks

  • The 25 software tasks cover representative operations such as browser search and navigation, email management, media editing, and Feishu communication and organization; a CapCut task uses GPT-4o for video-content questions.The video-understanding skill can answer questions such as when a ball enters a goal, and its response is used to complete the task.Source: §F.2 · pp. 52–53

F.3 Quantitative Evaluation · F.4 Implementation Details · F.5 Case Studies

CRADLE achieves 50% overall efficiency across 25 software-application tasks, while its implementation adapts screenshot-based interaction to application-specific visual and interaction challenges.

F.3 Quantitative Evaluation

  • Across 25 application tasks run five times each, CRADLE is evaluated on success rate, average steps and variance, and efficiency relative to human-performed expected steps.The paper also reports quantitative results on OSWorld, with details in Appendix G and an overview in Table 16.Source: §F.3 · pp. 53, 56
  • In Chrome task 3, CRADLE directly uses Ctrl + Shift + T to reopen the last closed page, a shortcut the paper says humans typically do not know.Shortcut calibration can also be used for application-specific controls, such as mapping CapCut’s Alt + arrow keys timeline navigation to time in a predefined skill.Source: §F.3 · p. 56, §F.4 · p. 60
  • 50% overall efficiency across diverse software tasks is reported, with frequent incorrect mouse positioning identified as the main cause of extra steps.Performance varies considerably by task; the evaluation measures success rate, average steps, and efficiency against human-estimated expected steps.Source: §F.3 · p. 56, Table 15 · p. 57

F.4 Implementation Details

  • GPT-4o exhibits spatial-reasoning and UI-identification limitations across applications, while screenshot augmentation and explicit Self-Reflection rules help mitigate selection and success-detection errors.For software tasks, pre- and post-action screenshots usually suffice; dynamic games require multiple frames, and CapCut uses few-shot examples because its UI provides limited feedback.Source: §F.4 · pp. 56–59
  • SAM2SOM uses SAM-generated, numerically tagged bounding boxes from screenshots rather than OS-specific UI APIs, making the augmentation more general but producing more candidate boxes.The paper presents SAM2SOM as a mitigation for current spatial-reasoning issues, not as a core contribution.Source: §F.4 · p. 57, Figure 49 · p. 59
  • Application-specific screenshot handling includes two SAM2SOM label styles, a Feishu watermark filter, and a prominently redrawn magenta mouse pointer.The Feishu filter reduces unnecessary boxes from 216 to 166 in the shown example; the pointer is included in augmented screenshot variants.Source: §F.4 · p. 58, Figure 46 · p. 58, Figure 48 · p. 59
  • Software operations generally map directly to mouse and keyboard actions, so CRADLE adds application-specific predefined skills rather than requiring direct skill generation.Examples include learned shortcuts and complex skills that use GPT-4o to extract information from a video for a CapCut task.Source: §F.4 · p. 60
  • For software, CRADLE can execute two actions in sequence, while labeled-box mouse skills are converted into ordinary mouse actions using each box’s centroid.Task Inference decomposes the next step from the task and subtask descriptions and proposes a new plan when the prior decomposition is unreasonable.Source: §F.4 · p. 60

F.5.1 Task Hardness · F.5.2 Tool Use in CapCut

The Outlook case shows how interface-specific reasoning and verification can make a simple task difficult, while CapCut demonstrates how video-frame analysis and reflection support tool-assisted editing. Across both cases, CRADLE combines task decomposition with low-level action planning and self-reflection.

F.5.1 Task Hardness

  • In Outlook, CRADLE achieved a 40% success rate and 45.45% efficiency on task 1—creating and sending a new e-mail with subject “Hello friend”—which the paper says may seem lower than expected.Source: §F.5.1 · p. 61
  • Task Inference consistently decomposes email creation into four steps, but Action Planning must translate each step into mouse and keyboard operations.The steps are creating an email, adding the recipient, writing the title, and sending it.Source: §F.5.1 · p. 61
  • CRADLE tends to click Outlook’s “New mail” button rather than use a shortcut, making spatial-reasoning or icon-recognition errors possible.The paper notes that such issues led to integrating SAM2SOM into the framework.Source: §F.5.1 · p. 61
  • Outlook’s recipient UI can confuse CRADLE: contact selection takes at least four unintuitive steps, while assistive pop-ups can mislead GPT-4o and obscure the subject area.Typing the address can trigger pop-ups that GPT-4o mistakes for evidence that the address is already entered or duplicated; the contact-selection flow can exceed the allowed steps.Source: §F.5.1 · p. 61
  • Subject positioning errors occur less often, and a disappearing send interface does not always mean success; Self-Reflection detects failed subtask attempts and guides Action Planning forward.It provides failure rationales even when Information Gathering and Action Planning repeat mistakes.Source: §F.5.1 · p. 61

F.5.2 Tool Use in CapCut

  • For CapCut task 5, CRADLE uses video-frame analysis to find when the ball enters the goal, then plans to crop and export the video.It samples frames at 1 fps, records the identified moment in Episodic Memory, and executes go_to_timestamp(seconds=8), delete_right(), and export_project().Source: §F.5.2 · pp. 61–62
  • Few-shot learning in Self-Reflection lets CRADLE expect CapCut’s main window after export_project() and use that screen to verify task success.This expected-screen information supports overall task success detection.Source: §F.5.2 · p. 62

F.6 Limitations of GPT-4o … G.5.2 Planning with Self-reflection

The OSWorld passages describe Cradle’s screenshot-based setup and case studies, while identifying challenges in GUI understanding, precise interaction, and GPT-4o’s attention to visual context. In the examples, grounding supports a successful Save action, and reflection helps recover from command errors and verify task completion.

F.6 Limitations of GPT-4o

  • GPT-4o can misread non-standard interfaces and visual noise, overlook missing context, and misjudge whether visual edits succeeded.Examples include treating a Save dialog as ready without a filename, suggesting a nonexistent File-menu shortcut, and choosing the wrong visual outcome; CapCut’s varied effect workflows also complicate action planning.Source: §F.6 · pp. 62–63

G.1 Introduction to OSWorld

  • OSWorld is a multimodal-agent computer environment designed to provide a unified platform for evaluating open-ended tasks across applications.Source: §G.1 · p. 63

G.2 OSWorld Tasks

  • OSWorld comprises 369 real-world computer tasks, mostly on Ubuntu Linux with a smaller Windows set, each paired with a natural-language instruction and a manually crafted evaluation script.Tasks were collected from the authors and sources including forums, tutorials, and guidelines.Source: §G.2 · p. 63

G.3 Implementation Details

  • Cradle follows screenshot-only GCC in OSWorld, using GPT-4o and PyAutoGUI; grounding maps labels and boxes to coordinates, while reflection checks action and task success and infeasibility.The implementation uses the default experimental settings and the same prompt templates for all 369 tasks.Source: §G.3 · pp. 64–65

G.4 Application Target and Setting Challenges

  • OSWorld evaluations expose challenges in GUI understanding and operational knowledge, while agents also struggle with lengthy observation and action records.The passage calls for stronger visual and GUI interaction abilities, and agent architectures that support autonomous exploration and synthesis.Source: §G.4 · pp. 65–66

G.5.1 Information Gathering

  • With SAM grounding, the agent identified the Save button as box 32, clicked it, and successfully saved the PDF.The bounding box helped identify the GUI element and determine its position for mouse control.Source: §G.3 · p. 65, §G.5.1 · p. 66, Figure 56 · p. 66

G.5.2 Planning with Self-reflection

  • After command errors, the agent inspected the directory, tried alternatives, and verified that file1 had been copied to dir1, dir2, and dir3 before marking the task successful.The task instruction requested copying file1 to all three directories; the passage reports two initial attempts with errors and warnings, followed by a new command and a file check.Source: §G.5.2 · p. 66

G.6 Quantitative Evaluation · H Cradle Prompts

This section reports OSWorld evaluation findings, including stronger performance in Chrome and GIMP and limitations in VLC and VSC, and documents game- and application-specific prompts. These prompts structure screenshot- and history-based state assessment, progress reasoning, planning, action selection, and reflection into grounded next actions.

G.6 Quantitative Evaluation

  • The agent performed better in Chrome and GIMP, while differences in OS, Writer, and VSC were less statistically significant because those domains had fewer tasks.The evaluation followed OSWorld’s experimental settings and ran each experiment only once; Table 16 lists domain-level success rates.Source: §G.6 · p. 66, Table 16 · p. 67
  • The complex pipeline and limitations of current grounding tools and GPT-4 hindered performance in VLC and VSC, which the authors identify as future directions for real-world use.Source: §G.6 · p. 68

H Cradle Prompts

  • The appendix exemplifies prompts for each framework module and states that all prompts and customizations are included in Cradle’s open-source GitHub repository.Source: H Cradle Prompts · p. 68

H.1 Prompts for RDR2

  • RDR2 information-gathering prompts classify screenshots and task horizon, identify relevant objects, and extract on-screen text into categories such as status, notifications, guidance, and dialogue.The prompts also specify how to translate keyboard or mouse guidance into executable control code.Source: §H.1 · pp. 68–71
  • RDR2 prompts assess whether the last action succeeded and why, summarize recent events as a timeline, and use task context, memory, reflection, and available actions to plan the next move.For movement, the reflection prompt treats unchanged character position across sequential frames as failure; the inference prompt requests a history summary of at least 10 sentences.Source: §H.1 · pp. 73–75
  • RDR2 action prompts require selecting only valid actions in Python format and applying explicit rules for navigation, minimap use, obstacles, following, and recovery.The reasoning prompt also asks about recent history, self-reflection, the previous action, and relevant screenshot targets or bounding boxes before choosing the next action.Source: §H.1 · pp. 76–77

H.2 Prompts for Cities: Skylines

  • Cities: Skylines information gathering extracts the screenshot description, budget, population, error messages, and construction information into labeled fields.Construction information is textual, including cost or placement guidance, and absent errors or construction information should be reported as null.Source: §H.2 · p. 78
  • The self-reflection prompt judges action success from the executed action and textual construction information, with explicit placement rules and a requirement that only confirmation completes a building subtask.It also asks for a likely failure cause, subtask completion, and whether the subtask itself is reasonable.Source: §H.2 · p. 79
  • Task inference summarizes prior progress and proposes a concrete next subtask, following a supplied task decomposition and rules for utilities, electricity, and initial road construction.The prompt asks whether the previous subtask is complete and directs the agent to continue it if not, rather than assuming completion from a successful action alone.Source: §H.2 · pp. 80–82
  • Action planning uses task, coordinates, action history, reflection, error and construction information, and the valid action set to reason about the next step.Before placing anything, the agent must check whether the corresponding menu is open; otherwise it should open that menu first.Source: §H.2 · pp. 82–83
  • Cities: Skylines action prompts require valid Python-formatted actions and constrain road placement, retries, and placement confirmation to avoid invalid or repeated moves.Roads must connect to existing roads, follow grid coordinates, and be at least 300 pixels long; failed placements should be retried with a position change of at least 100 pixels, while successful placements should be confirmed or canceled.Source: §H.2 · pp. 83–84

H.3 Prompts for Stardew Valley

  • Stardew Valley information-gathering prompts divide screenshots into a 3x5 grid and request cell descriptions plus structured date, energy, weather, dialogue, and other fields.They identify date and time in the upper-right, energy in the bottom-right, and require “null” when weather, dialogue, or other information is absent.Source: §H.3 · p. 85
  • Stardew self-reflection prompts assess whether the last action succeeded, identify a likely failure cause, and determine whether the subtask and target task are complete.Movement is unsuccessful only if the character's position stays unchanged across sequential frames; uncertain actions are treated as successful, and an unavailable interaction calls for moving elsewhere.Source: §H.3 · p. 86
  • Stardew task-inference prompts maintain a precise history of progress and propose a concrete next subtask suited to the current state.For cultivation, the history tracks watering day by day, while the prompt says seeds need watering once daily and directs the agent to sleep if already watered that day.Source: §H.3 · pp. 87–88
  • Stardew action-planning prompts use task state, toolbar, prior actions, reflection, and a valid action set to select the next action while enforcing game-specific tool and movement rules.They limit output to two Python-formatted actions, distinguish do_action() from use_tool(), require selecting toolbar items first, and forbid repeating a failed action or moving after get_out_of_house().Source: §H.3 · pp. 89–90
  • The screenshot-description prompt divides the image into a 3×5 grid and requests cell-by-cell descriptions plus date/time, energy, weather, dialogue, and other information.It specifies where date/time and energy appear and requires a fixed output format.Source: §H.3 · pp. 90–91
  • The self-reflection prompt assesses whether the last action succeeded, why it may have failed, and whether the subtask and target task are complete.It treats uncertain action failures as successes and asks for a single likely cause when failure is established.Source: §H.3 · p. 92
  • The task-inference prompt summarizes task progress and successful events day by day, then reuses or proposes a feasible subtask based on the current situation.For crop watering, it tracks watering by day, instructs watering only once daily, and directs the agent home to sleep after watering.Source: §H.3 · pp. 93–94
  • The action-planning prompt uses the current subtask, toolbar, recent history, and valid skill set to reason about and output the next game action.Its rules limit output to at most two Python-formatted actions and constrain tool selection, movement, obstacle clearing, and avoiding repeated failed actions.Source: §H.3 · pp. 94–97
  • The shopping information-gathering prompt requests a left-to-right, top-to-bottom description of a 5×3 screenshot grid focused on the current subtask.It also provides the current subtask as context for identifying relevant screenshot contents.Source: §H.3 · pp. 97–98
  • Stardew Valley prompts extract date and time, energy, weather, and dialogue from screenshots, with “null” used when applicable information is absent.Source: §H.3 · p. 98
  • Stardew self-reflection checks whether actions succeeded from sequential frames, diagnoses likely failure causes, and verifies store entry or purchases where relevant.Source: §H.3 · p. 99
  • Stardew task inference summarizes prior progress and selects a concrete, feasible next subtask, using completion evidence and situation-specific shopping or travel rules.Source: §H.3 · pp. 100–101
  • Stardew action planning requires valid Python actions, at most two per response, and checks reachability, contact, prior failures, and task relevance before acting.Source: §H.3 · pp. 102–103

H.4 Prompts for Dealer’s Life 2

  • Dealer’s Life 2 prompts describe the screenshot’s page and task-relevant information, capture the bank balance as budget, and provide task, action, and history context for reflection.Source: §H.4 · pp. 104–105
  • Dealer’s Life 2 prompts require history summaries to track task progress and successful events day by day, then assess or propose a concrete, feasible subtask.For watering seeds, the summary should record watering frequency and estimate days until harvest; subtask reasoning should check whether the previous subtask finished or remains suitable.Source: §H.4 · pp. 105–106
  • Dealer’s Life 2 action planning uses screenshots, task and history context, and a valid action set to select executable actions, with trading rules based on customer role and item details.The prompts also ask the agent to judge action success, identify likely failure causes, and determine whether subtasks or the target task are complete.Source: §H.4 · pp. 105, 107, 108

H.5 Prompts for Software Applications

  • Chrome information-gathering prompts request detailed screenshot descriptions, one-by-one descriptions of labeled bounding boxes, and a relevant target object or null.The prompts instruct the agent to describe cursor proximity and avoid unsupported assumptions about layout or uncertain elements.Source: §H.5 · p. 109
  • Chrome self-reflection prompts assess the last action using sequential screenshots, mouse-position changes, visible typed text, UI effects, and any execution error report.A separate task-level success check considers the task description, history, screenshots, and success-detection flag, and requests missing steps when unsuccessful.Source: §H.5 · p. 111
  • Chrome task-inference prompts use recent screenshots, event descriptions, and prior history summaries to propose the next suitable subtask for the target task.Source: §H.5 · p. 112
  • History prompts require tracking completed and pending subtasks, summarizing successful events, and choosing a concrete next subtask suited to the current situation.The subtask should contribute to the overall task and be completable within a few actions; the prompts also require a prescribed response format.Source: §H.5 · pp. 112–113
  • Chrome action planning combines task context, screenshot descriptions and labels, action history, and a valid action set to select and explain the next executable action.The prompt prefers keyboard actions, requires checking the screenshot and relevant label IDs rather than assuming UI layout, and specifies Python-format actions from the valid set.Source: §H.5 · pp. 113–114
  • Outlook information gathering asks for a detailed screenshot-based UI description and identification of a relevant visible target, or an explicit null or off-screen response.It also requests reasoning for the selected target or for why no target needs detection.Source: §H.5 · pp. 116–117
  • Outlook self-reflection checks whether the last action produced the intended UI change, diagnoses a likely failure cause, and assesses success for the overall task.It directs the agent to inspect sequential screenshots, not rely solely on textual history, and avoid repeating an ineffective action before trying something else.Source: §H.5 · p. 118
  • Outlook prompts summarize task progress and completed actions, then choose or revise a concrete subtask based on prior reasoning and reflection.They require tracking unfinished prerequisites and avoiding assumptions about information not previously provided.Source: §H.5 · p. 120
  • Outlook action planning uses screenshots, task history, success status, and available skills to select precise next actions, preferring applicable keyboard shortcuts.The prompt also guards against repeating ineffective actions, requires checking UI labels and text-field focus, and restricts output to valid actions.Source: §H.5 · pp. 121–123
  • CapCut information gathering requires detailed screenshot and cursor descriptions, a one-by-one inventory of labeled bounding boxes, and identification of a relevant target object when needed.The target-object prompt permits “null” when no explicit object is specified or detection is unnecessary.Source: §H.5 · p. 124
  • CapCut self-reflection evaluates whether an executed action had the correct effect using task context, action implementation, errors, history, success status, and the valid action set.It also supplies current-versus-previous screenshot and mouse-position sameness flags.Source: §H.5 · p. 125
  • Self-reflection prompts assess whether the last action succeeded using sequential screenshots and action-specific checks, while task-level detection determines whether the overall task succeeded.When unsuccessful, the prompt asks for a likely cause; overall-task assessment also identifies missing steps or returns only “SUCCESSFUL” when complete.Source: §H.5 · pp. 125–126
  • CapCut task-inference prompts ask the model to summarize progress, assess whether a subtask is finished, and propose a concrete next subtask grounded in the current task and history.The prompts also describe task-specific procedures, including switching to a relevant panel before dragging an object onto the timeline.Source: §H.5 · pp. 126–128
  • A screen-classification prompt asks the model to select one screenshot class and return it in a prescribed JSON format.The prompt specifies the output type as “screen_classification” and gives “map” as the output format example.Source: §H.5 · p. 128
  • Action-planning prompts combine task state, screenshots, history, available skills, and success detection to choose a valid next action or stop when the task is complete.They prioritize keyboard actions, discourage repeating unsuccessful actions, and prescribe UI-specific handling such as selecting material panels before dragging media to the timeline.Source: §H.5 · pp. 128–131
  • Meitu information-gathering prompts request detailed screenshot descriptions, descriptions of every labeled bounding box, and identification of a task-relevant object when detection is needed.They require “null” when no explicit object is specified or object detection is unnecessary.Source: §H.5 · pp. 131–132
  • The Meitu reflection prompt checks the last action against sequential screenshots and action-specific success rules, then assesses overall task completion and missing steps.It asks for a likely cause when an action fails and requires an empty action when the task is judged successful.Source: §H.5 · pp. 133–134
  • The Meitu task-inference prompt summarizes progress and successful events, tracks completed subtasks, and proposes or reuses a concrete next subtask based on the current state.It directs the model to ignore further history questions if the task is complete and to choose a straightforward next subtask without unsupported assumptions.Source: §H.5 · pp. 134–135
  • The Meitu action-planning prompt selects only valid skills, favors keyboard shortcuts and Enter, uses labeled UI targets for mouse actions, and avoids repeating failed actions.It requires clicking the relevant input before scrolling or typing and specifies executable Python-format actions.Source: §H.5 · pp. 135, 137
  • The Feishu information-gathering prompt requests detailed screenshot descriptions focused on relevant UI state, open panels, the active area, and the cursor’s nearby target.It cautions against assumptions about layout and asks that bounding-box overlays be excluded from the description.Source: §H.5 · pp. 138–139
  • Self-reflection evaluates the last action against sequential screenshots, diagnoses a likely cause of failure, and separately judges whether the overall task succeeded.For mouse movement, unchanged or incorrect cursor movement counts as failure; the success assessment concerns the overall task, not only individual actions.Source: §H.5 · pp. 140–141
  • Task inference summarizes successful events and completed or pending subtasks, then decides whether to reuse the previous subtask or propose a concrete next one.The prompt asks for task progress and history-based reasoning without assumptions about information not previously stated.Source: §H.5 · p. 142
  • Action planning grounds the next action in the screenshot, task history, and valid skill set, requiring Python-formatted actions and correct input focus before typing.It also requires using a labeled click skill when a bounding box is available, and adapting after coordinate or availability failures.Source: §H.5 · pp. 143–145
Loading 2403.03186v3…