PaperlayerView exact revision on arXiv

Source-linked AI summary

A data-driven approach for learning to control computers

Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, Timothy Lillicrap

arXiv:2202.08137v2cs.LG

TL;DR

Computer control aims to let machines perform everyday digital tasks using human-compatible keyboard-and-mouse interaction, but scalable methods for this setting remain underexplored. The paper combines reinforcement learning with behavioural cloning from human-computer interaction data and finds state-of-the-art and human-level mean performance across MiniWob++, with strong cross-task transfer. These results support a unified human-agent interface as a route toward broader computer control.

  • Problem

    Digital device control needs agents that turn natural-language goals into actions through human-compatible interfaces, while this domain has received relatively less attention.

    Source: §1 · p. 1

  • Method

    The paper trains agents with a straightforward mixture of behavioural cloning and reinforcement learning, co-training across tasks to improve efficiency and build broad capabilities.

    Source: §2.5 · p. 5

  • Results

    The method surpasses previous state-of-the-art performance by a wide margin and achieves human-level mean performance across MiniWob++, with performance continuing to improve using datasets up to 400× larger than prior studies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate the usefulness of a unified human-agent interface and suggest a scalable route toward controlling computers more generally as humans do.

    Source: §1 · p. 2, §4 · p. 9

    MiniWob++ evaluates turning language goals into browser actions but hides the complexity of understanding human intent, language, and judgement together.

    Source: §4 · p. 8

Abstract

from arXiv · show

It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-scale expert demonstrations and human judgements of interactive behaviour, which are two ingredients that have driven much recent success in AI. Here we investigate the setting of computer control using keyboard and mouse, with goals specified via natural language. Instead of focusing on hand-designed curricula and specialized action spaces, we focus on developing a scalable method centered on reinforcement learning combined with behavioural priors informed by actual human-computer interactions. We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark, a challenging suite of computer control problems, and find strong evidence of cross-task transfer. These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers. Altogether our results suggest a formula for achieving competency beyond MiniWob++ and towards controlling computers, in general, as a human would.

1. Introduction

Digital device control combines human-compatible multimodal actions with abundant human data, yet has received less attention than other AI domains. This work studies scalable keyboard-and-mouse control using reinforcement learning and behavioural cloning, finding that substantially larger human datasets enable state-of-the-art and human-level MiniWob++ performance.

Motivation

  • Large-scale human demonstrations and judgements are key ingredients behind recent progress in models for language, code, and multimodal interactive behaviour.The paper extends this data-informed perspective to computer control.Source: §1 · p. 1
  • Digital device control offers expressive, composable, human-compatible affordances and scales well because it relies primarily on digital information.The study focuses on keyboard and mouse actions with pixel and Document Object Model observations.Source: §1 · p. 1

Benchmark

  • MiniWob++ provides instruction-following tasks involving clicking, typing, and form-filling, with programmatically defined rewards for initial computer-control research.The benchmark is positioned as a step toward more open-ended interactions involving natural-language goals and human performance judgements.Source: §1 · p. 1

Method

  • The method combines reinforcement learning with behavioural cloning using aligned human and agent keyboard-and-mouse action spaces.The approach avoids hand-designed curricula and specialized action spaces while retaining desirable data- and compute-scaling properties.Source: §1 · p. 1

Results

  • 400× larger human trajectory datasets than prior studies yield continuing performance improvements and surpass previous state of the art, reaching human-level mean performance across MiniWob++.The authors identify human-data scale as the primary missing factor in the simple imitation-plus-reinforcement-learning approach.Source: §1 · p. 2

2. Methods

The methods establish a human-compatible browser-control benchmark, secure low-latency environment, multimodal CC-Net agent, large-scale demonstrations, and jointly trained BC/RL procedure across MiniWob++.

2.1. MiniWob++

  • MiniWob++ provides browser-based instruction-following tasks with programmatic rewards, while the study uses mouse and keyboard actions rather than DOM-specific actions.This interface aligns agent actions with human demonstrations and is intended to transfer beyond compact web DOMs.Source: §2.1 · p. 2

2.2. Environment Interface

  • The environment uses a sandboxed Chrome browser, direct Chrome DevTools Protocol access, and X11 mouse, keyboard, and framebuffer interfaces for secure, human-equivalent interactions.The stack also targets low latency and supports full-browser controls, demonstrations, and context-sensitive cursor rendering.Source: §2.2 · p. 3

2.3. Agent Architecture

  • CC-Net combines ResNet visual features, cross-attended language inputs, an 8-layer multimodal transformer, previous-action embeddings, and two LSTMs.Its policy outputs action type, cursor coordinates, keyboard-key index, and task-field index, using attention to select task-field strings.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4. Human Data Collection

  • The study collected over 2.4 million demonstrations from 77 participants, totaling approximately 6300 hours, after observing that some tasks require practice or specialized knowledge.Human trajectories were filtered for success before behavioral cloning.Source: §2.4 · p. 5, §2.5 · p. 5

2.5. Training

  • Agents were trained by co-training behavioral cloning and VMPO reinforcement learning with equally weighted losses across all tasks.The demonstrations were split into train and test sets, filtered by final reward, and stripped of up to ten consecutive no-op steps.Source: §2.5 · p. 5
  • Co-training all tasks was chosen because it improves per-task training efficiency through transfer and supports one agent with many capabilities.The reported conditions used hand-tuned settings and one seed per condition because of computational requirements.Source: §2.5 · p. 5

3. Results

The combined behavioural-cloning and reinforcement-learning agent reaches human-level mean performance and exceeds aggregated prior state of the art across MiniWob++, while also showing transfer across tasks. Results further identify dataset scale, DOM observations, and mouse-keyboard actions as important design factors.

3.1. Human-Level Performance on MiniWob++

  • The agent exceeds aggregated prior state-of-the-art performance by a wide margin and achieves human-level mean performance across MiniWob++.This performance is enabled by combining behavioural cloning and reinforcement learning.Source: §3.1 · p. 5
  • Humans and the agent perform nearly perfectly on most tasks, although the agent scores zero on simon-says and terminal while exceeding mean human performance on moving-items.The agent’s failure on simon-says is linked to observing the environment at about 2Hz, while human difficulty on moving-items is likely related to network latency.Source: Figure 4 · p. 6, §3.1 · p. 6

3.2. Task Transfer

  • Co-training one agent on all 104 tasks is significantly more data-efficient than training separate agents per task, providing evidence of cross-task transfer.With a fixed training-step budget per task, the multi-task regime produces a strong performance boost as a function of frames consumed on each task.Source: §3.2 · p. 6, Figure 5 · p. 6

3.3. Scaling

  • Human trajectory data remains crucial: using 1/1000 of the dataset causes rapid overfitting and no significant gain over reinforcement learning alone, while performance improves across more than three orders of magnitude.The authors note that algorithmic or architectural changes might improve low-data performance, but collecting more data is simpler.Source: §3.3 · pp. 6–7, Figure 7 · p. 7
  • The book-flight task, with over 1.9 billion possible action permutations, is solved by combining behavioural cloning and reinforcement learning, demonstrating the value of human data for exploration.The result indicates that behavioural priors can make a difficult exploration problem tractable for the combined method.Source: Figure 6 · p. 7

3.4. Input and Output Ablations

  • Removing DOM observations causes a 75% performance drop, whereas removing pixels is less harmful because MiniWob++ canvas information is represented in the DOM.This also exposes a boundary: more general computer-control tasks may require substantially better pixel-only performance.Source: §3.4 · p. 7, Figure 8 · p. 8
  • Without environment-provided text input choices, the agent can still solve form-filling tasks by learning from human trajectories to highlight and drag text into the required field.This behaviour would not be straightforward in the original Selenium environment.Source: §3.4 · p. 7, Figure 8 · p. 8
  • Actions targeting specified DOM elements are less effective than baseline mouse and keyboard actions, partly because the alternative action space cannot express clicking locations, dragging, or highlighting.The mismatch between recorded human trajectories and available DOM actions further nudges imitation toward suboptimal trajectories.Source: §3.4 · p. 7, Figure 8 · p. 8

4. Discussion, Limitations, & Related Work

The paper argues that large-scale human demonstration data makes straightforward deep RL effective for keyboard-and-mouse computer control, while MiniWob++ remains a limited test of broader device assistance. It reports strong benchmark performance but identifies data demands, unexplored performance improvements, and unresolved human-intent challenges.

Device assistance

  • MiniWob++ evaluates scripted language goals with programmatic rewards, leaving human intent, language, and preferences for future work.The paper frames useful device assistance as requiring both understanding human intent and converting language goals into device actions.Source: §4 · p. 8

Related work

Limitations

  • 400× more human data than the initial collection was required to achieve human-level performance, while collecting more data could be cheaper than developing low-data algorithms.The authors describe this as a one-time cost and note that it remains much smaller than recent language-modeling datasets.Source: §4 · p. 8

Performance

  • The method is deliberately simple, but task sampling, objective tuning, and decaying KL penalties remain unexplored avenues for improving performance.The authors specifically mention over-representing harder tasks and regularizing RL toward a pretrained behavioural-cloning policy.Source: §4 · pp. 8–9

Discussion

  • The approach resolves the original MiniWob hypothesis that deep RL can solve keyboard-and-mouse computer tasks when augmented with sufficient demonstration data.The authors characterize the method as straightforward rather than algorithmically novel.Source: §4 · p. 9

Conclusion

  • The authors expect assistance on even a small fraction of daily digital-device tasks to support a cycle of human feedback, agent improvement, and new capabilities.This conclusion presents iterative feedback as a prospective benefit rather than an achieved result.Source: §4 · p. 9

5. Contributions

The project was conceived, managed, implemented, analyzed, and written by distinct overlapping author groups. The listed contributions span environment and infrastructure development, agent research, experimentation, analysis, and manuscript preparation.

Contributions

  • The contributors divided work across project conception, management, infrastructure, agent architecture, experiments, analysis, and manuscript writing.The contribution statement assigns these roles to specific author groups.Source: §5 · p. 9
  • A large author group developed the environment, data collection, experiment, and agent infrastructure.The statement credits GT, AS, TP, AM, PCH, PG, RC, and JA with these activities.Source: §5 · p. 9
  • Agent architecture, experiments, and data analysis were developed or conducted by AS, DR, and PCH.The same authors also contributed to the manuscript alongside DR and TL.Source: §5 · p. 9
Loading 2202.08137v2…