PaperlayerView exact revision on arXiv

Source-linked AI summary

D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI

Suhwan Choi, Jaeyoon Jung, Haebin Seong, Minchan Kim, Minyeong Kim, Yongjun Cho, Yoonshik Kim, Yubeen Park, Youngjae Yu, Yunsung Lee

arXiv:2510.05684v3cs.AIcs.CVcs.RO

TL;DR

Embodied AI lacks internet-scale training data because physical trajectories are costly to collect. D2E standardizes desktop interactions, pseudo-labels gameplay, and pretrains representations that transfer across manipulation and navigation tasks.

  • Problem

    Embodied AI datasets remain small, domain-specific, and fragmented because physical trajectories require specialized hardware, costly human operation, and complex annotation pipelines.

    Source: §1 · p. 1

  • Method

    D2E standardizes desktop demonstrations with OWA, uses Generalist-IDM to pseudo-label gameplay, and pretrains vision-action representations for robotics transfer.

    Source: §1 · pp. 2–3

  • Results

    96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation are reported for desktop-pretrained VAPT.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Desktop-pretrained sensorimotor patterns can enhance performance in embodied AI domains, offering a practical alternative to costly physical data collection.

    Source: §1 · p. 2

    Real-world validation covers only a single task setting, leaving broader manipulation and navigation validation for future work.

    Source: J LIMITATIONS · p. 30

Abstract

from arXiv · show

Large language models leverage internet-scale text data, yet embodied AI remains constrained by the prohibitive costs of physical trajectory collection. Desktop environments -- particularly gaming -- offer a compelling alternative: they provide rich sensorimotor interactions at scale while maintaining the structured observation-action coupling essential for embodied learning. We present D2E (Desktop to Embodied AI), a framework that demonstrates desktop interactions can serve as an effective pretraining substrate for robotics embodied AI tasks. Unlike prior work that remained domain-specific (e.g., VPT for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a complete pipeline from scalable desktop data collection to verified transfer in embodied domains. Our framework comprises three components: (1) the OWA Toolkit that unifies diverse desktop interactions into a standardized format with 152x compression, (2) the Generalist-IDM that achieves strong zero-shot generalization across unseen games through timestamp-based event prediction, enabling internet-scale pseudo-labeling, and (3) VAPT that transfers desktop-pretrained representations to physical manipulation and navigation. Using 1.3K+ hours of data (259 hours of human demonstrations and 1K+ hours of pseudo-labeled gameplay), our 1B-parameter model achieves 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation, matching or surpassing models up to 7x larger, such as \pi_{0} (3.3B) and OpenVLA (7B). These results demonstrate that sensorimotor primitives learned from digital interactions transfer effectively to real-world physical tasks, establishing desktop pretraining as a practical paradigm for embodied AI. All resources are publicly available at https://worv-ai.github.io/d2e.

1 INTRODUCTION

D2E treats abundant desktop interaction as a scalable pretraining source for embodied AI, addressing costly physical-trajectory collection. Its pipeline standardizes desktop data, pseudo-labels gameplay, and evaluates transfer to robotics.

1 INTRODUCTION

  • Desktop interactions offer abundant, standardized trajectories while preserving the observation–action coupling needed for embodied learning.Physical trajectories require specialized hardware, costly human operation, and complex annotation pipelines.Source: §1 · p. 1
  • Timestamp-aware next-event prediction lets Generalist-IDM generalize to unseen games and pseudo-label over 1,000 hours of YouTube gameplay.The model is described as substantially outperforming specialist baselines with minimal compute requirements.Source: §1 · pp. 2–3
  • 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation are reported for VAPT trained on 1.3K hours of desktop data.The results are presented as evidence of desktop-to-robotics transfer.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • OWA synchronizes desktop events and compresses storage up to 152× while supporting 335 hours of demonstrations across 31 games and applications.The collection involved 14 annotators over one month, compared with DROID’s 50 collectors across 13 institutions over 12 months for a dataset of comparable scale.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses real-robot data collection and desktop action modeling, but desktop resources remain domain-specific or lack open, standardized coverage. D2E relates its multi-game dataset and event-timestamp prediction to these gaps.

Collecting Data for Vision-Action Pretraining

  • Desktop datasets and toolkits remain limited; VPT is Minecraft-specific, SIMA keeps data proprietary, and PLAICraft focuses on Minecraft, while D2E offers an open dataset spanning 31 games.The passage identifies broad cross-application generalization as requiring unified schemas covering diverse desktop applications.Source: §2 · p. 3

Inverse Dynamics Models

  • Unlike tick-based IDMs that predict at fixed intervals, D2E predicts events and timestamps to avoid no-op ticks and improve temporal resolution in sparse-event settings.VPT [Baker et al., 2022] pseudo-labeled Minecraft gameplay, while D2E explores a Generalist-IDM across multi-game desktop settings.Source: §2 · p. 3

Collecting Data for Vision-Action Pretraining

  • OWA’s recording and storage architecture synchronizes multimodal desktop streams and uses OWAMcap with H.265 media references to reduce storage while retaining event fidelity.Reported reductions include 152× for the VPT dataset and 34.45× for CS:GO.Source: Figure 2 · p. 4

3 OPEN-WORLD AGENTS TOOLKIT

The OWA Toolkit standardizes desktop interaction capture and data processing across applications. Its recording, storage, and loading pipeline is designed to support scalable vision-action pretraining.

3.1 OCAP: SYNCHRONIZED DESKTOP RECORDER

  • The ocap recorder synchronizes video, audio, keyboard, and mouse streams and supports real-time FHD/QHD recording at 60 Hz on consumer GPUs.Hardware acceleration keeps overhead low so normal user activities remain unaffected.Source: §3.1 · p. 4

3.2 OWAMCAP: STANDARDIZED DATA FORMAT

  • OWAMcap standardizes screen, keyboard, and mouse event schemas and uses MediaRef video storage, achieving 217× compression over raw captures and 68× over PNG.The schemas support unified processing, while MediaRef uses H.265 and maintains sufficient visual quality for agent training.Source: §3.2 · pp. 4–5

3.3 OPTIMIZED DATA PIPELINE

  • The four-stage data pipeline transcodes media, converts events to HuggingFace datasets, packs episode-preserving FSLDataset sequences, and loads media on the fly with adaptive batch decoding.FSLDataset coalesces fine-grained random access; adaptive decoding addresses the need to seek to keyframes in compressed video.Source: §3.3 · p. 5
  • 4.77 it/s with one dataloading worker matches the baseline’s 4.55 it/s with 16 workers; optimized decoding reaches 119.16 img/s, 10.2× baseline.The decoding benchmark also reports 18.73 KB average disk read per image, 3.4× less than baseline and 41× less than TorchCodec.Source: §3.3 · p. 5, Table 2 · p. 6, Table 1 · p. 6

3.4 COLLECTING HUMAN DEMONSTRATIONS AT SCALE

  • The toolkit collected 335 hours of human demonstrations across 31 games, spanning diverse genres and interaction styles.Gameplay was selected for behavioral diversity and reduced privacy concerns; the toolkit can also capture tasks such as web surfing and productivity applications.Source: §3.4 · pp. 5–6

4 GENERALIST INVERSE DYNAMICS MODEL

The Generalist-IDM uses timestamp-based event prediction to infer actions from desktop interactions and extend the training corpus with pseudo-labeled gameplay. Its temporal-offset objective uses future observations, and its generalization supports pseudo-labeling across diverse games.

4 GENERALIST INVERSE DYNAMICS MODEL

  • A Generalist-IDM trained on a multi-domain desktop corpus is intended to generalize across heterogeneous interaction patterns, including environments unseen during training.This capability provides a pathway to pseudo-labeling gameplay beyond the Minecraft-only scope of VPT [Baker et al., 2022].Source: §4 · p. 6

4.1 TIMESTAMP-BASED EVENT TOKENIZATION

  • Timestamp-based event prediction records each event and its time, preserving asynchronous alignment across screen, keyboard, and mouse streams while avoiding empty no-op ticks.The approach avoids the fixed prediction windows used by tick-based methods.Source: §4.1 · pp. 6–7

4.2 NEP-τ: NEXT-EVENT PREDICTION WITH TEMPORAL OFFSET

  • NEP-τ shifts the observation window forward by τ steps to incorporate future observations without jointly encoding entire past and future trajectories.The objective predicts actions from preceding observations and actions; performance stabilizes at τ ≥100 ms, so the experiments use τ = 100 ms.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 PSEUDO-LABELING WITH YOUTUBE GAMEPLAY VIDEOS

  • The Generalist-IDM enables pseudo-labeling YouTube gameplay across diverse games, adding 1,055 hours across twenty publicly shared titles without filtering domain-specific interfaces.The pipeline retrieves licensed long-form gameplay at 20 Hz, converts frames to screen events, predicts keyboard and mouse events, and applies consistency checks.Source: §4.3 · pp. 7–8

5 RESULTS

The Generalist-IDM performs across in-distribution and unseen games, while VAPT transfers desktop-pretrained representations to simulated and real-world robotics tasks. Results vary by task: pseudo-labels improve navigation, but do not add manipulation gains.

5.1 PERFORMANCE OF THE GENERALIST-IDM

  • The Generalist-IDM improves in-distribution Pearson correlation by 39.5 points on Stardew Valley X and keyboard accuracy by 57.6 points on Brotato.The evaluation spans six 2D and 3D games and compares the generalist model with specialist models trained individually for each game.Source: Table 4 · p. 8, Table 5 · p. 8, §5.1 · p. 8
  • On unseen games, the Generalist-IDM reaches 63% keyboard accuracy in Battlefield 6 and nearly 28% in Ogu and the Secret Forest, up from about 12% for the specialist.A few-shot prefix improves predicted mouse scale ratio in Battlefield 6, while Ogu and the Secret Forest tests transfer across a larger domain gap.Source: §5.1 · p. 8, Table 6 · p. 9

5.2 TRANSFERABILITY TO DOWNSTREAM TASKS

  • The authors identify action-modality alignment, goal-directed sequential decision-making, and diversity across 20 games as properties that may facilitate transfer.They also report that VAPT-initialized models converge immediately while baseline models show an initial plateau.Source: §5.2 · p. 9
  • 96.6% on LIBERO Total and 83.3% on CANVAS navigation; pseudo-labels add gains for navigation, not manipulation.On LIBERO, VAPT without pseudo-labels reaches 96.6% Total; on CANVAS, adding pseudo-labeled demonstrations raises performance from 75.3% to 83.3%.Source: Table 7 · p. 9, §5.2 · pp. 9–10, Table 9 · p. 10
  • On Meta-World, VAPT improves average success by roughly 5% over the InternVL3-1B baseline, with the gap largest in Hard and Very Hard tasks.For Very Hard tasks, the reported comparison is 8.0% versus 20.0–24.0%.Source: §5.2 · pp. 9–10, Table 8 · p. 10
  • In the SO101 real-world pick-and-place task, both VAPT variants reach 80% success, compared with 70% for the InternVL3-1B baseline.The evaluation used 30 rollouts per policy across five cube starting positions.Source: Table 8 · p. 10, §5.2 · p. 10

6 CONCLUSION

D2E proposes desktop interactions as a scalable pretraining resource for embodied AI and reports transfer to manipulation and navigation benchmarks. The authors release the tools, datasets, and models publicly.

6 CONCLUSION

  • D2E addresses the cost of large-scale physical interaction data by using desktop interactions as a pretraining substrate for embodied AI.The framework combines the OWA Toolkit, Generalist-IDM, and VAPT.Source: §6 · p. 10
  • Using 1.3K+ hours of human and pseudo-labeled data, D2E reports 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation.The conclusion describes these results as evidence that digital sensorimotor patterns can improve embodied AI benchmarks.Source: §6 · p. 10
  • The authors publicly release the tools, datasets, and models to support further investigation of desktop-to-embodied transfer.They frame desktop data as a practical, scalable resource that does not rely on prohibitively expensive physical data collection.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

This appendix documents D2E’s reproducible data resources and the recording and data-pipeline choices that support scalable desktop interaction capture. It describes storage efficiency, recorder capabilities, and optimizations for training-time media access.

REPRODUCIBILITY STATEMENT

  • The complete code, datasets, model weights, training configurations, preprocessing pipelines, and evaluation scripts are publicly released.Compute-constrained researchers can also access smaller dataset subsets and checkpoints for partial reproduction and ablations.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 FORMAT COMPARISON

  • Prior desktop formats trade off recording efficiency, typed-stream indexing, crash safety, or storage efficiency; OWAMcap addresses the lack of a desktop-specific standard.The comparison contrasts monolithic image-frame tables and VPT’s MP4–JSONL pairs with the indexing and compression rationale for OWAMcap.Source: §A.1 · p. 16

A.2 COMPRESSION EFFICIENCY

  • OWAMcap reduces CS:GO storage from 689 GiB to 20 GiB (34.45×) and VPT storage from 1.06 TiB to 7.12 GiB (152×).The CS:GO reduction comes from video encoding instead of raw image buffers; the VPT reduction reflects MCAP’s efficiency relative to JSONL.Source: §A.2 · p. 16

A.3 VIDEO COMPRESSION PERFORMANCE

  • Video encoding provides higher compression than other formats while maintaining visual quality, and ocap stores media in H.265 by default.The comparison covers a 12-second, 60 Hz Minecraft desktop capture at 1920×1080; H.265 uses hardware acceleration.Source: §A.3 · p. 16, Table 10 · p. 16

A.4 OCAP ARCHITECTURE

  • ocap combines screen, audio, keyboard, and mouse capture in OWAMcap, using Windows APIs and GStreamer for synchronized, crash-safe recording.The implementation uses DXGI for hardware-accelerated screen capture, WASAPI for low-latency audio, and H.265/HEVC encoding.Source: §A.4 · p. 17, Figure 5 · p. 17

A.5 SCREEN CAPTURE PERFORMANCE BENCHMARKS

  • 6× faster performance than common alternatives is reported for ocap’s screen capture benchmark.The benchmark caption identifies an Intel i5-11400 with GTX 1650 as the test hardware.Source: Table 11 · p. 17, §A.5 · p. 17

A.6 COMPARISON WITH EXISTING RECORDERS

  • ocap is the only compared framework reported to combine synchronized multimodal recording, crash safety, and efficient compression.The comparison is shown in Figure 6.Source: §A.6 · p. 17, Figure 6 · p. 18

A.7 DATA PIPELINE OPTIMIZATIONS

  • The media pipeline improves random access by fixing x264 keyframes at 30-frame intervals and disabling B-frames, then batch-decoding all sample frames in one operation.FSLDataset packing preserves episode coherence by concatenating chronological events and padding sequences to a uniform length.Source: §A.7 · pp. 18–19

A.8 BENCHMARK CONFIGURATION

The appendix specifies benchmark setups and dataset preparation procedures, from media and model throughput measurements to collection, calibration, conversion, and temporal preprocessing.

A.8 BENCHMARK CONFIGURATION

  • The media benchmark uses 64 five-minute Minecraft episodes at 640×360 and 20 Hz, measuring image throughput and average disk bytes read per image.The training benchmark uses InternVL3-1B on one H100 and measures iterations per second across dataloader-worker counts; media configurations are run three times.Source: §A.8 · p. 19

B.1 COLLECTION AND QUALITY ASSURANCE

  • Collection combines automated privacy detection and manual review with automated checks for alignment or corruption and human review of behavior realism and fidelity.The resulting recordings include navigation, application switching, text input, menu interactions, and multi-step tasks.Source: §B.1 · p. 19

B.2 ANNOTATOR CALIBRATION AND PROTOCOLS

  • Contributors calibrated refresh rate, resolution, cursor fidelity, and input mapping before recording, then followed standardized prompts for game scenarios.Sessions were captured at FHD or QHD 60 Hz with synchronized mouse and keyboard traces; calibration was repeated after hardware changes.Source: §B.2 · pp. 19–20, Table 12 · p. 20

B.3 CONVERTED DATA

B.4 PREPROCESSED DATASET

  • Preprocessing shifts action labels by τ = 100 ms while leaving observations unchanged, and filters extended inactive segments.The same temporal offset is used to preprocess both generalist and specialist IDM training data.Source: §B.4 · p. 21

B.5 PSEUDO-LABELED DATASET

This appendix describes the pseudo-label video collection pipeline and the event representation used to model desktop interactions. It also details Generalist-IDM inference, training, and generation procedures.

B.5 PSEUDO-LABELED DATASET

  • The pipeline curates over 1,000 hours of gameplay footage for pseudo-labeling from targeted searches and bulk downloads, restricted to 480p.Searches use the query template “GAME_NAME no commentary”; effective pseudo-labeled hours by game are summarized in Table 14.Source: §B.5 · p. 21, Table 14 · p. 21

C EVENT TOKENIZATION DETAILS

  • Desktop logs become chronological streams of discrete observation and action events, allowing a decoder-only transformer to model asynchronous modalities with fine-grained temporal alignment.Event tokens distinguish screen updates from keyboard and mouse inputs; a common structure delimits events and carries timestamps, types, and event-specific details.Source: C EVENT TOKENIZATION DETAILS · p. 22, §C.2 · p. 23

C.1 EVENT TOKEN

  • The event vocabulary includes modality and boundary tokens, virtual-key and press/release tokens, and numeric encodings for mouse deltas, scroll values, and timestamps.Timestamp bases cover a 10-second window at 10 ms resolution; mouse movement encoding supports signed values within ±1999 pixels.Source: §C.1 · p. 22

C.3 SCREEN EVENTS; C.4 KEYBOARD EVENTS

  • Screen events pair a timestamp with image tokens, while keyboard events pair a timestamp with a virtual key code and press/release action.The examples encode screen images using 256 visual tokens and identify VK_32 as the spacebar.Source: §C.3 · p. 23, §C.4 · p. 24

C.5 MOUSE EVENTS

  • Mouse events encode movement direction and magnitude, button flags, and optional scroll data; the example decodes movement of +2 and −19 pixels at timestamp 245.Scroll data is included only when button flags indicate scroll-wheel activity.Source: §C.5 · p. 24

D MODEL ARCHITECTURE DETAILS

  • Generalist-IDM uses InternVL3-1B, with Intern-ViT as the vision encoder and Qwen2.5 as the language backbone; streaming inference predicts action events until a temporal stopping condition or invalid event.The inference procedure trims its context to the maximum length and returns the predicted action events.Source: D MODEL ARCHITECTURE DETAILS · pp. 24–25

E TRAINING DETAILS

  • Generalist-IDM training uses 8 H100 GPUs for about 24 hours, and the described training schedules include separate human-data and pseudo-labeled-data stages for VAPT.The appendix specifies five epochs for Generalist-IDM and one pseudo-labeled-data epoch followed by three human-data epochs for VAPT with pseudo data.Source: E TRAINING DETAILS · p. 25

F.1 GENERATION METHODS

  • Autoregressive generation resamples and tokenizes synchronized streams, maintains a sliding context window, and decodes generated sequences back into MCAP events.For YouTube pseudo-labeling, the pipeline generates two-minute segments while excluding the first minute and last two minutes of videos; evaluation does not use teacher forcing.Source: §F.1 · pp. 25–26

F.2 EVALUATION METRICS

The evaluation defines mouse and keyboard metrics over non-overlapping 50 ms bins, then tests how temporal offset affects Generalist-IDM predictions. The results support using at least 100 ms of future context, with 100 ms adopted as the default.

F.2 EVALUATION METRICS

  • The evaluation aggregates events in non-overlapping 50 ms bins; Pearson correlation measures mouse-displacement correspondence, while scale ratio measures magnitude mismatch and is normalized to be at least 1.Keyboard accuracy averages binary correctness over key-bin pairs with an event in either source or prediction; mouse button and scroll accuracy use per-bin matching.Source: §F.2 · p. 26

G.1 ABLATION STUDY ON TEMPORAL OFFSETS

  • τ = 0 causes Pearson correlations to collapse near zero, action-scale errors to grow by more than an order of magnitude, and keypress accuracy to drop sharply.The ablation evaluates multiple offsets on six in-distribution games.Source: §G.1 · p. 27
  • Performance stabilizes at τ ≥100 ms, with only minor variation from 100 to 200 ms; τ = 100 ms is adopted as the default.A 50 ms offset improves stability but remains suboptimal, particularly for keypress prediction; no offset from 100 to 200 ms consistently dominates.Source: §G.1 · p. 27

G.2 TRAINING LOSS CURVES

VAPT initialization improves fine-tuning optimization on manipulation and navigation benchmarks: its loss falls smoothly from the start and remains below the baseline.

G.2 TRAINING LOSS CURVES

  • The comparison evaluates the baseline InternVL3-1B without desktop pretraining against VAPT w/o pseudo on LIBERO and CANVAS.Curves are smoothed with an exponential moving average using α = 0.10.Source: §G.2 · p. 27
  • VAPT models show smooth loss reduction from the start, while the baseline plateaus at high loss for about 1,000 steps.The plateau is observed in LIBERO-Spatial and other benchmarks.Source: §G.2 · p. 27
  • VAPT maintains lower loss than the baseline throughout training.The authors suggest this indicates better-aligned representations for embodied control.Source: §G.2 · p. 27

H.1 ROBOT MANIPULATION

This appendix describes manipulation and navigation evaluation, ethics practices, and a scope boundary: real-world validation covers only one task setting.

H.1 ROBOT MANIPULATION

  • LIBERO success rates average 500 episodes per suite; Meta-World covers four difficulty levels and evaluates each task over 10 episodes.The difficulty levels are Easy, Medium, Hard, and Very Hard.Source: §H.1 · pp. 28–29
  • VAPT models show immediate convergence on all four LIBERO suites, unlike the baseline’s initial plateau.Figure 7 reports the training loss curves for Spatial, Goal, Long, and Object.Source: Figure 7 · p. 29
  • The SO101 real-world evaluation uses 208 demonstrations and tests each model in 30 rollouts on a pick-and-place task.The task is to pick a blue cube and place it in a white box.Source: §H.1 · pp. 29–30

H.2 ROBOT NAVIGATION

  • CANVAS navigation compares models trained on COMMAND, with evaluations repeated three times per test dataset using randomized initial orientations.Models are fully unfrozen during training.Source: §H.2 · p. 30

I ETHICS STATEMENT

  • The ethics statement reports informed consent and data review for personal information, permissively licensed public videos, and planned release of code, tools, and weights.The authors acknowledge potential dual-use risks.Source: I ETHICS STATEMENT · p. 30

J LIMITATIONS

  • Real-world validation is limited to one pick-and-place task setting, leaving broader robot-task validation for future work.The authors also note that gaming-focused data may not cover the full range of desktop activities relevant to general-purpose robotics.Source: J LIMITATIONS · p. 30
Loading 2510.05684v3…