PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models

Qi Lyu, Jiahua Dong, Hao Shen, Xudong Wang, Hongyuan Yu, Baichen Liu, Henghui Ding, Zhi Han, Nicu Sebe, Ivan Laptev, Fahad Shahbaz Khan, Salman Khan

arXiv:2609.40219v1cs.CVcs.AI

TL;DR

Existing World Action Models do not explicitly reuse action experience across manipulation tasks, despite the potential for cross-task relationships to guide skill learning. The paper introduces a visually conditioned Action Experience Dictionary and motion-aware transition loss; its experiments report improved manipulation performance across simulation benchmarks and real-world cross-embodiment evaluations.

  • Problem

    World Action Models do not explicitly reuse historical action experience across manipulation tasks, while task relationships could help guide target-task skill learning.

    Source: §1 · pp. 1–2

  • Method

    The Action Experience Dictionary encodes historical trajectories as shared action embeddings, conditions retrieved embeddings on visual context, and uses motion-aware transition supervision.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    92.8% on RoboTwin 2.0 and 98.8% on LIBERO are the highest reported average success rates; experiments also report improved performance in real-world cross-embodiment evaluations.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The experiments support using shared, visually conditioned action experience and motion-aware supervision for manipulation across simulation benchmarks and real-world embodiments.

    Source: §5 · p. 10

    A finite history window limits access to longer-term dependencies.

    Source: B LIMITATION · p. 14

Abstract

from arXiv · show

World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experience across manipulation tasks. Furthermore, existing WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction, as redundant background elements interfere with the extraction of key visual information. To address these challenges, we develop a novel Action Experience Dictionary (AED) that encodes historical physical action trajectories into shared action embeddings to support skill reuse and model cross-task relationships. Specifically, we first aggregate historical actions to align with visual observations and retrieve action embeddings from the AED using a pretrained action tokenizer. Subsequently, we visually condition the pooled embeddings through cross-attention and prepend them to noisy action tokens, providing interaction context and action intent for prediction. To model action-related motion and reduce reliance on irrelevant background cues, we introduce a motion-aware transition loss that supervises visual feature change prediction over random temporal intervals. Experiments on simulation benchmarks and in real-world cross-embodiment settings verify the effectiveness of our AED. The anonymous project website is available at \href{https://github.com/JiahuaDong/AED}{AED}.

1 INTRODUCTION

World Action Models couple action generation with future visual prediction, but cross-task reuse of action experience remains underexplored. The paper introduces a shared Action Experience Dictionary (AED), visually conditioned action embeddings, and motion-aware transition supervision to support manipulation skill learning.

1 INTRODUCTION

  • WAMs couple action generation with future visual prediction, but cross-task semantic relationships that could guide manipulation skill learning remain underexplored.Historical actions alone are insufficient because similar motions can serve different purposes depending on manipulated objects and their spatial relationships.Source: §1 · pp. 1–2
  • Placing a bowl on a plate involves substantial reuse of both transporting and dipping action patterns represented in the AED.The figure identifies f ⋆[0]–f ⋆[2] as transporting and f ⋆[3]–f ⋆[7] as dipping patterns.Source: Figure 1 · p. 2
  • A learnable Action Experience Dictionary encodes historical physical trajectories as shared action embeddings for WAMs.Past actions are aggregated over intervals aligned with visual observations, and a pretrained action tokenizer retrieves task-relevant embeddings.Source: §1 · p. 2
  • Retrieved action embeddings are pooled, visually conditioned through cross-attention, and prepended to noisy action tokens to provide interaction context and action intent.This design lets the action expert use inter-task relationships when predicting subsequent action chunks.Source: §1 · p. 2
  • A motion-aware transition loss predicts visual feature changes over random temporal intervals, encouraging action-related motion representations and reducing reliance on irrelevant background cues.The loss uses starting-interval visual features and corresponding AED-conditioned hidden states.Source: §1 · p. 2

2 RELATED WORK

Prior work explores structured action representations, transferable latent actions, and visual dynamics for robot control. The paper distinguishes its approach by explicitly reusing historical action experience across tasks and conditioning retrieved embeddings on visual context.

2 RELATED WORK

  • Prior VLA work includes frequency-space action tokenization, task- or motion-centric latent actions, language-compatible action representations, and decoupled visual-dynamics and inverse-action learning.The cited examples are FAST, UniVLA, ViPRA, VLM2VLA, and DeFI, respectively.Source: §2 · pp. 2–3
  • WAMs use visual prediction to capture physical and temporal structure, with approaches spanning text-conditioned video planning, video generative priors, and joint video-action modeling.More recent examples integrate understanding, video, and action experts or use video co-training for control.Source: §2 · p. 3
  • Unlike prior WAMs focused on task-specific behaviors or transferable latent action spaces, this method encodes historical trajectories in shared AED embeddings and visually conditions retrieved actions.Its overview also pairs this representation with motion-aware transition loss to encode action-relevant motion.Source: §2 · p. 3, Figure 2 · p. 3

3 METHODOLOGY

The method jointly trains video and action prediction, then adds a shared Action Experience Dictionary (AED) to reuse historical action patterns across manipulation tasks. Visual conditioning and a motion-aware transition loss guide action prediction toward task-relevant context and motion.

3.1 PRELIMINARIES

  • The preliminary setup uses flow matching to jointly predict future video latents and an action chunk from noisy targets.The targets are future video and an action chunk, with the policy also using the robot’s proprioceptive state.Source: §3.1 · pp. 3–4

3.2 ACTION EXPERIENCE DICTIONARY (AED)

  • The AED is shared across manipulation tasks and stores action-pattern embeddings retrieved from historical trajectories aggregated to align with visual observations.The aggregation sums arm commands over each observation interval while retaining the final gripper command.Source: §3.2 · p. 4
  • Cross-attention conditions retrieved action embeddings on compressed historical visual features, and the resulting embeddings are prepended to noisy action tokens for action-chunk prediction.The action embeddings query visual tokens, which provide keys and values.Source: §3.2 · p. 5
  • The motion-aware transition loss predicts visual feature changes over randomly sampled future intervals using AED-conditioned action hidden states, encouraging action-relevant motion features and reducing reliance on background cues.The theorem states that the sampled loss is an unbiased estimator of its sampling-weighted risk, which bounds temporal composition error.Source: §3.2 · p. 6

3.3 TRAINING AND INFERENCE

  • The overall training objective combines flow matching for the video and action branches with the motion-aware transition loss, weighted by λm = 0.01.At inference, Gaussian-noise latents are denoised for K Euler steps to produce an action chunk; neither video decoding nor the transition predictor is required.Source: §3.3 · p. 6

4 EXPERIMENTS

Across simulation benchmarks and real-world OOD settings, the experiments report strong manipulation success, while ablations and analyses examine component contributions, efficiency, cross-task transfer, and motion-aware transitions.

4.1 IMPLEMENTATION DETAILS

  • Training uses 8 A100 GPUs for 10 LIBERO epochs and 32 H100 GPUs for 5 RoboTwin 2.0 epochs; real-world tests use Spirit AI MOZ1 and ROKAE AR5.The method reports success rates on LIBERO, RoboTwin 2.0, and LIBERO Plus.Source: §4.1 · p. 7

4.2 MAIN COMPARISON RESULTS

  • Across Spirit AI MOZ1 and ROKAE AR5, AED achieves higher success than existing methods under clutter, low light, and unseen-object conditions.The evaluations also include the representative task “Put the mushroom and orange in the basket.”Source: §4.2 · pp. 7–8
  • 92.8% on RoboTwin 2.0 and 98.8% on LIBERO are the highest reported average success rates.On LIBERO, the method improves over Fast-WAM by 1.2 percentage points on average; on RoboTwin 2.0, it leads the strongest baseline by 0.6 points on average.Source: §4.2 · p. 8, Table 1 · p. 8
  • 86.2% overall on LIBERO-Plus beats Fast-WAM and π0.5 without embodied policy pretraining; AED leads Camera, Light, and Noise, and wins six of seven categories.Robot is the sole category where it does not outperform Fast-WAM.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY AND EFFICIENCY ANALYSIS

  • 97.4% on LIBERO-10 is the full-model success rate; removing MT loss yields 96.8%, while removing AED lowers success by 2.6% versus the full model.Removing visually conditioned action embeddings further degrades performance.Source: §4.3 · pp. 8–9, Table 3 · p. 8
  • 167.3 to 708.5 ms latency as denoising steps rise from 1 to 15; peak GPU memory remains 24.1 GB.These LIBERO-10 measurements disable future-frame decoding and remove the transition predictor during inference.Source: §4.3 · p. 9

4.4 ANALYSIS OF ACTION EXPERIENCE DICTIONARY (AED)

  • Training on five target tasks plus five auxiliary source tasks consistently raises success over target-only training, indicating positive transfer of shared skill patterns.The five target tasks are evaluated in both training conditions.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 ANALYSIS OF MOTION-AWARE TRANSITION (MT) LOSS

  • Direct t1-to-t3 feature prediction and composed t1-to-t2-to-t3 transitions produce similar visualizations and identify action-relevant objects.With MT-loss guidance, the policy learns to suppress background interference.Source: Figure 6 · p. 9, §4.5 · p. 10

5 CONCLUSION

The paper presents AED for learning manipulation skills from historical trajectories and reports improved performance in simulation and real-world cross-embodiment evaluations.

5 CONCLUSION

  • AED encodes historical trajectories into shared action embeddings conditioned on visual context, while a motion-aware transition loss predicts visual feature changes over random temporal intervals.Experiments across simulation benchmarks and real-world cross-embodiment evaluations report improved manipulation performance over baseline approaches.Source: §5 · p. 10

AI USE STATEMENT

The authors disclose using generative AI tools for manuscript-preparation tasks and state that they reviewed suggestions and independently checked technical and reported content.

AI USE STATEMENT

  • Generative AI tools assisted with literature and experimental-record organization, table formatting, language revision, and discussion of possible refinements.The authors state that AI-assisted suggestions were carefully reviewed.Source: AI USE STATEMENT · p. 10
  • The authors independently checked equations, implementation details, experimental procedures, citations, reported results, and interpretations.Source: AI USE STATEMENT · p. 10
  • The authors state that they remain responsible for the final methodology, experiments, results, code, and manuscript.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study covers simulated and real-world robotic manipulation, with real-world experiments conducted under bounded workspaces, operating constraints, and human supervision.

ETHICS STATEMENT

  • The work studies robotic manipulation in both simulated environments and real-world settings.Source: ETHICS STATEMENT · p. 10
  • Real-world experiments use bounded workspaces, predefined operating constraints, and appropriate human supervision.Source: ETHICS STATEMENT · p. 10
  • The study does not intentionally collect personal or sensitive information; visual observations are limited to the robot and its surrounding workspace.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The supplementary materials document experimental settings and procedures intended to support reproduction of the reported experiments.

REPRODUCIBILITY STATEMENT

  • Supplementary materials provide model configurations, optimization settings, preprocessing procedures, and inference parameters.They also include per-task benchmark results, data-collection information, robot-platform details, ablations, and qualitative analyses.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The anonymous project page provides code and robotic manipulation demonstrations.The passage identifies the supplementary project page but does not provide its address.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The materials document settings and procedures required to reproduce the reported experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

B LIMITATION

The method’s training cost can be mitigated by caching frozen visual features, while its finite history window limits access to longer-term dependencies. The configuration trains the experts and added modules while keeping the video VAE and vision encoder frozen.

B LIMITATION

  • A finite history window limits access to longer-term dependencies, while loading the frozen visual model may slow training optimization.The authors suggest caching visual features in advance to address the optimization slowdown.Source: B LIMITATION · p. 14

C.1 MODEL AND REPRESENTATION CONFIGURATION

  • Training updates both experts, the AED, visual memory and action–visual fusion modules, and the transition predictor.The video VAE and LingBot-vision encoder remain frozen.Source: §C.1 · p. 14
  • The AED contains 2,048 valid 1,024-dimensional entries and a separate padding entry.FAST+ token indices retrieve dictionary embeddings, which are aggregated with temporal positional encoding.Source: §C.1 · p. 14
  • Historical visual features are compressed into 128 embeddings, then fused with action history into eight prefix tokens for the action expert.The frozen LingBot encoder supplies visual targets for transition supervision.Source: §C.1 · p. 14

C.2 OPTIMIZATION AND TRAINING CONFIGURATION

The appendix specifies training, preprocessing, and deployment settings, then reports LIBERO and RoboTwin results and describes real-world data and platforms.

C.2 OPTIMIZATION AND TRAINING CONFIGURATION

  • LIBERO training uses 10 epochs and RoboTwin training uses 5, with AdamW and a cosine-decay learning-rate schedule.Training uses A100 and H100 GPUs, respectively, and assigns the motion-transition loss a weight of 0.01.Source: §C.2 · p. 14

C.3 OBSERVATION AND ACTION PREPROCESSING

  • Single-arm inputs combine third-person and wrist views; dual-arm inputs combine a head view and one wrist view per arm.Images are resized to 224 × 224; single-arm action histories contain 32 steps grouped into eight four-step intervals before FAST+ tokenization.Source: §C.3 · p. 14

C.4 INFERENCE AND DEPLOYMENT

  • At inference, the policy predicts 32 actions using 10 flow-matching ODE steps, executes the first 10 at 30 Hz, then replans.The pipeline conditions joint video and action generation on observations, proprioceptive state, instructions, and historical action embeddings.Source: §C.4 · pp. 14–15

D.1 LIBERO

  • The 42K checkpoint achieves 98.80% success across 2,000 standard LIBERO trials.Table 6 reports per-task success rates across 40 tasks, with suite averages over 500 trials each.Source: §D.1 · p. 15, Table 6 · p. 15

D.2 ROBOTWIN 2.0

  • The method achieves 93.18% Clean and 92.34% Random success on RoboTwin 2.0, averaging 92.76% across 10,000 trials.Table 7 reports per-task rates for all 50 tasks; the evaluation uses one checkpoint and 100 trials per task in each setting.Source: §D.2 · p. 16, Table 7 · p. 16

E.1 TRAINING DATA OVERVIEW

  • The real-world data overview covers ten Spirit AI MOZ1 manipulation tasks, with data collected using a VR device.Figure 7 presents six high-camera frames per task from episode start to finish.Source: §E.1 · p. 17, Figure 7 · p. 18

E.2 ROBOT PLATFORMS

  • Real-world experiments use Spirit AI MOZ1 and ROKAE AR5-5 0.7 platforms, with a host computer sending predicted actions to the robots.Camera images and natural-language task instructions are processed by the model during deployment.Source: §E.2 · pp. 17–18

F.1 CROSS-TASK SIMILARITY OF ACTION CONTENT EMBEDDINGS

Across manipulation tasks, action-content embeddings show stronger similarity for the same skill, and task windows share AED entries. Additional analyses show compatible transition predictions and stage-dependent visual responses associated with historical action embeddings.

F.1 CROSS-TASK SIMILARITY OF ACTION CONTENT EMBEDDINGS

  • Same-skill task pairs have higher median action-content similarity than different-skill pairs across all six categories, with the largest gaps for grasping (0.173) and releasing (0.163).Both comparison groups are cross-task; the plotted samples are weighted average cosine similarities for task pairs.Source: §F.1 · p. 19, Figure 9 · p. 19
  • The overlapping similarity distributions indicate graded skill relationships, and the analyzed structure is present in pooled action content before positional encoding or visual cross-attention.The paper presents this as support for historical trajectories as reusable action experience across tasks.Source: §F.1 · p. 19
  • Shared AED access and similarity distributions provide complementary views of cross-task action experience; a token ID need not denote an entire high-level skill.The analysis distinguishes repeated dictionary access from the organization of pooled, projected action-content embeddings.Source: §F.1 · p. 20

F.2 SHARED AED ENTRIES ACROSS MANIPULATION TASKS

  • AED entries overlap across task windows with different objects and goals, including ID 300 in all four windows and other entries shared by subsets.The reported frequencies for ID 300 in T1–T4 are 75%, 100%, 25%, and 100%, respectively.Source: §F.2 · p. 20
  • Shared action content is paired with visual conditioning to provide interaction context and action intent for a target task.The authors connect this combination to the fact that similar motions can serve different purposes depending on objects and spatial relationships.Source: §F.2 · p. 20

G.1 TRANSITION PREDICTION

  • Direct and composed transition predictions show similar spatial organization and broadly reproduce ground-truth endpoint feature patterns across four examples.Agreement between routes supports temporal composition consistency: subdividing an interval yields a compatible estimate of the resulting visual state.Source: Figure 10 · p. 20, §G.1 · p. 21

G.2 VISUAL CONDITIONING

  • Visual responses associated with historical action embeddings shift from nearby objects toward gripper–object interactions and the plate as the trajectory progresses.This stage-dependent pattern is consistent with visual conditioning supplying object and spatial context for interpreting reusable motions.Source: §G.2 · p. 21, Figure 11 · p. 21

H.1 EXPERIMENTAL PROTOCOL

The appendix evaluates ablation choices under a fixed LIBERO-10 protocol, covering action aggregation, temporal sampling, visual encoding, memory-query count, and history-prefix design.

H.1 EXPERIMENTAL PROTOCOL

  • The ablation uses seed 0, 32 predicted actions per call, ten executed actions before replanning, and 50 trials per task across ten LIBERO-10 tasks; 97.4% is the shared full-model reference.SR denotes success rate in percent.Source: §H.1 · p. 21

H.2 ACTION AGGREGATION AND TEMPORAL SAMPLING

  • Agg→Tok (Ours) outperforms joint tokenization of four actions before pooling and MLP encoding of aggregated continuous actions.The protocol aggregates each four-action interval before FAST+ tokenization.Source: §H.2 · p. 22
  • The 96.8% temporal-sampling result jointly removes random sampling and the motion-aware transition objective.Compared configurations include Start→Span, Span→Start, a fixed window, and no interval sampling.Source: §H.2 · p. 22

H.3 SAMPLING WINDOW DISTRIBUTION

  • For V=8, P(∆=1)=37.04% while P(∆=7)=2.04%, so Start→Span sampling emphasizes short transitions but retains positive probability for every valid longer interval.Short windows are valid at more start offsets, including late starts with fewer remaining choices.Source: §H.3 · p. 22

H.4 VISUAL ENCODER AND QUERY COUNT

  • The visual-encoder ablation compares LingBot-Vision with DINOv3, while query-count ablations test 32, 64, 128, and 256 learnable queries, with 128 proposed.Source: §H.4 · p. 23

H.5 PREFIX CONTENT AND INJECTION

  • Vision-only uses historical visual features; Action-only uses historical action embeddings without VC, with reported settings of 95.6% when MT is also removed and 94.8% when AED, VC, and MT are removed.Source: §H.5 · p. 23
  • Prefix injection prepends history tokens to the action sequence, whereas Cross-attn conditions the action expert through separate residual cross-attention modules.Source: §H.5 · p. 23

Related papers

Loading 2609.40219v1…