PaperlayerView exact revision on arXiv

Source-linked AI summary

HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy

Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

arXiv:2510.00695v3cs.ROcs.CV

TL;DR

Current-observation-only VLAs may miss the history needed for robotic manipulation. HAMLET adds moment tokens and lightweight memory to pretrained VLAs, improving performance across real-world and simulation tasks.

  • Problem

    VLAs typically predict actions from the current observation alone, although manipulation can depend on preceding context such as whether an object has already been grasped.

    Source: §1 · p. 1

  • Method

    HAMLET adapts pretrained VLAs with time-contrastively initialized moment tokens and lightweight memory that integrates tokens across timesteps for action prediction.

    Source: §1 · p. 2

  • Results

    HAMLET improves performance across real-world and simulation VLAs, including a 47.2% gain over the naively fine-tuned VLA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework provides a plug-in way to add history-awareness without additional large-scale pretraining, with reported benefits across diverse VLA policies.

    Source: §1 · pp. 2–3

    HAMLET adds training cost for time-contrastive initialization and may not directly extend to auto-regressive VLAs.

    Source: C DISCUSSION · p. 19

Abstract

from arXiv · show

Inherently, robotic manipulation tasks are history-dependent: leveraging past context could be beneficial. However, most existing Vision-Language-Action models (VLAs) have been designed without considering this aspect, i.e., they rely solely on the current observation, ignoring preceding context. In this paper, we propose HAMLET, a scalable framework to adapt VLAs to attend to the historical context during action prediction. Specifically, we introduce moment tokens that compactly encode perceptual information at each timestep. Their representations are initialized with time-contrastive learning, allowing them to better capture temporally distinctive aspects. Next, we employ a lightweight memory module that integrates the moment tokens across past timesteps into memory features, which are then leveraged for action prediction. Through empirical evaluation, we show that HAMLET successfully transforms a state-of-the-art VLA into a history-aware policy, especially demonstrating significant improvements on long-horizon tasks that require historical context. In particular, on top of GR00T N1.5, HAMLET achieves an average success rate of 76.4% on history-dependent real-world tasks, surpassing the baseline performance by 47.2%. Furthermore, HAMLET pushes prior art performance from 64.1% to 66.4% on RoboCasa Kitchen (100-demo setup) and from 95.6% to 97.7% on LIBERO, highlighting its effectiveness even under generic robot-manipulation benchmarks.

1 INTRODUCTION

Current-frame-only VLAs can miss history needed for manipulation, while naively adding past frames raises computational costs. HAMLET adds learned moment tokens and lightweight memory, improving results across real-world and simulation tasks.

1 INTRODUCTION

  • Current-frame-only VLAs can miss essential context, such as whether an occluded object has already been grasped, even though that history determines the next action.Robotic manipulation tasks are described as intrinsically history-dependent.Source: §1 · p. 1
  • Appending four past frames slows the VLA forward pass by about 35% and raises peak memory use by about 3.6×, motivating history-aware adaptation without pretraining from scratch.The larger memory footprint also restricts feasible batch sizes.Source: §1 · p. 1
  • HAMLET combines timestep-specific moment tokens initialized with time-contrastive learning and lightweight memory that aggregates tokens into history-aware action features.The tokens emphasize task-relevant dynamics while suppressing redundant information, and the framework is designed to integrate with diverse VLAs without additional large-scale pretraining.Source: §1 · pp. 2–3
  • HAMLET improves performance by 47.2% over the naively fine-tuned VLA and raises RoboCasa Kitchen success from 64.1% to 66.4% and CogACT SimplerEnv-Bridge success from 52.1% to 63.5%.The results support improvements across real-world tasks and different VLA backbones.Source: §1 · p. 2

2 RELATED WORKS

Prior work develops generalist VLAs and memory mechanisms, but memory design for large pre-trained VLAs remains under-explored. HAMLET targets this gap by adding learnable tokens and lightweight memory without retraining the backbone.

2 RELATED WORKS

  • Generalist VLAs transfer pretrained VLM representations to action prediction, but typically generate actions from the current observation alone.Recent approaches condition diffusion or flow-matching action experts on VLM representations.Source: §2 · p. 3
  • Earlier robot learning often relies on task-specific datasets covering a narrow distribution of environments, objects, and instructions, while generalist policies use internet-scale VLM priors.This context motivates adapting pretrained VLMs for low-level robot control.Source: §2 · p. 3
  • Robotics memory methods include recurrent and Transformer policies, but many use less generalizable architectures or require training from scratch, leaving memory for large pretrained VLAs under-explored.HAMLET instead augments pretrained VLAs with learnable tokens and lightweight memory without retraining.Source: §2 · p. 3

3 METHOD

HAMLET augments a VLA’s current-timestep representation with compact representations of past timesteps, then aggregates them into a temporally informed condition for action prediction. Its method combines moment-token compression with a memory module that consolidates those tokens across time.

3 METHOD

  • HAMLET compresses each timestep before retaining history because raw observations impose latency and memory costs and may include redundant or static content.The method targets informative past representations rather than storing every raw observation.Source: §3.1 · p. 4, §3 · p. 4

3.1 CONTEXT COMPRESSION VIA MOMENT TOKENS

  • At each timestep, learnable moment tokens are appended to the VLM input and their resulting representations serve as compact, context-aware summaries of the scene.The tokens attend to the current visual observation and task instruction before their representations are stored for aggregation.Source: §3.1 · p. 4
  • Time-contrastive initialization aligns a moment-token representation with an augmented view of the same timestep while distinguishing it from other timesteps.This encourages timestep-specific cues while suppressing static components; the VLM is frozen during this initialization stage.Source: §3.1 · pp. 4–5

3.2 MEMORY CONSOLIDATION VIA MEMORY MODULE

  • A lightweight Transformer uses causal self-attention to selectively integrate moment tokens across timesteps, rather than relying on simple concatenation.The memory module attends to informative past timesteps while ignoring less relevant ones.Source: §3.2 · p. 5
  • The memory module’s history-augmented moment-token feature is concatenated with the VLM representation and sent to the action expert to predict the next k actions.The history matrix stacks the most recent T moment tokens, with L = T · nm total tokens.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments test HAMLET on history-dependent real-world tasks, generic simulation benchmarks, another VLA backbone, and efficiency and transfer. HAMLET improves results across these settings, with analyses examining what its memory uses and which design choices matter.

4.1 EXPERIMENTAL SETUPS

  • Setups span three handcrafted tabletop tasks and RoboCasa Kitchen, LIBERO, and SimplerEnv-Bridge; HAMLET follows each backbone’s fine-tuning setup with four moment tokens, four history steps, and a two-layer Transformer.The primary backbone is GR00T N1.5, with CogACT used to assess generalization to another VLA.Source: §4.1 · pp. 5–7

4.2 MAIN RESULTS

  • HAMLET improves GR00T N1.5’s real-world performance across tasks, with an average improvement of 47.2% over the base model.The evaluation averages success rates over 24 trials per task; Table 1 reports partial and full completion rates.Source: Table 1 · p. 6, §4.2 · p. 7
  • On LIBERO, HAMLET raises the prior best score from 95.6% to 97.6%, while multi-frame inputs reduce baseline performance on RoboCasa and LIBERO.The reported multi-frame declines are 3.3% on RoboCasa (100 demos) and 8.8% on LIBERO.Source: Table 2 · p. 7, §4.2 · p. 7
  • On SimplerEnv-WidowX, HAMLET raises CogACT’s average full success rate from 52.1% to 63.5%.HAMLET also improves partial success rates across all four tasks; the multi-frame baseline does not consistently improve success.Source: Table 3 · p. 7, §4.2 · p. 8

4.3 MORE ANALYSIS

  • Moment tokens attend to changing, task-relevant regions, while memory attention varies by episode context and can emphasize a timestep needed for the next action.In Cover-and-Stack, memory attends more to the earlier timestep when the blue cube was visible.Source: Figure 4 · p. 8, §4.3 · p. 8
  • At history length 8, multi-frame inference requires roughly 2.4× the latency and 7× the memory of vanilla inference, while HAMLET’s reported overhead is 1.07×.The efficiency measurements average per-timestep latency and peak memory on RoboCasa.Source: §4.3 · p. 8, Table 4 · p. 9
  • Ablations identify the memory module’s removal as the largest performance drop; TCL initialization consistently helps, and performance peaks at token length 8 before declining.Among memory architectures, the Transformer achieves the highest average success rate; moment concatenation yields almost no gains over baseline.Source: §4.3 · p. 9, Table 5 · p. 9
  • A memory module trained on LIBERO and frozen for RoboCasa achieves success comparable to in-distribution training, without additional adaptation training.Table 6 describes the LIBERO-pretrained module as providing gains on RoboCasa.Source: Table 6 · p. 9, §4.3 · p. 9

5 CONCLUSION

HAMLET adapts pretrained VLAs to use historical information through a lightweight memory module integrating learnable moment tokens. The paper reports improvements on long-horizon real-world tasks and standard simulation benchmarks while maintaining computational efficiency.

5 CONCLUSION

  • HAMLET equips pretrained VLAs to use historical information through a lightweight memory module integrating learnable moment tokens.The framework is presented as an alternative to costly retraining from scratch.Source: §5 · p. 10
  • HAMLET achieves significant improvements on long-horizon real-world tasks and standard simulation benchmarks while maintaining computational efficiency.Source: §5 · p. 10
  • The authors hope history-aware off-the-shelf VLA policies will help tackle more complex robotic manipulation tasks reliably and effectively in practice.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

The experiments evaluate five diffusion-based VLAs, using checkpoints obtained from their official repositories.

A.1 MODEL DETAILS

  • The evaluated diffusion-based VLAs are GR00T N1, GR00T N1.5, π0, π0-FAST, and CogACT.Source: §A.1 · p. 14
  • All evaluated model checkpoints come from their official repositories.Source: §A.1 · p. 14

A.2 DATASETS

The evaluation spans history-dependent real-world tabletop tasks and simulation benchmarks. The real-world tasks use 50 training demonstrations each and a 700-timestep evaluation limit; simulations cover RoboCasa, LIBERO, and SimplerEnv-Bridge.

A.2 DATASETS

  • The real-world evaluation uses three handcrafted tabletop tasks on a Franka Research 3 arm with table-mounted and wrist cameras.The tasks are Pick-and-Place Twice, Cover-and-Stack, and Swap Cubes; each has 50 training demonstrations, and trials are capped at 700 timesteps.Source: §A.2 · p. 14, Figure 6 · p. 14, Figure 7 · p. 15
  • Real-world episode frame-count statistics are reported as means, maxima, and standard deviations to inform the evaluation timestep limit.Source: §A.2 · p. 14, Table 7 · p. 15
  • Simulation evaluation covers RoboCasa, LIBERO, and SimplerEnv-Bridge, with LIBERO run using GR00T N1.5 and SimplerEnv-Bridge using CogACT.RoboCasa contains 24 kitchen tasks across 120 scenes, while LIBERO contains 40 tasks in four suites; each RoboCasa task is evaluated over 50 episodes.Source: Figure 5 · p. 14, §A.2 · p. 15

A.3 IMPLEMENTATION

HAMLET uses four moment tokens and a two-layer Transformer memory module, with time-contrastive initialization based on augmented same-frame positives and temporally separated negatives. A naïve multi-frame baseline instead concatenates historical observations or conditions the action expert with a cognition token.

A.3 IMPLEMENTATION

  • The implementation uses four moment tokens and a two-layer Transformer as the memory module.Source: §A.3 · p. 15
  • Time-contrastive initialization uses stochastically augmented views of the same frame as positives and frames more than 16 timesteps apart as negatives.Positive augmentations include photometric distortions, Gaussian blur or noise, and small-patch occlusions; the contrastive temperature is 0.07.Source: §A.3 · pp. 15–16
  • The naïve multi-frame baseline concatenates past views for GR00T N1.5, while CogACT uses its cognition token to condition the action expert.Source: §A.3 · p. 16

A.4 TRAINING DETAILS

GR00T N1.5 and CogACT are fine-tuned using official recipes, with HAMLET components trained under model-specific schedules. The reported compute costs distinguish memory-module fine-tuning from standard training and TCL initialization.

A.4 TRAINING DETAILS

  • The experiments fine-tune GR00T N1.5 and CogACT according to their official training recipes.Table 8 summarizes modification costs by model parameters, inference time, and training time; inference runs on one A100, with training on four H200s except GR00T N1.5 on four A100s.Source: §A.4 · p. 16, Table 8 · p. 16
  • Both backbones use a default history length of 4; GR00T N1.5 freezes the VLM and moment tokens during fine-tuning, while CogACT keeps them trainable.GR00T N1.5 uses 60k fine-tuning steps; CogACT uses 20k.Source: §A.4 · p. 16

A.5 COMPUTATIONAL RESOURCES •

  • GR00T N1.5 fine-tuning with the memory module takes ~16 hours versus ~14 hours standard; CogACT takes ~9 versus ~4 hours.TCL initialization adds ~5 hours for GR00T N1.5 and requires ~9 hours for CogACT.Source: §A.5 · p. 16

B.1 QUANTITATIVE RESULTS

The quantitative analyses report gains from HAMLET on autoregressive VLA benchmarks, evaluate TCL initialization, and test a longer-horizon task. Additional tables provide full RoboCasa results and comparisons of TCL pair-construction methods.

B.1 QUANTITATIVE RESULTS

  • The appendix supplies complete RoboCasa Kitchen per-task success rates for HAMLET and GR00T N1.5 across dataset sizes.Table 9 is identified as the full results table for different dataset sizes.Source: §B.1 · p. 16, Table 9 · p. 17
  • TCL achieves the highest average success rate among tested initializations and improves results across datasets, with gains up to +2.7% in RoboCasa 30-demo training.The TCL comparisons cover RoboCasa Kitchen at 30, 100, and 300 demonstrations and LIBERO suites.Source: §B.1 · p. 17, Table 12 · p. 18, Table 11 · p. 18
  • On this long-horizon task, HAMLET significantly improves over the baseline despite needing to memorize more than 14 inference steps while explicitly retaining only 4 steps by default, exceeding the original window size.Source: §B.1 · p. 18
  • HAMLET gains 9.0% on LIBERO-Long with OpenVLA and 5.8% on RoboCasa Pick-and-Place tasks with π0-FAST.Table 14 covers OpenVLA on LIBERO and π0-FAST on RoboCasa Kitchen (100 demos).Source: §B.1 · p. 18

B.2 QUALITATIVE RESULTS

Qualitative analyses show TCL-initialized moment tokens focusing on task-relevant regions and the memory module attending to past timesteps when needed. The paper also discusses limitations, future directions, and the stated use of language models in manuscript preparation.

B.2 QUALITATIVE RESULTS

  • TCL-initialized moment tokens attend more to task-relevant regions, including the object to be grasped, than randomly initialized tokens.The attention comparison is shown on RoboCasa input frames.Source: §B.2 · pp. 18–19, Figure 8 · p. 20
  • On Swap Cubes, the memory module attends to the timestep when the blue cube was placed before selecting which cube to pick next, but less during the initial move.Figure 9 contrasts a history-critical decision with a case where recalling history is less critical.Source: §B.2 · p. 19, Figure 9 · p. 20
  • Rollouts show HAMLET better handling multi-step dependencies and occlusions than GR00T N1.5, while the naïve multi-frame baseline often fails to recover from failure states.The paper suggests simply appending multi-frame inputs might degrade generalizability; Figures 10–13 provide rollout examples.Source: §B.2 · p. 19, Figure 10 · p. 21, Figure 11 · p. 21

C DISCUSSION

  • HAMLET incurs additional training cost for TCL initialization, and its extension to autoregressive VLAs may not be direct.The authors report applicability to recent diffusion-based VLAs.Source: C DISCUSSION · p. 19
  • The authors identify scaling to larger manipulation datasets and exploring deeper or hybrid memory architectures as future directions.They report that components such as the memory module can transfer to unseen datasets.Source: C DISCUSSION · p. 19

D THE USE OF LARGE LANGUAGE MODELS

Loading 2510.00695v3…