PaperlayerView exact revision on arXiv

Source-linked AI summary

FAST: Efficient Action Tokenization for Vision-Language-Action Models

Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, Sergey Levine

arXiv:2501.09747v1cs.ROcs.LG

TL;DR

Autoregressive VLAs require action tokenization schemes that map continuous robot actions to discrete symbols, but per-dimension, per-timestep binning performs poorly for high-frequency dexterous data. The paper introduces FAST, which compresses action chunks using the discrete cosine transform and byte-pair encoding, and FAST+ as a universal tokenizer. Combined with π0, FAST-based autoregressive VLAs match diffusion-based VLAs across varied tasks while reducing training time by up to 5x.

  • Problem

    Per-dimension, per-timestep binning often performs poorly on high-frequency dexterous robot data because correlated action tokens weaken next-token prediction.

  • Method

    FAST compresses continuous action chunks with the discrete cosine transform and byte-pair encoding, while FAST+ is trained as a universal tokenizer on diverse robot trajectories.

  • Results

    FAST-based autoregressive VLAs match diffusion-based VLA performance across varied tasks while reducing training time by up to 5x.

  • Takeaways & Limitations

    FAST provides a broadly applicable tokenizer for autoregressive VLA training across diverse robot action sequences and control frequencies.

Abstract

from arXiv · show

Autoregressive sequence models, such as Transformer-based vision-language action (VLA) policies, can be tremendously effective for capturing complex and generalizable robotic behaviors. However, such models require us to choose a tokenization of our continuous action signals, which determines how the discrete symbols predicted by the model map to continuous robot actions. We find that current approaches for robot action tokenization, based on simple per-dimension, per-timestep binning schemes, typically perform poorly when learning dexterous skills from high-frequency robot data. To address this challenge, we propose a new compression-based tokenization scheme for robot actions, based on the discrete cosine transform. Our tokenization approach, Frequency-space Action Sequence Tokenization (FAST), enables us to train autoregressive VLAs for highly dexterous and high-frequency tasks where standard discretization methods fail completely. Based on FAST, we release FAST+, a universal robot action tokenizer, trained on 1M real robot action trajectories. It can be used as a black-box tokenizer for a wide range of robot action sequences, with diverse action spaces and control frequencies. Finally, we show that, when combined with the pi0 VLA, our method can scale to training on 10k hours of robot data and match the performance of diffusion VLAs, while reducing training time by up to 5x.

I. INTRODUCTION

FAST addresses the difficulty of tokenizing high-frequency robot actions for autoregressive VLAs by compressing action trajectories in frequency space. FAST+ extends this approach across diverse robots and action spaces, while π0-FAST matches diffusion-VLA performance with substantially lower training cost.

  • Results: π0-FAST matches π0 diffusion-VLA performance on dexterous and long-horizon manipulation tasks while training 5x faster.The result is reported for a generalist robot policy based on FAST.
  • Motivation: High-frequency action chunks can produce highly correlated tokens, allowing trivial copying strategies that undermine autoregressive next-token learning.Prior policies commonly use per-dimension, per-timestep binning, which can generate hundreds of tokens per action chunk and slow inference.
  • FAST: FAST compresses continuous robot action trajectories with the discrete cosine transform before autoregressive token prediction.The method is designed to reduce correlation between consecutive tokens and supports dexterous, high-frequency tasks where standard discretization fails.
  • FAST+: FAST+ is trained on 1M real robot action trajectories spanning diverse embodiments, action spaces, and control frequencies.It is presented as an off-the-shelf tokenizer for a wide range of robot action sequences.
  • FAST: FAST outperforms popular binning tokenization schemes, particularly on high-frequency robot data.The comparison is illustrated for autoregressive Transformers used in dexterous robot control.
  • Related work: The paper positions FAST among action representations that avoid the hand-designed controllers required by semantic action representations and the high-frequency failures of vector quantization.These comparisons motivate FAST as a broadly applicable action-tokenization approach.

IV. CASE STUDY: HOW DOES TOKENIZATION AFFECT VLA TRAINING?

A didactic interpolation study isolates how action tokenization affects autoregressive learning as sampling frequency increases. Naive binning degrades sharply at high frequencies, whereas compression-oriented tokenization is intended to preserve a stronger learning signal.

  • Experimental setup: The case study uses a synthetic cubic-spline dataset to emulate predicting high-frequency action chunks from conditioning information.Target sequences interpolate four randomly generated points and are discretized into 256 bins per element under the naive scheme.
  • Findings: As sampling frequency increases, binning-tokenized models show steeply increasing prediction error and eventually copy the first action.The underlying data distribution remains unchanged across sampling rates, so the observed degradation is attributed to the training setup rather than greater data complexity.
  • Mechanism: Highly redundant high-frequency trajectories reduce the marginal information carried by each action token, weakening autoregressive learning.The case study motivates compressing the action signal into fewer high-information tokens.
  • Approach: The proposed remedy is to compress continuous time series before designing the action-tokenization algorithm and universal tokenizer.The section introduces this pipeline in stages: time-series compression, action tokenization, and universal tokenizer training.

A. Time-Series Compression via Discrete Cosine Transform

FAST compresses continuous action sequences in frequency space using the discrete cosine transform, then quantizes and BPE-encodes the resulting coefficients into action tokens. This compression reduces sequence redundancy while preserving an invertible, efficient tokenization pipeline.

  • Time-Series Compression via Discrete Cosine Transform: DCT represents continuous signals as sums of cosine components, with low frequencies capturing overall shape and high frequencies reflecting sharp jumps.The transform is used as the basis for compressing robot action time series.
  • Time-Series Compression via Discrete Cosine Transform: FAST normalizes each action dimension, applies DCT separately, and scale-and-rounds coefficients to trade off compression rate against lossiness.Quantile normalization maps the 1st and 99th training-data quantiles to [−1, …, 1] and improves robustness to outliers.
  • Time-Series Compression via Discrete Cosine Transform: Quantized DCT matrices become sparse, so FAST flattens them with low-frequency components first and uses BPE to compress the integer sequence.BPE merges frequent coefficient combinations and squashes zero-valued components into dense action tokens.
  • Time-Series Compression via Discrete Cosine Transform: FAST’s operations are invertible and use two relatively insensitive hyperparameters: DCT rounding scale 10 and BPE vocabulary size 1024.The same values are used across the single-dataset tokenization experiments.
  • Time-Series Compression via Discrete Cosine Transform: DCT-compressed tokens maintain constantly low autoregressive prediction error across a wide range of sampling frequencies in the toy time-series experiment.This contrasts with naïve binning, whose error increases sharply as sampling frequency rises.

C. A Universal Robot Action Tokenizer

FAST+ extends FAST into a universal robot action tokenizer by training its BPE vocabulary on diverse cross-embodied action data. The released processor can tokenize new robot action chunks directly or fit a dataset-specific tokenizer.

  • A Universal Robot Action Tokenizer: The only learned component of FAST is the BPE vocabulary, which must be trained for each new dataset when using dataset-specific tokenization.This training typically takes only a few minutes but adds friction to applying FAST.
  • A Universal Robot Action Tokenizer: FAST+ is trained on approximately 1M 1-second action chunks spanning single-arm, bimanual, and mobile robots, varied action spaces, and control frequencies.The mixture includes joint and end-effector control action spaces.
  • A Universal Robot Action Tokenizer: The released FAST+ AutoProcessor applies the pre-trained universal tokenizer to new robot action chunks without fitting a new tokenizer.The processor is packaged for convenient use on arbitrary robot action chunks.
  • A Universal Robot Action Tokenizer: For best compression, FAST recommends quantile-normalizing actions to [−1, …, 1] and tokenizing 1-second chunks.The module also supports fitting a new FAST tokenizer on an action dataset.

VI. EXPERIMENTS

The experiments evaluate FAST and FAST+ with autoregressive VLA backbones across seven environments covering dexterous real-robot tasks, simulation, and zero-shot generalization. The study compares tokenizers, ablates design choices, and tests scaling against diffusion-based VLAs.

  • Experimental Setup: FAST is evaluated with π0 and OpenVLA backbones against alternative action tokenization schemes and key design ablations.The experiments also compare FAST-tokenized π0 models with a state-of-the-art π0 flow-matching VLA.
  • Experimental Setup: The evaluation suite contains 7 tasks: 6 real-robot environments and 1 simulated environment.The tasks target both dexterous manipulation and generalization.
  • Experimental Setup: The suite includes dexterous tasks such as laundry folding and simulation benchmarks such as Libero, which reports average performance across four suites.Libero aggregates Libero-Spatial, Libero-Object, Libero-Goal, and Libero-10.
  • Experimental Setup: Zero-shot DROID evaluation tests a policy trained on the full DROID dataset in an entirely unseen tabletop environment without co-training or fine-tuning.The changed conditions include table setup, background, objects, viewpoint, and table height.
  • Experimental Setup: The generalist evaluation uses grocery bagging, toaster removal, and laundry folding to assess the most powerful VLA.These tasks are reserved for the generalist model evaluation described in Section VI-F.

B. Comparing Action Tokenizers for VLA Training

FAST compresses action sequences more effectively than naïve binning and supports policy training on high-frequency, dexterous tasks where naïve tokenization fails. It also generalizes across datasets and enables zero-shot table-top manipulation with DROID.

  • Compression efficiency: FAST uses fewer tokens per 1-second action chunk than naïve tokenization, especially for high-frequency tasks such as T-shirt folding.The reduction indicates more effective removal of action redundancy.
  • Compression efficiency: FAST achieves significant action-sequence compression across datasets at comparable reconstruction accuracy, though the compression is not entirely lossless.The compression–fidelity trade-off is controlled by the scale parameter γ.
  • Policy training: Naïve tokenization cannot make progress on the high-frequency Table Bussing and T-Shirt Folding tasks, whereas compression-based tokenization enables effective training.These tasks operate at 20Hz and 50Hz, respectively.
  • Policy training: FAST performs as well as or better than FSQ, particularly on dexterous, high-frequency tasks, while requiring no separate neural-network training.Figure 6 compares FAST, FSQ, naïve binning, and FAST+ across policy-training evaluations.
  • Generalization: FAST tokenization enables a strong DROID generalist policy to perform simple manipulation tasks zero-shot in unseen environments without fine-tuning.The policy was evaluated across environments on three university campuses.

C. Universal Action Tokenizer

FAST+ is a universal tokenizer trained on diverse real-robot trajectories and tested on unseen robot datasets. It provides broad compression while preserving policy performance close to dataset-specific FAST tokenizers.

  • Training and evaluation: FAST+ was trained on 1M real-robot action sequences spanning diverse morphologies, action spaces, and control frequencies.Evaluation datasets were entirely unseen during tokenizer training.
  • Compression performance: FAST+ reduces the number of action tokens by 2x across all tested datasets, with larger reductions on some datasets.The tests cover new robot setups without recomputing the tokenization.
  • Policy performance: Across all tasks, FAST+ closely matches dataset-specific FAST tokenizers in policy performance.This supports using FAST+ as a strong default tokenizer for robot action sequences.

D. Ablation Studies

The ablations examine whether FAST transfers across VLA backbones and whether BPE compression matters, while comparisons with diffusion π0 assess training efficiency and inference trade-offs. FAST improves OpenVLA training and converges faster than diffusion π0 on large datasets, but autoregressive inference is slower.

  • Backbone independence: FAST significantly boosts OpenVLA performance on high-frequency T-shirt folding, suggesting tokenization is independent of the underlying model backbone.The comparison uses OpenVLA’s naïve tokenization versus FAST+ with 1-second action chunks.
  • BPE ablation: Removing BPE worsens rollout performance and slows inference because repeated zero tokens dilute the learning signal and require hundreds of autoregressive predictions.The DCT transform still concentrates information in a few tokens, so the no-BPE variant remains better than naïve tokenization.
  • Comparison with diffusion: On small datasets below 50 hours, FAST and diffusion π0 perform comparably, while on Table Bussing FAST reaches high performance with 3x fewer training steps.In DROID evaluations, the FAST-based autoregressive model follows language instructions more closely.
  • Inference trade-off: FAST-based autoregressive inference takes approximately 750ms per 1-second chunk versus approximately 100ms for diffusion π0 on an NVIDIA 4090 GPU.The difference reflects 30–60 decoded action tokens and the full 2B-parameter backbone versus 10 diffusion steps and a 300M-parameter action expert.
  • Task capability: FAST enables autoregressive VLAs to perform complex, long-horizon, dexterous laundry folding that previous tokenization schemes could not.The result is illustrated by a π0-FAST rollout on laundry folding.
  • Comparison with diffusion: π0-FAST matches diffusion π0 performance while requiring significantly less training compute.The comparison reports mean performance and 95% confidence intervals.

F. Scaling Autoregressive VLAs to Large Robot Datasets

FAST scales autoregressive VLAs to large, cross-embodied dexterous datasets, matching diffusion performance while substantially reducing training compute. It also achieves strong performance on individual large datasets with faster convergence.

  • Conclusion: The broader summary reports that FAST enables autoregressive VLAs on dexterous tasks where prior tokenization schemes completely fail.Combined with π0, FAST supports generalist policies that rival diffusion VLAs while being faster to train.
  • Scope and future work: FAST+ remains experimentally bounded by policy evaluations on static robot manipulators, despite offline compression results for mobile robots, dexterous hands, and humanoids.Actual policy performance on those other morphologies remains future work.

APPENDIX

The appendix describes the universal tokenizer’s diverse training data and policy setup, alongside compression–reconstruction sweeps and decoding details. It also records the implementation choices used for reproducibility.

  • Universal tokenizer data: FAST+ training data spans multiple robot morphologies, action spaces, and control frequencies, with joint, end-effector world-frame, and camera-frame parameterizations.Actions are padded to 32 dimensions before tokenization.
  • Universal tokenizer data: The appendix lists datasets covering bi-manual, single-arm, and mobile robots across joint, end-effector, and camera-frame action spaces.The listed control frequencies range from 5 to 50.
  • Policy setup: Policies use π0 or OpenVLA backbones conditioned on images, natural-language instructions, and proprioceptive state.Images are encoded separately at 224x224 resolution before their tokens are concatenated.
  • Tokenizer evaluation: Compression–reconstruction tradeoffs are swept by varying FAST rounding scale, naïve-tokenization subsampling frequency, and FSQ latent-token count, with vocabulary size held constant.Output token count represents the compression level.
  • Optimization: Training uses a 1k-step linear warmup, constant learning rate 5e-5, AdamW, gradient clipping at 1, and EMA decay 0.999.AdamW uses β1 = 0.9 and β2 = 0.95 without weight decay.
  • Inference: Inference uses greedy autoregressive decoding, with β = 0.7 for specified bi-manual tasks to help policies leave the home position.Those tasks contain stationary action chunks at episode beginnings.

D. DROID Policy Setup

The DROID setup evaluates zero-shot policies across a structured suite of tasks, datasets, initial conditions, and progress-based scoring criteria. The appendix documents both training protocols and task-specific evaluation scenes.

  • Evaluation tasks: Table Bussing uses a single UR5e arm to place objects into a trash can or bin, scoring the percentage correctly handled in an unseen twelve-object scene.The scene includes challenging objects and placements such as utensils on trash.
  • Evaluation protocol: The evaluation materials specify sampled initial configurations and quantitative DROID setups, while scoring uses task-progress rubrics.The evaluation suite is summarized as TABLE II: DROID evaluation tasks.
  • Evaluation tasks: T-Shirt Folding uses a bimanual ARX robot and scores the percentage of five varying shirts successfully folded by a human rater.The shirts begin flat and vary in color and size.
  • Evaluation tasks: Grocery Bagging evaluates a pretrained single-arm UR5e on seven varied items, scoring the percentage placed into a paper grocery bag.The evaluation is performed out-of-the-box on models pretrained with the full mixture.
  • Evaluation tasks: Toast out of Toaster evaluates a bimanual Trossen ViperX on removing two toast pieces and placing them on a plate, with a four-point progress score.The setup mirrors the ALOHA configuration.
  • Evaluation tasks: Laundry Folding evaluates a bimanual ARX on folding clothing from a laundry bin, requiring unfurling and flattening before folding.The task is described as very challenging because the laundry is tangled.
  • DROID setup: DROID training uses 75k successful episodes and 21M samples, followed by zero-shot deployment in new scenes, camera angles, backgrounds, and objects.The quantitative suite contains 16 tasks and 44 trials per policy.

TOAST OUT

The appendix includes a compute-matched comparison of π0-FAST and diffusion π0, plus a universal-tokenizer evaluation-dataset table.

  • Compute-matched performance: Figure 15 compares π0-FAST with compute-matched diffusion π0 using mean performance and 95% confidence intervals.The figure caption states that π0-FAST clearly outperforms diffusion π0 at equal training compute because of faster convergence.
  • Universal tokenizer evaluation: The appendix identifies TABLE III as the universal-tokenizer evaluation-dataset table.The supplied passage names the table but does not describe its entries.
Loading 2501.09747v1…