PaperlayerView exact revision on arXiv

Source-linked AI summary

Direction-Scale Decomposition in Action Representation: Rethinking What to Tokenize for Vision-Language-Action Models

Yufei Duan, Hang Yin, Alberta Longhini, Chao Tang, Danica Kragic

arXiv:2609.28865v1cs.CVcs.RO

TL;DR

Raw pose-increment action tokens can be sensitive to execution speed and dataset-specific normalization, while action representation remains underexamined. The paper introduces Direction-Scale Decomposition (DSD), which separates directional and scale components before tokenization. It reports improved average LIBERO success with BIN and BEAST and a 10.3-percentage-point SimplerEnv gain for DSD-BIN over BIN under mixed-dataset training.

  • Problem

    Raw pose-increment tokens can vary with execution speed and dataset-specific normalization, while action representation remains underexamined.

    Source: §I · p. 1, A. Problem 1: irrelevant factors hide… · p. 3

  • Method

    DSD decomposes translation and rotation increments into direction–magnitude and axis–angle components before normalization, retaining magnitudes in separate scale channels.

    Source: geometric structure across… · p. 2

  • Results

    DSD improves average LIBERO success with both BIN and BEAST, and DSD-BIN exceeds BIN by 10.3 percentage points on SimplerEnv under heterogeneous co-training.

    Source: geometric structure across… · p. 2, B. Simulation: LIBERO · p. 6, C. SimplerEnv · p. 7

  • Takeaways & Limitations

    The findings support DSD as an effective representation for discrete-token VLAs and motivate controlled studies across larger dataset mixtures, additional tokenizers, and broader embodiments.

    Source: §VIII · p. 8

    Mixed-dataset evaluation was restricted to five OXE datasets, leaving DSD’s effectiveness on larger mixtures, including the full OXE collection, untested.

    Source: §VII · p. 8

Abstract

from arXiv · show

Action representation plays a central role in discrete-token vision-language-action (VLA) learning but remains underexamined. Under conventional pose-increment representations, action tokens are sensitive to execution speed and dataset-specific normalization, potentially obscuring geometric structure shared across demonstrations and datasets. We introduce Direction-Scale Decomposition (DSD), an action representation that decomposes translation and rotation increments into direction and scale components before tokenization. DSD isolates motion direction while retaining magnitudes in separate scale channels. We evaluate DSD with uniform binning (BIN) and BEAST, a B-spline-based tokenizer, in simulation and real-world manipulation under both single-dataset and mixed-dataset training. On LIBERO, DSD improves average success rates with both tokenizers. On SimplerEnv, DSD-BIN outperforms BIN by 10.3 percentage points in overall success rate under mixed-dataset training. Real-robot experiments further show gains both with and without robotics pretraining. These results support DSD as an effective action representation for discrete-token VLA models and suggest its potential to mitigate performance degradation when training on large and diverse dataset mixtures. Our project page with additional resources is available at https://vla-dsd.github.io/

I. INTRODUCTION

Discrete-token VLA action representations must account for execution-speed and dataset-normalization variation, yet raw pose increments can obscure shared geometric patterns. The paper introduces Direction-Scale Decomposition (DSD) and reports gains across simulation, path tracking, and real-robot experiments.

I. INTRODUCTION

  • DSD–BIN achieves 20.5% lower mean DTW than BIN in triangle tracking.DSD separates translation into direction and magnitude and rotation into axis and angle before tokenization.Source: Figure 1 · p. 1, geometric structure across… · p. 2
  • Raw pose-increment tokens can vary with execution speed and dataset-specific normalization, while mismatched normalization can distort motion direction during transfer.The paper frames these issues as obstacles to exposing shared geometric structure across demonstrations and datasets.Source: §I · p. 1

geometric structure across demonstrations despite variations in execution speed and dataset-specific normalization?

  • DSD improves average LIBERO success with both BIN and BEAST, while DSD-BIN exceeds BIN by 10.3 percentage points on SimplerEnv under heterogeneous co-training.Real-robot gains are also reported both without robotics pretraining and after fine-tuning pretrained checkpoints.Source: geometric structure across… · p. 2
  • DSD factorizes translation and rotation increments into direction–magnitude and axis–angle pairs before scale normalization, retaining magnitudes in separate channels.The authors describe DSD as an analytic representation compatible with different tokenization schemes.Source: geometric structure across… · p. 2

II. RELATED WORK

The related work distinguishes VLA action-prediction approaches and tokenization families, while the preliminaries describe dataset-specific normalization, tokenization, and decoding. DSD is presented as a representation that can complement different tokenizers.

A. Vision-language-action models

  • RT-2 [2] and OpenVLA adapt pretrained vision-language models for action prediction, while π0 [5] uses flow matching and π0.5 [10] combines discrete-token pretraining with flow-matching post-training.These approaches generally use dataset-specific normalization.Source: A. Vision-language-action models · p. 2

B. Action Tokenization

  • Uniform binning, structured compression (FAST [7], BEAST), and learned codebooks (VQ-BeT, ActionCodec [12]) are distinct action-tokenization families; the cited structured methods operate on dataset-normalized actions.The learned methods require a separately trained quantizer and may produce unreliable codes for action chunks outside the quantizer’s training distribution.Source: B. Action Tokenization · p. 2
  • KAT, SpatialVLA [14], and MoMo [15] exploit geometric or dynamical structure, whereas DSD analytically decomposes relative translation and rotation before normalization and can complement different tokenizers.The passage distinguishes DSD as an action representation rather than a tokenizer.Source: B. Action Tokenization · p. 2

III. PRELIMINARIES

  • Dataset-specific statistics normalize action coordinates, and tokenizers map normalized action chunks to token sequences that are decoded and denormalized using target-setup statistics.The resulting physical commands depend on both the tokenizer and the denormalization statistics.Source: §III · pp. 2–3

IV. LIMITATIONS OF CURRENT ACTION REPRESENTATIONS

The section identifies three problems with current action representations and states that it examines how they can affect the model.

IV. LIMITATIONS OF CURRENT ACTION REPRESENTATIONS

  • The section frames current action representations as having three problems and directs the discussion to their effects on the model.The passage refers to Fig. 2 but does not specify the three problems.Source: §IV · p. 3

A. Problem 1: irrelevant factors hide action patterns

The section describes how speed variation and dataset-specific normalization can make action tokens inconsistent or distort transferred motion. It then introduces a decomposition of scale and normalization as the next topic.

A. Problem 1: irrelevant factors hide action patterns

  • Spatially similar paths executed at different speeds can receive widely separated tokens, making task-relevant action patterns harder to distinguish from velocity variation.The passage notes this can occur across teleoperators and even across demonstrations from one teleoperator.Source: A. Problem 1: irrelevant factors hide… · pp. 3–4

B. Problem 2: Inconsistency in dataset normalization

  • Dataset-specific normalization can assign different token targets to the same physical action, requiring mixed-dataset training to associate one motion with distinct targets.The passage connects this inconsistency with difficulty reusing shared action patterns across datasets.Source: B. Problem 2: Inconsistency in dataset… · p. 4

C. Problem 3: direction distortion in action transfer

  • A token pattern learned under source-dataset normalization can produce a different physical motion after target-side denormalization, including a changed action direction.The mismatch can arise from coordinate-specific offsets and scales.Source: C. Problem 3: direction distortion in… · p. 4
  • A positive z displacement can become a negative physical displacement after mismatched normalization and denormalization, even without clipping or quantization.The example uses Bridge normalization followed by Berkeley-UR5 statistics.Source: C. Problem 3: direction distortion in… · p. 4
  • A shared binning range across datasets and translation axes must accommodate the largest motion scale, leaving smaller-range datasets fewer effective bins.This reduces effective quantization resolution for those smaller action ranges.Source: C. Problem 3: direction distortion in… · p. 4

V. DECOMPOSED SCALE AND NORMALIZATION

  • The following section introduces Direction-Scale Decomposition, its conversion from raw actions, and implementation remarks for training.Source: §V · p. 4

A. Direction–scale action representation

Direction–Scale Decomposition (DSD) represents translation and rotation increments through direction and scale components before normalization and tokenization. It retains the gripper command and the physical motion’s six degrees of freedom.

B. Conversion between raw and DSD representations

Raw pose increments are converted into translation and relative rotation, then represented by direction and scale components. The DSD representation can be reconstructed as a physical command and used with existing tokenizers.

  • The conversion starts from a raw action’s translation increment and relative rotation, expressing translation through magnitude and unit direction and rotation through angle and unit axis.Source: B. Conversion between raw and DSD… · p. 4
  • When translation magnitude is zero, the fixed default direction e3 has no effect on the represented action.Source: B. Conversion between raw and DSD… · p. 4
  • A numerically stable matrix-to-axis–angle conversion is used near zero and π, with a fixed sign convention resolving equivalent axes at π.Source: B. Conversion between raw and DSD… · p. 4
  • Reconstruction converts the rotation to roll–pitch–yaw and retains the gripper command, yielding a raw action for the same physical command.Source: B. Conversion between raw and DSD… · pp. 4–5
  • Fig. 3 shows demonstrations at five execution speeds and 10 rollouts per tokenizer; BEAST and DSD-BEAST each have two safety-guard terminations.Gold dashed curves mark the reference trajectory, and crosses mark terminations.Source: Figure 3 · p. 5
  • DSD scale channels are normalized to [−1,1], using shared parameters during co-training and target-specific parameters during fine-tuning; directions need no dataset-specific normalization.For DSD-BIN, scales and directional components are discretized into 256 equal-width bins; decoded scales are denormalized before motion reconstruction.Source: B. Conversion between raw and DSD… · p. 5

C. Stability: masking direction loss at small scale

Direction targets become noise-sensitive when translation magnitude or rotation angle approaches zero. The method reduces direction-token loss weight below a scale threshold while keeping a nonzero learning signal.

VI. EXPERIMENTS

The experiments test DSD against raw actions, tokenizer compatibility, heterogeneous co-training, and pretrained-checkpoint fine-tuning; the speed-varying tracking study tests path-shape retention.

VI. EXPERIMENTS

  • In-repository policies use Florence-2-Base, predict 20-step action chunks, and compare BIN and DSD-BIN with BEAST and DSD-BEAST.Each action dimension uses 256 bins; BEAST represents each chunk using 10 B-spline basis functions.Source: §VI · p. 5

A. Motion Tracking with Speed Varying Demonstrations

  • The speed-variation study uses 50 demonstrations across five velocities and evaluates 10 rollouts per method with DTW on normalized path shape and traversal order.The demonstrations comprise five velocities with 10 repetitions each; the evaluation also reports a demonstration-calibrated score.Source: A. Motion Tracking with Speed Varying… · pp. 5–6
  • DSD-BIN cuts mean DTW distance 20.5% versus BIN, and DSD-BEAST cuts it 10.1% versus BEAST across 10 rollouts.Table I reports average and best-case performance, including failed rollouts.Source: Table I · p. 6, A. Motion Tracking with Speed Varying… · p. 6
  • DSD raises average tracking score by 9.23 points with BIN and 11.23 with BEAST, supporting better path-geometry preservation under speed variation.The corresponding best-score improvements are 14.30 and 6.21 points, respectively.Source: A. Motion Tracking with Speed Varying… · p. 6

B. Simulation: LIBERO

On LIBERO, DSD improves average success with both BIN and BEAST; DSD-BEAST improves over BEAST across all four suites.

B. Simulation: LIBERO

  • DSD-BEAST achieves the highest Long-suite success among the compared methods at 88.6%.The evaluation covers LIBERO’s Spatial, Object, Goal, and Long suites, with 500 trials per suite.Source: B. Simulation: LIBERO · p. 6, Table II · p. 6
  • Without robotics pretraining, DSD-BIN averages 92.3% success, exceeding pretrained π-FAST at 85.6% and matching pretrained π0.π0 uses flow matching for continuous action prediction.Source: B. Simulation: LIBERO · p. 6, Table II · p. 6
  • DSD-BIN improves over BIN on three suites by 3.8 percentage points in average success, while DSD-BEAST improves over BEAST on all four by 3.9 points.Table II reports LIBERO success rates; “P” denotes robotics pretraining.Source: Table II · p. 6, B. Simulation: LIBERO · p. 6

C. SimplerEnv

In SimplerEnv, DSD-BIN benefits from heterogeneous co-training while BIN declines; real-robot experiments also show gains with and without robotics pretraining. The evaluations cover multiple tasks and training conditions.

C. SimplerEnv

  • SimplerEnv compares BridgeV2-only fine-tuning with a five-dataset mixture across three embodiments, using four tasks and 75 matched trials per task.The real-robot evaluation uses 30 trials per method on cube stacking, cloth folding, tissue sweeping, and spoon replacement, including contact-rich interactions and behaviors beyond pick-and-place.Source: C. SimplerEnv · p. 6, D. Real-robot experiments · p. 7
  • Under five-dataset co-training, DSD-BIN beats BIN by 10.3 percentage points and reaches 43.3% overall, highest among Table III methods.With BridgeV2-only training, DSD-BIN exceeds BIN by 0.7 percentage points; adding four datasets raises DSD-BIN from 40.7% to 43.3% while BIN falls from 40.0% to 33.0%.Source: C. SimplerEnv · pp. 6–7, Table III · p. 7

D. Real-robot experiments

  • Across four real-robot tasks, DSD raises average success by 13.3 points without pretraining and 11.7 points after robotics pretraining.DSD-BIN outperforms BIN on each task, and DSD-BIN-P outperforms BIN-P on each task; Figure 4 reports per-task rates and four-task averages.Source: Figure 4 · p. 7, D. Real-robot experiments · p. 7

C. SimplerEnv

  • Co-training improves DSD-BIN’s carrot and eggplant scores but reduces its spoon and stack scores.The DSD-BIN mixture achieves the highest overall success among Table III methods despite task-level variation.Source: C. SimplerEnv · p. 7

VII. DISCUSSION

The discussion frames DSD as exposing geometric structure amid speed and normalization variation, while noting limits in dataset and evaluation scope. The conclusion reports gains in simulation and on real robots and motivates broader studies.

VII. DISCUSSION

  • DSD exposes geometric structure that raw coordinate-based representations may obscure under execution-speed variation and dataset-specific normalization.The discussion connects this representation to inconsistent token targets and motion-direction distortion during transfer.Source: §VII · p. 8
  • The mixed-dataset findings are consistent with DSD-BIN benefiting from four added datasets while BIN’s overall performance declines.These results are limited to the tested mixture and do not establish performance on larger mixtures.Source: §VII · p. 8
  • The mixed-dataset evaluation covers five OXE datasets, leaving larger mixtures, including full OXE, untested; SimplerEnv gains also vary across tasks.The direction-learning threshold may require tuning across training stages; fine-tuning thresholds are generally smaller than the pretraining threshold.Source: §VII · p. 8
  • Evaluation is limited to BIN and BEAST, Franka and WidowX in simulation, and Franka in the real world; broader embodiments and bimanual manipulation remain future directions.The discussion also identifies uncertain metric scale in monocular egocentric demonstrations as an area for future evaluation.Source: §VII · p. 8

VIII. CONCLUSION

  • Simulation and real-robot experiments show improvements in average success and normalized path fidelity, with benefits after fine-tuning representation-matched pretrained checkpoints.In the tested five-dataset mixture, DSD-BIN benefits from additional heterogeneous data while BIN’s overall performance declines.Source: §VIII · p. 8
Loading 2609.28865v1…