PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory-Level Continuous Action Representation for Robotic Manipulation

Tong Yang, Jingkai Jia, Yuecheng Xu, Xueyao Chen, Chi Zhang, Wenqiang Zhang

arXiv:2608.24111v1cs.RO

TL;DR

Existing visuomotor action representations can become redundant as sampling frequency rises and may rely on predefined temporal parameterizations. CAT encodes fixed-window trajectories into continuous latent tokens with frequency-aware temporal coordinates, and CAT-based policies consistently outperform competitive baselines across simulated and real-world settings.

  • Problem

    Timestep-level representations grow with sampling frequency even when task-relevant trajectory variation does not, while some alternatives rely on predefined temporal parameterizations.

  • Method

    CAT encodes fixed-real-time-window action trajectories into continuous latent tokens using frequency-aware positional encoding, reconstruction, and regularization.

  • Results

    CAT-based policies consistently outperform competitive VQ-based and continuous-control baselines across simulated and real-world manipulation tasks under varying control frequencies.

  • Takeaways & Limitations

    Trajectory-level continuous representations support visuomotor policy learning across varying control frequencies and long-horizon real-robot manipulation tasks.

Abstract

from arXiv · show

We propose CAT, a trajectory-level continuous action representation framework for robotic manipulation. Existing visuomotor systems often entangle action representation with control frequency or rely on fixed temporal parameterizations. This leads to representational redundancy at high sampling rates and limits the modeling of critical motion. CAT instead encodes action trajectories within a fixed real-time interval into a set of continuous latent tokens. To ensure temporal consistency across varying control frequencies, we further incorporate a frequency-aware positional encoding that establishs a shared temporal coordinate system. Trajectory-level regularization further stabilizes the latent representation. This approach prevents representation growth with timestep density and avoids reliance on predefined temporal parameterizations. Extensive system-level evaluations on LIBERO, MimicGen, and real-world long-horizon manipulation tasks demonstrate that CAT-based policies consistently outperform both competitive VQ-based and continuous visuomotor baselines under matched training settings. Across various model backbones and control frequencies, CAT consistently improves success rates. These results highlight the advantages of trajectory-level continuous action modeling for scalable robotic manipulation across varying control rates.

1 Introduction

CAT addresses frequency-dependent redundancy and uneven temporal importance in action trajectories by encoding fixed-window trajectories into continuous latent tokens. Its frequency-aware representation supports consistent control across sampling rates, and evaluations report stronger performance than competitive baselines.

  • Motivation: Localized contact transitions and corrective adjustments can matter more for task success than smooth, repetitive trajectory regions.This temporal imbalance can dilute learning signals for shorter but consequential action segments.
  • Motivation: Higher sampling frequencies expand timestep-level representations even when task-relevant trajectory variation does not increase proportionally.This creates temporal redundancy and motivates alternatives to fixed temporal grids or predefined reparameterizations.
  • CAT: CAT encodes fixed-real-time-window trajectories into a constant number of continuous latent tokens across control sampling rates.Frequency-aware positional encoding normalizes timestep indices into shared temporal coordinates while preserving frequency-dependent variation.
  • CAT: CAT learns its representation with reconstruction and regularization objectives and integrates the resulting latent tokens with a flow-matching policy learner.The representation does not rely on predefined temporal parameterizations.
  • Evaluation: CAT-based policies outperform competitive VQ-based and continuous-control baselines across simulated and real-world robotic manipulation tasks.The reported gains persist under varying control frequencies and in long-horizon real-robot settings.

2 Related Work

Prior visuomotor work includes autoregressive and diffusion-based paradigms, while CAT provides a fixed-size continuous trajectory representation for varying sampling rates. CAT’s overview uses frequency-aware encoding and decoding to connect action trajectories with compact latent representations.

  • Vision-Language-Action Models: Vision-Language-Action models map multimodal inputs to action outputs and commonly use autoregressive or diffusion-based paradigms.Autoregressive approaches typically predict actions step by step using discrete tokens.
  • CAT: CAT’s frequency-aware transformer encoder compresses trajectories into latent tokens, while its decoder reconstructs continuous trajectories for downstream generation.Both components incorporate frequency-scaled timestep embeddings and can integrate with diffusion-based VLA frameworks.
  • CAT: CAT represents continuous action trajectories over a fixed real-time interval with token dimensionality constant across sampling rates.This avoids timestep-level discretization and predefined temporal or spectral bases.

3 Method

CAT represents fixed-window action trajectories with a fixed number of continuous latent tokens, aligns different control rates in shared temporal coordinates, and regularizes the latent space for discriminability. It integrates these representations into diffusion-based VLA policies while preserving trajectory decoding across varying control rates.

  • Structural Decoupling: CAT encodes trajectories within a fixed real-time window into K continuous latent tokens, with K remaining independent of timestep density.The method uses N = f_sT timesteps while keeping K fixed across sampling rates, with K << N.
  • Trajectory Encoding and Decoding: The encoder aggregates full action sequences into latent tokens, while the decoder reconstructs continuous actions from register tokens and timestep-specific action embeddings.Each action embedding corresponds to one timestep and is decoded into a reconstructed action; causal temporal attention regulates decoder information flow.
  • Frequency Alignment: Frequency-aware positional encoding normalizes timestep indices by control frequency to align trajectories sampled at different rates in shared temporal coordinates.The normalized coordinate t/f_s is mapped through sinusoidal and cosine components parameterized by frequency coefficients.
  • Latent Stability: CAT combines reconstruction loss with trajectory-level contrastive regularization to learn a compact yet discriminative continuous latent representation.The contrastive term penalizes small distances between trajectory embeddings, while reconstruction preserves intra-trajectory cohesion.
  • Integration into Diffusion-Based VLA: Within diffusion-based VLA systems, CAT replaces timestep-level action parameterization with fixed-length continuous latent trajectories that are decoded into actions during inference.The diffusion model refines noisy latent trajectories into clean representations, which CAT decodes into continuous actions under varying control rates.

4 Experiments

The experiments test whether trajectory-level continuous latent actions improve policy learning, frequency robustness, and long-horizon deployment. Across simulated and real-world evaluations, CAT consistently improves performance over timestep-level, discretization-dependent, and baseline representations.

  • Experimental Design: The evaluation covers LIBERO, MimicGen, RoboTwin 2.0, and four real-world long-horizon manipulation tasks under controlled comparisons.Experiments compare trajectory-level continuous latents with timestep-level, discretization-dependent, and fixed-basis representations under matched conditions.
  • LIBERO: 90.8% versus 87.3%: SmolVLA-CAT improves LIBERO average success rate by 3.5 points over timestep-level SmolVLA.Under the same backbone and policy architecture, SmolVLA-CAT also consistently outperforms SmolVLA-FAST and SmolVLA-VQ-VLA.
  • LIBERO: 96.5% versus 94.2%: integrating CAT into diffusion-based π0 improves performance, indicating gains across model scales.The controlled LIBERO results support trajectory-level continuous latents as an effective policy-learning inductive bias.
  • MimicGen: 86.9% versus 85.8%: DP-CAT improves MimicGen average success by 1.1 points under matched parameter counts.The largest task improvement is on Stack 3, where DP-CAT reaches 85% versus 76% for DP, a 9-point gain.
  • RoboTwin 2.0: At 10Hz, 16.7Hz, 25Hz, and 50Hz, SmolVLA-CAT gains +10.3, +4.0, +10.2, and +5.3 points over SmolVLA, respectively.CAT outperforms the baseline in 24 of 32 task–frequency pairs, while the best frequency varies by task.
  • Real-World Long-Horizon Manipulation: 60.5 versus 45.0: CAT improves the average normalized real-world score by 15.5 points and outperforms the baseline on all four tasks.The gains are +14 on Stack-5, +12 on Drawer, +8 on Rope, and +28 on Flower across sequential manipulation scenarios.
  • Ablation Studies: F-RoPE is more effective when data and evaluation involve varying sampling rates, according to the frequency-related results.The ablation results also show non-monotonic effects of register count and best performance at λreg = 0.1.
  • Ablation Studies: Causal decoder attention improves average success rate by 2.25 points over full attention across all task suites.This supports explicit temporal directionality during decoding for trajectory-level action representations.

5 Conclusion

CAT represents action trajectories over fixed real-time intervals with continuous latent tokens, supporting policy operation across varying control frequencies. Integrated with diffusion-based policy learning, it improves performance across simulated and real-robot manipulation tasks.

  • CAT represents action trajectories within a fixed real-time interval using continuous latent tokens.
  • The representation supports policy operation under varying control frequencies.
  • Integrated with diffusion-based policy learning, CAT consistently improves performance across simulated benchmarks and real-robot manipulation tasks.

A Implementation Details

This section specifies the architecture and training details used across experiments for implementation clarity.

  • The section provides architecture details used across all experiments.
  • The section provides training details used across all experiments.
  • The stated purpose is implementation clarity.

A.1 Architecture

CAT uses an encoder-decoder Transformer architecture to aggregate action-chunk information and autoregressively reconstruct latent representations.

  • CAT uses an 8-layer Transformer encoder and an 8-layer Transformer decoder, each with hidden size 512 and 8 attention heads.
  • The encoder uses full attention to aggregate information across the entire action chunk.
  • The decoder uses an asymmetric causal mask that permits all encoder outputs but only past decoder positions.

A.2 Training Pipeline

CAT training proceeds in two stages: learning continuous trajectory representations, then training a downstream policy with the pretrained CAT kept fixed.

  • The first stage trains CAT to learn continuous latent representations of action trajectories.
  • The second stage integrates pretrained CAT into a policy model for downstream control-task training.
  • During second-stage policy training, CAT remains fixed.

A.3 Simulation Experiments

Simulation experiments cover LIBERO, MimicGen, and RoboTwin 2.0 under dataset- and frequency-specific training protocols. Evaluation uses success rate with horizons and executed actions scaled by control frequency.

  • LIBERO uses 1,693 preprocessed expert demonstrations in LeRobot format, matching the dataset version used by SmolVLA.
  • CAT and the integrated SmolVLA policy are each trained on LIBERO for 100k steps with batch size 64 and chunk size 16.
  • LIBERO evaluation uses an action horizon of 8, predicting 16 future actions per step and executing the first 8 with a fixed random seed.
  • MimicGen uses an 8-task Robosuite subset, with CAT trained for 200 epochs before integration into a flow-matching diffusion policy trained for 300 epochs.
  • RoboTwin 2.0 provides trajectories at 10 Hz, 16.7 Hz, 25 Hz, and 50 Hz, with 50 episodes collected per frequency and 1-second action chunks.
  • Success rate is the primary metric; models predict ⌊fs⌋ future actions and execute the first ⌊fs/2⌋, while episode limits scale relative to 16.7 Hz.

A.4 Real-World Experiments

The real-world experiments pretrain CAT on OpenX, fine-tune it for each task using 20 Hz tele-operated data, and then train the SmolVLA-CAT policy.

  • CAT is pretrained on OpenX for 400k steps using 1-second action chunks, then fine-tuned separately for each real-world task.
  • Task-specific fine-tuning uses tele-operated data collected at 20 Hz for 1k steps with 16-step action chunks.
  • SmolVLA-CAT is trained for 180k steps with learning rate 2e−4, batch size 64, and an action chunk size specified in the experimental protocol.

B Qualitative Results

Qualitative results visualize successful rollouts across simulation and real-world manipulation tasks. Each rollout row shows progression from an initial state to a final state, with representative task suites spanning the benchmarks.

  • Each qualitative rollout row represents a single task, with the leftmost frame showing the initial state and the rightmost frame showing the final state.
  • LIBERO: LIBERO visualizations cover the Spatial, Object, Goal, and Long task suites using uniformly sampled frames from successful episodes.
  • MimicGen: MimicGen visualizations cover Coffee, Hammer, Mug, Nut, Square, Threading, Stack, and Stack3, with frames sampled from successful episodes.
  • RoboTwin 2.0: RoboTwin 2.0 visualizations cover eight manipulation tasks, and all rollouts are evaluated at a test-time control frequency of 16.7Hz.
  • Real-world tasks: The real-world setup includes four tasks: placing a black bowl on a plate, placing orange juice in a basket, placing a wine bottle on a rack, and closing a microwave after inserting mugs.
Loading 2608.24111v1…