PaperlayerView exact revision on arXiv

Source-linked AI summary

ReAT: Residual Action Tokenization for Autoregressive Robot Policies

Dijia Zhan, Zhuo Li, Xuemiao Xu, Sun Zheng, Jinyi Li, Wenjun Wang, Jie Tang

arXiv:2610.13093v2cs.RO

TL;DR

Existing action tokenizers can leave each token’s contribution to the reconstructed action implicit, despite supplying autoregressive policies with discrete targets. ReAT constructs each token from the remaining error in the decoded draft and trains prefixes to reconstruct complete chunks. Across the evaluated tasks, 8-token ReAT outperforms the evaluated baselines, while 4-token ReAT surpasses 8-token action-tokenization baselines on LIBERO.

  • Problem

    Existing tokenization objectives do not explicitly define each token’s contribution relative to actions already reconstructed, leaving the role of successive tokens implicit.

    Source: §1 · p. 2

  • Method

    ReAT encodes the remaining error against the decoded prefix as the next token and uses prefix reconstruction supervision so each nonempty prefix reconstructs a complete action chunk.

    Source: §1 · p. 2

  • Results

    With 8 tokens, ReAT outperforms evaluated action-tokenization and diffusion-policy baselines across three simulation benchmarks and real-world bucket stacking; 4-token ReAT surpasses evaluated 8-token action-tokenization baselines on LIBERO.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results support explicit residual correction as a design principle for structuring how successive action tokens contribute to reconstruction.

    Source: §7 · p. 9

    Physical evaluation covers one task with 20 trials per configuration, and additional tasks and hardware platforms would clarify deployment generality.

    Source: §6 · p. 9

Abstract

from arXiv · show

Autoregressive robot policies rely on action tokenizers to convert continuous action chunks into compact sequences of discrete prediction targets. Existing tokenizers can reconstruct action chunks from discrete codes, but often leave implicit what each individual token contributes to the reconstructed action. We introduce Residual Action Tokenization (ReAT), which gives each token an explicit role in refining the current action reconstruction. It uses the decoded reconstruction as feedback to construct the next token from the remaining action error. This closes the loop between encoding and decoding, making each new token responsive to the errors left by the actual decoded prefix. Across 19 manipulation tasks spanning three simulation benchmarks and real-world settings, 8-token ReAT policies outperform the evaluated baselines in task success; on LIBERO, even 4-token ReAT surpasses the evaluated 8-token action-tokenization baselines at half the token budget. These results suggest a path toward action tokenization that supports self-corrective policy learning, helping autoregressive policies learn to progressively refine their own action plans.

1 INTRODUCTION

ReAT addresses the implicit role of individual tokens in existing action representations by constructing each successive token from the error left by the decoded prefix. Across the evaluated settings, ReAT improves task success, including a LIBERO comparison using half the token budget of the evaluated action-tokenization baselines.

1 INTRODUCTION

  • Existing tokenizers can leave each token’s contribution relative to actions already reconstructed implicit, despite reconstruction, ordering, and prefix-supervision objectives.The remaining action error offers an explicit basis for constructing subsequent tokens.Source: §1 · p. 2
  • ReAT encodes the remaining error against the decoded action draft, making each successive token a correction that reflects quantization and decoding effects.The draft is updated after each token and used to determine the next token’s target.Source: §1 · p. 2
  • With 8 tokens, ReAT outperforms evaluated action-tokenization and diffusion-policy baselines across three simulation benchmarks and real-world bucket stacking; 4-token ReAT surpasses evaluated 8-token action-tokenization baselines on LIBERO.The evaluation spans 19 manipulation tasks.Source: §1 · pp. 2–3
  • Prefix reconstruction supervision makes every nonempty token prefix a complete action chunk, allowing one autoregressive policy to operate at different token budgets without retraining.The paper identifies prefix supervision as enabling effective control with shorter token sequences.Source: §1 · p. 2

2 RELATED WORK

Related work advances robot action generation and the compactness or ordering of action representations. ReAT focuses on structuring discrete action targets so successive tokens explicitly correct the decoded action reconstruction.

Generative robot policies

Action tokenization and ordered prefixes

Residual coding and reconstruction feedback

  • Residual quantization approximates latent features, whereas recurrent image codecs feed decoded-output errors into compression; ReAT brings reconstruction feedback to action tokenization.The passage distinguishes latent approximation error from error remaining in decoded actions.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define the observation and action-chunk setting for autoregressive robot policies. A tokenizer supplies prefix-decodable training targets, and the architecture depicts how policies use them.

3 PRELIMINARIES

  • An observation contains camera images and proprioception, and the policy predicts a chunk of H actions with d control dimensions from an observation history.The robot executes the first H_e ≤ H commands before observing and replanning.Source: §3 · p. 3
  • A tokenizer maps an expert action chunk to tokens, with every nonempty prefix required to decode into a complete chunk used as an autoregressive policy target.At deployment, the policy predicts tokens from observations and the decoder reconstructs actions without access to the expert chunk.Source: §3 · p. 3
  • The architecture shows residual tokenization with reconstruction feedback, draft-conditioned decoding, and an observation-conditioned policy whose predicted prefixes are decoded by the frozen tokenizer.Source: Figure 3 · p. 4

4 RESIDUAL ACTION TOKENIZATION

ReAT constructs tokens in a reconstruction-feedback loop, then decodes each token as a correction to a complete action draft. Prefix-level training and policy learning support action chunks at different token budgets.

4 RESIDUAL ACTION TOKENIZATION

  • Starting from a zero draft, ReAT encodes the remaining error, refines the draft with each token, and reuses the updated reconstruction to construct the next token.The loop constructs training labels; at deployment, the policy predicts tokens and only the decoder is needed to reconstruct actions.Source: §4 · p. 4

4.1 RESIDUAL TOKEN CONSTRUCTION

  • At stage k, the encoder maps the difference between the expert chunk and the prior draft to a quantized token, rerunning on the updated residual after decoding.The stage-specific encoder readout produces the policy label token index and the quantized embedding used by the decoder; the residual is measured against the actual decoded reconstruction.Source: §4.1 · p. 4

4.2 RECONSTRUCTION-CONDITIONED REFINEMENT

  • The decoder combines each token embedding with the current draft to predict a chunk-wide correction, so the same token can update different drafts differently.Every stage produces a complete H-step chunk; token count controls refinement depth, not chunk duration.Source: §4.2 · p. 5

4.3 TRAINING FOR PROGRESSIVE RECONSTRUCTION

  • Training samples a prefix length uniformly and compares its decoded chunk with expert actions, while a progress term penalizes increases in successive-prefix motion and control error.The regularizer also penalizes correction magnitude and temporal spread; the progress term encourages, but does not guarantee, improvement with each added token.Source: §4.3 · p. 5

4.4 AUTOREGRESSIVE POLICY LEARNING AND INFERENCE

  • After tokenizer training, the tokenizer is frozen and the observation-conditioned policy learns target tokens; at inference, an externally selected token budget decodes to a complete chunk without retraining the policy.The controller executes the first H_e actions before replanning.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments evaluate ReAT across simulation benchmarks and real-world bucket stacking, examining policy success, token-budget trade-offs, inference cost, reconstruction, and design choices.

5.1 BENCHMARKS, DATA, AND EVALUATION PROTOCOL

  • Evaluation spans 19 manipulation tasks across LIBERO-10, MetaWorld, RoboCasa, and real-world bucket stacking, with five simulation rounds of 50 episodes per task.Each simulation round’s success is averaged equally across tasks; reported results are the mean ± SEM across five round-level averages.Source: §5.1 · p. 6

5.2 POLICY PERFORMANCE AND TOKEN-BUDGET TRADE-OFFS

  • At eight tokens, ReAT exceeds the strongest baseline by 3.28, 1.80, and 2.00 percentage points on LIBERO, MetaWorld, and RoboCasa, respectively.The comparison covers the three simulation benchmarks.Source: Table 2 · p. 7, §5.2 · p. 7
  • Two-token ReAT lowers inference latency to 7.6–11.0 ms, while eight-token ReAT takes 22.2–25.4 ms versus OAT’s 18.2–22.6 ms.Latency includes observation encoding, token generation, and action decoding; the results characterize a control–latency trade-off, not a uniform speed advantage.Source: Table 3 · p. 7, §5.2 · p. 7
  • Increasing ReAT from two to eight tokens reduces LIBERO reconstruction MSE from 0.009491 to 0.001763, reaching 20.1% below OAT at eight tokens.FAST and Bin obtain lower MSE with longer sequences but lower policy success, so reconstruction fidelity alone does not determine control performance.Source: §5.2 · pp. 7–8

5.3 REAL-WORLD BUCKET STACKING

  • Eight-token ReAT achieves 70% success, compared with 40% for OAT and 35% for QueST in real-world bucket stacking.Success is not monotonic across token budgets.Source: Table 4 · p. 8, §5.3 · p. 8
  • Additional observations show Bin repeatedly approaches the bucket rim without aligning its grasp, while FAST encounters policy-generated token sequences with an incorrect number of action coefficients.A held-out validation check confirms FAST’s policy-generation failure mode, although expert-action tokens decode successfully.Source: §5.3 · p. 8

5.4 ABLATION STUDIES

  • The 1000-code FSQ configuration performs best on LIBERO at eight tokens, and increasing capacity further does not improve policy success.Source: Table 5 · p. 9, §5.4 · p. 9
  • Feedback-off reduces eight-token LIBERO success from 71.16% to 60.24% and performs worse at every tested budget.Full-length-only training retains similar eight-token performance but performs poorly at shorter prefixes, supporting prefix supervision.Source: Table 6 · p. 9, §5.4 · p. 9

6 DISCUSSION AND LIMITATIONS

ReAT gives tokens a corrective role, but its evaluation and control setup leave practical boundaries.

6 DISCUSSION AND LIMITATIONS

  • The same token can produce different updates for different drafts, making its role corrective rather than a fixed motion meaning.Source: §6 · p. 9
  • Budgets are externally selected and chunks are generated before execution; learned stopping, asynchronous refinement, and inter-chunk continuity remain open directions.Source: §6 · p. 9
  • Output bounds do not guarantee safe motion, and feedback does not eliminate policy prediction errors; physical evaluation covers one task with 20 trials per configuration.Additional tasks and hardware platforms would clarify deployment generality.Source: §6 · p. 9

7 CONCLUSION

ReAT constructs action tokens from decoded-prefix residuals and turns them into draft-conditioned corrections. Experiments demonstrate control performance across simulation and real-world bucket stacking, while LIBERO ablations support reconstruction feedback and prefix supervision.

7 CONCLUSION

  • ReAT builds each token from the residual left by the decoded prefix and converts it into a draft-conditioned correction.Source: §7 · p. 9
  • Prefix supervision lets the same autoregressive policy use multiple token budgets while preserving the action horizon.Source: §7 · p. 9
  • The experiments support explicit residual correction as a design principle that structures how successive tokens contribute to action reconstruction.Source: §7 · p. 9

ETHICS STATEMENT

The work uses existing robot demonstration datasets, simulation, and a physical bucket-stacking task, without a new human-subject study.

ETHICS STATEMENT

  • The evaluation uses existing robot demonstration datasets, simulation, and a physical bucket-stacking task.Source: ETHICS STATEMENT · p. 10
  • The study reports no new human-subject study.Source: ETHICS STATEMENT · p. 10
  • Physical deployment requires robot-specific safety measures beyond bounded action outputs, and dataset reuse is subject to original licenses and attribution requirements.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement directs readers to the sections and appendices containing the method, evaluation protocol, implementation details, data interfaces, uncertainty estimation, timing protocol, and model provenance.

REPRODUCIBILITY STATEMENT

A.1 ARCHITECTURE AND OPTIMIZATION

Appendix A documents ReAT’s architecture and optimization, training and checkpoint selection, action representation, recurrent decoder, tokenizer objective, and token construction at deployment.

A.1 ARCHITECTURE AND OPTIMIZATION

  • The main configuration uses eight content tokens, a 1,000-code vocabulary, width-256 encoder and decoder, and a four-layer width-256 policy.Source: §A.1 · p. 13, Table 7

A.2 TRAINING DURATION AND CHECKPOINT EVALUATION

  • Tokenizer checkpoints are selected by validation reconstruction MSE, while policy checkpoints use validation cross-entropy or denoising MSE; training runs for 500 epochs.Source: §A.2 · p. 13

A.3 ACTION COORDINATES AND RECURRENT STATE

  • Actions are normalized per channel; translation channels use cumulative sums, rotations retain per-step values, and recurrent motion and control states form each draft.Source: §A.3 · p. 14

A.4 ENCODER FEATURES AND BOUNDED DECODER UPDATES

  • At each stage, the encoder quantizes a residual feature and the decoder combines its embedding with the current draft to propose bounded updates.Source: §A.4 · p. 14

A.5 EXPANDED TOKENIZER OBJECTIVE

  • The tokenizer objective combines prefix reconstruction, progress penalties against increased motion or control error, and locality and edit regularizers.Source: §A.5 · p. 15

A.6 TOKEN CONSTRUCTION AND DEPLOYMENT DETAILS

  • Training differentiates through recurrent updates with straight-through quantization; deployment predicts content tokens and decodes them without expert actions or residual encoding.Source: §A.6 · p. 15

B.1 TASKS AND ACTION INTERFACES

The appendix specifies task interfaces and evaluation procedures, reports FAST sequence-validity and tokenizer-reconstruction findings, and tests whether matched action coordinates explain ReAT’s LIBERO advantage.

B.1 TASKS AND ACTION INTERFACES

  • The tasks span LIBERO, MetaWorld, and RoboCasa, with benchmark-specific action interfaces and demonstration data described for RoboCasa.Source: §B.1 · pp. 15–16

B.2 OBSERVATION PROCESSING AND RESET HANDLING

  • Methods share benchmark-specific camera and proprioceptive inputs, while short-prefix OAT evaluations reuse its eight-token policy and tokenizer without retraining.Source: §B.2 · p. 16

B.3 STATISTICAL REPORTING

  • Success is averaged across tasks and episodes within each round, then across five round-level means; tables report standard error across those rounds.Source: §B.3 · p. 16

B.4 FAST DECODING VALIDITY ON BUCKET STACKING

  • 87.67% of top-k policy-generated FAST sequences and 83.67% of greedy sequences are invalid, while all 300 expert-action sequences decode successfully.Source: §B.4 · p. 16

C INFERENCE-LATENCY MEASUREMENT PROTOCOL

D PER-DIMENSION TOKENIZER RECONSTRUCTION

E OAT WITH MATCHED ACTION COORDINATES

Related papers

Loading 2610.13093v2…