Source-linked AI summary
Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models
Yuxin Yang, Gaohan He, Changxue Guan, Hangming Liu
TL;DR
Autoregressive VLA control needs action tokens that preserve motion while remaining predictable, yet tokenizers are often evaluated mainly by reconstruction fidelity. The paper compares analytical, linear, and nonlinear representations under a unified interface across rate-distortion, sequence diagnostics, and closed-loop rollouts. Rankings vary by criterion: Temporal-DCT leads PCA in mean seen-task success despite higher nominal reconstruction error, while the autoencoder’s lowest reconstruction error does not make it the strongest policy.
Problem
Action-tokenizer evaluation often emphasizes compression or reconstruction quality, although autoregressive control also requires predictable tokens and decoded actions that do not amplify discrete prediction errors.
Source: §1 · p. 1Method
The study compares Temporal-DCT, PCA, and an MLP autoencoder under a matched tokenization interface and evaluates rate-distortion, sequence predictability, quantization sensitivity, and closed-loop performance.
Source: §1 · p. 1Results
Temporal-DCT achieves 86.2% mean seen-task success versus PCA’s 83.2% across three seeds, a 3.0-pp advantage whose ordering reverses in one seed.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
Reconstruction fidelity alone cannot select a policy representation; evaluation should separate geometric fidelity, sequence predictability, and decoder stability.
Source: §6 · p. 4The study uses one VLA backbone, three training seeds, and fixed K = 14 scalar quantization; the observed ordering should not be assumed to transfer unchanged.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Discrete action tokenization is central to autoregressive vision-language-action (VLA) models, yet action representations are often evaluated primarily through reconstruction fidelity. We ask which representation properties actually matter for closed-loop control by comparing fixed analytical, data-driven linear, and nonlinear neural representations under a unified tokenization interface. Across rate-distortion analysis, sequence-modeling diagnostics, and 3,500 LIBERO rollouts, representation rankings change with the evaluation criterion. PCA achieves lower nominal reconstruction error than Temporal-DCT, but produces less predictable token sequences and 3.0 percentage points lower mean seen-task success across three policy-training seeds, with the policy ordering reversing in one seed. In a matched seed-42 ablation, an autoencoder further reduces reconstruction error yet does not yield the strongest policy and exhibits greater sensitivity to discrete token perturbations. These findings show that reconstruction fidelity alone cannot reliably select action representations for autoregressive control, motivating joint evaluation of geometric fidelity, sequence predictability, decoder stability, and closed-loop performance.
1. INTRODUCTION
The paper asks which action-representation properties matter for autoregressive control beyond reconstruction fidelity. It compares three representation families under matched evaluation and finds that representation rankings depend on the criterion: lower reconstruction error does not guarantee stronger closed-loop performance.
1. INTRODUCTION
- Action tokenizers must preserve task-relevant motion while producing compact, predictable sequences, because their representation can affect closed-loop execution beyond compression efficiency.The paper frames tokenization as an interface coupling geometric representation, autoregressive sequence modeling, and error propagation into physical actions.Source: §1 · p. 1
- The study compares fixed Temporal-DCT, data-driven PCA, and a nonlinear MLP autoencoder under matched tokenization and policy-evaluation protocols.It examines rate-distortion behavior, sequence predictability, quantization sensitivity, and closed-loop performance.Source: §1 · p. 1
- Temporal-DCT exceeds PCA by 3.0 pp in mean seen-task success across three seeds despite higher nominal reconstruction error, while the ordering reverses in one seed.In the matched ablation, the autoencoder has the lowest reconstruction error but not the highest success rate, and its larger perturbation gain identifies decoder stability as another property hidden by reconstruction MSE.Source: §1 · pp. 1–2, Figure 1 · p. 2
2. BASIS ACTION TOKENIZATION PIPELINE
The pipeline converts continuous action chunks into discrete tokens using analytical, data-driven linear, or nonlinear representations. It standardizes chunks, projects them into a compact representation, quantizes the coefficients, and reconstructs action commands from the tokens.
2.1. Trajectory Chunking and Normalization
- Chunks contain 16 steps across 7 action dimensions, are flattened to 112 values, and are standardized using training-task statistics.Pose commands are scaled using training bounds, gripper commands remain in [0, 1], and normalizers are fitted only on Tasks 0–8.Source: §2.1 · p. 2
2.2. Analytical, Linear, and Neural Tokenizers
- Temporal-DCT applies independent temporal transforms per actuator, PCA captures empirical cross-axis correlations, and the MLP autoencoder learns a nonlinear 14D bottleneck.DCT retains frequency components in priority order across channels; PCA uses leading covariance eigenvectors, while the autoencoder is a reconstruction-optimized baseline.Source: §2.2 · p. 2
2.3. Symmetric Uniform Quantization
- Projection coefficients are standardized, clipped to [−5, +5], and uniformly quantized into V = 256 discrete token values at the nominal 8-bit setting.The token budget is K coefficients; dequantized coefficients are projected back for linear bases or decoded through the autoencoder, then actions are de-standardized.Source: §2.3 · pp. 2–3
3. OFFLINE REPRESENTATION ANALYSIS
The offline analyses show that representation rankings depend on quantization and transfer setting: PCA often improves nominal reconstruction, while Temporal-DCT is more robust at coarse precision and yields more predictable sequences.
3.1. Rate–Distortion and Quantization Sensitivity
- At K = 14 and 8-bit quantization, PCA reduces reconstruction MSE by 8.3% versus Temporal-DCT (0.0083 vs. 0.0091), but Temporal-DCT is consistently better under 2-bit quantization.PCA outperforms Temporal-DCT in 14 of 20 evaluated configurations; the advantage reverses under extreme 2-bit quantization.Source: §3.1 · p. 3
3.2. Cross-Task and Cross-Dataset Generalization
- In leave-one-task-out LIBERO-Spatial evaluation, PCA wins 6 of 10 held-out tasks and reduces macro-average MSE by 6.2%.The PCA basis is refitted on the other nine tasks in each offline fold; this differs from the downstream policy protocol, which excludes Task 9 from fitting and training.Source: §3.2 · p. 3
- On LIBERO-Goal, Temporal-DCT has 22.9–42.2% lower MSE at K ∈ {7, 14}, while PCA improves MSE by up to 28.3% at K = 28.On LIBERO-Object, neither basis consistently dominates.Source: §3.2 · p. 3
3.3. Sequence Predictability
- Despite higher nominal reconstruction error, Temporal-DCT lowers position-wise entropy by 0.75 bits and perplexity by 12.5–20.7%; seed-42 Task 0 teacher-forced policy loss is 3.419 vs. 3.513 nats/token.The table covers LIBERO-Spatial at K = 14 and V = 256; policy loss is teacher-forced on seed-42 Task 0.Source: §3.3 · pp. 3–4, Table 3 · p. 3
4. DOWNSTREAM ROBOTIC POLICY EVALUATION
In matched LIBERO-Spatial policy evaluations, Temporal-DCT has higher mean seen-task success than PCA despite worse nominal reconstruction, while the autoencoder minimizes reconstruction error without leading control and shows greater perturbation sensitivity.
4.1. Evaluation Protocol
- The evaluation uses 3,500 LIBERO-Spatial rollouts with Qwen3-VL-4B, 50 initial states per task and seed, and seven runs.Task 9 is excluded from fitting; policies predict K = 14 tokens and decode T = 16 actions.Source: §4.1 · p. 4
4.2. Closed-Loop Control
- Temporal-DCT reaches 86.2% mean seen-task success versus PCA’s 83.2%, a 3.0-pp aggregate advantage, but the ordering reverses for seed 44.DCT leads by 6.2 and 5.3 pp in seeds 42 and 43; PCA leads by 2.7 pp in seed 44. With three independent training seeds, the cross-seed mean gap is descriptive, not an inferential claim over optimization randomness.Source: §4.2 · p. 4
4.3. Reconstruction–Control Misalignment
- The autoencoder reduces post-quantization MSE by 36.7% versus PCA and 41.9% versus Temporal-DCT, yet its 82.7% seen-task success lies between PCA (80.0%) and DCT (86.2%).The reconstruction ranking AE>PCA>DCT does not match the control ranking DCT>AE>PCA.Source: §4.3 · p. 4
- At |δzj| = 4, the autoencoder’s perturbation gain is 0.0431 versus 0.0336 for PCA/DCT, with 26.5–30.3% larger mean amplification than the linear bases.The diagnostic perturbs token coordinates, clips to vocabulary range, and averages across chunks, coordinates, and both signs.Source: §4.3 · p. 4, Table 4 · p. 4
4.4. Synthesis of Representation Trade-offs
- Across the evaluated representations, reconstruction quality does not track control rankings, so tokenizer evaluation should separate geometric fidelity, sequence predictability, and decoder stability.PCA has lower nominal reconstruction error than DCT, while DCT yields more predictable sequences and higher mean seen-task success; the autoencoder minimizes reconstruction MSE but has the highest perturbation gain.Source: §4.4 · p. 4, §4.2 · p. 4, §4.3 · p. 4
5. LIMITATIONS
The study’s findings are bounded by its evaluation setting and unresolved transfer questions.
5. LIMITATIONS
- The study evaluates LIBERO-Spatial with one VLA backbone, three training seeds, and fixed K = 14 scalar quantization; its observed ordering should not be assumed to transfer unchanged.Morphological diversity, policy scale, learned codebooks, and BPE compression remain open; perturbation gain is local rather than causal, and the Task 9 floor precludes held-out conclusions.Source: §5 · p. 4
6. CONCLUSION
In this controlled autoregressive VLA study, reconstruction rankings diverge from control rankings.
6. CONCLUSION
- DCT leads PCA by 3.0 pp in mean seen-task success despite worse reconstruction, while the autoencoder minimizes distortion without leading control.Reconstruction alone therefore cannot select a policy representation; the study calls for sequence-aware diagnostics and paired closed-loop evaluation.Source: §6 · p. 4