PaperlayerView exact revision on arXiv

Source-linked AI summary

Mimic Intent, Not Just Trajectories

Renming Huang, Chendong Zeng, Wenjing Tang, Jintian Cai, Cewu Lu, Panpan Cai

arXiv:2602.08602v3cs.RO

TL;DR

Imitation-learning policies, including VLAs, often struggle with environmental variation and skill transfer because they mimic raw trajectories without explicitly modeling intent. MINT separates intent from execution through multi-scale spectral tokenization and reports stronger benchmark robustness, one-shot transfer, and real-robot performance.

  • Problem

    Current VLA policies often generalize poorly to environmental variations and new task instances, while raw-trajectory imitation does not explicitly model underlying intent.

  • Method

    MINT uses a spectrally disentangled multi-scale tokenizer and coarse-to-fine autoregressive policy to separate low-frequency intent from high-frequency execution details.

  • Results

    MINT achieves state-of-the-art benchmark performance, 15% higher success under stronger disturbances, 60% higher one-shot transfer performance, and 29% higher real-robot performance than π0.5.

  • Takeaways & Limitations

    The intent representation supports robust adaptation and one-shot transfer across new layouts, tasks, environments, and physical systems.

Abstract

from arXiv · show

While imitation learning (IL) has achieved impressive success in dexterous manipulation through generative modeling and pretraining, state-of-the-art approaches like Vision-Language-Action (VLA) models still struggle with adaptation to environmental changes and skill transfer. We argue this stems from mimicking raw trajectories without understanding the underlying intent. To address this, we propose explicitly disentangling behavior intent from execution details in end-2-end IL: Mimic Intent, Not just Trajectories(MINT). We achieve this via multi-scale frequency-space tokenization, which enforces a spectral decomposition of action chunk representation. We learn action tokens with a multi-scale coarse-to-fine structure, and force the coarsest token to capture low-frequency global structure and finer tokens to encode high-frequency details. This yields an abstract Intent token that facilitates planning and transfer, and multi-scale Execution tokens that enable precise adaptation to environmental dynamics. Building on this hierarchy, our policy generates trajectories through next-scale autoregression, performing progressive intent-to-execution reasoning, thus boosting learning efficiency and generalization. Crucially, this disentanglement enables one-shot transfer of skills, by simply injecting the Intent token from a demonstration into the autoregressive generation process. Experiments on several manipulation benchmarks and on a real robot demonstrate state-of-the-art success rates, superior inference efficiency, robust generalization against disturbances, and effective one-shot transfer.

I. INTRODUCTION

MINT addresses poor adaptation and skill transfer by separating behavioral intent from execution details through frequency-space action tokenization. Its coarse-to-fine intent representation supports progressive generation, robustness, and one-shot transfer.

  • Current VLA and imitation-learning policies often generalize poorly to environmental variations and new task instances because they mimic raw trajectories without modeling intent.
  • Existing action-tokenization methods commonly compress trajectories without explicitly aligning tokens with interpretable behavioral concepts such as intent.
  • MINT uses multi-scale frequency-space tokenization to separate low-frequency global behavior structure from high-frequency execution details.
  • Progressive prediction from the Intent token to Execution tokens supports intent-to-execution reasoning, sample efficiency, and stable long-horizon generation.
  • 15% higher success rates under stronger LIBERO-Plus disturbances and 60% higher one-shot transfer performance demonstrate improved robustness and transfer over reported baselines.Real-robot experiments used around 20 demonstrations per task and outperformed π0.5 by 29%.

III. OVERVIEW

MINT is a two-stage framework comprising a spectrally disentangled tokenizer and a policy that predicts action tokens progressively from coarse to fine. The tokenizer structures demonstrations into intent and execution representations that the policy decodes into executable trajectories.

  • MINT first trains SDAT to learn structured multi-scale action representations, then trains a policy to generate SDAT tokens through progressive intent-to-execution reasoning.
  • SDAT segments demonstrations into overlapping chunks, transforms them with the DCT, and learns a discrete action codebook using a VQ-VAE architecture.
  • The coarsest single token represents global low-frequency structure, while finer Execution tokens model residual information through progressively increasing capacity.
  • Progressive frequency-domain reconstruction constrains coarse tokens to explain dominant low-frequency components and finer tokens to specialize in high-frequency residuals.
  • The policy conditions action-token prediction on visual observations, language instructions, and robot proprioceptive state, then decodes the tokens into action trajectories.

B. Multi-Scale Residual Quantization

SDAT recursively quantizes latent action features across increasing resolutions, forming a coarse-to-fine multi-scale token representation. Scale-wise spectral reconstruction assigns global structure to early scales and detail to later residual scales.

  • SDAT decomposes an encoded action embedding into K discrete token maps at increasing resolutions using shared-codebook multi-scale residual quantization.
  • At each scale, residual features are interpolated, quantized, embedded through the shared codebook, projected to the original resolution, and subtracted from the residual.
  • The tokenizer initializes encoded action features and progressively accumulates quantized residuals across scales before returning tokens, spectra, and reconstructions.
  • Each cumulative representation is decoded into a progressively refined action sequence and transformed into the frequency domain for comparison with the ground-truth spectrum.
  • The scale-wise spectral objective encourages early scales to capture low-frequency global structure while later scales model high-frequency details.

D. Training Objective

The training objective combines scale-wise spectral reconstruction with codebook, commitment, and auxiliary action-reconstruction losses. The formulation also uses stop-gradient and a weighting factor.

  • SDAT trains with scale-wise spectral reconstruction, codebook, commitment, and auxiliary l1 reconstruction losses.The objective prioritizes spectral reconstruction while regularizing discrete codebook learning and action reconstruction.
  • The loss includes separate codebook and commitment terms for learning and stabilizing the discrete action representation.
  • An auxiliary l1 reconstruction term supplements the frequency-domain objective.
  • The formulation uses stop-gradient and a weighting factor α.

V. MINT POLICY LEARNING

MINT predicts multi-scale action token maps autoregressively from coarse to fine, then combines overlapping action predictions using intent similarity. This supports smooth execution and long-horizon generation.

  • Next-Scale Autoregressive Modeling: MINT predicts action tokens across scales autoregressively, using coarser token maps as prefixes for finer-scale prediction.Each scale is modeled as a token map, and predictions within a scale are generated in parallel.
  • Next-Scale Autoregressive Modeling: A hybrid attention mask restricts scale k to attend to token maps from scales s≤k during training.The policy uses cross-entropy between predicted and ground-truth token maps.
  • Intent-Based Action Ensemble: At time t, overlapping predictions for the same action are combined into a final executed action.The predicted action at time t is conditioned on the corresponding observation and neighboring observation histories.
  • Intent-Based Action Ensemble: Ensemble weights adapt to cosine similarity between the current intent token and historical intent tokens.The temperature β controls how strongly intent similarity affects weight assignment.
  • Intent-Based Action Ensemble: Intent-based action ensembling improves action stability and long-horizon task success in empirical studies.

C. Model Architectures

MINT is instantiated in lightweight and large-scale architectures, combining different pretrained or frozen perceptual-language components with an action expert. Evaluation covers benchmarks, disturbances, real-world settings, and one-shot transfer.

  • Model Variants: MINT-30M is a lightweight decoder-only Transformer trained from scratch with approximately 30M trainable parameters.It uses frozen SigLIP, DINOv2, and BERT encoders, with language and visual features injected through FiLM layers.
  • Model Variants: MINT-4B combines a PaliGemma-2.6B vision-language model, a SigLIP visual encoder, and an approximately 300M-parameter action expert.The action expert is trained from scratch and predicts multi-scale action tokens autoregressively.
  • Evaluation: The framework is evaluated on standard manipulation benchmarks and LIBERO-Plus under severe disturbances.
  • Evaluation: Experiments also cover real-world environments and one-shot skill transfer in simulation.

A. Performance Comparison

MINT consistently matches or surpasses state-of-the-art methods across standard manipulation benchmarks, under LIBERO-Plus distribution shifts, in one-shot transfer, and on real-world tasks. Its reported results include improved robustness to disturbances, effective transfer across novel task conditions, and strong performance on high-precision manipulation.

  • Standard Benchmarks: MINT consistently matches or surpasses current state-of-the-art approaches across LIBERO, CALVIN, and MetaWorld benchmarks.The comparison includes pretrained VLA models, action-tokenization methods, and classic imitation learning approaches.
  • Robustness and Generalization: MINT-30M and MINT-4B outperform prior baselines under LIBERO-Plus camera-viewpoint and robot-initialization shifts while maintaining stable performance under scene and sensor perturbations.The benchmark evaluates seven perturbation dimensions, including viewpoints, robot states, lighting, backgrounds, sensor noise, layouts, and language instructions.
  • One-Shot Transfer: Intent-based specification enables effective one-shot transfer across unseen tasks, novel layouts, and extended-horizon sequences, whereas language-based one-shot finetuning is limited.MINT-30M can diverge on new layouts and fail to capture the required behaviors for extended-horizon tasks, while MINT-Zero-30M conditions on an intent token extracted from one demonstration.
  • Real-World Experiments: MINT-4B significantly outperforms baselines on four real-world tasks, including an unseen cup-stacking task requiring high-precision alignment and geometric re-orientation.The real-world evaluation covers three seen behaviors and one unseen behavior, with Bayesian posterior analysis used for statistical comparison.

E. Ablation Studies

Ablations show that spectral decomposition and intent-based ensembling are central to MINT’s performance, while the learned intent space forms coherent behavioral clusters.

  • Efficacy of Scale-Wise Spectral Decomposition: 93.4% on LIBERO-Long and 4.54 length on CALVIN were achieved with Scale-Wise Spectral Loss, outperforming scale-wise time-domain constraints.The time-domain constraints reached 82.8% and were reported to overfit high-frequency noise.
  • Efficacy of Scale-Wise Spectral Decomposition: MINT’s spectral objective produces coherent latent-space behavioral clusters, unlike fragmented standard time-domain reconstruction.Figure 6 visualizes clusters using t-SNE of action chunks colored by S1 tokens.
  • Impact of Intent-based Action Ensemble: The Intent-Based Action Ensemble achieved 93.2% success on LIBERO-Long and 4.57 average sequence length on CALVIN.It dynamically modulates aggregation weights based on intent compatibility and outperformed Temporal and Action-based baselines.
  • Limitations and Future Work: The conclusion identifies dataset-limited intent diversity as a limitation and proposes broader network data and recombined intent tokens as future directions.These directions are presented as ways to broaden task coverage and synthesize novel long-horizon behaviors.

APPENDIX

The appendix details MINT’s training configurations, implementation variants, and model components, including lightweight and large-scale architectures.

  • Action Tokenizer: The action encoder and spectrum decoder use 1D CNNs, with separate projections and early Group CNN processing for translation, rotation, and gripper states.Exponential Moving Average updates are used for codebook stability and to prevent codebook collapse.
  • MINT-30M: MINT-30M is a 30M-parameter architecture trained from scratch without a VLM backbone.It uses BERT for language, frozen visual encoders, and FiLM conditioning for language-controlled behavior.
  • MINT-4B: MINT-4B uses a PaliGemma-2.6B vision-language model, a SigLIP visual encoder, and an approximately 300M-parameter action expert.The action expert performs next-scale autoregressive prediction over multi-scale action tokens.
  • Training Recipes: Training recipes for SDAT and both MINT model variants are documented in Tables V and VI.The tables specify benchmark-specific SDAT recipes and recipes for MINT-30M and MINT-4B.

4) Hyperparameter Details:

The appendix describes model reproducibility details and evaluation setups spanning simulated, perturbed, and real-world manipulation benchmarks.

  • Hyperparameter Details: The visual encoders contain 400 million parameters for SigLIP and 300 million parameters for DINOv2.These parameter counts are provided to support reproducibility.
  • Reported Analyses: Tables VII and VIII report learning-efficiency comparisons and ablations over spectral scales and action-chunk horizons.Table IX reports performance comparisons across LIBERO, CALVIN, and MetaWorld.
  • LIBERO Benchmark: LIBERO evaluation uses 270k combined training samples and evaluates five-task sequences on split D across 500 rollouts.Reported metrics include full-sequence success rate and average successfully completed tasks per episode.
  • LIBERO-Plus and Real-World Evaluation: LIBERO-Plus evaluates robustness under perturbations including initial pose, language, lighting, background texture, and sensor noise.The benchmark reports binary success per episode.
  • Real-World Benchmark: The real-world benchmark uses a 6-DoF Piper-X arm and includes placement, stacking, and insertion tasks with varied object conditions.For the first three tasks, 20 demonstrations per task produce 5.4K samples recorded at 10 Hz over 90-frame horizons.
  • Real-World Results: MINT maintains stable control under distribution shifts and successfully completes a zero-shot cup-stacking task without additional demonstrations.The real-world examples are presented as qualitative validation of generalization and robustness.

C. More Ablation Studies

Additional analyses show faster convergence, a trade-off between model expressiveness and stability, improved reconstruction analysis, stable intent clustering, and broad benchmark strength.

  • Learning Efficiency: MINT converges significantly faster than baseline methods, and MINT-30M gains performance rapidly without pre-training.Learning efficiency is measured through success rates at different training iterations.
  • Ablations: Moderate numbers of spectral scales and intermediate action horizons provide the best trade-off between expressiveness, optimization stability, planning, and prediction accuracy.Excessively long horizons degrade performance because they increase modeling difficulty.
  • Reconstruction Analysis: Reconstruction analysis evaluates how spectral-scale count affects fidelity and compares reconstructed trajectories with ground-truth actions.The analysis combines reconstruction-error measurements with qualitative trajectory visualization.
  • Intent Token Structure: S1 intent tokens form stable clusters for behaviors such as object pickup, forward motion, and rotational manipulation across LIBERO and CALVIN.The clusters are reported to reflect task-level abstractions rather than dataset-specific artifacts.
  • Benchmark Results: Full benchmark results show consistently strong performance across LIBERO, CALVIN, and MetaWorld, especially in long-horizon and multi-task settings.The comparison includes both pre-trained and non-pre-trained baselines.
  • LIBERO-Plus Analysis: LIBERO-Plus results are additionally broken down across LIBERO suites and perturbation types for finer-grained comparison.The expanded baseline table enables comparisons under identical perturbation settings.

E. Statement on the Use of Large Language Models

The manuscript reports limited LLM assistance for improving written expression, while the authors exclusively conceived, developed, and validated the research.

  • LLM assistance was limited to grammar, sentence flow, and clarity improvements.
  • The authors conceived and developed the underlying research concepts.
  • The authors exclusively validated the research conclusions.
Loading 2602.08602v3…