Source-linked AI summary
StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation
Mingyu Liu, Jiuhe Shu, Hui Chen, Zeju Li, Canyu Zhao, Jiange Yang, Shenyuan Gao, Hao Chen, Chunhua Shen
TL;DR
Robotics needs state representations that are both compact and expressive for efficient world modeling. StaMo learns compact states from static images and uses their differences as latent actions; it reports improved LIBERO and real-world performance with minimal inference overhead.
Problem
Robotics state representations must be compact for efficient future prediction yet expressive enough to encode task-critical information, a balance existing approaches often miss.
Source: §1 · p. 1Method
StaMo learns compact states from images with a Diffusion Autoencoder and treats differences between state encodings as latent actions.
Source: §1 · p. 1Results
+11.6% in LIBERO and +31% in real-world deployments are reported for VLM architectures using StaMo states, with negligible inference-speed overhead.
Source: §1 · p. 1Takeaways & Limitations
Motion can emerge from differences between image-derived states, offering an alternative to learning latent actions explicitly from video sequences.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
A fundamental challenge in embodied intelligence is developing expressive and compact state representations for efficient world modeling and decision making. However, existing methods often fail to achieve this balance, yielding representations that are either overly redundant or lacking in task-critical information. We propose an unsupervised approach that learns a highly compressed two-token state representation using a lightweight encoder and a pre-trained Diffusion Transformer (DiT) decoder, capitalizing on its strong generative prior. Our representation is efficient, interpretable, and integrates seamlessly into existing VLA-based models, improving performance by 14.3% on LIBERO and 30% in real-world task success with minimal inference overhead. More importantly, we find that the difference between these tokens, obtained via latent interpolation, naturally serves as a highly effective latent action, which can be further decoded into executable robot actions. This emergent capability reveals that our representation captures structured dynamics without explicit supervision. We name our method StaMo for its ability to learn generalizable robotic Motion from compact State representation, which is encoded from static images, challenging the prevalent dependence to learning latent action on complex architectures and video data. The resulting latent actions also enhance policy co-training, outperforming prior methods by 10.4% with improved interpretability. Moreover, our approach scales effectively across diverse data sources, including real-world robot data, simulation, and human egocentric video.
1. Introduction
StaMo addresses the need for state representations that are both compact for efficient prediction and expressive enough to retain task-critical information. It encodes static images into compact states, from which motion emerges as a latent difference, and reports improved VLA performance and generalization.
1. Introduction
- Compact state representations must preserve actionable information for efficient future prediction while remaining expressive enough to encode goal states, interactions, and spatial relationships.The introduction contrasts this need with low-dimensional trajectories and flow fields, which capture basic motion but lack semantic richness.Source: §1 · p. 1
- +11.6% in LIBERO and +31% in real-world deployments are reported for VLM architectures using StaMo states, with negligible inference-speed overhead.The representation is learned with a Diffusion Autoencoder finetuned on robotics data and uses a pretrained DiT decoder.Source: §1 · p. 1
- StaMo learns states from individual images, and motion emerges as the difference between start and goal encodings rather than being explicitly modeled from sequences.Linear interpolation between the encodings is reported to produce smooth, plausible, dynamically consistent motion trajectories.Source: §1 · pp. 1–2
- Compared with video-based latent-action learning, the state-centric approach is reported as more training-efficient and less affected by ambiguity from motion variation in video.The passage describes video methods as requiring complex temporal models and often producing ambiguous, coarse-grained actions.Source: §1 · p. 2
- StaMo’s latent actions show strong generalization and sim-to-real transferability, and co-training experiments report superior performance over traditional and more complex latent-action approaches.The framework is also presented as an intermediate representation for world modeling and bridging vision-language models with action experts.Source: §1 · pp. 2–3
2. Related Works
Related work spans representations that trade expressive state information against compactness, VLA world modeling, and latent actions learned from data. StaMo is presented as combining compact, expressive states with motion represented by differences between them.
2.1. Robot Representation Learning
- StaMo is presented as balancing compactness and expressiveness, unlike motion representations that lack rich state information or detailed image encodings that are too redundant for motion-by-difference.The cited Figure 2 visualizes representations along the expressiveness–compactness spectrum.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2. World Modeling in VLA
- Prior VLA world-modeling approaches either decode full images at inference or use redundant states, while StaMo is described as compact, faster at inference, and adaptable to unseen scenes without encoder fine-tuning.The passage identifies works [12] [27] [45] [55] as exploring image-based world modeling.Source: §2.2 · p. 3
2.3. Latent Action Learning
- Latent actions learned from unlabeled data have been used in downstream tasks, but many methods rely on designed architectures and continuous video frames, bringing sampling sensitivity, temporal bias, and blurry or poorly interpretable actions.The passage cites discrete approaches [9] [10] [11] [15] [41] [53] and continuous approaches [20] [52].Source: §2.3 · p. 3
- StaMo instead proposes that actionable representations can emerge from natural-image collections, with the paper reporting better interpretability and generalization for these image-emergent actions.This challenges the conventional dependence on video sequences for learning latent actions.Source: §2.3 · p. 3
3. Method
StaMo compresses images into compact states that also encode motion through differences between states. It uses these representations for future-state prediction and for co-training policies with action-labeled robot data and action-less video.
3.1. Compress Image into Compact State
- StaMo compresses observations into as few as two 1024-dimensional tokens using a frozen DINOv2 feature extractor, transformer compressor, and DiT decoder.The decoder reconstructs the original image observations.Source: §3.1 · p. 4
3.2. Interpolating Motion from States
- Motion is the difference between consecutive compact state tokens, combining a representation of scene content with a compact description of dynamics.The paper contrasts this with compact motion representations that lack rich state context and expressive state representations that are computationally burdensome and do not intrinsically encode motion.Source: §3.2 · p. 4
3.3. StaMo for Efficient World Modeling
- StaMo equips a VLA with a lightweight MLP head to predict future states alongside actions, using a composite objective for both tasks.The authors report that future-state prediction improves task success rates.Source: §3.3 · pp. 4–5
3.4. StaMo for Latent Motions Co-Training
- For co-training, StaMo state differences between consecutive video frames provide pseudo-action labels that can be learned jointly with ground-truth robot actions.The authors report substantial gains over prior methods and describe the emergent motions as robust proxies for actions.Source: §3.4 · p. 5
4. Experiments
StaMo reconstructs robotic manipulation images with high quality on in-domain and out-of-domain data. Its latent-space interpolation produces continuous plausible motion, and token differences transfer across simulation and real-world settings.
4.1. Qualitative Analysis of StaMo
- StaMo produces high-quality reconstructions on both in-domain and out-of-domain data, with Figure 3 showing ground-truth and predicted images alongside PSNR and SSIM.The quantitative reconstruction comparison is reported in Table 1.Source: §4.1 · p. 5, Figure 3 · p. 5, Table 1 · p. 5
- Latent-space interpolation yields decoded images with continuity and plausible motion, while token-difference motions transfer across sim-to-sim, sim-to-real, and real-to-sim settings.The reported transfer results indicate that learned motion is not scene-specific.Source: §4.1 · p. 5
4.2. World Modeling Results
On LIBERO, StaMo representations support world modeling in OpenVLA and OpenVLA-OFT, with the preferred predictive target depending on action horizon. StaMo also adds less inference overhead than other world-modeling methods.
4.2. World Modeling Results
- The LIBERO evaluation retrains OpenVLA and OpenVLA-OFT to predict StaMo states; OpenVLA-OFT* uses only the third-person camera and omits the wrist camera.Each task is evaluated over 1,000 rollouts, with results reported as percentage success rates.Source: §4.2 · pp. 5–6
- For short-horizon OpenVLA, motion representation performs better; for longer-horizon OpenVLA-OFT, the state representation is recommended.The authors relate motion to incremental delta end-effector pose targets and state to stable goal conditioning.Source: §4.2 · p. 6, Table 2 · p. 6
4.3. Inference Speed Comparison
- StaMo introduces significantly less inference overhead because models predict compact tokens rather than decoding full images.The inference-frequency comparison is presented in Table 3, which is not among the supplied passages.Source: §4.3 · p. 6
4.4. Zero-Shot Transfer to New Environment
The section reports StaMo’s performance in zero-shot transfer to ManiSkill and policy co-training, alongside its inference-speed comparison with other world-model methods.
4.4. Zero-Shot Transfer to New Environment
- StaMo introduces minimal computational overhead and achieves higher inference frequency than previous methods, including direct use of DINOv2 features.The comparison deployed all models on a single RTX 4090 GPU.Source: Table 3 · p. 6
- StaMo achieves stronger performance than competing models when LIBERO-trained models are transferred directly to ManiSkill in a zero-shot setting.Each task was evaluated over 100 trials.Source: §4.4 · p. 6, Table 4 · p. 6
4.5. Policy Co-training Results
- StaMo significantly outperforms LAPA and ATM in policy co-training using per-task data with StaMo-generated pseudo-actions.The setup used 10 robot trajectories and 40 video trajectories per task; success rates were averaged across three runs of 20 trials per task.Source: §4.5 · p. 6, Table 5 · p. 6
4.6. Action Linear Probing Results
The action linear-probing evaluation tests whether differences between StaMo state latents encode predictable robot motion. StaMo has the lowest reported MSE across the evaluated horizons and baselines.
4.6. Action Linear Probing Results
- The protocol encodes start and goal images from sampled robot-trajectory tuples into a latent action, then trains an MLP to predict the intervening ground-truth action sequence.The StaMo encoder is frozen, and prediction error is measured by MSE.Source: §4.6 · p. 7
- A low LP-MSE score supports that differences between StaMo latent states contain informative, linearly separable information about robotic motion.After training, the linear head can map current and goal observations to an executable action trajectory.Source: §4.6 · p. 7
- StaMo achieves the lowest linear-probing MSE across all evaluated horizons, outperforming the compared baselines.The baselines include pixel-wise image differences, DINOv2 feature differences, and LAPA; nongenerative methods were evaluated at 1, 2, 4, and 8 steps, while LAPA used its native 1-step horizon.Source: Figure 4 · p. 7, §4.6 · p. 7
4.7. Goal-Conditioned Task Planning
The goal-conditioned task-planning comparison evaluates StaMo and DINO-WM on ManiSkill, which was absent from both models’ pretraining datasets.
4.7. Goal-Conditioned Task Planning
- The ManiSkill evaluation compares goal-conditioned task-planning performance for StaMo and DINO-WM, with results reported as success rates over 100 trials per task.Only the linear head was trained; both encoders remained frozen.Source: §4.7 · p. 7, Table 6 · p. 7
4.8. Real World Experiments
The real-world benchmark evaluates StaMo’s role in world modeling and decision-making across six manipulation tasks.
4.8. Real World Experiments
- The benchmark covers three short-horizon and three long-horizon tasks, using 50 human demonstrations per task and evaluating performance over 20 trials.Results are reported as percentage success rates in Table 7.Source: §4.8 · p. 7
4.9. Ablation Study
The ablations examine how latent dimension, training-data diversity, and decoder pretraining relate to StaMo’s reconstruction and real-world task performance. More diverse training data steadily improves performance, while the pretrained decoder provides an advantage that training from scratch does not match.
4.9.1. Ablation of Token Dimension
- Across 256, 512, and 1024 hidden dimensions, dimension has limited impact on reconstruction performance with two latent tokens.The authors suggest the pretrained DiT decoder’s generative prior offsets potential information loss at lower dimensions; reconstruction is evaluated with PSNR and SSIM.Source: §4.9.1 · p. 7, Table 8 · p. 8
4.9.2. Ablation of Additional Training Data
- Adding DROID, OXE, and egocentric human data to simulation data steadily improves StaMo’s real-world task performance.The comparison covers simulation only, simulation plus DROID, then OXE, and finally egocentric data.Source: §4.9.2 · p. 8, Figure 6 · p. 8
4.9.3. Ablation of Pretrained Diffusion Decoder
- The pretrained diffusion decoder converges faster and achieves performance that a decoder trained from scratch does not match.The authors attribute the advantage to pretraining on large-scale natural-image datasets, which supplies a prior beneficial for representing robotics data.Source: §4.9.3 · p. 8, Table 9 · p. 8
5. Conclusion
StaMo extracts reusable latent action vectors from static images using a transformer-based tokenizer and diffusion decoder. Linear interpolation in the latent space yields smooth, physically plausible motions, and the vectors generalize zero-shot to unseen scenarios.
5. Conclusion
- StaMo extracts reusable latent action vectors from static images, and linear interpolation in their latent space produces smooth, physically plausible motions that generalize zero-shot to unseen scenarios.The authors suggest this points to large-scale visual models implicitly learning a linearized dynamics manifold and offers a pathway toward unsupervised skill discovery from diverse visual data.Source: §5 · p. 8