Source-linked AI summary
CARE: Multi-Task Pretraining for Latent Continuous Action Representation in Robot Control
Jiaqi Shi, Xulong Zhang, Xiaoyang Qu, Jianzong Wang
TL;DR
Action supervision constrains VLA development, while latent-action models face issues including weak action encoding and shortcut learning. CARE uses multi-task VLM pretraining with keypoint prediction, then shows stronger simulation success than prior label-free approaches while remaining competitive with action-label-based methods.
Problem
VLA development is constrained by the cost and time required for large-scale action supervision, and latent-action models face bias propagation, weak action encoding, and shortcut-learning risks.
Source: §1 · p. 1Method
CARE integrates latent-action training into VLM pretraining and uses frame prediction and keypoint trajectory prediction as multiple objectives.
Source: §1 · p. 1Results
CARE achieves higher success rates than other label-free pretraining approaches across four tasks while remaining competitive with action-label-based methods.
Source: §1 · p. 2, §3.2 · p. 3Takeaways & Limitations
CARE produces latent actions that more effectively encode motions and shows improved explicit action representation with reduced shortcut learning.
Source: §1 · p. 1, §4 · p. 4A performance gap remains compared with action-label-based pretraining.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Recent advances in Vision-Language-Action (VLA) models have shown promise for robot control, but their dependence on action supervision limits scalability and generalization. To address this challenge, we introduce CARE, a novel framework designed to train VLA models for robotic task execution. Unlike existing methods that depend on action annotations during pretraining, CARE eliminates the need for explicit action labels by leveraging only video-text pairs. These weakly aligned data sources enable the model to learn continuous latent action representations through a newly designed multi-task pretraining objective. During fine-tuning, a small set of labeled data is used to train the action head for control. Experimental results across various simulation tasks demonstrate CARE's superior success rate, semantic interpretability, and ability to avoid shortcut learning. These results underscore CARE's scalability, interpretability, and effectiveness in robotic control with weak supervision.
1. INTRODUCTION
CARE addresses the cost of action supervision by introducing label-free VLM pretraining with multiple objectives, including keypoint trajectory prediction. Simulations show advantages over prior label-free approaches while remaining competitive with action-label-based pretraining.
1. INTRODUCTION
- Action supervision is costly and time-consuming, while latent-action models also face bias propagation, weak action encoding, and shortcut-learning risks.Precise annotations such as joint angles or end-effector trajectories can be impractical to obtain at scale.Source: §1 · p. 1
- CARE integrates latent action model training into VLM pretraining, reducing the VLA training process from four stages to three.The framework uses instruction-image pairs during pretraining rather than action labels.Source: §1 · p. 1, Figure 1 · p. 2
- CARE uses multiple decoders and keypoint trajectory prediction to strengthen explicit latent-action encoding and mitigate shortcut learning.The trajectory objective emphasizes action-driven positional changes alongside next-frame reconstruction.Source: §1 · p. 1
- CARE provides significant advantages over prior label-free approaches in simulations while remaining competitive with action-label-based methods.The reported comparison includes higher success rates than other pretraining approaches without action labels.Source: §1 · pp. 1–2
2. METHODOLOGY
CARE first pretrains a latent VLM without action labels, then fine-tunes it with a small labeled robot dataset. Its multi-task objective predicts both next-frame visual features and point trajectories.
2. METHODOLOGY
- CARE separates unsupervised latent-VLM pretraining from supervised fine-tuning on a small robot dataset.The two stages are latent VLM pretraining and action-head fine-tuning.Source: §2 · p. 2
2.1. Latent Vision-Language Model
- The latent-action prediction task takes an observation image and natural-language instruction as input, with action placeholders appended to the prompt.The model uses the hidden states corresponding to the action dimensions as its continuous latent action representation.Source: §2.1 · p. 2
2.2. Continuous Latent Action with Multi-task Learning
- CARE jointly trains latent actions to predict next-frame visual features and the next-frame coordinates of 256 evenly distributed points tracked by Co-Tracker.The model fuses current-frame features and latent actions for frame prediction, and current point coordinates and latent actions for point prediction.Source: §2.2 · p. 2
- An uncertainty-weighted loss combines the visual-feature and point-coordinate prediction objectives, using task-independent uncertainty.The model treats point coordinates with a Gaussian likelihood and visual features with a softmax-based likelihood.Source: §2.2 · p. 3
2.3. Fine-tuning VLA with action labels
- A lightweight residual MLP action head maps latent representations to real-valued robot actions, trained with L1 regression on labeled demonstrations.The model is adapted with LoRA fine-tuning.Source: §2.3 · p. 3
3. EXPERIMENT
The experiments evaluate CARE’s task performance, latent-action interpretability, shortcut learning, and multi-task pretraining using LIBERO and action-labeled fine-tuning data. Results compare CARE with label-free and action-label-based approaches and test the contributions of its pretraining objectives.
3.1. Setups
- Pretraining uses about 240k robot and human video trajectories or clips, followed by fine-tuning on a 3% uniform sample of RT-1; evaluation covers four 10-task LIBERO suites.The experiments assess success rate, linear-probe LP-MSE, semantic accuracy, and S-PCFC shortcut learning.Source: §3.1 · p. 3
3.2. Main Results
- Against autoregressive LAPA and CoMo, CARE leads across all four LIBERO suites and remains competitive with action-label-based methods, exceeding OpenVLA by 1.4% in Goal and 12.5% in Long.Compared with diffusion-policy CoMo, CARE is lower on Goal and Object but higher on Spatial and Long; success rates also improve as pretraining data scales.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. Ablation Study on Interpretability
- CARE’s latent actions yield lower LP-MSE than CoMo and LAPA on Goal, Spatial, and Long, while 9-frame semantic prediction reaches 84.2% on LIBERO Goal.The 9-frame result is comparable to prediction using 10 consecutive frames.Source: §3.3 · p. 4
3.4. Ablation Study on Shortcut Learning
- S-PCFC scores were 0.992 for LAPA, 0.892 for CoMo, and 0.833 for CARE, with the lower CARE score indicating weaker shortcut learning.The authors attribute CARE’s result to modifying pretraining with VLM-based latent-action extraction.Source: §3.1 · p. 3, §3.4 · p. 4
3.5. Ablation Study on Multi-task Learning
- Multi-task Prismatic-7B reached 77.7% SR with 0.046 latent loss, versus 67.8% SR with 0.012 loss for frame prediction and 54.3% SR with 0.009 loss for point tracking.The authors interpret the results as synergistic benefits of multi-task pretraining; keypoint tracking’s lower success rate supports prior findings favoring frame prediction.Source: §3.5 · p. 4, Table 4 · p. 4
4. CONCLUSION
CARE learns continuous latent actions through multi-task pretraining with two decoders. It outperforms prior methods relying on a separate latent action model in simulation, though a gap remains relative to action-label-based pretraining.
- The authors identify action chunking and multidimensional perceptual perspectives as future directions for narrowing the remaining performance gap to action-label-based pretraining.Source: §4 · p. 4