Source-linked AI summary
SymVD: Symmetric Vision Language Action Distillation for Robot Manipulation
Hyewon Choi, Donggyu Kim, Soojean Han
TL;DR
Adapting broadly generalizing VLA models to changed environments can require substantial retraining, motivating more efficient compact policies. SymVD distills a VLA teacher into an equivariant student using symmetry-aware supervision and adaptive RL weighting, and experiments report improved sample efficiency and generalization over SAC and conventional distillation, with teacher quality remaining a limitation.
Problem
VLA adaptation to real-world environments or task shifts can require substantial additional data and retraining, while RL-based distillation remains sensitive to teacher quality.
Method
SymVD uses an equivariant actor-critic, symmetry-aware teacher-action alignment, group-averaged distillation, and adaptive weighting between distillation and reinforcement learning.
Results
Experiments on BulletArm manipulation tasks show SymVD consistently outperforms no-distilled SAC and conventional distillation in sample efficiency and generalization.
Takeaways & Limitations
Symmetry-consistent supervision with adaptive weighting supports compact policy transfer while reducing redundant exploration across symmetry-related configurations.
Takeaways & Limitations
RL-based distillation remains sensitive to teacher quality, and weak teachers can fail to converge or excessive teacher reliance can hinder stable improvement in complex tasks.
Abstract
from arXiv · showhide
While pretrained Vision-Language-Action (VLA) models offer broad generalization capabilities in robotic manipulation tasks, adapting them to real-world environments or handling task shifts often requires substantial additional data and retraining. To address this, we propose Symmetric VLA Distillation (SymVD), a distillation framework that transfers knowledge from a large VLA teacher to a compact student policy by explicitly exploiting geometric symmetries in manipulation tasks, such as rotational and reflectional invariance. SymVD employs an equivariant actor-critic architecture and trains the student using a symmetry-aware objective that aligns with teacher actions under group-invariant properties. We demonstrate that by enforcing the policy to respect equivariance, SymVD reduces redundant exploration across configurations related by group transformations and improves sample efficiency during distillation. To further stabilize and improve distillation, SymVD introduces an adaptive weighting scheme that dynamically balances the distillation objective and reinforcement learning updates based on training progress, enabling robust transfer even when the teacher signal is imperfect or misaligned. Experimental results on robotic manipulation tasks demonstrate that SymVD consistently improves over standard distillation and also outperforms SAC in terms of sample efficiency and generalization to previously unseen symmetric transformations of the environment.
1 Introduction
VLA models generalize broadly but can require substantial retraining after real-world or task shifts. SymVD combines symmetry-aware policy distillation, an equivariant student, and adaptive weighting to improve efficient transfer.
- VLA models can generalize across tasks, objects, and commands, but real-world or task changes may require substantial retraining data.
- Policy distillation transfers partially optimized teacher actions to compact students, but domain or task shifts can still demand heavy exploration.
- SymVD designs the student policy to respect geometric symmetries through a group-equivariant architecture for high-dimensional continuous control.
- SymVD combines symmetry-constrained supervised alignment with reinforcement learning and adaptively balances both objectives during training.
- The framework aims to reduce trial-and-error exploration and improve transfer to unseen configurations related by symmetry transformations.
2 Related Work
Prior work uses policy distillation and reinforcement learning to adapt generalist policies into compact agents, while geometric symmetries support robotic generalization and sample efficiency. However, teacher quality remains a central challenge for RL-based distillation.
- Policy distillation transfers expert knowledge into compact student models, while reinforcement learning supports experience-based behavior acquisition and task-specific reward signals.
- Recent RL-based distillation frameworks target faster exploration and improved deployability for lightweight policies derived from generalist models.
- Suboptimal or biased teacher demonstrations can degrade student performance, motivating more resilient distillation objectives.
- Geometric symmetries have been used in robotic perception to improve generalization and sample efficiency through equivariant and invariant representations.
3 Method
SymVD formulates manipulation as a symmetry-constrained MDP and equips the student with an equivariant actor–critic architecture. Its group-averaged distillation supervises invariant action components while leaving directional components to reinforcement learning, with adaptive weighting balancing both objectives.
- Problem formulation: SymVD models manipulation with state and action transformations that preserve rewards and dynamics, allowing experience to generalize across physically equivalent configurations.The symmetry group acts on states through L_g and actions through R_g.
- Equivariant agent design: The method focuses on planar rotational symmetries and approximates SO(2) with the discrete cyclic group C4, representing 90° rotations.The construction can be extended to finer cyclic discretizations such as C8.
- Equivariant agent design: The actor is constrained to satisfy πξ(L_g(s)) = R_g(πξ(s)), while the critic produces invariant values for jointly transformed states and actions.The planar action component rotates with the workspace, whereas auxiliary action dimensions remain invariant.
- Group-averaged distillation: Because pointwise teacher matching can conflict with an equivariant student, SymVD uses group averaging and masks distillation to invariant auxiliary action components.For rotational groups, direct matching would cancel the planar directional component in expectation, so reinforcement learning learns that component instead.
- Theoretical justification: Under a G-invariant MDP, an optimal G-equivariant policy exists, motivating the restriction of the student to symmetry-consistent policies.This justification assumes invariance of the MDP, initial distribution, and terminal condition.
- Adaptive weighting: The actor objective combines the SAC actor loss with a time-varying distillation term whose weight is initialized at 1 and reduced as the student approaches target performance.This adaptive weighting balances teacher supervision against reinforcement learning updates.
4 Experiment
Experiments across seven BulletArm manipulation tasks compare conventional and symmetry-aware distillation, including equivariant students and adaptive weighting. SymVD achieves faster convergence and the highest evaluation reward across tasks, while adaptive weighting generally improves rewards but can degrade later training with complex tasks or weak teachers.
- Experimental setup: Experiments evaluate three configurations: non-equivariant SAC without distillation, conventional pointwise distillation, and SymVD with equivariant SAC, symmetry-aware distillation, and adaptive weighting.The study uses seven closed-loop BulletArm tasks with a 7-DoF arm and gripper holding state.
- Baseline experiment: SymVD consistently outperforms naïve distillation on the evaluated tasks, and both distillation variants outperform non-distilled SAC in the baseline comparison.In the Drawer Opening task, no-distillation and naïve distillation fail to make meaningful learning progress, whereas SymVD continues improving and surpasses the fine-tuned VLA reference.
- Lightweight teachers: Across all seven tasks, SymVD converges faster and achieves the highest evaluation reward compared with all baselines.Figure 3 measures evaluation rewards every 500 training steps, averages results over 10 seeds, and shows standard error across runs.
- Lightweight teachers: Task difficulty changes distillation’s role: it supports early exploration on easier tasks but shapes inductive bias on difficult alignment tasks.SymVD removes pose- and direction-specific teacher biases by aligning supervision only within symmetry-consistent subspaces.
- Lightweight teachers: Equivariant SAC with naïve distillation can match or underperform SAC because the symmetry-agnostic objective conflicts with the architecture’s structured inductive bias.This limitation appears in the block pulling and picking tasks.
- Adaptive vs. non-adaptive distillation: Adaptive distillation weighting yields higher evaluation rewards than constant λ = 1 across most tasks, but later performance can degrade on Block Stacking.A weak teacher with a 40% success rate failed to converge there, so a stronger teacher exceeding 62% success was used instead.
- Limitations and future work: SymVD currently focuses on action-space symmetry and is proposed for future extension to richer symmetry groups, diverse embodiments, and state-level transformations.These extensions are described as future work rather than evaluated capabilities.
5 Conclusion
SymVD transfers pretrained VLA knowledge to compact reinforcement learning policies by combining equivariant actor–critic design, symmetry-aware distillation, and adaptive weighting. On BulletArm tasks, it outperforms no-distilled SAC and conventional distillation in sample efficiency and generalization, including with lightweight or low-quality teachers.
- SymVD transfers pretrained VLA knowledge to compact reinforcement learning policies by leveraging geometric symmetries in robotic manipulation.
- Its equivariant actor–critic architecture and symmetry-aware objective align teacher supervision with the student’s representational constraints.
- Adaptive weighting balances distillation and reinforcement learning updates, enhancing training stability when teacher signals are imperfect.
- SymVD consistently outperforms no-distilled SAC and conventional distillation in BulletArm sample efficiency and generalization.
- The results support task symmetries as a practical strategy for compressing generalist VLA knowledge.
A Vision-Language-Action Fine-Tuning
The VLA fine-tuning procedure adapts OpenVLA-7B to BulletArm by pairing task descriptions with action sequences and optimizing an L1 action-prediction objective. Training follows the OpenVLA-OFT recipe with LoRA for up to 200K steps.
- OpenVLA-7B is fine-tuned with LoRA following the officially released OpenVLA-OFT recipe.
- Each PyBullet task name is converted into a short textual manipulation description used only for the VLA-based teacher.
- The model tokenizes task text with the corresponding action sequence and predicts multi-step action chunks using an L1 regression objective.
- Training runs for up to 200K steps with a learning rate of 5 × 10^-4.
B Training for Lightweight Teacher Policies
Lightweight teacher policies are pretrained with DrQ-fD or TD3+BC, using planner-generated demonstrations to stabilize training. The study compares SymVD against naive distillation applied to the SAC actor, while reporting teacher rewards separately.
- Pretrained teacher policies use either DrQ-fD or TD3+BC.
- DrQ-fD adds an agent–demonstration mean squared error loss to DrQ’s image-based, augmentation-based off-policy training.
- TD3+BC combines TD3 with behavior cloning to constrain the learned policy toward demonstration data.
- More than 500 planner-generated demonstration episodes prefill the replay buffer for stable lightweight-teacher training.
- Naive distillation applies an objective matching the student SAC actor’s mean action to the VLA teacher action.