PaperlayerView exact revision on arXiv

Source-linked AI summary

VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RL

Namiko Saito, Kinam Kim, Heecheol Kim, Katsushi Ikeuchi, Yasuyuki Matsushita

arXiv:2609.30868v1cs.RO

TL;DR

Contact-rich VLA execution can remain imprecise, while physical-robot RL is costly and safety-critical. VLaRL trains residual corrections in simulation using the frozen VLA’s vision-language latents and a mapper toward real latents. It improves real-world success across four tasks and two backbones, without real-world RL or online adaptation.

  • Problem

    Contact-rich VLA execution can remain imprecise, and learning corrective behavior with physical-robot RL requires repeated interaction with substantial cost and safety concerns.

    Source: §I · p. 1

  • Method

    VLaRL trains a latent-conditioned residual policy in simulation for a frozen VLA, mapping simulated VLM latents toward the real latent distribution for transfer.

    Source: §I · pp. 1–2

  • Results

    VLaRL improves real-world success across all eight combinations of four contact-rich tasks and two VLA backbones, while ablations show the importance of latent alignment.

    Source: §I · p. 2

  • Takeaways & Limitations

    Residual policies also improved execution on unseen objects while retaining the frozen VLA’s instruction-dependent nominal behavior.

    Source: §VII · p. 7

    Mapper training requires approximately corresponding sim–real trajectories built through task-specific digital-twin reconstruction, and it does not explicitly address physical-property discrepancies such as contact dynamics or friction.

    Source: C. Limitations and Future Directions · p. 6

Abstract

from arXiv · show

Vision-language-action (VLA) models provide broad, instruction-conditioned manipulation behaviors, but their physical execution can remain imprecise during contact-rich interaction. Residual reinforcement learning (RL) can correct such errors while keeping the VLA frozen, but real-robot RL is costly and safety-critical. We propose VLA Latent-Conditioned RL (VLaRL), which enables residual RL for frozen VLAs to be trained in simulation and deployed on real robots without real-world RL or online adaptation. The key challenge is transferring the learned residual policy despite the visual gap between simulation and reality. Rather than requiring pixel-level visual correspondence, VLaRL uses the VLA's internal vision-language latent representation to condition residual control and as the sim-to-real transfer interface, and learns a lightweight mapper that transforms simulation-derived latents toward the real latent distribution. Across four contact-rich manipulation tasks and two VLA backbones, VLaRL improves real-world success in all task-backbone combinations, while controlled ablations demonstrate the importance of both latent conditioning and latent alignment for transferring simulation-trained residual control.

I. INTRODUCTION

VLaRL addresses imprecise contact-rich execution by learning residual corrections for a frozen VLA in simulation and transferring them through aligned vision-language latents. It improves real-world success across four tasks and two VLA backbones without real-world RL or online adaptation.

I. INTRODUCTION

  • Contact-rich manipulation can fail despite appropriate nominal behavior because small errors in contact location, alignment, or sustained contact can cause task failure.Correcting such errors through imitation learning requires demonstrations of the desired corrections and updating the base policy.Source: §I · p. 1
  • VLaRL keeps the VLA frozen and learns only corrective actions, using the pretrained policy’s task-relevant nominal behavior as a basis for reward-optimized residual control.This avoids relearning the complete policy.Source: §I · pp. 1–2
  • VLaRL uses the frozen VLA’s internal vision-language representation as the residual-control interface, reusing pretrained visual abstraction rather than raw images or a separate geometric state interface.The representation retains task-relevant visual and language context, but simulated and real observations can still produce different latent distributions.Source: §I · pp. 1–2
  • A lightweight mapper shifts simulation-derived latents toward the real latent distribution, allowing the residual policy to train on mapped simulated latents and consume real VLM latents at deployment.Deployment requires neither real-world RL nor online adaptation.Source: §I · pp. 1–2, Figure 1 · p. 1
  • VLaRL improves real-world success in all eight task–backbone combinations, and removing latent mapping substantially degrades performance.The evaluation covers Flower and GR00T N1.7; ablations highlight the importance of latent alignment.Source: §I · p. 2

II. RELATED WORK

Prior work spans generalist VLAs, residual refinement methods, and sim-to-real transfer approaches. VLaRL differs by training residual control entirely in simulation using the frozen VLA’s internal representation and latent mapping for real-robot transfer.

A. Vision-Language-Action Models

  • VLA research has developed generalist policies across architectures, while VLaRL keeps the VLA frozen as the nominal controller and learns residual corrections through simulation-based RL.Prior examples include RT-1, RT-2, Open X-Embodiment, OpenVLA, Octo, Flower, GR00T, and π-series models [1],,,,,, [6], [10], [11].Source: A. Vision-Language-Action Models · p. 2

B. Reinforcement Learning for VLA Refinement

  • Prior residual-refinement approaches use state-based simulation followed by distillation, physical-robot RL, RL-generated data, or VLA-initialized students; VLaRL instead trains residual control in simulation on the frozen VLA’s internal representation.Its latent mapping supports transfer without real-world RL –[20], [11], [8].Source: B. Reinforcement Learning for VLA… · p. 2

C. Sim-to-Real Transfer and Representation Alignment

  • Sim-to-real methods commonly use domain randomization or compact geometric representations, while real-to-sim work reconstructs simulation environments; VLaRL transfers through mapped pretrained VLM latents instead.The residual policy directly consumes real VLM latents, without requiring pixel-level correspondence or an explicit object-state interface at deployment.Source: C. Sim-to-Real Transfer and… · p. 2

III. METHOD

VLaRL retains the frozen VLA action head as the nominal controller and supplies its intermediate vision-language representation to residual control. Its workflow maps simulated latents during training, then deploys the residual policy on real latents without the mapper.

III. METHOD

  • VLaRL treats the frozen VLA as a vision-language encoder and an action head, reusing the intermediate representation for residual control while retaining the action head as nominal controller.The encoder processes visual observations and language instructions.Source: A. Overview · p. 2
  • The residual policy receives a fixed-dimensional VLM latent, robot proprioceptive state, and measured wrist force as inputs.The latent is reduced from the VLM token matrix to a D-dimensional vector; the residual scaling factor α is fixed across tasks.Source: A. Overview · p. 3
  • The four-stage workflow fine-tunes and freezes the VLA, learns a mapper toward real latents, trains the residual policy in simulation with the VLA and mapper frozen, and deploys the frozen VLA and residual policy.At deployment, the residual policy uses real-image latents directly and the mapper is not used.Source: A. Overview · p. 3, Figure 2 · p. 3

B. Step 1: VLA Fine-Tuning and Digital Twin Reconstruction

The VLA is fine-tuned on real-robot demonstrations, and those demonstrations support approximate digital twins for simulation and sim–real latent mapping. A mapper aligns simulated VLM representations toward the real latent distribution before residual-policy training.

B. Step 1: VLA Fine-Tuning and Digital Twin Reconstruction

C. Step 2: Sim-to-Real VLM Latent Mapping

  • The mapper transforms simulated token representations toward the real latent distribution, reducing representation shift for the VLA action head and aligning residual-policy inputs across domains.It operates on simulated tokens before pooling and is frozen during residual RL.Source: C. Step 2: Sim-to-Real VLM Latent… · p. 3
  • Because reconstructed sim–real trajectories provide approximate rather than exact frame-level correspondence, the mapper uses optimal-transport-based distribution alignment.The transport cost incorporates normalized trajectory progress, and entropy regularization enables efficient Sinkhorn-iteration computation.Source: C. Step 2: Sim-to-Real VLM Latent… · p. 3

D. Step 3: Simulation-Trained Residual Reinforcement Learning

The residual policy is trained in simulation using mapped VLA representations and the nominal action, alongside proprioceptive and force inputs. Asymmetric critics and demonstration-derived targets support learning, while the VLA and mapper stay frozen.

D. Step 3: Simulation-Trained Residual Reinforcement Learning

  • The residual actor receives the mean-pooled mapped simulated latent together with the nominal VLA action, proprioceptive state, and Cartesian force.The frozen VLA produces the nominal action from the mapped simulated token representation.Source: D. Step 3: Simulation-Trained Residual… · pp. 3–4
  • An asymmetric actor–critic gives the critic privileged task state available in simulation, while the actor uses only observations available at real-world deployment.Demonstration-derived residual targets are used as an auxiliary regularization term.Source: D. Step 3: Simulation-Trained Residual… · p. 4
  • The VLA and latent mapper remain frozen throughout RL training; only the residual policy is optimized.Source: D. Step 3: Simulation-Trained Residual… · p. 4

E. Step 4: Real-Robot Deployment

At deployment, the frozen VLA processes the real observation and instruction to produce a nominal action and real token representation. The residual policy uses a pooled real latent and does not use the mapper.

E. Step 4: Real-Robot Deployment

IV. EXPERIMENTAL SETUP

Experiments use a Franka Research 3 robot and MuJoCo, with four contact-rich tasks and two VLA backbones. Training uses a learned latent mapper and simulation-trained residual policies with specified inputs, optimization, and reward components.

A. Robot Setup and Manipulation Tasks

  • Experiments use a Franka Research 3 robot in the real world and MuJoCo simulation, with 32 teleoperated demonstrations collected per task.FoundationPose reconstructs object position and environment for digital twins; the demonstrations also support VLA fine-tuning, latent mapping, and residual targets.Source: A. Robot Setup and Manipulation Tasks · p. 4
  • The four tasks are button pressing, block pushing, cup stacking, and drawer closing, covering precise contact, sustained contact, spatial alignment, and articulated contact, respectively.Source: A. Robot Setup and Manipulation Tasks · p. 4
  • Unseen-object transfer tests block pushing on novel star- and triangle-shaped blocks and cup stacking on novel sky-blue and green cups, without additional residual RL or real-world adaptation.Cup tests use both language-specified role assignments.Source: A. Robot Setup and Manipulation Tasks · p. 4

B. Training Details

  • The mapper is a four-layer Transformer encoder trained with Sinkhorn divergence, while the residual actor is regularized using demonstration-derived targets and residual-action magnitude.The mapper operates before pooling with 16 attention heads and feed-forward dimension twice the VLA token dimension.Source: B. Training Details · p. 4
  • Residual policies train for 60,000 environment steps using TD3, with normalized Cartesian action, end-effector pose, gripper state, and 3D Cartesian force as inputs.The actor and twin critics are four-layer ReLU MLPs with hidden width 1024; critics also receive privileged simulation task state.Source: B. Training Details · p. 4

C. Evaluation Protocol

The evaluation tests simulation-trained residual control on real robots across task–backbone combinations, with controlled ablations and randomized conditions. VLaRL improves real-world success throughout, although simulation and real-world performance do not always track together.

C. Evaluation Protocol

  • Initial positions were randomized by ±3 cm and orientations by ±15°, real-world conditions used 40 trials, and all models remained frozen without real-world RL or online adaptation.Drawer opening was also randomized by ±3 cm; simulation results were reported over three seeds.Source: C. Evaluation Protocol · p. 4
  • The controlled Flower ablations removed either the VLM latent or the mapper while keeping the same evaluation protocol and success criteria as VLaRL.The no-latent variant retained the nominal VLA action, proprioception, and force; the no-mapper variant used unaligned simulated VLM latents.Source: C. Evaluation Protocol · p. 4
  • Fig. 3 pairs real and simulated left- and hand-camera observations at the same timestep, while Table I presents the evaluation tasks and success criteria.The figure shows four manipulation tasks and their digital-twin observations.Source: Figure 3 · p. 5, Table I · p. 5

A. Simulation-Trained Residual Control Improves Real-World Execution

  • In simulation, Flower block-pushing success rose from 32.5% with the base VLA to 42.5% with the mapper and 67.5% with VLaRL.The mapper improved nominal VLA success across all task–backbone combinations, and residual RL further improved success across all combinations.Source: A. Simulation-Trained Residual Control… · p. 5
  • VLaRL improved real-world success in all eight task–backbone combinations, including Flower button pressing (67.5% to 100.0%), block pushing (22.5% to 50.0%), and GR00T N1.7 cup stacking (17.5% to 45.0%).The simulation-trained residuals transferred without real-world RL or online adaptation.Source: A. Simulation-Trained Residual Control… · p. 5
  • Simulation and real-world performance did not always follow the same trend; Flower’s higher real-world base success on cup stacking left only a small additional real-world improvement from the residual.The passage identifies contact dynamics and the distribution of VLA actions as other sim-to-real differences that latent alignment does not eliminate.Source: A. Simulation-Trained Residual Control… · p. 5

B. Ablation Analysis

Ablations indicate that both the mapper and VLM latent contribute to real-world residual performance. On unseen objects, VLaRL improves real-world success for block pushing and cup stacking without additional residual RL or real-world adaptation.

B. Ablation Analysis

  • Without the mapper, block pushing and cup stacking failed in all 40 real-robot trials, versus 50.0% and 70.0% success, respectively, with full VLaRL.The result indicates residual policies trained on unaligned simulated VLM representations do not directly transfer to real representations.Source: B. Ablation Analysis · p. 5
  • Removing the VLM latent lowered real-world success on all four tasks, including button pressing (100.0% to 70.0%) and block pushing (50.0% to 20.0%).The ablation retained nominal VLA action, proprioception, and force.Source: B. Ablation Analysis · p. 5
  • Integrated-gradient attribution assigned the VLM latent the largest share of residual-action norm across tasks and backbones: 72.1%–89.7% for Flower and 84.0%–90.9% for GR00T N1.7.Force attribution rose for sustained-contact block pushing, while proprioception became relatively more important for position-sensitive cup stacking.Source: B. Ablation Analysis · p. 5

C. Unseen-Object Transfer

  • On unseen objects, VLaRL raised real-world block-pushing success from 40.0% (16/40) to 57.5% (23/40) and cup-stacking success from 55.0% (22/40) to 62.5% (25/40), without additional residual RL or real-world adaptation.The evaluations used unseen shaped objects for pushing and unseen-colored cups for stacking, with balanced placement and object-role conditions.Source: C. Unseen-Object Transfer · pp. 5–6

VI. DISCUSSION

VLaRL uses VLM latents to condition residual corrections and align simulation training with real-robot deployment. The discussion highlights why this interface matters and identifies constraints on scaling and generalization.

A. Why Latent-Conditioned Residual RL?

  • VLM latents combine visual observations with language instructions, giving residual control context that nominal actions alone do not expose.This is useful when corrections depend on the instructed target or manipulation role; explicit geometric states instead require task-relevant objects and variables to be specified and estimated.Source: A. Why Latent-Conditioned Residual RL? · p. 6

B. Simulation-Only Residual Learning

  • Mapping simulation-derived representations toward the real latent distribution addresses the shift that can prevent simulation-trained residual corrections from transferring.The approach aims to use simulation interaction without real-world RL or online adaptation.Source: B. Simulation-Only Residual Learning · p. 6

C. Limitations and Future Directions

  • Mapper training requires approximately corresponding sim–real trajectories built through task-specific digital-twin reconstruction.The paired-data requirement is incurred during training, and reducing or eliminating it would make the approach easier to scale.Source: C. Limitations and Future Directions · p. 6
  • The mapper addresses visual representation differences but does not explicitly address mismatches in contact dynamics, friction, compliance, or other physical properties.These remaining differences may explain discrepancies between simulation and real-world performance.Source: C. Limitations and Future Directions · p. 6
  • Each VLA backbone requires its own mapper, and residual RL is trained separately for each task category.The discussion identifies transferable alignment and shared residual policies across backbones and skills as directions toward more general control.Source: C. Limitations and Future Directions · p. 6
  • Unseen-object evaluation covers only a limited range of variations and does not establish open-world generalization.The reported improvements suggest local execution can improve while preserving instruction-dependent nominal behavior for unseen targets.Source: C. Limitations and Future Directions · pp. 6–7

VII. CONCLUSION

VLaRL augments frozen VLAs with latent-conditioned residual control and a mapper that aligns simulation-derived latents toward the real distribution. Across four contact-rich tasks and two backbones, it improves real-world success, with ablations highlighting the importance of latent conditioning and mapping.

VII. CONCLUSION

  • VLaRL uses a frozen VLA’s vision-language representation as the residual-control interface and maps simulation-derived latents toward the real distribution.Residual RL is trained in simulation and deployed without real-world RL or online adaptation.Source: §VII · p. 7
  • Across four contact-rich tasks and Flower and GR00T N1.7, VLaRL consistently improves real-world success, while ablations show the importance of latent conditioning and sim-to-real mapping.The conclusion also reports improved execution on unseen objects while retaining the frozen VLA’s instruction-dependent nominal behavior.Source: §VII · p. 7
  • The conclusion presents simulation-trained residual control as a practical way to combine pretrained VLA behavior with precise real-world manipulation.The unseen-object evaluation changes appearance and language-assigned roles for cup tasks, and object geometry for pushing.Source: §VII · pp. 7–8, Figure 8 · p. 7

Related papers

Loading 2609.30868v1…