PaperlayerView exact revision on arXiv

Source-linked AI summary

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies

Kian Hosseinkhani, Qinhe Peng, George Shramko, Mehran Aghabozorgi, Jianing Qian, Tristan Engst, Alireza Moazeni, Dinesh Jayaraman, Ke Li

arXiv:2609.10915v1cs.ROcs.CV

TL;DR

VLA policies face an inference bottleneck because continuous action heads require iterative sampling, slowing real-time control while single-step regression risks mode collapse. IMLE-VLA replaces that head with a cIMLE-trained single-step conditional generator, achieving leading LIBERO success and faster inference while retaining robustness under perturbations and improving real-world efficiency.

  • Problem

    Iterative diffusion or flow-matching action heads slow VLA inference, while naive single-step regression can collapse multimodal action distributions.

  • Method

    IMLE-VLA replaces π0.5’s iterative action head with a single-step conditional generator trained using cIMLE for mode coverage.

  • Results

    IMLE-VLA achieves 98.0% average success on LIBERO, leads in inference frequency, retains π0.5’s LIBERO-plus robustness, and reduces real-world VLA-only wall-clock by 3.9×–6.6×.

  • Takeaways & Limitations

    A cIMLE action head can combine single-step inference with multimodal action coverage in a generalist VLA.

Abstract

from arXiv · show

Vision-language-action (VLA) policies leverage pretrained vision-language backbones to achieve strong cross-task generalization. A leading design couples this backbone with a dedicated continuous action head trained via diffusion or flow matching. However, such heads rely on iterative multi-step sampling, for example 10 Euler steps in $π_{0.5}$. This creates an inference bottleneck that produces stop-and-go movement in the robot and slower task completion. We introduce IMLE-VLA, which replaces the iterative action head with a single-step conditional generator trained via conditional Implicit Maximum Likelihood Estimation (cIMLE). The cIMLE objective promotes multimodal action coverage, avoiding the mode collapse of naive regression heads while eliminating multi-step sampling entirely. When IMLE-VLA is applied to $π_{0.5}$, it increases inference frequency 3.67x (55 Hz vs. 15 Hz), enabling up to 11x higher action throughput. On the 40-task LIBERO benchmark, IMLE-VLA achieves the highest average success rate (98.0%) among all baselines while leading in inference frequency. Under the test-time perturbations of LIBERO-plus, IMLE-VLA retains $π_{0.5}$'s robustness while other baselines degrade sharply, confirming that the cIMLE head preserves generalization. Real-world experiments on a Franka Emika Panda across four tasks demonstrate smoother motion (2.2x to 3.0x lower jerk) and faster task completion, with IMLE-VLA outperforming $π_{0.5}$ on every task and reducing average VLA inference time per episode by 3.9x to 6.6x. Videos and code are available at https://kianhk6.github.io/IMLE-VLA/

I. INTRODUCTION

IMLE-VLA addresses the latency and multimodality trade-off in VLA action generation by replacing iterative sampling with a single-step cIMLE generator. It improves inference speed and benchmark performance while preserving robustness under perturbations.

  • VLA inference is a bottleneck because large vision-language backbones and sequential continuous-action sampling create latency in high-frequency robot control.Continuous-action heads may require multiple sequential forward passes, such as 10 Euler steps.
  • IMLE-VLA replaces multi-step diffusion or flow-matching sampling with a single-step conditional generator trained using cIMLE for explicit mode coverage.The objective targets multimodal action distributions without relying on naive single-step regression, which can collapse toward average behavior.
  • 3.67× higher inference frequency, 55 Hz versus 15 Hz, addresses the stop-and-go behavior caused by slow action generation.With a longer execution horizon, the increase compounds to up to 11.0× higher action throughput.
  • 98.0% average success makes IMLE-VLA the highest-performing method on the 40-task LIBERO benchmark while leading in inference frequency.On LIBERO-plus, it also retains π0.5’s robustness while other baselines degrade sharply.
  • Real-world evaluation on a Franka Emika Panda measures inference-time reductions of up to 6.6×, alongside smoother and faster robot behavior.The reported wall-clock measurement excludes robot execution and other non-inference overhead.

II. RELATED WORK

Prior acceleration methods use system optimization, distillation, or teacher-free regression, but often retain sequential sampling, teacher biases, reduced capacity, or mode-collapse risks. IMLE-VLA applies cIMLE single-step generation to a generalist VLA rather than a specialist policy.

  • System-level optimizations: System-level kernel tuning can accelerate inference, but sequential diffusion-head steps remain a bottleneck; the two approaches are complementary.The paper reports further amplification from efficient Triton kernels.
  • Distillation methods: Distillation accelerates VLAs by training students to imitate teachers, but speedups can inherit teacher-distribution biases or reduce success through lower capacity.Shallow-π is described as a lower-capacity student whose speedup costs success relative to its teacher.
  • Teacher-free single-step methods: OpenVLA-OFT uses an L1 regression head for single-step generation, but L1 can collapse to the conditional median under multimodal action distributions.Its reported viability is attributed to OpenVLA’s 7B backbone capacity.
  • From specialists to a generalist: IMLE-VLA extends cIMLE-based single-step multimodal generation from specialist robotics policies to a generalist VLA.It uses internet-scale vision-language pretraining to generalize across tasks and embodiments.

III. METHOD

IMLE-VLA couples a frozen vision-language backbone with a lightweight action head and trains that head for single-step multimodal action generation using cIMLE. The method is designed to remove iterative action-generation cost while retaining the backbone’s pretrained representations.

  • IMLE-VLA trains a single-step action head with a conditional IMLE objective to eliminate iterative action-generation steps.The framework is presented as a coupling of a frozen VLM backbone and a lightweight action head.
  • The method preserves the high-capacity pretrained VLM while focusing training on the action-generation component.

A. Single-Step VLA Architecture

The architecture freezes a high-capacity VLM and uses its embeddings, together with Gaussian noise, to condition a lightweight generator. In one forward pass, the generator produces complete multimodal action chunks rather than iteratively denoising them.

  • The frozen VLM encodes visual observations and language instructions, while a lightweight generator produces actions conditioned on those embeddings.Freezing the backbone retains pretrained spatial and semantic features and limits training to the action head.
  • A single forward pass maps VLM embeddings and Gaussian noise to a complete action chunk A ∈ R^C×D.C is the prediction horizon and D is the action-space dimensionality.
  • Sampling different latent noise vectors enables diverse candidate action chunks that represent multimodal conditional behavior.This stochastic generation avoids regressing to one average outcome.
  • The action head is initialized from a pretrained π0.5 checkpoint and fine-tuned with cIMLE while the VLM remains frozen.

B. Conditional IMLE (cIMLE)

cIMLE turns conditional action generation into a single-step mapping from latent noise to action chunks while preserving multimodality. Nearest-neighbour assignment prevents averaging distinct valid actions into one collapsed behavior.

  • B. Conditional IMLE (cIMLE): cIMLE generates action chunks in one step by mapping latent noise to actions conditioned on VLM embeddings, eliminating iterative sampling.The method replaces repeated numerical integration with direct latent-to-action generation.
  • B. Conditional IMLE (cIMLE): Each ground-truth action is assigned to its nearest candidate among m samples generated from the same observation.The assignment is performed before the gradient update, which minimizes distance only to the selected candidate.
  • B. Conditional IMLE (cIMLE): With m=1, cIMLE reduces to L2 regression, whose optimum is the conditional mean and can produce invalid actions between modes.Averaging multiple valid behaviors may match none of the demonstrated action sequences.
  • B. Conditional IMLE (cIMLE): For m>1, nearest-neighbour assignment lets different latent codes cover different action modes instead of collapsing them into one average.The outer objective requires coverage across ground-truth behaviors while unassigned candidates remain available for other modes.
  • B. Conditional IMLE (cIMLE): The training procedure reuses each VLM embedding across candidates and performs assignment without gradients, keeping cIMLE computationally lightweight.The authors report m=2 as an effective balance between expressivity and efficiency.

IV. EXPERIMENTS

The experiments evaluate whether single-step cIMLE improves inference efficiency while preserving VLA policy quality. They measure success, throughput, VLA-only inference time, robustness under distribution shift, and real-world behavior.

  • IV. EXPERIMENTS: The experimental goal is to determine whether cIMLE can raise inference frequency while maintaining or improving VLA success rates.The study includes both simulation and a real-world robot evaluation.
  • IV. EXPERIMENTS: Task success rate is the primary policy-quality metric, while action throughput equals inference frequency multiplied by execution horizon H.VLA-only wall-clock measures forward-pass time per successful episode, excluding robot execution and other non-inference overhead.
  • IV. EXPERIMENTS: All experiments use a receding-horizon loop in which one forward pass predicts a chunk and the robot executes its first H actions before replanning.A larger H means fewer replanning events between observations.
  • IV. EXPERIMENTS: The evaluation spans inference frequency, 40-task LIBERO performance, LIBERO-plus distribution-shift robustness, and four real-world manipulation tasks.These axes compare efficiency, simulated task success, robustness, and physical-robot behavior.

A. Inference frequency

IMLE-VLA removes the iterative flow-matching bottleneck while retaining the π0.5 VLM backbone. It reaches 55 Hz, compared with 15 Hz for canonical π0.5, improving the control loop without reducing model capacity.

  • A. Inference frequency: Higher inference frequency supports smoother motion and more responsive control by enabling tighter robot control loops.The paper connects slow inference cycles with stop-and-go pauses and reduced reactivity to environmental changes.
  • A. Inference frequency: The benchmark includes mandatory tokenization and proprioceptive-state costs at every forward call under deployment-realistic conditions.Measurements use a single NVIDIA L40S GPU, natural task-specific prompts, and standard two-view image inputs.
  • A. Inference frequency: The 10-step flow-matching loop limits optimized π0.5 implementations to 15–25 Hz, so system-level tuning alone cannot remove the algorithmic ceiling.PyTorch torch.compile reaches 20 Hz and dedicated Triton kernels reach 25 Hz.
  • A. Inference frequency: IMLE-VLA preserves the exact VLM backbone architecture while replacing the iterative action head with a single-step conditional generator.The change increases inference frequency without reducing model capacity or backbone scale.
  • A. Inference frequency: 55 Hz is achieved by IMLE-VLA versus 15 Hz for canonical π0.5, a 3.67× inference-frequency increase.Table I averages measurements over 50 episodes across 10 LIBERO-Long tasks on an NVIDIA L40S.

B. Simulation: LIBERO Benchmark

On LIBERO, IMLE-VLA combines the highest reported success with leading inference speed, while retaining robustness under distribution shift and substantially increasing throughput and reducing VLA-only wall-clock.

  • Success rate comparison: 98.0% average success at H=10 makes IMLE-VLA the highest-performing compared method while it leads inference frequency at 3.67× π0.5.The benchmark covers 40 tabletop manipulation tasks across four suites, with 50 episodes per task.
  • Robustness under distribution shift: Across four perturbation axes and five severity levels, IMLE-VLA retains π0.5’s robustness while other baselines degrade as severity increases.LIBERO-plus perturbs background, robot initial state, language, and layout.
  • Action throughput: 11.0× higher action throughput at H=30 compounds IMLE-VLA’s single-step inference advantage with the longer execution horizon.At H=10, the policy already delivers 3.67× higher throughput than π0.5 from single-step generation alone.
  • VLA-only wall-clock: 10.5× lower VLA-only wall-clock at H=30 combines roughly three times fewer forward passes with faster per-pass inference.At H=10, VLA-only wall-clock is already reduced by 3.6×.

C. Method Analysis

The method analysis shows that longer horizons trade reactivity for throughput, while cIMLE’s sample-factor choice preserves success and avoids the failure associated with standard regression.

  • Horizon ablation: 97.1% success at H=30 accompanies 11.0× higher throughput and exceeds π0.5’s 96.1% at the same horizon.H=10 reaches 98.0%, but performance gradually degrades as the horizon grows.
  • Horizon ablation: 98.0% success at H=10 is the benchmark peak, whereas larger horizons reduce reactivity and degrade performance.Each replan cycle executes H open-loop actions before conditioning on a new observation.
  • cIMLE sample factor ablation: m=1 sharply reduces cIMLE to standard regression, while m=2 and m=5 are nearly identical; the method therefore uses m=2.The selected value retains strong performance while minimizing the cost of generating additional candidates.

D. Real-world experiments

Across single-step, multi-step, and reactive real-world tasks, IMLE-VLA produces faster, smoother, and more successful actions than π0.5. Its higher inference frequency improves motion quality and enables more responsive adaptation to changing targets.

  • Real-world results: IMLE-VLA outperforms π0.5 on every real-world task, reduces VLA-WC by 3.9×–6.6×, and produces 2.2×–3.0× lower jerk.Results use 20 episodes per task on a Franka Emika Panda; VLA-WC measures VLA forward-pass time per successful episode.
  • Multi-step manipulation: Faster inference lets IMLE-VLA follow direct trajectories and recover from errors more effectively during multi-step manipulation.On swap pineapple and cube, it avoids repeated direction changes; on pineapple in cabinet, its shorter replan interval enables recovery after a slip.
  • Dynamic reactivity: In the moving-plate task, IMLE-VLA’s shorter wall-clock replanning cycle enables continuous tracking and adaptation despite its longer action horizon.More frequent perception–action closure helps the policy respond to the plate’s changing motion in real time.
  • Dynamic reactivity: Slower π0.5 replanning acts on increasingly stale observations, causing repeated reaches toward outdated target locations and task failure.The delay between observing the moving plate and responding to its updated position degrades interception and grasp attempts.

V. CONCLUSION

IMLE-VLA replaces iterative flow-matching action heads with a single-step cIMLE generator, improving inference speed while preserving multimodal action coverage. It achieves strong simulation and real-world performance, with smoother motion and lower VLA-only runtime than π0.5.

  • Conclusion: IMLE-VLA replaces iterative flow-matching heads with a single-step cIMLE generator, achieving 3.67× higher inference frequency and up to 11.0× higher action throughput.The approach explicitly optimizes for mode coverage while eliminating multi-step sampling.
  • Conclusion: IMLE-VLA achieves the highest average LIBERO success rate at 98.0% while leading in inference frequency.The method modifies only the action head, supporting its use as a drop-in replacement for iterative heads in existing VLAs.
  • Conclusion: On a Franka Emika Panda, IMLE-VLA outperforms π0.5 on every task, produces smoother motion, and reduces VLA-only wall-clock time by 3.9×–6.6×.The conclusion attributes these real-world gains to higher inference frequency.
Loading 2609.10915v1…