PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflex: Enabling Fast and Predictive Vision-Language-Action Models for Reaction-Critical Manipulation

Yuxuan Chen, Wanruo Zhang, Xiao Li

arXiv:2608.14379v1cs.ROcs.AI

TL;DR

Existing benchmarks largely overlook reaction-critical manipulation and inference latency in dynamic settings. The paper introduces ReflexBench and ReflexVLA with latency-aware evaluation, future prediction, temporal fusion, and inference optimization, achieving the highest average ReflexBench success rate at 50.4% while remaining competitive on static manipulation.

  • Problem

    Existing manipulation benchmarks largely overlook reaction-critical scenarios and inference latency, limiting evaluation of VLA performance in dynamic real-world settings.

  • Method

    The paper introduces ReflexBench and ReflexVLA, combining latency-aware evaluation with latent future prediction, multi-frame temporal fusion, and inference optimization.

  • Results

    ReflexVLA achieves the highest overall ReflexBench average success rate at 50.4% across six tasks while maintaining competitive static-benchmark performance.

  • Takeaways & Limitations

    ReflexBench evaluates latency-aware dynamic manipulation, while ReflexVLA provides competitive reaction-critical performance with a lightweight 1B-parameter model.

  • Takeaways & Limitations

    Future prediction and temporal fusion are added only during fine-tuning, and advanced inference mechanisms such as RTC are not explored.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have recently achieved promising performance in robotic manipulation. However, existing benchmarks mainly evaluate generalization on static manipulation tasks and largely overlook dynamic interaction scenarios. To address this gap, we present ReflexBench, a benchmark for reaction-critical manipulation. ReflexBench contains six dynamic tasks and introduces an evaluation framework that decouples simulator stepping from robot control while supporting configurable latency under synchronous and asynchronous inference. Building upon ReflexBench, we propose ReflexVLA, an efficient VLA model designed for reaction-critical manipulation without large-scale robot-data pretraining. ReflexVLA enhances temporal reasoning through latent future prediction and multi-frame temporal fusion within the vision backbone, while reducing deployment latency through batched visual encoding and CUDA Graph replay. Experiments show that ReflexVLA consistently improves dynamic manipulation performance while maintaining competitive accuracy on standard static manipulation benchmarks, and real-world experiments further demonstrate its effectiveness under practical deployment conditions. Project website: https://reflexvla.github.io

I. INTRODUCTION · II. RELATED WORK

The paper identifies a gap in VLA evaluation and deployment for reaction-critical manipulation, where rapid responses and future-state reasoning are essential. It introduces ReflexBench and ReflexVLA to address this gap through dynamic-task evaluation, predictive modeling, and latency optimization.

  • I. INTRODUCTION: Current VLA models struggle with manipulation scenarios requiring rapid reaction and future-state reasoning.Many approaches condition actions on current observations, while inference latency delays perception-to-execution in dynamic environments.
  • I. INTRODUCTION: Existing research explores dynamic-manipulation benchmarks, future prediction, and efficient VLA inference as complementary directions.These efforts include specialized benchmarks, future-aware capabilities, architectural optimization, model compression, and system-level acceleration.
  • I. INTRODUCTION: ReflexBench introduces six simulation-based tasks requiring efficient execution and anticipation of future environmental dynamics.The benchmark targets representative reaction-critical manipulation challenges.
  • I. INTRODUCTION: ReflexBench incorporates latency effects and aligns evaluation with real-world execution delays to better reflect practical deployment conditions.Its evaluation is designed to account explicitly for delays affecting robot control.
  • I. INTRODUCTION: ReflexVLA is a fast and predictive VLA framework combining future latent prediction, multi-frame history modeling, and inference latency optimization.Latent prediction encourages reasoning about near-future visual states, while historical observations provide richer temporal information.
  • I. INTRODUCTION: The paper investigates the factors enabling VLA policies to succeed in reaction-critical tasks through ReflexBench and ReflexVLA.ReflexVLA is designed for fast and predictive robot control.
  • I. INTRODUCTION: The proposed designs are evaluated extensively in simulation and real-world environments.The experiments demonstrate the effectiveness of the proposed designs.

A. Benchmarks for Dynamic Manipulation … A. Task List

The paper motivates ReflexBench as a latency-aware benchmark for reaction-critical manipulation, addressing the limits of static evaluations and simulation settings that pause during inference. It defines six dynamic tasks requiring rapid inference under changing environmental conditions and highlights efficiency as important for deployment.

  • A. Benchmarks for Dynamic Manipulation: Existing dynamic-manipulation benchmarks study object motion, latency, or spatiotemporal reasoning, but ReflexBench explicitly evaluates latency under synchronous and asynchronous execution.This provides a more faithful evaluation of latency-sensitive robotic manipulation.
  • B. Efficient Vision-Language-Action Models: Efficiency is increasingly important for deploying larger VLA models, motivating approaches that reduce model size, training cost, or inference overhead.Examples include SmolVLA and VLA-Adapter for size and training costs, and FASTER and VLASH for system- or architecture-level efficiency.
  • III. REFLEXBENCH: Most manipulation benchmarks overlook reaction-critical scenarios because their simulators pause during policy inference, ignoring perception-execution delays.As a result, they do not accurately assess VLA performance in dynamic real-world settings.
  • III. REFLEXBENCH: ReflexBench incorporates realistic latency effects and targets tasks requiring future-aware decision making and efficient policy execution.The benchmark is specifically designed for reaction-critical manipulation.
  • A. Task List: ReflexBench contains 6 manipulation tasks in which policies must infer quickly under dynamic environmental conditions.The task suite includes conveyor-belt pick-and-place, ball catching, whack-a-mole, rolling-ball interception, ball throwing, and rotating peg insertion.
  • A. Task List: The task suite covers moving-object interactions, including picking a cube from a conveyor, catching an incoming ball, and intercepting a rolling ball before it falls.These tasks require responding to objects whose positions change over time.
  • A. Task List: It also tests short-lived and continuously changing targets through whack-a-mole, ball throwing, and rotating peg insertion.Targets may disappear if unreached, while the peg must be inserted into a continuously rotating socket.

B. Data Collection · C. Latency-Aware Evaluation

ReflexBench addresses the difficulty of collecting demonstrations for dynamic manipulation and evaluates policies under realistic, configurable inference latency. Its protocol decouples environmental simulation from robot control and supports both synchronous and asynchronous deployment paradigms.

  • B. Data Collection: Dynamic manipulation demonstrations require continuous adaptation to moving objects and time-dependent interactions, unlike scripted or teleoperated static-benchmark trajectories.The dynamic tasks involve environmental dynamics that make successful execution substantially more challenging.
  • B. Data Collection: The data collection pipeline decomposes each task into sequential phases with distinct subgoals, such as pre-alignment, target tracking, and timed insertion.In Rotating Peg Insertion, these phases organize the task around temporally distinct objectives.
  • B. Data Collection: For dynamic objects, the planner predicts future trajectories from real-time position and velocity to plan actions toward anticipated future states.A task-specific approach is trained when planning alone cannot handle complex interaction dynamics.
  • C. Latency-Aware Evaluation: ReflexBench decouples simulation from robot control, allowing environmental dynamics to evolve independently during policy computation for more realistic evaluation.This differs from simulation benchmarks that pause the environment during inference.
  • C. Latency-Aware Evaluation: ReflexBench models inference latency under synchronous and asynchronous inference, trading execution blocking against potentially stale actions.Synchronous inference waits for policy completion, whereas asynchronous inference overlaps action execution with policy computation.
  • C. Latency-Aware Evaluation: A latency blocking mechanism reproduces inference-blocking deployment by inserting a specified idle period before executing the predicted action chunk.Under synchronous inference, the simulator pauses for observation and policy querying, then resumes before the robot waits and acts.
  • C. Latency-Aware Evaluation: Latency can be manually varied without changing the policy or hardware, enabling controlled analysis of its effect on task performance.ReflexBench also derives simulation delays from measured real-world policy latency using the simulator’s Real-Time Factor (RTF), where RTF compares elapsed simulation time with corresponding wall-clock execution time.

IV. REFLEXVLA · A. Latent Future Prediction

ReflexVLA is a compact vision-language-action model that combines fused visual-language processing with latent future prediction for reaction-critical manipulation. Its future-prediction objective aligns semantic scene forecasting with action chunks, encouraging representations that capture dynamic-scene evolution.

  • IV. REFLEXVLA: ReflexVLA builds on a compact vision-language-action architecture with fused DINOv2 and SigLIP visual encoders, a Qwen2.5-0.5B language backbone, and learnable action queries.Visual inputs use 224 × 224 resolution, and the model predicts action chunks through a continuous regression head.
  • A. Latent Future Prediction: Dynamic manipulation motivates predicting future states rather than relying only on the current observation.The latent future prediction objective is introduced to learn predictive scene representations.
  • IV. REFLEXVLA: The ReflexVLA design integrates latent future prediction, multi-frame temporal fusion, and inference latency optimization for fast, anticipatory decisionmaking.The supplied overview identifies these components as the model’s central elements for reaction-critical robotic manipulation.
  • A. Latent Future Prediction: ReflexVLA predicts future observations in a semantic latent space extracted by a frozen visual encoder instead of generating future images directly.This avoids the computational expense and low-level appearance focus associated with direct future-image prediction.
  • A. Latent Future Prediction: For each training sample, future observations over horizon H are encoded using a frozen DINOv3 model.The encoded future representations provide targets for the auxiliary prediction objective.
  • A. Latent Future Prediction: ReflexVLA appends H learnable future tokens, matching the prediction horizon to the action chunk size so each token corresponds to one future control step.This alignment connects future-state prediction with action generation and downstream control.
  • A. Latent Future Prediction: The predicted feature for each future token is trained against its corresponding future visual representation with a masked cosine similarity loss.The mask indicates whether the corresponding future observation is valid.
  • A. Latent Future Prediction: Future prediction is jointly optimized with action prediction, with λ_future controlling its contribution to the combined objective.This training requirement encourages representations that capture the temporal evolution of dynamic scenes.

B. Multi-Frame Temporal Fusion

ReflexVLA fuses historical visual information inside the vision backbone using intermediate features and causal temporal attention, then exposes only the fused current-frame representation to the language model. This preserves short-term motion information while keeping the language model’s visual-token count unchanged from single-frame input.

  • Motivation and limitations: Multi-frame token concatenation incurs high computational cost from quadratic attention and leaves cross-frame interactions to implicit language-model learning.With V views, T frames, and P tokens per image, concatenation produces VTP visual tokens.
  • Temporal fusion design: ReflexVLA performs temporal fusion within the vision backbone and passes only the fused current-frame representation to the language model.Fusion uses intermediate features because they retain richer local appearance and motion cues than deeper semantic representations.
  • Temporal fusion design: Causal temporal attention summarizes each patch’s motion history while preventing information leakage from future observations during online deployment.Features from the same spatial position across historical frames form temporal trajectories, which are normalized, projected to a lower-dimensional space, and combined with temporal positional embeddings.
  • Temporal fusion design: The fused current-frame tokens are projected into the language-model embedding space and inserted into the multimodal sequence for action prediction.The number of visual tokens consumed by the language model remains identical to the single-frame setting, avoiding substantial language-model-side overhead from multi-frame images.

C. Inference Latency Optimization

ReflexVLA reduces deployment latency by optimizing visual processing and GPU execution, improving responsiveness for dynamic manipulation. These system-level changes preserve the policy architecture, learning objective, and multimodal reasoning process.

  • Visual and GPU Optimization: ReflexVLA targets visual processing and GPU execution to reduce end-to-end inference latency for real-world dynamic manipulation.Inference latency is critical because actions generated too slowly can cause strong policies to fail.
  • Visual Encoding: Batching all view-frame images into one visual encoder invocation replaces V × T separate forward passes and reduces per-frame execution overhead.The resulting features are reshaped for temporal fusion across views, frames, patches, and feature dimensions.
  • CUDA Graph Replay: ReflexVLA captures its fixed complete inference pipeline as a CUDA Graph instead of repeatedly launching individual GPU kernels each control cycle.The captured pipeline includes visual encoding, temporal fusion, multimodal projection, language model inference, and action prediction.
  • CUDA Graph Replay: During execution, new observations are copied into pre-allocated buffers and the captured graph is replayed directly, eliminating repeated kernel scheduling and runtime dispatch overhead.The graph produces the predicted action chunk from the updated observations.
  • System-Level Impact: The optimizations preserve the policy architecture, learning objective, and original multimodal reasoning process while reducing deployment overhead for latency-sensitive robotic manipulation.They substantially improve execution efficiency without modifying policy learning.

V. EXPERIMENTS

The experiments investigate factors affecting reaction-critical manipulation success and evaluate ReflexVLA against existing baselines, static-task performance, component contributions, and real-world manipulation.

  • The experiments examine how different factors influence success rates on reaction-critical manipulation tasks.
  • They compare ReflexVLA with existing VLA baselines on ReflexBench.
  • They assess whether ReflexVLA maintains competitive performance on standard static manipulation tasks.
  • They evaluate ReflexVLA’s performance in real-world manipulation tasks.

A. Setup

The setup compares ReflexVLA against lightweight, dynamic-manipulation, and large-scale VLA baselines using success rate, model size, and inference latency. Policies share a six-task dataset and standardized asynchronous evaluation conditions.

  • Baselines: ReflexVLA is compared with lightweight VLA-Adapter and SmolVLA, dynamic DynamicVLA and PUMA, and large-scale OpenVLA-OFT and π0.5.The baselines span representative model scales and dynamic-manipulation methods.
  • Metrics: Task performance is measured by task success rate, while model size indicates deployment cost and end-to-end inference latency is used in the latency ablation.
  • Implementation Details: All VLA policies are trained as a single policy on the same dataset, with 200 demonstration episodes for each of the 6 tasks.
  • Implementation Details: ReflexVLA uses 2 consecutive temporal observations and λfuture = 0.05, while all policies use asynchronous inference with an action chunk size of 8 and an action horizon of 2.Evaluation runs on a single NVIDIA RTX 5880 Ada GPU, with each success rate computed over 150 evaluation episodes per task and reported across three runs.

B. Main Results

Main results show that high-frequency asynchronous inference with larger action chunks and shorter horizons is most effective for reaction-critical manipulation. ReflexVLA achieves the highest average success rate on ReflexBench with a 1B-parameter model while retaining competitive performance on LIBERO.

  • Inference and control settings: High-frequency asynchronous inference increasingly outperforms synchronous inference as observation-action mismatch decreases.At low frequencies, asynchronous inference is particularly harmed by mismatch in rapidly changing environments; higher frequencies enable continuous action execution.
  • Inference and control settings: Larger chunk sizes and shorter action horizons generally improve asynchronous performance at 30 Hz, leading to the choice of chunk size 8 and action horizon 2.The optimal chunk size varies slightly across tasks, but these settings are used in subsequent ReflexBench experiments for fair comparisons.
  • ReflexBench results: 50.4% average success rate makes ReflexVLA the highest-performing method on ReflexBench while using only a 1B-parameter model.The result is reported across all six ReflexBench tasks and exceeds the evaluated baselines.
  • Static benchmark results: 97.2% average success rate shows that ReflexVLA remains competitive on the static LIBERO benchmark despite being designed for dynamic manipulation.Its performance is comparable to the best-performing VLAs, with only a marginal difference from the VLA-Adapter backbone.

C. Ablation Study

The ablation study progressively adds latent future prediction, temporal fusion, and inference-latency optimization to ReflexVLA. These components improve success rate while reducing deployment latency for reaction-critical manipulation.

  • Progressive ablation: ReflexVLA’s progressive ablation introduces each proposed component incrementally from a baseline model.The study evaluates latent future prediction, temporal fusion, and inference-latency optimization in sequence.
  • Latent future prediction: Frozen DINOv3 features stabilize latent future prediction and improve success rate from 36.8% to 62.8% with almost no additional inference overhead.Trainable visual targets instead cause substantial performance degradation because representation and prediction objectives are jointly optimized.
  • Temporal fusion: Temporal fusion on intermediate visual features improves motion understanding.The paper evaluates three temporal fusion strategies after adding the stabilized latent future prediction module.
  • Inference latency optimization: Batched visual encoding and CUDA Graph replay increase success rate to 73.8% while reducing inference latency from 125.1 ms to 65.0 ms.The result supports the importance of deployment-latency reduction for reaction-critical manipulation.

D. Real-World Experiments · E. Limitations · VI. CONCLUSION

Real-world experiments on an AgileX Piper arm show ReflexVLA outperforming SmolVLA across three dynamic tasks and matching PUMA overall despite a much smaller model. The paper concludes with ReflexBench and ReflexVLA while identifying fine-tuning-only temporal components and limited inference paradigms as key limitations.

  • D. Real-World Experiments: All policies are trained with teleoperation demonstrations, comparing ReflexVLA against SmolVLA and PUMA.Each policy is evaluated 20 times on every task.
  • D. Real-World Experiments: ReflexVLA consistently outperforms SmolVLA across all three real-world tasks.The paper attributes this performance to temporal modeling and latency-aware designs.
  • D. Real-World Experiments: Despite being much smaller than PUMA, ReflexVLA achieves comparable overall performance on dynamic real-world manipulation tasks.These results are reported in Table IV, with representative executions shown in Figure 6.
  • E. Limitations: Latent future prediction and multi-frame temporal fusion are introduced only during fine-tuning rather than large-scale pretraining, potentially limiting their full potential.This is identified as the first limitation requiring future investigation.
  • E. Limitations: The study evaluates naive synchronous and asynchronous inference but does not explore more advanced mechanisms such as RTC.This is identified as a second limitation.
  • VI. CONCLUSION: The paper presents ReflexBench with six dynamic tasks and a latency-aware framework decoupling simulator stepping from robot control.It also presents ReflexVLA, which integrates latent future prediction, multi-frame temporal fusion, and inference optimization for dynamic manipulation.
Loading 2608.14379v1…