Source-linked AI summary
How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perf
Wenqi Jiang, Jason Clemons, Karu Sankaralingam, Christos Kozyrakis
TL;DR
VLA inference must satisfy real-time constraints, yet its performance remains poorly understood across the combinatorial space of model and system configurations. The paper introduces VLA-Perf, an analytical performance model, and uses it to systematically study model and deployment choices. The study distills 15 takeaways for designing future VLA models and inference systems, while limiting its scope primarily to manipulation-task inference and excluding several real-world latency sources.
Problem
VLA inference performance is poorly understood across the combinatorial space of model architectures, accelerators, execution locations, and networks, despite real-time requirements.
Method
The paper develops VLA-Perf, an analytical roofline-based model that predicts optimal latency and throughput for arbitrary model–system combinations.
Results
The systematic study covers model and system configurations and distills 15 key performance takeaways for future VLA models and inference systems.
Takeaways & Limitations
Designing real-time VLA systems requires jointly considering model scaling, architecture, context, execution strategy, deployment placement, hardware, and network conditions.
Takeaways & Limitations
The study focuses primarily on manipulation-task VLA models and excludes robot execution latency and sensor latency from its inference analysis.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have recently demonstrated impressive capabilities across various embodied AI tasks. While deploying VLA models on real-world robots imposes strict real-time inference constraints, the inference performance landscape of VLA remains poorly understood due to the large combinatorial space of model architectures and inference systems. In this paper, we ask a fundamental research question: How should we design future VLA models and systems to support real-time inference? To address this question, we first introduce VLA-Perf, an analytical performance model that can analyze inference performance for arbitrary combinations of VLA models and inference systems. Using VLA-Perf, we conduct the first systematic study of the VLA inference performance landscape. From a model-design perspective, we examine how inference performance is affected by model scaling, model architectural choices, long-context video inputs, asynchronous inference, and dual-system model pipelines. From the deployment perspective, we analyze where VLA inference should be executed -- on-device, on edge servers, or in the cloud -- and how hardware capability and network performance jointly determine end-to-end latency. By distilling 15 key takeaways from our comprehensive evaluation, we hope this work can provide practical guidance for the design of future VLA models and inference systems.
1 Introduction
VLA inference must meet real-time constraints, but its performance landscape spans a large combinatorial space of models and deployment systems. VLA-Perf enables systematic analysis of these combinations and yields 15 practical insights for future VLA models and serving systems.
- Research gap: Real-time VLA inference is difficult to characterize because performance depends on combinatorial choices across models, accelerators, execution locations, and networks.The same VLA model can exhibit performance differences of multiple orders of magnitude across inference systems.
- Real-time target: The study defines 10 Hz as acceptable inference frequency and 100 Hz as high-performance, relative to typical 24–60 Hz camera rates.These thresholds frame the paper’s real-time performance analysis.
- Study scope: The evaluation studies model scaling, architectural choices, long-context inputs, asynchronous inference, dual-system pipelines, deployment placement, hardware, and networks.The study summarizes its findings as 15 key performance takeaways.
- VLA-Perf: VLA-Perf is an analytical, roofline-based model that predicts optimal inference latency and throughput for arbitrary model–system combinations.It supports varying model sizes and architectures as well as diverse deployment configurations.
- Coverage: VLA-Perf supports analysis across stateless and long-context inference, action chunk sizes, asynchronous inference, and diverse hardware, locations, and network configurations.The tool is open-sourced for further performance analysis.
2 Background and Motivation
VLA systems combine visual-language models, action-generation mechanisms, and deployment choices that jointly determine real-time inference performance. Existing efficiency techniques reduce computation or overlap inference with execution, motivating a comprehensive performance analysis across model and system configurations.
- VLA models: VLA models support embodied agents by combining visual perception and language understanding with physical action generation.Recent systems have shown strong capabilities in manipulation tasks using robotic arms and humanoid robots.
- Model architecture: VLA architectures use either autoregressive action generation or a VLM backbone paired with a diffusion-based action expert.Diffusion-style models iteratively refine actions using a configurable number of denoising steps.
- Action prediction: Action chunking predicts multiple future actions in one inference, with chunk size denoting the sequence length.This can reduce the required inference frequency while supporting smooth execution.
- Real-time constraint: Real-time VLA inference targets 10–100 ms latency to match visual signal ingestion.The system should aim to provide action guidance at the camera ingest rate.
- Efficiency techniques: Efficiency techniques include smaller or quantized models, layer skipping, fewer denoising steps, parallel decoding, action chunking, and inference–execution overlap.Dual-system pipelines can run a lightweight action expert more frequently than a larger VLM backbone.
- Research gap: A comprehensive study is needed because VLA performance varies across on-device, edge-server, and cloud-server deployments and across hardware and network configurations.Exhaustive empirical evaluation is costly because the model–system space includes many accelerator, location, network, and future-model variants.
3 Analyzing VLA Inference Performance with VLA-Perf
VLA-Perf is a roofline-based analytical model for estimating VLA inference performance across arbitrary model and system combinations. It parameterizes model computation, hardware capabilities, and network movement, then validates its predictions against optimized π0 inference.
- Overview: VLA-Perf analyzes performance across arbitrary combinations of VLA models and inference systems under an accuracy-sufficient deployment assumption.The analysis focuses on latency and throughput rather than task success rate.
- Modeling framework: The model represents each VLA component as operators and estimates end-to-end latency from model computation plus data movement across components.Model components may exchange images, vision tokens, KV caches, or action predictions locally or over a network.
- Inputs: VLA-Perf exposes model parameters including architectures, sequence lengths, denoising steps, action chunk size, and action dimensionality.System parameters include accelerator peak FLOP/s, memory bandwidth, and network specifications.
- Operator modeling: Each operator’s latency is modeled with a roofline approach using computation and memory-access characteristics of the operator and hardware.The model accounts for FLOPs, accessed bytes, peak compute throughput, and memory bandwidth.
- Validation: 73.3∼82.6% of the theoretical roofline is achieved by optimized π0 inference, with the gap narrowing when processing three camera frames.Validation compares VLA-Perf predictions with Triton-based RTX 4090 measurements.
- Limitations: VLA-Perf abstracts hardware microarchitecture and software overheads, including instruction scheduling, kernel launches, operating-system interference, and runtime-library costs.It assumes maximum theoretical compute capability and memory bandwidth are attainable for every operator.
4 Evaluation and Takeaways
The evaluation uses VLA-Perf to study model-design and deployment questions governing real-time inference. It organizes experiments around model scaling and configuration, inference placement, hardware, networking, and achievable inference rates.
- Model design: The evaluation examines how future VLA models can meet real-time latency constraints.The model-side questions cover scaling, context length, action-expert architecture, denoising, action chunking, and asynchronous or dual-system inference.
- Model design: The study asks whether VLA models can process thousands of visual frames while remaining practically feasible.Long-context video inputs are treated as a distinct model-design question.
- Model design: The evaluation compares autoregressive and diffusion-based action experts and studies the effects of denoising steps and action chunk size.These questions target how action-generation choices affect inference performance.
- Deployment: The systems analysis considers on-device, server, and device-server collaborative inference across available hardware capabilities and network performance.It asks how deployment choices support workloads ranging from 10 Hz to 100 Hz.
- Evaluation organization: Experiments begin with a π0 baseline, then vary model configurations before incorporating inference placement and network latency for deployment-oriented analysis.The organization spans §§4.2–4.11.
4.1 Evaluation Setup
The evaluation spans VLA model variants, GPU capabilities, inference locations, and network settings, using latency and throughput as the principal performance measures. Baseline experiments characterize π0 across hardware and analyze compute- versus memory-bound behavior.
- Models and inference systems: The study evaluates π0-derived VLA models on edge, consumer, and datacenter GPUs across multiple inference locations.The considered hardware includes Jetson Thor, RTX 4090, A100, H100, and B100; systems vary inference location and server-side network configuration.
- Performance metrics: System latency measures elapsed time from visual observation to action prediction, while throughput measures single-robot inferences per second.For synchronous inference, throughput is the inverse of latency; asynchronous throughput can exceed that inverse, and robot execution latency is excluded.
- Baseline π0 performance: 61.7–314.4 Hz: A100, H100, and B100 match or exceed common RGB-camera rates, whereas Jetson Thor reaches 19.0 Hz.The datacenter-GPU range is at least comparable to 24–60 Hz camera frame rates, while Jetson Thor falls below most camera rates.
- Hardware bottlenecks: Action prediction is memory-bound across hardware, while vision and VLM inference are generally compute-bound except on Jetson Thor.Vision and VLM components have higher operator intensity than the action expert, reflecting their processing of many more tokens.
- Scaling context: Larger model sizes increase inference latency proportionally across VLA components.The evaluation measures component-level latency while scaling model variants, including larger vision encoders, language models, and action experts.
4.3 Scaling Model Sizes Under Real-Time Constraints
Scaling VLA models increases component latency approximately linearly, but datacenter GPUs retain real-time capability for substantially larger models than edge and consumer GPUs. On B100, even the largest evaluated model remains near the 10 Hz real-time target.
- Experimental design: The section evaluates scaled vision encoders, language models, and action experts derived from the π0 architecture.The language-model family includes 7B, 13B, and 70B variants, while the vision encoder is expanded to a 1.1B-parameter model.
- Model scaling: Latency of each VLA component scales approximately linearly with increasing model size.Across components, larger models impose proportionally higher computational costs and inference latency.
- Hardware limits: 9.6 Hz: B100 sustains real-time inference for the largest 81B model, which is 30× larger than π0.The result demonstrates that modern datacenter GPUs can accommodate substantially larger VLA models under real-time constraints.
- Hardware limits: RTX 4090 runs out of memory for π0-XL at 16.7B parameters, while Jetson Thor reaches only 2.1 Hz on π0-XL.Jetson Thor struggles to provide real-time performance even when sufficient memory capacity is available.
4.4 Long-Context VLA Inference
Long-context VLA inference becomes increasingly difficult as accumulated visual history grows, with datacenter GPUs supporting substantially longer real-time contexts than edge and consumer GPUs. The study also finds denoising steps, rather than action chunk size, dominate diffusion-VLA latency.
- Long-context scaling: Jetson Thor and RTX 4090 support real-time long-context inference only to roughly 100 past timesteps.At this context length, their inference frequency is around 8 Hz, below the paper’s 10 Hz acceptable target but near it.
- Long-context scaling: 11.7 Hz: B100 sustains real-time inference with 1K past timesteps, but falls to 1.2 Hz at 10K steps.The 10K-step setting no longer meets the paper’s real-time requirement.
- Diffusion controls: Denoising steps substantially affect action-expert and end-to-end latency, whereas action chunk size has only a marginal effect.Each denoising step incurs a forward pass, while varying action chunk size across the evaluated range changes latency negligibly.
- Diffusion controls: 5×: Increasing diffusion steps from 10 to 50 raises action-prediction latency proportionally and increases overall VLA latency by 2.15×.This example uses the default action chunk size of 50.
4.6 Diffusion-Based vs. Autoregressive Action Prediction
Diffusion-based VLAs generally provide much lower inference latency than vanilla autoregressive VLAs when action chunking is used. Autoregressive models become competitive for small action-token counts or with parallel decoding.
- Diffusion-based VLA inference with action chunking is one to two orders of magnitude faster than vanilla autoregressive inference.On B100, the standard diffusion model reaches 3.2 ms latency with chunk size 50, versus 327.6 ms for the classic autoregressive model.
- 3.2 ms is 102.4× faster than the classic autoregressive model’s 327.6 ms latency at chunk size 50.
- Autoregressive VLAs are also competitive when parallel decoding is enabled.
- Autoregressive VLAs can slightly outperform large diffusion models when generating a small number of action tokens, although standard diffusion remains faster.The comparison considers action dimensionalities from 7 DoF to over 40 DoF without action chunking.
4.8 Device-Server Collaborative Inference
Device–server collaboration generally underperforms server-only inference and often underperforms device-only inference because transferring the VLM KV cache can dominate latency. Asynchronous inference instead improves server-side throughput, particularly over slower wireless networks.
- Device–server collaboration is always slower than server-only inference and usually slower than device-only inference.The exception to the latter comparison is a fast wired Ethernet 10G connection.
- KV-cache downloads make collaboration especially slow over wireless networks, taking 12.4 ms on Ethernet 10G, 43.7 ms on WiFi 7, and 257.7 ms on 5G.
- Asynchronous throughput is bounded by the lower of GPU inference throughput and network transmission throughput.
- Asynchronous inference improves server-side throughput by 2.63∼5.99× under WiFi 7, 5G, and 4G, while wired networks show similar synchronous and asynchronous throughput.
- System 1–System 2 asynchrony yields hardware- and network-dependent gains, reaching 2.24× on B100 with fast 10G Ethernet at a 10 Hz cap.On Jetson Thor, the gains are 1.46× at 5 Hz and 1.30× at 10 Hz.
4.11 Supporting High-Performance VLA Inference up to 100 Hz
The 10 Hz target is achievable across the evaluated deployment settings with suitable hardware and networking, whereas 100 Hz requires stronger accelerators, model-level optimization, or asynchronous inference. Cloud deployments particularly depend on asynchrony for high-frequency operation.
- Jetson Thor achieves 19 Hz for π0, exceeding the 10 Hz target, but reaching 100 Hz requires roughly a 5× improvement.Suggested model-level changes include reducing model size, decreasing diffusion steps, and lower-precision quantization.
- An RTX 4090 can achieve 10 Hz over slow 4G, while 100 Hz requires B100-class acceleration or model-level optimization and faster networking.For B100, 100 Hz further depends on wired Ethernet or high-quality wireless connectivity such as WiFi 7.
- Cloud inference can reach 10 Hz with good networking, but 100 Hz generally requires asynchronous inference.Synchronous B100 cloud inference reaches only 42.8 Hz even with a fast network, whereas asynchronous execution reaches 314.4 Hz with WiFi 7 or better.
5 Conclusion and Future Work
The paper presents VLA-Perf as a comprehensive analytical study of VLA inference across model and system configurations, distilling practical guidance for real-time design. Its scope is limited to manipulation-focused inference and excludes broader end-to-end robotic latency.
- VLA-Perf systematically studies model configurations and deployment systems across hardware, inference placement, and network conditions.
- The study distills 15 takeaways intended to guide future VLA model and inference-system design.
- The evaluation focuses primarily on manipulation tasks and excludes domains such as autonomous driving, quadrupeds, and drones.Those settings may involve stronger on-device requirements and additional components such as SLAM and specialized control modules.
- The analysis omits robot execution and sensor latency, so it does not model complete end-to-end robotic behavior.
A Detailed System and Model Parameters
This section presents the hardware performance configuration used in the evaluation and the parameter specifications of the π0 model components.
- The section organizes the evaluation setup around detailed hardware performance and model-parameter configurations.
- The evaluation uses GPU hardware configurations documented in Table 10.
- Table 11 reports parameter specifications for π0 model components, excluding the vocabulary table.