PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware

Borui Xu, Hongshi Tan, Wenqi Jiang, Bingsheng He, Ming Wu, Jason Zeng, Michael Heinrich

arXiv:2610.34717v2cs.ROcs.AI

TL;DR

Prior VLA edge-inference studies cover few action-generation forms, at most three phases, and no same-model optimization comparisons, leaving latency design unclear for closed-loop robots. VLA-HWBench characterizes the complete observation-to-action path of state-of-the-art VLA models on edge devices, timing seven stages from CPU preprocessing through residual overhead. Across 18 open models on two edge platforms, latency follows action-generation iterations rather than model size, with parameter count explaining at most 15% of variance.

  • Problem

    Prior VLA edge-inference studies cover few action-generation forms, at most three phases, and no same-model optimization comparisons, leaving latency design unclear for closed-loop robots.

    Source: §1 · pp. 1–2

  • Method

    VLA-HWBench characterizes the complete observation-to-action path of state-of-the-art VLA models on edge devices, timing seven stages from CPU preprocessing through residual overhead.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across 18 open models on two edge platforms, latency follows action-generation iterations rather than model size, with parameter count explaining at most 15% of variance.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Future edge VLAs should reduce action-generation iterations, extend kernel fusion and low-bit execution to more models and platforms, and prioritize memory bandwidth and a fast host CPU.

    Source: §1 · p. 3

    The kernel-fusion and 4-bit-weight backend is available only for π0.5 and GR00T N1.7 on Jetson.

    Source: §1 · p. 2

Abstract

from arXiv · show

Vision-language-action (VLA) models increasingly run on edge devices aboard robots, where inference latency constrains closed-loop robot control. However, it remains unclear how to design VLA models, their inference software and edge hardware so that robots can run VLA inference in real time. Most VLA models report only task success, while prior studies cover limited action-generation strategies, divide inference into at most three stages, and lack comparisons of inference optimizations on the same models. Therefore, we introduce VLA-HWBench, a benchmark that measures 18 open VLA models on NVIDIA Jetson AGX Thor and AMD Ryzen AI Max+ PRO 395, with a seven-stage inference breakdown. Our study reveals three insights. First, action-generation strategy matters more than model size. It dominates 14 of 19 Jetson configurations, latency varies by $82\times$ on Jetson and $157\times$ on AMD, and parameter count explains at most 15% of latency variance. Models that generate long action chunks in a few flow-matching steps keep up with control, whereas token-by-token decoding does not. Second, kernel fusion with 4-bit weights helps most, running $\pi_{0.5}$ and GR00T N1.7 2.6-2.9$\times$ faster than PyTorch with torch.compile, but it exists for only these two models and only on the Jetson. Optimization also moves the time to the stages it shortens least, as CPU preprocessing grows from 12% to 39% of the GR00T N1.7 call. Third, after optimization, memory bandwidth limits only the stages that process few tokens on the Jetson, such as the denoising steps, whereas software limits all other stages, from unfused kernels to serial preprocessing on the host CPU. Overall, low-latency edge VLAs should use few action-generation iterations, inference software should extend kernel fusion and low-bit weights to more models and platforms, and edge hardware needs high memory bandwidth and a fast host CPU.

1 INTRODUCTION

Low-latency edge inference matters because observations can become stale before actions execute. VLA-HWBench profiles the full inference path and finds that action-generation strategy, optimization, and platform resources shape latency more than model size alone.

1 INTRODUCTION

  • Prior studies cover few action-generation forms, divide inference into at most three phases, and do not compare optimization gains on the same models.They also omit CPU preprocessing timing.Source: §1 · p. 2
  • VLA-HWBench measures 18 open models on Jetson AGX Thor and Ryzen AI Max+ PRO 395, breaking each call into seven timed stages and testing optimizations on two models.The benchmark uses a common reference configuration and timing boundary, with optional optimizations disabled.Source: §1 · p. 2, Figure 2 · p. 3
  • Action generation dominates 14 of 19 Jetson configurations; latency spans 77 ms to 6.3 s on Jetson (82×) and 114 ms to 17.9 s on AMD (157×), with parameter count explaining at most 15% of variance.Figure 1 plots end-to-end latency against loaded parameter count across both platforms.Source: §1 · p. 2, Figure 1 · p. 2
  • On Jetson, kernel fusion with 4-bit weights makes π0.5 and GR00T N1.7 2.6–2.9× faster than compilation, but is available only for these two models.CPU preprocessing grows from 12% to 39% of the GR00T N1.7 call after optimization.Source: §1 · p. 2
  • After optimization, memory bandwidth limits few-token stages on Jetson, while unfused kernels, missing low-precision support, and serial CPU preprocessing constrain other stages.Vision encoding and long prefills reach only 6–42% of the hardware's allowed performance.Source: §1 · p. 3
  • Future edge VLAs should reduce action-generation iterations rather than simply shrink models, while software extends fusion and low-bit execution and hardware prioritizes bandwidth and host-CPU speed.The paper also calls for optimizing host preprocessing alongside GPU execution.Source: §1 · p. 3

2 RELATED WORK

Prior work studies VLA action-generation strategies, edge inference, and runtime optimizations, but typically evaluates limited model sets or targeted backends. VLA-HWBench compares optimization options stage by stage across fixed models and two platforms.

2 RELATED WORK

  • Released VLAs generate actions through flow matching, parallel decoding, token-by-token decoding, or reasoning traces, but are commonly compared by task success rather than inference cost.VLA-HWBench measures the inference cost of each action-generation form on edge hardware.Source: §2 · p. 3
  • Prior edge-inference work profiles limited sets of models or architectures, while optimization backends are evaluated only on the models they target.The studies differ in model coverage, action-generation stages, platforms, backends, and weight precision.Source: §2 · pp. 3–4
  • Runtime options include fused-kernel backends, C++ runtimes, compilation, and lower precision; VLA-HWBench compares released-model options stage by stage on fixed weights across models and platforms.Related work also explores compressed action tokens, fewer denoising steps, real-time chunking, and asynchronous inference.Source: §2 · pp. 3–4

3 STUDYING VLA INFERENCE LATENCY ON EDGE DEVICES

The study treats a VLA call as observation-to-action latency divided into seven stages, then defines metrics for latency, optimization gains, and bottlenecks. Its design compares reference configurations across two platforms and attributes timings to model stages.

3.1 RESEARCH PROBLEM

  • A VLA maps camera images, robot state, and language instructions to an H-action chunk; call latency runs from host-side observation to host-side output across seven stages.Action generation runs N sequential iterations, with N varying by generation strategy.Source: §3.1 · p. 4

3.2 METRICS

  • Q1 compares stage shares, call latency, and time per action against LIBERO’s 50 ms control interval; Q2 measures optimization speed-up and action consistency.A model keeps up only when T/H < 50 ms.Source: §3.2 · p. 4
  • Q3 uses a roofline model to distinguish hardware-bound GPU stages near compute or memory limits from software-bound stages far from those limits.The memory-versus-compute boundary is the ridge point P/B.Source: §3.2 · p. 4

3.3 DESIGN PRINCIPLES

  • The design compares each model in its own unoptimized reference configuration, adds optimizations one at a time, and synchronizes stage boundaries to check attribution.Stages captured in one CUDA graph are separated by the graph’s kernels.Source: §3.3 · p. 4
  • The two platforms pair Arm and x86 hosts with GPUs that differ 4.4× in peak 16-bit throughput but only 1.1× in memory bandwidth.Table 2 records loaded-model parameters, weight types, and LIBERO evaluation chunk sizes.Source: §3.3 · p. 4, Table 2 · p. 5

4 VLA-HWBENCH BENCHMARK DESIGN

VLA-HWBench covers open models, multiple runtime backends, two edge platforms, and a staged measurement protocol. It measures latency on a common workload and profiles GPU and CPU work to attribute costs.

4.1 MODEL COVERAGE

  • The benchmark selects 18 open models released in 2025 or 2026 that cover four action-generation forms; MolmoAct2’s two action heads yield 19 reference configurations.Models use their LIBERO-10 checkpoints and release-specific evaluation settings.Source: §4.1 · p. 5

4.2 RUNTIME BACKENDS

  • Four runtime backends range from PyTorch-based LeRobot and authors’ code to FlashRT’s fused kernels and TensorRT’s compiled inference engines.FlashRT offers FP8 and NVFP4 weights; the authors’ code uses each release’s pinned library versions.Source: §4.2 · p. 5

4.3 PLATFORMS

  • The benchmark runs on Jetson AGX Thor and Ryzen AI Max+ PRO 395, whose GPUs share memory with their CPUs.Table 3 lists platform specifications.Source: §4.3 · p. 5, Table 3 · p. 6

4.4 MEASUREMENT PROTOCOL

  • Figure 3 breaks down the 19 reference configurations by stage on both platforms, assigning each fused stage to the stage that contains it.Reference runs use the stated input and timing protocol.Source: Figure 3 · p. 6, §4.4 · p. 6
  • Each configuration receives the same LIBERO-10 observation and up to 100 synchronized timed calls; the reported latency is the median.The protocol also tests 24 frames from one episode, times model stages, profiles GPU kernels, and measures CPU preprocessing substeps.Source: §4.4 · p. 6

5 BENCHMARK RESULTS

The benchmark traces VLA latency across action-generation strategies, optimizations, and hardware bottlenecks. Latency varies widely, and its dominant costs shift as software optimization shortens some stages more than others.

5.1 WHERE DOES THE TIME GO IN VLA MODELS, AND WHAT SETS THEIR LATENCY?

  • Latency spans 77 ms (X-VLA) to 6.3 s (VLA-0) on Jetson (82×) and 114 ms to 17.9 s on AMD (157×); only 10/19 and 9/19 configurations keep LIBERO pace.Figure 4 summarizes the reference-config latency distributions and their relationship to action-chunk size H.Source: Figure 4 · p. 7, §5.1 · p. 7
  • Action generation is the largest stage in 14 of 19 Jetson configurations, consuming 42–99% of the call; parallel decoders and Evo-1 are exceptions.π0.5 on AMD also spends more time in vision encoding than action generation.Source: §5.1 · p. 7
  • Only 9% of Jetson and 15% of AMD latency variance is explained by parameter count; latency instead follows action-generation iterations.On Jetson, a flow-matching step costs 4–107 ms, while autoregressive decoding costs 20–52 ms per token.Source: §5.1 · p. 7

5.2 HOW MUCH DO OPTIMIZATIONS HELP, AND WHERE DOES THE TIME MOVE?

  • Optimization makes π0.5 7.2× and GR00T N1.7 2.6× faster on Jetson; kernel fusion with 4-bit weights gives the largest gain but is available only for these models there.Figure 5 shows optimization stages and the resulting latency and stage-time changes.Source: Figure 5 · p. 8, §5.2 · p. 8
  • GR00T N1.7 CPU preprocessing grows from 12% to 39% of the call, while optimization widens the AMD-to-Jetson latency gap from 2.0–2.5× to 5.7–7.8×.Figure 5c,d shows that stage times shrink unevenly.Source: §5.2 · p. 8

5.3 WHAT LIMITS THE REMAINING TIME AFTER OPTIMIZATION?

  • On Jetson, denoising steps that process 10–41 tokens are memory-bound, operating far below the FP8 and NVFP4 ridge points.FlashRT steps take 1.6–1.8× the weight-read bound; the AMD steps remain 2.8–3.3× below the roof.Source: §5.3 · p. 8
  • Prefill and vision encoding remain software-limited in several configurations: long prefills incur kernel overhead, and vision towers miss hardware throughput due to format and operator support or kernel costs.Short GR00T N1.7 prefills are memory-bound on Jetson, while its AMD GEMMs use a fifth of peak bandwidth.Source: §5.3 · p. 9
  • CPU preprocessing is the largest GR00T N1.7 stage on Jetson at 39% of the fastest call, limited by serial host work and thread wake-ups rather than GPU execution.A third of the preprocessing time goes to waking PyTorch’s 14 worker threads; the AMD host finishes the serial work 5× faster.Source: §5.3 · p. 9

6 CONCLUSION

VLA-HWBench measures released VLA inference stage by stage across 18 open models and two edge platforms. Its results show that action-generation iterations, backend software, and platform jointly shape latency, with different bottlenecks remaining after optimization.

6 CONCLUSION

  • Across 18 open models on two edge platforms, latency differs by up to 157× and follows action-generation iterations rather than model size.A fused backend with 4-bit weights gives the largest gain, 2.6–2.9× over PyTorch with torch.compile, but is available only on Jetson for two models.Source: §6 · p. 10
  • After optimization, memory bandwidth limits few-token stages such as Jetson denoising, while software limits the rest.These insights are intended to guide edge VLA model, inference-software, and hardware design.Source: §6 · p. 10
  • VLA-HWBench measures where inference time goes stage by stage on edge devices.The benchmark covers released VLA models across two edge platforms.Source: §6 · p. 10

A ADDITIONAL BENCHMARK DETAILS

The additional benchmark details document configuration attributes, defaults, and model-use indicators for the reference models.

A.1 MODEL CONFIGURATIONS

  • Table 4 records software, camera views, chunk length H, and execution horizon He for each of the 19 reference configurations.The π0.5 and pi0-FAST checkpoints ship with LeRobot compilation enabled, and MolmoAct2 ships with CUDA graphs enabled; reference runs switch these off.Source: §A.1 · p. 14, Table 4 · p. 14
  • The configuration table distinguishes each model’s shipped software defaults from the benchmark’s reference settings.The reference configurations disable the listed compile or CUDA-graph defaults.Source: §A.1 · p. 14
  • Table 5 reports repository GitHub stars and LIBERO-checkpoint Hugging Face downloads for the 18 models.The counts are all-time totals as of 2026-09-26; a dash indicates a checkpoint whose downloads are not counted by Hugging Face.Source: §A.1 · p. 14, Table 5 · p. 15

A.2 STAGE DEFINITIONS

The appendix specifies how inference stages map to model operations and how reference latency is reported. It also defines the per-action measure used to compare latency with LIBERO’s execution rate.

A.2 STAGE DEFINITIONS

  • Vision encoding, prefill, and action generation are mapped to model-specific calls or operations, with captured-graph work apportioned by kernel time.For LeRobot, prefill is the language-model call that fills the key–value cache; action generation is the remainder of the policy call.Source: §A.2 · p. 15, Table 6 · p. 15

B LATENCY OF THE REFERENCE CONFIGURATIONS

  • Time per action divides latency by chunk size H; values above 50 ms indicate actions are generated more slowly than LIBERO executes them.The table also supplies the latency values underlying Figures 1 and 4a,b.Source: B LATENCY OF THE REFERENCE… · p. 15
  • Table 7 reports median latency, 5th–95th percentile, time per action, and AMD-to-Jetson latency ratio for the 19 configurations.Configurations are sorted by Jetson latency.Source: Table 7 · p. 16

C STAGE TIMES

The appendix reports instrumented stage times on both platforms, linear step/view sweeps, and frame-to-frame latency spread.

C STAGE TIMES

  • Stage-time tables cover Jetson and AMD reference configurations; for LeRobot, a second instrumented run splits policy-call time into vision encoding, prefill, and action generation.The second run's median latency is within 3.1% of the first run.Source: C STAGE TIMES · p. 16, Table 8 · p. 16, Table 9 · p. 17

D STEP AND VIEW SWEEPS

  • Table 10 fits latency as T = a + bN over denoising steps or camera views to estimate per-step costs.Repeated sweeps are averaged, and R2 is the lowest among them; a dash marks configurations not available or not swept on that platform.Source: D STEP AND VIEW SWEEPS · p. 17, Table 10 · p. 17

E FRAME-TO-FRAME VARIATION

  • Table 11 reports the spread of per-frame median latency across 24 frames from one episode, relative to the median.Asterisked values use runs with default CUDA graphs because the graph-off configuration was measured only on the benchmark observation.Source: E FRAME-TO-FRAME VARIATION · p. 18, Table 11 · p. 18

F OPTIMIZATION DETAILS

The appendix catalogs optimization latency comparisons, compile-mode details, and action agreement and variability across optimized configurations.

F OPTIMIZATION DETAILS

  • Table 12 lists latency before and after each measured optimization and the resulting speed-up, including options for models outside the main optimization study.These include compile options, an fp32-to-bf16 cast, default CUDA graphs, and fast modes that reuse or skip reasoning; the fast modes need their own task-success evaluation.Source: F OPTIMIZATION DETAILS · p. 18, Table 12 · p. 19
  • π0.5 compiled actions differ by up to 2.0 between compiled processes on Jetson and from eager actions on AMD, versus 0.07 between repeated calls within one process; SmolVLA repeated calls differ by up to 1.96.Table 14 summarizes action agreement before and after optimization over 11 frames; FlashRT calibrates FP8 scales at load time on the benchmark observation, also the first of those frames.Source: F OPTIMIZATION DETAILS · p. 18, Table 14 · p. 20
  • Table 13 compares compile modes with eager reference configurations; GR00T N1.7 compiles only its action head, and traced frames count frames captured by the compiler.Aligned rows use FlashRT's workload of 10 actions and two image slots on both platforms.Source: F OPTIMIZATION DETAILS · p. 18, Table 13 · p. 19

G BOTTLENECK ANALYSIS

The bottleneck analysis compares GPU-stage times with compute and memory bounds and breaks stage time down by kernel class.

G BOTTLENECK ANALYSIS

  • The analysis derives floating-point operations and weight bytes from eager-run GEMM shapes, counted in each configuration's kernel number format.FlashRT uses mixed FP8 and NVFP4 weights across model components, while compiled configurations retain their loaded-model formats.Source: G BOTTLENECK ANALYSIS · p. 20
  • FlashRT computes GR00T N1.7 denoising modulation and context keys and values once per call, whereas Isaac-GR00T recomputes them each step.Stage times inside captured graphs use the relevant trace-kernel time or a share of the timed graph; GR00T N1.7's FlashRT step time is the slope of its step sweep.Source: G BOTTLENECK ANALYSIS · p. 20
  • Table 15 compares each fastest-configuration GPU stage with compute and memory bounds, while Figure 7 splits stage time by kernel class and normalizes it to the larger bound.Action generation is measured per denoising step.Source: Table 15 · p. 21, Figure 7 · p. 21

Related papers

Loading 2610.34717v2…