Running VLAs at Real-time Speed

• The falling pen leaves little time to initiate the catch after it is observed.Source: Figure 1 · p. 1
• 30 FPS inference lets the VLA process every frame in the camera stream.Source: Figure 1 · p. 1
• End-to-end reaction time can be shorter than 200 ms, which the passage compares with an average human in this test.Source: Figure 1 · p. 1

• A VLA forward pass typically takes hundreds of milliseconds, while processing every 30-FPS frame requires latency below about 33 ms to avoid dropped-frame delays.Source: §1 · p. 1
• 27.3 ms inference with two input views on a single RTX 4090 is reported as significantly faster than openpi's official inference.Table 1 assumes empty prompt text and a 63-chunk length.Source: §1 · p. 1, Table 1 · p. 1
• The trained π0 policy achieved a 100% success rate catching a falling pen released from a higher disturbed position.The experiment used hundreds of grasping data collected by automatic rules.Source: §1 · p. 2
• The proposed Full Streaming Inference mode maps VLA input and output frequencies into a control system capable of generating signals at up to 480 Hz.Source: §1 · p. 2
• Extending the framework to more challenging real-time tasks remains future work, including possible 1–10 Hz high-level VLM planning and multisensory inputs to the action expert.Source: §1 · p. 2

• π0 combines a vision-language model (VLM) with an action expert (AE) coupled through a mixture-of-experts architecture.The VLM backbone is initialized with 3B-parameter PaliGemma; the AE has 300M parameters and uses flow matching to predict action chunks.Source: §2 · p. 2
• The inference breakdown begins from naive PyTorch, removes redundant computation and CPU overhead, then optimizes kernels and establishes a lower bound.Source: Figure 2 · p. 2
• Multiview images and the task prompt route to the VLM, while robot states and action noises route to the AE.Source: §2 · p. 2

3. Eliminating the Overheads

• The naive PyTorch implementation takes over 100 ms; the first optimization steps target CPU overhead and redundant computation.Source: §3 · p. 2

3.1. Removing CPU Overhead

• CUDA-graph replay launches recorded kernels through the GPU and driver, removing Python execution overhead and speeding inference by around twofold.The approach relies on constant kernel code and buffer pointers; the transformer blocks have no dynamic branches.Source: §3.1 · p. 3

3.2. Simplifying the Graph

• Graph simplifications fold RMS affine parameters into a linear layer, fold action-time projections, and fuse QKV projections, reducing operators, MACs, and kernel launches.The passage reports these modifications reduce inference time by 7–8 ms.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3. Avoiding Other Overheads

• System-level measures include choosing a camera resolution near 224×224, hand-writing resize code, using pinned memory and static CPU buffers, and zero-copy camera frames.Correctly implemented image resizing takes less than 60 μs on desktop x86 CPU; the paper does not count this time in the rest of its measurements.Source: §3.3 · p. 3

4.1. Tuning Tile Parameters of GEMM

• About 1.5 ms is saved by manually tuning GEMM tiling with Triton; omitting the last LLM attention and FFN layer saves a further 0.7 ms.The LLM runs 17 rather than 18 attention and FFN layers because only KV caches are passed to the action expert.Source: §4.1 · p. 4

4.2. Fusing Gated Linear Layers

• 1.7 ms: Fusing the two gated FFN projections lets their matmuls run in parallel and coalesces memory operations, improving inference performance.Only the combined result is written back, reducing memory operations and improving tensor utilization.Source: §4.2 · p. 4

4.3. Partial Split-k

• A 512 × 1152 × 1152 GEMM is split into independent 512 × 1152 × 1024 and 512 × 1152 × 128 matmuls, gaining less than 0.1 ms.The first part uses 64 × 64 tiles; the second uses 32 × 32 blocks with split-2 along K to distribute work across the 128 SMs.Source: §4.3 · p. 4

4.4. Fusing the Scalar Operations

• Fusing bias, residual, and activation operations into GEMMs, and handling RMS norm statistics separately, reduces memory footprint; this step is attributed roughly 4 ms of remaining gain.The authors say the gain from this step is hard to estimate.Source: §4.4 · p. 5

5.1. Roofline of the GEMMs

• The GEMM roofline lower bound is the maximum of memory-operation and compute time, using HBM bandwidth and tensor-core cycles.The estimate counts memory traffic for the second matrix, since activations can reside in L2 while network parameters are too large to cache fully; the authors sum operator bounds because most vision-encoder and LLM operations are compute-limited while most action-expert operations are bandwidth-limited.Source: §5.1 · pp. 5–6

5.2. Synchronization Overhead

• For 1378 matmuls, the CUDA-graph estimate of synchronization overhead is 1.72 ms, while a software barrier reduces the estimate to 0.86 ms.The software-barrier approach is faster in the estimate, but operator fusion with it produced negative performance gains, so it serves only as a lower-bound indicator.Source: §5.2 · p. 6
• At most 30% improvement remains relative to measured performance, suggesting the implementation is close to the authors’ estimated optimum.The comparison uses the performance in Fig. 2 and lower bounds updated to include 0.86 ms synchronization overhead.Source: §5.2 · p. 6

6.1. Overlapped Streaming Inference

• Concurrent execution of IO-bound AE and compute-bound VLM kernels can support 30 VLM runs and 480 AE runs per second, motivating an additional 480 Hz control loop.The proof of concept uses two CUDA streams and older KV cache for parallel AE kernels; its timing approaches the 1/30-second bound.Source: §6.1 · pp. 6–7

6.2. Re-thinking Action Expert: Up to 480 Hz

• A 480 Hz control loop requires high-frequency inputs to the AE, which can gradually generate action chunks and react before the VLM observes those inputs.Force sensors can sample above 2 kHz; motor current or resistance-based tactile signals are alternatives when force sensors are unavailable.Source: §6.2 · p. 7
• The proposed design pairs a 480 Hz AE stream and trajectory buffer with a 30 Hz VLM stream that supplies KV cache.AEs receive prior AE output and high-frequency sensor inputs, process up to 64 tokens, and update the trajectory buffer; the AE window can cover four frames when tokens map one-to-one to nodes.Source: §6.2 · p. 8
• The framework runs a fast sensor-to-action loop alongside a slower image-driven loop, with optimistic feedback times of 2 ms and 1/30 second, respectively.The loops run in parallel; a persistent megakernel is suggested as a possible implementation, but implementation details are outside the report's scope.Source: §6.2 · p. 8

6.3. Fusing VLMs: Going below 1Hz

• Visual encoding can share model-weight loading with text inference, yielding an additional autoregressive text stream of 30 tokens per second.The authors propose using the stream for user interaction or reasoning and say the added MACs should have little effect because visual tokens are numerous.Source: §6.3 · p. 8

6.4. Summary

• The report sketches three parallel feedback loops: force at 480 Hz, vision at 30 Hz, and text below 1 Hz, while omitting implementation details.The authors identify implementation details as work for subsequent studies.Source: §6.4 · p. 8

7.1. Setup and Data Collection

• The setup tests whether a vertically aligned second grabber closes in time to catch a released pen, with opening or closing taking 60 ms.The camera observes the second grabber and pen but not the release; its measured delay is about two frames.Source: §7.1 · p. 8
• The training set contains 600 episodes, with target-grabber states post-processed into one-dimensional trajectories.The network receives the current and previous camera frames as two views, providing a hint about pen speed.Source: §7.1 · pp. 8–9

7.2. Training and Inference

• The system uses separate camera, inference, and output threads, with the inference thread processing the latest frame and overwriting timestamped output states.The authors wait for GPU inference time to stabilize before releasing the pen.Source: §7.2 · p. 9

7.3. Result

• All 10 consecutive falling-pen catches succeed, and the authors interpret even a single catch as verification of low-latency VLA inference rather than a demanding learning result.They report that humans commonly need about 30 cm of pen travel to react, the distance used in the experiment.Source: §7.3 · p. 9
• The authors conclude that the system reacts as quickly as a human while containing billions of parameters, and suggest studying VLAs on time-critical tasks.They characterize the catching result as trivial from a learning perspective, but informative about implementation latency.Source: §7.3 · p. 9

Manually-designed Kernels

• Manual kernel optimization frameworks improve scheduling and GEMM performance, but can leave kernel-launch and repeated global-memory-movement penalties.The passage notes that attention-focused methods such as FlashAttention and FlashDecoding do not address other AE components such as FFN.Source: §8 · p. 9

Scheduling-based approaches

• Scheduling-based compilers enable loop tiling, fusion, and layout transformations, but depend on explicit per-kernel algorithms and the quality of their schedules.Source: §8 · p. 9

Superoptimization Approaches

• Tensor superoptimization searches across thread, tile, and operator levels, but the cited approaches may require long searches or deliver low acceleration ratios, limiting real-time practicality.The passage also notes Mirage's reliance on statistical equivalence checking for correctness.Source: §8 · p. 9

9.1. Visual-based Latency: Going to 60-120 FPS

• Higher camera FPS and more views would raise encoding compute; low-bit quantization is proposed because the reported results use BF16 and encoding is compute-bound.The authors identify 8-bit multiplication as a possible way to increase available compute.Source: §9.1 · p. 10
• Adaptive view selection could fuse active-camera information into fewer tokens and may boost FPS; the authors identify 60 FPS as the next stage and 120 FPS as beyond human.The discussion notes that people reportedly distinguish 30 FPS from 60 FPS playback.Source: §9.1 · p. 10

9.2. Larger Models: Going to 7B and Beyond

• 7B is proposed as the next milestone from the current 3B π0; RTX 5090 bandwidth may ease AE scaling, though its learning benefit is unclear.The authors note that the RTX 5090 has 1.79TB/s bandwidth and that the impact of scaling AEs on learning remains uncertain.Source: §9.2 · p. 10

9.3. More Fine-grained Feed-back Loop

• The finer-feedback direction remains unresolved: no method is known for layers to use the latest signal and emit intermediate outputs faster, while collecting kilo-sample-per-second demonstrations is challenging.The current construction still has a loop running faster than the AE.Source: §9.3 · p. 10