Source-linked AI summary
StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models
Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han, Zhi Hou, Hengshuang Zhao
TL;DR
Single-frame VLA models such as π0.5 lack historical context for temporal memory and precise spatial perception. StreamPI adds streaming multimodal temporal modeling without new parameters, using instruction-anchored attention and random-interval training; it consistently outperforms π0.5 across real-robot and LIBERO tasks.
Problem
Single-frame VLA models lack historical observations, limiting temporal memory and precise spatial perception in robot manipulation.
Method
StreamPI streams instruction-anchored image-text units with bidirectional intra-pair and causal inter-pair attention, using pretrained weights without additional parameters.
Results
StreamPI consistently outperforms π0.5 on real-robot precise-perception and memory-dependent tasks and on the LIBERO benchmark.
Takeaways & Limitations
StreamPI provides flexible single-frame and multi-frame temporal reasoning while preserving persistent instruction grounding during execution.
Takeaways & Limitations
Training requires loading all frames, making extremely long temporal horizons costly, and random-interval training does not fully address extreme deployment asynchrony.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
1 Introduction
StreamPI extends single-frame VLA models with streaming temporal reasoning while preserving instruction grounding and pretrained representations. It combines cached temporal context with random-interval training, and outperforms π0.5 across memory-dependent and precise-perception tasks.
- Single-frame VLA models lack historical context, limiting temporal memory and precise spatial perception.
- Bidirectional attention within image-text pairs enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference.
- Cached Key&Value representations keep streaming inference efficient, while random-interval training addresses asynchronous observation timing during deployment.
- StreamPI inherits π0.5 pretrained weights without additional parameters and supports both single-frame and multi-frame inference.
- Experiments on real-robot tasks and LIBERO show that StreamPI consistently outperforms π0.5 on spatial-precision and memory-dependent tasks.
- StreamPI treats each visual observation and language instruction as an atomic temporal unit, maintaining persistent task awareness during execution.
2 Related Work
Prior work shows that temporal context benefits robot manipulation, but existing approaches can incur high computational costs or weaken instruction conditioning. StreamPI addresses these issues with streaming inference and random-interval training for asynchronous deployment.
- Temporal context improves action consistency, task success, spatial precision, and long-horizon reasoning in robot manipulation.
- Existing multi-frame VLA methods either use costly full attention over historical tokens or decouple visual observations from language instructions.
- Streaming inference research demonstrates length extrapolation and efficient processing of long sequences without recomputation.
- StreamPI uses random-interval streaming training to improve robustness to variable-rate asynchronous observation streams on physical robots.
3 Method
StreamPI extends single-frame VLA models with streaming temporal reasoning by organizing observations and instructions into anchored pairs and applying hierarchical attention. Its parameter-free design combines cached streaming inference with interval randomization and temporal masking for asynchronous deployment.
- Instruction-Anchored Temporal Modeling: StreamPI represents each time step as an atomic pair of multi-view visual observations and its language instruction, preserving instruction anchoring across the temporal horizon.This design addresses instruction dilution when historical visual tokens accumulate.
- Instruction-Anchored Temporal Modeling: Bidirectional attention fuses vision and language within each pair, while causal attention across pairs aggregates past observations without future-frame leakage.The resulting representation conditions autoregressive action generation.
- Parameter-Free Temporal Extension: StreamPI introduces no additional parameters because its hierarchical attention is implemented by restructuring the existing backbone’s attention mask and using LLM length extrapolation.The method inherits pretrained weights and supports extended token sequences.
- Random-Interval Streaming Training: Random-interval sampling perturbs the base inter-frame interval within clipped bounds, exposing training to diverse temporal spacing and improving robustness to asynchronous observation arrival.The strategy targets the mismatch between fixed-interval training and variable-rate real-robot deployment.
- Random-Interval Streaming Training: Temporal masking randomly hides the earliest frames in a streaming sequence, simulating incremental observation patterns during training.It is combined with causal attention masking to align training with streaming inference.
- Streaming Inference: Streaming inference caches past Key&Value representations so only newly arriving frames are encoded, avoiding the growing inference cost of repeatedly processing the full observation history.Naive temporal concatenation instead causes sequence length, memory use, and latency to grow with the temporal horizon.
4 Experiments
Experiments evaluate StreamPI on real-robot manipulation, LIBERO, and CALVIN, including memory-dependent, precise-perception, and long-horizon tasks. Ablations test attention direction, interval randomization, and cross-stream generalization.
- Experimental setup: StreamPI is evaluated on real-robot tasks, LIBERO’s four suites, and CALVIN’s multi-task sequences using success-rate-based comparisons.LIBERO contains Spatial, Object, Goal, and Long suites; CALVIN evaluates sequences of up to five consecutive tasks.
- Real-robot results: 36.6% and 33.3% gains are reported on Rolling Object Grasping and Cup Hiding and Retrieval over the single-frame baseline.These memory-dependent tasks require tracking moving objects or recalling concealed-object locations across observations.
- Real-robot results: 26.7% and 32.0% gains are reported on Pen Insertion into Narrow Bottle and Cup Insertion into Cup Sleeve over π0.5.The tasks require fine-grained depth, geometric alignment, or sub-centimeter spatial precision.
- CALVIN benchmark: StreamPI (T = 5) reaches a 4.547 average sequence length on CALVIN, versus 4.313 for π0.5 and 4.090 for MemoryVLA.On the fifth task, success is 85.0% for StreamPI, compared with 79.5% for π0.5 and 69.4% for MemoryVLA.
- Ablation studies: Causal intra-pair attention reduces LIBERO-Long performance by 5.6% at T=5, while inter-pair aggregation adds 3.0% on LIBERO-Long.The ablation supports bidirectional image-text fusion within temporal units and causal aggregation across units.
- Ablation studies: Random-interval training raises average LIBERO success from 96.4 to 97.5 at T=3 and from 97.0 to 98.3 at T=5.The comparison is between fixed-interval and random-interval training under the same temporal length.
5 Conclusion
StreamPI equips VLA models with temporal awareness for robot manipulation through atomic image-text temporal units, causal attention, and random-interval training. Experiments on real-robot tasks and LIBERO show consistent gains over single-frame baselines, while the framework adds no parameters.
- StreamPI equips VLA models with robust temporal awareness for robot manipulation without introducing additional parameters.It treats image-text pairs as atomic temporal units and combines intra-pair bidirectional attention with inter-pair causal attention.
- Random-Interval Streaming Training exposes the model to diverse temporal spacings to reduce mismatch between fixed-interval training and asynchronous deployment.
- Extensive real-robot and LIBERO experiments show StreamPI consistently outperforms the single-frame baseline on perception-sensitive and memory-dependent tasks.
- The appendix covers implementation details, additional real-world results, qualitative visualizations, limitations, future directions, broader impacts, and a video demonstration.
B Implementation Details
Algorithm 1 describes streaming inference with a KV-cache: each incoming observation is paired with the language instruction, encoded using cached history, and used to predict an action chunk.
- Streaming inference receives an observation stream, language instruction, and maximum cache size T, then outputs predicted action chunks.
- Each current observation is encoded with the language instruction and cached representations to obtain a fused representation h_tn.
- When the cache would exceed T, StreamPI flushes it and resets the cache length before continuing inference.
- After updating the cache, the model predicts action A_n from h_tn and the cache, dispatches it, and advances to the next observation.
B.1 Streaming Inference
Streaming inference maintains low latency as temporal context expands on a real-robot platform, with Table 6 reporting mean latency and standard deviation across 20 trials.
- 20 trials on a real-robot platform with one NVIDIA GeForce RTX 4090 measured mean streaming-inference latency with standard deviation.
- 94.4 ± 3.4 ms is the single-frame baseline latency, while 3 frames require 97.9 ± 5.1 ms.
- 3 frames add only 3.5 ms of latency over the single-frame baseline.
- Table 6 compares inference time across different numbers of streaming frames.
B.2 Real-world Experiments
Real-world experiments evaluate StreamPI on spatial-precision and memory-dependent manipulation tasks using AgileX PiperX arms, multiple cameras, and success rate. Reported comparisons show gains over π0.5 on memory, dynamic-motion, and perception-sensitive tasks.
- Hardware Setup: Experiments use AgileX PiperX 6-DoF arms, an Aloha-style leader-follower setup, and three RealSense cameras.
- Tasks: Four real-world tasks comprise two spatial-precision tasks—cup and pen insertion—and two memory-dependent tasks—rolling-object grasping and Shell Game.
- Evaluation: Success rate is defined from binary outcomes, assigning 1 point for success and 0 points for failure.
- Memory-dependent Tasks: 33.3% improvement over π0.5 is reported for the Shell Game.
- Memory-dependent Tasks: 36.6% improvement over π0.5 is reported for picking up the rolling bottle, a dynamic task requiring temporal information to predict object motion.
- Spatial-precision Tasks: The pen-insertion task requires precise perception and temporal information, while StreamPI shows stronger perception than π0.5 for distant cup-sleeve locations.
C Visualization Comparisons
StreamPI succeeds on visualization tasks requiring precise perception, temporal information, or memory, where π0.5 struggles with spatial estimation, motion, or state tracking.
- Cup Insertion into Cup Sleeve: StreamPI accurately locates distant objects and places a cup into a sleeve, while π0.5 struggles with spatial estimation.
- Pen Insertion into Narrow Bottle: StreamPI completes pen transfer between bottles, whereas π0.5 confuses picking up the pen with inserting it because it lacks temporal information.
- Pick up the rolling bottle: StreamPI predicts object movement in advance and catches a rolling bottle that π0.5 misses because of slow reaction.
- Shell Game: Across four consecutive Shell Game trials, StreamPI accurately identifies the target-containing cup despite repeated exchanges.
D Limitations and Future Work
StreamPI remains costly for extremely long temporal horizons and does not fully resolve extreme deployment asynchrony; future work targets longer-horizon training and cache pruning.
- Limitations: Training StreamPI requires loading all frames, making extremely long temporal horizons prohibitively costly.
- Limitations: Random-interval streaming training improves robustness to variable frame rates but does not fully address extreme asynchrony during real-robot deployment.
- Future Work: Future work will support horizons longer than 100 frames through more efficient training with lower computational cost.
- Future Work: Adaptive KV cache pruning is planned to preserve representation quality for ultra-long horizons with negligible additional inference time.
E Broader Impacts
StreamPI is presented as suitable for real-world embodied tasks because its temporal modeling handles asynchronous observations without additional parameters, while broader deployment raises labor-market and safety concerns.
- StreamPI’s parameter-free streaming design and asynchronous-observation robustness lower barriers to deploying foundation models in real-robot systems.
- More capable manipulation robots may affect labor markets in routine industrial tasks.
- Robustness to real-robot out-of-distribution scenarios remains critical for avoiding safety risks in operation.