PaperlayerView exact revision on arXiv

Source-linked AI summary

TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation

Jiarui Yang, Yehao Lu, Yuning Su, Yu Zhong, Yufeng Xie, Yazhou Zhang, Haiyu Lan, Kaixiang Lu, Peiwen Lin, Chuang Wang, Junwei Liang, Enyu Li

arXiv:2608.26821v1cs.RO

TL;DR

TemporalFlow-VLA addresses the limited action usefulness of naively added historical frames, especially when multi-stage states look similar but arise from different execution histories. It learns two compact temporal queries from offline robot-surface flow supervision and exposes them to the action expert, achieving strong LIBERO and RoboTwin performance with ordered-history sensitivity and deployment-time geometric processing avoided.

  • Problem

    Simply adding historical frames does not reliably encode recent physical change for action prediction, creating ambiguity when visually similar multi-stage states require history-dependent actions.

  • Method

    TemporalFlow-VLA uses recorded robot states, robot geometry, and calibrated cameras to supervise two execution-aligned temporal queries that compress historical RGB features for the action expert.

  • Results

    TemporalFlow-VLA achieves 97.63±0.26% average success on LIBERO and 85.5%/84.2% Clean/Randomized success across 12 RoboTwin tasks, with clearest gains on multi-stage manipulation.

  • Takeaways & Limitations

    Controlled interventions show that action prediction depends on both historical content and temporal order, while temporal conditioning can retain single-frame-level server-side sampling latency through asynchronous caching.

Abstract

from arXiv · show

Vision-language-action (VLA) models leverage pretrained vision-language representations for robot control, yet simply adding historical frames does not reliably capture recent physical change. This is especially problematic in multi-stage manipulation, where visually similar states may require different actions depending on prior execution. To address this challenge, we present TemporalFlow-VLA, which learns compact execution history through physically grounded temporal supervision. Using recorded robot states, robot geometry, and calibrated cameras, we construct robot-surface temporal flow as a training-only target and supervise two execution-aligned temporal queries that provide structured history to the action expert. The geometric supervision path is not evaluated at deployment. TemporalFlow-VLA achieves 97.63 +/- 0.26% average success on LIBERO, including 96.60 +/- 0.87% on LIBERO Long, and 85.5%/84.2% Clean/Randomized success across 12 RoboTwin tasks. It shows its clearest advantage over prior methods on longer-horizon, multi-stage manipulation. Controlled history interventions show that action prediction depends on both historical content and temporal order. With asynchronous feature caching, temporal conditioning maintains single-frame-level server-side sampling latency without additional historical-encoding overhead. Overall, TemporalFlow-VLA provides a compact, physically grounded interface for exploiting ordered execution history without explicit motion estimation or geometric processing at deployment.

I. INTRODUCTION

TemporalFlow-VLA addresses the difficulty of using execution history for control by learning compact, physically grounded temporal representations rather than simply adding frames. Its design targets ambiguities in visually similar multi-stage states and preserves ordered history for action generation.

  • Motivation: Visually similar robot states can require different actions because current RGB, language, and robot state omit how the state was reached.Examples include failed grasps resembling pre-grasp states and identical end-effector poses corresponding to approach, carrying, or recovery.
  • Motivation: Shuffling three historical frames leaves offline action flow-matching loss nearly unchanged, while removing history increases it by about 4.6%.The diagnostic indicates that an unconstrained history baseline uses history but is largely insensitive to its chronological order.
  • Approach: TemporalFlow-VLA uses robot-surface motion as training supervision for two compact, chunk-aligned temporal queries exposed to the action expert.Recorded joint states, URDF geometry, and camera calibration generate the temporal target offline; dense flow is not evaluated at deployment.
  • Results: TemporalFlow-VLA achieves 97.63 ± 0.26% average success on LIBERO and 84.2% across 12 challenging randomized RoboTwin 2.0 tasks.The clearest gains occur on long- and multi-stage manipulation.
  • Approach: The method combines a parallel temporal pathway with joint masked attention and asynchronous historical-feature caching to reduce temporal-conditioning overhead.Historical image patches reach the action expert through the supervised query interface, while cached features overlap historical encoding with action execution.

B. External Memory and Explicit Motion

Prior methods organize, retrieve, or expose temporal motion through visual or feature-based mechanisms, but TemporalFlow-VLA assigns compact history an explicit physical target. This target is intended to make recent execution evolution more control-aligned than generic temporal representations.

  • External Memory: Memory-based approaches retrieve or organize temporal context, whereas TemporalFlow-VLA learns the local execution history immediately preceding replanning with an explicit physical target.The comparison includes keyframe retrieval, video and language memory, and recency-biased prefix-state reuse.
  • Explicit Motion: Motion-centric methods expose tracked trajectories, trajectory fields, codec motion vectors, or optical flow, while TemporalFlow-VLA uses deterministic robot-surface flow.These approaches differ in whether motion is presented as a visual prompt, tokenized field, hindsight prior, or visual chain-of-thought.
  • Compact Temporal Representations: Feature-based temporal methods learn compact representations through chunking, contrastive moment tokens, working-memory retrieval, or recurrent memory.Their temporal content is shaped by native VLM features, action supervision, reconstruction, or recurrence rather than a prescribed physical change.
  • Paper Positioning: TemporalFlow-VLA asks whether compact latent history can be assigned an explicit, control-aligned physical target.Q15 and Q8 are introduced as the paper’s execution-aligned temporal queries.

A. Overview

TemporalFlow-VLA preserves the base current-observation policy while adding a parallel pathway that compresses recent RGB history into two supervised temporal queries. Robot kinematics and calibrated rendering provide offline robot-surface flow targets for those queries.

  • A. Overview: The base policy predicts an action chunk from the current RGB observation, language instruction, robot state, and diffusion timestep.The temporal pathway additionally receives observations at t −15, t −8, and t.
  • A. Overview: Q8 represents motion from t −8 to t, while Q15 captures evolution from t −15 to t and can build on Q8’s short-range summary.For a 16-step action chunk, the two timestamps approximately cover the previous chunk’s beginning and midpoint.
  • B. Kinematics-Grounded Robot-Surface Temporal Flow: Robot states, link geometry, and camera calibration transport visible robot surface points from a source image into a target image.The method recovers each point in its link frame, applies target-time forward kinematics, and projects it into the target image.
  • B. Kinematics-Grounded Robot-Surface Temporal Flow: Correspondences are retained only when target projection is in bounds, belongs to the same link, and has a 3-D residual no larger than 5 mm.This filtering supports valid source-to-target robot-surface displacement computation.
  • B. Kinematics-Grounded Robot-Surface Temporal Flow: Valid robot flows are averaged over non-overlapping 14×14 regions to form a 16 × 16 × 2 target, with coverage gating the auxiliary loss.The temporal pathway still receives full RGB, so object and scene changes remain available to the action objective; labels are generated offline.

C. Hierarchical Temporal Queries and Joint Attention

TemporalFlow-VLA organizes historical information into two supervised, hierarchically connected temporal queries and exposes history to the action expert only through them. Query-specific flow reconstruction preserves distinct temporal scales while joint masked attention blocks direct historical-patch access.

  • Hierarchical Temporal Queries: Learnable q8 and q15 tokens are appended to the standard π0.5 prefix containing current-image, language, and robot-state inputs.Historical frame-identity embeddings are added only to the historical patches.
  • Hierarchical Temporal Queries: Q8 summarizes recent motion from t −8 to t, while Q15 integrates t −15 to t and can build on Q8.The directed hierarchy keeps short- and long-range temporal content distinct.
  • Temporal-Flow Supervision: Separate query-conditioned decoders reconstruct interval-specific flow from each query and its source-frame spatial states.The decoders use query-based FiLM modulation and pointwise MLPs without parameter sharing, convolution, or upsampling.
  • Joint Masked Attention: Action tokens access historical visual information only through Q8 and Q15, not directly through historical image patches.The structured mask preserves the original VLA context while exposing only the two temporal queries.
  • Temporal-Flow Supervision: The training objective jointly optimizes action flow matching and temporal-flow reconstruction, with λtemp set to 1.0 in main experiments.This supervision ties Q8 and Q15 to their intended temporal scales and motion semantics.

D. Asynchronous Historical-Feature Caching

Asynchronous caching moves historical visual encoding off the synchronous replanning path. Historical features are computed during execution, timestamped, and retrieved later, leaving current-frame encoding and action computation on the critical path.

  • Cache Construction: Historical observations are written to a timestamped ring buffer while the robot executes the current action chunk, then encoded by a background process.This overlaps historical feature extraction with ongoing action execution.
  • Cache Retrieval: Fixed action-chunk length and Q8/Q15 offsets allow the t −15 and t −8 features needed at replanning to be precomputed during the preceding chunk.At replanning, cached historical features are retrieved directly while the current observation is encoded synchronously.
  • Latency: With asynchronous caching, only current-frame encoding, the joint transformer, and action generation remain on the synchronous critical path.Without caching, synchronous latency includes three visual encodings.
  • Latency: Tvision, Tjoint, and Taction denote visual-encoding, joint-transformer, and action-generation latency, respectively.Tjoint includes the temporal queries.
  • Cache Validity: At episode start, unavailable history is filled with the earliest observation and marked invalid, while stale features are evicted outside the required window.The cache is bypassed until both required historical frame tokens are available.

A. Experimental Setup

The experiments combine simulation benchmarks, controlled ablations, inference-efficiency evaluation, and real-robot experiments. LIBERO uses joint training across four suites, while RoboTwin uses 12 jointly trained tasks with clean and randomized demonstrations.

  • Evaluation Scope: Experiments cover simulation benchmark evaluation, controlled ablations, inference efficiency, and real-robot experiments.This evaluation design spans performance, mechanism, deployment cost, and physical-robot settings.
  • LIBERO: LIBERO policies are jointly trained on Spatial, Object, Goal, and Long for 30k steps and evaluated with 500 rollouts per suite and seed.Reported results use mean ± SD over seeds 0, 2, and 5.
  • RoboTwin: RoboTwin policies are jointly trained on 12 tasks for 60k steps from 50 clean and 500 randomized demonstrations.The setup evaluates both clean and randomized demonstration conditions.

B. Simulation Benchmark Evaluation

TemporalFlow-VLA performs strongly on LIBERO and RoboTwin, with its clearest gains on longer-horizon, multi-stage manipulation. On RoboTwin, performance improves with execution horizon, while LIBERO Long shows the clearest advantage over prior systems.

  • RoboTwin: 87.5% at H=3 exceeds the runner-up average by 14.5 points, while 80.8% at H=2 exceeds it by 5.5 points.The H=1 average is not the best, indicating the strongest gains occur at longer horizons.
  • LIBERO: 96.60 ± 0.87% is achieved on LIBERO Long, 2.1 percentage points above the strongest listed prior mean result.The method averages 97.63±0.26% across the standard LIBERO suites over three seeds.
  • Interpretation: The advantage is concentrated in multi-stage manipulation rather than already saturated short-horizon LIBERO suites.This pattern is also reflected by the horizon-dependent RoboTwin results.

C. Ablation Studies

Controlled ablations show that action prediction uses both the content and order of recent history, while flow supervision and two temporal scales provide complementary benefits. The history perturbation diagnostic is offline and should not be interpreted as online success.

  • History Content and Order: Both removing and shuffling history increase action flow-matching loss on all six tested RoboTwin tasks.Shuffling is most harmful on five tasks, whereas put bottles (dustbin) is more sensitive to history removal.
  • Caveat: The history intervention is an offline action-loss diagnostic showing sensitivity to temporal assignment, not a proxy for online success.The diagnostic therefore supports temporal-order sensitivity without directly establishing online task improvement.
  • Flow Supervision: Two queries without flow reach 83.2%/83.6% Clean/Randomized versus 79.8%/81.5% for Multi-frame.These results compare the query bottleneck against raw multi-frame history.
  • Flow Supervision: Adding flow supervision reaches 85.5%/84.2% Clean/Randomized and adds 3.7/1.5 points at H=2.Because the no-flow variant uses the same two query slots and history window, the gap isolates flow supervision from temporal capacity and context length.
  • Temporal Scale: Q8 alone reaches 84.8%/84.0% Clean/Randomized versus 83.9%/83.0% for Q15, while both supervised scales perform best overall.The results indicate complementary short- and longer-range temporal context.

D. Inference Efficiency

Asynchronous caching reuses historical visual features during execution, reducing server-side policy sampling latency while current-observation encoding and action sampling remain on the foreground path.

  • Caching Results: Caching reduces mean latency from 68.10 to 62.78 ms, saving 5.32 ms per replan.Historical tokens are precomputed during the preceding 16-step action chunk and retrieved at replanning.
  • Caching Results: Cumulative sampling time decreases from 1.021 to 0.942 s over the measured 15-replan segment, a 79.8 ms (7.8%) reduction.The evaluation uses matched successful LIBERO Long rollouts on an RTX 4090 after warm-up.
  • Caching Operation: At episode start, caching is bypassed until both required historical tokens are available.Subsequent replans reuse cached history, removing redundant encoding while current-observation encoding and action sampling remain foreground operations.

E. Real-Robot Evaluation

Real-robot tests on AgiBot A3 show that TemporalFlow-VLA improves success on two three-stage manipulation tasks, with the larger gain on cup stacking. The study also identifies fixed temporal-scale selection as an open boundary for generality.

  • Evaluation setup: The physical evaluation uses head- and chest-view RGB observations, while only the head view constructs training-time temporal-flow supervision.Each task contains 280 demonstrations; evaluation uses 45 trials per method and task.
  • Task results: 77.8% mean success on Three-Cup Stacking improves over 57.8% for the baseline, a gain of 20.0 points.The task requires preserving progress across sequential placements and alignment steps.
  • Task results: 97.8% mean success on Two-Bottle Packing improves over 86.7% for the baseline, a gain of 11.1 points.
  • Physical transfer: Across the physical setting, TemporalFlow-VLA consistently improves over the baseline and remains useful under real-world observation and execution variation.The deployment policy does not require geometric inputs.
  • Limitation: The fixed historical observations leave the optimal temporal horizon and sampling granularity for different manipulation tasks unresolved.The authors identify systematic study of history number and temporal spacing as future work.
Loading 2608.26821v1…