PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Stream Memory for Vision-Language-Action Policies

Yang Wang, Ping Liu, Ruiyu Wang, Xuhua Chen, Lingfeng Zhang, Tao Zhang

arXiv:2610.13262v1cs.ROcs.CV

TL;DR

Memory-dependent manipulation asks policies to distinguish execution stages and recover earlier visual evidence that current observations may hide. This paper pairs recurrent execution context with task-conditioned access to historical frames; across 16 RoboMME tasks, it reaches 49.22% Mean TSR versus 44.51% for FrameSamp+Modul.

  • Problem

    VLA models do not determine which episodic information a policy should retain, despite memory-dependent tasks requiring both execution-stage information and earlier visual evidence.

    Source: §I · p. 1

  • Method

    The paper combines a recurrent Gated DeltaNet temporal token supervised by progress and action-conditioned reconstruction with task-conditioned frame retrieval and multi-scale spatial tokens.

    Source: §I · p. 2, A. Dual-stream memory · p. 3, D. Linear multi-scale fusion · p. 3

  • Results

    49.22% Mean TSR versus 44.51% for FrameSamp+Modul, with improvements on fourteen tasks and all four suite means across sixteen RoboMME tasks.

    Source: D. Full-benchmark comparison · p. 5

  • Takeaways & Limitations

    Reference gained 8.98 percentage points, consistent with detailed historical evidence, while Counting gained 5.61 points, consistent with execution-context support.

    Source: D. Full-benchmark comparison · p. 5

    Shell success declined from 53.3% to 40.0%; maintaining target identity through three cup swaps motivates explicit identity tracking.

    Source: E. Real-robot evaluation · p. 7

Abstract

from arXiv · show

Memory-dependent manipulation requires a policy to track execution state and recover visual evidence that is no longer observable. We propose dual-stream memory for vision-language-action policies, assigning these requirements to complementary representations. A task-agnostic Gated DeltaNet compresses history into one temporal token, supervised by progress prediction and action-conditioned feature reconstruction. A task-conditioned retriever retains frames above a fixed relevance threshold; linear multi-scale fusion encodes each frame into sixteen spatial tokens. We then freeze the temporal compressor and jointly optimize retrieval and policy learning. Across sixteen RoboMME tasks, the complete model achieves 49.22% mean success versus 44.51% for the published FrameSamp+Modul reference, improving fourteen tasks and all four suite means. On four real-robot tasks, mean success reaches 62.5% versus 37.5%, with the largest gains on repeated actions requiring execution-stage discrimination. These results support combining execution context with selective visual access, while failures under cup swaps expose persistent identity tracking as an unresolved requirement.

I. INTRODUCTION

Long-horizon manipulation can demand different actions from similar observations because of execution stage or earlier visual evidence. The paper separates execution context, frame relevance, and within-frame detail across complementary memory representations.

I. INTRODUCTION

  • Similar observations can require different actions depending on prior execution, while occlusion and reidentification can hide visual evidence needed later.Repeated actions call for execution-stage information; target reidentification calls for distinguishing visual detail.Source: §I · p. 1
  • Recurrent compression may lose target-identifying detail, while uniform sampling can miss brief evidence and spatial pooling can discard distinguishing detail.The design therefore distinguishes execution context, frame relevance, and within-frame detail.Source: §I · pp. 1–2
  • The proposed task-agnostic Gated DeltaNet compressor uses progress prediction and action-conditioned feature reconstruction to supervise one temporal token.The objectives encourage execution context and reconstructable visual history in that token.Source: §I · p. 2
  • Task-conditioned threshold retrieval adapts retained-frame count to available evidence, and linear multi-scale fusion combines coarse context and fine detail in sixteen tokens per frame.Together with recurrent temporal memory, these components provide selected spatial evidence.Source: §I · p. 2

A. Memory for robot policies

The related work spans episodic memory for robot policies, recurrent compression and selective access, and spatial representations. This paper distinguishes recurrent execution state from retrieved visual evidence and enriches retained frames across spatial scales.

A. Memory for robot policies

  • CALVIN and LIBERO study sequential manipulation and knowledge transfer [13], [14], while RoboMME targets Counting, Permanence, Reference, and Imitation [15].MemoryVLA, MEM, and NativeMEM organize perceptual, cognitive, or multi-scale memory [10], [11], [12].Source: A. Memory for robot policies · p. 2

B. Recurrent compression and selective access

  • Prior work extends historical access through attention, recurrence, or compression [18], [19], [20],, while DeltaNet and GDN use delta-rule updates and data-dependent forgetting, [26].This design supervises GDN for execution progress and reconstructable visual history, with a separate spatial bank for direct evidence access.Source: B. Recurrent compression and selective… · p. 2

C. Spatial representations for memory

  • Patch encoders and multi-scale representations preserve spatial structure [29], [35], [36],; here DINOv3 supports compression and SigLIP retrieval and fusion within sixteen slots.The fusion enriches retained frames without enlarging their token budget.Source: C. Spatial representations for memory · pp. 2–3

III. METHOD

The method pairs recurrent compression of visual history with task-relevant frame selection and spatial encoding. Its recurrent state is trained to represent execution progress and recoverable visual history.

A. Dual-stream memory

  • The temporal stream compresses a stridesampled front-view prefix into c_t, while the spatial stream selects and encodes task-relevant frames; both streams are causal.DINOv3 supplies one global token per frame for compression, while frozen SigLIP supplies language-aligned patch features for localization and spatial memory.Source: A. Dual-stream memory · p. 3

B. Recurrent execution context

  • Progress supervision encourages the recurrent state to distinguish execution stages, while action-conditioned reconstruction constrains it to retain information about earlier visual appearance.The compressor processes history at stride three through a six-layer GDN; its auxiliary heads are removed after pretraining, and an adapter maps its output to a policy-width token.Source: B. Recurrent execution context · p. 3

C. Task-conditioned threshold retrieval

Task-conditioned retrieval selects historical frames using relevance scores and a fixed threshold rather than a fixed frame count. Shared spatial and temporal supervision trains the score grid to localize relevant patches and events.

C. Task-conditioned threshold retrieval

  • Retaining frames above a fixed relevance threshold lets memory length adapt to available evidence, unlike uniform sampling or fixed-count top-k retrieval.The threshold is applied to normalized LSE logits, and retained frames stay in chronological order.Source: C. Task-conditioned threshold retrieval · p. 3
  • Frozen SigLIP patch features, temporal attention, and a task-goal query define patch scores and frame-level log-sum-exp logits for selection.The query and keys produce the relevance scores used at inference.Source: C. Task-conditioned threshold retrieval · p. 3
  • Spatial mask targets and temporal keyframe intervals supervise the same score grid through patch and interval cross-entropy losses.For ShellGame, annotations include target–cup associations, exchanges, and final cup configuration; labels are generated offline from observed prefixes.Source: C. Task-conditioned threshold retrieval · p. 3

D. Linear multi-scale fusion

Linear multi-scale fusion combines coarse context and fine spatial detail in sixteen tokens per retained frame. It builds on aligned feature grids and maps the fused features into policy tokens.

D. Linear multi-scale fusion

  • Frozen SigLIP 16×16 features are mean-pooled into aligned 8×8, 4×4, and 2×2 grids.Sixteen base slots combine each slot with its four fine children and coarse parent.Source: D. Linear multi-scale fusion · p. 3
  • Each retained frame contributes sixteen tokens, combining coarse context and fine detail through independently projected scales.The fused features receive positional encoding and a feature encoder to produce 1,024-dimensional policy tokens.Source: D. Linear multi-scale fusion · p. 3
  • Independent linear projections map the scales to a shared feature width; identity initialization for P4 and zero initialization for other projections recover the base representation.The spatial memory bank contains 16mt tokens in total.Source: D. Linear multi-scale fusion · p. 3

E. Policy integration and training

The policy reads both spatial and temporal memory through separate cross-attention paths. Training first pretrains the temporal compressor, then freezes it while jointly training retrieval and the policy.

E. Policy integration and training

  • The method uses the RoboMME π0.5 policy stack with a Gemma prefix and flow-matching action expert [15],, [41]; Mpatch and Mtemp denote the spatial memory bank and temporal token.Source: E. Policy integration and training · p. 4
  • The temporal gate is initialized to 0.01 and modulates the normalized action representation before its feed-forward layer.Source: E. Policy integration and training · p. 4
  • Stage 1 pretrains GDN with LC; Stage 2 freezes GDN and trains retrieval and the π0.5 policy with Laction + LR.SigLIP and MiniLM remain frozen, memory resets between episodes, and spatial storage and readout costs depend on the retained-frame count.Source: E. Policy integration and training · p. 4

A. Setup

The evaluation covers component variants, sixteen RoboMME tasks, and four real-robot tasks. Results use a published reference whose checkpoint aggregation differs from the authors’ aggregation.

A. Setup

  • The complete-model evaluation covers sixteen RoboMME tasks and four real-robot tasks; Stage 2 uses 80,000 steps, batch size 64, and learning rate 5×10−5.The retrieval corpus contains 6,400 episodes and 131 task goals.Source: A. Setup · p. 4
  • Task success rate uses 50 episodes per task for each of three evaluation seeds, and suite means weight tasks equally.FrameSamp+Modul results use published checkpoint aggregation that differs from the authors’ aggregation, so comparisons are not matched architectural effects.Source: A. Setup · p. 4
  • Baselines include π0.5 without memory and NativeMEM, a learned policy-aligned memory-token baseline alongside sampled-history FrameSamp+Modul.The NativeMEM implementation uses 30,000 tokenizer-training steps followed by 60,000 policy-training steps.Source: A. Setup · p. 4

B. Component comparisons

The component comparisons show gains from recurrent execution context, threshold-based frame retrieval, and multi-scale spatial detail, with some task-level declines for retrieval and fusion.

B. Component comparisons

  • Counting TSR increased 6.94 percentage points with GDN added to sampled spatial memory; StopCube improved most, by 15.33 points.The StopCube result is consistent with recurrent context supporting different stopping decisions at visually similar stages.Source: B. Component comparisons · p. 4
  • Permanence TSR increased 4.22 percentage points with threshold retrieval, improving three tasks; ButtonUnmaskSwap gained most (+7.11 points).VideoUnmask declined by 1.34 points.Source: B. Component comparisons · p. 4
  • Reference TSR increased 4.50 percentage points with linear fusion; VideoPlaceOrder improved most (+12.00 points), while VideoRepick declined by 1.11 points.The fusion variant modifies sampled-frame features without GDN or threshold retrieval.Source: B. Component comparisons · p. 4

C. Analysis of Learned Memory and Retrieval

The analysis probes whether recurrent memory captures execution progress and whether compressed states retain recoverable visual evidence. Retrieval aligns annotated evidence at both frame and patch scales, while examples reveal task-dependent errors and multiple temporal relevance peaks.

Execution-stage readout

  • Progress readout MSE was 0.03430 over 46,800 valid frames, averaged within tasks and then equally across sixteen tasks.The readout was trained with SmoothL1 loss.Source: C. Analysis of Learned Memory and… · p. 4
  • Progress traces tracked references on InsertPeg and MoveCube, fluctuated more on BinFill and PickHighlight, and underestimated terminal progress on PatternLock.These readouts indicate recurrent state contains information correlated with execution progress.Source: C. Analysis of Learned Memory and… · p. 4, Figure 3 · p. 5

Action-conditioned visual history

  • Action-conditioned reconstruction examples showed 38 matches and 10 mismatches across 48 displayed slots for InsertPeg, PatternLock, and SwingXtimes.The examples illustrate visual history recoverable from the compressed state.Source: C. Analysis of Learned Memory and… · p. 4

Temporal and spatial localization

  • Temporal and spatial Hit@1 were 96.88% and 95.00%, with AP of 96.53% and 92.24%, respectively.Hit@1 tests the top-ranked frame or patch against annotated evidence; AP evaluates ranking beyond the top prediction.Source: C. Analysis of Learned Memory and… · p. 4
  • Four training examples showed concentrated relevance around annotated evidence, with multiple peaks across each observed history.Visualization thresholds apply to probabilities, whereas inference thresholds normalized LSE logits.Source: C. Analysis of Learned Memory and… · pp. 4–5

D. Full-benchmark comparison

The complete model outperformed published FrameSamp+Modul across RoboMME and real-robot evaluations, though component gains were not uniformly additive and Shell remained challenging.

D. Full-benchmark comparison

  • 49.22% mean TSR versus 44.51% for published FrameSamp+Modul, with gains on fourteen tasks and all four RoboMME suite means.The comparison uses the published RoboMME checkpoint aggregation, which differs from the authors’ evaluation protocol.Source: D. Full-benchmark comparison · p. 5, Table II · p. 7
  • The combined model’s benefits were not additive: Counting and Permanence remained below their single-component variants, while VideoUnmaskSwap and PatternLock declined by 1.44 and 1.56 percentage points.Reference exceeded the linear-fusion-only variant.Source: D. Full-benchmark comparison · p. 5
  • NativeMEM improved ten tasks over the no-memory policy but declined on six, including three Counting tasks, so compressed memory alone did not consistently improve performance under this recipe.Source: D. Full-benchmark comparison · p. 5
  • 62.5% mean TSR on four real-robot tasks exceeded FrameSamp+Modul by 25.0 percentage points, with 24/30 and 28/30 successes on Pick×3 and Swing×2.Both repeated-action tasks revisit similar observations at stages requiring different subsequent actions.Source: Table III · p. 6, E. Real-robot evaluation · p. 7
  • Shell success declined from 53.3% to 40.0%; the task requires maintaining target identity through three cup swaps.The authors identify explicit identity tracking across interactions as a next step.Source: E. Real-robot evaluation · p. 7
  • The reported gains are bounded by the evaluation protocols; matched full-system ablations and broader task and hardware coverage are needed to assess component contributions and generalization.Source: §V · p. 7

Related papers

Loading 2610.13262v1…