PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCAT: Remember, Count, and Time: Structured Recurrent Memory for Robot Manipulation

Pankhuri Vanjani, Mostafa Hatab, Can Mizrakli, Vaisakh Shaj, Zhuoyue Li, Moritz Reuss, Rudolf Lioutikov

arXiv:2609.35200v1cs.ROcs.AIcs.CVcs.LG

TL;DR

In memory-dependent manipulation, the current observation may not reveal task state, so action choice can depend on earlier observations and events. ReCAT uses a recurrent memory with five Mamba-2 layers and one causal-attention layer, retaining recurrent states and an attention cache as its history representation. On real-robot tasks, the best ReCAT variant reaches 66.7% success, versus 8.3% for X-VLA-H, the strongest baseline without persistent memory.

  • Problem

    In memory-dependent manipulation, the current observation may not reveal task state, so action choice can depend on earlier observations and events.

    Source: §I · p. 1

  • Method

    ReCAT uses a recurrent memory with five Mamba-2 layers and one causal-attention layer, retaining recurrent states and an attention cache as its history representation.

    Source: B. ReCAT · p. 3

  • Results

    On real-robot tasks, the best ReCAT variant reaches 66.7% success, versus 8.3% for X-VLA-H, the strongest baseline without persistent memory.

    Source: C. Results and Discussion · p. 5

  • Takeaways & Limitations

    Within evaluated configurations, the hybrid observation encoder and every-block memory cross-attention had the highest average success, while update-rule results varied by task.

    Source: §V · p. 7

    The real-robot evaluation covers one task per memory demand with 20 rollouts each, and the joint memory ablation leaves the individual contributions of recurrence and attention unresolved.

    Source: §VI · p. 7

Abstract

from arXiv · show

Memory-dependent manipulation requires robots to make decisions using information that is no longer available to their current sensors, such as recalling an earlier visual cue, tracking task progress, counting repeated events, or estimating elapsed time. We present ReCAT, a language-conditioned policy with structured recurrent memory. An instruction-conditioned encoder forms features from the current observation. A recurrent memory integrates the observation stream through Mamba-2 layers and one causal attention layer. A flow-matching Transformer decoder reads the current and the historical representation through separate cross-attention in every block. ReCAT reaches 95.3\% average success on LIBERO and 62.4\% on RMBench, with the best or tied-best result on six of nine tasks. On three real-robot tasks probing spatial recall, event counting, and interval timing, the best ReCAT variant reaches 66.7\% average success, against 8.3\% for the strongest short-history baseline. Controlled comparisons within ReCAT show that the observation encoder and every-block memory conditioning are needed for this performance. They also show that update rules developed for efficient sequence modeling behave differently as robot memory: additive updates have the highest observed success on counting and timing, and delta-rule updates on spatial recall. Project website is at https://intuitive-robots.github.io/ReCAT

I. INTRODUCTION

Memory-dependent manipulation requires combining current perception with earlier task events, because visually similar observations can call for different actions. ReCAT pairs direct access to the present with a compact recurrent history, and the paper evaluates it on simulation benchmarks and real-robot memory tasks.

I. INTRODUCTION

  • Visually similar observations can require different actions after one versus two scoops, making task history necessary for manipulation.In PLANT, the robot scoops again after one scoop but puts down the shovel after two.Source: §I · p. 1, Figure 1 · p. 1
  • ReCAT sends instruction-conditioned features directly to the action decoder and through a recurrent memory, which the decoder reads separately from current features.Its memory combines recurrent layers with one causal attention layer, and the flow-matching decoder uses separate cross-attention pathways.Source: §I · pp. 1–2
  • 66.7% average success on three real-robot memory tasks compares with 8.3% for the strongest short-history baseline.The tasks probe spatial recall, event counting, and interval timing.Source: §I · p. 2
  • The evaluation covers LIBERO, RMBench, and three real-robot tasks, while controlled comparisons examine encoding, update rules, memory capacity, and decoder conditioning.Stage-wise evaluation distinguishes manipulation failures from failures that require using history.Source: §I · p. 2

II. RELATED WORK

Prior visuomotor policies represent history with recent-observation windows, memory banks, or compact recurrent states. ReCAT builds on recurrent memory while varying both its update rule and how the decoder combines memory with the current observation.

A. Memory in Visuomotor Imitation Learning

  • Robot policies represent the past by attending to recent visual tokens in fixed windows [13], [14], retrieving stored observations from memory banks, [15], [16], or carrying compact learned states through time [2], [12], [17].Source: A. Memory in Visuomotor Imitation… · p. 2
  • ReCAT addresses an open design question by making both the recurrent update rule and the decoder’s memory pathway configurable.Its memory stacks recurrent layers with one causal attention layer, while separate cross-attention pathways carry current observations and memory into each decoder block.Source: A. Memory in Visuomotor Imitation… · p. 2

B. Recurrent Memory Architectures for Robot Manipulation

  • Robot policies use SSMs for action generation and history encoding, but the cited approaches fix a recurrent backbone and a route from memory to action –[22].Examples include Mamba-based action backbones, temporal encoders, full-history conditioning, and test-time fast-weight adaptation.Source: B. Recurrent Memory Architectures for… · p. 2
  • ReCAT mixes recurrence with causal attention across time, using recurrence for constant-cost progress tracking and attention to access a specific earlier observation.The passage describes additive, delta-rule, and richer-dynamics update families, while noting that fixed-size state can struggle with exact token recall –[31].Source: B. Recurrent Memory Architectures for… · p. 2

A. Problem Formulation

The formulation treats language-conditioned manipulation as partially observed: the current observation may not reveal task state, so action selection depends on the observation history. ReCAT represents that history compactly while retaining direct access to current features.

A. Problem Formulation

  • The setup is language-conditioned imitation learning from demonstrations containing task instructions, camera and proprioceptive observations, and demonstrated actions.An action chunk contains K consecutive actions.Source: A. Problem Formulation · p. 2
  • The overview routes each frame feature both directly to the action decoder and into recurrent memory, whose readout receives separate cross-attention in every decoder block.The decoder backbone is shared across experiments, while ablations vary the encoder mixer, update rule, memory dimensions, and decoder conditioning.Source: Figure 2 · p. 3
  • When the current observation is insufficient to determine task state, visually similar views can require different actions based on prior events.The policy is therefore modeled as depending on the observation history and instruction, rather than the current observation alone.Source: A. Problem Formulation · p. 3

B. ReCAT

ReCAT encodes each current observation into a feature used both for action generation and recurrent memory. A flow-matching decoder reads current and historical representations through separate pathways, while memory configuration and inference are specified explicitly.

B. ReCAT

  • The action decoder predicts an action chunk from frame feature zt and memory readout mt, while zt also drives memory updates.The memory updates its retained state at each step and returns a history-conditioned readout.Source: B. ReCAT · p. 3
  • Frozen DynaFLIP image and T5 instruction encoders, adapted with LoRA, feed a language-conditioned compressor that preserves global context and local spatial detail.The encoder combines visual, instruction, and proprioception tokens; five Mamba-2 layers and one bidirectional-attention layer produce zt.Source: B. ReCAT · p. 3
  • The temporal memory has six width-768 blocks: five Mamba-2 layers and one causal-attention layer, with recurrent states and an attention cache forming its retained state.The cache grows by one 768-dimensional entry per step; the memory is reset between episodes, and mt is the normalized final-block output rather than the state itself.Source: B. ReCAT · p. 3
  • Memory capacity varies by block count and state width, while the recurrent update can use Mamba-2, Mamba-3, or Gated DeltaNet-2 with other components fixed.The compared rules differ in how their writes affect stored information; their described semantics are intended behavior, not a guarantee about trained policies.Source: B. ReCAT · p. 3, Table I · p. 4
  • Each four-block decoder layer cross-attends separately to current feature zt and memory readout mt after action-token self-attention.It predicts 16-action chunks using flow matching, with flow time conditioning each block through adaptive layer normalization.Source: B. ReCAT · pp. 3–4
  • Training scans each demonstration causally over its full episode; inference uses four Euler steps and executes four actions before replanning.The frame encoder runs and memory advances at every control step, including during chunk execution.Source: B. ReCAT · p. 4

IV. EVALUATION

The evaluation spans simulation benchmarks and real-robot memory-dependent tasks, asking about ReCAT’s effectiveness and how design choices affect performance.

IV. EVALUATION

  • The evaluation asks whether ReCAT is effective on standard and memory-dependent manipulation.This is RQ1 across two simulation benchmarks and three real-robot tasks.Source: §IV · p. 4
  • RQ2 examines how the observation encoder and memory conditioning affect performance.Source: §IV · p. 4
  • RQ3 examines how recurrent update rules and memory capacity affect different memory demands.Stage-wise behavior and computational cost are also reported separately.Source: §IV · p. 4

A. Experimental setup

The setup evaluates ReCAT on two simulation benchmarks and three real-robot tasks, with protocols and metrics specified for each setting.

A. Experimental setup

  • LIBERO tests performance when the current observation is largely sufficient, while RMBench includes nine long-horizon bimanual tasks with actions dependent on past events.LIBERO includes Spatial, Object, Goal, and LIBERO-10; RMBench has five M(1) and four M(n) tasks.Source: A. Experimental setup · p. 4
  • Three real-robot tasks test spatial recall, event counting, and interval timing under real-world execution variability.The setup uses a Franka Panda with fixed side and wrist RGB cameras and joint-space proprioception.Source: A. Experimental setup · p. 4
  • SPONGE requires returning a moved sponge to its unmarked original location after it leaves direct view.Success requires landing within 2–4 cm of its original centre.Source: A. Experimental setup · p. 5
  • PLANT requires counting two visually similar scoops before switching from scooping to planting.The sequence includes five stages and episodes of about 1,200 steps.Source: Figure 3 · p. 4, A. Experimental setup · p. 5
  • POT TIMER requires waiting about 30 seconds in a nearly unchanged scene before adding pepper to the pot.A wait counts as correct at 29–31 seconds.Source: A. Experimental setup · p. 5
  • Each task and configuration uses one seed, 500 training epochs, and 20 randomized-position rollouts; evaluation reports success, latency, and SPARC smoothness.With 20 rollouts, one rollout equals five percentage points, within rollout noise.Source: A. Experimental setup · p. 5

B. Baselines

The baselines cover no-memory, short-history, and explicit attention-based memory policies, with comparisons constrained by robot deployment feasibility.

B. Baselines

  • Baselines include no-memory DP and X-VLA, short-history DP-H and X-VLA-H, and GMP, which retrieves from stored history using cross-attention and a calibrated binary gate.The short-history variants use the last eight observations; DP-H uses a shared trainable ResNet-18 and X-VLA-H compresses them into eight history tokens.Source: B. Baselines · p. 5
  • Robot comparisons use deployable policies on a 4060Ti, excluding multi-billion-parameter memory-augmented VLAs; DSSP [10] is discussed as concurrent work without an official implementation.DP has 268M parameters, X-VLA has 0.9B, and GMP is the closest published memory policy at comparable size.Source: B. Baselines · p. 5

C. Results and Discussion

ReCAT achieves its strongest reported results on real-robot tasks requiring retained history, while remaining competitive on standard manipulation benchmarks. Ablations show performance varies with encoder, memory conditioning, update rule, and capacity.

Stage-wise results

  • Short-history baselines often complete observation-driven stages but fail at history-dependent decisions, including returning the sponge, stopping after two scoops, or waiting correctly.X-VLA-H places the sponge in 15 of 20 rollouts but returns it correctly in only 2; it scoops at least once in 19 of 20 PLANT rollouts but completes the task in 2, and waits correctly in 1 of 20 POT TIMER rollouts.Source: C. Results and Discussion · pp. 5–6, Table IV · p. 7

Benchmark results

Update rule

  • Update rules have task-dependent effects: additive Mamba variants score 85% and 90% on PLANT, while Gated DeltaNet-2 reaches 40% on SPONGE.All three rules wait correctly in every POT TIMER rollout; the proposed explanation that additive writes accumulate scoop evidence remains a hypothesis because internal counts were not measured.Source: C. Results and Discussion · p. 6

Memory capacity

  • Reducing memory depth from six blocks to two lowers PLANT success from 85% to 65%, while state-width results are non-monotonic in a single training run.PLANT success at widths 32, 64, 128, and 256 is 15%, 20%, 85%, and 0%, respectively; the authors do not conclude that larger states hold less.Source: C. Results and Discussion · p. 6

Real-robot results

  • 66.7% average success for the best ReCAT variant on three real-robot tasks, versus 8.3% for X-VLA-H, the strongest baseline.ReCAT variants reach 50.0–66.7%; the reported Wilson confidence intervals for the best variant and X-VLA-H do not overlap.Source: C. Results and Discussion · p. 5, Table IV · p. 7

Memory conditioning

  • 66.7% average success with memory cross-attention in every decoder block exceeds late fusion at 38.3% and AdaLN at 31.7%.On PLANT, every-block cross-attention reaches 85%, while no alternative exceeds 10%; SPONGE is an exception, with late fusion at 25% and every-block cross-attention at 20%, a one-rollout difference.Source: C. Results and Discussion · p. 6, Table V · p. 7

Observation encoder

  • 66.7% average success with the hybrid Mamba-attention encoder contrasts with 25.0% for a full-Transformer encoder; removing the memory or multimodal encoder yields 0% on all three tasks.The no-memory comparison tests the temporal module as a whole and does not isolate recurrent-layer and attention-layer contributions.Source: C. Results and Discussion · p. 6, Table VI · p. 7

Computational efficiency

V. CONCLUSIONS

ReCAT combines recurrent memory with separate current-observation and history pathways for action generation. It performs strongly across benchmarks and real-robot memory tasks, while the analysis identifies configuration-dependent effects of the encoder, decoder conditioning, and update rule.

V. CONCLUSIONS

  • 95.3% on LIBERO, 62.4% on RMBench, and 66.7% across three real-robot tasks; latency is 59 ms per step, lowest among compared memory-based policies.The real-robot tasks probe spatial recall, counting, and timing.Source: §V · p. 7
  • Within evaluated configurations, the hybrid observation encoder and every-block memory cross-attention gave the highest average success; additive rules led on counting and timing, and the delta rule on spatial recall.The decoder reads current observations and memory through separate cross-attention in every block.Source: §V · p. 7
  • Stage-wise results show short-history policies execute immediate manipulation but fail at decisions that depend on earlier information.ReCAT’s memory integrates the observation history through Mamba-2 layers and one causal attention layer.Source: §V · p. 7

VI. LIMITATIONS AND FUTURE WORK

The real-robot evaluation is narrow, and the memory ablation does not isolate its recurrent and attention components. Spatial-return manipulation remains a trade-off, motivating broader evaluation and further work under limited demonstrations.

VI. LIMITATIONS AND FUTURE WORK

  • Real-robot evaluation covers one task per memory demand, with 20 rollouts each, leaving generalization across tasks and embodiments for broader study.The authors also call for evaluation across multiple counts and durations distinguished by language instructions.Source: §VI · p. 7
  • The memory ablation removes recurrent and attention layers together, so their individual contributions remain open.The authors identify a remaining trade-off between memory use and fine-grained spatial control, and call for improving spatial-return success under limited demonstrations and larger task diversity.Source: §VI · pp. 7–8
Loading 2609.35200v1…