PaperlayerView exact revision on arXiv

Source-linked AI summary

LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Qixin Sun, WenLong Gao, Xiansheng Chen, Zhongyuan Wang, Si Liu, Pengwei Wang

arXiv:2610.21362v1cs.ROcs.AI

TL;DR

Robotic manipulation can depend on interactions no longer visible in the current observation. LTM stores history as compact latent transitions and integrates them into π0.5 without memory-specific semantic annotations or external reasoning-model calls. With DreamDojo transitions, it reports 58.54% success on RoboMME and 80.5% on LIBERO-Mem.

  • Problem

    A current observation can omit earlier interactions needed for action, while visual histories, language summaries, and recurrent states each have stated representational trade-offs.

    Source: §1 · p. 1

  • Method

    LTM encodes consecutive observations as compact latent transitions and supplies them as an ordered memory prefix to the policy, using a frozen encoder and lightweight adapter.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    With DreamDojo transitions, LTM-π reaches 58.54% success on RoboMME and 80.5% on LIBERO-Mem at the original data scale.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Across evaluated history-dependent manipulation tasks, compact latent transitions provide an effective episodic memory representation, with temporal composition and informative event boundaries remaining directions for improvement.

    Source: §6 · p. 14

    Memory quality depends on the pretrained latent action encoder, which may inadequately represent major scene changes such as newly appearing objects; sparse visual anchors may not fully address this.

    Source: §5 · p. 13

Abstract

from arXiv · show

Robotic manipulation often requires remembering interactions that are no longer visible. We introduce Latent Transition Memory (LTM), which stores history as compact latent transitions between consecutive observations. Instantiated on $\pi_{0.5}$ as LTM-$\pi$, our method uses a frozen latent action encoder, a lightweight adapter, and causal KV caching for incremental inference without memory-specific semantic annotations or external reasoning-model calls. With DreamDojo transitions, LTM-$\pi$ achieves 58.54% success on RoboMME and 80.5% on LIBERO-Mem, performing best among baselines without external reasoning models or memory-specific annotations. Ablations support the contribution of transition content beyond temporal cues and the efficiency of delayed memory admission. Scaling RoboMME demonstrations tenfold improves success from 57.67% to 87.50%, while a Cup Swap diagnostic highlights the role of process supervision in temporal composition. Under a 200-token memory budget, success remains 84.00%, versus 87.50% without compression. Finally, real-robot counting success improves from 11.25% to 82.50%. Together, these results demonstrate the effectiveness, scalability, and efficiency of latent transition memory, highlighting its capacity for complex temporal information, bounded-context operation, and real-world robotic control.

1 Introduction

Latent Transition Memory (LTM) represents interaction history as compact transitions between consecutive observations, addressing cases where the current view omits information needed for action. LTM-π combines these transitions with a lightweight adapter and causal caching, with reported gains across benchmarks, scaling, compression, and real-robot counting.

1 Introduction

  • LTM stores compact latent transitions between consecutive observations as history, using a frozen latent action encoder and lightweight adapter without memory-specific semantic annotations or external reasoning-model calls.Causal attention supports persistent KV caching, while delayed transition prefixes allow encoding to overlap action execution.Source: §1 · p. 2
  • 58.54% average success on RoboMME versus 17.93% for memory-free π0.5, and 80.5% on LIBERO-Mem versus 42.5% for the strongest included baseline.These results use DreamDojo transitions and the original benchmark data.Source: §1 · p. 2
  • Replacing observation-dependent transitions with fixed random representations lowers RoboMME success from 57.67% to 44.50%; a five-step lag reduces estimated encoder blocking from 114.25 ms to 9.47 ms as success changes from 57.88% to 57.67%.The ablation supports transition-specific content beyond temporal-progress cues, while delayed admission reduces blocking with a small success change.Source: §1 · p. 2
  • Tenfold RoboMME data expansion raises success from 57.67% to 87.50%, with gains across all four tracks; 120K training steps on original data alone yield no appreciable gain.The expanded setting uses 1,000 rather than 100 episodes per task family and trains for 120K rather than 40K steps.Source: Figure 1 · p. 1, §1 · p. 2
  • In the Cup Swap diagnostic, process supervision recovers final states from latent transitions with 97.93% accuracy, versus 2.40% with final-state-only supervision, while swap tasks remain below their non-swap counterparts.VideoUnmaskSwap and ButtonUnmaskSwap reach 74% and 62%, respectively, versus 100% for their non-swap counterparts.Source: §1 · p. 2
  • A 200-token budget retains 84.00% success versus 87.50% uncompressed, while real-world repeated pick-and-place counting success rises from 11.25% with memory-free π0.5 to 82.50% with LTM-π.Token merging reduces mean transition-token count by at least 38.26% on profiled training queries.Source: §1 · p. 2

2 Related Work

Related VLA memory work addresses partial observability through recurrent states, memory tokens, retrieval, object-centric representations, or higher-level reasoning. LTM differs by exposing ordered transitions directly to the action policy, using pretrained latent action models to encode observed changes.

2 Related Work

  • VLA policies can require events absent from the current observation, and RoboMME and LIBERO-Mem evaluate this need for execution memory.The cited approaches include tokenized-action and flow-matching policies, as well as generalist models trained across heterogeneous data.Source: §2 · p. 3
  • Prior VLA memories retain history through sequential or learned memory states, retrieved visual evidence, spatial or object-centric representations, or cached image and action tokens.Other approaches incorporate motion or imagined futures, combine recent video with long-term textual memory, or retrieve keyframes for a higher-level model to guide a policy.Source: §2 · p. 3
  • Several systems delegate historical interpretation to a higher-level model or interface, whereas LTM provides observed transitions directly to the action policy without separate reasoning calls or intermediate language generation during control.SimpleMemVLA uses offline cloud-VLM subtask annotations; T2Mem updates fast-weight memory through test-time gradients, while LTM appends frozen pairwise codes without such updates.Source: §2 · p. 3
  • Latent action models infer compact transition codes from observation pairs, but LTM uses pretrained models only to encode observed transitions into episodic memory rather than reconstructing past frames or generating future video.Prior uses include controllable video generation, policy pretraining, and world-model pretraining.Source: §2 · p. 3

3 Method

LTM represents interaction history as compact latent transitions between consecutive observations, which are projected into memory tokens for a π0.5 policy. Delayed admission and causal KV caching support incremental inference, while sparse visual anchors supplement memory for video-conditioned tasks.

3.1 Latent-Transition-Based In-Context Memory for VLAs

  • LTM encodes each consecutive-observation pair as a compact latent transition, then projects it into a memory token for policy conditioning.The decoder reconstructs the later observation conditioned on the earlier one, encouraging the latent bottleneck to capture temporal change; transition codes are low-dimensional and depend only on two observations.Source: §3.1 · p. 4

3.2 LTM-𝜋

  • LTM-π instantiates the memory mechanism on π0.5, using a frozen latent encoder and a lightweight adapter trained without language summaries or intermediate semantic supervision.A two-layer MLP maps normalized 32-dimensional transition vectors into memory tokens in the backbone’s 2048-dimensional hidden space.Source: §3.2 · p. 5
  • At inference, transitions are encoded asynchronously and admitted after a fixed lag into a persistent causal KV cache, overlapping encoding with action execution.For n cached prefix tokens and b new tokens, caching changes per-chunk backbone self-attention cost from O((n + b)^2) to O(nb + b^2); for fixed b, this is linear in cached history length.Source: §3.2 · p. 5
  • For video-conditioned tasks, sparse visual features from demonstration and execution sequences supplement transition memory to help align differing scene layouts.A minimal choice uses the first frame of each sequence as a visual anchor; these anchors are retained in the persistent prefix alongside demonstration tokens.Source: Figure 2 · p. 4, §3.2 · p. 5

4 Experiment

The experiment section evaluates LTM-π’s effectiveness, information content, and memory cost in simulation and on a real robot.

4.1 Simulation

  • Simulation experiments report RoboMME and LIBERO-Mem results and isolate the effects of memory components and inference lag.The section introduces the benchmarks and evaluation protocol before reporting results.Source: §4 · p. 6
  • Further simulation tests examine demonstration scaling, latent-transition swap information, and manipulation under a fixed token budget.Source: §4 · p. 6

4.2 Real-robot evaluation

  • The final experiment section evaluates repeated pick-and-place manipulation on a real robot.Source: §4 · p. 6

4.1 Evaluation in Simulation

LTM-π performs strongly across RoboMME and LIBERO-Mem, and broader demonstrations further improve RoboMME results. Ablations, a Cup Swap diagnostic, and token compression tests examine transition content, temporal composition, and bounded memory.

4.1.1 Settings

  • The evaluation covers 16 history-dependent RoboMME tasks and 10 long-horizon LIBERO-Mem tasks, with comparisons against published baselines.The benchmarks target memory for earlier observations or demonstrations, including object identity, interactions, and state changes.Source: §4.1.1 · pp. 6–7

4.1.2 Main Results

  • 58.54% RoboMME success with DreamDojo ranks first among compared methods using neither external reasoning models nor memory-specific annotations, while 80.5% on LIBERO-Mem exceeds its strongest included baseline.The RoboMME result is 1.71 percentage points above T2Mem; the LIBERO-Mem result is 38.0 points above MemoryVAM (UNet).Source: §4.1.2 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 8

4.1.3 Ablations

  • Removing latent transitions lowers average RoboMME success from 57.67% to 23.13%; LT-only retains 49.13%, while fixed random transitions reach 44.50%.The results indicate that transition content contributes beyond temporal cues, while visual anchors provide complementary grounding.Source: §4.1.3 · pp. 8–9, Table 3 · p. 9
  • At lag k=5, success changes from 57.88% at k=0 to 57.67%, while estimated LAM blocking falls from 114.25 ms to 9.47 ms.At k=10, estimated blocking is zero with 56.38% success; blocking is an overlap estimate, not measured end-to-end latency.Source: §4.1.3 · p. 9, Table 4 · p. 9

4.1.4 Data Scaling on RoboMME

  • 87.50% RoboMME success at 10× data improves on 57.67% at 1×, while extending original-data training alone yields no appreciable gain.All four tracks improve; Reference gains 47.50 points, and the comparison with PonderPounce is descriptive because training and supervision budgets differ.Source: §4.1.4 · pp. 9–10, Table 5 · p. 9

4.1.5 Diagnostic Study: Can Latent Transitions Represent “Swap”?

  • 97.93% final-state recovery under process supervision contrasts with 2.40% under final-state-only supervision in the six-cup Cup Swap diagnostic.The diagnostic uses a separate latent-fusion architecture and frozen VLM backbone; low training accuracy for final-state-only supervision may contribute to the gap.Source: §4.1.5 · pp. 10–11, Figure 3 · p. 10, Table 6 · p. 11

4.1.6 Token Compression: Can Transition Memory Remain Effective Under a Fixed Budget?

  • Under a 200-token budget, merging the most similar adjacent projected tokens bounds execution-memory length while retaining temporal order among remaining segments.The operation averages a selected pair and removes an event boundary; repeated equal-weight merges are lossy and do not track constituent transition counts.Source: §4.1.6 · pp. 11–12
  • 84.00% success with 200-token compression compares with 87.50% uncompressed on RoboMME-10x, retaining 96.00% of the reference success rate.Several tasks remain at 100%, while swap-task declines are consistent with sensitivity to temporal boundaries; single-seed evaluation does not resolve smaller gains.Source: §4.1.6 · p. 12, Table 8 · p. 12

4.2 Real-World Evaluation

The real-world evaluation uses a Galbot G1 for repeated pick-and-place counting. LTM-π achieves higher success than memory-free π0.5, with shorter motion-onset delays also observed qualitatively.

4.2.1 Settings

  • The Galbot G1 evaluation used 200 teleoperated trajectories across targets of one to four objects, with only its arms and grippers actuated.The dataset contains 222,834 frames recorded at 30 FPS; real-world trials used no anchor frames.Source: §4.2.1 · p. 13

4.2.2 Results

  • Across four target quantities, LTM-π achieved an unweighted average success rate of 82.50%, versus 11.25% for memory-free π0.5, a gain of 71.25 percentage points.Source: §4.2.2 · p. 13, Table 9 · p. 13
  • LTM-π showed shorter motion-onset delays than the memory-free policy in trials with long initial static segments, but latency was not measured separately.The authors describe this qualitative pattern as consistent with temporal-progress cues from the memory sequence.Source: §4.2.2 · p. 13

5 Limitation

LTM depends on the information preserved by its pretrained latent action model. The authors identify substantial scene changes, such as new objects, as a boundary for transition representations.

5 Limitation

  • Memory quality depends on the information preserved by the pretrained latent action model.The encoder supplies the transition representations used by the method.Source: §5 · p. 13
  • Latent transitions may inadequately represent substantial scene changes, including the appearance of new objects where visual and semantic information matters.The limitation concerns scene content not captured by local interaction dynamics.Source: §5 · p. 13
  • Sparse visual anchors may not fully address this boundary; combining transition memory with visual memory is proposed as a direction for more robust long-horizon manipulation.The proposed combination would integrate dynamic changes with visual context.Source: §5 · p. 13

6 Conclusion

The conclusion reports gains on benchmark and real-world manipulation tasks using ordered latent transition memory. It also highlights demonstration scaling, process supervision, and bounded-context compression as findings and directions for development.

6 Conclusion

  • At the original data scale, LTM-π with DreamDojo transitions reaches 58.54% success on RoboMME and 80.5% on LIBERO-Mem.The method uses frozen latent action encoders, a lightweight adapter, and cache-compatible causal attention, without memory-specific semantic annotations or an external reasoning model during control.Source: §6 · p. 14
  • 87.50% success versus 57.67% follows a tenfold increase in RoboMME data and training from 40K to 120K steps; longer training on original data alone yields no appreciable gain.Gains occurred across all four tracks, with the largest improvement on Reference.Source: §6 · p. 14
  • 97.93% final-state accuracy versus 2.40% under final-state-only supervision in the Cup Swap diagnostic highlights intermediate supervision for temporal composition.Swap-dependent state tracking remains challenging after data scaling.Source: §6 · p. 14
  • A 200-token budget retains 84.00% success on RoboMME-10x versus 87.50% uncompressed, while reducing mean transition-token count by at least 38.26% on profiled training queries.The conclusion notes that cache refresh costs must be considered when translating token-count reductions into runtime gains.Source: §6 · p. 14
  • 82.50% average success on real-world repeated pick-and-place counting, versus 11.25% with memory-free π0.5.The conclusion presents compact latent transitions as effective episodic memory for the evaluated history-dependent manipulation tasks.Source: §6 · p. 14

A Cup Swap Diagnostic Experiment Details

The Cup Swap diagnostic tests recovery of final ball occupancy from a sequence of swaps using latent transitions. Its setup separates motion reasoning from visual recognition of the initial configuration.

Task and dataset

  • The synthetic Cup Swap dataset contains 14,000 videos with six cups and one to five balls, followed by 5–20 randomly generated swaps.The task is to recover final ball occupancy after composing the complete sequence; the split is 11,176 training, 1,375 validation, and 1,449 test examples.Source: A Cup Swap Diagnostic Experiment Details · p. 18

Latent-transition extraction

  • Adjacent frame pairs are encoded as 32-dimensional transitions with stride 1, retaining only transitions sourced at or after reveal frame 42.The initial occupancy is supplied separately as text, and the reveal transition is excluded from the latent stream.Source: A Cup Swap Diagnostic Experiment Details · p. 18

Input and latent-memory fusion

  • The initial cup occupancy is provided as text, while every four consecutive latents are compressed into one projected memory token for the Qwen3-VL-2B-Instruct model.The latent stream uses eight layers of self-attention and the language states read it through eight layers of cross-attention.Source: A Cup Swap Diagnostic Experiment Details · p. 18

Supervision variants

  • Process supervision tracks the swap process and evolving state before final-state prediction, whereas the Vanilla objective supervises only final cup occupancy.The process-supervised objective is CoT-like in function but does not generate a natural-language chain of thought.Source: A Cup Swap Diagnostic Experiment Details · p. 18

Training configuration

  • The process-supervised run uses eight H100 80GB GPUs with global batch size 32, while the Qwen backbone remains frozen.The main learning rate is 1 × 10^-4 and the classification-head learning rate is 3 × 10^-4.Source: A Cup Swap Diagnostic Experiment Details · p. 18

Additional metrics

  • 97.93% final-state accuracy and 97.52% full exact accuracy are reported for process supervision, versus 2.40% best exact accuracy for the Vanilla six-cup objective.A simpler three-cup, single-ball, fixed-five-swap classifier is not compared directly because its state space and task difficulty are much smaller.Source: A Cup Swap Diagnostic Experiment Details · p. 18

B Real-World Evaluation

The real-world evaluation measures object-counting accuracy in packaging-bag trials, with success requiring an exact count and a stationary return to the initial pose. It reports per-trial transferred-object counts and provides representative videos.

B Real-World Evaluation

  • A trial is correct only when the transferred-object count matches the instruction and the robot returns to its initial pose and remains stationary for at least 10 seconds.Each model is evaluated for 20 trials per target quantity, except memory-free π0.5 at quantities 3 and 4, which received 10 trials each because it repeatedly continued picking after the requested count.Source: B Real-World Evaluation · p. 19
  • Table 10 lists per-trial transferred-object counts; bold entries match the instructed quantity, and unperformed trials are excluded from the accuracy denominator.Trial indices are local to each model–quantity setting.Source: Table 10 · p. 19, B Real-World Evaluation · p. 19
  • Representative videos include successful LTM-π pick-and-place trials and a memory-free π0.5 failure.The videos are available in the project repository.Source: B Real-World Evaluation · p. 19

Related papers

Loading 2610.21362v1…