PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Training as Residual Memory for Robot Policies

Haoxuan Wang, Gengyu Zhang, Ramana Rao Kompella, Gaowen Liu, Yan Yan

arXiv:2610.04701v1cs.RO

TL;DR

Long-horizon robot policies need historical information that current observations may not recover, while storing more history is costly and existing memory designs either retain selected records or compress history. TTT-RM uses reconstruction residuals to train fast weights to complement retrieved memory, and improves performance across simulation and real-world tasks.

  • Problem

    Long-horizon manipulation is partially observable, and existing policies use limited context or bounded memory that may omit or evict task-relevant historical information.

    Source: §1 · p. 1

  • Method

    TTT-RM trains a history decoder to reconstruct past representations from current context and uses the resulting residual to supervise fast weights that complement retrieved memory during action generation.

    Source: §1 · p. 2

  • Results

    TTT-RM improves real-world task progress by 10% to 26.5% and consistently outperforms diverse baselines across memory-intensive simulation benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TTT-RM generalizes across multiple memory designs and supports sustained execution on a three-minute, eight-stage real-world stowing task.

    Source: §6 · p. 10

Abstract

from arXiv · show

Memory is essential for long-horizon robotic manipulation, where successful actions may depend on past events that are no longer recoverable from the current observation. As episodes grow longer, however, retaining the full history becomes increasingly costly, creating a fundamental scalability challenge for memory-augmented policies. Existing approaches address this challenge by either storing selected past observations in a bounded memory bank or compressing interaction history into a fixed-size parametric state through Test-Time Training (TTT). Yet these formulations do not explicitly distinguish between historical information that can already be recovered from the policy's current context and information that must persist beyond it. We introduce TTT-RM, which repurposes TTT as Residual Memory, using fast weights not to generically compress history but to complement a bounded memory bank by preserving task-relevant historical information that cannot be recovered from the policy's current context. Concretely, TTT-RM learns a history decoder that reconstructs historical representations from the current observation and retrieved memory. The resulting reconstruction residual captures what this context fails to explain and serves as the learning target for TTT. The TTT slow weights are optimized against this residual target so that online fast-weight updates learn to encode complementary historical information over time. The fast-weight state is then queried to produce a residual memory representation that conditions action generation. Extensive experiments on memory-intensive simulation benchmarks and real-world tasks show that TTT-RM consistently improves across multiple memory designs, outperforms diverse baselines, and supports sustained execution on a three-minute, eight-stage stowing task.

1. Introduction

Long-horizon robotic manipulation requires policies to use historical evidence because the current observation may not reveal the task state needed for the next action. TTT-RM combines bounded Explicit Memory with Test-Time Training specialized to preserve history that the available context cannot recover, and improves performance across simulation and real-world tasks.

1. Introduction

  • Task-relevant information may be spread across interaction history, including occluded objects, delayed instructions, and completed subgoals, making memory necessary for history-dependent action prediction.Current observations alone may not contain the information needed to determine the next action.Source: §1 · p. 1
  • Explicit Memory retains selected observations in a bounded bank, while TTT integrates history into fixed-size fast weights; TTT-RM uses the latter for information unrecoverable from the existing context.The figure contrasts sparse access to retained records with residual-history retention.Source: §1 · p. 2, Figure 1 · p. 2
  • TTT-RM trains a history decoder to reconstruct past representations from the current observation and retrieved memory, then uses the reconstruction residual to train fast weights to encode complementary history.The queried fast-weight state produces a Residual Memory representation that is combined with retrieved memory to condition action generation.Source: §1 · p. 2
  • TTT-RM consistently outperforms diverse baselines across memory-intensive simulation benchmarks and improves real-world task progress by 10% to 26.5%.It also supports sustained execution on a three-minute, eight-stage stowing task.Source: §1 · p. 2

2. Related Work

Prior robot-memory work stores selected history for retrieval, while Test-Time Training uses fast weights as an online memory state. TTT-RM builds on both lines but trains fast weights to preserve residual information unavailable from current context and retrieved memory.

2.1. Explicit Memory for Robot Learning

  • Explicit Memory approaches retain selected history as retrievable records, varying in what they store and how they organize it across visual, semantic, event, and temporal representations.Examples include keyframes, visual anchors, task-critical events, gist tokens, and separate short- and long-term stores.Source: §2.1 · pp. 2–3

2.2. Test-Time Training and Fast-Weight Memory

  • Robot-learning TTT methods use fast weights as memory for history-dependent control, whereas TTT-RM trains them to preserve information unrecoverable from current observation and retrieved memory.RoboTTT compresses interaction history into recurrent fast-weight states, while WAM-TTT meta-trains adaptive memory from unlabeled human videos at deployment.Source: §2.2 · p. 3

3. Preliminaries

The preliminaries formulate memory as an episode-scoped state for policies operating under partial observability. They describe Explicit Memory as bounded retrieval and TTT as online fast-weight updates with fixed memory footprint.

3. Preliminaries

  • Under partial observability, similar current observations can follow different histories and require different actions, so policies use an episode-scoped memory state that carries information from prior interactions.The state is updated causally across policy calls and reset at the beginning of each episode.Source: §3 · p. 3
  • Explicit Memory retains a bounded set of historical representations for direct policy access, with its memory budget limiting the selected prior visual-feature addresses.The policy's memory state is the retained memory set.Source: §3 · p. 3
  • TTT encodes interaction history in fast weights updated online through an inner-loop objective, enabling a fixed-size memory state independent of episode length.The write associates a key with its target value; the updated model is queried to read from memory, while fast weights reset each episode and update at every policy call.Source: §3 · pp. 3–4

4. Methodology

TTT-RM uses a history decoder to identify what the policy’s current observation and retrieved memory cannot recover, then trains TTT fast weights to preserve that residual information for action generation. Its training separates decoder learning, Residual Memory learning, and full-policy fine-tuning; inference removes the decoder and updates fast weights once per policy call.

4. Methodology

  • TTT-RM combines Explicit Memory with TTT-based Residual Memory, using decoder-defined reconstruction residuals to target history missing from the policy’s current context.The resulting fast-weight readouts condition action generation alongside retrieved memory.Source: §4 · p. 4, §4.1 · p. 4

4.1. History Decoder Learning

  • The history decoder reconstructs historical representations from the current observation and retrieved memory, distinguishing recoverable history from information the existing context leaves unexplained.Its fixed visual targets are formed by projecting frozen-encoder features into the TTT representation space, and the decoder is frozen after training.Source: §4.1 · pp. 4–5

4.2. Test-Time Training as Residual Memory

  • TTT writes incoming visual features into fast weights and is trained to make address-conditioned readouts approximate the history decoder’s unexplained residuals.Residual readouts subtract the episode-initial fast-weight response, and the resulting representations condition action generation through an attention adapter.Source: §4.2 · pp. 5–6

4.3. Training and Inference

  • Training proceeds by fitting and freezing the decoder, training Residual Memory components with inherited Explicit Memory and action policy frozen, then fine-tuning the full policy.At inference, the decoder is discarded, fast weights reset at each episode and updated once per policy call; the updated state is reused across that call’s flow-integration steps.Source: §4.3 · p. 6

5. Experiments

Experiments span two memory-intensive simulation benchmarks and three real-world manipulation tasks, with baseline comparisons and targeted ablations. TTT-RM shows consistent gains across these settings.

5. Experiments

  • Evaluation covers RoboMME’s 16 tasks, RMBench’s nine dual-arm tasks, and three real-world tasks, using common benchmark protocols and 20 varied-configuration trials per real-world method and task.The real-world study uses approximately one hour of demonstrations per task.Source: §5.1 · p. 7, §5.2 · p. 9

5.1. Simulation Benchmarks

  • Moderate λres values yield consistent gains, plain EM+TTT adds limited or no improvement, and sequential Stage 2+3 training performs best across both EM variants.Stage 2 freezes the inherited policy; the authors hypothesize this warm-up reduces early optimization competition before Stage 3 joint fine-tuning.Source: Figure 3 · p. 8, §5.1 · pp. 8–9
  • TTT-RM improves every Explicit Memory configuration by 2.2–7.1 percentage points and wins all 16 configuration–suite comparisons; FrameSamp w/ Modul rises from 42.8% to 45.4%.On RoboMME, standalone TTT has mixed performance across integration designs.Source: §5.1 · p. 8
  • TTT-RM improves over EM by 6% on M(1) and 5% on M(n) in RMBench.Standalone TTT is mixed across the two settings, while EM is the strongest baseline in both.Source: Table 2 · p. 8, §5.1 · p. 8

5.2. Real-World Experiments

  • Across the three real-world tasks, TTT-RM beats every baseline at every progress threshold; mean progress rises from 47.0% to 73.5% on Pick N pens and 50.5% to 77.0% on Stow objects.The progress profiles report the fraction of trials reaching or exceeding each task-progress threshold.Source: §5.2 · p. 9, Figure 4 · p. 10

6. Conclusion

TTT-RM uses residual-supervised Test-Time Training to preserve task-relevant history unavailable from the current observation and Explicit Memory. It generalizes across multiple memory designs, consistently outperforms diverse baselines, and supports a three-minute, eight-stage real-world stowing task.

  • TTT-RM generalizes across multiple memory designs and consistently outperforms diverse baselines.Source: §6 · p. 10
  • TTT-RM supports sustained execution on a three-minute, eight-stage real-world stowing task.Source: §6 · p. 10
  • A history decoder reconstructs historical representations from available context, and its residual supervises Residual Memory learning.The resulting representation is combined with retrieved memory to condition action generation.Source: §6 · p. 10
Loading 2610.04701v1…