Source-linked AI summary
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Kuan Fang, Alexander Toshev, Li Fei-Fei, Silvio Savarese
TL;DR
Long-horizon tasks in partially observable environments require agents to retain and use observations from far in the past, while existing memory policies can be rigid or limited. SMT embeds each observation into a growing scene memory and applies attention to exploit spatio-temporal dependencies; across visual-navigation tasks, it outperforms reactive and memory-based policies.
Problem
Long-horizon decision making in partially observable environments depends on distant observations, while existing memory-based policies suffer from limited flexibility and scalability.
Method
SMT stores separately embedded observations in a scene memory and uses an attention-based policy network to compute actions from the current observation and memory.
Results
SMT considerably and consistently outperforms existing reactive and memory-based policies across roaming, coverage, and search visual-navigation tasks.
Takeaways & Limitations
Attention over a generic observation memory provides a policy for exploiting long-term spatio-temporal dependencies in long-horizon robot tasks.
Takeaways & Limitations
Existing memory-based policies are limited by domain-specific memory structures, while SMT assumes a discrete action space under noisy dynamics in its navigation setup.
Abstract
from arXiv · showhide
Many robotic applications require the agent to perform long-horizon tasks in partially observable environments. In such applications, decision making at any step can depend on observations received far in the past. Hence, being able to properly memorize and utilize the long-term history is crucial. In this work, we propose a novel memory-based policy, named Scene Memory Transformer (SMT). The proposed policy embeds and adds each observation to a memory and uses the attention mechanism to exploit spatio-temporal dependencies. This model is generic and can be efficiently trained with reinforcement learning over long episodes. On a range of visual navigation tasks, SMT demonstrates superior performance to existing reactive and memory-based policies by a margin.
1. Introduction
Long-horizon navigation in partially observable environments requires agents to retain detailed histories, but recurrent and structured memory policies can struggle with long-term dependencies, flexibility, and scalability. SMT stores separately embedded observations and uses attention to exploit spatio-temporal dependencies, with linear-complexity factorization for large memories, and outperforms existing policies across three tasks.
- Long-horizon navigation requires agents to maintain detailed memories of past observations and actions in unseen, partially observable environments.
- RNN policies compress history into a fixed-size state vector and often fail to capture long-term dependencies.
- Existing external memories improve over reactive and RNN policies but impose domain-specific structures such as fixed 2D layouts and geometry-based updates.
- SMT separately embeds each observation, stores the embeddings in a scene memory, and defers aggregation until action computation.
- Attention aggregates visual and geometric information from the scene memory, enabling SMT to exploit long-term spatio-temporal dependencies without a predefined environment structure.
- Linear complexity is achieved through memory factorization when many observations are stored, while the memory retains only one embedding vector per step.
- SMT considerably and consistently outperforms reactive and memory-based policies on roaming, coverage, and search tasks.
2. Related Work
Prior robot-navigation policies commonly use recurrent or structured external memories, but these approaches face long-sequence optimization difficulties or rigid memory designs. SMT instead stores observations separately and uses attention for generic visual navigation over long horizons.
- Memory-based policy using RNN: RNN navigation policies, commonly using LSTMs, face optimization difficulties over long sequences during backpropagation through time.
- Memory-based policy using RNN: SMT stores each observation separately and aggregates information only when computing an action, avoiding reliance on backpropagation through time for extracting spatio-temporal dependencies.
- External memory: External-memory models in natural language processing are generally attention-based and designed for long sequences or large sets.
- External memory: Autonomous-agent methods often organize memory as fixed-size 2D grids, allocating equal capacity across environmental regions and limiting flexibility.
- External memory: More generic agent memories include topological maps, attention decoders with LSTM controllers, and Differentiable Neural Computers, but some retain recurrent or optimization challenges.
- Visual Navigation: SMT is evaluated on visual-navigation tasks without scene-specific environment information or a map provided beforehand.
- At each time step, SMT embeds the observation, adds it to scene memory, and uses the full memory with the current observation to produce an action.
3. Method
SMT addresses long-horizon navigation by retaining separately embedded observations and using attention to relate the current observation to the accumulated scene memory. Its factorized variant reduces the encoder’s quadratic cost, while avoiding backpropagation through time supports longer-dependency training.
- 3.1. Problem Setup: The task is modeled as navigation in unseen environments under a POMDP with multimodal observations and noisy discrete actions.Observations contain RGB, depth, semantic segmentation, pose, and the previous action; actions are forward, left, or right.
- 3.2. Scene Memory Transformer: SMT stores every past observation as a separate embedded memory element instead of compressing history into one recurrent state.The memory is generic, has no imposed geometric structure, and is updated by adding the current observation embedding.
- 3.2.2. Attention-based Policy Network: The policy self-attends over memory to capture spatio-temporal dependencies, then decodes an action distribution using the current observation as the query.The encoder transforms each memory element using other past observations, while the decoder uses the encoded memory as context.
- 3.2.2. Attention-based Policy Network: The attention network uses projected queries, keys, and values with residual layers and layer normalization to transform memory and observation representations.The attention block applies attention followed by a fully connected layer, with residual connections and normalization.
- 3.2.3. Memory Factorization: SMT’s encoder costs O(|M|^2) and its decoder costs O(|M|), making encoder self-attention potentially prohibitive as long-horizon memory grows.Memory factorization replaces self-attention with attention through a compressed memory of representative centers selected using farthest point sampling, yielding O(|M||M_tilde|).
- 3.3. Training: SMT avoids backpropagation through time, making optimization more stable and less computationally heavy and enabling exploitation of longer temporal dependencies.The memory grows linearly with episode length but stores only low-dimensional embeddings, allowing hundreds of observations on device.
4. Experiments
Experiments evaluate SMT on roaming, coverage, and search, showing consistent gains over reactive and memory-based policies. Ablations and qualitative analyses examine memory capacity, modalities, components, and learned navigation behavior.
- Task Setup: SMT is evaluated on three visual navigation tasks—roaming, coverage, and search—with task-specific reward functions and shared partially observable environments.The tasks increase in complexity from collision-free movement to spatial coverage and semantic object search.
- Comparative Evaluation: 14% improvement on coverage and 5% improvement on search represent SMT’s largest relative gains over other memory-based approaches.Across all tasks, SMT outperforms the other memory-based models; roaming methods perform in a similar range.
- Comparative Evaluation: Attention-based memory access improves all tasks over SM + Pooling, with larger gaps on semantic search; multiheaded attention and residual layers also improve coverage and search.These comparisons isolate benefits from attention-based access and from the richer encoder design.
- Comparative Evaluation: 18% coverage improvement over reactive policies, 17% over LSTM policies, and 23% over FRMQN accompany a 5% increase in found objects for search.These task metrics indicate gains in both spatial exploration and semantic discovery.
- Ablation Analysis: Larger scene memories improve all tasks: roaming benefits up to 300 elements, while coverage and search continue improving with larger capacities.The ablation directly tests whether SMT uses the memory capacity supplied to it.
- Ablation Analysis: Last action, pose, and depth are crucial across tasks, while removing RGB and segmentation hurts search more than geometry-focused tasks; the encoder adds 23.7 reward in search.The reported drops are 10 for roaming and 15 and 20 for search when segmentation and RGB observations are removed.
- Qualitative Results: Qualitatively, SMT forms compact roaming trajectories, switches from explored to unvisited rooms, and moves directly between discovered targets.These behaviors rely on memorized spatial coverage and target-object information across long episodes.
5. Conclusion
The paper introduces Scene Memory Transformer, a memory-based policy that uses attention to aggregate observation history for long-horizon robotic tasks. Trained with deep Q-learning, it achieves higher performance than established methods across visual navigation tasks.
- Scene Memory Transformer aggregates observation history with attention mechanisms to exploit spatio-temporal dependencies in long-horizon robotic tasks.
- The policy is trained on several visual navigation tasks using deep Q-learning.
- Evaluation shows that SMT achieves higher performance than other established methods.
A. Environment Details
Experiments simulate a mobile Fetch robot with head-mounted RGBD sensing and noisy planar motion. Images are rendered at 640 × 480 resolution with depth restricted to the sensor’s 5-meter operating range.
- Experiments simulate a mobile base of the Fetch robot equipped with a Primesense Carmine 1.09 short-range RGBD sensor.
- The visual observations are rendered at 640 × 480 resolution.
- Depth values are rendered only for points within 5 meters of the camera, alongside a binary valid-depth mask.
- The planar Fetch dynamics add zero-mean Gaussian noise with standard deviation 0.5 rad/s to both wheels.
B. Analysis of Memory Factorization
SMT factorizes growing memories using representative centers selected by farthest point sampling. Compared with recent-history and static-center baselines, FPS performs especially well on coverage and search, while positional embeddings are evaluated under noisy dynamics.
- Representative memory centers should cover past observations and remain dynamically updated as the memory grows.
- FPS compares against Window, which retains the most recent history, and Static, whose inducing points remain fixed during testing.
- With 100 representative centers, FPS matches Static on roaming and outperforms both baselines on coverage and search.
- Figure 7 compares three positional embedding methods while sweeping noise standard deviation from 0.0 to 1.0.
C. Robustness to Noisy Dynamics
SMT estimates pose from actions and previous pose estimates when ground-truth poses are unavailable, then evaluates temporal embeddings under noisy dynamics. Exponential embeddings encode recency, and temporal information improves performance in clean and noisy observations.
- Pose is estimated recursively from each action and the previous estimated pose instead of retrieving ground-truth poses.
- SMT uses exponential time-step embeddings to provide temporal information for the policy.
- Exponential embeddings are monotonic and represent how recent each time step is, unlike periodic sinusoidal embeddings.
- The evaluation compares no embedding, sinusoidal, and exponential embeddings while sweeping noise standard deviation from 0.0 to 1.0.
- Temporal information improves performance with clean observations and helps leverage noisy environment dynamics.
D. More Visualization
The appendix visualizes agent behaviors in roaming, coverage, and search tasks using top-down trajectory views and task-specific annotations.
- Visualization: Figures 8–10 visualize agent trajectories for the roaming, coverage, and search tasks.The trajectories are shown from a top-down view.
- Visualization: Green curves show trajectories, while white and black dots mark starting and ending points.Navigable areas are dark purple, and red lines indicate collision boundaries.
- Task-specific annotations: Coverage visualizations mark covered cells in pink, while search visualizations mark target objects in yellow.