Source-linked AI summary
HitMem: Hierarchical Temporal 3D Memory with Multi-Modal Context-Aware Retrieval for Dynamic Environments
Ruijie Tang, Chenye Zou, Guoquan Wu, Jun Wei, Wei Chen, Jiaxin Zhu
TL;DR
Dynamic object displacement exposes the static-world assumptions of existing 3D memories, causing memory conflicts and inefficient relocation. HitMem combines hierarchical temporal 3D memory, temporal decay, and context-aware retrieval, and evaluations on Dyna-THOR report improved relocation, exploration, and task execution. Its evaluation uses a simulator-based benchmark and standardized perception, reasoning, and navigation conditions.
Problem
Existing 3D memory methods assume static environments, limiting robust handling of displaced objects and leaving benchmarks insufficiently dynamic.
Method
HitMem combines hierarchical spatial-semantic-temporal memory, temporal decay, and two-stage retrieval using external-agent trajectories and semantic class affinities.
Results
Evaluations on Dyna-THOR show improved relocation accuracy, exploration efficiency, adaptability, and long-term task execution for HitMem.
Takeaways & Limitations
HitMem provides a dynamic-environment memory and retrieval framework that targets displaced-object localization while reducing redundant exploration.
Abstract
from arXiv · showhide
Executing long-term tasks in dynamic environments requires embodied agents to maintain robust and adaptive 3D scene representations. However, most existing 3D memory frameworks rely on static world assumptions. When objects are displaced by human activities or unobserved events, agents encounter memory-observation conflicts and often require costly geometric recomputations or inefficient global re-exploration. To address this, we propose HitMem, a hierarchical temporal 3D memory framework with a multi-modal context-aware retrieval mechanism. Through continuous perception, HitMem unifies semantic and spatial information into a lightweight topological graph that captures support relationships, while a temporal decay mechanism dynamically regulates memory activeness to mitigate the impact of stale representations. In addition, the multi-modal context-aware retrieval mechanism defaults to filtering candidates using integrated semantic, spatial, and temporal memory features, and activates a specialized two-stage retrieval process when object displacement is detected. This process combines spatial constraints inferred from external agent trajectories with semantic common sense grounded in class affinities, efficiently identifying high-probability candidate regions. Extensive evaluations on our constructed Dyna-THOR benchmark demonstrate that HitMem significantly improves object relocation accuracy, reduces exploration costs, and enhances task execution performance in dynamic environments.
1 Introduction
Existing 3D memory methods assume static environments, creating conflicts with observations after object displacement and making relocation inefficient. HitMem addresses this with hierarchical temporal memory and context-aware retrieval that combines spatial, semantic, and temporal cues.
- Motivation: Existing methods assume static worlds, so displaced objects create memory conflicts and obsolete representations.Dynamic environments can require costly relational or geometric recomputation, while temporal dynamics are often unmodeled.
- Motivation: Human or unobserved events can displace objects, making exhaustive global search or single-modality retrieval inadequate.Effective relocation requires combining spatio-temporal dynamics with semantic common sense.
- HitMem: HitMem unifies semantic, spatial, and temporal information in a lightweight hierarchical representation with detailed low-level 3D attributes.Its temporal decay mechanism reduces reliance on obsolete observations by modulating memory-node activeness over time.
- HitMem: The retrieval system first filters candidates using hierarchical memory features and temporal activeness, then invokes two-stage retrieval for detected displacement.The displacement strategy prunes regions using external-agent trajectories and ranks candidate carriers through class-affinity analysis.
- Evaluation: Dyna-THOR introduces external-agent interactions that relocate task-relevant objects during execution, enabling evaluation under unpredictable dynamics.Experiments report improved relocation accuracy, exploration efficiency, and adaptability for HitMem.
2 Related Work
Prior embodied-agent memories and 3D representations struggle with efficient, persistent retrieval and real-time updates in dynamic environments. HitMem combines hierarchical temporal memory with trajectory- and class-affinity-based retrieval to localize displaced objects more efficiently.
- 3D Scene Representations: Dense reconstruction captures geometry but lacks high-level semantics, while VLM-enhanced 3D methods remain expensive and difficult to update in real time.These limitations reduce suitability for semantic queries and dynamic environments.
- Memory for Embodied Tasks: Short-term and image-based long-term memories provide limited persistence or efficiency for reliable long-term embodied tasks.The supplied related-work passage introduces these limitations without detailing all downstream consequences.
- HitMem: HitMem uses hierarchical memory with temporal dynamics and multi-modal retrieval for rapid target localization.Its two-stage strategy integrates external-agent trajectories with semantic class-affinity scanning to reduce the search space.
3 Method
HitMem builds a hierarchical temporal 3D memory that combines semantic, spatial, and temporal information for dynamic-scene retrieval. It uses structured support relationships, memory activeness, and context-aware relocation to update and search scene representations efficiently.
- Hierarchical 3D Representation: HitMem continuously constructs an object-centered hierarchical memory graph from RGB-D observations, camera poses, semantic embeddings, and 3D geometric attributes.Each node stores semantic and spatial information derived from detected instances.
- Hierarchical 3D Representation: Carrier and carried-object nodes form a directed topological structure based on semantic carrier classification and geometric support relationships.Support combines proximity, horizontal coverage, vertical support, and height consistency, with edges added when the maximum score exceeds θsup.
- Hierarchical 3D Representation: Unassigned objects remain in memory and are reevaluated when new carriers appear, enabling localized node insertion or deletion instead of global graph reconstruction.This dynamic Cr → Cd structure is designed to reduce update costs as exploration continues.
- Temporal Memory: A temporal decay mechanism assigns each memory node an activeness score that decreases exponentially with time since its last observation.The decay rate depends on structural role and scene scale; carrier objects use zero decay, while carried objects use λbaseη(Sscene).
- Temporal Memory: External-agent nodes are chronologically organized into trajectories that provide physical context for multimodal retrieval and target relocation.Visual instance tracking can separate nodes belonging to multiple external agents and construct independent trajectories.
- Multi-Modal Context-aware Retrieval: HitMem first queries the hierarchical graph using semantic relevance and temporal activeness, then activates trajectory- and class-affinity-based relocation after detecting displacement.The retrieval score combines cosine similarity with activeness, and up to three high-ranked candidates are selected before active verification.
4 Experiments
Experiments evaluate HitMem on Dyna-THOR, comparing it with existing methods, ablating its components, and testing scalability across scene sizes and temporal decay rates. Results show stronger task performance and efficiency in dynamic object-relocation settings, with decay-rate effects depending on spatial scale.
- Benchmark and setup: Dyna-THOR evaluates long-term tasks in interactive scenes where external agents manipulate or relocate task-relevant objects without notifying the ego-agent.The benchmark contains 12 interactive scenes across four room categories, with five long-term tasks per scene.
- Comparison with baselines: HitMem significantly outperforms all baselines in success rate and execution efficiency on Dyna-THOR.Comparisons include DELTA, ConceptGraphs, DovSG, and DynaMem under matched processing and navigation constraints.
- Comparison with baselines: Static methods struggle with displaced objects, while DynaMem’s exhaustive global search can exceed the maximum step limit during relocation.DovSG supports dynamic updates but refreshes memory only from local observations, limiting its handling of larger displacements.
- Ablation study: Removing hierarchical memory degrades success rate and navigation efficiency, while removing temporal decay severely reduces SPL by treating stale and fresh observations equally.Without decay, agents repeatedly navigate to outdated coordinates and waste steps on invalid object entries.
- Ablation study: Removing both context-aware retrieval and temporal decay produces the largest degradation because blind global searches exhaust the step limit.The full strategy uses external-agent trajectories and semantic affinity to predict probable target locations.
- Scalability analysis: Decay-rate sensitivity is minor in small scenes but becomes critical in multi-room environments as reduced revisit frequency causes memories to decay.Smaller λ improves persistence, whereas excessively large λ over-prioritizes recent observations and can sharply reduce success rates.
5 Conclusion
The paper presents HitMem as a hierarchical temporal memory framework for embodied agents in dynamic environments. It combines semantic, spatial, and temporal memory with context-aware retrieval, and evaluations show adaptability and reduced exploration costs.
- Conclusion: HitMem unifies semantic, spatial, and temporal information in a lightweight topological graph and modulates memory-node activeness through temporal decay.It also uses two-stage retrieval that combines external-agent trajectories with semantic class affinity.
- Conclusion: Evaluations show that HitMem adapts to dynamic changes, scales across scenarios, and reduces exploration costs during complex long-term task execution.
A Dyna-THOR Benchmark
Dyna-THOR is built on AI2-THOR to evaluate high-level planning and long-term memory under externally induced object changes. Its navigation-centric action abstraction uses GoToObject and GoToPosition instead of primitive movement commands.
- Action abstraction: Dyna-THOR abstracts primitive low-level actions into GoToObject and GoToPosition navigation-centric action primitives.The abstraction encapsulates Move and Rotate commands through high-level interfaces.
- Agent configuration: The benchmark initializes an Ego-Agent and an External Agent with identical manipulation capabilities but different navigation privileges.The external agent can use ground-truth object IDs through GoToObject to localize and manipulate task-relevant objects.
A.2 Benchmark Composition
Benchmark episodes specify deterministic initial conditions, linguistic task instructions, external-agent interventions, and goal-based evaluation details. Expert demonstrations define minimal high-level execution sequences, while ego-agents must handle displaced targets without access to those trajectories.
- Episode composition: Each episode records initial 3D positions, orientations, camera horizon, and posture for both agents to establish a deterministic starting context.
- Episode composition: Long-term tasks use multiple natural-language sub-goals with varied linguistic formulations to test semantic understanding and robustness.Examples include explicit and contextualized requests to find and retrieve a book.
- Episode composition: External-agent action sequences deliberately manipulate task-relevant objects before ego-agent execution, while the ego-agent cannot access the expert trajectories.These interventions introduce dynamics intended to mimic human behavior and obey natural physical laws.
- Episode composition: Episodes define ground-truth terminal object states and record execution steps for calculating success and planning efficiency.
- Expert demonstrations: Minimum Execution Steps counts low-level primitives in an expert trajectory that uses ground-truth object IDs for direct target localization.The expert sequence is the minimal set of high-level actions required to achieve the task goals.
- Expert demonstrations: A long-term task is complete only when all constituent sub-goals succeed despite positional shifts caused by external-agent interference.Figure 6 illustrates representative expert-demonstration episodes and their sub-goal structure.
B.1 Impact of Descriptions and LLMs
Task success depends strongly on both instruction clarity and LLM choice. Explicit descriptions improve planning, while redundant model-generated actions can increase exploration costs and execution risk.
- Explicit and intuitive task descriptions yield higher success rates, whereas ambiguous instructions markedly reduce performance.
- LLMs differ in task understanding because their inference capabilities and action-generation strategies vary.
- Gemini-2.5-Pro’s redundant action sequences increase exploration costs, lengthen trajectories, and risk exceeding maximum step limits.
- Practical deployment requires balancing model capability, planning fidelity, and response latency for the task and resource setting.
- Task instructions should be precise and minimize semantic ambiguity to reduce execution risks.
B.2 Resilience to Imperfect Tracking
HitMem remains effective when external-agent tracking is incomplete or noisy, and it generalizes to dynamic object relocation on ALFRED. Semantic guidance complements trajectory information when tracking fails.
- 46.67% SR is achieved by the affinity-only setting under off-screen displacement, outperforming the Trajectory-only setting.
- Dropping 30% of trajectory points or adding Gaussian noise with σ=0.02 causes only a marginal performance drop.
- The results indicate that HitMem does not rigidly depend on perfect trajectory tracking.
- On ALFRED’s valid_unseen split of 85 tasks with relocated targets, HitMem outperforms dynamic memory baselines on both SR and SPL.
B.4 Analysis of Failure Cases
Qualitative failures arise from perception, memory encoding, and visual ambiguity, despite continuous observation enabling adaptation to some object displacements. These cases identify recognition and target-disambiguation boundaries.
- Success Case: Continuous observation enables the agent to detect object displacement and update its memory during successful execution.
- Perception Failure: Slender object geometry can cause perception failure even when the target is within view, preventing a valid memory entry.
- Memory Encoding Failure: A missed GarbageCan detection prevents memory encoding and leads to prolonged, futile exploration without a retrieval index.
- Visual Ambiguity: Visual ambiguity causes incorrect localization when a semantically related distractor is favored over the true target.
- Visual Ambiguity: An underspecified “Cup” instruction allowed a Mug distractor to outrank the true Cup through higher semantic affinity.
B.5 More Details in Experiments
The experiments standardize perception and evaluation conditions while examining parameter choices and scene-category variation. Performance is strongest in bedrooms and weaker in kitchens.
- 400×400-pixel ego-centric views balance visual inference accuracy with computational efficiency and real-time responsiveness.
- Core thresholds, weights, and decay parameters were selected using prior practices and empirical tuning.
- Pre-exploration uses Fast Frontier-based Exploration with a real-time occupancy grid, and its steps count toward total cost.
- Bedroom scenes achieve optimal comprehensive performance, while Kitchen scenes are relatively weaker.
- Performance differences across scenes are attributed to variation in object categories, scales, and shapes affecting visual-recognition robustness.
B.6 Limitations
The paper identifies limitations in its methodology and benchmark design, spanning foundation-model dependence, insufficient planning verification, and limited benchmark diversity.
- B.6 Limitations: Memory construction accuracy and efficiency are constrained by the capabilities of underlying foundation vision models.Perception errors from these models inevitably propagate to the memory module.
- B.6 Limitations: Task planning lacks a priori feasibility checking and closed-loop validation before execution.Verification currently occurs only during execution, leaving the system susceptible to planning hallucinations involving logically invalid or physically impossible actions.
- B.6 Limitations: Dyna-THOR requires broader task diversity to cover a wider spectrum of dynamic interaction scenarios.The authors specifically identify benchmark comprehensiveness as an area requiring further expansion.