3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning

• Existing object-centric graphs lose spatial detail through restrictive textual relationships, while dense representations are computationally expensive and neither models unexplored regions.These limits make nuanced spatial queries and active exploration difficult.Source: §1 · p. 2
• Memory Snapshots use informative multi-view images to represent explored regions, capturing visible objects, their surroundings, and spatial cues.The introduction illustrates that an image can show whether space in front of an armchair is sufficient for a coffee table.Source: §1 · p. 2
• Frontier Snapshots represent glimpses of unexplored regions, letting agents choose between acting on accumulated knowledge and exploring for new information.Source: §1 · p. 2
• Incremental memory aggregation and Prefiltering provide real-time memory construction and retrieval as the scene memory grows during lifelong exploration.Prefiltering selects relevant memories during decision-making to manage the expanding memory.Source: §1 · pp. 2–3
• Across three benchmarks, 3D-Mem significantly enhances agents’ reasoning and lifelong exploration capabilities in 3D environments.Source: §1 · p. 2

• Dense 3D representations ground 2D foundation-model features in scenes but have high resource costs and limited dynamic updates; scene graphs offer compact, dynamically updated alternatives but oversimplify relationships [32].3D-Mem instead uses snapshots to visually capture spatial and semantic relations.Source: §2 · p. 3
• VLM exploration methods use consecutive observations to predict actions [43] or visual prompting and semantic maps [28], but their memories are short-term or target-specific.The paper characterizes 3D-Mem as lifelong and target-agnostic.Source: §2 · p. 3
• Image-based topological maps such as TSGM and RoboHop build navigation graphs but do not fully capture objects and their relations; 3D-Mem clusters co-visible objects for tasks beyond route planning.The passage also states that its retrieval mechanism scales to lifelong exploration.Source: §2 · p. 3

3.1. 3D-Mem Construction

• Memory Snapshots represent co-visible objects with multi-view images that retain object details, spatial relationships, and room-level context.Each snapshot captures a cluster of objects and their surroundings.Source: §3.1 · p. 4, §3.1.2 · p. 4

3.1.1. Memory Snapshot Formulation

• The snapshot set is a minimal subset of observations covering salient objects, with each object assigned to one snapshot.Objects assigned to a snapshot may remain visible in other snapshots.Source: §3.1.1 · p. 4

3.1.2. Co-Visibility Clustering

• Co-Visibility Clustering assigns object clusters to frames containing them, splitting clusters when no frame covers them and merging snapshots that share a frame.Among eligible frames, it selects the one containing the most objects, breaking ties by summed detection confidence.Source: §3.1.2 · p. 4

3.2. 3D-Mem with Frontier-based Exploration

• The exploration pipeline adapts memory construction for frontier-based exploration and adds retrieval as the memory grows and decision-making becomes computationally heavy.It updates frontier and memory snapshots from stepwise observations.Source: §3.2 · p. 4

3.2.1. Introduction of Frontier Snapshot

• A Frontier Snapshot represents an unexplored region with its region, a navigable location, and an image captured toward that region.Frontier and memory snapshots are raw images usable as VLM input.Source: §3.2.1 · pp. 4–5

3.2.2. Incremental Construction of 3D-Mem

• Incremental construction updates objects from local egocentric views, reclusters newly detected and previously assigned objects, and revises frontiers as regions change.Objects are restricted by a maximum distance; frontiers may be modified, removed, or newly added.Source: §3.2.2 · p. 5

3.2.3. Memory Retrieval with Prefiltering

• Prefiltering ranks object categories for the instruction and retains only snapshots containing the top K categories, reducing resource use.The VLM ranks categories from the instruction and the current object set.Source: §3.2.3 · p. 5

3.3. Exploration and Reasoning with 3D-Mem

• For question answering, the VLM uses retrieved memory and frontier snapshots to decide whether to explore or answer; for object navigation, it selects an object snapshot.Frontiers are reached by navigating toward their locations, with stopping limits of 1.0m traveled or 0.5m from the target.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

4. Experiments

• The evaluation covers active EQA, episodic-memory EQA without exploration, and lifelong GOAT-Bench navigation, with benchmark-specific adaptations.The experiments also include ablations of components and hyperparameter choices in the appendix.Source: §4 · p. 6

4.1. Active Embodied Question Answering

• On A-EQA, 3D-Mem significantly outperforms previous methods in both accuracy and efficiency.The main-paper evaluation uses a 184-question subset; questions require exploration for object recognition, functional reasoning, or spatial understanding.Source: Table 1 · p. 6, §4.1 · pp. 6–7

4.2. Episodic-Memory Embodied Q&A

• On EM-EQA, 3D-Mem and Multi-Frame outperform caption-based methods using approximately three frames, while 3D-Mem surpasses Multi-Frame in frame efficiency.The benchmark supplies observation trajectories and camera poses, so this evaluation tests scene representation without requiring exploration.Source: Table 2 · p. 7, Figure 4 · p. 7, §4.2 · pp. 7–8

4.3. GOAT-Bench

• On the GOAT-Bench Val Unseen subset, 3D-Mem achieves the highest accuracy and efficiency scores among compared methods.Without memory, performance declines for both GPT-4o and LLaVA-7B, particularly in SPL; at K = 10, prefiltering retains an average of 4.66 snapshots from 16.58 snapshots per episode.Source: Table 3 · p. 8, §4.3 · p. 8

5. Conclusion

• 3D-Mem stores robust scene information in informative multi-view snapshot images.The conclusion describes these images as the framework's scene-memory representation.Source: §5 · p. 8
• Frontier-based exploration lets agents use explored-region memory to solve tasks or explore further to expand scene knowledge.Source: §5 · p. 8
• Incremental construction and efficient memory retrieval make 3D-Mem an effective memory system for lifelong agents.Source: §5 · p. 8

6. Full-Set Evaluation

• Resource limitations restrict main-paper evaluations of A-EQA and GOAT-Bench to subsets.Source: §6 · p. 9
• On whole-set A-EQA, 3D-Mem scores 53.3 LLM-Match and 38.0 LLM-Match SPL.Source: §6 · p. 9
• On subset A-EQA, 3D-Mem scores 52.6 LLM-Match and 42.0 LLM-Match SPL.Source: §6 · p. 9

7.2. Decision Frequency

• In the alternative setting that waits until reaching a navigation target before deciding again, A-EQA LLM-match and SPL are 50.5 and 36.2, respectively; the default permits en-route correction but can oscillate between frontiers.The demo uses the alternative setting to avoid oscillation during longer exploration episodes.Source: §7.2 · p. 9

7.1. Detailed Experiment Results

• On A-EQA, 3D-Mem outperforms ConceptGraph with Frontier Snapshots on spatial understanding and object localization, and is comparable in other question categories.The category table reports LLM-Match Score and SPL; the authors attribute the spatial-category gains to foreground relationships and room-level cues in Memory Snapshots.Source: §7.1 · p. 9, Table 4 · p. 10
• 3D-Mem consistently outperforms both methods across all modalities in Success Rate and SPL.The authors link this to recalling previously observed regions and objects.Source: §7.1 · p. 9, Table 5 · p. 10
• On GOAT-Bench, 3D-Mem outperforms ConceptGraphs in Object Category and Language modalities, but has slightly lower Image-modality SPL despite similar Success Rate.The authors suggest that viewpoint differences between memory snapshots and image prompts may distract current VLMs in complex scenes.Source: §7.1 · p. 9, Table 5 · p. 10

7.3. Limitations

• 3D-Mem is designed for static environments, depends on detection and VLM reasoning, requires precise agent location, and has noticeable embodied-QA latency.VLM inference is identified as the primary latency bottleneck.Source: §7.3 · p. 10

Dataset Issues

• Some A-EQA questions are vague and permit multiple reasonable answers, while an overconfident VLM prediction can end an episode prematurely.The authors classify these as dataset issues.Source: §8 · p. 11

Limitations of the VLM

• VLM failures include incorrect answers despite selecting the right snapshot, especially for small targets at 360×360 resolution, and selection of an entirely wrong snapshot.The authors attribute these cases to limited perception capabilities.Source: §8 · p. 11

Limitations of the Object Detection Model

• Incorrect detector labels can confuse the VLM when they align with an expected answer, while undetected targets may be absent from memory and prevent selection of a suitable snapshot.Prefiltering and VLM judgment often filter or ignore incorrect labels, but not in every case.Source: §8 · p. 11
• A missed target can still be answered when it remains visible in another snapshot that passes prefiltering.The authors cite these cases as evidence of greater robustness than traditional 3D scene graphs.Source: §8 · p. 11

Occupancy and Regions

• A 3D occupancy grid records room space with 0.1-meter voxels; navigable space is the unoccupied layer 0.4 meters above ground, split into explored and unexplored areas.The explored area is within 1.7 meters of the agent’s trajectory.Source: §9 · p. 11

Frontier Construction

• Frontiers are unexplored pixel clusters formed with DBSCAN, each paired with a boundary navigation location and an image observation.Clusters need more than 20 pixels; regions spanning over 150° in the agent’s view are split using K-Means.Source: §9 · p. 11

Frontier Updates and Navigation

• A frontier is updated when its region’s IoU with the previous region falls below 0.95; VLM prompts contain image observations, and a selected frontier’s location becomes the navigation target.Source: §9 · p. 11

Scope

• The current 3D-space representation does not support multi-floor scenes, limiting performance on A-EQA questions that require exploration across floors.Source: §9 · p. 11

10. More Details in Experiments

• At each step, the experiments use three egocentric views, detect objects from 1280×1280 images before resizing frames to 360×360, and allow up to 50 steps per task.Frontier snapshots are initially captured at 360×360; detection uses YOLOv8x-World with 200 ScanNet classes.Source: §10 · p. 11

10.1. Implementation Details for A-EQA

• A-EQA uses three egocentric observations per step, a 3.5 maximum object distance for scene-graph inclusion, and ten prefiltered classes.Source: §10.1 · pp. 11–12
• A-EQA failure examples cover vague questions, VLM perception or snapshot-selection errors, and detector-label errors that can mislead the VLM.Source: Figure 5 · p. 12, Figure 6 · p. 12, Figure 7 · p. 13

10.2. Implementation Details for EM-EQA

• For EM-EQA, the authors construct scene memory from RGB-D observations and camera poses, then prefilter snapshots at K values of 1, 2, 3, 5, or 10 for GPT-4o.Source: §10.2 · pp. 13–14
• The EM-EQA figures show missed detections and an example where targets remain visible in other snapshots, alongside an overview of frontier-based exploration.In that example, the VLM still answers successfully using other memory snapshots.Source: Figure 9 · p. 14, Figure 10 · p. 14, Figure 11 · p. 15

10.3. Implementation Details for GOAT-Bench

• GOAT-Bench navigation prompts cover category, language-description, and image targets; the agent navigates near an object after the VLM identifies it in a snapshot.The evaluation tests GPT-4o and LLaVA-7B, with the latter fine-tuned on a generated dataset.Source: §10.3 · p. 15

11. Details of the Active Exploration

• The VLM chooses a memory or frontier target; navigation positions are set around object clusters treated as 2D points, then movement triggers refreshed snapshots and frontiers.For a single-object snapshot, the observation distance is 0.75 meters; GOAT-Bench also allows the VLM to choose an object and navigates toward it.Source: §11 · p. 15

12.1. Training Dataset Collection

• Training data uses 3,669 navigation targets from 136 HM3D training scenes, with about four exploration paths of roughly 12 steps per target and GPT-4o-generated object rankings.Source: §12.1 · p. 16

12.2. Training Process

• Training jointly optimizes exploration and prefiltering: exploration predicts a memory or frontier choice, while prefiltering selects the top 10 helpful observed objects.The model is fine-tuned from LLaVA-1.5-7B for five epochs; snapshot and frontier image features are compressed to 3 × 3 tokens in training prompts.Source: §12.2 · p. 16

13.1. Ablation on Hyperparameter Choices

• Increasing observations need not help; larger max dist improves GOAT-Bench but hurts A-EQA, while more prefiltered classes improve both; selected settings are N=3 and max dist=3.5 meters.At K=10, prefiltering leaves an average of 3.26 snapshots (29.8% of total snapshots; 8.2% of frame candidates) for A-EQA and 4.66 (28.1%; 5.1%) for GOAT-Bench.Source: §13.1 · pp. 16–17, Figure 13 · p. 16, Figure 14 · p. 16

13.2. Ablation on Pipeline Components

• Choosing frontiers via the VLM outperforms always taking the nearest frontier on both benchmarks; a Prefiltering ablation was infeasible because VLM queries would exceed context length.Removing Frontier Snapshots and memory maintenance together also confirms the reported pattern, with the GOAT-Bench drop smaller as agents rely more on memory in mostly explored scenes.Source: Table 7 · p. 17, §13.2 · p. 17

14. Other Related Works

• Unlike 2D topological methods and, 3D-Mem aims for compact object coverage and relationships, while frontier snapshots and Prefiltering support active, lifelong exploration.Source: §14 · p. 17

15. Complete Prompts for VLMs

• The prompts distinguish prefiltering, A-EQA, and GOAT-Bench inputs; GOAT-Bench additionally supplies object crops taken from snapshot detection boxes.Source: §15 · p. 18, Figure 16 · p. 19, Figure 17 · p. 20