MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation

• Prior map-based navigation work has focused mainly on map construction and use, while maps offer a spatial abstraction and human interpretability for longer-horizon tasks.Source: §1 · p. 1
• The experiments assume perfect localization, separating the study of useful map information from noisy sensing and actuation.Source: §1 · pp. 1–2
• MultiON benchmarks navigation to episode-specific ordered object sequences in realistic 3D interiors, extending earlier visual navigation tasks.Source: §1 · p. 2
• MultiON varies difficulty through multiple goal objects while excluding higher-level understanding, planning, and reasoning factors that could confound map-memory comparisons.Source: §1 · p. 2
• Experiments show that performance drops dramatically with task complexity, simple semantic maps outperform more complex learned map agents, and even oracle maps achieve relatively low performance.Source: §1 · p. 2

• Unlike PointGoal, which supplies a displacement vector and largely removes the need for long-horizon planning, ObjectGoal directs agents to objects [2] [39] [14] [23].Source: §2 · p. 2
• The paper generalizes ObjectGoal navigation to systematically investigate spatial and semantic maps for grounding egocentrically acquired visual information.Source: §2 · p. 2
• Earlier long-horizon tasks include navigating to random city locations [33], answering questions involving two objects [50], and object search without controlled goal order or count [21].Source: §2 · p. 2
• MultiON combines episode-specific object goals with an ordered sequence in photorealistic 3D environments, requiring agents to retrieve previously encountered objects when they become goals.Source: §2 · p. 2
• The paper identifies a gap in systematic comparisons of map information, egocentrically acquired maps, oracle maps, and no-map representations for navigation.Source: §2 · p. 3

• An m-ON episode contains m ordered goal objects selected from k available objects, with m controlling navigation complexity and k ≥ m.Source: §3 · p. 3
• Agents receive RGB and depth images plus a one-hot current-goal vector, and must use FOUND near each goal before the sequence advances.Source: §3 · p. 3
• A different goal sequence is specified per episode, and goal objects are programmatically placed at controlled locations to control distances and episode complexity.Source: §3 · p. 3
• The base architecture encodes RGBD frames, an egocentric map, the current goal, and the previous action into a GRU state used by an actor-critic pair.Source: Figure 1 · p. 4
• The evaluation metrics are independent of reward formulations, allowing investigation of alternative reward structures and methods such as imitation learning.Source: §3 · p. 4

4.1 Base Agent Architecture

• Agents receive RGB, depth, and goal inputs; map-based models use an egocentric view of the global map, while a GRU embeds inputs and supplies implicit memory.The architecture combines image, map, goal, and previous-action embeddings to predict the action policy and value function.Source: §4.1 · p. 4

4 Agent Models

• The benchmark compares agents with implicit memory, oracle maps, and learned maps, plus random-action baselines.The models share a general network architecture, with map construction and use as the critical variation.Source: §4 · p. 4, §4.2 · p. 5

Implicit memory baselines

• NoMap(RNN), FRMQN, and SMT provide implicit-memory comparisons, using recurrent state or stored observation embeddings rather than explicit maps.FRMQN retrieves key-value memories with soft attention; SMT uses an encoder-decoder to retrieve stored observations.Source: Implicit memory baselines · p. 5

Oracle baselines

• OracleMap receives the full ground-truth global map, whereas OracleEgoMap reveals that map progressively from the agent’s egocentric perspective.OracleEgoMap accounts for visibility using ray tracing and the current depth frame.Source: Oracle baselines · p. 5

Learned map agents

• ObjRecogMap maps predicted visible-object categories, while ProjNeuralMap depth-projects image features into an egocentric map and registers them globally; EgoMap reads a semantic map with attention.ObjRecogMap assigns a no-object label when no goal is within its field of view and 2.5 m; ProjNeuralMap uses element-wise max-pooling when registering features into filled cells.Source: Learned map agents · pp. 5–6, Figure 2 · p. 6

Random baseline agents

• Rand chooses randomly among movement and FOUND actions; Rand+OracleFound instead invokes FOUND when within a threshold distance of the current goal.Episodes terminate on success or after 2,500 steps.Source: Random baseline agents · p. 6

4.3 Training

• Training uses PPO for 40 million frames across 16 parallel threads, with a goal-discovery reward, geodesic-distance progress, and negative slack encouraging shorter paths.The specified reward values are rgoal = 3.0 and α = −0.01; ObjRecogMap’s recognition network receives cross-entropy supervision on the ground-truth goal category.Source: §4.3 · p. 6

5.1 Datasets

• Episodes sample starts and goals on one floor with navigable paths and 2–20m geodesic intervals; Figure 3 contrasts agent and shortest paths.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 Metrics

• SUCCESS requires ordered goal completion, PROGRESS counts the fraction found, and SPL/PPL measure path efficiency, with PPL weighting path length by progress.Source: §5.2 · p. 7

5.3 Example episodes

• NoMap(RNN) tends to wander and earn low PPL even when it reaches goals; calling FOUND away from the current goal terminates the episode in failure.Source: §5.3 · p. 7

5.4 Quantitative evaluation

• OracleMap SUCCESS falls from 0.94 (1-ON) to 0.79 (2-ON) and 0.62 (3-ON), below the stated exponential-decay expectation.Source: §5.4 · p. 8
• Object-category information contributes more than occupancy; Occ+Obj may trail Obj because it had not fully converged within 40 million training steps.Source: §5.4 · p. 8
• Across test tasks, performance decreases as objects are added, while maps help considerably; OracleMap performs best, with ObjRecogMap close to OracleEgoMap.Source: §5.4 · p. 8, Table 2 · p. 9
• ObjRecogMap outperforms ProjNeuralMap and EgoMap on 1-ON and 2-ON, but trails them on 3-ON; ProjNeuralMap and EgoMap perform similarly overall.Source: §5.4 · p. 9

• MultiON supports systematic study of semantic-map navigation; experiments find that simple semantic integration can outperform more complex learned maps, while performance still falls sharply with modest increases in task difficulty.Source: §6 · p. 9

• The authors call for studying robotic systems in simulation first and then in controlled real environments to minimize deployment risks.They note that imperfect agents may cause injuries or damage.Source: Broader Impact · p. 10
• Potential applications include assisting older adults, delivering items, and moving warehouse goods.Source: Broader Impact · p. 10
• Success in these directions may displace workers and create economic difficulties in occupations amenable to automation.Source: Broader Impact · p. 10

• Table 3 summarizes definitions for important symbols used in the main paper and supplement.Source: §A.1 · p. 14, Table 3 · p. 14
• The subscript t denotes notation corresponding to time step t.Source: Table 3 · p. 14

• Each map-based model uses a 300 × 300 global grid with 0.8m × 0.8m cells, revealed or built over time depending on the model.The map-cell contents and update process vary by model.Source: §A.2 · p. 14
• OracleEgoMap reveals an oracle-derived map over time, storing occupancy and object-category channels while limiting discovery to visible locations.Locations must be within a 79° field of view, within 5m, and unobstructed; discovered cells remain discovered for the episode.Source: §A.2 · p. 14
• ObjRecogMap predicts object categories from egocentric RGB views and stores only category information in its map cells.A classifier is trained with ground-truth object categories; when multiple objects are visible, training targets the closest one.Source: §A.2 · pp. 14–15
• ProjNeuralMap projects image features onto a 7 × 13 egocentric map using downsampled depth, then registers it into the global map by element-wise max-pooling.The local map covers 5.6m in front and to both sides; points farther away are not projected.Source: §A.2 · p. 15

• Episode distributions tend toward longer oracle shortest-path geodesic distances from 1-ON to 2-ON and 3-ON.Source: §A.3 · p. 15, Figure 5 · p. 16
• Train, validation, and test episode distributions generally correspond within each m-ON, despite small variations.Source: §A.3 · p. 15
• Each scene has 50,000 training episodes and 12,500 validation and test episodes; Figure 5 bins episodes by oracle shortest-path geodesic distance.The figure's horizontal axis is geodesic distance and its vertical axis is the fraction of episodes in each histogram bin.Source: Figure 5 · p. 16

Backtracking analysis

• Previously seen goals significantly improve later subgoal success for OracleEgoMap, ProjNeuralMap, and ObjRecogMap, with the largest gains for ObjRecogMap.NoMap(RNN) and OracleMap show no improvement in this comparison.Source: Table 4 · p. 17, §A.4 · p. 17

Testing agent generalization

• Agents trained on 1-ON have the lowest generalization capability and fail on more complex tasks; 2-ON and 3-ON agents transfer better, especially to simpler tasks.The authors suggest 1-ON agents lack training to continue past the first found goal.Source: Table 5 · p. 17, §A.4 · pp. 17–18

Analysis of agent performance vs episode length

• Average PPL generally declines as episode distance increases; OracleMap retains relatively high PPL, trains fastest, and reaches the best overall performance.NoMap(RNN) has relatively low PPL and often fails to reach any goals on harder episodes.Source: §A.4 · p. 18, Figure 6 · p. 19, Figure 7 · p. 20

Oracle map ablations

• DynamicOracleMap outperforms the original setup, while OnlyOracleMap underperforms RGBD agents and OracleMapHiddenObjects matches the object-present model.The map-only deficit is attributed primarily to ineffective scene navigation rather than difficulty calling FOUND at the right time.Source: §A.4 · p. 22, Table 6 · p. 22

Varying map resolution and target object shapes

• Stacking high- and low-resolution maps did not improve OracleMap; varying target shapes improved 1-ON but worsened 2-ON and 3-ON.The shape-varying setting may make goals easier to identify by camera height but harder to distinguish from one another.Source: §A.4 · pp. 22–23

Dynamically updating maps

• Dynamically retaining only the current goal embedding does not outperform base OracleEgoMap or ObjRecogMap, which retain embeddings of discovered targets.ProjNeuralMap cannot use this variant because it does not store object embeddings.Source: §A.4 · pp. 22–23, Table 7 · p. 23

Maximum episode length determination

• The 2,500-step limit has negligible impact under proportional thresholds; 3-ON episodes average 276.2 steps and have a median of 151.Most episodes terminate through a wrong FOUND action rather than reaching the maximum time limit.Source: §A.4 · p. 23

Effect of having a hard FOUND action

• Allowing even one wrong FOUND action significantly increases performance metrics, suggesting that premature FOUND calls end many episodes.The comparison uses OracleMap on 3-ON agents trained under the usual single-error termination rule.Source: §A.4 · p. 23, Table 9 · p. 24