Source-linked AI summary
ObjectReact: Learning Object-Relative Control for Visual Navigation
Sourav Garg, Dustin Craggs, Vineeth Bhat, Lachlan Mares, Stefan Podgorski, Madhava Krishna, Feras Dayoub, Ian Reid
TL;DR
Image-relative topological navigation relies on subgoal images tied to pose and embodiment, while object-relative control uses map objects as subgoals. The paper combines a relative 3D scene graph with a WayObject Costmap-conditioned controller, reporting stronger performance on challenging navigation tasks and near-invariance to sensor-height changes across mapping and execution.
Problem
Image-relative topological navigation depends on subgoal images tied to robot pose and embodiment, limiting the invariance of its world representation.
Source: §1 · pp. 1–2Method
The paper builds a relative 3D object scene graph and trains ObjectReact to predict trajectories from a WayObject Costmap without explicit RGB input.
Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5Results
Object-relative control performs better than image-relative control on challenging tasks, while its performance is nearly invariant to sensor-height changes that lower image-relative SPL by an absolute 48%.
Source: §5.1 · p. 6, §5.2 · p. 7Takeaways & Limitations
Abstract
from arXiv · showhide
Visual navigation using only a single camera and a topological map has recently become an appealing alternative to methods that require additional sensors and 3D maps. This is typically achieved through an "image-relative" approach to estimating control from a given pair of current observation and subgoal image. However, image-level representations of the world have limitations because images are strictly tied to the agent's pose and embodiment. In contrast, objects, being a property of the map, offer an embodiment- and trajectory-invariant world representation. In this work, we present a new paradigm of learning "object-relative" control that exhibits several desirable characteristics: a) new routes can be traversed without strictly requiring to imitate prior experience, b) the control prediction problem can be decoupled from solving the image matching problem, and c) high invariance can be achieved in cross-embodiment deployment for variations across both training-testing and mapping-execution settings. We propose a topometric map representation in the form of a "relative" 3D scene graph, which is used to obtain more informative object-level global path planning costs. We train a local controller, dubbed "ObjectReact", conditioned directly on a high-level "WayObject Costmap" representation that eliminates the need for an explicit RGB input. We demonstrate the advantages of learning object-relative control over its image-relative counterpart across sensor height variations and multiple navigation tasks that challenge the underlying spatial understanding capability, e.g., navigating a map trajectory in the reverse direction. We further show that our sim-only policy is able to generalize well to real-world indoor environments. Code and supplementary material are accessible via project page: https://object-react.github.io/
1 Introduction
Visual topological navigation offers a single-camera alternative to dense 3D mapping, but image-relative control ties subgoals to the agent’s pose and embodiment. The paper introduces object-relative control and a topometric object-level map, with tasks designed to test navigation beyond repeating a prior trajectory.
Motivation
- Single-camera topological navigation learns control from current and subgoal images, but image-level representations remain tied to the robot’s pose and embodiment.Earlier topological approaches were often limited to teach-and-repeat.Source: §1 · pp. 1–2
Evaluation tasks
- The task set includes teach-and-repeat imitation, a previously seen but unvisited goal, a shortcut around a longer prior route, and reverse-direction travel.These tasks vary how the target route relates to the mapped trajectory.Source: Figure 1 · p. 2
Object-relative control
- Object-relative control conditions on visible object subgoals, avoiding a separate subgoal image and its image-matching requirement.The paper proposes this to support new routes and cross-embodiment generalization across mapping and execution.Source: §1 · p. 2
Contributions
- The proposed navigation stack combines a relative 3D scene graph, a WayObject Costmap-conditioned ObjectReact controller, and tasks testing understanding from limited prior experience.The map connects objects using topological connectivity across images and relative 3D information within images.Source: §1 · p. 2
2 Related Work
Prior navigation work uses image nodes, object-level subgoals, or semantic relations, with differing map representations and controller designs. This paper distinguishes its approach through object connectivity and a controller conditioned on object-path costs rather than an explicit RGB input.
Image-Relative Subgoal Control
- Image-relative methods use a topological map of images and combine a selected subgoal image with the current view to produce control.Prior work also spans lookup, learned, and visual-servoing controllers.Source: §2 · pp. 2–3
Object-Relative Subgoal Control
- RoboHop, PixNav, and TANGO use objects, pixels, or 3D points as local subgoals, but their controllers have stated limitations in collisions, scene-layout overfitting, or dependence on visible traversability.The paper instead targets a learned continuous controller conditioned on currently viewed objects.Source: §2 · p. 3
Objects and Semantics for Navigation in Unseen Environments
- Many unseen-environment methods learn semantic or spatial relations, but their graph nodes are often images tied to agent poses rather than object-connected scene graphs.The paper also distinguishes seen-environment path search from predicting a proxy for global planning in unseen environments.Source: §2 · p. 3
Prior Experience-based Navigation
- Prior-experience navigation includes teach-and-repeat and policies learned from large real-world datasets, yet remains image-relative; this paper separates embodiment-invariant perception from trajectory-invariant control learning.Its perception uses open-set, zero-shot models, while control uses a WayObject Costmap without explicit RGB input.Source: §2 · p. 3
3 Approach
The approach represents object connectivity in a relative 3D scene graph, uses object matches and path costs to form local control inputs, and trains ObjectReact on WayObject Costmaps. Its phases combine object-level mapping, goal-path planning, and trajectory prediction.
3 Approach
- The approach replaces 2D object-center triangulation with relative 3D connectivity to reduce geometric ambiguity in shortest-path planning, then learns a controller to react to objects.It retains object-level localization and planning while changing the map connectivity and controller.Source: §3 · p. 4
3.1 Mapping Phase: Relative 3D Scene Graph
- Image segments become graph nodes with binary masks and farthest-point 3D coordinates projected from monocular depth using a fixed focal length.The map represents relative object layout without estimating camera pose or global scene geometry.Source: §3.1 · p. 4
- Within-image object pairs receive Euclidean 3D edge weights; across consecutive images, feature-matched segment correspondences form zero-cost inter-image edges.Inter-image matching uses SuperPoint keypoints and descriptors with LightGlue; the zero weight is equivalent to merging nodes for planning.Source: §3.1 · pp. 4–5
3.2 Execution Phase: Object Localizer, Global Planner, and Local Controller
- At execution, query objects are matched to map nodes, and Dijkstra’s algorithm supplies path lengths to the goal, selecting the least-cost match when several exist.Those path lengths and segmentation masks provide the inputs for local trajectory prediction.Source: §3.2 · p. 5
3.3 Training Phase: The ObjectReact Controller
- The WayObject Costmap combines object masks with per-image-normalized path costs encoded as sine-cosine channels, retaining spatial and semantic information for a variable number of objects.The representation is designed to avoid bias from absolute episode costs and to support control without an explicit RGB input.Source: §3.3 · p. 5
- ObjectReact adapts the GNM training pipeline with HM3D data and a custom multi-channel goal encoder; training costmaps use simulator instances and depth, with outlier-cost augmentation.Training and validation episodes are sampled from 145 IIN-train scenes; 30% of segments are randomly assigned the outlier cost during augmentation.Source: §3.3 · pp. 5–6
4 Experimental Setup
Experiments use HM3D trajectories and evaluate navigation across sensor heights, with ground-truth map-image localization. Four tasks vary whether the agent repeats, extends, shortcuts, or reverses its prior route.
Evaluation Dataset
- Evaluation uses 36 IIN-validation scenes, one episode per scene, with a prior map trajectory and starts at least 5 m geodesically from the goal.The agent receives the map image index closest to its current 2D position as ground-truth localization.Source: §4 · p. 6
Evaluation Metrics
- Success requires reaching within 1 m of the goal in at most 300 steps; performance is reported as SPL and Soft-SPL over 72 runs.Each episode is tested at execution sensor heights of 1.3 m and 0.4 m, while map images use a fixed height of 1.3 m.Source: §4 · p. 6
Tasks
- The four tasks are Imitate, Alt Goal, Shortcut, and Reverse, testing prior-route repetition, an unvisited seen goal, a shortcut, and travel opposite the prior route.Some episodes are excluded for particular tasks when manual inspection finds an invalid object goal or path.Source: §4 · p. 6
5 Results and Discussion
Object-relative control matches image-relative control on Imitate and substantially outperforms it on harder navigation tasks, while remaining nearly invariant to sensor-height changes. Ablations favor ObjectReact over alternative controllers, 3D over 2D map connectivity, and conditioning without explicit RGB input.
5.1 Image-Relative vs. Object-Relative
- Object-relative control matches image-relative control on Imitate but substantially outperforms it on harder tasks, reaching unvisited goals and handling shortcut and reverse routes.Table 1 compares SPL and SSPL across four navigation tasks, averaged over two sensor heights.Source: §5.1 · pp. 6–7, Table 1 · p. 7
- On the challenging tasks, absolute performance remains below Imitate, largely because robust perception is lacking rather than because of the learned controller.Using simulator ground-truth object instances for segmentation and matching yields high performance across tasks.Source: §5.1 · p. 7
5.2 Embodiment Variations based on Robot Height
- 48% absolute SPL drop occurs for image-relative control when execution height changes to 0.4m from a 1.3m map; object-relative control is almost invariant.The authors attribute the latter's slight benefit at lower height to greater visibility of immediately traversable regions.Source: §5.2 · p. 7, Table 2 · p. 7
5.3 Ablation Studies
- ObjectReact outperforms RoboHop and PixNav by large margins when all three receive the same WayObject Costmap input.This comparison uses ground-truth object instances and relative 3D edge-based map connectivity.Source: §5.3 · p. 8, Table 3 · p. 8
- Relative 3D object connectivity is more informative than 2D Delaunay-based connectivity for the map edges.ObjectReact defines intra-image edges using 3D distances between object pairs.Source: §5.3 · p. 8, Table 3 · p. 8
- Removing explicit RGB input improves generalization; the controller instead conditions on WayObjects in the current observation.The authors speculate that RGB input can encourage overfitting to training-set object instances.Source: §5.3 · p. 8, Table 3 · p. 8
6 Conclusion and Future Work
The conclusion presents an object-relative navigation pipeline built around a relative 3D Scene Graph and a controller conditioned on a WayObject Costmap. It identifies limitations and outlines possible future extensions of the representation.
6 Conclusion and Future Work
- The proposed pipeline combines a relative 3D Scene Graph map with a learned local controller conditioned on currently viewed objects through a WayObject Costmap.The paper reports that subgoal conditioning affects performance on challenging tasks and cross-embodiment generalization, particularly across mapping and execution phases.Source: §6 · p. 8
- The authors identify a lack of robust perception as a limitation and propose generating the costmap from language, exploration objectives, cross-sensor modalities, or a vision LLM.They also suggest that future work could investigate links to landmark-based navigation strategies observed in human trials.Source: §6 · p. 8
Appendix
- The appendix provides implementation and experimental details, limitations, and additional results and analyses, especially concerning real-world and simulator demonstrations.Source: Appendix · p. 9
A Implementation Details
The WayObject Costmap represents currently viewed object masks together with their path lengths to a long-horizon goal. Each object's path length is encoded and combined with its mask to form the representation.
A.1 WayObject Costmap Representation
- Given binary masks for currently viewed objects and their scalar path lengths to the long-horizon goal, the method constructs a path-length encoding and a WayObject Costmap.The stated outputs are E with shape Nm×D and W with shape H×W×D.Source: §A.1 · p. 9
- Each object's path length is normalized per image and rescaled before encoding; L marks the shortest path length and zero marks outliers.The outliers correspond to undetected or unmatched image segments.Source: §A.1 · p. 9
- The path-length encoding uses D = 8, Z = 10000, and L = 100.Source: §A.1 · p. 9
A.2 Model Architecture and Loss functions
The controller adapts GNM to predict local waypoints from a WayObject Costmap rather than an explicit goal image. Mapping, localization, planning, and execution provide the inputs and control pipeline for this approach.
A.2 Model Architecture and Loss functions
- ObjectReact uses a custom ResNet costmap encoder to predict 10 future 2D waypoints, omitting GNM’s distance-to-image-goal output.The costmap has the same resolution as GNM images, with pixels encoding locally normalized path lengths.Source: §A.2 · p. 9
A.3 Mapping
- For mapping, SAM2 generates masks; CLIP removes floor and ceiling segments, SuperPoint-LightGlue matches frames, and Depth Anything supplies local 3D projections.SAM2 uses 16 points per side and one additional cropping layer with a downsample factor of 2.Source: §A.3 · pp. 9–10
A.4 Localization and Planning
- Localization matches query segments to map segments and assigns path lengths; unlocalized segments may inherit a median path length through tracking across eight image-history frames.Undetected or unlocalized and untracked segments are encoded as outliers with value 0.Source: §A.4 · p. 10
A.5 Control Execution
- The controller converts the last predicted waypoint into linear and angular velocities, averages predictions across five observations, and clips velocity ranges for execution.For real-world runs, velocities are executed on the Unitree Go1 through its HighCmd interface.Source: §A.5 · p. 10
A.6 Parameters: Image resolution, field-of-view and agent radius
- Simulator mapping and execution use 120° field-of-view images at 320×240 resolution with a 0.75 m agent radius, while training uses 85×65 images and a 0.3 m radius.The training image resolution matches GNM’s.Source: §A.6 · p. 10
B.1 Baselines
- PixNav receives a goal-pixel mask and RGB-image history, then selects from discrete actions; its evaluation begins with a visible intermediate goal updated by a done command or full memory buffer.The evaluation uses the official model checkpoint.Source: §B.1 · pp. 10–11
- RoboHop’s zero-shot controller weights segment-center offsets from the image center by normalized path length using a softmax with temperature 5 and fixed gain 0.4.The resulting displacement is scaled by image width.Source: §B.1 · p. 11
B.2 Excluded episodes in evaluation
The evaluation excludes manually identified invalid or unsuitable episodes. Quantitative analyses report similar performance for two GNM variants and roughly linear scaling of offline mapping and planning with map size.
B.2 Excluded episodes in evaluation
- Manual inspection excluded 6 Imitate, 16 Alt-Goal, 13 Shortcut, and 9 Reverse episodes, each task starting with 36 episodes.Reasons included invalid or unreachable goals, goals failing task criteria, and simulator rendering issues.Source: §B.2 · p. 11
C.1 Quantitative Analyses
- The simulator-trained and real-world-trained GNM models perform similarly; the challenging-task gap is attributed to world representation and subgoal conditioning, not training environment or embodiment diversity.Table 5 compares ObjectReact with image-relative GNM models trained on simulator and real-world data.Source: §C.1 · p. 11, Table 5 · p. 12
- Offline map and plan time scales roughly linearly with image or object-node count; a three-floor, 30×36 m office building was mapped in 5 minutes.The reported planning assumes the map is already given; online mapping would amortize this cost throughout the run.Source: §C.1 · p. 11, Table 4 · p. 12
C.2 Qualitative Results and Analyses
Qualitative trials include Unitree Go1 navigation and simulator runs, with examples of successful goal-reaching behavior and failures related to obstacle avoidance and perception. Demonstration visuals pair real-world and simulator examples.
C.2.1 Simulator Deployment
- Simulator demonstrations include successful ObjectReact runs using FastSAM and SuperPoint-LightGlue rather than ground-truth object instances.A ground-truth-instance run is included to show the perception-quality gap during deployment.Source: §C.2.1 · p. 12
- Figure 3 pairs a real-world demonstration video on the left with a simulator video on the right; the marked localized image is the closest map match.The figure illustrates the accompanying demonstration videos.Source: Figure 3 · p. 12
C.2 Qualitative Results and Analyses
- On the Unitree Go1, the demonstrated run turns toward lower-cost objects, navigates around an obstacle, then reaches the goal object.The illustrated trajectory shows these actions at 5, 20, and 45 seconds, respectively.Source: §C.2 · p. 12, Figure 4 · p. 13
C.2.1 Simulator Deployment
- Reported simulator failures include obstacle-avoidance and perception errors, especially when facing away from the goal or deviating from the mapping run.Goal-segment matching failures are more common in Alt-Goal and Reverse when the goal lies near the mapping perspective’s periphery.Source: §C.2.1 · pp. 12–13
C.2.2 Real-World Deployment
Real-world demonstrations show ObjectReact handling changes in routes, goals, embodiment, lighting, and obstacles, while the discussion identifies limits in perception, mapping, control, and exploration.
C.2.2 Real-World Deployment
- Real-world trials rerouted around an unmapped obstacle and demonstrated phone-to-quadruped transfer, peripheral-goal navigation, shortcuts, lighting changes, and adaptation to new obstacles.The shortcut trial reached the goal by a direct path despite a longer, winding mapping route.Source: §C.2.2 · pp. 13–14
D.1 Perception
- With inferred perception, object-relative control outperformed image-relative control, but challenging-task performance remained subpar, with SAM recall, FastSAM mask quality, and data association still problematic.The simulator ground-truth perception results are an optimistic upper bound for the learned controller.Source: §D.1 · p. 14
D.2 Mapping and Planning
- High-degree floor and ceiling nodes can create misleading shortcuts; removing them is a symptomatic fix, while spatial extent or local 3D boundaries could improve path lengths.The approach removes nodes whose CLIP vision embeddings match “floor” or “ceiling.”Source: §D.2 · p. 14
D.3 Learnt Controller
- Because the controller lacks history context, frame-to-frame costmap variation from segmentation or matching failures can destabilize control; execution-time smoothing stabilized the costmap.The authors also suggest learning the dynamics or smoothing the goal encoding.Source: §D.3 · p. 14
D.4 Changes in the Map and Dynamic Objects
- Significant scene rearrangement is likely to cause failures, while apparent dynamic-obstacle avoidance is not trained behavior and depends on the image matcher’s true-negative detection.Severe occlusion and mismatching of dynamic obstacles are associated with failures.Source: §D.4 · p. 15
D.5 Exploration
- The initial map stays fixed during episodes; exploration when lost or when a goal is absent from the map remains a possible extension, for which WayObject Costmaps are a proposed starting point.The agent may still progress in a new region when very few objects match the map.Source: §D.5 · p. 15
D.6 Real-world Deployment
Real-world deployment differs from simulation in camera parameters and perception, and shortest-path training can leave little room for error around obstacles; the authors identify related failure modes and possible improvements.
D.6 Real-world Deployment
- Although demonstrations show real-world generalization, deployment changes camera parameters and uses a perception pipeline with failure modes unlike ground-truth simulator training.The deployed pipeline uses FastSAM and LightGlue, while training uses ground-truth simulator perception with path-length perturbations.Source: §D.6 · p. 15
- Cloning shortest-path demonstrations encourages tight turns around objects and leaves little obstacle margin, sometimes causing collisions during execution.The demonstrations also omit examples of taking longer routes around obstacles.Source: §D.6 · p. 15
- Perception failures can create broad, low-cost costmap regions, and recovery after the agent leaves the mapped area is not explicitly demonstrated in training data.Failures include missed or mismatched goal and obstacle segments.Source: §D.6 · p. 16
- Proposed improvements include real-world training data, obstacle-margin-aware demonstrations, training with inferred perception, and further perception-pipeline improvements.These are the authors’ stated plans for addressing deployment limitations.Source: §D.6 · p. 16