PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation

Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, Ian Reid

arXiv:2405.05792v1cs.ROcs.AIcs.CVcs.HCcs.LG

TL;DR

Image-based topological maps lack explicit object-level representation and connectivity, while metric maps rely on geometric optimization. RoboHop builds a topological graph from open-vocabulary image segments and uses connected segments for planning and localization. Preliminary trials indicate zero-shot real-world navigation without 3D maps, image poses, or a learned policy.

  • Problem

    Image-discretized topological maps lack explicit representation and cross-image association of physical entities, limiting their semantic expressivity.

    Source: §I · p. 1

  • Method

    RoboHop builds a topological graph of semantically meaningful, open-vocabulary image segments, linking segments across observations and neighboring segments within images.

    Source: §I · p. 1

  • Results

    Preliminary segment-level hopping trials indicate zero-shot real-world navigation without 3D maps, image poses, or a learned policy.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The approach demonstrates segment-based mapping, planning, and localization for object-goal navigation, with preliminary real-world hopping trials.

    Source: §VI · p. 7

    Navigation can fail when repetitive structures cause segment-association mismatches; the current method cannot handle dynamic environmental changes, and relational queries can fail when metric information is needed to judge whether objects are next to each other.

    Source: §V · p. 7

Abstract

from arXiv · show

Mapping is crucial for spatial reasoning, planning and robot navigation. Existing approaches range from metric, which require precise geometry-based optimization, to purely topological, where image-as-node based graphs lack explicit object-level reasoning and interconnectivity. In this paper, we propose a novel topological representation of an environment based on "image segments", which are semantically meaningful and open-vocabulary queryable, conferring several advantages over previous works based on pixel-level features. Unlike 3D scene graphs, we create a purely topological graph with segments as nodes, where edges are formed by a) associating segment-level descriptors between pairs of consecutive images and b) connecting neighboring segments within an image using their pixel centroids. This unveils a "continuous sense of a place", defined by inter-image persistence of segments along with their intra-image neighbours. It further enables us to represent and update segment-level descriptors through neighborhood aggregation using graph convolution layers, which improves robot localization based on segment-level retrieval. Using real-world data, we show how our proposed map representation can be used to i) generate navigation plans in the form of "hops over segments" and ii) search for target objects using natural language queries describing spatial relations of objects. Furthermore, we quantitatively analyze data association at the segment level, which underpins inter-image connectivity during mapping and segment-level localization when revisiting the same place. Finally, we show preliminary trials on segment-level `hopping' based zero-shot real-world navigation. Project page with supplementary details: oravus.github.io/RoboHop/

I. INTRODUCTION

RoboHop represents environments as graphs of semantically meaningful image segments, addressing the limited object-level expressivity of image-based topological maps. The representation supports segment-level navigation planning and localization through connected segment tracks and neighboring segments.

I. INTRODUCTION

  • Segment tracks and intra-image hops support navigation plans from open-vocabulary queries without a learned policy.The map connects source and target nodes specified through text-based queries.Source: Figure 1 · p. 1, §I · p. 1
  • Image-discretized topological maps lack explicit representation and cross-image association of physical entities, limiting their semantic expressivity.Existing maps encode some spatial reasoning through image-level connectivity.Source: §I · p. 1
  • RoboHop uses semantically meaningful, open-vocabulary image segments as graph nodes, linking segments across observations and neighboring segments within each image.The within-image links use pixel centroids; the approach contrasts with pixel-level features.Source: §I · p. 1
  • Neighborhood aggregation updates segment descriptors, supporting robot localization through segment-level retrieval.The representation describes a continuous sense of place through persistent segments and their neighboring nodes.Source: §I · p. 1
  • The paper’s contributions include segment-based mapping and connectivity, semantically interpretable navigation plans, and preliminary zero-shot real-world navigation trials.Plans use text queries to define object-level source and target nodes.Source: §I · pp. 1–2

II. RELATED WORK

Prior mapping methods trade metric precision, semantic information, computational demands, and object-level connectivity. RoboHop instead uses semantically meaningful, open-vocabulary segments as nodes in a purely topological map.

Mapping

  • Metric mapping methods can be accurate but computationally intensive and limited in semantics, while purely topological maps simplify computation but lack explicit object-level connectivity.Hybrid approaches add semantics but can remain computationally intensive.Source: §II · p. 2

Navigation

  • Image-as-node navigation maps lack explicit object-level reasoning unless combined with 3D input, and learned control policies may require environment- and embodiment-specific data.Map-based strategies are described as modular, scalable, and interpretable.Source: §II · p. 2
  • RoboHop uses segment tracks and segment-servoing subgoals in a purely topological map with semantically meaningful, open-vocabulary-queryable nodes.As segments disappear from parts of an image, other segments can match to the next image, allowing hopping across nodes.Source: §II · p. 2

III. ROBOHOP

The RoboHop pipeline comprises mapping, localization, planning, navigation, and open-vocabulary natural-language querying.

III. ROBOHOP

  • The pipeline includes a mapping module.Mapping is one of the five modules identified for RoboHop.Source: §III · p. 2
  • Localization and planning are named modules in the RoboHop pipeline.The passage identifies both as pipeline components.Source: §III · p. 2
  • Navigation and open-vocabulary natural-language querying complete the listed pipeline modules.The pipeline description lists these alongside mapping, localization, and planning.Source: §III · p. 2

A. Mapping

RoboHop represents an environment as a topological graph built from image segments, with segment-level and within-image connections. Segment descriptors incorporate neighboring nodes to add place context.

A. Mapping

  • Image segments form the nodes of a topological map, using zero-shot segmentation to avoid tying the representation to a closed set of recognized objects.The map is defined as G = (N, E), with N and E representing nodes and edges.Source: A. Mapping · p. 2
  • Each segment node includes a mask centroid and an L2-normalized descriptor aggregated from pixel-level DINO or DINOv2 features.CLIP descriptors are also used to exclude segments highly similar to labels for stuff such as floor, ceiling, and wall.Source: A. Mapping · p. 2
  • Within-image edges connect segment centroids through Delaunay triangulation, while inter-image edges associate segment descriptors across image pairs.The association considers images within a temporal window; if no match passes the threshold, the highest-similarity pair links to the next image to keep the graph connected.Source: Edges · p. 2
  • When no segment match passes the threshold, the highest-similarity pair is linked to the next image; the map does not define loop-closure edges.Loop-closure edges are identified as a possible way to enhance the map with shortcuts.Source: Edges · p. 2
  • Neighborhood aggregation through graph convolutions adds place context to segment descriptors, with successive layers incorporating neighbors of neighbors.Aggregation is applied to both the map and query image with lmax = 2.Source: Node Descriptor & Aggregation · pp. 2–3

B. Localization

Localization is performed by retrieving map nodes that match segment descriptors in a query image. The descriptors include neighborhood context, which improves localization ability.

B. Localization

  • For each query-image segment, localization matches its descriptor against map segment nodes and accepts matches above a similarity threshold.The method uses node retrieval for localization.Source: B. Localization · p. 3
  • The authors report that descriptor richness together with a simple threshold provided high-quality retrieval.They note that more sophisticated retrieval methods are available.Source: B. Localization · p. 3
  • Neighbor-informed segment descriptors add place context and improve localization ability.The descriptors are updated through neighborhood aggregation described in Eq. 2.Source: B. Localization · p. 3

C. Global Planning

The map's segment interconnectivity is used to generate navigation plans as tracks that hop continuously between segments as they enter and leave the field of view.

C. Global Planning

  • Navigation plans use segment tracks, with continuous hopping from one track to another as segments enter and leave the field of view.Source: C. Global Planning · p. 3

1) Edge Weighting:

Dijkstra planning uses edge weights that favor connections across images, producing persistent segment tracks as navigation sub-goals.

1) Edge Weighting:

  • Dijkstra planning assigns weights of 0 to inter-image edges and 1 to intra-image edges, so shortest-path search prefers connections across images.Source: 1) Edge Weighting · p. 3
  • The weighted paths yield tracks of persistent entities that serve as navigation sub-goals, with hopping across those sub-goals leading to the destination.Source: 1) Edge Weighting · p. 3
  • These edge weights are used for navigation planning, not for node descriptor aggregation.Source: 1) Edge Weighting · p. 3

2) Planning Strategy:

The planning strategy uses segment-level hops, with discrete and continuous control modes, and supports natural-language queries for choosing object goals and planning paths.

2) Planning Strategy:

  • Intra-DT is the default planning mode, using Delaunay Triangulation to connect intra-image segments.The section defines two plan-generation variants based on how intra-image edges are connected.Source: 2) Planning Strategy · p. 3

1) Discrete Control Mode:

  • Discrete control matches each planned segment to the current observation, using the best match to track the object or determine when to advance to the next node.The best-match similarity determines lost versus track state; horizontal pixel offset guides motion, and segment-size ratio signals a successful hop.Source: 1) Discrete Control Mode · p. 4

2) Continuous Control Mode:

  • Continuous control matches query segments to a local submap and uses path lengths to weight horizontal offsets toward the goal until the minimum matched path length reaches zero.Source: 2) Continuous Control Mode · p. 4

E. Querying the Map with Open Vocabulary

IV. EXPERIMENTS AND RESULTS

  • The experiments are designed to validate segment-level localization, hopping-based planning, and object-level control.Source: §IV · p. 5

A. Segment-Level Data Association

The section evaluates segment-level association with indoor instance ground truth and assesses localization on a campus dataset; it also presents qualitative planning results on two datasets.

A. Segment-Level Data Association

  • Indoor ground-truth instance associations enable direct evaluation, while outdoor evaluation uses downstream localization because comparable ground truth is unavailable.Source: A. Segment-Level Data Association · p. 5

1) Object Instance and Category Recognition:

  • The indoor association evaluation uses 544 SAM segment views of 68 objects from 18 categories, with top-1 descriptor-matching accuracy against ground-truth objects.The views are extracted from GibsonEnv panoramas at a 120-degree field of view; segments under 0.2% of image area are excluded.Source: 1) Object Instance and Category… · p. 5
  • DINO performs better than CLIP in instance- and category-level recognition; the examples include category recognition despite instance-level failures among nearby chairs.Source: 1) Object Instance and Category… · p. 5, Figure 4 · p. 5

2) Segment-level Topological Localization:

B. Planning

  • Qualitative pipeline results construct a segment-level map, retrieve source and target nodes by text and CLIP similarity, then plan between them on PanoContext-Living and GPCampus-DayLeft.PanoContext-Living represents a rotation-based traversal with loop closure between its final and first frames; GPCampus-DayLeft is a forward-moving traverse.Source: B. Planning · p. 6

1) PanoContext-Living:

The section illustrates segment-level paths in panoramic rotation and forward-moving navigation, then reports preliminary continuous-control trials in simulation and the real world. Across the examples, the plans differ with edge definitions and weighting, and Intra-DT uses multiple objects as intermediate hops.

1) PanoContext-Living:

  • Intra-All found the wraparound Window-to-Sofa path via Chair, while a partial Chair match and a full Chair with wheels match led to practically similar yaw-step paths to Television.The Chair query matched a partial visual observation, unlike Chair with wheels, which matched the full chair.Source: 1) PanoContext-Living · p. 6, Figure 6 · p. 6
  • Intra-DT paths span more objects than Intra-All for smoother transitions, and use ubiquitous carpet nodes as low-cost hops when reaching Television.The carpet remained in the common segments after preprocessing.Source: 1) PanoContext-Living · p. 6

C. Navigation:

  • Figure 8 depicts Habitat continuous control, with matched-segment pixel offsets weighted by path length to aggregate angular velocity.The figure encodes path length from red to green and shows navigation toward a green painting goal.Source: Figure 8 · p. 7, C. Navigation · p. 7

2) GPCampus-DayLeft:

  • For the Z block-to-Dustbin traverse, Intra-DT first hops to a door with a persistent track, Intra-All hops to a closet track, and DA-All rapidly hops without persistent tracks.The DA-All baseline includes an unthresholded inter-image edge for every segment, so planning can avoid intra-image edges.Source: 2) GPCampus-DayLeft · pp. 6–7, Figure 7 · p. 6

C. Navigation

  • Preliminary trials used segment-level mapping and planning for zero-shot navigation in both the real world and simulation.In Habitat, a PID controller converts horizontal pixel offset into yaw velocity while forward translation remains fixed at a small velocity.Source: C. Navigation · p. 7

V. LIMITATIONS

The paper identifies constraints in segment association, dynamic-scene handling, segmentation consistency, and relational queries that need metric information. It concludes with preliminary zero-shot navigation trials and outlines possible extensions to visual servoing, local metric information, and scene graphs.

V. LIMITATIONS

  • Navigation depends strongly on segment-level data association, which can fail on repetitive structures; the method also cannot currently handle dynamic environmental changes.The authors report LightGlue performing better than DINOv2 for association in highly aliased environments.Source: §V · p. 7
  • Imperfect repeatable segmentation on revisits can distort area ratios and affect forward/backward motion and hop-state decisions; ground or wall segments may also persist after preprocessing.The authors note depth information could address the revisitation issue while retaining the topological map.Source: §V · p. 7
  • LLM-handled relational queries can fail when metric information is needed to determine whether two objects are next to each other.Source: §V · p. 7

VI. CONCLUSION AND FUTURE WORK

  • The conclusion presents preliminary segment-level hopping trials as enabling zero-shot navigation without 3D maps, image poses, or a learned policy.Source: §VI · p. 7
  • Future work proposes visual-servoing feedback, local node- and edge-level metric information, and semantic node labels for 3D scene graphs.Source: §VI · p. 7
Loading 2405.05792v1…