PaperlayerView exact revision on arXiv

Source-linked AI summary

Clio: Real-time Task-Driven Open-Set 3D Scene Graphs

Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, Luca Carlone

arXiv:2404.13696v4cs.RO

TL;DR

Robots need maps whose semantic granularity supports their tasks, but choosing that granularity is not addressed by a general framework. Clio uses an Information Bottleneck-based incremental clustering method to build task-relevant 3D scene graphs online. Its experiments report compact maps, closed-set performance on par with the state of the art, and support for real-robot task execution.

  • Problem

    The appropriate object and scene-structure granularity depends on the robot’s tasks, but map representations must choose what to retain to support those tasks.

    Source: §I · p. 2

  • Method

    Clio applies incremental Agglomerative Information Bottleneck clustering to task-agnostic 3D primitives, then integrates the task-relevant objects and regions into an online scene graph.

    Source: §I · p. 3, §VIII · p. 9

  • Results

    Clio constructs more parsimonious and useful maps, performs on par with the state of the art in closed-set settings, clusters meaningful regions, and supports task execution on real robots.

    Source: §VI · p. 6

  • Takeaways & Limitations

    Clio’s task-relevant scene graph supports real-time onboard pick-and-place mobile manipulation on Spot.

    Source: §VIII · p. 9

    Clio inherits foundation-model vulnerabilities, including strong sensitivity to prompt tuning.

    Source: §VII · p. 8

Abstract

from arXiv · show

Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and mapping. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us with a fundamental question: what is the right granularity for the objects (and, more generally, for the semantic concepts) the robot has to include in its map representation? While related work implicitly chooses a level of granularity by tuning thresholds for object detection, we argue that such a choice is intrinsically task-dependent. The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks. We show that this problem can be naturally formulated using the Information Bottleneck (IB), an established information-theoretic framework. The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into task-relevant objects and regions and executes incrementally. The third contribution is to integrate our task-driven clustering algorithm into a real-time pipeline, named Clio, that constructs a hierarchical 3D scene graph of the environment online using only onboard compute, as the robot explores it. Our final contribution is an extensive experimental campaign showing that Clio not only allows real-time construction of compact open-set 3D scene graphs, but also improves the accuracy of task execution by limiting the map to relevant semantic concepts.

I. INTRODUCTION

A map is useful insofar as it supports the robot’s tasks, but the right object granularity depends on which tasks it must perform. Clio formulates task-driven mapping with the Information Bottleneck, applies incremental Agglomerative IB, and builds real-time 3D scene graphs onboard.

I. INTRODUCTION

  • A piano may be one object for moving, keys for playing, or keys and strings for tuning, so useful map granularity depends on the task.The appropriate representation remains ill-posed until the tasks it must support are specified.Source: §I · p. 2
  • The task-driven mapping problem compresses task-agnostic primitives into a minimal representation sufficient to complete natural-language tasks, formulated using the Information Bottleneck.The primitives include 3D object segments and obstacle-free places, which are clustered into relevant objects and regions.Source: §I · p. 3
  • Agglomerative IB uses CLIP embeddings to derive the required probability densities and runs incrementally, with computational complexity that does not increase with environment size.The algorithm is applied to task-driven 3D scene understanding.Source: §I · p. 3
  • Clio builds a hierarchical 3D scene graph online as the robot explores, retaining task-relevant objects and regions while running onboard in real time with lightweight foundation models.Tasks are specified in natural language at the beginning of operation.Source: §I · p. 3
  • Experiments report compact open-set scene graphs and improved task-execution accuracy from limiting maps to relevant objects and regions, including real-time onboard mapping and manipulation on Spot.The evaluation includes Replica and four real environments: an apartment, an office, a cubicle, and a large-scale building scene.Source: §I · p. 3

II. RELATED WORK

Prior work applies foundation models and open-set semantics to 3D mapping, while task-driven approaches vary in scope and computational cost. Clio frames scene understanding as compressing task-agnostic primitives into task-relevant concepts, balancing compactness with information about the task.

II. RELATED WORK

  • Vision-language models and class-agnostic segmentation support open-set understanding, while related work explores 3D scene understanding, robot planning, and saliency.Clio seeks task-driven rather than visual-saliency-driven maps.Source: §II · p. 3
  • Foundation-model mapping work includes text-queryable radiance fields and language maps, incremental neural semantic maps, hierarchical maps, and implicit grids queried with CLIP.These approaches include LERF, LangSplat, and methods cited as.Source: §II · p. 3
  • Open-set 3D mapping includes CLIP/SAM-based object clustering in ConceptGraphs, while another hierarchical scene-graph approach is reported not to run in real time.ConceptGraphs optionally uses a large vision-language model to describe objects from multiple views.Source: §II · p. 3
  • Prior task-driven representation work includes Information Bottleneck clustering, visual-observation compression, and affordance-based regions and maps.The cited approaches include IB for image context, VLM attribution, and affordance work.Source: §II · pp. 3–4

III. PROBLEM FORMULATION: TASK-AWARE 3D SCENE UNDERSTANDING

  • The paper identifies a lack of a general framework for choosing task-dependent semantic granularity, with prior task-driven frameworks described as narrow or computationally expensive.Natural-language task lists Y relate visual data to tasks, while over-segmented 3D objects and obstacle-free places form task-agnostic primitives X.Source: §III · p. 4
  • The formulation compresses task-agnostic primitives X into task-relevant clusters X̃ that retain information about tasks Y, using the Information Bottleneck.The resulting clusters group 3D segments into objects and places into regions; β balances compression against task relevance.Source: §III · p. 4

IV. TASK-DRIVEN CLUSTERING

The section adapts Agglomerative Information Bottleneck clustering to task-driven scene representations, using task relevance and spatial adjacency to decide which primitives to merge. Its incremental form supports clustering as the map grows by updating only affected connected components.

Agglomerative Information Bottleneck

  • Clustering uses geometric adjacency as an inductive bias, permitting nearby primitives to merge while avoiding merges between distant segments.The method extends Agglomerative IB to iteratively merge neighboring primitives.Source: §IV · p. 4
  • The algorithm greedily merges adjacent clusters with the smallest weight, which measures the dissimilarity between their task-relevance distributions.The merge weight uses Jensen-Shannon divergence, and graph edges specify allowable merges.Source: §IV · p. 4
  • The algorithm stops when fractional information loss δ(k) exceeds a threshold, which controls compression from the original primitives to fully merged primitives.A threshold of 0 returns the original primitives; 1 returns fully merged primitives.Source: §IV · p. 4

Incremental Agglomerative IB

  • Incremental clustering processes connected components independently and recomputes the stopping measure only for components affected by new measurements.This supports online clustering as new primitives arrive while avoiding work on unaffected components.Source: §IV · pp. 4–5

Task-Relevant Conditional Distributions

  • CLIP similarities to task embeddings define primitive relevance, while a null task captures primitives below the relevance threshold and top-ranked task similarities are retained.The resulting conditional probabilities provide the task-relevance distributions used by clustering.Source: §IV · p. 5

V. CLIO: REAL-TIME TASK-DRIVEN OPEN-SET 3D SCENE GRAPHS

Clio has a frontend that builds task-agnostic object and place primitives and a backend that clusters objects and regions according to user-specified tasks. The frontend constructs these representations from RGB-D input while incrementally maintaining object primitives and a background mesh.

A. Clio Frontend

  • The frontend processes RGB-D images and poses into semantic segments, tracks, 3D object primitives, a place graph, and a background mesh; the backend clusters objects and regions based on tasks.Object primitives are reconstructed from track frames with averaged CLIP features, while background reconstruction is coarser and performed for each incoming frame.Source: Figure 3 · p. 5, §V · p. 5
  • Place primitives form a graph by incrementally computing and sparsifying a Generalized Voronoi Diagram, with each node assigned averaged CLIP features from images where its centroid is visible.The place graph follows the Hydra approach.Source: A. Clio Frontend · p. 5

B. Clio Backend

Clio’s backend clusters object primitives for task-relevant mapping, and its experiments find that task-informed approaches improve open-set precision while retaining fewer objects. Across the evaluated datasets, Clio generally outperforms baselines, with real-time online performance close to batch performance in most cases.

B. Clio Backend

  • Clio builds a hierarchical 3D scene graph online from a natural-language task list, retaining task-relevant objects and regions.The backend clusters overlapping object primitives using p(y|x), with Clio-batch clustering all scene primitives and Clio-online reclustering only affected connected components after new measurements.Source: B. Clio Backend · p. 5
  • Clio clusters place primitives into regions at each backend update.The supplied passage describes this place-clustering operation but does not provide further details about its edge construction.Source: B. Clio Backend · p. 5

VI. EXPERIMENTS

  • Clio’s evaluated settings include Replica and four real environments, with onboard mapping demonstrated on a Spot quadruped with a robotic arm.The open-set object-clustering evaluation uses office, apartment, cubicle, and five-floor university-building datasets; ground-truth object boxes were annotated for the first three.Source: A. Open-Set Object Clustering Evaluation · p. 6

A. Open-Set Object Clustering Evaluation

  • The evaluation reports open-set Recall and Precision, their harmonic-mean F1, average IOU, estimated-object count, and runtime per frame.Strict detections require both centroid-containment conditions; relaxed detections require at least one.Source: A. Open-Set Object Clustering Evaluation · p. 6
  • Comparisons include ConceptGraphs, Khronos, Clio primitives without IB clustering, task-aware baseline variants, and both batch and online Clio.ConceptGraphs uses CLIP in place of LLaVA+GPT for these queries; task-aware baselines filter mapped objects by task similarity.Source: A. Open-Set Object Clustering Evaluation · p. 6
  • Task-informed approaches improve open-set precision while retaining far fewer objects, with Clio retaining an order of magnitude fewer than task-agnostic baselines in some cases.Task-aware Khronos and ConceptGraphs have strictly worse open-set recall than their task-agnostic versions because they filter objects but do not form task-dependent objects.Source: A. Open-Set Object Clustering Evaluation · p. 6
  • Clio generally outperforms baselines across datasets and metrics, ranking first or second in all but Office IOU and strict open-set recall cases.Clio runs in a fraction of a second and is around six times faster than ConceptGraphs; batch and online versions perform similarly in most cases.Source: Table I · p. 7
  • For condiment tasks, Clio groups varied packets as one object for general retrieval but splits them by sauce type for tasks requiring specific condiments.The figure compares clustering under two task sets, with 14 additional shared tasks included but not displayed.Source: Figure 4 · p. 6, A. Open-Set Object Clustering Evaluation · p. 7

B. Closed-Set Object Evaluation

Clio’s closed-set evaluation finds comparable class-mean recall on Replica, while its room-region results vary with scene structure. Task prompts also produce regions at different granularities using the same parameters.

B. Closed-Set Object Evaluation

  • Clio achieves comparable mAcc to leading methods on Replica; OpenMask3D has superior f-mIOU but requires a full 3D reconstruction.The evaluation uses 8 Replica scenes; the authors state the results indicate task-aware clustering does not degrade closed-set performance.Source: Table II · p. 7, B. Closed-Set Object Evaluation · p. 7

C. Open Vocabulary Places Clustering

  • Clio (average) outperforms Hydra and Clio (closest) on room segmentation in Office and Building, but performs relatively poorly in Apartment.Results are averaged over 5 trials; Table III reports standard deviations for all metrics.Source: Table III · p. 7, C. Open Vocabulary Places Clustering · p. 7
  • Apartment’s geometrically distinct rooms are over-segmented by Clio, while connected, semantically similar Office regions are under-segmented relative to Hydra.The authors associate the Apartment result with high precision but low recall, and the Office result with lower recall.Source: Figure 5 · p. 8
  • On the Office scene, room-label prompts and more granular mixed room/object prompts produce regions reflecting their different task granularities with unchanged parameters.The figure provides qualitative examples of the resulting regions.Source: C. Open Vocabulary Places Clustering · p. 8, Figure 5 · p. 8

D. Online Evaluation on Spot

Spot trials demonstrate real-time Clio mapping and grasping, with most reported failures not arising from selecting the wrong Clio object. The paper also identifies limits in foundation-model sensitivity and task complexity.

D. Online Evaluation on Spot

  • Seven trials used mapping followed by planning; the repeated manipulation trials ran with the laptop off-board, while an attached video shows onboard computation.The authors cite logistical challenges, including battery life, for running the laptop off-board during trials.Source: D. Online Evaluation on Spot · p. 8, Figure 6 · p. 8
  • 57% of 21 grasp attempts succeeded, rising to 71% when Spot API failures to pick up correctly identified objects are disregarded.Clio selected the wrong target object once; the trial breakdown distinguishes selection, detection, navigation, and Spot failures.Source: Figure 6 · p. 8

VII. LIMITATIONS

  • Clio remains vulnerable to foundation-model prompt tuning, may over-cluster objects with similar task relevance, and currently targets relatively simple single-step tasks.The authors also identify averaging CLIP vectors when merging primitives as an area for improvement.Source: §VII · p. 8

VIII. CONCLUSION

  • Clio constructs task-relevant scene graphs as the robot explores and has been demonstrated in real time onboard Spot for pick-and-place mobile manipulation.The conclusion presents this as a demonstration of the system’s relevance for robotics.Source: §VIII · pp. 8–9

APPENDIX

Agglomerative IB hard-clusters primitives into a compressed representation while preserving task-relevant information. Its incremental version reprocesses affected connected components as new measurements arrive.

A. Agglomerative Information Bottleneck

  • Agglomerative IB merges primitives into hard clusters, compressing the initial signal while preserving information about the task-relevance variable until a compression threshold is reached.The threshold regulates compression relative to preserving information about the tasks.Source: A. Agglomerative Information Bottleneck · p. 10

B. Incremental Agglomerative IB

  • Under a uniform primitive distribution, the incremental method uses a reweighted stopping condition that yields the same result as full-graph Agglomerative IB.When new primitives arrive, Clio updates their subgraphs and reruns clustering only on affected connected components.Source: B. Incremental Agglomerative IB · p. 10

C. Office, Apartment, and Cubicle Datasets

The office, apartment, cubicle, and building datasets are collected with an Intel RealSense D455; the resulting scene graphs are visualized.

C. Office, Apartment, and Cubicle Datasets

D. Office Scene Task List

The office task list covers 33 distinct objects and natural-language requests spanning retrieval, manipulation, organization, counting, and cleaning.

D. Office Scene Task List

  • The office task list contains 33 distinct objects assigned across its tasks.The listed task-object counts appear in parentheses.Source: D. Office Scene Task List · p. 10
  • Requests include retrieving specific items, such as a black Expo marker, a tractor painting, and a Signals and Systems textbook.The list also includes requests to move a magazine rack and organize dishes or hardware tools.Source: D. Office Scene Task List · pp. 10–11
  • Some tasks involve multiple assigned objects, including paper-cutting tools, printer tasks, and counting solid-core wood doors, each with three.Other listed tasks have one or two objects assigned.Source: D. Office Scene Task List · pp. 10–11

E. Apartment Scene Task List

The apartment, cubicle, and building task lists span household items, workplace objects, and larger equipment, with task-object counts varying by scene.

E. Apartment Scene Task List

  • The apartment list covers 28 distinct objects, including requests to retrieve kitchen items, personal belongings, and snacks, and to move clothes or dishes.The kitchen-knife task has three assigned objects, while bringing a pillow has two.Source: E. Apartment Scene Task List · p. 11

F. Cubicle Scene Task List

  • The cubicle list has 18 objects, with one corresponding object for each task, including notebooks, waste bins, backpacks, and tools.Requests also include cleaning backpacks and moving hats.Source: F. Cubicle Scene Task List · p. 11

G. Building Scene Task List

  • The building list includes tasks for items such as a jet engine, CO2 tanks, cleaning supplies, and stairs; some tasks have many relevant occurrences.Its scene-graph example draws objects as cubes, places as spheres, and regions as cubes, with places and regions colored by their closest task.Source: G. Building Scene Task List · pp. 11–12, Figure 8 · p. 12

H. Open Vocabulary Tasks on OpenCLIP model

The experiments vary the image model and task prompts, and include visual examples of task-driven region clustering in the office dataset.

H. Open Vocabulary Tasks on OpenCLIP model

  • OpenCLIP ViT-H-14 produces higher cosine similarities for both relevant and irrelevant image-primitive/task pairings, so the experiments raise the null task value and threshold α to 0.26.Because this model requires more compute, the authors run Clio-batch but not Clio-online; Table IV covers open-set task-query object localization across three datasets and includes this model.Source: H. Open Vocabulary Tasks on OpenCLIP… · p. 12, Table IV · p. 13

I. Closed-Set Places Clustering Task List

  • The closed-set places-clustering prompts use images of scene types such as kitchens, bedrooms, workspaces, and hallways.The listed prompts vary by scene, with the building list also including a workshop or machine shop and an aircraft hangar or garage.Source: §I · pp. 12–13

J. Places Clustering Results Visualization

Related papers

Loading 2404.13696v4…