Source-linked AI summary
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, Liam Paull
TL;DR
ConceptGraphs addresses the scalability and structural limitations of dense semantic 3D representations by constructing open-vocabulary, object-centric scene graphs. It integrates geometric mapping with language and vision-language foundation models, supports language-based retrieval and planning, and demonstrates robustness across manipulation and navigation tasks.
Problem
Dense 3D representations with per-point semantic features consume excessive memory, scale poorly to large scenes, and lack structure for dynamic map updates.
Method
ConceptGraphs constructs open-vocabulary 3D scene graphs whose object nodes combine geometric and semantic features, while graph edges encode relationships using off-the-shelf language and vision-language models without training or finetuning.
Results
ConceptGraphs demonstrates robustness and extensibility across real-world manipulation and navigation tasks, with spatial relationships labeled at 90% average accuracy.
Takeaways & Limitations
The scene graph provides a low-memory, structured representation that supports efficient task planning and free-form language querying over large scenes.
Takeaways & Limitations
Node-captioning errors, missed small or thin objects, duplicate detections, and the computational and economic costs of multiple model inferences remain limitations.
Abstract
from arXiv · showhide
For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )
I. INTRODUCTION
ConceptGraphs addresses the need for scalable, open-vocabulary, task-useful 3D representations by organizing object semantics and spatial relationships in a graph. It combines 2D foundation models with object-centric 3D mapping for perception and planning.
- Motivation: Robotic scene representations should scale with environment size and operation duration, support novel concepts at inference time, and offer task-appropriate detail.The requirements span dense geometry for mobility and manipulation and more abstract representations for other tasks.
- Motivation: Existing dense 3D semantic representations assign feature vectors to every point, consuming excessive memory and hindering scalability and dynamic map updates.Their lack of structure also makes decomposition difficult for robotics.
- Contribution: ConceptGraphs represents each object as a graph node with geometric and semantic features, while inter-object relationships are encoded by graph edges.The resulting abstraction is open-vocabulary and structured for perception and planning.
- Contribution: The approach integrates geometric cues from traditional 3D mapping with semantic cues from vision and language foundation models using off-the-shelf models without training or finetuning.Language tags and descriptions enable free-form language querying.
- Evaluation: ConceptGraphs is evaluated on real-world wheeled and legged robots across manipulation, navigation, localization, map updates, and complex language-query tasks.These experiments examine both perception and planning capabilities.
II. METHOD
ConceptGraphs incrementally converts posed RGB-D observations into object-centric 3D entities by combining segmentation, visual features, geometric-semantic association, and multi-view fusion. It then captions the mapped objects with vision-language models for language-based scene understanding.
- Object-centric mapping: Given posed RGB-D frames, ConceptGraphs incrementally adds incoming observations to existing objects or instantiates new ones in an object-centric 3D map.Each mapped object stores a point cloud and semantic feature vector.
- Object-centric mapping: Class-agnostic segmentation identifies candidate regions, whose visual descriptors are extracted and whose masked pixels are projected into a denoised 3D point cloud.The system uses geometric projection and DBSCAN denoising before mapping.
- Object association: New detections are associated with overlapping mapped objects using the sum of semantic and geometric similarity, with greedy matching or new-object initialization when similarity is insufficient.The geometric term uses nearest-neighbor overlap, while the semantic term compares visual descriptors.
- Object fusion: Associated detections are fused by updating the mapped object’s semantic feature and unioning its point cloud, followed by downsampling to remove redundant points.This maintains a compact multi-view object representation.
B. Scene Graph Generation
ConceptGraphs completes the scene graph by estimating spatial relationships among mapped objects and connecting the resulting representation to language-driven task execution. The graph supports object retrieval and downstream navigation or manipulation.
- Scene graph generation: Potential graph edges are generated from spatial overlap by computing pairwise 3D bounding-box IoU and pruning the dense graph with a minimum spanning tree.This produces a refined set of candidate connections among object nodes.
- Scene graph generation: An LLM extends the nominal spatial edge type to relationships expressed in natural language, such as storage or recycling relations between objects.These relations contribute to the open-vocabulary graph.
- Language-based planning: For a natural-language task query, an LLM selects the most relevant object from JSON descriptions containing each node’s bounding box and caption.The selected object’s 3D pose is passed to a downstream grasping or navigation pipeline.
- Implementation: The implementation uses SAM, CLIP, LLaVA, and GPT-4, with 2.5 cm voxel size and nearest-neighbor threshold settings.ConceptGraphs-Detector instead combines RAM image tagging with Grounding DINO bounding boxes and merges background objects separately.
III. EXPERIMENTS
ConceptGraphs is evaluated as an open-vocabulary 3D mapping system using human-assessed scene-graph accuracy, semantic segmentation, and text-query retrieval. Its graph edges are highly accurate, while node-label errors primarily reflect LVLM captioning.
- Scene Graph Construction: 90% average edge precision indicates that ConceptGraphs estimates spatial relationships accurately.Human evaluators assessed the estimated relationships between scene-graph nodes.
- Scene Graph Construction: About 70% node-label accuracy is reported, with most errors attributed to the employed LVLM.Both CG and CG-D discover valid objects while producing only 0–5 duplicate detections per scene.
- Semantic Segmentation: The study evaluates open-vocabulary semantic segmentation on Replica alongside scene-graph construction.The segmentation evaluation compares privileged finetuned methods with zero-shot methods evaluated off the shelf.
- Complex Semantic Queries: ConceptGraphs is tested on descriptive, affordance, and negation queries using Replica scenes and a real-world REAL Lab scan.The evaluation includes staged clothes, tools, and toys in the lab scan.
- Complex Semantic Queries: CLIP excels on descriptive queries, whereas the LLM performs better on complex affordance and negation queries but depends on caption accuracy.For example, the LLM retrieves a roll of tape for a broken-zipper query, while CLIP retrieves a backpack.
D. Complex Visual-Language Queries
ConceptGraphs supports complex visual-language queries by combining a 3D scene graph with LLM-based reasoning and robot perception. Experiments demonstrate abstract-object retrieval, navigation, traversability reasoning, and open-vocabulary pick-and-place across multiple robot platforms.
- Complex Visual-Language Queries: The Jackal navigates to objects selected from abstract user queries and can answer visual queries using an LVLM description of the camera image.Given a picture of Michael Jordan and the query “Something this guy would play with,” it finds a basketball.
- LLM-Grounded Robot Reasoning: ConceptGraphs combines scene-graph captions with LLM reasoning to infer object locations and whether objects can be safely pushed or traversed.These inferred properties support object-search and traversability-estimation tasks.
- Object Search: For object search, the robot checks whether the retrieved object is at its expected location and queries an LLM for an alternative location when necessary.The prompt encourages consideration of typical containers or storage locations based on other object captions.
- Traversability Estimation: Traversability estimation is demonstrated in a scenario where the robot pushes multiple objects aside to create a path to a goal state.The authors present grounding LLM knowledge in a 3D map as granting capabilities similar to experience-based traversability learning.
- Open-Vocabulary Pick and Place: Spot Arm responds to the query “cuddly quacker” by grasping a duck plush toy and placing it in a nearby box.A supplementary experiment similarly directs grasping of a mango using the query “something healthy to eat.”
G. Localization and Map Updates
ConceptGraphs supports object-based localization, map updates, object search, and traversability planning through interactions between its scene graph and language models. Real-robot demonstrations show recovery from misplaced-object assumptions and obstacle-aware path planning, while limitations remain in captioning, detection, and inference cost.
- Localization and map updates: ConceptGraphs supports 3-DoF localization and remapping in AI2Thor using a particle filter over a pre-built scene graph.Detections are matched against mapped objects to weight pose hypotheses, while missing objects can be removed and new objects added.
- Object search: Object search combines LLM object selection with LVLM validation and replanning toward plausible containers when the target is absent.The demonstrations redirect searches for a NASA shirt and red-and-white sneakers to a laundry bag and shoe rack.
- Traversability estimation: Traversability planning uses LLM predictions from node captions to add non-traversable objects to the costmap and reach the goal through safe alternatives.The Jackal passes through a curtain and pushes a basketball while avoiding bricks, an iron dumbbell, and a flower pot.
- Limitations: Failure modes include inaccurate node captions, missed small or thin objects, duplicate detections, and potentially significant LVLM and proprietary LLM costs.Incorrect detections can affect planning when they are crucial to task success.
- Related approaches: ConceptGraphs is compared with related open-vocabulary object-factorization approaches whose CLIP-based representations show similar difficulties on complex affordance and negation queries.The comparison passage characterizes those approaches as comparable to the system’s CLIP variant.
- Conclusion: The conclusion identifies dynamic scene updates, temporal dynamics, and evaluation in less structured environments as continuing directions or scope boundaries.The paper reports utility across manipulation and navigation while proposing future assessment in more challenging settings.
A1. CONTRIBUTION STATEMENT
This appendix records contributors’ roles in developing, integrating, deploying, evaluating, and documenting ConceptGraphs. It also describes the object-captioning workflow, including multi-view selection, LVLM caption generation, and GPT-4 summarization.
- Contribution statement: Qiao led object-based mapping, localization, map updates, object captioning prototypes, and segmentation experiments.
- Contribution statement: Ali developed mapping prototypes, coordinated real-robot experiments, integrated LLM planners, ran object-retrieval experiments, and supported hardware setup.
- Contribution statement: Sacha led robot navigation, object search, and traversability deployment, while Krishna developed mapping, scene-graph, and vision-language interfaces.
- Contribution statement: The project included manipulation integration, mobile-manipulation deployment, qualitative comparison with learned approaches, and advisory support for real-world demonstrations.These roles involved multiple contributors across robotic platforms and evaluation activities.
- Node captioning: Object captions are generated from up to 10 informative views per object, passed through LLaVA-7B and summarized into coherent tags by GPT-4.The selected views maximize the number of noise-free 3D points contributed by each image segment.
- Node captioning: The caption summarizer uses repeated mentions and container or surface context to produce concise object descriptions or reject invalid detections.
A3. LLM PLANNER: IMPLEMENTATION DETAILS
The LLM planner receives a JSON text representation of scene-graph objects and returns structured answers to natural-language queries. The format preserves object geometry, tags, captions, relevance rankings, achievability, and explanations.
- Planner interface: GPT-4 is instantiated with a system prompt to parse the JSON scene description and answer user questions in JSON format.The planner is configured with an 8K-token context length, with larger-context variants also tested.
- Scene representation: The paper provides sample serialized entries containing object ids, bounding-box geometry, tags, and captions for objects such as a dresser, vase, and light fixture.
- Scene representation: Each scene-graph object is serialized with a unique id, 3D bounding-box extents, bounding-box center, object tag, and caption.
- Planner outputs: Planner outputs include an interpreted query, relevant object ids, query achievability, ranked final objects, and an explanation.Final objects are ordered from most to least relevant whenever possible.
A4. EXAMPLE TEXT QUERIES
The evaluation queries span descriptive references, affordances, and negations across Replica scenes and a real lab scan. Examples combine object identity with spatial relations, intended use, physical attributes, and exclusions.
- REAL Lab queries: REAL Lab descriptive queries include concrete target objects such as red-and-white sneakers, a NASA t-shirt, a Rubik’s cube, a basketball, and a laundry bag.
- Descriptive queries: Replica descriptive queries identify objects through appearance, identity, and spatial relations such as next to, in front of, above, and on top of.Examples include furniture, screens, plants, rugs, and wall or floor relationships.
- Affordance queries: Replica affordance queries ask for objects serving functions such as watching news, telling time, sitting, storing cups, adding light, or opening with keys.
- Negation queries: Replica negation queries request objects that differ from references by properties including size, weight, rigidity, softness, or transparency.
- REAL Lab queries: REAL Lab affordance queries describe uses including laptop disassembly, CPU cooling, carrying books, securing a broken zipper, and understanding computer operation.
- REAL Lab queries: REAL Lab negation queries exclude categories or properties, including basketball, formal shoes, umbrellas, red-and-white shoes, green materials, soda, weights, and backpacks.
A5. NAVIGATION EXPERIMENTS
The navigation experiments use ConceptGraphs to locate queried objects and estimate traversability, then incorporate those results into Jackal navigation. The system combines scene-graph reasoning with onboard visual verification and costmap updates.
- Setup: Jackal navigation begins from a lidar-built REAL Lab pointcloud and a downprojected 2D costmap, while task-relevant objects are added through ConceptGraphs.The robot uses a VLP-16 lidar, a forward-facing Realsense D435i camera, and Open3d SLAM for the initial map.
- Object search: For object search, an LLM Planner selects a likely object location, and LLaVA checks whether the target is visible from that viewpoint.If the target is not visible, it is removed from the scene graph and the planner proposes another likely location or container.
- Traversability: Traversability estimation asks GPT whether the robot can traverse or push an object, given its description, tags, and 5-pound pushing limit.The prompt specifies a maximum pushing capacity of 5 pounds or 2.27 kg.
- Costmap integration: Non-traversable object pointclouds are downprojected into the Jackal costmap before navigation toward a specified room pose.This converts semantic traversability judgments into navigation obstacles.
- Model and mapping configuration: The experiments use a local offboard LLaVA instance when needed and otherwise rely on the GPT-4 API for language-model queries.Scene mapping uses an Azure Kinect and RTAB-Map to obtain camera poses and the scene pointcloud.
A6. LIMITATIONS
The authors identify captioning errors for small objects and explicitly limit the paper’s scope regarding improvements to LLM-based planning over 3D scene graphs.
- Captioning errors: LLaVA-7B misclassifies a non-negligible number of small objects as toothbrushes or pairs of scissors.The authors expect more capable vision-language models to alleviate this issue and improve node and edge precision.
- Planning scope: The work does not explicitly investigate improving LLM-based planning over 3D scene graphs.The authors point to SayPlan for using hierarchical scene-graph structure for efficient planning.