Source-linked AI summary
ConceptFusion: Open-set Multimodal 3D Mapping
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, Antonio Torralba
TL;DR
Existing 3D semantic maps are often closed-set and limited in their query modalities, while foundation models lack direct pixel-level features for precise 3D localization. ConceptFusion derives zero-shot pixel-aligned features and fuses them into dense 3D maps, enabling open-set multimodal queries; it reports over 40% mIoU improvement over supervised or finetuned counterparts for long-tailed concepts.
Problem
Existing 3D semantic maps are largely closed-set and support limited query modalities, while foundation models generally produce image-level rather than pixel-aligned features.
Method
ConceptFusion combines global and local foundation-model features into pixel-aligned embeddings and fuses them into dense 3D maps for zero-shot multimodal querying.
Results
More than 40% mIoU improvement is reported over supervised or finetuned counterparts for long-tailed concepts, alongside evaluations across real-world, simulated, manipulation, and driving settings.
Takeaways & Limitations
The method supports zero-shot spatial reasoning over a broad range of concepts using text, image, audio, and click queries.
Takeaways & Limitations
ConceptFusion requires substantial memory for dense maps with high-dimensional embeddings, and its features lack compositionality and negation understanding while inheriting foundation-model biases.
Abstract
from arXiv · showhide
Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs
I. INTRODUCTION
ConceptFusion addresses the limitations of closed-set, single-modality 3D maps by combining open-set foundation-model features with zero-shot, multimodal querying. Its pixel-aligned approach preserves fine-grained concepts better than finetuned alternatives and supports broad evaluation across datasets and robotic tasks.
- Motivation and contributions: Foundation models provide broad multimodal concept knowledge, but image-level embeddings alone do not provide the pixel- or object-level localization needed for 3D mapping.Existing pixel-aligned approaches can lose fine-grained concepts during finetuning.
- Motivation and contributions: ConceptFusion builds open-set 3D maps that support zero-shot queries from text, images, audio, and clicks.The representation is designed to capture concepts at varying levels of detail without retraining for each task.
- Motivation and contributions: ConceptFusion introduces a mechanism that derives pixel-aligned local features from foundation models producing only global image vectors.The approach combines region-level information with global image context and uses generic object masks and bounding boxes.
- Motivation and contributions: More than 40% mIoU improvement is reported over supervised or finetuned counterparts for capturing long-tailed concepts.The method is motivated by observations that finetuned pixel-aligned models may forget concepts retained by their base CLIP models.
- Evaluation scope: The evaluation includes the UnCoCo dataset, real-world and simulated object search, manipulation, and autonomous driving tasks.UnCoCo contains 78 household or office objects and more than 500K multimodal queries.
- Relation to prior work: ConceptFusion extends dense SLAM and multiview fusion by incorporating pixel-aligned foundation features into 3D maps.Related systems include scene-specific learned fields, distilled 3D representations, and approaches limited to fewer modalities or different feature extractors.
IV. THE ConceptFusion APPROACH
ConceptFusion constructs an open-set multimodal 3D map by generating pixel-aligned foundation features and fusing them with dense geometric reconstruction. The map stores concept vectors at 3D points so queries from multiple modalities can be matched spatially.
- Map representation: Given RGB-D observations, ConceptFusion builds a 3D map whose points store geometry, confidence, optional color, and concept vectors.Queries are represented as modality-specific vectors from foundation-model encoders.
- Pixel-aligned feature computation: Global and local features are combined using similarities between local and global embeddings and among local features to produce pixel-aligned features.The resulting fusion weights combine image-level context with region-level information.
- Pixel-aligned feature computation: The method computes global image, local region, and pixel-aligned embeddings using an off-the-shelf foundation model.Class-agnostic mask proposals define regions, whose bounding boxes are encoded to obtain local embeddings.
- 3D feature fusion: Pixel-aligned concept features are fused into the global 3D map alongside depth and optional color through a dense reconstruction pipeline.Image pixels are associated with map points after pose transformation and depth-noise filtering.
- Feature behavior: The approach is designed to retain fine-grained and long-tailed concepts that finetuned pixel-aligned models may forget.Figure 4 contrasts ConceptFusion with LSeg and OpenSeg on diet coke, lysol, and yogurt.
- 3D feature fusion: Each pixel-grounded feature receives confidence α = e^-γ^2/2σ^2, where γ is radial distance and σ = 0.6.The confidence is based on normalized distance from the camera center.
B. Computing pixel-aligned features
ConceptFusion constructs pixel-aligned features by combining global image context with local region information, preserving long-tailed concepts without finetuning. These features are mapped to pixels and fused into 3D representations.
- Fusing global and local features: ConceptFusion combines global image-level context with local region-level information to construct pixel-aligned features.The approach addresses the loss of fine-grained concepts observed after finetuning pixel-aligned models.
- Feature extraction: A foundation model extracts global, pixel-aligned, and local embeddings from each input image.The global embedding represents the entire image, while local embeddings are obtained from class-agnostic region proposals.
- Feature extraction: Class-agnostic mask proposals define regions whose bounding boxes are processed by the foundation model to obtain local embeddings.This supplies region-specific features even when the foundation model natively produces image-level representations.
- Fusing global and local features: Each region’s pixel-aligned embedding is a weighted combination of global and local embeddings, with the local weight based on global and inter-local similarities.The mixing weight is constrained to [0, 1] and uses a temperature τ set to 1 in reported results.
- Pixel alignment: The resulting region features are normalized, assigned to their pixels, and accumulated when pixels belong to multiple regions.This produces pixel-level embeddings suitable for subsequent 3D fusion.
- Long-tailed concepts: ConceptFusion remains zero-shot and captures fine-grained concepts such as diet coke, lysol, and yogurt better than finetuned alternatives.The reported comparison attributes the difference to concept forgetting during finetuning of LSeg and OpenSeg.
C. Multimodal querying over 3D feature-fused maps
ConceptFusion fuses features into queryable 3D maps that support text, click, image, and audio inputs, along with composed spatial-relation queries. The representation also supports reasoning over spatial attributes and objects not co-observed in a single image.
- Multimodal querying: A fused 3D map scores each map point against a modality-specific query embedding using cosine similarity.Text, click, image, and audio queries use CLIP text, clicked-point, image-level CLIP, and AudioCLIP embeddings, respectively.
- 3D spatial reasoning: 3D feature fusion enables reasoning about objects that were never co-observed in one image and about spatial attributes available only in 3D.These attributes include relative position, orientation, support, and containment.
- 3D spatial reasoning: Spatial comparator modules compose queries using relation signatures and return scalar or boolean values for relations such as distance, left/right, and support.An optional language-model parser converts natural-language questions into comparator compositions.
V. CASE STUDIES
ConceptFusion is evaluated as an open-set multimodal 3D map across text, image, and audio queries, using UnCoCo and established datasets. It outperforms competing approaches on multimodal object localization.
- Experimental setup: ConceptFusion evaluates open-set 3D maps with text, image, click, and audio queries across multiple datasets and tasks.The benchmark includes real-world and simulated environments, with comparisons to LSeg-3D, OpenSeg-3D, and MaskCLIP-3D.
- Text queries: ConceptFusion accurately localizes objects from long-form text queries, including queries with confounding attributes.On ScanNet, the comparison shows ConceptFusion assigning the highest response to the queried map points even when competing methods are distracted.
- UnCoCo dataset: UnCoCo contains 78 household and office objects, 20 RGB-D sequences, and more than 500,000 multimodal queries.The dataset provides 2D and 3D object masks and supports text, click, image, and audio queries.
- Image queries: ConceptFusion outperforms finetuned foundation models on image queries in both 3D mIoU and detection accuracy.Performance remains broadly consistent across text and image modalities for most approaches.
- Audio queries: ConceptFusion outperforms a privileged AudioCLIP baseline for localizing objects from source-ambiguous and ecological audio queries.Its fused global and local context performs better than purely local features.
B. Open-Set semantic segmentation on other datasets
ConceptFusion is evaluated on semantic segmentation and robotic case studies spanning established datasets, tabletop manipulation, autonomous navigation, and LLM planning. These studies demonstrate open-set queries in varied environments.
- Semantic segmentation: ConceptFusion outperforms the closest zero-shot baseline, MaskCLIP, by a large margin across ScanNet, Replica, and SemanticKITTI.It is also competitive with privileged baselines that use finetuned CLIP models for semantic segmentation.
- Tabletop rearrangement: Zero-shot tabletop rearrangement uses natural-language commands to manipulate previously unseen objects, including branded products and character toys.The experiments use a UR5e manipulator and include distractors that challenge perception and manipulation planning.
- LLM integration: The ConceptFusion-Agent supplies an LLM planner with nearby room and object locations by querying the open-set map.The GenericLLM-Agent fails the specified task without an explicit 3D map representation.
- Autonomous navigation: ConceptFusion supports open-set autonomous navigation by matching a text query to a destination in a fused 3D map.A drive-by-wire vehicle uses the selected location to generate waypoints and navigate autonomously.
D. Ablation analyses
Ablations examine pixel-aligned feature design, 3D lifting, segmentation models, spatial reasoning, and integrations with multilingual foundation models. The results identify design components and settings that affect performance.
- Pixel-alignment design choices: Fusing image-level and region-level CLIP features is key to performance, while the uniqueness term suppresses redundant mask proposals.The full pixel-aligned feature-fusion pipeline achieves the best ablation performance.
- 3D versus 2.5D: Single-image 2.5D mapping performs poorly on spatial queries requiring scene-level context, especially when objects were never co-observed.It performs better on support and containment relations that can largely be inferred from image observations, but fails on room-level containment.
- Segmentation model: Using SAM for class-agnostic instance masks provides a substantial performance boost over Mask2Former.The comparison is reported on the Replica dataset.
- LLM integration: Integrating ConceptFusion with an LLM planner complements task-level planning with natural-language access to 3D perception.The case study uses open-set map queries to provide spatial context for planning.
- Multilingual abilities: ConceptFusion extends to multilingual queries by computing pixel-aligned features from multilingual CLIP models without additional finetuning.The same concept can therefore be queried in several languages.
APPENDIX
The appendix records contributors, funding, implementation details, and the dense 3D feature-fusion procedures used for indoor and outdoor datasets.
- Contributions and support: The appendix credits contributors for system implementation, dataset curation, robotics experiments, LLM integration, and real-time deployment.It also acknowledges project funding and external discussions and feedback.
- 3D feature fusion details: Indoor feature fusion is implemented on dense reconstruction maps using point-based fusion so nearby surface patches share surfels.This reduces map elements and increases the effective number of pixels contributing to each map element.
- 3D feature fusion details: Outdoor sequences are registered into a global frame with LegoLOAM, associating image features with corresponding lidar-derived 3D locations.The procedure projects lidar depths onto the image plane before feature association.
A4. PIXEL-ALIGNED FEATURE EXTRACTION
ConceptFusion extracts pixel-aligned features using class-agnostic instance proposals and foundation-model representations, then supports zero-shot 3D spatial queries through language parsing.
- Class-agnostic instance proposals provide the regions over which ConceptFusion aligns features for pixel-level 3D mapping.
- ConceptFusion uses DINO, CLIP, and related vision-transformer models to obtain semantic features.
- The evaluations focus on foreground objects while excluding specified background classes in indoor and outdoor scenes.
- An LLM parses natural-language spatial queries into primitive 3D spatial comparator function signatures, which are executed directly.
- 100 spatial queries were parsed correctly in the reported evaluation.
Supplementary material for “ConceptFusion:
The supplementary material provides implementation details, feature-fusion procedures, and additional evaluation information across indoor and outdoor mapping settings.
- The supplement includes implementation details, ablations, and additional UnCoCo examples alongside an anonymized webpage with qualitative results and videos.
- Indoor feature fusion is implemented on dense reconstruction with point-based fusion that shares surfels across nearby surface patches.
- Outdoor mapping incrementally registers point clouds globally using LegoLOAM odometry and associates image features with projected lidar map points.
A2. PIXEL-ALIGNED FEATURE EXTRACTION
ConceptFusion uses class-agnostic Mask2Former instance proposals rather than downstream modules trained with instance-segmentation ground truth.
- Mask2Former uses a Swin-L backbone and produces 100 class-agnostic mask proposals per image.
A3. FOUNDATION MODELS USED
The system draws on DINO, CLIP, vision-transformer variants, and AudioCLIP, while evaluating selected foreground objects across indoor, outdoor, simulated, and tabletop settings.
- The foundation-model families used are DINO with associated vision transformers, and CLIP with its variants.
- AudioCLIP is included as a publicly available model trained on AudioSet.
- Evaluations focus on foreground objects and ignore specified background classes for indoor and outdoor scenes.
- The evaluation covers selected ScanNet and Replica scenes, qualitative ICL and AI2-THOR settings, and SemanticKITTI frames containing foreground objects.
- UnCoCo contains 78 objects across 20 tabletop RGB-D sequences, with two sequences reserved for tuning and 18 used for validation.
- Table A.1 lists the objects from the UnCoCo sequences used for evaluation.