Source-linked AI summary
Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces
Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, Francis Engelmann
TL;DR
Traditional 3D scene graphs do not represent small interactive elements and functional relationships needed for higher-level scene reasoning. OpenFunGraph uses foundation models to construct open-vocabulary functional 3D scene graphs, and experiments on SceneFun3D and FunGraph3D show significant improvements over adapted baselines with demonstrated downstream applications.
Problem
Traditional 3D scene graphs are restricted mainly to object nodes and spatial relationships, while functional relationships and interactive elements remain insufficiently represented.
Method
OpenFunGraph leverages VLMs and LLMs to construct functional 3D scene graphs without task-specific training, modeling objects, interactive elements, and their functional relationships.
Results
OpenFunGraph significantly outperforms adapted Open3DSG and ConceptGraph baselines on the extended SceneFun3D and FunGraph3D datasets.
Takeaways & Limitations
Functional 3D scene graphs support flexible querying and downstream 3D question answering and robotic manipulation.
Abstract
from arXiv · showhide
We introduce the task of predicting functional 3D scene graphs for real-world indoor environments from posed RGB-D images. Unlike traditional 3D scene graphs that focus on spatial relationships of objects, functional 3D scene graphs capture objects, interactive elements, and their functional relationships. Due to the lack of training data, we leverage foundation models, including visual language models (VLMs) and large language models (LLMs), to encode functional knowledge. We evaluate our approach on an extended SceneFun3D dataset and a newly collected dataset, FunGraph3D, both annotated with functional 3D scene graphs. Our method significantly outperforms adapted baselines, including Open3DSG and ConceptGraph, demonstrating its effectiveness in modeling complex scene functionalities. We also demonstrate downstream applications such as 3D question answering and robotic manipulation using functional 3D scene graphs. See our project page at https://openfungraph.github.io
1. Introduction
The paper extends 3D scene graphs from static spatial relationships to functional representations of objects, interactive elements, and their relationships. OpenFunGraph uses foundation models to address limited training data, evaluates on two datasets, and outperforms adapted baselines.
- Motivation: Traditional 3D scene graphs primarily represent objects and spatial relationships, limiting finer-grained functional reasoning.They omit small interactive elements and relationships such as a switch controlling a light.
- Approach: OpenFunGraph leverages VLMs and LLMs to infer functional scene graphs without task-specific training data.The approach uses foundation-model knowledge to address the scarcity of annotations for complex functional relationships.
- Core idea: Functional 3D scene graphs represent objects, interactive elements, and functional relationships in a unified structure.The representation extends traditional scene graphs with interactive sub-parts and non-spatial relationships.
- Evaluation: The evaluation uses an extended SceneFun3D dataset and the newly collected FunGraph3D dataset with functional annotations.FunGraph3D contains high-precision 3D laser scans and accurately registered data.
- Results and applications: OpenFunGraph outperforms adapted Open3DSG and ConceptGraph baselines and supports downstream 3D question answering and robotic manipulation.These results and applications demonstrate the representation’s use for functional scene understanding.
2. Related Work
Prior work advances 3D scene understanding, open-vocabulary perception, affordance prediction, and scene graphs, but largely remains object-centered or spatial. OpenFunGraph instead targets scene-level interactive elements and functional relationships through an open-vocabulary pipeline.
- 3D indoor scene understanding: 3D indoor scene understanding has progressed from closed-set segmentation to open-vocabulary perception and visual-language grounding.These directions primarily address object-level recognition and grounding.
- Affordance understanding: Affordance methods predict interaction properties from images, videos, or 3D representations, often at the object level.OpenFunGraph extends this focus to scene-level interactive elements and remote operations.
- 3D scene graphs: Traditional 3D scene graphs organize indoor entities and model inter-object relationships, whereas functional graphs add interactive-element nodes and functional edges.This distinction separates spatial scene organization from functional interaction modeling.
- OpenFunGraph: OpenFunGraph detects objects and interactive elements, describes nodes with language, and infers functional relationships using foundation models.Its pipeline combines RAM++, GroundingDINO, GPT, and LLAVA for these stages.
- Comparison: Compared with Open3DSG and ConceptGraph, OpenFunGraph uses adaptive detection and description for interactive elements plus sequential functional reasoning.The baselines emphasize object nodes or spatial relationships and are less suited to broad functional modeling.
3. Problem Formulation
A functional 3D scene graph is a directed graph that jointly represents scene objects, interactive elements, and functional relationships between them. The task is to construct this graph from posed RGB-D observations of an unseen indoor environment.
- Graph definition: A functional 3D scene graph is defined as G = (O, I, R), with objects O, interactive elements I, and functional relationships R.Relationships point from interactive-element nodes to object nodes.
- Interactive elements: Interactive elements are components that agents interact with to trigger functions, including handles, knobs, and buttons.Examples include opening a cabinet or turning off a light.
- Functional relationships: Functional relationships are either local connections within an object or remote relationships in which an element operates an object from a distance.Examples are a door-handle connection and a TV-remote-control connection.
- Task definition: Given posed RGB-D frames from an unseen indoor environment, the task is to infer the functional relationships among detected objects and interactive elements.The output is the corresponding functional 3D scene graph.
4. Method
OpenFunGraph builds functional 3D scene graphs through foundation-model-based node detection, language description, and sequential functional-edge reasoning. It fuses multi-view observations, handles small interactive parts with targeted prompts and context, and combines local with confidence-aware remote relationships.
- Pipeline: OpenFunGraph detects objects and interactive elements, describes each node in language, and infers functional relationships between them.The method uses foundation-model knowledge to operate in an open-vocabulary setting with limited training data.
- Node Candidate Detection: Multi-view node construction projects 2D detections into 3D and fuses them using geometric consistency.Each candidate stores a 3D point cloud, bounding box, image assets, and confidence information.
- Node Candidate Detection: Object candidates are generated by using RAM++ object tags to prompt GroundingDINO for 2D boxes, masks, and confidence scores.The resulting detections form the object side of the graph’s node candidates.
- Node Candidate Detection: GPT-4 proposes interactive-element tags for valid object candidates, which are combined into prompts such as “door. handle” for GroundingDINO.This strategy targets small functional parts and empirically improves their detection accuracy.
- Node Candidate Description: Node descriptions combine selected multi-view evidence, LLAVA captions, and GPT-4 summarization into unified natural-language descriptions.Views are selected using semantic confidence and geometric contribution, while interactive-element boxes are enlarged to provide context.
- Functional Edges: Sequential reasoning first infers local rigid connections and then constructs remote functional relationships.Local reasoning uses spatial overlap, descriptions, and 3D boxes; remote reasoning assigns confidence scores so agents can prioritize higher-confidence interactions.
- Functional Edges: The final predicted graph combines the nodes and relationships produced by the local and remote reasoning stages.This yields a unified functional scene graph for the input environment.
5. Data Collection
FunGraph3D combines high-fidelity 3D scans, RGB-D video, egocentric interaction videos, and functional scene-graph annotations to capture relationships that static observations cannot fully reveal. The dataset covers diverse real-world indoor scenes with open-vocabulary objects, interactive elements, and functional relationships.
- Dataset modalities: FunGraph3D combines high-fidelity 3D scans, RGB-D video sequences, egocentric human-scene interaction videos, and functional 3D scene-graph annotations.These modalities support both geometric reconstruction and functional relationship labeling.
- Data acquisition: Laser scanning captures fine geometric details, including small interactive elements such as knobs and buttons, at 5 mm resolution.Multiple scanner positions are used to achieve high scene coverage.
- Dataset modalities: Egocentric videos help resolve ambiguous functional connections, such as which light switch controls a ceiling light.Physical interactions provide evidence that static 3D observations may not reveal.
- Annotation: Annotators label object and interactive-element instances, assign free-form labels, and connect each interactive element to the object it controls with a relationship description.The process extends the SceneFun3D annotation tool and supports construction of ground-truth functional 3D scene graphs.
- Dataset statistics: FunGraph3D contains 14 in-the-wild scenes: 6 kitchens, 2 living rooms, 3 bedrooms, and 3 bathrooms.The dataset includes 201 interactive elements, 228 functional relationships, and 146 objects of interest with open-vocabulary labels and relationships.
6. Experiments
Experiments evaluate OpenFunGraph on SceneFun3D and FunGraph3D using node and triplet metrics, compare adapted ConceptGraph and Open3DSG baselines, and test downstream applications. OpenFunGraph achieves higher node and triplet recall, while ablations show benefits from interactive-element detection, sequential reasoning, and confidence-aware remote reasoning.
- Experimental Setup: Evaluation uses an extended SceneFun3D dataset, FunGraph3D, and a functional scene-graph metric extending Open3DSG Recall@K with node detection.The metric separately evaluates object and interactive-element recall and requires all triplet components to be retrieved for triplet success.
- Node Evaluation: OpenFunGraph surpasses ConceptGraph* by 160% on SceneFun3D and 176% in R@3 on FunGraph3D for node evaluation.With IED, ConceptGraph* improves node recognition but remains 22% and 43% below OpenFunGraph in R@3 on the two datasets.
- Node Evaluation: OpenFunGraph achieves 95% and 29% higher R@3 than Open3DSG*† and Open3DSG* on SceneFun3D, and 174% and 66% higher on FunGraph3D.The comparison reflects stronger recognition of interactive elements through open-vocabulary inference and the specified node description stage.
- Triplet Evaluation: OpenFunGraph outperforms ConceptGraph* + IED by 76% and 189% in R@5, and Open3DSG*† by 179% and 308% on SceneFun3D and FunGraph3D.The authors attribute these triplet results to accurate node recognition and sequential reasoning for functional inference.
- Ablation Studies: Removing assistive object tags reduces node R@3 by 19% and 10%, while direct all-node reasoning reduces triplet R@5 by 42% and 32%.Randomly selecting remote connections further decreases triplet R@5 by 7% and 11% on SceneFun3D and FunGraph3D, respectively.
- Downstream Applications: Functional 3D scene graphs support JSON-based 3D inventory question answering and robotic manipulation guided by queried interactive elements.The robot locates the referenced element through the graph and then navigates to interact with it.
7. Conclusion
The paper introduces Functional 3D Scene Graphs to jointly represent objects, interactive elements, and their functional relationships in 3D indoor environments. It also presents an open-vocabulary inference pipeline, a high-fidelity annotated dataset, and downstream demonstrations in question answering and robotic manipulation.
- Functional 3D Scene Graphs jointly model objects, interactive elements, and their functional relationships in 3D indoor environments.
- The open-vocabulary pipeline leverages foundation-model common-sense knowledge to infer functional 3D scene graphs and support flexible querying.
- The paper develops a high-fidelity dataset of real-world 3D indoor environments with multimodal data and functional annotations for systematic benchmarking.
- Experiments on the proposed and existing datasets significantly outperform baselines, while downstream demonstrations cover 3D question answering and robotic manipulation.