Functional-SLAM: Interaction-Aware Mapping with Online Functional Scene Graphs

1 Introduction

• Existing functional-scene-graph methods often rely on known poses, depth, or offline reconstruction, which can incur latency and deployment costs incompatible with immediate interaction during exploration.Source: §1 · p. 1
• Online mapping must handle pose drift and refinement for small interaction elements, while single-frame edge assignments can be ambiguous under occlusion, overlap, and competing object candidates.Source: §1 · pp. 1–2
• Functional-SLAM is presented as the first framework to maintain a functional scene graph as an online SLAM state, constructing stable maps online with competitive accuracy and efficiency over offline methods.The framework recursively fuses perception results into persistent functional nodes and stable edges.Source: §1 · p. 2
• Anchor-keyframe geometry, functional-context-constrained association, and temporal relation posteriors support persistent nodes and stable functional edges.Source: §1 · p. 2
• Stable nodes and functional topology supplement visual loop-closure candidates in repetitive-appearance or degraded-texture scenes, improving pose-estimation accuracy.Source: §1 · p. 2

2 Related Work

• SLAM methods span classical tracking and mapping, dense geometric and learned representations, and semantic, object-level, and open-vocabulary maps, but still lack operable elements and functional relations.Source: §2 · p. 2
• Functional 3D scene graphs represent objects, interaction elements, and their functional relations; later work adds element modeling, hierarchy, probabilistic reasoning, and task-relevant graphs.Source: §2 · p. 3
• Existing functional-graph methods often require known poses or offline reconstruction, limiting their fit for real-time exploration.Source: §2 · p. 3

3.1 Overview

• Functional-SLAM uses MASt3R-SLAM tracking and keyframe selection alongside open-vocabulary observations, recursively fusing them into a persistent graph under optimized poses.The graph represents interaction elements, objects, and their functional relations.Source: §3.1 · p. 3

3.2 Tracking

• MASt3R-SLAM supplies poses, pointmaps, confidence, and keyframe decisions for lifting frame-level functional observations into 3D, while its geometric tracking objective remains unchanged.Open-vocabulary labels and DeepSeek-based reasoning identify perception targets, which SAM3 localizes and segments.Source: §3.2 · pp. 3–4

3.3 Online Functional Graph Mapping

• The mapping stage combines anchor-keyframe geometry with functional-context constraints to associate observations with persistent nodes, and accumulates multi-frame evidence to reduce incorrect relation affiliations.This addresses pose-induced misalignment and single-frame ambiguity from occlusion and local overlap.Source: §3.3 · p. 4

3.3.1 Node Association

• Anchor-keyframe geometry is recovered using the latest pose, while node association combines geometric, semantic, and relation-context cues to match observations with persistent nodes.Association scores use projection overlap, 3D center distance, scale-adaptive gating, label consistency, and functional-affiliation compatibility.Source: §3.3.1 · pp. 4–5

3.3.2 Temporal Functional-Edge Posterior and Graph Commitment

• The system lifts single-frame relation observations to persistent evidence and commits an interaction element’s most reliable relation only after sufficient multi-frame support and consistent recent selections.Each interaction element retains only one relation to avoid conflicting affiliations.Source: §3.3.2 · p. 5

3.4 Functional-Topology-Assisted Loop Closure

• The system combines visual loop-closure candidates with object-level and functional-topology candidates from its online graph, then applies geometric verification and loop-closure optimization.Graphlet-based object and functional scores identify candidate keyframe pairs; surviving pairs are selected by thresholds and top-K ranking.Source: §3.4 · p. 6

4.1 Experimental Setup

• 18 sequences per dataset were selected as continuous segments with valid objects, interaction elements, and relations; evaluation uses ATE RMSE and node/triplet Recall@K.Comparisons include pose-estimation and mapping baselines, plus offline functional-graph methods.Source: §4.1 · pp. 6–7

4.2 Localization

• 16.3% lower ATE RMSE than MASt3R-SLAM on FunGraph3D; SceneFun3D averages 33.25 mm ATE RMSE while preserving geometric-backbone performance.Functional-topology-assisted loop closure supplements visual candidates under repetitive appearance or degraded texture, correcting accumulated drift.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Functional Scene Graph

• On FunGraph3D, Functional-SLAM constructs more accurate, stable online graphs; on SceneFun3D, pose drift limits Ours-pose results against some GT-pose offline baselines.Qualitative comparisons show more complete nodes and more accurate relations, especially for small interaction elements and complex object affiliations; other methods are shown under GT pose.Source: §4.3 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.4 Ablation Study

• Removing node stabilization raises ATE and lowers triplet recall; removing temporal edge posteriors lowers triplet accuracy, while removing functional loop closure degrades localization and graph quality.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 Real-World Scenes

• Handheld iPhone sequences in intrinsics-free monocular mode demonstrate camera tracking and online mapping despite dense tiny elements, severe occlusion, and sparse observations.Figure 4(B) shows the global trajectory, while panels C and D show local details.Source: §4.5 · p. 9, Figure 4 · p. 9

4.6 Runtime Performance

• Online pose baselines run slightly faster but cannot build functional graphs; Functional-SLAM is more efficient than offline functional graph methods.Runtime may be further improved by constructing functional observations only on keyframes.Source: §4.6 · p. 9, Table 4 · p. 9

• Functional-SLAM continuously maintains a functional scene graph as an online map state during SLAM.Source: §5 · p. 9
• The system recursively fuses open-vocabulary perception into persistent functional nodes and stable edges, then uses functional topology to assist loop-closure candidate generation.Source: §5 · p. 9
• Reliable visual tracking and open-vocabulary perception are required; large inter-frame gaps, textureless regions, or severe motion blur may degrade pose estimation and graph construction.Source: §5 · p. 9

A Additional Qualitative Results

• Functional-SLAM constructs more complete and stable graphs with online estimated poses, while offline methods show more missed detections, incorrect associations, or incomplete relations.Under Functional-SLAM-pose, OpenFunGraph often produces few or no nodes and edges, while FunGraph and KeySG can produce cluttered nodes and unstable functional ownership.Source: A Additional Qualitative Results · p. 18
• Anchor-keyframe geometry, context-constrained association, and temporal relation posteriors help preserve node identities and suppress erroneous edges across frames.These mechanisms support structurally consistent graphs despite pose drift or locally unstable geometry.Source: A Additional Qualitative Results · p. 18
• The qualitative advantage extends beyond node or triplet recall to graph interpretability and temporal stability.The comparisons also show offline methods are more prone to missing nodes or edges and confused relations when stable poses and online maintenance are unavailable.Source: A Additional Qualitative Results · pp. 18–19
• Figure 5 compares input images and functional graphs from Functional-SLAM and offline methods across test scenes and pose settings.Functional-SLAM’s graphs are reported as more complete and stable under online estimated poses.Source: A Additional Qualitative Results · p. 18, Figure 5 · p. 19

B Runtime Optimization

• The optimization performs semantic acquisition and graph maintenance on keyframes rather than every frame, and lowers SAM3 processor resolution to reduce inference cost.The full version runs open-vocabulary perception and graph updates on every frame; the optimized version skips these operations on non-keyframes.Source: B Runtime Optimization · p. 19
• 1.027 FPS on FunGraph3D, a 2.45× improvement over the unoptimized version; on 6kitchen/video0, speed rises from 0.377 FPS to 0.920 FPS.Keyframe-only perception and graph updates reduce repeated semantic processing and association work; SAM3 resolution is reduced from 1008 to 672.Source: B Runtime Optimization · pp. 19–20
• 0.92 FPS versus 0.02, 0.06, and 0.05 FPS for OpenFunGraph, FunGraph, and KeySG, respectively, with Ours w/ RT retaining higher overall node and triplet recall under the same Ours-pose setting.On FunGraph3D, Ours w/ RT still outperforms most offline results even when they use GT poses.Source: B Runtime Optimization · p. 20, Table 7 · p. 21
• ATE RMSE increases from 16.495 mm to 17.579 mm after runtime optimization, while overall node and triplet recall remain higher than offline methods under the same Ours-pose setting.The paper describes the accuracy loss as controllable and reports that Ours w/ RT remains more efficient than the offline methods.Source: B Runtime Optimization · p. 20

C Manipulation Task Examples

• The system uses a wrist-mounted RGB-D camera for online perception and graph construction, with a calibration board to recover scene scale for executable robot coordinates.A language-model-based query interface retrieves object and interaction-element nodes, their functional links, and corresponding 3D poses for robot actions.Source: C Manipulation Task Examples · p. 21
• In the query–explore–execute strategy, the robot executes directly when the graph contains the required object, interaction element, and relation chain; otherwise it explores and updates the graph.Execution begins once the required nodes and relations appear, rather than waiting for complete offline reconstruction.Source: C Manipulation Task Examples · pp. 21–22
• The robot explores until the kettle and handle relation are available, then uses the graph for kettle lifting and subsequent drawer, jar-lid, and cup tasks.Figure 6 shows the online graph overlaid on the map and the multi-step manipulation examples.Source: C Manipulation Task Examples · p. 21, Figure 6 · p. 22

D Failure Cases and Limitations

• Large camera-pose drift can shift back-projected geometry and destabilize associations, particularly for small, sparsely observed interaction elements.Under obvious drift, unstable reconstruction, and sparse observations, the system may produce association failures, duplicate nodes, or incorrect functional edges.Source: D Failure Cases and Limitations · p. 23
• Triplet recall can remain low even when relation semantics are predicted stably, because matching requires both endpoint nodes and the relation to be correct.The paper identifies lower node-association recall as a bottleneck, particularly for small interaction elements and sequences with severe pose drift.Source: D Failure Cases and Limitations · p. 23
• The system targets static or quasi-static exploration and does not explicitly update graph state when manipulation changes objects or their functional relations.Examples include opening doors or drawers, changing switches, and relocating movable objects.Source: D Failure Cases and Limitations · p. 23

E Supplementary Evaluation with a Unified SAM3 Front End

• Under Ours-pose, Functional-SLAM achieves the best overall node and triplet recall on both SceneFun3D and FunGraph3D after baseline front ends are unified with SAM3.The result supports the claim that its advantage mainly comes from online graph maintenance, including geometry synchronization, persistent node maintenance, and temporal relation stabilization. Replacing front ends may instead reduce recall because downstream graph pipelines depend on their original output distributions; pose errors can make tighter SAM3 masks especially problematic for small interaction elements.Source: E Supplementary Evaluation with a… · pp. 24–25, Table 8 · p. 25, Table 9 · p. 25

F.1 Functional Graph Representation

• Functional-SLAM represents the graph as G = (U, O, R) for readout, while internally maintaining an O-C-U hierarchy that preserves functional carriers between objects and interaction units.The carrier layer provides finer structural cues for functional-topology-assisted loop closure, and can be collapsed to object–interaction-unit relations for evaluation.Source: §F.1 · pp. 25–26

F.2 Functional-Context-Constrained Node Association

• Node association matches observations only to persistent nodes of the same role and uses geometry, semantics, and functional-parent context to reduce cross-instance confusion among similar interaction elements.The system solves one-to-one matching with the Hungarian algorithm; parent consistency is especially useful for distinguishing interaction units associated with adjacent doors, drawers, or devices.Source: §F.2 · pp. 26–28

F.3 Functional-Topology-Assisted Loop Closure

• Functional topology supplements visual loop candidates using local node graphlets, with candidate rankings enhanced when matched structures preserve functional-edge endpoints, types, and directions.Topology proposes candidates rather than validating loops: candidates must still pass geometric verification before becoming back-end constraints.Source: §F.3 · pp. 28–30

F.4 Remote Relations

• Remote functional relations between spatially separated objects, such as switches controlling lights, are first treated as label-level candidates and promoted to instance-level relations using node association and multi-frame evidence.Because these non-contact relations are not reliably inferred from overlap, containment, or proximity alone, the system handles them separately from local relations.Source: §F.4 · p. 30
• Open-vocabulary label-level candidates are associated with persistent source–target nodes and scored using detection confidence, co-visibility, geometric maturity, and relation-specific distance plausibility.Detection and confidence provide the strongest current-frame evidence; co-visibility and maturity strengthen support without acting as decisive requirements.Source: §F.4 · p. 30
• Geometric maturity favors anchor-fused geometry over stable anchors, candidate anchors, and world-coordinate-only estimates, while distance priors vary by relation type.Contact and installation relations prefer nearby endpoints; control and power-supply relations allow greater separation.Source: §F.4 · p. 31
• Instance-pair candidates compete in a temporal posterior, and an edge is committed only after continuous support across frames and stability over a recent window.Insufficient evidence leaves the relation tentative rather than committing it as a stable edge.Source: §F.4 · p. 31
• Unresolved early evidence is cached until uniquely associable, while template-recalled candidates remain weak priors subject to maturity, recent co-visibility, spatial checks, and temporal accumulation.Evidence with multiple possible endpoint pairs remains unresolved and is not written into the graph.Source: §F.4 · p. 31

F.5 Parameter Sensitivity Analysis

• Maximum variations were 5.3% for average ATE RMSE, 6.8% for Overall R@* in Node Evaluation, and 4.7% for Overall R@* in Triplet Evaluation.The analysis perturbed main hyperparameters individually by ±20% on FunGraph3D.Source: §F.5 · p. 31
• Robustness combines multi-cue node association and multi-frame edge commitment, while MASt3R geometric verification checks supplementary loop-closure candidates.The authors attribute performance gains primarily to online graph maintenance and functional topology, rather than reliance on specific hyperparameters.Source: §F.5 · pp. 31–32

F.6.1 Baseline Output Adaptation.

• Baseline adaptation changes only evaluation representations, not predicted nodes, positions, or edges, and does not use ground-truth relation labels.This conservative conversion addresses differences between methods’ graph representations and the benchmark protocol.Source: §F.6.1 · p. 32
• For KeySG, an LLM maps predicted functional-part and parent-object labels into benchmark-style functional relation text without changing nodes or edge endpoints.KeySG’s structural ownership edges otherwise differ from the benchmark’s interactive relation semantics.Source: §F.6.1 · p. 32
• For FunGraph, evaluation retains small-part–object edges, converts them to functional triplets, and maps predicted affordances to benchmark labels and relation descriptions.The conversions use FunGraph’s own outputs rather than ground-truth relation labels.Source: §F.6.1 · pp. 32–33

F.6.2 Relation Evaluation Protocol.

• Because benchmark relations are natural-language descriptions, wording and granularity differences can make semantically similar predictions miss fixed-vocabulary top-K retrieval.The examples include differing object complements and levels of description detail.Source: §F.6.2 · p. 33
• Relations count as semantically matched when top-K retrieval succeeds or predicted-to-ground-truth BERT cosine similarity is at least 0.70.The auxiliary criterion applies only after endpoint nodes have matched, leaving node, geometry, and association requirements unchanged.Source: §F.6.2 · p. 33

F.7 Prompts

• A scene-classification prompt infers kitchen, bathroom, livingroom, bedroom, or unknown from detected tags to select later prompts.The inferred type informs the scene-level atlas and incremental reasoning prompts.Source: §F.7 · p. 33
• The scene-level atlas organizes objects, functional carriers, interaction units, local chains, and remote dependencies, with relation descriptions supplying structured priors.O, C, and U denote objects, intermediate interaction-carrying structures, and directly operable units; remote relations capture dependencies such as a switch controlling a light.Source: §F.7 · pp. 33–34
• The frame-level prompt filters current tags before generating local structures and remote candidates, excluding generic categories and scene-conflicting tags.Remote endpoints are restricted to a whitelist from the atlas.Source: §F.7 · p. 34