Functionality understanding and segmentation in 3D scenes

Contribution

• Fun3DU is the first approach specifically designed for functionality understanding in 3D scenes.It targets functional objects such as handles, knobs, and buttons, including objects not explicitly named in task descriptions.Source: §1 · pp. 1–2

Evaluation

• Fun3DU is training-free and evaluated on SceneFun3D, which contains 230 scenes and over 3000 task descriptions.The benchmark includes challenging cases involving similar appearances, ambiguous descriptions, and world knowledge.Source: §1 · pp. 1–2

Method

• Fun3DU parses task descriptions with Chain-of-Thought reasoning to identify contextual and functional objects before segmentation.The method uses a language model to interpret functionality and context, including cases where the functional object is implicit.Source: §1 · pp. 1–2
• The method segments contextual objects to select informative views, segments functional objects in those views with a 2D VLM, and aggregates masks into a 3D point cloud.Multi-view masks are lifted through point-to-pixel correspondences and combined using multi-view agreement.Source: §1 · p. 2

Results

• +13.2 mIoU on average over open-vocabulary 3D segmentation baselines demonstrates Fun3DU’s stronger functionality-segmentation performance.The comparison uses adapted open-vocabulary 3D segmentation methods as baselines.Source: §1 · p. 2

Functionality segmentation

• SceneFun3D benchmarks fine-grained functional-element segmentation in indoor scenes using 2D images and high-resolution 3D point clouds.Its task descriptions may omit the object to segment and require world knowledge, making it relevant to embodied-AI applications.Source: §2 · p. 2

Open-vocabulary 3D segmentation

• Existing open-vocabulary 3D methods often segment common household furniture instead of small functional objects and remain insufficient even after task-specific fine-tuning.OpenMask3D and LERF perform poorly on functionality segmentation, while fine-tuned OpenMask3D improves only slightly.Source: §2 · p. 2
• Open-vocabulary 3D segmentation methods typically fuse 2D multi-view masks with 3D point-cloud proposals or learnable pooling.Their 3D components are often biased toward large objects, which contributes to difficulty with small object parts.Source: §2 · p. 2

Fun3DU

• Fun3DU uses Chain-of-Thought reasoning for complex descriptions and performs functional-element segmentation in 2D before aggregating results onto the point cloud.This design avoids relying solely on 3D models biased toward large household objects.Source: §2 · pp. 2–3
• Fun3DU’s four-module pipeline interprets tasks, selects views using contextual-object masks, segments functional objects with a VLM, and lifts masks into 3D.The view-selection stage discards views where the contextual object is absent or occluded.Source: Figure 2 · p. 3

3.1. Overview

• Fun3DU takes a point cloud, posed scene images, and a task description, then outputs a 3D mask of the functional object.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Problem formulation

• The problem distinguishes the functional object to segment from its contextual object, which contains or relates to it in the scene.Source: §3.2 · p. 3

3.3. Task description understanding

• The LLM uses task decomposition, contextual-object information, a predefined action space, and Chain-of-Thought reasoning to resolve omitted or ambiguous functional objects.The method addresses incorrect abstraction levels and context hallucinations by imposing a stopping criterion and selecting the first object in the inferred hierarchy as context.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4. Multi-view functionality understanding

• Fun3DU selects views where the contextual object is present and visible, then prompts a VLM with the task and functional object to locate points for fine-grained masks.Visibility scoring combines mask coverage, confidence, centrality, and angular distribution before retaining the highest-scoring views.Source: §3.4 · pp. 4–5

3.5. Point cloud functionality segmentation

• The method lifts functional masks into 3D, counts multi-view pixel-to-point agreements, normalizes point scores, and thresholds them to produce the final mask.This aggregation is intended to mitigate false positives from inaccurate individual 2D masks.Source: §3.5 · p. 5

4.1. Experimental setup

• SceneFun3D contains 30 split0 scenes and 200 split1 scenes, averaging 1800 RGBD views and 15 task descriptions per scene, with AP, AR, and mIoU evaluation.Ground-truth masks can include multiple functional objects corresponding to one task, such as two handles on a drawer.Source: §4.1 · pp. 5–6

4.2. Quantitative results

• Fun3DU is the only method exceeding 0.4 AP25 and 0.2 mIoU on both SceneFun3D splits, outperforming the closest competitor by 32.9 and 23.1 AP25 points and 15 and 11.4 mIoU points on split0 and split1.Baselines show high recall but near-zero precision, reflecting undersegmentation and difficulty capturing fine-grained functional objects.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.3. Qualitative results

• Qualitative examples show baselines selecting contextual objects instead of fine-grained targets, whereas Fun3DU usually segments the intended functional objects but still produces occasional spurious or incorrect masks.Errors arise from imprecise VLM points, small or similarly textured objects, and visual ambiguity between nearby objects.Source: §4.3 · pp. 6–7, Figure 5 · p. 7

4.4. Ablation studies

• Replacing VLM functional-object segmentation with open-vocabulary segmentation drops AP25 by 19.4 and AR25 by 17.3, while simplifying the task description drops AP25 by 14.4 and AR25 by 12.3.The ablations attribute these losses to weaker reasoning and removal of contextual information needed to distinguish similar objects.Source: §4.4 · p. 8, Table 3 · p. 8
• Randomly sampling 100 input images instead of the default 50, rather than using view selection, drops performance relative to the standard method by 12.8 AP25 and 4.9 AR25, showing view quality outweighs quantity.Combining detection confidence with contextual-mask distance and angle obtains optimal view-selection performance.Source: §4.4 · p. 8
• Increasing the number of selected views improves multi-view consensus and removes spurious predictions, while 10 or fewer views retain nonzero performance and more than 50 views provide only marginal gains.With τ = 0.7, 10 views produce mIoU above 10 and 4 or fewer views produce mIoU above 7; the standard setting remains 50 views for computational trade-offs.Source: §4.4 · p. 8, Figure 6 · p. 8

• Fun3DU is a training-free functionality-segmentation method built from a large language model and a pretrained vision-language model with world knowledge.Source: §5 · p. 8
• Fun3DU can fail when the contextual object’s location is insufficient to identify the functional object, especially in ambiguous environments.The proposed extension adds other objects mentioned in the task description as priors, targeting ambiguous multi-room cases.Source: §5 · pp. 8–9

• The supplementary material covers functionality-segmentation methods, deeper ablation studies, additional qualitative results, and hardware and implementation details.Source: Supplementary Material Functionality… · p. 11

1.1. Methods details

• OpenMask3D combines 3D mask proposals with projected-view SAM masks and CLIP features, then retrieves masks using text-feature similarity.The reproduced setup uses the original implementation and ScanNet200 checkpoint without retraining.Source: §1.1 · p. 11
• OpenIns3D transfers labels from 2D localized masks to 3D through its Mask, Snap, and Lookup modules, while the study instead uses SceneFun3D frames in its baseline.The standard configuration uses Yolo-world with CLIP-based ranking and filtering.Source: §1.1 · p. 11, §1.2 · p. 12
• LERF lifts CLIP-aligned pixel relevancy maps from neural-radiance-field views into 3D, using task descriptions to obtain final point-cloud masks.Models are trained per sequence when camera matching provides sufficient valid views.Source: §1.1 · p. 12

1.2. Additional results

• Concatenating contextual and functional object names reduces OpenMask3D AP25 by 0.2 and AR25 by 7.2, and OpenIns3D AR25 by 12.8.LERF remains on par, while the results suggest baseline text encoders can use the richer original task context.Source: Table 1 · p. 11, §1.2 · p. 12
• OpenIns3D’s Snap rendering module causes AR25 to drop by 46.6 points because its views emphasize common furniture rather than small functional objects.SceneFun3D’s original frames often provide close-up views that better support functional-object localization.Source: Table 2 · p. 11, §1.2 · p. 12

2. Extended ablation studies

• The extended study evaluates mIoU across different view counts and thresholds while also reporting average processing time for view retrieval, functional segmentation, and multi-view agreement.Task-description understanding and contextual-object segmentation are excluded from the timing because they are constant.Source: §2 · p. 12, Table 3 · p. 12

2.1. Hyperparameter sensitivity analysis

• A 100-view setting improves mIoU by 0.6 but nearly doubles processing time from 118.4 to 204.6 seconds, motivating the 50-view standard.The study treats 50 views as the performance–runtime trade-off.Source: §2.1 · p. 12, Table 3 · p. 12
• Sampling 200 views lowers mIoU by 1.3 relative to the default τ = 0.7 because additional views make spurious masks easier to retrieve.The authors therefore retain 50 sampled views in the standard setting.Source: §2.1 · p. 12

3.1. Task description understanding

• The LLM usually produces useful object hierarchies for view selection, but can reverse the hierarchy or choose the ceiling light instead of the wall-mounted light switch.Small wording changes such as “left” versus “right” can also produce substantially different interpretations.Source: §3.1 · p. 13

3.2. Score-based view selection

• Ranking views with detection, distance, and angle scores avoids selecting views that look confident but are distant, occluded, or poorly framed.Weighting all three scores can retain correct masks that confidence-only ranking would discard.Source: §3.2 · p. 13

3.3. Functional object segmentation

• Molmo supplies points and SAM converts them into functional-object masks, while task examples include drawer handles, telephone dials, and remote-control power buttons.The qualitative pipeline uses language-derived object hierarchies to locate the relevant scene context before segmentation.Source: §3.3 · pp. 13–14
• The qualitative cases illustrate that the method’s segmentation target is the interaction-relevant part, such as a drawer handle, dial pad, or power button, rather than necessarily the whole object.The task-solving sequences explicitly distinguish contextual objects from acted-on parts.Source: §3.3 · p. 14
• Multi-view agreement removes spurious radiator-part masks by accumulating 2D masks and thresholding their 3D agreement.This aggregation supports robust separation when several visually similar parts resemble the functional dial.Source: §3.3 · p. 14
• The VLM can identify a heavily occluded remote and recognize a visually variable humidifier button, after which SAM produces the corresponding masks.These examples extend beyond small knobs and buttons to occluded and appearance-diverse objects.Source: §3.3 · p. 14

3.4. Point cloud segmentation

• Fun3DU is the only method to accurately segment the light switch in one split0 example, while other baselines often segment contextual objects or miss relevant points.Across the remaining examples, Fun3DU achieves precision above 70 and recall above 65, though its light-switch mask also includes the surrounding panel.Source: §3.4 · p. 14
• On harder split1 scenes, Fun3DU locates small handles and valves that baselines miss, but can confuse a lock with a nearby handle or segment an entire phone instead of its dial pad.For an exercise-bike task, segmenting both knobs still yields mIoU 44.12.Source: §3.4 · pp. 14–15

4. Implementation details

• The implementation uses public HuggingFace models for OWLv2, SAM, and Molmo and runs experiments on an NVIDIA A100 GPU.The method is implemented in PyTorch.Source: §4 · p. 15