PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Functionality Segmentation

Jiaying Lin, Dan Xu

arXiv:2603.23478v1cs.CV

TL;DR

3D functionality segmentation must ground implicit natural-language instructions in precise masks of fine-grained interactive elements, but existing pipelines suffer from visual blindness and passive, single-scale frame selection. UniFunc3D uses a unified training-free MLLM with active spatial-temporal grounding and coarse-to-fine perception. On SceneFun3D, it achieves state-of-the-art performance across metrics and surpasses both training-free and training-based methods by a large margin.

  • Problem

    Functionality segmentation requires grounding implicit instructions to fine-grained interactive parts, while existing fragmented methods lack active spatial-temporal reasoning and can suffer cascading errors.

  • Method

    UniFunc3D uses a unified training-free MLLM that jointly performs semantic, temporal, and spatial reasoning through active coarse-to-fine grounding and multi-view verification.

  • Results

    UniFunc3D achieves state-of-the-art performance across all SceneFun3D metrics, surpassing both training-free and training-based methods by a large margin.

  • Takeaways & Limitations

    The unified active-observer design addresses visual blindness and cascading errors while preserving global context for fine-grained spatial reasoning.

  • Takeaways & Limitations

    Challenges remain for extremely small functional parts occupying less than 0.1% of image area and for scenes with severe occlusion.

Abstract

from arXiv · show

Functionality segmentation in 3D scenes requires an agent to ground implicit natural-language instructions into precise masks of fine-grained interactive elements. Existing methods rely on fragmented pipelines that suffer from visual blindness during initial task parsing. We observe that these methods are limited by single-scale, passive and heuristic frame selection. We present UniFunc3D, a unified and training-free framework that treats the multimodal large language model as an active observer. By consolidating semantic, temporal, and spatial reasoning into a single forward pass, UniFunc3D performs joint reasoning to ground task decomposition in direct visual evidence. Our approach introduces active spatial-temporal grounding with a coarse-to-fine strategy. This allows the model to select correct video frames adaptively and focus on high-detail interactive parts while preserving the global context necessary for disambiguation. On SceneFun3D, UniFunc3D achieves state-of-the-art performance, surpassing both training-free and training-based methods by a large margin with a relative 59.9\% mIoU improvement, without any task-specific training. Code will be released on our project page: https://jiaying.link/unifunc3d.

1. Introduction

3D functionality segmentation must infer implicit interactive targets from natural-language instructions while resolving fine-grained spatial and temporal evidence. UniFunc3D replaces fragmented, passive processing with unified active multimodal reasoning and coarse-to-fine grounding.

  • Functionality segmentation identifies how to interact with objects, such as locating a wall switch for “turn on the ceiling light,” rather than merely naming objects.
  • Existing pipelines decompose instructions with a text-only LLM, so incorrect contextual or functional-object identification can propagate to the final mask.Fun3DU may identify the device instead of the socket for an instruction involving a device and a left socket.
  • Passive heuristic frame selection processes isolated single frames, causing spatial-temporal inconsistencies when target and contextual objects are not visible together.These methods do not aggregate temporal information across views for spatial disambiguation or visibility.
  • Single-scale processing can miss small functional parts because the model cannot adaptively focus on important frames at higher resolution.The paper describes these parts as occupying tiny regions that may appear as imperceptible noise.
  • UniFunc3D consolidates semantic, temporal, and spatial reasoning in one training-free MLLM that adaptively selects informative video content using direct visual evidence.This design targets the perception-reasoning gap and seeks to prevent cascading errors from blind reasoning or heuristic frame selection.
  • Its coarse-to-fine strategy surveys video at low resolution, then processes a dense high-resolution temporal window to resolve small parts while retaining global context.The approach includes active spatial-temporal grounding and visual mask verification.
  • UniFunc3D achieves state-of-the-art SceneFun3D performance, surpassing training-free and training-based methods without task-specific training.The reported contribution describes this result as a large-margin improvement.

2. Related Work

3D functionality segmentation extends affordance understanding from object-centric recognition to fine-grained functional elements implicitly referenced by task descriptions. UniFunc3D addresses limitations in prior training-free and training-based approaches with unified visual-temporal-spatial reasoning.

  • SceneFun3D requires segmenting fine-grained elements such as handles, knobs, and switches from natural-language descriptions that may not name them explicitly.Its 230 high-resolution indoor scenes contain over 3,000 task descriptions requiring world knowledge and spatial reasoning.
  • Fun3DU uses a four-stage training-free pipeline beginning with text-only reasoning to identify contextual and functional objects.The cited passage describes this as the first dedicated method for the SceneFun3D benchmark.
  • Fun3DU reasons without visual input initially and processes frames independently, limiting handling of ambiguous cases and temporal spatial disambiguation.
  • Training-based methods such as TASA and AffordBot require task-specific data, point clouds, and ground-truth point-cloud annotations for training.The passage also states that they lack generalization to unseen domains.
  • UniFunc3D jointly performs visual reasoning, temporal grounding, and spatial localization in a single forward pass while remaining fully training-free.This is presented as eliminating visual blindness and information loss across pipeline stages.
  • Open-vocabulary 3D segmentation methods commonly combine 3D proposal modules with 2D multi-view RGB modules, but struggle with small functional parts.The cited passage attributes this difficulty to pretraining bias toward large furniture rather than small functional elements.

3. Method

UniFunc3D unifies visually grounded reasoning, active spatial-temporal grounding, mask verification, and multi-view 3D lifting to segment functional objects from natural-language tasks. Its coarse-to-fine process selects informative frames at low resolution, refines localization in dense high-resolution windows, and preserves scene context for disambiguation.

  • 3.1. Problem Formulation: Functionality segmentation infers the interactive object and specific part from task intent, then outputs a 3D mask of the functional object.The target may be implicit, such as identifying a wall switch for “turn on the ceiling light,” and may require resolving among similar instances using spatial constraints.
  • 3.3. Unified MLLM with Visually Grounded Reasoning: UniFunc3D uses one MLLM to jointly identify the functional object, select informative frames, and ground affordance points from task descriptions and video.This replaces the separation between text-only reasoning and independent visual perception with a single visually grounded reasoning chain.
  • 3.4. Active Spatial-Temporal Grounding: Round 1 surveys low-resolution frames across offset-based samples, jointly predicts the functional object and affordance points, and retains valid candidate frames for refinement.Multiple temporal offsets provide complementary slices, while the model directly selects the key frame rather than using handcrafted detection scores.
  • 3.5–3.6. Mask Verification and 3D Lifting: SAM3 generates candidate masks from predicted points, and the same MLLM verifies overlay masks before verified masks undergo multi-view agreement and 3D lifting.Verification checks that the highlighted region is the target functional object and excludes parent or containing objects; multi-view consensus suppresses outliers before producing the final point-cloud mask.
  • 3.4. Active Spatial-Temporal Grounding: Round 2 extracts dense temporal windows around candidate frames at native high resolution, enabling fine localization while retaining complete scene context.The resolution transition provides a zoom-in mechanism without external cropping, and multi-view temporal evidence helps resolve missing context and partially visible targets.

4. Experiments

Experiments on SceneFun3D show that UniFunc3D outperforms existing methods across reported metrics, while ablations support its two-stage, multi-sampling, and temporal-window design.

  • Main Results: UniFunc3D achieves the best performance across all metrics on both SceneFun3D splits, outperforming training-free and training-based methods.
  • Main Results: 14.34 AP50 points and 17.71 AP25 points improve over Fun3DU on split0 for UniFunc3D-30B, with a 9.1-point mIoU gain.These correspond to relative gains of 84.9%, 53.2%, and 59.9%, respectively.
  • Qualitative Results: UniFunc3D resolves spatial disambiguation and small functional parts more accurately than AffordBot and Fun3DU in qualitative comparisons.For a cabinet-drawer query, it identifies the correct top-left knob while the other methods select incorrect targets.
  • Ablation Study: The two-stage coarse-to-fine approach substantially improves performance over one-stage processing by combining global temporal coverage with high-resolution detail.One-stage high-resolution processing degrades performance, while low-resolution one-stage processing lacks sufficient spatial detail.
  • Ablation Study: Adding multisampling and temporal-window processing provides complementary benefits, with the full model achieving the best performance.
  • Ablation Study: Adding a temporal window at K = 2 raises AP50 from 15.51 to 21.12 and AP25 from 28.31 to 39.10.The gains are +5.61 AP50 and +10.79 AP25; the temporal window also makes verification more effective.
  • Ablation Study: Increasing sampling from K = 1 to K = 2 yields substantial gains, while further increasing to K = 4 provides only marginal improvement.Without diverse temporal candidates, single-iteration sampling can produce spatially inconsistent predictions.

5. Conclusion

UniFunc3D unifies semantic reasoning, temporal grounding, and spatial localization in one multimodal large language model for 3D functionality segmentation.

  • UniFunc3D consolidates semantic reasoning, temporal grounding, and spatial localization into a single multimodal large language model.
  • Experiments on SceneFun3D show state-of-the-art performance across all metrics against both training-free and training-based methods.

Segmentation

The supplementary document provides additional analyses, ablations, experimental details, and discussion of limitations and future work.

  • The supplementary material includes efficiency analysis, base-model ablations, experimental details, and limitations and future work.

6. Additional Experiments and Discussions

Additional experiments show that UniFunc3D is faster than Fun3DU, remains strong across backbone configurations, and improves localization when its grounding components are retained. The authors also identify remaining challenges for extremely small or heavily occluded functional parts.

  • Efficiency Analysis: Over 3.2 times speedup is achieved over Fun3DU while maintaining superior accuracy.
  • Efficiency Analysis: Approximately 26 minutes per scene are required by Fun3DU because of its multi-stage processing pipeline.
  • Ablation Study on Base Models: UniFunc3D surpasses TASA in 2 out of 3 metrics and exceeds AffordBot across AP50, AP25, AR50, AR25, and mIoU.
  • Limitations and Future Work: UniFunc3D cannot reliably handle extremely small functional parts below 0.1% of image area or scenes with severe occlusion.
  • Additional Visual Comparisons for Ablation Models: The full model reaches an mIoU of 0.523 for the cabinet-drawer query, while removing temporal windows, multi-sampling, or verification causes mislocalization or incorrect targets.

7. Additional Qualitative Results

Additional qualitative comparisons show UniFunc3D producing more accurate masks for fine-grained functional parts across diverse indoor scenes. It is especially effective for distinguishing adjacent or spatially related drawers, handles, and cabinet regions.

  • Additional Qualitative Results: UniFunc3D outperforms other baselines across diverse scenes containing different functional objects.
  • Additional Qualitative Results: UniFunc3D more accurately localizes cabinet handles and specific drawer sections than multi-stage baselines such as Fun3DU.
  • Additional Qualitative Results: UniFunc3D distinguishes target functional regions among multiple adjacent drawers, while AffordBot occasionally fails on small-scale objects.
  • Additional Qualitative Results: Figures 7–9 present five representative queries across four methods, with GT denoting ground truth.
Loading 2603.23478v1…