PaperlayerView exact revision on arXiv

Source-linked AI summary

Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description

Mahmoud Ahmed, Junjie Fei, Jian Ding, Eslam Mohamed Bakr, Mohamed Elhoseiny

arXiv:2405.18937v2cs.CVcs.CL

TL;DR

Fine-grained part and material understanding with segmentation grounding remains a challenge for 3D multimodal models. The paper introduces PaPGD, the 3DCoMPaT-GrIn dataset, and Kestrel, a model using point features and query refinement. Kestrel outperforms the reported 3D baselines on part-aware language comprehension and segmentation grounding metrics.

  • Problem

    Existing 3D MLLMs often fail to capture fine-grained object-part details and material properties, which are needed for precise real-world interaction.

    Source: §1 · p. 2

  • Method

    The paper introduces PaPGD and 3DCoMPaT-GrIn, and proposes Kestrel, which uses point features, an LLM, feature propagation, and query refinement for part-aware descriptions and segmentation grounding.

    Source: §1 · p. 2

  • Results

    Kestrel outperforms other 3D baselines across language comprehension and segmentation grounding metrics, achieving mIoU 86.70, AP50 54.42, and Recall 67.37.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    PaPGD and 3DCoMPaT-GrIn provide a task and dataset for training and evaluating part-aware 3D language understanding and segmentation grounding.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this paper, we introduce Part-Aware Point Grounded Description (PaPGD), a challenging task aimed at advancing 3D multimodal learning for fine-grained, part-aware segmentation grounding and detailed explanation of 3D objects. Existing 3D datasets largely focus on either vision-only part segmentation or vision-language scene segmentation, lacking the fine-grained multimodal segmentation needed for robotic navigation and interaction in real-world environments. To address this gap, we present the 3DCoMPaT Grounded Instructions (3DCoMPaT-GrIn) Dataset, a comprehensive resource that pairs rich point cloud descriptions with corresponding part-level segmentation masks. This dataset encompasses extensive samples designed for both PaPGD and fine-grained single-part grounding tasks. To tackle the inherent challenges of grounding objects and generating grounded descriptions at the part level, we propose Kestrel, a part-aware 3D multimodal large language model that integrates an advanced language model for nuanced language comprehension with multi-level point feature propagation and query refinement mechanism to enhance spatial reasoning at the part level. The extensive experiments demonstrate that Kestrel effectively bridges the gap between part-aware language understanding and 3D segmentation grounding, paving the way for more robust and interpretable 3D object comprehension that meets the demands of real-world robotic applications. Project page at https://feielysia.github.io/Kestrel.github.io/

1. Introduction

The introduction identifies fine-grained, material-aware part understanding and segmentation grounding as an unresolved challenge for 3D vision-language models. It introduces PaPGD, 3DCoMPaT-GrIn, and Kestrel as a task, dataset, and model addressing that challenge.

1. Introduction

  • Existing 3D MLLMs often miss fine-grained object-part details and material properties needed for precise real-world interaction.The introduction motivates compositional reasoning with a teapot-grasp example: identifying a wooden handle as a secure grip.Source: §1 · pp. 1–2
  • PaPGD requires generating a detailed description of an input point cloud and linking each part-level phrase to its segmentation mask.Source: §1 · p. 2
  • 3DCoMPaT-GrIn provides 111,514 training and 9,449 validation samples for PaPGD, direct segmentation, and reasoning segmentation.The dataset builds on 3DCoMPaT and combines point clouds with part annotations and material information to produce grounded descriptions.Source: §1 · p. 2
  • Kestrel combines a point encoder, LLM, point feature propagation module, and 3D segmentation decoder to generate part descriptions and segmentation queries.Its query-refinement mechanism incorporates upsampled PFPM features to improve the final masks.Source: §1 · p. 2
  • The paper presents PaPGD, 3DCoMPaT-GrIn, and Kestrel as contributions for material-aware, part-level segmentation and description.Kestrel combines query refinement with multi-level feature decoding for language understanding and segmentation grounding.Source: §1 · p. 2

2. Related Work

Related work spans 2D and 3D multimodal grounding as well as vision-language datasets, but the cited approaches generally do not provide fine-grained part-level segmentation grounding with detailed explanations.

2. Related Work

  • 2D grounding approaches include location-token and bounding-box prediction, polygon masks, and segmentation models paired with textual descriptions, while most still lack part-level reasoning.The discussion cites VisionLLM, LISA [28], and GLaMM [50] as examples of these approaches.Source: §2 · p. 2
  • Existing 3D MLLMs address scene or object understanding and grounding, while PARIS3D [26] focuses on single-part grounding and may be affected by domain shifts.The paper states that these models do not offer fine-grained part-aware segmentation grounding and detailed 3D-object explanations.Source: §2 · p. 2
  • Prior 3D vision-language datasets support captioning, question answering, or object localization, but few provide precise part localization and segmentation grounding.The cited 3D RPSeg dataset [26] covers one part per sample and is limited in shapes and parts per category; 3DCoMPaT [54] and PartNet [40] provide part-aware segmentation data.Source: §2 · p. 3

3. 3DCoMPaT-GrIn

3DCoMPaT-GrIn pairs point clouds with detailed grounded descriptions and part-level masks, and includes separate PaPGD and single-part grounding subsets. Its construction combines annotated parts and materials, rendered views, and GPT-4o-generated descriptions and prompts.

3. 3DCoMPaT-GrIn

  • 3DCoMPaT-GrIn uses 3DCoMPaT part and material annotations, each corresponding to a point-cloud segmentation mask, to ground references in descriptions.The dataset limits annotations to at most six prominent parts per point cloud, selected by visible area in rendered images.Source: §3 · p. 3
  • For PaPGD, four rendered views and part-material mappings are provided to GPT-4o to generate descriptions covering shape, parts, materials, colors, and potential functionality.Thirty user instructions guide the task; the subset contains 80,760 training and 6,770 validation description pairs.Source: §3 · p. 3
  • The direct segmentation subset has 8,076 training and 677 validation samples, while reasoning segmentation has 22,678 training and 2,002 validation samples covering 125 parts.Direct segmentation uses sampled part annotations and 15 prompt templates; reasoning prompts and responses are generated per part and manually filtered for identical functionality categories.Source: §3 · p. 3
  • Human annotators iteratively verify validation descriptions for ground-truth parts, associated materials, and representative colors using eight rendered views per point cloud.Across all three subsets, the dataset contains 111,514 training and 9,449 validation samples.Source: §3 · p. 3

4. Method

Kestrel combines a 3D multimodal language model with a segmentation decoder to generate part-level descriptions and masks. Its query-refinement mechanism uses multi-level point features to refine segmentation queries, while training jointly targets text and mask prediction.

4.1. Kestrel

  • Kestrel pairs a point encoder and LLM with a segmentation decoder that converts [SEG] token embeddings into initial queries for point-wise masks.The point encoder and LLM generate detailed part-level descriptions from point clouds and instructions.Source: Figure 2 · p. 4, §4.1 · p. 4
  • PFPM progressively upsamples point features, which combine with intermediate queries to form attention masks that guide query refinement and final part-mask prediction.The decoder integrates features at multiple levels; after two refinement stages, refined queries and the final upsampled features produce the mask prediction.Source: §4.1 · pp. 4–5

4.2. Training Objective

  • Kestrel is trained end to end to generate detailed descriptions and point-wise part masks using text cross-entropy, binary cross-entropy, and Dice losses.The default loss weights are 1.0 for cross-entropy, 2.0 for binary cross-entropy, and 0.5 for Dice loss.Source: §4.2 · p. 5

5. Experiments

Experiments evaluate Kestrel on part-aware description and single-part grounding, then examine query refinement and behavior under distribution shifts. Across the reported tasks, Kestrel achieves strong language and segmentation results, with application tests showing generalization to new domains and real-world inputs.

5. Experiments

  • Experiments assess language with BLEU-1, ROUGE-L, Sentence-BERT, SimCSE, and 3D-CALC, and segmentation with mIoU, AP50, and Recall.The evaluation includes description, direct and reasoning segmentation, ablations, and application tests.Source: §5 · pp. 5–6

5.1. Part-Aware Point Grounded Description

  • Kestrel achieves the highest mIoU (86.70), AP50 (54.42), and Recall (67.37), with mIoU 3.19% above the strongest baseline; it also leads on 3D-CALC.Traditional language metrics are comparable to PointLLM-7B and ShapeLLM-7B; 3D-CALC evaluates part, material, and composition accuracy.Source: §5.1 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6

5.2. Single-Part Segmentation Grounding

  • In direct segmentation, Kestrel leads all reported metrics, surpassing SegPoint by 8.44% in mIoU and 8.18% in AP50.On reasoning segmentation, it surpasses SegPoint by 6.60% in mIoU and 8.17% in AP50; on RPSeg3D, it outperforms PARIS3D after one epoch of fine-tuning on RPSeg3D.Source: §5.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

5.3. Ablation Studies

  • Two query-refinement levels achieve the highest mIoU for Grounded Description, Direct Segmentation, and Reasoning Segmentation.One refinement level improves results over none; the ablation holds the number of upsampling decoding levels constant.Source: §5.3 · p. 7, Table 5 · p. 8

5.4. Application

  • A checkpoint trained only on 3DCoMPaT-GrIn segments parts such as a pulling lever and wheel on Objaverse, demonstrating robustness to domain shift.For multi-object testing, Kestrel segments parts shared across two objects or appearing in only one.Source: §5.4 · pp. 7–8, Figure 4 · p. 8
  • Kestrel also handles noisy and incomplete ScanObjectNN inputs in real-world demonstrations.The demonstrations evaluate model-generated grounded descriptions.Source: §5.4 · p. 8, Figure 5 · p. 8

6. Conclusion

The paper introduces PaPGD, 3DCoMPaT-GrIn, and Kestrel, and reports that Kestrel outperforms existing methods on segmentation-grounding metrics and earns the highest GPT-based scores for part-aware language comprehension.

6. Conclusion

  • Kestrel surpasses existing methods on segmentation-grounding metrics across PaPGD, Direct Segmentation, and Reasoning Segmentation, and achieves the highest GPT-based part-aware language-comprehension scores.The conclusion also presents the work as a foundation for future research in fine-grained 3D object interaction and grounding.Source: §6 · p. 8
  • 3DCoMPaT-GrIn provides part-level grounded descriptions for over 120K samples to support training and evaluation of part-aware 3D vision-language understanding.Source: §6 · p. 8
  • PaPGD is introduced as a task for precise part-level segmentation grounding and detailed object comprehension, and Kestrel uses query refinement to connect language understanding with segmentation grounding.Source: §6 · p. 8

Supplementary Material

The Mask3D baseline uses language-model segmentation tokens as positional queries for a pretrained 3D segmentation decoder, which predicts part masks from voxel features.

A. Mask3D Baseline

  • Mask3D processes 8,192 RGB points with a voxel-based ResUNet encoder and segmentation decoder, using intermediate voxel resolutions from 600 to 8,192.The encoder and decoder are pretrained on ScanNet200; the stated voxel size is 0.01.Source: A. Mask3D Baseline · p. 12
  • FVL’s [SEG] hidden states are projected into the decoder’s query embedding space and used as positional queries to predict part masks aligned with the point cloud.Source: A. Mask3D Baseline · p. 12
  • The baseline combines FVL’s language understanding with Mask3D’s spatial reasoning and segmentation through decoder-produced part masks.Source: A. Mask3D Baseline · p. 12

B. Additional Ablation Studies

The ablations find that Uni3D with an MLP projector performs best overall, a small amount of direct-segmentation data offers a favorable trade-off, and combining all three data subsets yields the best results.

Model Architecture

  • Uni3D with an MLP projector achieves the best overall performance across the evaluated metrics, with Uni3D outperforming PointBert and MLP outperforming Q-Former on mIoU.The authors suggest the MLP produces tokens closer to those used during decoder upsampling.Source: B. Additional Ablation Studies · p. 12

Dataset Amount

  • Adding 8K direct-segmentation samples helps part grounding, while increasing the subset to 40K raises training cost and slightly degrades grounded-description performance.The authors characterize the smaller addition as a good overall-performance trade-off.Source: B. Additional Ablation Studies · p. 12

Additional qualitative results

Dataset Distribution

  • Adding the direct-segmentation subset boosts performance particularly on its task, and incorporating reasoning-segmentation data yields the best results across the tested distributions.The authors conclude that coverage of grounded descriptions, direct segmentation, and reasoning segmentation is important for robust part-aware understanding.Source: Table 8 · p. 12, B. Additional Ablation Studies · p. 13

C. PartNet-Mobility

On a sample of PartNet-Mobility categories, Kestrel uses one unified model across categories, while PartSTAD has the highest overall score but trains a separate model per category.

C. PartNet-Mobility

  • PartSTAD has the highest overall score, but Kestrel uses one unified model and its few-shot setting outperforms or matches prior methods on several categories, including Lamp and Chair.The passage reports strong overall performance and generalization across diverse part-segmentation tasks with limited supervision.Source: C. PartNet-Mobility · p. 14

D. Qualitative Examples

The qualitative examples show Kestrel producing detailed part-level explanations with corresponding 3D segmentation masks, and grounding single-part instructions in the object.

D. Qualitative Examples

  • Fig. 6 illustrates detailed part-level descriptions of 3D objects, with a corresponding predicted 3D position represented by a segmentation mask for each phrase.The examples present the masks alongside the generated explanations.Source: D. Qualitative Examples · p. 14
  • Fig. 7 presents single-part segmentation grounding results for instructions about a part.Kestrel predicts the part’s position from the text input.Source: D. Qualitative Examples · p. 14
  • The examples demonstrate Kestrel interpreting part-aware instructions and user intent to locate the requested part in 3D space.Source: D. Qualitative Examples · p. 14

E. Data Collection

The data collection pipeline uses GPT-4o and shape metadata to create part- and material-aware captions, then structures prompts for multi-part description grounding and single-part segmentation.

E.1. GPT4 collection

  • Shape metadata supplies part–material pairs, while GPT-4o uses that text and eight rendered views to caption each shape’s parts, materials, and colors.The prompts for captioning and reasoning segmentation are shown in Tables 10 and 11.Source: §E.1 · p. 14

E.2. Prompts

  • The grounded-description dataset uses one of 30 predefined instructions with a GPT-4o-generated caption to guide part-specific mask generation.The instructions are paired with captions for grounding descriptions.Source: §E.2 · p. 14
  • The reasoning-segmentation dataset targets one part using a randomly selected instruction and response template; direct segmentation uses 15 prompts and 11 templates.These structured prompts cover both multi-part and single-part grounding tasks.Source: §E.2 · pp. 14–15

E.3. Human Evaluation

The validation process checks grounded descriptions for completeness and agreement with part materials and colors, while the examples section reports dataset scale and illustrates part-level grounding outputs.

E.3. Human Evaluation

  • Annotators evaluated 6,770 validation descriptions for coverage of ground-truth parts and materials, color accuracy, and extra part names requiring correction.Samples with inaccuracies were corrected by rerunning the GPT-4o pipeline.Source: §E.3 · p. 15, Figure 9 · p. 16

E.4. Examples

  • The collection includes 88,836 training and 6,770 validation samples for part-aware descriptions, plus 677 validation samples for single-part grounding.Source: §E.4 · p. 15
  • The examples show colored point clouds paired with grounded descriptions that capture object components and their positions.Additional qualitative examples illustrate phrase-linked masks and single-part grounding from text instructions.Source: §E.4 · p. 15

Related papers

Loading 2405.18937v2…