PaperlayerView exact revision on arXiv

Source-linked AI summary

Part-X-MLLM: Part-aware 3D Multimodal Large Language Model

Chunshi Wang, Junliang Ye, Yunhan Yang, Yang Li, Zizhuo Lin, Jun Zhu, Zhuo Chen, Yawei Luo, Chunchao Guo

arXiv:2511.13647v1cs.CV

TL;DR

Part-X-MLLM addresses the challenge of part-aware 3D interaction by generating executable language programs from point clouds and prompts. Its plans unify generation, grounded QA, and localized editing; on UniPart-Bench, the paper reports improvements in box and voxel metrics. The conclusion also identifies inference latency, segmentation depth, and possible language-capability loss as limitations.

  • Problem

    Existing 3D systems lack a unified language-native way to identify meaningful parts, ground references to them, and produce executable interactions.

    Source: §1 · p. 2

  • Method

    Part-X-MLLM uses separate geometry and appearance encoders and an autoregressive decoder to generate structured part-box and edit-operation programs for downstream geometry modules.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Part-X-MLLM improves Voxel Recall/IoU and BBox IoU on UniPart-Bench while using executable programs to unify generation, QA, and localized editing.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The paper presents a single language-native planning interface that can control compatible geometry engines for part-centric 3D tasks.

    Source: §1 · p. 2

    Longer output sequences slow inference, BBox-confidence segmentation remains relatively shallow, and 3D-task fine-tuning may reduce the base LLM’s general language capabilities.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce Part-X-MLLM, a native 3D multimodal large language model that unifies diverse 3D tasks by formulating them as programs in a structured, executable grammar. Given an RGB point cloud and a natural language prompt, our model autoregressively generates a single, coherent token sequence encoding part-level bounding boxes, semantic descriptions, and edit commands. This structured output serves as a versatile interface to drive downstream geometry-aware modules for part-based generation and editing. By decoupling the symbolic planning from the geometric synthesis, our approach allows any compatible geometry engine to be controlled through a single, language-native frontend. We pre-train a dual-encoder architecture to disentangle structure from semantics and instruction-tune the model on a large-scale, part-centric dataset. Experiments demonstrate that our model excels at producing high-quality, structured plans, enabling state-of-the-art performance in grounded Q\&A, compositional generation, and localized editing through one unified interface. Project page: https://chunshi.wang/Part-X-MLLM/

1 INTRODUCTION

Part-X-MLLM addresses the lack of part-aware, language-native 3D interaction by representing diverse tasks as executable plans. Its design combines persistent part references, separate geometry and appearance encoding, controllable semantic granularity, and a part-centric benchmark.

Motivation

  • Holistic 3D generators often treat assets as monolithic, while existing systems lack a unified language interface for naming parts, grounding references, and producing executable edits.The introduction frames this structural opaqueness as a constraint on fine-grained understanding, compositional editing, and procedural animation.Source: §1 · p. 2

Approach

  • Part-X-MLLM unifies generation, editing, and question answering by translating instructions and 3D input into programs containing part boxes, persistent references, descriptions, and edit operators.The model-agnostic token interface is intended to drive downstream geometry modules.Source: §1 · p. 2

Architecture

  • The dual encoder separates geometry (XYZ and normals) from appearance (RGB), with reported consistent gains over a single-encoder baseline across grounding, captioning, and part Q&A.The introduction attributes representational conflict to combining structure and semantics in one encoder.Source: §1 · p. 2

Semantic granularity

  • Text-semantic clustering of part boxes lets users move between coarse components and fine-grained parts through the same programmatic interface.The introduction describes this as avoiding a need to predefine the number of parts or manually merge masks.Source: §1 · p. 2

Benchmark

  • UniPart-Bench contains 30k entries across 11 task families and evaluates plan quality and downstream performance with geometric and linguistic metrics.The benchmark is introduced to assess structured, part-centric tasks.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses 3D understanding, generation, part decomposition, and editing through distinct representations and interfaces. The paper positions its framework as a language-native planning interface that connects part-aware plans to geometry execution.

2.1 MULTIMODAL UNDERSTANDING

  • Early 3D multimodal models align point clouds with language for captioning, QA, and reasoning, while scene-level systems ground dialogue through identifiers, superpoints, or spatial features.The cited work also includes programmatic scene abstractions and language-embedded scene reconstruction.Source: §2.1 · p. 3

2.2 3D GENERATION

  • 3D generation spans score-distillation optimization, sparse voxel representations, and compact vector sets, trading among 2D-model guidance, geometric detail, computational cost, and scalability.The passage notes high-resolution voxel methods can be computationally costly, while vector sets can sacrifice surface detail fidelity.Source: §2.2 · p. 3

2.3 PART GENERATION

  • Part generation methods lift 2D cues into 3D or generate parts natively, while OmniPart combines autoregressive box planning with TRELLIS-based synthesis.The passage notes weak 3D constraints for 2D-driven methods and describes additional direct 3D approaches.Source: §2.3 · p. 3

2.4 3D EDITING

  • Editing methods range from optimization-based pipelines to faster feed-forward or training-free approaches, but the cited methods lack a language-native part planner that emits executable, spatially grounded edits.Figure 2 depicts the proposed pipeline: separate geometry and appearance encoding, prompt fusion, autoregressive plan generation, and geometry-head execution.Source: §2.4 · p. 3, Figure 2 · p. 4

3 METHODOLOGY

Part-X-MLLM treats part-aware 3D interaction as structured language planning, then delegates geometry synthesis and editing to downstream modules. Its pipeline combines dual-stream encoding, program generation, and a two-stage training curriculum.

3 METHODOLOGY

  • The methodology combines a unified geometry-and-language architecture, multi-stage training, and pre-existing geometry engines as execution backends.Figure 2 presents the overall framework.Source: §3 · p. 4

3.1 MOTIVATION

  • The design targets precise language control over meaningful parts, where artists and downstream applications need addressable structure rather than holistic shape synthesis.The motivation gives examples including swapping a handle without changing the body and reasoning about graspable components.Source: §3.1 · p. 4

3.2 UNIFIED ARCHITECTURE FOR PART-AWARE PLANNING

  • A structure encoder processes XYZ and normals while a semantic encoder processes RGB; a pretrained-LLM decoder generates programs with box-coordinate and edit-operation tokens.The planning language makes diverse tasks a single instruction-following problem; separate streams can distinguish structurally similar but visually different parts.Source: §3.2 · p. 4

3.3 DOWNSTREAM GEOMETRY INTERFACES

  • Downstream synthesis uses planned boxes as spatial guides, while editing uses boxes to define cuboid masks for localized changes that preserve untouched regions.The output is designed for geometry modules that interpret its geometric and semantic content.Source: §3.3 · pp. 4–5

3.4 END-TO-END TASK REALIZATION

  • The planning language supports part-aware mesh generation, grounded answers with persistent BBox references, and auto-located edits executed with masked commands.Text-description similarity using CLIP embeddings can cluster fine parts into coarser components, letting users adjust output detail without manually defining part counts or merging masks.Source: §3.4 · p. 5, Figure 3 · p. 5

3.5 MULTI-STAGE INSTRUCTION TUNING

  • Training first pretrains a geometry-only structure encoder for BBox localization, then instruction-tunes a dual-encoder LLM with an RGB semantic encoder and task-specific tokens.Stage 1 uses RGB-less point clouds and retains the specialized structure-encoder weights; Stage 2 freezes that encoder and the original LLM token embeddings while training the semantic encoder, decoder layers, and new token embeddings.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate structured part plans and downstream tasks on UniPart-Bench. Results cover part and object understanding, generation and editing, granularity control, and confidence-aware segmentation.

4.1 IMPLEMENTATION DETAILS; 4.2 DATASET AND EVALUATION PROTOCOL

  • The evaluation uses an 85,771-object part-centric dataset and a 400-object held-out benchmark; training runs in two stages, and downstream tasks use external geometry modules.The dataset averages 23 parts per object and includes annotations at coarse and fine text granularities.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 PART-AWARE GENERATION AND EDITING

  • Qualitative decompositions capture fine-grained details and preserve structural integrity, while baselines often produce fragmented or inaccurate parts.Bounding-box generation is evaluated with BBox IoU, Voxel Recall, and Voxel IoU against PartField and OmniPart.Source: §4.3 · p. 7, Figure 4 · p. 7, Table 1 · p. 7
  • The model interprets instructions to add, remove, or modify parts while preserving the rest of the object's structure.Figure 5 presents qualitative part-aware editing results.Source: §4.3 · p. 7, Figure 5 · p. 8
  • CLIP-similarity clustering progressively merges parts into coarser components, reducing the count from 22 to 2 without manual intervention.The process supports control over output detail.Source: §4.3 · p. 7, Figure 6 · p. 8
  • Generated boxes and token probabilities support fine-grained, confidence-aware face segmentation without additional training.The assignment maps each face to a single bounding box.Source: §4.3 · p. 8, Figure 7 · p. 8
  • The dual encoder improves box-listing IoU by +7.06 and consistently outperforms the single-encoder baseline across evaluated tasks.The comparison is reported on UniPart-Bench.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 PART AND OBJECT UNDERSTANDING

  • +18.7 SBERT, +25.5 SimCSE, +21.3 BLEU-1, +13.0 ROUGE-L, and +14.2 METEOR over the strongest baseline in part-level Q&A.The evaluation uses UniPart-Bench.Source: Table 3 · p. 9, §4.4 · p. 9
  • +10.4 SBERT, +9.4 SimCSE, +18.8 BLEU-1, +20.2 ROUGE-L, and +15.2 METEOR over best prior scores in overall object captioning.The evaluation probes holistic object understanding.Source: §4.4 · p. 9, Table 4 · p. 10

5 CONCLUSION

Part-X-MLLM represents 3D interaction as executable program generation, using part AABBs to unify generation, QA, and localized editing. The conclusion notes inference, segmentation-quality, and language-capability limitations.

5 CONCLUSION

  • Part-X-MLLM emits part-AABB programs that geometry engines execute, unifying generation, QA, and localized editing while improving benchmark metrics.The conclusion also points to controllable granularity in Appendix A.2.1.Source: §5 · p. 10
  • Longer sequences slow inference; compaction and hierarchical grouping mitigate latency, while BBox-based confidence segmentation remains shallow and 3D fine-tuning may reduce general language capability.Source: §5 · p. 10

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

The authors state that LLMs were used only for minor language editing, not method design or experimental work.

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • LLMs were used only for minor language editing, such as grammar and readability improvements; technical contributions were the authors' work.Source: §A.1 · p. 15

A.2.1 SEMANTIC PART CLUSTERING ALGORITHM

The algorithm clusters part bounding boxes into coarser semantic components using text and spatial features, then merges each cluster into a bounding box. It requires neither manual intervention nor a predefined number of clusters.

A.2.1 SEMANTIC PART CLUSTERING ALGORITHM

  • The post-processing pipeline extracts features, clusters parts, and merges each cluster into a coarser bounding box without manual intervention or a predefined cluster count.Its final output is a set of K merged boxes representing a semantically grouped decomposition.Source: §A.2.1 · p. 15
  • Each part’s hybrid feature combines a CLIP-encoded text description with a spatial feature derived from its box center and size, normalized across the object’s parts.The semantic and spatial features are combined with weighting factor α and the resulting vector is L2-normalized.Source: §A.2.1 · p. 15
  • DBSCAN groups feature vectors by distance threshold ϵ and minimum-points parameter minPts, yielding a variable number of clusters without specifying their number in advance.It also returns noise points.Source: §A.2.1 · p. 15
  • For each cluster, the algorithm merges its boxes by taking component-wise minima and maxima, producing one bounding box per cluster.The merged boxes form a coarser, semantically grouped object decomposition.Source: §A.2.1 · p. 15
  • Semantic clustering flexibly adjusts generated assets from fine-grained parts toward coarser semantic components.The paper presents this as bridging fine-grained part generation and high-level semantic understanding.Source: §A.2.1 · p. 15

A.3 ADDITIONAL QUALITATIVE RESULTS

The qualitative examples show Part-X-MLLM answering part-aware questions with detailed descriptions and precise bounding-box grounding.

A.3 ADDITIONAL QUALITATIVE RESULTS

  • Part-aware Q&A examples show detailed, box-annotated answers that accurately describe object parts in response to user queries.The figure caption characterizes the answers as more accurate and descriptive, with precise part grounding indicated by bounding-box tokens.Source: §A.3 · p. 16, Figure 9 · p. 16
  • The examples illustrate responses to user queries that accurately describe object parts.The figure caption describes the answers as both more accurate and more descriptive.Source: §A.3 · p. 16, Figure 9 · p. 16
  • The answers use bounding-box tokens to indicate precise grounding to object parts.This is the grounding mechanism highlighted in the Figure 9 caption.Source: Figure 9 · p. 16

A.4 CONFIDENCE-AWARE FACE SEGMENTATION FROM BOUNDING BOXES

The paper uses bounding-box token probabilities to estimate confidence for predicted boxes, then assigns mesh faces to boxes using containment and confidence rules. This produces a deterministic face-to-box assignment for fine-grained segmentation.

A.4 CONFIDENCE-AWARE FACE SEGMENTATION FROM BOUNDING BOXES

  • The segmentation process uses generated bounding boxes and decoding token probabilities without additional training.The paper presents this as a downstream application of the model’s autoregressive output.Source: §A.4 · p. 16
  • Each bounding box’s confidence is the arithmetic mean of the probabilities of its constituent tokens, typically six.The score reflects the model’s certainty in that box prediction.Source: §A.4 · p. 16
  • A mesh face is a candidate for a box when its centroid lies within that box’s volume.This criterion assigns faces based on inferred bounding boxes.Source: §A.4 · p. 16
  • For overlapping boxes, the algorithm assigns a face to the smallest box when one box contains the other, or to the highest-confidence box otherwise.The rules prioritize specific, fine-grained predictions in containment cases.Source: §A.4 · p. 17
  • The rules yield a deterministic assignment of each face to one box, producing a fine-grained object segmentation.The paper illustrates the resulting segmentation in Figure 7.Source: §A.4 · p. 17

A.5 ANALYSIS OF SPECIAL TOKEN EMBEDDINGS

The appendix analyzes special-token embeddings and documents dataset construction, instruction templates, and UniPart-Bench results. The token visualization shows distinct functional groups and ordered coordinate representations.

A.5 ANALYSIS OF SPECIAL TOKEN EMBEDDINGS

  • Point, Box, and Edit tokens cluster separately; 128 coordinate tokens form an ordered manifold, and similar edit start/end tokens lie close together.The authors interpret this structure as evidence that the model learns coordinate order and a representation of its executable grammar.Source: §A.5 · p. 17, Figure 10 · p. 17

A.6 DATASET CONSTRUCTION AND LABELING

  • The dataset contains 85,771 objects averaging 23 parts each, with AABBs, coarse part names, fine descriptions, object captions, and part-aware Q&A.Annotations are serialized with the unified box-token grammar.Source: §A.6 · pp. 17–18

A.6.1 MODEL-ASSISTED LABELING

  • A model-assisted labeling pipeline uses full-asset images and per-part crops with a structured schema and deterministic JSON responses.The tool calls an external LMM; labels include a short part name, a fine-grained description, and a confidence flag.Source: §A.6.1 · p. 18

A.6.2 BUILDING INSTRUCTION-FOLLOWING SAMPLES

  • Instruction samples serialize six quantized coordinates between box tokens and use edit-verb tokens for addition, deletion, and modification programs.Eleven templates span box listing, grounding, box-to-text, part-aware Q&A, and edit programs.Source: §A.6.2 · p. 18

A.7 DATASET STATISTICS

  • The corpus covers everyday objects and scenes, with the appendix identifying its top 19 categories.Task-family counts are summarized before and after optional balancing.Source: §A.7 · p. 19

A.9 PROMPT TEMPLATES FOR DATA CONSTRUCTION

  • The appendix supplies prompt templates for the 11 task types, including prompts for box listing, coarse and fine grounding, part descriptions, Q&A, and edits.The examples include single-part locating prompts and requests to add, delete, or modify parts.Source: §A.9 · p. 19, §A.9.4 · p. 23, §A.9.5 · p. 24

A.8 COMPREHENSIVE RESULTS ON UNIPART-BENCH

  • Language-intensive tasks T7 Part QA and T10 Edit—Add score highest on SBERT/SimCSE, while T0/T2/T10 show strongest IoU alignment.Blank text or IoU entries are intentional when the ground-truth task lacks that modality.Source: §A.8 · p. 19, Table 7 · p. 20

Related papers

Loading 2511.13647v1…