PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Single Object: Learning 3D Relations with Large Language Models

Kohsuke Ide, Ryousuke Yamada, Yue Qiu, Xianzheng Ma, Yoshihiro Fukuhara, Hirokatsu Kataoka, Yutaka Satoh

arXiv:2608.15710v1cs.CVcs.AIcs.CL

TL;DR

Existing 3D-LLMs emphasize single-object description or scene context rather than detailed object-to-object comparison. This paper introduces MO3D, Multi-3DLLM, and application-driven benchmarks, and reports that the model outperforms baselines on MO3D and transfers positively to single-object classification.

  • Problem

    Existing object-centric and scene-level 3D models are not designed for detailed object-to-object comparison, and comparison-focused architectures and instruction data are lacking.

    Source: §1 · pp. 1–2

  • Method

    The paper pairs the MO3D multi-object comparison instruction dataset and two application-driven benchmarks with Multi-3DLLM, a minimally adapted 3D-LLM.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Multi-3DLLM outperforms baselines on MO3D and performs strongly on Shape Mating and Change Captioning, with positive transfer to single-object and point-wise benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates practical utility for multi-object 3D reasoning on complex application-driven tasks.

    Source: §6 · p. 8

    The pipeline relies on existing captions and may inherit biases in description granularity.

    Source: §6 · p. 8

Abstract

from arXiv · show

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

1. Introduction

The paper targets the lack of fine-grained comparison and relational reasoning across 3D objects. It introduces MO3D and two application benchmarks, Shape Mating and Change Captioning, alongside Multi-3DLLM, a minimally modified 3D-LLM for inter-object reasoning.

Motivation

  • Object-centric PointLLM [53] and ShapeLLM [40] train largely on isolated objects, while scene-level models [51] retain coarse semantics, leaving local geometry and fine-grained comparison unsupported.The paper identifies the missing ingredients as an architecture for fine-grained cross-object relationships and an instruction dataset teaching geometric comparison.Source: §1 · pp. 1–2

Dataset

  • MO3D uses balanced, order-invariant prompts over semantically grouped objects to produce up to 70k positional, comparative, and holistic geometry-discrimination queries.It groups two or three Objaverse-Cap3D [13] [33] point clouds using CLIP [43], then synthesizes grounded question–answer pairs from multi-view renderings and hierarchical constraints.Source: §1 · p. 2

Model

  • Multi-3DLLM augments PointLLM [53] with shallow scalar-gated attention over concatenated patch tokens, capturing cross-object dependencies while retaining local geometry with minimal architectural change.The paper contrasts patch-level processing with object-level aggregation, which smooths details; it presents the modification as architecture-agnostic.Source: §1 · p. 2

Results

  • 50.7% to 54.2% zero-shot single-object classification improvement accompanies Multi-3DLLM’s substantial MO3D gains over strongest baselines and Mini-App-specialized models.Trained on a mixed data suite, it also performs strongly on Shape Mating, Change Captioning, and standard point-wise benchmarks.Source: §1 · p. 2

2. Related Work

Prior 3D-LLMs separate high-fidelity single-object understanding from scene-level multi-object processing, while 2D-VLMs offer a comparison blueprint without direct access to 3D structure. These limitations motivate combining patch-level fidelity with multi-object capability for fine-grained geometric comparison.

Object-Centric 3D-LLMs and Scene-Level 3D-LLMs

  • Object-centric 3D-LLMs excel at describing and recognizing individual objects but are not designed to compare distinct objects, while scene-level models lose patch details through object-level pooling.The paper positions its approach between these paradigms, combining object-centric patch-level fidelity with scene-level multi-object capability.Source: §2 · pp. 2–3

Multi-image and Multi-entity VLMs

  • Instruction-tuned and multi-image 2D-VLMs, including InstructBLIP [12], Flamingo [3], and Qwen-VL [5], provide a blueprint for following instructions and comparing visual inputs.Comparison also has precedent in 2D change captioning [26] [38] and compositional reasoning [46] [47].Source: §2 · p. 3
  • Despite strong language and semantic priors, 2D-VLMs lack direct access to 3D structure and struggle with fine-grained geometry-centric tasks.They can compare semantics or context from projections [3] [5] [12] [14] [31] [56], but this blueprint is insufficient when strict 3D geometric fidelity is required.Source: §2 · p. 3

3. MO3D: Multi-Object in 3D Dataset

MO3D is an instruction-tuning dataset for multi-object relational understanding, built from 3D objects and captions to support comparison, shape mating, and change captioning. Its construction combines language and multi-view visual inputs, while two mini-app benchmarks test practical geometric reasoning.

3. MO3D: Multi-Object in 3D Dataset

  • MO3D provides approximately 70k high-quality QA pairs, each involving 2–3 point clouds, for training and evaluating multi-object relational understanding.The dataset is built on Objaverse-Cap3D [13] [33] and targets multi-object comparison, shape mating, and change captioning.Source: §3 · p. 3
  • Its two-stage pipeline extracts caption attributes into six comparison categories, then generates questions for semantically related object groups using captions and multi-view renderings.Weighted sampling balances geometry, taxonomy, function, material, style, and color; a strict prompt hierarchy guides clear, visually grounded questions. The pipeline uses Qwen2-72B-Instruct [56], GPT-4, and dual sampling inspired by the complementary pair approach [19].Source: §3 · p. 3
  • MO3D organizes questions into positional, comparative, and holistic understanding, progressing from role-based referencing to property comparison and whole-set Yes/No reasoning.These task types form a structured curriculum for developing comparison capabilities.Source: §3 · pp. 3–4
  • The final 70k examples are split into training and test sets, and a human audit found 81.0% unanimous agreement across 500 randomly sampled test QA pairs.Balanced sampling yields a near 50:50 Yes/No ratio in the holistic subset; disagreements largely reflected ambiguity in abstract shapes or occlusions.Source: §3 · p. 4

3.1. Mini-Apps: Application-Driven Benchmarks

  • The Shape Mating benchmark tests whether a model can identify a compatible pair among three component point clouds and explain its geometric rationale.It uses complementary parts created by procedurally cutting Thingi10K meshes [62] with the process from Neural Shape Mating [10], and presents four-choice scenarios with one ground-truth pair and three incompatible parts.Source: §3.1 · p. 4
  • Change Captioning evaluates geometric-edit grounding using anchor, positive, and deliberately confusing negative point clouds paired with a human-written instruction.Derived from the ShapeNet [7] variation of ShapeTalk, it includes verification of whether a candidate follows the instruction and delta captioning that describes the edit from anchor to positive.Source: §3.1 · p. 4

4. Multi-3DLLM Architecture

Multi-3DLLM extends PointLLM [53] to multiple objects, combining independently encoded patch tokens through cross-object attention while retaining local geometry. Its two-stage training first aligns point-cloud features with the LLM, then fine-tunes the model on a mixture of relational tasks.

4. Multi-3DLLM Architecture

  • A scalar-gated PIT applies self-attention across patch tokens; initializing its gate near zero preserves pre-trained alignments as the model learns cross-object relations.Patch-level interaction preserves fine-grained local geometry, unlike prior scene-level work that compresses each object to one vector [51].Source: §4 · p. 4

4.1. Training Strategy

  • Phase 1 freezes the point-cloud encoder and LLM while training only the MLP projector on a large-scale brief-description instruction dataset.This phase aligns point-cloud features with the LLM text-embedding space, following PointLLM’s first training stage [53].Source: §4.1 · p. 4

4. Multi-3DLLM Architecture

  • Each object is encoded and projected independently, and its patch tokens are concatenated into a set-wide representation before entering the LLM.Figure 3 depicts the independent encoding and projection of N point clouds followed by interaction over the concatenated tokens.Source: §4 · p. 4, Figure 3 · p. 5

4.1. Training Strategy

  • Phase 2 unfreezes the projector, PIT, and LLM for instruction-tuning on an approximately 150K-pair mixture of MO3D, Shape Mating, and Change Captioning data.The mixture samples approximately 63K MO3D, 44K SM, and 40K CC pairs; response-token negative log-likelihood is minimized across both phases.Source: §4.1 · p. 5

5. Experiments

Experiments show Multi-3DLLM outperforming baselines on MO3D and key mini-application tasks, while task-mixture training transfers positively to zero-shot classification. Ablations associate patch-level interactions with relational-task gains and multitask training with sparser attention.

5. Experiments

  • Experiments cover MO3D, SM, CC, and ModelNet40, comparing point-cloud 3D-LLMs with rendered-view LLaVA [31] and Molmo [14], and adapting 3D baselines through point-cloud concatenation.Choice-task Reasoning Accuracy is evaluated on correctly answered samples; a human audit of 300 responses showed 91.0% unanimous agreement with the LLM evaluator.Source: §5 · pp. 5–6

5.1. Multi-Object Comparison (MO3D)

  • 56.3 Positional (M) and 81.7 Holistic (B) put Multi-3DLLM ahead of all baselines on MO3D; its Comparative (M) 33.8 exceeds ShapeLLM’s 11.2 and LLaVA’s 11.7.The strongest 2D baseline on Comparative (M) is LLaVA at 11.7, while the strongest 3D baseline is ShapeLLM at 11.2.Source: Table 1 · p. 6, §5.1 · p. 6

5.2. Performance on Mini-Applications

  • On SM, Multi-3DLLM reaches 37.1% Selection (S), above all baselines and the 25% chance level; on CC Delta Caption (M), it scores 51.0% while baselines are near zero.All models perform near the 50.0% chance baseline on CC Verification (B).Source: Table 1 · p. 6, §5.2 · p. 7

5.3. Evaluation on Zero-Shot Classification

  • PointLLM w/ MI falls to 2.1 for M=1 in the 3-Input task, while fine-tuned models achieve stable positional accuracy, surpass PointLLM on single-object classification, and retain similar performance with or without PIT.The results indicate that positional grounding must be learned; the PIT block is not required for high-level semantic classification and does not degrade that capability.Source: Table 2 · p. 6, §5.3 · p. 7

5.4. Ablation Studies

  • Ours (PIT) improves MO3D over No-Interaction by 10.8 points on Positional (M) and 12.0 on Comparative (M), and leads SM with 37.1 Selection and 36.8 Rationale.The Object-Level variant’s mean-pooling and broadcast updates wash out token-level cues, whereas patch-specific updates preserve local correspondences such as mating interfaces.Source: §5.4 · p. 8
  • Multitask training reduces Effective Rank by 50.7%, processing less than half as many points while achieving higher overall performance; the authors hypothesize fine-grained tasks promote focus on salient geometry.The attention comparison contrasts a model trained on MO3D + Mini-apps with a MO3D-only baseline; Figure 4 visualizes the resulting more concentrated attention.Source: §5.4 · p. 8, Figure 4 · p. 8, Table 4

6. Conclusion

The paper extends 3D-LLMs to multi-object comparison, shape mating, and change captioning with MO3D and Multi-3DLLM, which outperforms prior methods and demonstrates practical utility. It identifies limitations in caption granularity, token budgets, and real-world point density, and proposes future directions to address them.

Conclusion

  • Multi-3DLLM, trained on a holistic data mixture with minimal patch-level adaptation, significantly outperforms prior methods such as PointLLM.MO3D provides a robust benchmark for multi-object point-cloud tasks.Source: §6 · p. 8
  • The framework extends conventional 3D-LLMs beyond single objects to multi-object comparison, shape mating, and change captioning, demonstrating utility on complex application-driven tasks.The MO3D dataset provides a benchmark for this expanded task scope.Source: §6 · p. 8
  • The pipeline inherits biases from existing captions’ description granularity, while larger scenes pose token-budget trade-offs and real-world scans pose point-density challenges.Future work will explore hierarchical part-based annotations, efficient token scaling, and robust real-world domain adaptation.Source: §6 · p. 8

Supplementary Material · A. Implementation Details · Model Checkpoints

The system uses publicly available pretrained models from the PointLLM framework: Vicuna-7B-v1.5 for language, Point-BERT pretrained via ULIP2 for point-cloud encoding, and OpenCLIP ViT-L/14 for classification and grouping.

Model Checkpoints

A.1. Training Hyperparameters · Hardware · B. Dataset Generation Details

The appendix reports the optimizer and learning-rate schedule used for two-stage training, along with the hardware and durations for its two phases.

A.1. Training Hyperparameters

  • Training used AdamW with a cosine learning-rate scheduler and a 0.03 warmup ratio.The hyperparameters for the two-stage strategy are detailed in Table 5.Source: §A.1 · p. 12

Hardware

  • Training on 8 NVIDIA H200 GPUs (140GB VRAM) took 70 minutes for feature alignment and 12 hours for holistic-mixture training.The holistic mixture included MO3D, Shape Mating, and Change Captioning.Source: Hardware · p. 12

B.1. MO3D Dataset … C.1. Baselines & Input Format Fairness

The sections describe construction of MO3D and both mini-app datasets, a leakage-free split, and modality-specific baseline inputs. They also report that Multi-3DLLM outperforms a text-only baseline across MO3D metrics.

B.1. MO3D Dataset

  • MO3D derives six reasoning categories from 70k Objaverse-Cap3D captions and contains approximately 70k examples, balanced across three tasks and near-even positive and negative object groupings.Weighted sampling prioritizes geometric and structural categories (30.4%) over color (4.9%).Source: §B.1 · pp. 12, 14
  • MO3D forms two- or three-object groups from caption-similar neighbors, balances positive and negative sampling, and uses visually grounded, category-targeted QA generation with shuffled object order.Three-object groups enable outlier and majority reasoning beyond pairwise comparison; post-hoc correction targets near 50:50 yes/no balance in the holistic subset.Source: §B.1 · pp. 12–14

B.2. Mini-App A: Shape Mating Details

  • Shape Mating uses Thingi10K meshes cut with five geometries and 8,192 surface points per part; hard negatives and structured mismatch tags support four-choice, rationale-based QA.The cut-shell operation preserves surface-shell inputs to match Point-BERT’s surface-point pretraining; mismatch tags cover cut, object, phase, and same-side incompatibility.Source: §B.2 · pp. 14–15

B.3. Mini-App B: Change Captioning Details

  • Change Captioning uses Anchor-Positive-Negative triplets with same-anchor hard negatives, randomized Anchor-Candidate order for verification, and GPT-4o-mini-generated rationales and delta captions.Verification instructions average 64.3 words because they include the full list of geometric requirements to check.Source: §B.3 · p. 15, Figure 8 · p. 15

B.4. Data Splitting

  • A graph-based split assigns each connected component of co-occurring or source-related 3D assets wholly to Train or Test, preventing related instances from leaking across splits.The same leakage-free strategy applies to MO3D, Shape Mating, and Change Captioning.Source: §B.4 · pp. 15–16

C.1. Baselines & Input Format Fairness

  • Object-centric baselines receive separately normalized, axis-translated point clouds, while scene-level baselines receive target centroids or boxes; despite localization, scene-level models underperform on MO3D and Shape Mating.The section attributes this shortfall to object-level token pooling smoothing the fine-grained local geometry required for detailed comparison.Source: §C.1 · p. 16, Table 10 · p. 16
  • 56.3 vs 52.0 on Positional M and 33.8 vs 28.9 on Comparative M: Multi-3DLLM outperforms Vicuna-7B using ground-truth captions, and leads it across all reported MO3D metrics.The compared scores are Multi-3DLLM 56.3 and Vicuna-7B 52.0 on Positional M, and Multi-3DLLM 33.8 and Vicuna-7B 28.9 on Comparative M.Source: Table 10 · p. 16, §C.1 · p. 16

C.2. Architectural Ablation: Interaction Mechanics … E.1. LLM-based Evaluation Prompts

The extended analyses test how interaction granularity, training stages, object-count scaling, prompting, and evaluation affect Multi-3DLLM’s geometric reasoning and transfer. They find that preserving patch-level geometry and broad 3D-text alignment matters, while targeted inference and evaluation protocols support broader assessment.

C.2. Architectural Ablation: Interaction Mechanics

  • On Shape Mating Selection, Object-Level (Max) and Micro-Token score 23.9 and 24.5, below both PIT and No-Interaction despite strong MO3D semantic comparison.The results indicate that compressing local geometry into object-wise slots loses the spatial correspondence needed for mating, even with 32 micro-tokens.Source: §C.2 · pp. 16–17, Table 11 · p. 17

C.3. Training Strategy

  • 1-Stage PIT scores 66.6 Selection and 56.0 Delta Captioning, but MO3D Positional falls from 56.3 to 47.6 and Comparative from 33.8 to 24.5 versus 2-Stage.The authors conclude that Phase 1’s broad 3D-text pre-alignment is essential to prevent task-specific shortcuts and support generalized comparison.Source: §C.3 · p. 17

C.4. Robustness to Object Count (Scaling to 4–5 Objects)

  • Naive scaling yields 32% and 14% accuracy on queries about the 4th and 5th objects, while test-time coarse-to-fine filtering reaches 50% and 51%.The training-free procedure extracts referenced objects from ordinal terms, adds top-2 CLIP [43] candidates, and remaps ordinals before inference.Source: §C.4 · p. 17

D.1. Impact of Two-Turn Conversational Reasoning

  • Under the single-turn protocol, Multi-3DLLM achieves 37.1% Shape Mating Selection accuracy; the two-turn protocol separates selection from rationale generation.The combined-output objective lets long rationale tokens dominate cross-entropy loss, potentially penalizing short selection tokens; two-turn prompting decouples these outputs.Source: §D.1 · pp. 17–18

D.2. Transfer to Real-World Scans

  • With point densities matched, zero-shot real-world selection accuracy is 36.3% versus a 25% chance level, and fine-tuning on 5K samples raises it to 62.0%.LLaVA-7B remains near 25% across settings; ScanObjectNN’s roughly 2,048 points motivate density matching against an encoder pretrained on 8,192 points.Source: Table 12 · p. 18, §D.2 · p. 18

E.1. LLM-based Evaluation Prompts

  • The evaluation prompts use visual grounding for MO3D semantic accuracy, component recall and contradiction checks for Delta Captioning, and logical consistency for reasoning accuracy.In a human audit of 300 responses, the LLM judge received 91.0% unanimous support and 98.0% support from at least one of three auditors.Source: §E.1 · p. 18

E.2. Standard NLP Metrics

Standard NLP metrics can diverge from semantic evaluation in these tasks: fluent text may be geometrically incorrect, while correct descriptions may differ from ground-truth wording. The section therefore underscores the need for LLM-based metrics alongside standard NLP measures.

E.2. Standard NLP Metrics

  • Generative tasks are evaluated with BLEU-4, ROUGE-L, METEOR, and SimCSE.Source: §E.2 · p. 18
  • In Shape Mating, LLaVA reaches 67.65 SimCSE despite near-zero Selection accuracy, while w/o PIT slightly leads Multi-3DLLM on MO3D Positional BLEU-4 but trails on Semantic Accuracy (M).The discrepancies suggest that plausible-sounding text can be geometrically wrong, and that correct descriptions may differ from the ground-truth wording.Source: §E.2 · p. 19
  • These metric discrepancies underscore the need for LLM-based semantic metrics to benchmark geometric correctness accurately.The passage characterizes the Shape Mating outputs as “hallucinated fluency” and suggests the w/o PIT baseline relies on safe linguistic patterns.Source: §E.2 · p. 19

F. Additional Qualitative Results

The section presents qualitative comparisons on MO3D and the two mini-applications, illustrating examples and a stated failure case. The examples cover object positioning, comparison, holistic reasoning, shape mating, and change captioning.

F. Additional Qualitative Results

Related papers

Loading 2608.15710v1…