PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Single Object: Learning 3D Relations with Large Language Models

Kohsuke Ide, Ryousuke Yamada, Yue Qiu, Xianzheng Ma, Yoshihiro Fukuhara, Hirokatsu Kataoka, Yutaka Satoh

arXiv:2608.15710v1cs.CVcs.AIcs.CL

TL;DR

Existing 3D-LLMs are not designed for fine-grained comparison and relational reasoning across multiple objects. This paper introduces MO3D and Multi-3DLLM with application-driven benchmarks, and reports that the resulting model outperforms prior methods across multi-object and geometric tasks while improving single-object classification.

  • Problem

    Existing 3D-LLMs lack architectures and instruction data for fine-grained geometric comparison and relational reasoning across multiple objects.

  • Method

    The paper introduces MO3D, Multi-3DLLM with minimal patch-level adaptation, and Shape Mating and Change Captioning benchmarks for multi-object geometric reasoning.

  • Results

    Multi-3DLLM outperforms all baselines across MO3D metrics and application-driven tasks, while zero-shot single-object classification improves from 50.7% to 54.2%.

  • Takeaways & Limitations

    The framework extends 3D-LLMs toward multi-object comparison and demonstrates positive transfer to standard single-object and point-wise benchmarks.

  • Takeaways & Limitations

    The pipeline inherits potential biases from existing captions, while scaling to more objects and real-world scans raises token-budget and point-density challenges.

Abstract

from arXiv · show

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

1. Introduction

The paper targets 3D-LLMs’ inability to perform fine-grained comparison and relational reasoning across objects, proposing MO3D, Multi-3DLLM, and application-driven mini-applications to address this gap. Experiments show existing models struggle on geometry-aware multi-object tasks, while the proposed approach achieves strong performance and positive transfer to single-object benchmarks.

  • Motivation: Existing 3D-LLMs largely describe isolated objects or use coarse scene semantics, leaving fine-grained cross-object geometric relationships insufficiently modeled.Object-centric training data lacks collective or comparative descriptions, while scene-level processing can discard local geometry needed for precise comparison.
  • Dataset and benchmarks: MO3D provides large-scale instruction data for multi-object 3D reasoning, alongside Shape Mating and Change Captioning benchmarks.MO3D groups semantically related point-cloud instances and uses multi-view renderings with hierarchical constraints to prompt relational reasoning.
  • Model: Multi-3DLLM augments PointLLM with a lightweight patch-interaction transformer that captures cross-object, patch-level dependencies while preserving pretrained alignments.It concatenates patch tokens from all objects and applies a shallow, scalar-gated self-attention block, providing an architecture-agnostic minimal modification.
  • Results: State-of-the-art 3D-LLMs and competitive 2D-VLMs perform poorly on MO3D’s open-ended relational tasks, exposing limitations in multi-object settings.The paper attributes the proposed model’s gains to data that forces resolution of subtle geometric contrasts rather than reliance on textual co-occurrence.
  • Results: 50.7% to 54.2%: zero-shot single-object classification improves after training, indicating positive transfer without forgetting.Mixed-data training also yields strong performance on Shape Mating, Change Captioning, and standard single-object and point-wise benchmarks.

2. Related Work

Prior 3D-LLM research has largely followed an object-centric paradigm, while scene-level and 2D-VLM approaches provide multi-entity reasoning but lose fine-grained geometric fidelity. This work targets the gap between local patch-level detail and multi-object comparison, motivated by the documented difficulty of 2D-VLMs on geometry-centric 3D tasks.

  • Object-Centric 3D-LLMs: Object-centric 3D-LLMs established direct point-cloud-to-LLM pipelines and pursued universal object understanding, embodied interaction, and training efficiency.PointLLM established the direct pipeline, ShapeLLM introduced a new 3D encoder, and MiniGPT-3D leveraged 2D knowledge for efficient training.
  • Object-Centric 3D-LLMs: Object-level pooling in scene-level models smooths local patch details needed for intrinsic geometric comparison, limiting fine-grained geometric analysis.The proposed direction combines object-centric patch-level fidelity with scene-level multi-object capability.
  • Multi-image and Multi-entity VLMs: Although 2D tasks established comparison through change captioning and compositional reasoning, 2D-VLMs lack direct access to the geometric fidelity required by strict 3D comparison.The supplied passage characterizes 2D-VLMs as strong at semantic or contextual comparison from projections but insufficient for geometry-centric tasks.

3. MO3D: Multi-Object in 3D Dataset

MO3D extends 3D-LLMs from single-object description to relational multi-object reasoning through an instruction dataset built from Objaverse-Cap3D. Its curriculum combines positional, comparative, and holistic understanding, with quality control and downstream geometric benchmarks for shape mating and change captioning.

  • Dataset Overview: MO3D is an instruction-tuning dataset for training and evaluating relational understanding through multi-object comparison, shape mating, and change captioning.It is built on Objaverse-Cap3D, pairing Objaverse 3D objects with detailed Cap3D captions.
  • Dataset Construction: The construction pipeline extracts caption attributes into six comparison categories, then uses semantically related object groups, captions, and point-cloud renderings to generate GPT-4 QA pairs.Qwen2-72B-Instruct performs attribute extraction before GPT-4 generates questions grounded in both text and visual inputs.
  • Task Curriculum: The dataset organizes comparison ability into positional, comparative, and holistic tasks that form a progressively structured curriculum.Positional understanding uses relative order and context-dependent references; comparative understanding analyzes shape, structure, and spatial properties; holistic understanding requires binary Yes/No judgments across the object set.
  • Dataset Statistics and Quality Control: 70k final examples are split into training and test sets, with near-50:50 Yes/No balance in the holistic subset and 81.0% unanimous agreement among three annotators auditing 500 test QA pairs.Reported disagreements mainly arose from intrinsic ambiguities, including subjective perceptions.
  • Mini-App A: Shape Mating: The Shape Mating benchmark tests geometric compatibility by asking models to identify a mateable pair among three component point clouds and explain the geometric rationale.It targets applications in robotics and industrial design and uses meshes from Thingi10K split with Neural Shape Mating’s procedural mesh-cutting process.
  • Mini-App A: Shape Mating: Shape Mating includes Verification, a binary Yes/No classification task, and Delta Captioning, a generative task describing geometric edits between anchor and positive models.Verification uses the anchor, instruction, and either a positive or negative candidate; Delta Captioning uses the anchor and positive models.

4. Multi-3DLLM Architecture

Multi-3DLLM extends PointLLM to independently encode N objects, concatenate their patch tokens, and inject cross-object relations with a Patch-Interaction Transformer. Its patch-level design preserves local geometry, while two-phase training adapts the model to multi-object relational tasks.

  • Architecture: Multi-3DLLM independently encodes each of N objects, projects each into patch-token sequences, and concatenates them into a set-wide representation.The concatenated representation contains NT tokens with dimension d.
  • Patch-Interaction Transformer: The PIT block performs self-attention jointly over all concatenated patch tokens, injecting cross-object relations before LLM processing.A scalar-gated residual controls the PIT contribution through γ.
  • Patch-Interaction Transformer: Initializing γ ≈0 preserves pre-trained alignments, enabling cross-object relation learning without catastrophic forgetting of foundational single-object knowledge.The residual formulation blends the original representation with the PIT-transformed representation.
  • Interaction Granularity: Patch-level interaction preserves fine-grained local geometry, unlike scene-level approaches that compress each object into a single vector.This design supports the complex geometric tasks motivating multi-object instruction tuning.
  • Training Procedure: Training first aligns features with frozen point-cloud encoder and LLM parameters, then jointly fine-tunes the projector, PIT block, and LLM on ≈150K mixed-task pairs.The mixture samples ≈63K MO3D, ≈44K SM, and ≈40K CC pairs; MO3D contains ≈70K pairs across all splits.

5. Experiments

Experiments evaluate Multi-3DLLM on MO3D, two geometry-focused mini-applications, and zero-shot ModelNet40 classification. Multi-3DLLM outperforms baselines on relational tasks, while patch-level interaction improves comparison and fine-tuned models avoid positional bias and forgetting.

  • MO3D: Multi-3DLLM consistently outperforms all baselines across four MO3D metrics, scoring 56.3 on Positional (M) and 81.7 on Holistic (B).ShapeLLM, the best 3D-LLM baseline, scores 22.6 and 49.8 on those metrics, respectively.
  • MO3D: 33.8 on Comparative (M) is nearly three times ShapeLLM’s score, demonstrating especially strong performance on open-ended comparison.The supplied results identify Comparative (M) as the task with the most pronounced gap over the next-best 3D baseline.
  • Mini-applications: 37.1% Selection (S) on Shape Mating exceeds every baseline, while baselines remain below the 25% four-way chance level.Baseline Reasoning scores are also described as near-zero, supporting the need for geometrically aware 3D-LLMs.
  • Zero-shot classification: 2.1 accuracy for M=1 in the 3-Input task shows PointLLM w/ MI’s positional bias, whereas fine-tuned models achieve stable positional accuracy and avoid catastrophic forgetting.The comparison indicates that multi-object positional grounding must be learned rather than added only through multi-input capacity.
  • Ablations: 10.8 points on Positional (M) and 12.0 points on Comparative (M) are the gains of Ours (PIT) over No-Interaction on MO3D.The ablation attributes these improvements to patch-level cross-object mixing, which is described as crucial for relational understanding.
  • Ablations: 25.0 versus 34.4 Selection accuracy and 23.7 versus 36.7 Rationale accuracy show Object-Level’s poorer SM performance than No-Interaction.These ablations support the value of patch-level interaction for the mini-application.

6. Conclusion

The paper extends 3D-LLMs from single-object descriptions to multi-object comparison and application-driven geometric tasks, introducing MO3D and Multi-3DLLM as the core benchmark and model.

  • 6. Conclusion: The framework expands conventional 3D-LLMs to multi-object point clouds for comparison, shape mating, and change captioning.It introduces MO3D as a benchmark and Multi-3DLLM with holistic mixture training and minimal patch-level adaptation.
  • 6. Conclusion: Multi-3DLLM significantly outperforms prior methods, including PointLLM, on the MO3D benchmark.The conclusion attributes this result to holistic data-mixture training and minimal patch-level adaptation.
  • 6. Conclusion: The model demonstrates practical utility on complex application-driven tasks, including shape mating and change captioning.These tasks extend evaluation beyond conventional single-object 3D-LLM capabilities.

Supplementary Material · A. Implementation Details · Model Checkpoints

The system is initialized from publicly available pretrained models in the PointLLM framework, combining Vicuna-7B-v1.5, Point-BERT, ULIP2 pretraining, and OpenCLIP ViT-L/14.

  • Model Checkpoints: The model checkpoints use Vicuna-7B-v1.5 as the language model, Point-BERT pretrained via ULIP2 as the point-cloud encoder, and OpenCLIP ViT-L/14 for classification and grouping.These components are publicly available pretrained models from the PointLLM framework.

A.1. Training Hyperparameters · Hardware · B. Dataset Generation Details

The paper specifies a two-stage training setup using AdamW, cosine learning-rate scheduling, and a 0.03 warmup ratio. Training ran on 8 NVIDIA H200 GPUs, with 70 minutes for feature alignment and 12 hours for holistic-mixture training; no dataset-generation details are provided in the supplied passages.

  • A.1. Training Hyperparameters: Training used a two-stage strategy whose hyperparameters are detailed in Table 5.The supplied passage does not reproduce Table 5’s individual hyperparameter values.
  • A.1. Training Hyperparameters: All models used the AdamW optimizer.
  • A.1. Training Hyperparameters: A cosine learning-rate scheduler was used for all models.
  • A.1. Training Hyperparameters: The training configuration used a warmup ratio of 0.03.
  • B. Dataset Generation Details: The supplied passages contain no dataset-generation details for section B.
  • Hardware: Training ran on 8 NVIDIA H200 GPUs, each with 140GB of VRAM.
  • Hardware: 70 minutes were allocated to feature alignment in Phase 1.
  • Hardware: 12 hours were allocated to training on the holistic mixture of MO3D, Shape Mating, and Change Captioning datasets.

B.1. MO3D Dataset … C.1. Baselines & Input Format Fairness

The paper builds balanced, comparison-focused 3D benchmarks through structured object grouping, targeted generation, hard negatives, and leakage-free evaluation. It also standardizes multi-object inputs for fair baseline comparisons while emphasizing genuine geometric reasoning.

  • B.1. MO3D Dataset: MO3D defines six semantically distinct reasoning categories from 70k Objaverse-Cap3D captions using Qwen2-72B-Instruct, manual grouping, and curation.These categories form the foundation of the balanced generation pipeline.
  • B.1. MO3D Dataset: MO3D forms groups with CLIP ViT-L/14 embeddings and Faiss nearest neighbors, using two- or three-object groups to support complex relational concepts.Three-object groups enable identifying semantic outliers or majority properties, which pairwise comparisons cannot express.
  • B.1. MO3D Dataset: GPT-4 generates visually grounded QA pairs under task-specific constraints, weighted category coverage, and post-hoc balance correction.Object order is randomly shuffled to prevent positional biases, and generation prioritizes geometric understanding over simpler visual cues.
  • B.2. Mini-App A: Shape Mating Details: Shape Mating uses Thingi10K meshes and five cut geometries, with cut shell operations preserving surface-only structure for Point-BERT-compatible inputs.The five geometries are Planar, Sine, Square, Pulse, and Parabolic.
  • B.2. Mini-App A: Shape Mating Details: Shape Mating constructs four-choice scenarios with valid pairs and targeted decoys, while structured error tags support natural-language explanations of non-mating cases.Error types include object mismatch, phase mismatch, and same-side topological incompatibility.
  • B.3. Mini-App B: Change Captioning Details: Change Captioning uses Anchor-Positive-Negative triplets from ShapeTalk and hard-negative sampling that selects challenging alternatives sharing the same Anchor.The contrastive construction makes verification instances more difficult than random negative sampling.
  • B.4. Data Splitting: All benchmarks use graph-based leakage-free splits that assign connected components of related 3D assets atomically to Train or Test.Edges connect assets appearing together in samples or sharing a source object, avoiding leakage from shared geometries or grouping scenarios.
  • C.1. Baselines & Input Format Fairness: For fair baseline comparisons, object-centric 3D-LLMs receive individually normalized point clouds concatenated with fixed-margin axis separation, preserving genuine 3D geometry processing.This formatting adapts single-point-cloud models such as PointLLM and ShapeLLM to multi-object tasks without relying only on language biases.

C.2. Architectural Ablation: Interaction Mechanics · C.3. Training Strategy · C.4. Robustness to Object Count (Scaling to 4–5 Objects)

The ablations show that fine-grained patch-level interaction is necessary for geometric correspondence, while two-stage training provides essential broad 3D-text grounding. Zero-shot scaling beyond the standard 2–3 objects exposes substantial positional-QA degradation under fixed-token compression.

  • C.2. Architectural Ablation: Interaction Mechanics: Table 11 compares mean- and max-pooling strategies across object-, micro-, and patch-level interaction granularities.Micro-Token uses 32 representative tokens per object, whereas PIT uses full patch-level interaction.
  • C.2. Architectural Ablation: Interaction Mechanics: Object-level interaction fails because compressing local geometry into object-wise slots creates an architectural bottleneck, not because of the pooling operation.Even 32 micro-tokens lose the spatial correspondence required for shape mating, and object-level performance is worse than the No-Interaction baseline.
  • C.3. Training Strategy: Phase 1 aligns point-cloud features with the LLM embedding space using 660K Objaverse-Cap3D brief-description instructions while training only the projector.Phase 2 then fine-tunes the projector, PIT block, and LLM while keeping the point encoder frozen.
  • C.3. Training Strategy: The 1-Stage variant learns limited Shape Mating patterns but fails to ground the diverse, open-ended concepts required for MO3D.It initializes the projector randomly and trains the full model directly on the holistic data mixture.
  • C.3. Training Strategy: The 2-Stage strategy makes Phase 1 essential by grounding the model in broad 3D-text semantics and preventing task-specific shortcuts.This foundation enables the generalized comparison capabilities reported in MO3D.
  • C.4. Robustness to Object Count (Scaling to 4–5 Objects): 32% and 14% positional-QA performance result when naively scaling to 4 and 5 objects, respectively, through Micro-Token compression under a fixed token budget.The decline occurs on queries referencing the 4th or 5th objects, motivating evaluation beyond the standard n ∈{2, 3} setting.

D. Advanced Analysis on Shape Mating … E.1. LLM-based Evaluation Prompts

The appendix analyzes Shape Mating under conversational reasoning and real-world transfer, then describes GPT-4o-mini prompts for semantic evaluation. Two-turn prompting and matched point densities substantially improve selection accuracy, while LLaVA-7B remains at chance.

  • D. Advanced Analysis on Shape Mating: Multi-3DLLM’s strict single-turn Shape Mating protocol required pair selection and a detailed geometric rationale in one response.The output combined a categorical pair choice with explanatory text.
  • D.1. Impact of Two-Turn Conversational Reasoning: 37.1% Selection accuracy was achieved by Multi-3DLLM under the constrained single-turn setting.This result corresponds to the protocol requiring selection and rationale together.
  • D.1. Impact of Two-Turn Conversational Reasoning: Combining selection and rationale creates objective imbalance because long rationale tokens dominate CE loss and penalize short selection tokens.The appendix therefore evaluates a two-turn conversational approach that decouples the task.
  • D.2. Transfer to Real-World Scans: Shape Mating zero-shot transfer was evaluated on the real-world scanned datasets ScanObjectNN and OmniObject3D.This tests whether the task extends beyond synthetic procedural data.
  • D.2. Transfer to Real-World Scans: Approximately 2,048 points in ScanObjectNN create a density bottleneck relative to the encoder’s 8,192-point pretraining, motivating OmniObject3D evaluation.The evaluation augments the transfer study to address mismatched point densities.
  • D.2. Transfer to Real-World Scans: 36.3% zero-shot and 62.0% after fine-tuning on 5K real-world samples were achieved when point densities were matched.The zero-shot result exceeds the 25% chance level.
  • D.2. Transfer to Real-World Scans: LLaVA-7B stayed at approximately 25% across all settings, matching the 25% chance rate.This contrasts with Multi-3DLLM’s zero-shot and fine-tuned transfer results.
  • E.1. LLM-based Evaluation Prompts: GPT-4o-mini evaluates open-ended MO3D responses as Correct or Incorrect using semantic equivalence, reasoning logic, and visual grounding.The evaluator receives ground-truth text and the model response, avoiding reliance on exact string matching.

E.2. Standard NLP Metrics

The evaluation reports BLEU-4, ROUGE-L, METEOR, and SimCSE for generative tasks across Mini-Applications and MO3D. Standard metrics can diverge from semantic performance, with high language similarity masking near-zero task accuracy.

  • Standard NLP Metrics: BLEU-4, ROUGE-L, METEOR, and SimCSE are reported for all generative tasks.These standard NLP metrics are used for the Mini-Applications and MO3D evaluations.
  • Metric Discrepancies: 67.65 SimCSE for LLaVA in Shape Mating coexists with near-zero Selection accuracy, revealing hallucinated fluency rather than reliable geometric reasoning.This discrepancy shows that plausible-sounding generations can score well on standard language metrics while failing the task’s semantic evaluator.

F. Additional Qualitative Results · System Prompt for MO3D QA Generation

The paper presents qualitative comparisons across MO3D and mini-application benchmarks, while detailing prompt designs intended to improve dataset quality, robustness, and evaluation. Examples show strong task-specific reasoning alongside remaining holistic failures.

  • F. Additional Qualitative Results: Qualitative examples compare the model with state-of-the-art baselines on MO3D and application-driven benchmarks.Figures 16–20 cover MO3D, while Figures 21–24 cover Shape Mating and Change Captioning.
  • System Prompt for MO3D QA Generation: The MO3D generation prompt creates positional, comparative, and holistic question–answer pairs from object descriptions and supplementary multi-view images.It prioritizes visual evidence for visual attributes, uses descriptions for non-visual semantics, and trusts images when they contradict descriptions.
  • System Prompt for MO3D QA Generation: Strict keyword constraints are enforced for target categories including Geometry and Material, with the same logic applied to Color, Function, Taxonomy, and Style.This design is intended to produce high-quality, non-ambiguous questions.
  • System Prompt for MO3D QA Generation: Prompt templates diversify task phrasing and randomize anchor–candidate order to improve robustness and prevent positional bias.Shape Mating uses 15 instruction variations, while Change Captioning separates Verification and Delta templates and adjusts wording for object order.
  • F. Additional Qualitative Results: Molmo correctly describes the second object’s humanoid structure and walking pose in the positional example.The answer identifies the figure’s stylized body, limbs, and walking configuration, matching the ground truth.
  • F. Additional Qualitative Results: Baseline answers misidentify the compared objects’ colors, failing to match the first object’s blue-green colors and the third object’s green-brown camouflage.Molmo reports shared green-heavy colors, while MiniGPT-3D and PointLLM compare the wrong objects or omit the camouflage distinction.
  • F. Additional Qualitative Results: The holistic example exposes a failure case in which the model incorrectly generalizes a cylindrical shape to all objects.The ground truth states that only the second object is cylindrical; the first is a bottle and the third is a canister.
  • F. Additional Qualitative Results: Shape Mating examples require selecting compatible pairs and explaining complementary geometric interfaces.Ground-truth examples include pair (1,3), whose pulsed ridges interlock, and pair (2,3), whose stepped contours and mirrored normals nest together.
Loading 2608.15710v1…