PaperlayerView exact revision on arXiv

Source-linked AI summary

An overview of 3D Vision-Language Models

Márcus Lobo, Vitor Matias, Afonso Paiva, Jeová Farias, Tiago Novello, Moacir Ponti

arXiv:2609.05583v1cs.CV

TL;DR

Traditional 3D learning is often tied to fixed task labels, while prior surveys do not jointly cover the main 3D vision-language components. This tutorial reviews 3D representations and alignment methods, then organizes model paradigms that progress toward open-vocabulary representations and generative language interfaces.

  • Problem

    Traditional 3D models typically map an input to a category from a fixed taxonomy and are not directly connected to natural-language semantics.

    Source: §I · p. 1

  • Method

    The tutorial reviews 3D representations and cross-modal alignment, organizing existing approaches into five paradigms from closed-set recognition to 3D VLLMs.

    Source: §I · p. 1, §II · p. 2, §III · p. 3

  • Results

    The taxonomy progresses from closed-set recognition through projection-based and joint-embedding models to 3DGS and 3D VLLMs, toward open-vocabulary representations and generative language interfaces.

    Source: §III · pp. 3–4

  • Takeaways & Limitations

    When CLIP image and text encoders transfer their visual-semantic structure to a 3D encoder, the objective enables zero-shot recognition and cross-modal retrieval.

    Source: C. Contrastive Alignment for 3D VLMs · p. 3

    Unifying 3D data and language remains constrained by the lack of scalable encoders that generalize across point clouds, meshes, Gaussian primitives, and structured 3D latents.

    Source: §V · p. 5

Abstract

from arXiv · show

Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

I. INTRODUCTION

Traditional 3D models commonly map inputs to fixed task-specific labels, while 3D VLMs align geometry with image and text embeddings. This tutorial reviews the representations, alignment methods, and model families behind that shift.

I. INTRODUCTION

  • Traditional 3D models typically predict categories from a fixed taxonomy, and their task-specific outputs are not directly connected to natural-language semantics.The tutorial contrasts this closed-set setting with open-world understanding.Source: §I · p. 1
  • CLIP-based 3D VLMs align geometric representations with pretrained image and text embeddings, supporting zero-shot classification, open-vocabulary comprehension, and cross-modal retrieval.Retrieval can use text or a 2D image to search for a 3D shape.Source: §I · p. 1
  • Existing surveys cover related topics separately but do not jointly connect 3D representations, CLIP-style alignment, 3D VLLMs, and language-aligned 3D Gaussian representations.The tutorial presents this combined scope as a gap in prior survey coverage.Source: §I · p. 1
  • The tutorial traces 3D VLM methods from closed-set recognition toward open-world understanding and reviews representations, contrastive alignment, and model families.Its roadmap includes distinguishing embedding-based 3D VLMs from generative 3D VLLMs.Source: §I · pp. 1–2

II. REPRESENTATIONS AND CROSS-MODAL ALIGNMENT

The alignment setting pairs a 3D object representation with an image and text description. Dedicated encoders and projection layers map these inputs into a shared, normalized embedding space.

II. REPRESENTATIONS AND CROSS-MODAL ALIGNMENT

  • The problem setting uses N triplets, each containing a 3D representation, a 2D image, and a textual description.The image may be rendered from the 3D representation from an arbitrary viewpoint.Source: §II · p. 2
  • Dedicated encoders process the 3D, image, and text inputs, with the 3D encoder architecture depending on the geometric representation.Examples include point clouds, multi-view images, meshes, and 3D Gaussian Splatting.Source: §II · p. 2
  • Projection layers map the encoded features to ℓ2-normalized 3D, image, and text embeddings in a common space.CLIP-based methods commonly initialize the image and text encoders from a pretrained vision-language model, while optimizing the 3D encoder to match its visual-semantic space.Source: §II · p. 2

A. 3D Inputs and Encoders

3D applications choose among geometric representations with different properties, and that choice determines the encoder used to extract 3D features. The section surveys multi-view images, point clouds, meshes, implicit fields, and 3D Gaussian Splatting.

A. 3D Inputs and Encoders

  • A 3D representation determines the encoder architecture, with choices spanning point clouds, meshes, 3D Gaussian Splatting, and multi-view renderings.These structures differ in geometric, appearance, and computational properties.Source: A. 3D Inputs and Encoders · p. 2, Figure 2 · p. 2
  • Multi-view images use a shared 2D encoder for each rendering, then aggregate view-level features into an object-level representation.The encoders can use convolutional or Vision Transformer backbones, including MVCNN.Source: A. 3D Inputs and Encoders · p. 2
  • Point clouds are native outputs of LiDAR and depth sensors, and their irregular, permutation-invariant structure motivates specialized point-wise, graph-based, transformer, and sparse-convolution encoders.These encoders differ in how they capture local neighborhoods and long-range geometric dependencies.Source: A. 3D Inputs and Encoders · p. 2
  • Meshes represent surfaces through vertices, edges, and faces, while MeshCNN operates on edges and MeshNet uses triangular-face features.Mesh connectivity poses challenges for learning operations.Source: A. 3D Inputs and Encoders · p. 2
  • Implicit fields represent surfaces as continuous occupancy or signed-distance fields, while 3D Gaussian Splatting uses explicit anisotropic Gaussians for real-time photorealistic rendering.Point-cloud encoders are also used for 3DGS tasks; TRELLIS encodes and decodes Gaussian splats into an embedding space for 3D content generation.Source: A. 3D Inputs and Encoders · p. 2

B. Geometric View of Contrastive Alignment

Contrastive alignment brings geometry, images, and language into a shared semantic space, where matching representations become more similar than nonmatching candidates. For 3D VLMs, symmetric cross-modal matching aligns 3D embeddings independently with image and text embeddings.

B. Geometric View of Contrastive Alignment

  • Contrastive learning maps modality-specific features into a shared semantic space, placing representations of the same object nearer than unrelated ones.This makes direct cross-modal comparison possible, including text-to-3D and image-to-3D retrieval and zero-shot classification.Source: B. Geometric View of Contrastive… · p. 3
  • Optimization raises a 3D anchor’s similarity to its paired image or text relative to other batch candidates, reducing their angular distance on the unit sphere.With ℓ2-normalized embeddings, inner product corresponds to cosine similarity; Figure 3 illustrates the geometric change.Source: B. Geometric View of Contrastive… · p. 3, Figure 3 · p. 3

C. Contrastive Alignment for 3D VLMs

  • For matched 3D, image, and text triplets, pairwise comparisons form 3D–image and 3D–text matrices, with diagonal matches and off-diagonal in-batch negatives.Figure 4 depicts the matrices and their positive and negative entries.Source: C. Contrastive Alignment for 3D VLMs · p. 3, Figure 4 · p. 3
  • Directional InfoNCE treats matching as an N-way cross-modal task, increasing the paired representation’s similarity against the remaining candidates in both directions.The temperature-scaled cosine similarity uses τ to control the concentration of the similarity distribution.Source: C. Contrastive Alignment for 3D VLMs · p. 3
  • When image and text encoders inherit CLIP’s visual-semantic structure, alignment transfers it to the 3D encoder, enabling zero-shot recognition and cross-modal retrieval.Methods differ in 3D representation, backbone, supervision, pair construction, and trainable components.Source: C. Contrastive Alignment for 3D VLMs · p. 3

III. A TAXONOMY OF 3D VISION-LANGUAGE MODELS

The tutorial organizes the evolution of 3D vision-language models into five paradigms, from closed-set recognition to 3D VLLMs. Across this progression, fixed task-specific outputs give way to open-vocabulary representations and generative language interfaces.

III. A TAXONOMY OF 3D VISION-LANGUAGE MODELS

  • The taxonomy contains five paradigms: closed-set recognition, projection-based adaptation, native joint embeddings, language-aligned 3DGS, and 3D VLLMs.Together, they organize the evolution of 3D vision-language models.Source: §III · p. 3
  • Across these paradigms, fixed task-specific outputs progressively give way to open-vocabulary representations and, more recently, generative language interfaces.Source: §III · p. 4

A. From Closed-set 3D Recognition to Language Alignment

Early 3D learning mapped geometric inputs to categories in a fixed taxonomy, using approaches such as rendered views or unordered point sets. Despite later advances in geometric encoding, outputs remained task-specific; vision-language pretraining instead uses a continuous semantic space.

A. From Closed-set 3D Recognition to Language Alignment

B. Projection-Based 3D VLM Adaptation

Projection-based 3D VLMs render geometry into views that frozen 2D encoders can process, providing an interface to pretrained models. Their performance depends on view selection, rendering quality, and the geometric information retained by projections.

B. Projection-Based 3D VLM Adaptation

  • Projection-based methods convert 3D geometry into rendered views that frozen 2D encoders can process.They provide an interface to pretrained 2D models, often without a dedicated 3D encoder.Source: B. Projection-Based 3D VLM Adaptation · p. 4
  • PointCLIP compares CLIP features from multi-view point-cloud depth maps with textual category embeddings.PointCLIP V2 uses more realistic projections and 3D-specific prompts, while EPCL explores frozen CLIP features for point-cloud understanding.Source: B. Projection-Based 3D VLM Adaptation · p. 4
  • Performance depends on viewpoint selection, rendering quality, and how much geometric information projections preserve.Source: B. Projection-Based 3D VLM Adaptation · p. 4

C. Joint Embedding Spaces

Joint-embedding methods train a dedicated 3D encoder to align native geometry with pretrained image and text representations. The paradigm spans tri-modal contrastive learning and scalable spatial encoders, but requires paired data, multimodal supervision, and costly pretraining.

C. Joint Embedding Spaces

  • Joint-embedding methods train a dedicated 3D encoder to align native geometry with pretrained image and text representations in a shared semantic space.Source: C. Joint Embedding Spaces · p. 4
  • ULIP established tri-modal alignment through 3D–image and 3D–text contrastive objectives, with later work scaling data, supervision, and architectures.OpenShape emphasized large, diverse datasets; Uni3D adapted scalable ViT backbones; ULIP-2 used automatically generated descriptions.Source: C. Joint Embedding Spaces · p. 4
  • Concerto and Utonia move toward general-purpose spatial encoders beyond joint-embedding alignment.Concerto combines 3D self-distillation with 2D–3D joint embedding and maps representations into CLIP’s language space; Utonia learns a unified self-supervised encoder across point-cloud domains.Source: C. Joint Embedding Spaces · p. 4
  • Joint-embedding methods preserve more geometric information than projection-based methods, but require large paired datasets, multimodal supervision, and costly contrastive pretraining.The supervision may be limited or noisy.Source: C. Joint Embedding Spaces · p. 4

D. 3D Gaussian Splatting Meets VLMs

Gaussian representations connect 3D structure and appearance with image supervision, supporting VLM alignment and retrieval. 3D VLLMs instead connect geometric representations to language models for generation and interaction, with challenges including hallucination and geometric fidelity.

D. 3D Gaussian Splatting Meets VLMs

  • 3D Gaussian Splatting represents scenes with renderable primitives encoding position, covariance, opacity, and color, coupling 3D structure to VLM-processed images.Source: D. 3D Gaussian Splatting Meets VLMs · p. 4
  • UniGS, TIGaussian, and CLIP-GS align Gaussian features with image and language embeddings using tri-modal matching, specialized latent branches, or efficient viewpoint-aware tokenization.CLIP-GS targets cross-modal retrieval.Source: D. 3D Gaussian Splatting Meets VLMs · p. 4
  • Gaussian representations couple geometry, appearance, and image supervision more closely than point clouds, but depend on reconstruction quality and can incur substantial storage or per-scene costs.Source: D. 3D Gaussian Splatting Meets VLMs · p. 4

E. 3D Vision Large Language Models

  • 3D VLLMs connect geometric representations to autoregressive language models, supporting captioning, question answering, instruction following, and spatial reasoning.Their architecture combines a 3D encoder or tokenizer, a learned projector, and a pretrained language backbone such as Qwen3-VL.Source: E. 3D Vision Large Language Models · p. 4
  • 3D VLLMs shift the objective from cross-modal comparison to language generation, enabling interaction while introducing challenges in hallucination, geometric fidelity, token efficiency, and 3D instruction data.PointLLM and ShapeLLM use a 3D encoder, projector, and language model; later work includes feature regularization, multi-object relations, discrete point-cloud tokens, and native 3D grounding.Source: E. 3D Vision Large Language Models · p. 4

IV. CURRENT TRENDS AND EMERGING APPLICATIONS

The tutorial surveys emerging 3D VLM applications spanning scene understanding, spatial reasoning, content generation, and embodied interaction. The described tasks range from object-level recognition and retrieval to scene-level question answering and manipulation.

IV. CURRENT TRENDS AND EMERGING APPLICATIONS

  • Emerging 3D VLM directions include scene understanding, spatial reasoning, content generation, and embodied interaction.Figure 5 presents these as five emerging directions, though the supplied discussion describes only some applications.Source: §IV · p. 5, Figure 5 · p. 5

A. Recognition, grounding and retrieval

  • Object-level tasks include zero-shot classification, cross-modal retrieval, and grounding, with outputs such as category scores, spatial regions, or ranked 3D assets.PointCLIP and PointCLIP V2 transfer CLIP through multi-view projections; ULIP, OpenShape, and Uni3D learn native language-aligned embeddings.Source: A. Recognition, grounding and retrieval · p. 5

B. Scene Understanding and Spatial Reasoning

  • Scene-level applications include 3D question answering, scene understanding, spatial reasoning, and text-, click-, or spatial-prompt interaction.Examples include Gaussian methods for segmentation, grouping, editing, and object isolation, as well as Multi-3DLLM and N3D-VLM for multi-object relations and geometric reasoning.Source: B. Scene Understanding and Spatial… · p. 5

C. 3D Generation

The tutorial reviews multimodal 3D generation and embodied interaction, then identifies limits in scalable cross-representation encoders and explicit geometry use by embodied VLA models.

C. 3D Generation

  • Text- or image-conditioned generation spans structured 3D outputs, while single-image reconstruction predicts geometry, texture, and layout.TRELLIS supports meshes, radiance fields, and 3D Gaussians; MeshGPT and MeshFlow are geometric generators rather than VLMs, and SAM 3D addresses image-based reconstruction.Source: C. 3D Generation · p. 5

D. Embodied AI

  • Most VLA systems rely primarily on 2D observations, while explicit 3D can add metric depth, object extent, and spatial relations for physical interaction.The tutorial notes challenges including action latency, limited 3D–language–action data, robustness to unseen environments, and preserving geometric accuracy during control.Source: D. Embodied AI · p. 5

V. CONCLUSION AND FUTURE DIRECTIONS

  • Scalable encoders that generalize across point clouds, meshes, Gaussian primitives, and structured 3D latents remain lacking, and embodied VLA models use explicit geometry only to a limited extent.The tutorial identifies efficient alignment, scalable 3D-language-action datasets, and robust transfer from synthetic assets to real-world sensor data as future-work directions.Source: §V · p. 5

Related papers

Loading 2609.05583v1…