PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

Sheng-Yu Huang, Jaesung Choe, Yu-Chiang Frank Wang, Cheng Sun

arXiv:2601.09575v1cs.CV

TL;DR

Open-vocabulary 3D scene understanding is constrained by embedding-based methods, costly annotations, and scene-specific training. OpenVoxel groups sparse voxels training-free, captions the resulting instances with VLMs and MLLMs, and retrieves targets through text-to-text reasoning. The authors report superior performance on OVS and RES, particularly for complex referring expressions.

  • Problem

    Embedding-based 3D methods limit arbitrary or nuanced queries, while human-annotated description-mask pairs and additional language-field training constrain use on new scenes.

  • Method

    OpenVoxel groups sparse voxels from SAM2 masks, constructs a captioned spatial scene map with VLMs and MLLMs, and performs direct text-to-text retrieval.

  • Results

    OpenVoxel outperforms previous studies on open-vocabulary semantic segmentation and referring expression segmentation, with particularly strong results reported for complex queries.

  • Takeaways & Limitations

    The framework provides training-free object-level grouping and caption-based scene understanding without human-annotated description-mask pairs or embedding alignment.

  • Takeaways & Limitations

    Grouping performance depends on instance separation in SAM2 masks, so sampling and merging settings must be adjusted carefully for each scene.

Abstract

from arXiv · show

We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.

1. Introduction

Open-vocabulary 3D understanding methods often rely on learned language embeddings and costly scene-specific annotations and training. OpenVoxel instead groups sparse voxels without training, captions object-level groups, and uses language reasoning for OVS and RES.

  • Neural rendering primitives support open-vocabulary 3D tasks, including semantic segmentation and open-vocabulary segmentation.
  • Learned CLIP- or DINO-aligned language fields limit arbitrary textual queries, while annotations and additional language-field training are labor-intensive and time-consuming.The cited passage reports more than one hour of additional training per scene.
  • OpenVoxel directly populates sparse voxel scenes with human-readable captions and performs text-to-text retrieval instead of learning a fixed text-embedding space.
  • Training-Free Sparse Voxel Grouping lifts and matches SAM2 per-frame segmentation maps to form view-consistent, object-level voxel groups.
  • OpenVoxel combines instance-level captions with LLM reasoning and reports superior performance on both OVS and RES, including complex and simple language queries.

2. Related Works

Prior language-aligned 3D representations use sentence-level embeddings and annotated sentence-object mask pairs for referring segmentation. OpenVoxel replaces these requirements with captions and direct MLLM text retrieval for more flexible complex-query reasoning.

  • Neural radiance fields provide high-quality novel-view synthesis but suffer from slow training and rendering.
  • ReferSplat aligns Gaussian features with sentence-level embeddings to predict object masks from natural-language sentence queries.
  • Existing methods rely on text encoders and human-annotated sentence-object mask pairs, limiting practicality on new scenes and nuanced or arbitrary queries.
  • OpenVoxel generates rich instance captions and performs direct MLLM text-to-text retrieval without annotation pairs or embedding alignment.

3. Preliminary

OpenVoxel starts from a pre-trained sparse voxel scene representation and organizes it into groups using SAM2-derived masks. It then renders group views and captions to construct a spatially indexed scene map for query-based retrieval.

  • The pre-trained sparse voxel model is reconstructed from multi-view images and supports efficient attribute assignment at 3D locations.
  • The framework includes training-free sparse voxel grouping and canonical scene map construction as distinct processing stages.
  • OpenVoxel builds a voxel group field from multi-view SAM2 masks, renders group images and masks, and records group positions and captions in a scene map.
  • Its rendered voxel colors follow classical volume rendering, where each voxel contributes according to opacity and accumulated ray weights.The contributing weight w_i(r) is also used during grouping.

4. Methodology

OpenVoxel groups sparse voxels into coherent 3D instances, constructs a captioned scene map, and supports direct natural-language retrieval without training language embeddings. Its pipeline progressively lifts 2D masks into 3D group fields, canonicalizes group captions, and matches canonicalized user queries.

  • Framework overview: OpenVoxel targets assigning descriptive language to sparse voxels for scene-map construction and open-vocabulary reasoning over category or referring-expression queries.The framework is designed to avoid gradient descent and latent-dimensional language representations.
  • Framework overview: The method comprises training-free sparse voxel grouping, canonical scene-map construction, and referring-query inference.These components respectively form instances, store captions with spatial information, and support natural-language querying.
  • Training-Free Sparse Voxel Grouping: Voxel group membership is represented by 3D coordinates pointing toward instance centers and updated in one training-free pass over all views.The design contrasts with methods that use gradient descent to train high-dimensional per-primitive features.
  • Training-Free Sparse Voxel Grouping: Group features encode each voxel’s instance centroid, while feature weights record confidence and a group dictionary stores unique instance centroids.These quantities are iteratively updated using instance information from each frame.
  • Canonical Scene Map Construction: For each group, DAM generates a detailed caption from masked multi-view images, and an MLLM such as Qwen3-VL rewrites it into a fixed canonical form.The canonical template includes category, appearance, function or part relation, and placement or relation, reducing subject ambiguity.
  • Referring Query Inference: Canonicalized queries are directly retrieved against the scene map, supporting both category-only OVS queries and detailed RES expressions without a learned embedding space.The query may include an optional target-view image and is rewritten into the same fixed template as group captions.

5. Experiments

OpenVoxel is evaluated on OVS and RES benchmarks using standard scene subsets and foundation models. It improves RES over ReferSplat and remains competitive on OVS, while qualitative examples show better localization of objects described by complex queries.

  • Evaluation settings: OpenVoxel is evaluated on LeRF-Mask and LeRF-OVS for OVS, and Ref-LeRF for RES across the listed indoor scenes.The RES subset contains sentence-level referring expressions paired with objects.
  • Implementation: OpenVoxel uses SAM2 for per-view segmentation and mask merging, DAM for captioning, and Qwen3-VL-8B-Instruct for caption refinement and target retrieval.The implementation is built on sparse voxel rasterization using PyTorch.
  • RES results: 13.2% improvement over the original ReferSplat result and 17.9% over ReferSplat* are reported on the Ref-LeRF RES evaluation.The authors attribute the reproduced baseline’s instability to overfitting sentence-level embeddings to seen descriptions.
  • OVS results: OpenVoxel performs competitively on both OVS benchmarks, including settings with simpler and less ambiguous queries.For LeRF-Mask, all current state-of-the-art methods, including OpenVoxel, exceed 70% mIoU.
  • Qualitative results: Qualitative RES examples show OpenVoxel selecting the correct toy pumpkin and sake cup when ReferSplat localizes partial or incorrect regions.These examples involve queries containing object attributes and spatial relationships.

Ablation Study.

The ablation study examines voxel grouping and language canonicalization, while the runtime comparison highlights OpenVoxel’s training-free efficiency. Canonicalizing captions and queries yields the best performance, and scene-map construction takes about three minutes.

  • Ablation study: The ablation begins without merging overlapping small masks or canonicalizing captions and queries, exposing retrieval difficulties from noisy groups and language misalignment.Explicit captioning and text-to-text retrieval alone remain insufficient in this baseline.
  • Ablation study: Canonicalizing the input query to match the caption form provides more precise alignment, reduces ambiguity, and produces the best performance.The study reports that the full set of designs is necessary for the strongest result.
  • Runtime comparison: About 3 minutes are required for voxel grouping and canonical scene-map construction, with per-query inference taking less than 1 second.The comparison is performed on a single RTX 5090 GPU.
  • Runtime comparison: OpenVoxel is reported as at least 10 × faster than other state-of-the-art methods in the runtime comparison.The comparison accounts for training needed to obtain semantic or language fields.

6. Conclusion

OpenVoxel groups and captions sparse voxel rasterization models to construct a canonical scene map and retrieve target objects from descriptive queries. The paper reports efficiency and capability through quantitative and qualitative experiments, while supplementary material identifies a separate limitations discussion.

  • 6. Conclusion: OpenVoxel groups sparse voxels by matching and merging per-view SAM2 masks in a training-free voxel-grouping process.The method operates on a pre-trained sparse voxel rasterization model.
  • 6. Conclusion: VLMs and MLLMs generate fixed-form captions for groups, whose positions and captions form the canonical scene map.The scene map supports subsequent query-based target retrieval.
  • 6. Conclusion: OpenVoxel performs referring-query inference by retrieving target objects from the scene map using an input descriptive query.The conclusion describes this as the final stage of the pipeline.
  • 6. Conclusion: Quantitative and qualitative experiments are used to verify OpenVoxel’s efficiency and capability.The supplementary material states that limitations are discussed in Section E.

A. Details

OpenVoxel’s implementation uses periodic SAM2-based mask merging, sampled views, and MLLM prompting to create consistent, view-independent captions and retrieve targets efficiently.

  • Mask merging: SAM2 re-prompting merges smaller masks when they lie largely inside larger masks, reducing noise and duplicated instance IDs.The procedure uses positive prompts for the group of interest, negative prompts for known groups, and merges a smaller mask when over 90% lies inside another.
  • Captioning and retrieval: DAM supplies detailed masked-region captions, while Qwen3-VL-8B performs canonical captioning, query refinement, and target retrieval.Canonical captioning uses darkened backgrounds and a red-dot highlight, then rewrites descriptions into structured category, appearance, function or part, and placement fields.
  • Limitations: The shared system prompts may be suboptimal because prompt design was not extensively explored, leaving prompt changes as a possible improvement.The authors use the same prompts for RES and OVS and report that users can modify them within the pipeline.
  • Efficiency: The pipeline samples at most 150 processed views per scene, merges masks every 1–5 steps, and uses eight frame-mask pairs per group.These choices reduce grouping, captioning, and retrieval time, while acknowledging that more visual information could improve performance.
  • Captioning and retrieval: Canonical captions must use a concrete subject noun, image-supported appearance details, optional function or part information, and view-independent placement.The prompt forbids generic fillers, unsupported guesses, camera-relative directions, and naming unmasked neighbors as the subject.
  • Captioning and retrieval: Caption generation scans masked frames for readable text and requires at least four distinct visual cues when supported.The rules prioritize color, material, texture, shape, subtype, printed content, state, function, part-of relations, and placement.

B.3. Evaluation protocols and results

The evaluation adapts voxel-to-point label transfer to account for OpenVoxel’s much denser voxel representation, then reports competitive semantic-segmentation results without ground-truth point priors.

  • Evaluation protocols: OpenVoxel evaluates point labels with Nearest, Majority of 25-NN, and Majority of 50-NN protocols because its voxel count exceeds ground-truth point-cloud counts.Scenes typically contain about 5M–10M OpenVoxel voxels versus roughly 50K–350K ground-truth points.
  • Evaluation protocols: The Nearest protocol assigns each point the label of its spatially nearest voxel, while neighborhood protocols use majority labels from nearby voxels.The described 50-NN variant uses the majority label among 50 spatially nearest voxels.
  • Results: Without ground-truth points as a prior, OpenVoxel outperforms all baselines in mIoU and is comparable in mAcc on ScanNet semantic segmentation.The paper presents this as evidence that the training-free approach may extend beyond OVS and RES.

C. Ablation study.

Ablations show that grouping, captioning, and MLLM choices materially affect RES, while the pipeline remains stronger than the reproduced ReferSplat baseline under alternative captioning models.

  • Segmentation model: Using SAM instead of SAM2 causes over-detailed fragmentations and about a 10% RES performance drop.The authors attribute the noisier grouping to SAM’s tendency to segment small fragments.
  • Captioning model: Either alternative captioning model still achieves higher RES mIoU than ReferSplat’s reported 29.2% and reproduced 24.5% results.Osprey’s inconsistent per-frame captions hinder summarization, whereas direct Qwen3-VL-8B captioning remains feasible for RES.
  • MLLM: Qwen3-VL-2B produces poor canonical captions and yields 9.98% mIoU, while Qwen3-VL-8B performs best under the same prompts.The smaller model tends to repeat words from raw captions, making target localization difficult.

D. Qualitative results

Qualitative RES examples show OpenVoxel retaining complex query information more effectively than ReferSplat, although view-dependent queries can retrieve an extra target.

  • RES comparisons: ReferSplat sometimes follows only a query’s object noun or color, whereas OpenVoxel locates the target using additional spatial and descriptive clues.Examples include distinguishing a smooth container near a sheep doll and a yellow countertop from nearby distractors.
  • Qualitative examples: Figures 8 and 9 provide qualitative RES results for the Teatime and Kitchen scenes.The captions identify these as qualitative results on the corresponding scenes.
  • RES comparisons: For view-dependent descriptions, OpenVoxel may retrieve two targets instead of one, but the correct target remains included.The authors present this as evidence of potential capability for view-specific tasks rather than as exact single-target retrieval.

E. Discussions and Limitations.

OpenVoxel’s training-free grouping is sensitive to scene-specific parameters and instance separation, requiring careful adjustment. Its instance-level grouping also limits segmentation of queried parts within larger objects.

  • OpenVoxel’s grouping process is relatively sensitive to parameters compared with end-to-end generalizable approaches.
  • ReferSplat struggles with additionally created queries for unseen target objects, even within a scene on which it was optimized.These queries lack ground-truth masks because they do not appear in the original Ref-LeRF annotations.
  • Frame sampling, merging frequency, and SAM2 parameters must be customized or adjusted carefully because instance separation affects OVS and RES performance.The authors report that sampling one frame per 3–5 frames and merging once per 3 steps generally work well, with reruns taking about 1 minute per scene.
  • Because grouping occurs before captioning and retrieval at the instance level, a query for part of a larger object segments the whole object.The paper gives the camera flashlight as an example of a queried part that would yield the complete camera segment.
Loading 2601.09575v1…