PaperlayerView exact revision on arXiv

Source-linked AI summary

3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering

Changwoo Baek, Kyeongbo Kong

arXiv:2608.01185v1cs.CVcs.LG

TL;DR

3D VLMs produce thousands of spatially structured tokens, making efficient compression necessary while spatial aggregation alone leaves object-level imbalance unresolved. 3DZip combines voxelization, feature-diversity-guided anchor selection, and spatially constrained merging, outperforming existing compression methods across three benchmarks while retaining 94.7% of original performance with 128 tokens and running 1.92× faster.

  • Problem

    3D VLM token compression must address both overlapping spatial observations and persistent object-level token imbalance, which spatial aggregation alone cannot resolve.

  • Method

    3DZip uses coarse voxelization, feature-diversity-guided anchor selection via a Determinantal Point Process, and spatially constrained token merging.

  • Results

    3DZip consistently outperforms existing compression methods across three 3D question answering benchmarks, retaining 94.7% of original performance with 128 tokens and achieving 1.92× faster inference.

  • Takeaways & Limitations

    3DZip provides an efficiency–accuracy trade-off for scalable multi-view 3D VLM deployment.

  • Takeaways & Limitations

    Coarse voxelization may attenuate fine-grained cues for small objects, while fixed hyperparameters may be suboptimal across scenes with different complexity.

Abstract

from arXiv · show

Recent 3D vision-language models (3D VLMs) construct geometry aware tokens by projecting 2D visual features into world coordinates, enabling spatial reasoning for tasks such as 3D question answering. However, this design generates thousands of tokens per scene, resulting in substantial computational and memory overhead. While token compression has been extensively studied in 2D VLMs, existing approaches rely on semantic relevance or attention-based selection that overlook the structured spatial nature of 3D tokens. Moreover, redundancy in 3D representations cannot be resolved by spatial proximity alone, as object-level token imbalance persists even after spatial aggregation. To address this, we propose 3DZip, a three-stage token compression framework that first applies coarse voxelization to remove point-level redundancy, then selects anchor tokens based on feature-space diversity via a Determinantal Point Process, and finally merges remaining tokens under spatial constraints to preserve geometric coherence. Experiments on three 3D question answering benchmarks demonstrate that 3DZip consistently outperforms existing compression methods, retaining 94.7% of the original performance with only 128 tokens, achieving a $1.92\times$ faster inference speed.

1 Introduction

Projection-based 3D VLMs enable spatial reasoning but create point- and object-level token redundancy, with severe object imbalance persisting after spatial aggregation. 3DZip addresses these issues through three-stage compression that combines voxel aggregation, feature-diversity anchor selection, and spatially constrained merging while preserving strong reasoning performance.

  • Motivation: Projection-based multi-view aggregation creates overlapping tokens from repeated observations of identical physical surfaces.These repeated observations correspond to nearly identical spatial locations and can be partially mitigated through voxel-based aggregation.
  • Motivation: Object-level token allocation remains highly imbalanced after voxel aggregation, with a small subset of objects dominating the token budget.Spatially driven diversity strategies such as XYZ-DPP alleviate this issue only partially because tokens remain concentrated on limited objects.
  • Motivation: Feature-level diversity increases object coverage to 70%, producing a more balanced token distribution than spatial aggregation alone.The finding motivates explicitly addressing different redundancy sources in multi-view 3D representations.
  • Results: 94.7% of original performance is retained with only 128 tokens, while inference runs 1.92× faster across three 3D question answering benchmarks.The paper reports that 3DZip consistently outperforms existing token compression methods while maintaining strong reasoning performance.
  • Method: 3DZip combines voxel aggregation, feature-diversity–based anchor selection, and spatially constrained merging in a structured three-stage compression framework.The framework aligns token selection with redundancy induced by multi-view aggregation in geometry-aware 3D VLM representations.

2 Related Works

Research on 3D vision-language modeling has progressed from transformer-based and LLM-integrated point-cloud methods to projection-based approaches using pretrained 2D models. These methods improve 3D reasoning but face complex pipelines, information loss, and severe token-computation overhead, motivating spatially aware compression.

  • 3D Vision-Language Models: Transformer-based models such as 3D-VisTA align 3D scenes and text through pretraining, while LL3DA, LEO, Chat-Scene, and Robin3D integrate point clouds into LLMs.Robin3D further improves 3D LLMs through robust instruction tuning, but these approaches rely on specialized mechanisms for geometric modeling.
  • 3D Vision-Language Models: Projection-based methods such as 3D-LLM and Scene-LLM aggregate pretrained 2D features into 3D space but require complex pipelines and external segmentation.These requirements are computationally intensive and can cause information loss.
  • 3D Vision-Language Models: LLaVA-3D integrates 3D positional embeddings into 2D patches, enabling explicit 3D modeling without dedicated encoders while preserving the original LMM’s reasoning capabilities.However, it generates thousands of tokens per scene, creating substantial computational and memory overhead.
  • Token Compression: Existing VLM token-compression methods primarily target redundant 2D image-text tokens, often selecting tokens through LLM cross-attention distributions during decoding.Their assumptions and decoding dependence do not directly address structured 3D representations.
  • Token Compression: DTC incorporates depth for pruning but remains focused on multi-view 2D VLM settings, whereas 3DZip jointly considers feature-space diversity and geometric constraints.This design aims to preserve both semantic representativeness and spatial coherence in projection-based 3D VLMs.

3 Preliminary: Projection-based 3D VLMs

Projection-based 3D VLMs lift multi-view 2D visual features into a shared 3D coordinate system, combining visual features with learned positional embeddings to form geometry-aware tokens. These tokens are aggregated across views, and compression seeks to reduce their cardinality while preserving the semantic and spatial cues needed for downstream reasoning.

  • Token construction: Multi-view RGB-D observations are encoded into 2D visual tokens and back-projected into world coordinates using depth maps and camera poses.The resulting tokens are augmented with 3D positional embeddings to incorporate geometric information.
  • Token construction: Each geometry-aware 3D token combines a visual feature with a learnable positional embedding of its 3D coordinate.Its representation consists of the augmented feature and the corresponding 3D position.
  • Token aggregation: Aggregating tokens across views produces a flattened 3D token set with N tokens, each represented by an augmented feature and a 3D position.The notation denotes the resulting tokens as v_i = (f̃_i, p_i), for i = 1,…,N.
  • Compression objective: Compression constructs a smaller set with N′ ≪ N while preserving information required for downstream reasoning and reducing computational cost.The target is to maintain approximately equivalent language-model outputs, g(V, t) ≈ g(V′, t), while retaining semantic and spatial cues.

4 Method

3DZip compresses dense geometry-aware 3D tokens through three stages that progressively remove point- and object-level redundancy while preserving semantic diversity and geometric consistency. It voxelizes tokens, selects diverse feature-space anchors with a DPP, and spatially merges compatible remaining tokens.

  • Stage 1: Coarse Voxelization: Coarse voxelization partitions 3D space into axis-aligned voxels and mean-pools tokens within each voxel to reduce overlapping point-level redundancy.The resulting voxel token set has N_v much smaller than the original token count N.
  • Stage 2: Feature-Diversity Guided Anchor Selection: DPP-based anchor selection chooses K representative voxel tokens by maximizing feature-set diversity through det(L_A), using a greedy Cholesky approximation.Voxel features are normalized before constructing the cosine similarity kernel.
  • Stage 3: Spatially Constrained Token Merging: Each non-anchor token is assigned to its most similar anchor in feature space, then merged only if its grid-space distance is within the spatial threshold τ_g.Tokens that fail the spatial constraint are discarded, while assigned tokens update their anchor by mean aggregation.
  • Stage 3: Spatially Constrained Token Merging: The compressed set retains one token per selected anchor, yielding N′ = K much smaller than N before being provided to the language model for downstream reasoning.Anchor positions are preserved during feature aggregation.

5 Experiments

Experiments evaluate 3DZip against spatial, semantic, and 2D-derived compression methods across ScanQA, SQA3D, and OpenEQA. 3DZip consistently preserves performance under tight token budgets while delivering substantial computational savings, with feature diversity, coarse voxelization, and spatially constrained merging each supported by ablations.

  • Experimental Setup: 3DZip is compared with voxelization, DTC, FastV, SparseVLM, VisionZip, and VisPruner using LLaVA-3D as the projection-based backbone.The evaluation examines both 3D-aware methods and methods extended from 2D VLM settings.
  • Overall Results: Across 128, 64, and 32 tokens, 3DZip achieves the best performance and retains the highest relative performance.Voxelization can overlook semantic representativeness, while DTC degrades as the token budget becomes more constrained.
  • Category-wise Results: 3DZip achieves the highest average OpenEQA score at every budget, reaching 64.2 for attribute recognition at 128 tokens and 50.5 for object recognition at 32 tokens.It closely matches the uncompressed attribute-recognition baseline of 64.0 and achieves the largest object-recognition margin of +6.7 at 32 tokens.
  • Ablation Studies: Feature-space distance outperforms spatial distance for DPP anchor selection across all token budgets, including 52.8 versus 50.1 EM at 64 tokens.Feature dispersion selects semantically distinct objects, mitigates long-tail allocation bias, and improves scene-wide object coverage.
  • Ablation Studies: Removing coarse voxelization lowers 64-token EM from 52.8 to 51.8, confirming that multi-view redundancy should be reduced before feature-diversity anchor selection.The ablation supports balancing redundancy reduction with local geometric preservation.
  • Efficiency: 128 tokens reduce FLOPs from 9.18T to 0.90T and KV cache size from 722MB to 101MB, while EM falls only from 55.7 to 53.2.Relative to LLaVA-3D’s 1410 tokens, this is a 90.9% token reduction, 90.2% FLOPs reduction, and 86.0% KV-cache reduction.

6 Limitations and Future Work

3DZip achieves a strong efficiency–accuracy trade-off but remains limited by potential loss of fine-grained cues and scene-insensitive hyperparameter settings.

  • Limitations: Coarse voxelization may attenuate fine-grained cues for small objects by merging their tokens with nearby objects or background regions.This issue is analyzed in Supplementary Section B.8.
  • Limitations: The hyperparameters δ and τg are fixed across scenes rather than adapted to scene complexity.Scene size can affect object density and inter-object distances, motivating scene-adaptive settings.

7 Conclusion

3DZip is a geometry-aware token compression framework for projection-based 3D VLMs that addresses distinct point-level and object-level redundancy in multi-view 3D representations. It shows that spatial aggregation alone cannot resolve object-level redundancy.

  • Redundancy Analysis: 3DZip targets point-level redundancy from overlapping observations and object-level redundancy from repeated surface tokens of the same object.These are identified as two distinct redundancy types inherent to multi-view 3D representations.
  • Redundancy Analysis: Spatial aggregation alone is insufficient to address object-level redundancy in multi-view 3D representations.Repeated surface tokens of the same object can remain redundant even after spatial aggregation.
  • Framework: 3DZip combines three complementary stages to compress tokens in projection-based 3D VLMs.The passage explicitly introduces the framework as a three-stage approach and names coarse voxelization as its first stage.

Supplementary Material Overview … A.3 Additional Metrics

The supplementary material extends the main evaluation with additional models, dense captioning, metrics, qualitative results, analyses, and experimental details. Across the reported evaluations, 3DZip generalizes beyond LLaVA-3D and improves captioning-oriented performance under token compression.

  • Supplementary Material Overview: The supplement provides additional experimental results, analyses, and implementation details supporting the main paper.
  • A Additional Results: Additional sections cover model evaluation, dense captioning, qualitative results, redundancy, diversity, distance metrics, spatial constraints, robustness, failure cases, and scene-size effects.
  • A.1 Evaluation on Additional Models: Under identical token budgets, 3DZip consistently outperforms all baselines across both additional projection-based 3D VLM architectures.
  • A.1 Evaluation on Additional Models: 3DZip is evaluated beyond LLaVA-3D on Video-3D-LLM and SR-3D, using 20 input views per scene and differing visual encoding and token-construction pipelines.
  • A.2 Evaluation on 3D Dense Captioning: The dense captioning evaluation uses Scan2Cap’s validation split with 7,023 samples across 102 scenes and retains the original LLaVA-3D architecture and training procedure.
  • A.2 Evaluation on 3D Dense Captioning: 3DZip consistently outperforms existing token compression methods on 3D dense captioning across multiple token budgets.
  • A.3 Additional Metrics: Beyond exact match, ScanQA evaluation reports BLEU-4, METEOR, ROUGE-L, and CIDEr, with 3DZip consistently achieving the best performance across all metrics.

A.4 Additional Qualitative Results · B Additional Analysis · B.1 Point-Level Token Redundancy Analysis

The additional results show that 3DZip retains semantically diverse tokens while adding complementary spatial context, and that voxel aggregation reduces point-level duplication with a measurable purity–redundancy trade-off. Further analysis characterizes how redundancy is measured and how spatial selection affects object-level token allocation.

  • A.4 Additional Qualitative Results: 3DZip retains semantically diverse tokens across scenes while incorporating complementary context through spatially constrained merging.Additional qualitative examples are provided for SQA3D, OpenEQA, and ScanQA.
  • A.4 Additional Qualitative Results: Qualitative examples illustrate the 3D tokens retained by 3DZip across indoor scenes in SQA3D.The examples are shown in Fig. 5.
  • A.4 Additional Qualitative Results: Qualitative examples illustrate the 3D tokens retained by 3DZip across indoor scenes in OpenEQA.The examples are shown in Fig. 6.
  • A.4 Additional Qualitative Results: Qualitative examples illustrate the 3D tokens retained by 3DZip across indoor scenes in ScanQA.The examples are shown in Fig. 7.
  • B.1 Point-Level Token Redundancy Analysis: Point-level redundancy arises when overlapping views generate near-duplicate tokens for the same physical surface point, and Stage 1 reduces it through voxel aggregation.The analysis defines PLR using multi-view tracks formed from matched SuperPoint keypoints, excluding singleton tracks.
  • B.1 Point-Level Token Redundancy Analysis: PLR decreases from 4.74 to 3.09/2.50/2.25 for δ=0.1/0.2/0.3 m, confirming that voxelization reduces geometric duplication.PLR measures the average number of distinct voxels occupied by each repeatedly observed physical point and is independent of the final token budget.
  • B.1 Point-Level Token Redundancy Analysis: 52.8 EM at δ=0.2 m is the peak under a 64-token budget, while voxelization reduces PLR by about 47% and preserves 0.915 object purity.Object purity nevertheless drops from 0.975 to 0.847 across the evaluated voxel sizes, revealing a trade-off between aggregation and object mixing.
  • B.1 Point-Level Token Redundancy Analysis: Voxel-only allocation exhibits a pronounced long-tail pattern, while XYZ-DPP reduces extreme concentration but still assigns tokens to a limited subset of objects.The comparison covers 1,729 foreground object instances across 67 ScanNet scenes from the SQA3D test split.

B.2 Object-Level Token Allocation Analysis … C.1 DPP Algorithm

Across object-level allocation, ablations, robustness tests, and implementation details, the analyses show that feature diversity complements spatial processing to reduce redundancy while preserving geometric and fine-grained information. The accompanying studies identify cosine similarity, spatial constraints, scene-dependent settings, and greedy DPP selection as important components and characterize remaining limitations.

  • B.2 Object-Level Token Allocation Analysis: 266 tokens can accrue to the largest object, while most objects receive fewer than 10 under voxel-only allocation, revealing pronounced object-level redundancy.The analysis covers 1,729 foreground instances across 67 ScanNet scenes using 0.2m voxelization and ground-truth instance labels.
  • B.2 Object-Level Token Allocation Analysis: 64% coverage (1,121/1,729) and a maximum of 19 tokens per object show that Feature-DPP reduces object-level imbalance by selecting semantically distinct tokens.Feature-DPP replaces spatial similarity with cosine similarity over CLIP features, indicating that feature-level diversity is needed beyond spatial uniformity.
  • B.3 Comparison of Diversity Strategies: DPP is evaluated against FPS and MMDP to test whether gains arise from the diversity objective rather than from a particular selection algorithm.FPS greedily selects tokens with large feature-space distance, whereas MMDP maximizes minimum pairwise distance.
  • B.4 Distance Metric Analysis: Cosine similarity achieves the best results among ℓ1, ℓ2, and cosine metrics, suggesting normalized feature-vector direction better captures token relationships.The comparison concerns feature-space distance during anchor selection.
  • B.5 Effect of Spatial Constraints: Performance is relatively robust across τg values, while very small thresholds restrict contextual aggregation and reduce complementary information merged into anchors.τg limits allowable grid-space distance between anchors and assigned tokens, preventing spatially distant merges.
  • B.6 Comparison with 2D Diversity-based Token Compression: 3DZip consistently outperforms DivPrune across all token budgets on ScanQA, SQA3D, and OpenEQA, supporting spatial awareness as a complement to feature diversity.DivPrune uses an MMDP objective but does not incorporate spatial structure.
  • B.7 Robustness of Stage 2 to Stage-1 Aggregation Strategies: Adding DPP improves EM for ConceptFusion from 50.3 to 52.7 (+2.4), OpenFusion++ from 49.8 to 52.3 (+2.5), and voxel-mean aggregation from 49.8 to 52.5 (+2.7).The results use a shared LLaVA-3D backbone, lifted 3D tokens, and a 64-token budget, showing Stage 2 remains beneficial across Stage-1 aggregation strategies.
  • B.8 Fine-grained Analysis and Failure Cases; B.9 Effect of Scene Size on Hyperparameters: 3DZip achieves best overall EM (23.3) and preserves Local struct. performance (22.4), but remains worse than w/o Voxel on Small obj. and Fine detail (19.5/30.5 vs. 21.6/31.7).The remaining loss is mainly attributed to coarse voxelization, with Voxel-only showing low small-object purity (0.070). Small scenes favor δ=0.1m and τg=1, whereas large scenes favor δ=0.2m and τg=7 at 32 tokens.

C.2 DPP Runtime Analysis … SR-3D

The paper evaluates DPP-based token selection alongside experimental details for LLaVA-3D, Video-3D-LLM, and SR-3D. DPP selection reduces runtime overhead after voxelization, while the compared systems use distinct 3D token construction and compression pipelines.

  • C.2 DPP Runtime Analysis: 18 ms: DPP-based selection takes half the 36 ms required by DTC’s iterative selection under the same token budget, K = 128.The lower selection time produces smaller overhead relative to total inference time.
  • D Experimental Details: Voxelization before anchor selection significantly reduces the number of tokens, keeping DPP-based selection overhead modest in practice.
  • D.1 Experimental Setup: LLaVA-3D projects multi-view RGB-D observations into a shared 3D coordinate system and produces geometry-aware 3D tokens from CLIP visual features and 3D positional embeddings.The positional embeddings are derived from depth and camera parameters.
  • LLaVA-3D: LLaVA-3D uses Voxelization Pooling for spatial aggregation and FPS Pooling for fixed-size token selection.The experiments use the LLaVA-v1.5-7B backbone and adopt Voxelization Pooling as the baseline token construction stage.
  • Video-3D-LLM: Video-3D-LLM models 3D scenes as video sequences, adds sinusoidal encodings from back-projected depth coordinates, and selects frames with greedy Maximum Coverage Sampling.It uses a SigLIP visual encoder and a Qwen2-7B language backbone.
  • SR-3D: SR-3D is built on NVILA-Lite-8B and supports single-view and multi-view inputs through a canonical positional feature for 3D reasoning.Its MLP projector contains a 3×3 convolutional downsampling layer, so token compression is applied after the projector stage.

D.2 Dataset Details … D.3 LLM-based Evaluation Prompt

The evaluation uses SQA3D, ScanQA, and the publicly comparable ScanNet subset of OpenEQA, with dataset sizes, scene coverage, question characteristics, and evaluation protocols specified. OpenEQA responses receive 1–5 LLM-based scores by comparison with ground-truth and acceptable answers.

  • D.2 Dataset Details: SQA3D’s test split contains 3,519 questions across 67 ScanNet scenes, with each question grounded in an agent’s 3D position and orientation.Questions cover six categories: What, Is/Are, How many, What color, Which, and Other.
  • SQA3D: SQA3D questions span six categories: Is/Are (33.1%), What (26.6%), How many (12.3%), Other (12.1%), Which (10.0%), and What color (6.0%).
  • OpenEQA: OpenEQA includes ScanNet with 1,079 questions across 89 scenes and HM3D with 557 questions across 63 scenes.Because official HM3D episode trajectory states are unavailable, evaluation uses only the ScanNet subset for fair comparison.
  • OpenEQA: The HM3D OpenEQA subset requires official episode trajectory states to render frames from its benchmark camera viewpoints, so it is excluded from evaluation.
  • D.3 LLM-based Evaluation Prompt: The OpenEQA evaluation prompt asks an AI assistant to compare a response with the question, correct answer, and additional correct answers, then output one integer from 1 to 5.A score of 5 indicates a perfect match with the answer or an extra acceptable answer, while 1 indicates complete disagreement.
  • ScanQA: ScanQA’s validation split contains 4,675 questions across 71 ScanNet scenes about object attributes, spatial relationships, or scene properties.BLEU-4, METEOR, ROUGE-L, and CIDEr are computed against all reference answers.
Loading 2608.01185v1…