Source-linked AI summary
SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering
Wenli Li, Kai Zhao, Haoran Jiang, Enquan Yang, Yi Su, Dan Zeng
TL;DR
Multi-view VLM-based 3D QA suffers from redundant visual tokens, while existing reduction methods may not preserve both critical objects and spatial coverage. SeGPruner combines attention-based saliency selection with geometry-guided diversification, achieving efficient inference and competitive 3D reasoning performance on ScanQA and OpenEQA.
Problem
Multi-view aggregation creates severe visual-token redundancy, while existing pruners may fail to retain semantically critical objects and sufficient spatial coverage for 3D reasoning.
Method
SeGPruner is a training-free framework that preserves salient tokens using attention importance and selects complementary tokens using semantic similarity and 3D geometric distance.
Results
SeGPruner achieves state-of-the-art performance on ScanQA and OpenEQA, retaining 23% of original visual tokens on ScanQA while outperforming the full-token base model.
Takeaways & Limitations
SeGPruner provides a strong accuracy–efficiency trade-off, reducing visual-token redundancy while maintaining broad scene coverage for 3D reasoning under constrained budgets.
Takeaways & Limitations
Most existing token reduction methods operate purely in 2D and therefore cannot exploit 3D spatial information in 3D QA.
Abstract
from arXiv · showhide
Vision-language models (VLMs) have been widely adopted for 3D question answering (3D QA). In typical pipelines, visual tokens extracted from multiple viewpoints are concatenated with language tokens and jointly processed by a large language model (LLM) for inference. However, aggregating multi-view observations inevitably introduces severe token redundancy, leading to an overly large visual token set that significantly hinders inference efficiency under constrained token budgets. Visual token pruning has emerged as a prevalent strategy to address this issue. Nevertheless, most existing pruners are primarily tailored to 2D inputs or rely on indirect geometric cues, which limits their ability to explicitly retain semantically critical objects and maintain sufficient spatial coverage for robust 3D reasoning. In this paper, we propose SeGPruner, a semantic-aware and geometry-guided token reduction framework for efficient 3D QA with multi-view images. Specifically, SeGPruner first preserves semantically salient tokens through an attention-based importance module (Saliency-aware Token Selector), ensuring that object-critical evidence is retained. It then complements these tokens with spatially diverse ones via a geometry-guided selector (Geometry-aware Token Diversifier), which jointly considers semantic relevance and 3D geometric distance. This cooperation between saliency preservation and geometry-guided diversification balances object-level evidence and global scene coverage under aggressive token reduction. Extensive experiments on ScanQA and OpenEQA demonstrate that SeGPruner substantially improves inference efficiency, reducing the visual token budget by 91% and inference latency by 86%, while maintaining competitive performance in 3D reasoning tasks.
1 Introduction
3D QA requires both semantic understanding and spatial reasoning, but multi-view VLM pipelines create redundant visual tokens. SeGPruner addresses this by preserving salient object evidence and adding spatially diverse tokens for efficient 3D reasoning.
- Motivation: 3D QA combines language queries with 3D scenes and requires semantic understanding alongside geometric relationship reasoning.The task supports applications including embodied intelligence, robotic navigation, interaction, and autonomous driving.
- Motivation: Multi-view images enable 3D QA with pretrained 2D VLMs, but aggregating viewpoints produces substantial visual-token redundancy.Multi-view images are readily captured in practice, motivating efficient use of pretrained 2D VLMs.
- Motivation: Existing reduction methods may overlook 3D geometry or fail to preserve salient tokens and diverse coverage, risking critical-information loss.Image retrieval reduces views without explicitly leveraging 3D geometry, while pruning and merging methods may inadequately cover key regions.
- Proposed Framework: SeGPruner combines semantic-aware token reduction with geometry-guided selection to reduce multi-view redundancy while preserving essential cues and spatial diversity.Its Saliency-aware Token Selector retains tokens for principal objects using attention-based importance, while the Geometry-aware Token Diversifier complements them with spatially diverse tokens.
- Proposed Framework: The Saliency-aware Token Selector retains tokens associated with semantically critical objects for object-centric 3D reasoning.It estimates token importance from attention scores.
- Proposed Framework: The Geometry-aware Token Diversifier combines semantic similarity with 3D distance to ensure broad spatial coverage under aggressive reduction.The selector uses a joint semantic-spatial criterion after projecting candidate tokens into a unified 3D coordinate space.
- Results: 23% of the original visual tokens were retained on ScanQA while outperforming the full-token base model.The experiments report state-of-the-art performance on both ScanQA and OpenEQA.
2 Related Work
Related work progresses from explicit 3D representations and two-stage descriptions toward multi-view images processed by pretrained 2D VLMs. Token reduction improves efficiency, but predominantly 2D methods cannot exploit the spatial structure central to 3D QA.
- 3D QA Representations: Early 3D QA methods used explicit 3D representations such as point clouds, with later approaches combining point clouds and multi-view inputs.
- Multi-view 3D QA: Multi-view images can be fed directly into existing 2D VLMs, benefiting from large-scale 2D pretraining without requiring explicit 3D representations.Earlier two-stage methods generated captions or scene descriptions before LLM reasoning, causing significant visual information loss.
- Token Reduction: Multi-view visual tokens redundantly represent backgrounds, flat surfaces, repetitive textures, and similar objects across viewpoints.This redundancy creates many visual tokens with low effective information density.
- Token Reduction: Training-free token reduction methods operate at inference time without modifying model parameters and include pruning and merging approaches.Pruning commonly uses attention or importance estimation as a proxy for token relevance.
- Token Reduction: Most existing token reduction methods operate purely in 2D, so they cannot exploit 3D spatial information in 3D QA.The resulting inability to model cross-view redundancy can produce suboptimal reasoning performance.
- Geometry-aware Reduction: Geometry-aware approaches use camera parameters, depth, or camera information, but image retrieval can retain token overlap and require task-specific view-selection modules.These requirements limit applicability to off-the-shelf pretrained VLMs.
3 Methodology
SeGPruner reduces multi-view visual-token redundancy through two cooperative stages: salient-token preservation and geometry-guided diverse-token selection. It constructs a unified 3D representation, selects semantically important tokens, adds complementary spatially diverse tokens, and feeds their ordered union to the LLM.
- Overall framework: SeGPruner decomposes token reduction into salient-token preservation and geometry-guided diverse-token selection.The two stages target semantically critical objects and broad spatial coverage, respectively.
- 3D-Aware Feature Construction: Multi-view visual features are back-projected into a unified 3D coordinate space using depth maps and camera poses.Patch-level coordinates provide spatial cues without requiring dense point-level representations.
- Selecting Salient Tokens: Attention-based importance scores rank visual tokens, after which the top ⌊rM⌋ tokens are retained as salient tokens.For encoders without a [CLS] token, importance is computed by averaging each token’s received attention from the final self-attention matrix.
- Selecting Diverse Tokens: The Geometry-aware Token Diversifier samples remaining tokens using a metric that combines normalized 3D distance with semantic similarity.It initializes with the highest-attention remaining token and iteratively selects the farthest candidate by minimum semantic-spatial distance.
- Inference Procedure: The selected important and diverse tokens replace the original visual sequence and are reordered before being fed to the LLM.The reduced sequence preserves the original token ordering expected by the language model for cross-modal reasoning.
4.1 Implementation Details
The experiments use a frozen LLaVA-OneVision-7B model with uniformly sampled multi-view images, while token selection balances spatial distance and semantic relevance.
- LLaVA-OneVision-7B remains frozen throughout all experiments.
- Each scene uses 12 RGB images sampled uniformly from different viewpoints.Images are resized to 384×384 before SigLIP feature extraction.
- 8,748 visual tokens are extracted per scene before token selection.
- The balancing parameter λ is fixed at 0.5 to trade off spatial distance and semantic relevance.The importance ratio is adjusted according to the retention ratio to balance important and diverse tokens.
- Evaluation covers ScanQA, built from ScanNet with approximately 8,000 scenes and over 41,000 question–answer pairs.The benchmark evaluates object localization, relational reasoning, and cross-view consistency.
4.2 Baseline
The baseline comparison spans explicit-3D QA models, video-based VLMs, and task-specialized 3D-LMMs, while token-pruning methods are compared under identical settings.
- ScanQA comparisons include task-specific 3D-QA models using explicit 3D information such as RGB-D-reconstructed point clouds.
- Video-based VLMs form a second comparison category, processing multi-frame inputs to implicitly capture 3D cues through temporal modeling.
- Task-specialized 3D-LMMs form a third category by adapting large vision-language models to 3D question answering.
- LLaVA-OneVision achieves competitive performance across ScanQA and OpenEQA without explicit 3D inputs or additional auxiliary information.It is therefore used as the fixed base VLM for token-reduction comparisons.
- DTC and VisPruner are compared with SeGPruner using the same base model and identical evaluation settings.
4.3 Evaluation Metrics
ScanQA uses exact-match accuracy, whereas OpenEQA uses LLM-based semantic matching under its official evaluation protocol.
- ScanQA uses Exact Match (EM@1) as its primary evaluation metric.EM@1 measures the percentage of predictions exactly matching ground-truth answers.
- OpenEQA uses LLM-Match to assess semantic consistency between predicted and reference answers.The metric is intended to better reflect semantic correspondence in open-vocabulary scenarios.
- OpenEQA evaluation follows the official protocol with the provided prompt and GPT-4 checkpoint gpt-4-1106-preview.
4.4 Experimental Results and Discussion
SeGPruner remains competitive under aggressive token reduction on both ScanQA and OpenEQA, where preserving semantic saliency and spatial diversity supports scene coverage.
- At 9% retention on ScanQA, SeGPruner preserves 95.3% of the original model performance.The method remains particularly robust under extreme reduction.
- The results associate spatially and semantically diverse token preservation with sufficient scene coverage for reliable 3D reasoning under severe reduction.
- At 8% retention on OpenEQA, SeGPruner outperforms VisPruner by 2.6 points and DTC by 0.6 points.
- At moderate OpenEQA retention ratios of 56% and 43%, SeGPruner and VisPruner achieve performance comparable to the base model.
4.5 Ablation Studies
Ablations show that saliency preservation and spatial diversity are complementary: each alone addresses one failure mode, while their combination supports robust reasoning and lower latency under token reduction.
- Qualitative ablation: Saliency-only selection preserves salient objects but can miss fine-grained details, whereas diversity-only selection improves spatial coverage but can overlook important tabletop regions.
- Qualitative ablation: The combined strategy retains salient objects, fine-grained structures, and more continuous large-object representations than either component alone.
- Overall ablation: At 23% token retention on ScanQA, combining both components improves EM@1 by 0.4 over the base model.
- Overall ablation: On OpenEQA, combining saliency and geometry retains approximately 98% of base-model performance under token reduction.
- Inference efficiency: SeGPruner achieves lower per-example latency than VisPruner at every token retention ratio.
- Diversity-aware selection: Geometry-aware diversification consistently outperforms semantic-similarity sampling across retention ratios, showing the value of spatial cues under severe reduction.
4.6 Qualitative Results.
Qualitative comparisons indicate that SeGPruner retains meaningful objects and fine-grained details while distributing tokens more completely across image and 3D spaces.
- Qualitative comparison: SeGPruner preserves meaningful objects and fine-grained details while avoiding excessive token allocation to large textureless regions.
- Qualitative comparison: Compared with VisPruner, SeGPruner maintains more even image-plane coverage and more complete 3D object structures.
- Qualitative comparison: The visual differences support more accurate object localization and spatial reasoning, including correct answers where VisPruner fails.
5 Conclusion
The conclusion presents SeGPruner as a training-free module that combines semantic selection with 3D geometric diversification for efficient multi-view 3D QA. Experiments support an accuracy–efficiency trade-off under constrained token budgets.
- Conclusion: SeGPruner is a training-free token reduction module for efficient multi-view 3D QA with off-the-shelf VLMs.
- Conclusion: The method first retains semantically relevant tokens, then selects spatially diverse tokens using 3D geometric cues.
- Conclusion: Experiments on ScanQA and OpenEQA show substantially fewer visual tokens and faster inference while maintaining competitive performance.
- Conclusion: Under aggressive token budgets, SeGPruner preserves broad scene coverage for 3D reasoning with a constrained inference budget.